Question: Following on from the Kubelet should accept YAML pod specifications. Currnelty it supports only JSON pod specifications.

<ins> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package config import ( "reflect" "testing" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/testapi" "github.com/ghodss/yaml" ) func noDefault(*api.Pod) error { return nil } func TestDecodeSinglePod(t *testing.T) { pod := &api.Pod{ TypeMeta: api.TypeMeta{ APIVersion: "", }, ObjectMeta: api.ObjectMeta{ Name:      "test", UID:       "12345", Namespace: "mynamespace", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSClusterFirst, Containers: []api.Container{{ Name:                   "image", Image:                  "test/image", ImagePullPolicy:        "IfNotPresent", TerminationMessagePath: "/dev/termination-log", }}, }, } json, err := testapi.Codec().Encode(pod) if err != nil { t.Errorf("unexpected error: %v", err) } parsed, podOut, err := tryDecodeSinglePod(json, noDefault) if testapi.Version() == "v1beta1" { // v1beta1 conversion leaves empty lists that should be nil podOut.Spec.Containers[0].Resources.Limits = nil podOut.Spec.Containers[0].Resources.Requests = nil } if !parsed { t.Errorf("expected to have parsed file: (%s)", string(json)) } if err != nil { t.Errorf("unexpected error: %v (%s)", err, string(json)) } if !reflect.DeepEqual(pod, podOut) { t.Errorf("expected:n%#vngot:n%#vn%s", pod, podOut, string(json)) } externalPod, err := testapi.Converter().ConvertToVersion(pod, "v1beta3") if err != nil { t.Errorf("unexpected error: %v", err) } yaml, err := yaml.Marshal(externalPod) if err != nil { t.Errorf("unexpected error: %v", err) } parsed, podOut, err = tryDecodeSinglePod(yaml, noDefault) if !parsed { t.Errorf("expected to have parsed file: (%s)", string(yaml)) } if err != nil { t.Errorf("unexpected error: %v (%s)", err, string(yaml)) } if !reflect.DeepEqual(pod, podOut) { t.Errorf("expected:n%#vngot:n%#vn%s", pod, podOut, string(yaml)) } } func TestDecodePodList(t *testing.T) { pod := &api.Pod{ TypeMeta: api.TypeMeta{ APIVersion: "", }, ObjectMeta: api.ObjectMeta{ Name:      "test", UID:       "12345", Namespace: "mynamespace", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSClusterFirst, Containers: []api.Container{{ Name:                   "image", Image:                  "test/image", ImagePullPolicy:        "IfNotPresent", TerminationMessagePath: "/dev/termination-log", }}, }, } podList := &api.PodList{ Items: []api.Pod{*pod}, } json, err := testapi.Codec().Encode(podList) if err != nil { t.Errorf("unexpected error: %v", err) } parsed, podListOut, err := tryDecodePodList(json, noDefault) if testapi.Version() == "v1beta1" { // v1beta1 conversion leaves empty lists that should be nil podListOut.Items[0].Spec.Containers[0].Resources.Limits = nil podListOut.Items[0].Spec.Containers[0].Resources.Requests = nil } if !parsed { t.Errorf("expected to have parsed file: (%s)", string(json)) } if err != nil { t.Errorf("unexpected error: %v (%s)", err, string(json)) } if !reflect.DeepEqual(podList, &podListOut) { t.Errorf("expected:n%#vngot:n%#vn%s", podList, &podListOut, string(json)) } externalPodList, err := testapi.Converter().ConvertToVersion(podList, "v1beta3") if err != nil { t.Errorf("unexpected error: %v", err) } yaml, err := yaml.Marshal(externalPodList) if err != nil { t.Errorf("unexpected error: %v", err) } parsed, podListOut, err = tryDecodePodList(yaml, noDefault) if !parsed { t.Errorf("expected to have parsed file: (%s)", string(yaml)) } if err != nil { t.Errorf("unexpected error: %v (%s)", err, string(yaml)) } if !reflect.DeepEqual(podList, &podListOut) { t.Errorf("expected:n%#vngot:n%#vn%s", pod, &podListOut, string(yaml)) } } </ins>


Question:
As the title suggests, I can find no documentation on KUBEADMISSIONCONTROL (from the apiserver configuration), and I don't know golang well enough to determine the answer (yet) from the source code. Google search yields only 3 hits today, and all of them are copies of the configuration file itself with no commentary. As an admin, I'd be grateful for simple documentation on all the variables, their possible settings, and what those really mean. If that already exists, please identify where it exists. Though my formal issue is the documentation, the problem that leads me here is that my pods can't query the apiserver. Given the following, my connection is reset: Any assistance would be greatly appreciated, though my formal issue remains the lack of thorough documentation on the configuration. I'll continue learning golang in order to try to figure out what my problem may be.
I agree a document is needed, will look to put something together describing what each option does.
Thanks! :)
That being said, I recommend we scope this issue to just documentation being needed, and if you need assistance with your other topic, treat it as a separate issue.
Quite reasonable! If you think that documentation will be before the weekend, I'll wait until that's available and see if I can figure it out myself; otherwise, I'll make a separate issue for that.

<ins> # Admission Controllers ## What are they? An admission control plug-in is a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object.  The plug-in code is in the API server process and must be compiled into the binary in order to be used at this time. Each admission control plug-in is run in sequence before a request is accepted into the cluster.  If any of the plug-ins in the sequence reject the request, the entire request is rejected immediately and an error is returned to the end-user. Admission control plug-ins may mutate the incoming object in some cases to apply system configured defaults.  In addition, admission control plug-ins may mutate related resources as part of request processing to do things like increment quota usage. ## Why do I need them? Many advanced features in Kubernetes require an admission control plug-in to be enabled in order to properly support the feature.  As a result, a Kubernetes API server that is not properly configured with the right set of admission control plug-ins is an incomplete server and will not support all the features you expect. ## How do I turn on an admission control plug-in? The Kubernetes API server supports a flag, ```admission_control``` that takes a comma-delimited, ordered list of admission control choices to invoke prior to modifying objects in the cluster. ## What does each plug-in do? ### AlwaysAdmit This plug-in will accept all incoming requests made to the Kubernetes API server. ### AlwaysDeny This plug-in will reject all mutating requests made to the Kubernetes API server.  It's largely intended for testing purposes and is not recommended for usage in a real deployment. ### DenyExecOnPrivileged This plug-in will intercept all requests to exec a command in a pod if that pod has a privileged container. If your cluster supports privileged containers, and you want to restrict the ability of end-users to exec commands in those containers, we strongly encourage enabling this plug-in. ### ServiceAccount This plug-in limits admission of Pod creation requests based on the Pod's ```ServiceAccount```. 1. If the pod does not have a ```ServiceAccount```, it modifies the pod's ```ServiceAccount``` to "default". 2. It ensures that the ```ServiceAccount``` referenced by a pod exists. 3. If ```LimitSecretReferences``` is true, it rejects the pod if the pod references ```Secret``` objects which the pods ```ServiceAccount``` does not reference. 4. If the pod does not contain any ```ImagePullSecrets```, the ```ImagePullSecrets``` of the ```ServiceAccount``` are added to the pod. 5. If ```MountServiceAccountToken``` is true, it adds a ```VolumeMount``` with the pod's ```ServiceAccount``` API token secret to containers in the pod. We strongly recommend using this plug-in if you intend to make use of Kubernetes ```ServiceAccount``` objects. ### SecurityContextDeny This plug-in will deny any ```SecurityContext``` that defines options that were not available on the ```Container```. ### ResourceQuota This plug-in will observe the incoming request and ensure that it does not violate any of the constraints enumerated in the ```ResourceQuota``` object in a ```Namespace```.  If you are using ```ResourceQuota``` objects in your Kubernetes deployment, you MUST use this plug-in to enforce quota constraints. It is strongly encouraged that this plug-in is configured last in the sequence of admission control plug-ins.  This is so that quota is not prematurely incremented only for the request to be rejected later in admission control. ### LimitRanger This plug-in will observe the incoming request and ensure that it does not violate any of the constraints enumerated in the ```LimitRange``` object in a ```Namespace```.  If you are using ```LimitRange``` objects in your Kubernetes deployment, you MUST use this plug-in to enforce those constraints. ### NamespaceExists This plug-in will observe all incoming requests that attempt to create a resource in a Kubernetes ```Namespace``` and reject the request if the ```Namespace``` was not previously created.  We strongly recommend running this plug-in to ensure integrity of your data. ### NamespaceAutoProvision (deprecated) This plug-in will observe all incoming requests that attempt to create a resource in a Kubernetes ```Namespace``` and create a new ```Namespace``` if one did not already exist previously. We strongly recommend ```NamespaceExists``` over ```NamespaceAutoProvision```. ### NamespaceLifecycle This plug-in enforces that a ```Namespace``` that is undergoing termination cannot have new content created in it. A ```Namespace``` deletion kicks off a sequence of operations that remove all content (pods, services, etc.) in that namespace.  In order to enforce integrity of that process, we strongly recommend running this plug-in. Once ```NamespaceAutoProvision``` is deprecated, we anticipate ```NamespaceLifecycle``` and ```NamespaceExists``` will be merged into a single plug-in that enforces the life-cycle of a ```Namespace``` in Kubernetes. ## Is there a recommended set of plug-ins to use? Yes. For Kubernetes 1.0, we strongly recommend running the following set of admission control plug-ins: ```shell --admission_control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota ``` </ins>


Question:
We seem to have an occasional consistency issue between the client and server versions on our Jenkins GCE e2e project. Build 6295 failed this morning because the client version was newer than the server version: The server version matches the build was was immediately running in 6294, which was aborted early. I wonder if (somehow) the cleanup didn't work properly? (Which is a bit worrying if so.) Here's the version from 6294: A short while later, build 6297 failed, this time with the client version older than the server version: This one makes a little bit more sense: there was a new build published between downloading the client/test tarballs and setting up the cluster. I'm guessing that somewhere in the cluster setup scripts the latest CI release file was read again, rather than using the version read for the client/test tarballs. (I haven't yet worked out where exactly it's read, but it should be an easier fix.) cc
A side issue: we should probably figure out why the build git client is being marked dirty. Since we're checking out the head version from GitHub, it should ideally be a clean client.
The -dirty bit is now fixed by .
This is caused by different Jenkins jobs uploading to the same GCS bucket and clobbering each other. In we see: Over in
This also explains the issues seen in ; the job is also running in the same project.
Ah, good sleuthing. We avoided this issue in the PR builds by setting (used ), but maybe we should always be setting to include the Jenkins job name (and optionally executor/node number).
The original idea was to run every e2e job in it's own project, to avoid these sorts of name clashes. But we seem to have regressed and now have many jobs sharing the same projects. Is project-per-job not a better approach? It also helps with resource leaks and the likes. Just a thought.
I'd argue this is actually uncovering a (dev) deployment bug; I should be able to deploy different versions of Kubernetes to the same project. I know you and I disagree (and probably always will) on the strategy here. :smile:
fwiw I think I agree with that if project-per-job avoids this issue and helps identify resource leaks, it's a pragmatic near-term improvement in the state of the world that would result in more benefit than the cost of changing all the everythings to support multi-job-in-project. Re: devs deploying different versions of k8s to the same project, P3 feature request?
I was just about to suggest the same thing. I will file a separate issue, and fix this in the short term an easier way. is out to make it easier to isolate a run to a project.
Also it seems like a reasonable thing (higher than P2) to want to deploy multiple clusters to the same project safely in prod, not just dev. Disagree?
Multiple clusters in prod using well-defined build versions such as v1.1.4 = P0. Multiple clusters in prod using our cooked-up test-building system = non-goal?
SGTM; this is prod code, though, not test code; it's used every time we call , regardless of whether it's a CI build or released version.
FYI: kubernetes-e2e-gce-reboot/ has one reboot test failed due to kube-proxy image change. But that build doesn't include pr yet.
looks like it. That run used version , but the first build with was .

KUBE_PROMPT_FOR_UPDATE=y KUBE_SKIP_UPDATE=${KUBE_SKIP_UPDATE-"n"} <del> # Suffix to append to the staging path used for the server tars. Useful if # multiple versions of the server are being used in the same project # simultaneously (e.g. on Jenkins). KUBE_GCS_STAGING_PATH_SUFFIX=${KUBE_GCS_STAGING_PATH_SUFFIX-""} </del> # How long (in seconds) to wait for cluster initialization. KUBE_CLUSTER_INITIALIZATION_TIMEOUT=${KUBE_CLUSTER_INITIALIZATION_TIMEOUT:-300}


Question:
We seem to have an occasional consistency issue between the client and server versions on our Jenkins GCE e2e project. Build 6295 failed this morning because the client version was newer than the server version: The server version matches the build was was immediately running in 6294, which was aborted early. I wonder if (somehow) the cleanup didn't work properly? (Which is a bit worrying if so.) Here's the version from 6294: A short while later, build 6297 failed, this time with the client version older than the server version: This one makes a little bit more sense: there was a new build published between downloading the client/test tarballs and setting up the cluster. I'm guessing that somewhere in the cluster setup scripts the latest CI release file was read again, rather than using the version read for the client/test tarballs. (I haven't yet worked out where exactly it's read, but it should be an easier fix.) cc
A side issue: we should probably figure out why the build git client is being marked dirty. Since we're checking out the head version from GitHub, it should ideally be a clean client.
The -dirty bit is now fixed by .
This is caused by different Jenkins jobs uploading to the same GCS bucket and clobbering each other. In we see: Over in
This also explains the issues seen in ; the job is also running in the same project.
Ah, good sleuthing. We avoided this issue in the PR builds by setting (used ), but maybe we should always be setting to include the Jenkins job name (and optionally executor/node number).
The original idea was to run every e2e job in it's own project, to avoid these sorts of name clashes. But we seem to have regressed and now have many jobs sharing the same projects. Is project-per-job not a better approach? It also helps with resource leaks and the likes. Just a thought.
I'd argue this is actually uncovering a (dev) deployment bug; I should be able to deploy different versions of Kubernetes to the same project. I know you and I disagree (and probably always will) on the strategy here. :smile:
fwiw I think I agree with that if project-per-job avoids this issue and helps identify resource leaks, it's a pragmatic near-term improvement in the state of the world that would result in more benefit than the cost of changing all the everythings to support multi-job-in-project. Re: devs deploying different versions of k8s to the same project, P3 feature request?
I was just about to suggest the same thing. I will file a separate issue, and fix this in the short term an easier way. is out to make it easier to isolate a run to a project.
Also it seems like a reasonable thing (higher than P2) to want to deploy multiple clusters to the same project safely in prod, not just dev. Disagree?
Multiple clusters in prod using well-defined build versions such as v1.1.4 = P0. Multiple clusters in prod using our cooked-up test-building system = non-goal?
SGTM; this is prod code, though, not test code; it's used every time we call , regardless of whether it's a CI build or released version.
FYI: kubernetes-e2e-gce-reboot/ has one reboot test failed due to kube-proxy image change. But that build doesn't include pr yet.
looks like it. That run used version , but the first build with was .

gsutil mb "${staging_bucket}" fi <del> local -r staging_path="${staging_bucket}/devel${KUBE_GCS_STAGING_PATH_SUFFIX}" </del> <ins> local -r staging_path="${staging_bucket}/${INSTANCE_PREFIX}-devel" </ins> SERVER_BINARY_TAR_HASH=$(sha1sum-file "${SERVER_BINARY_TAR}") SALT_TAR_HASH=$(sha1sum-file "${SALT_TAR}")


Question:
We seem to have an occasional consistency issue between the client and server versions on our Jenkins GCE e2e project. Build 6295 failed this morning because the client version was newer than the server version: The server version matches the build was was immediately running in 6294, which was aborted early. I wonder if (somehow) the cleanup didn't work properly? (Which is a bit worrying if so.) Here's the version from 6294: A short while later, build 6297 failed, this time with the client version older than the server version: This one makes a little bit more sense: there was a new build published between downloading the client/test tarballs and setting up the cluster. I'm guessing that somewhere in the cluster setup scripts the latest CI release file was read again, rather than using the version read for the client/test tarballs. (I haven't yet worked out where exactly it's read, but it should be an easier fix.) cc
A side issue: we should probably figure out why the build git client is being marked dirty. Since we're checking out the head version from GitHub, it should ideally be a clean client.
The -dirty bit is now fixed by .
This is caused by different Jenkins jobs uploading to the same GCS bucket and clobbering each other. In we see: Over in
This also explains the issues seen in ; the job is also running in the same project.
Ah, good sleuthing. We avoided this issue in the PR builds by setting (used ), but maybe we should always be setting to include the Jenkins job name (and optionally executor/node number).
The original idea was to run every e2e job in it's own project, to avoid these sorts of name clashes. But we seem to have regressed and now have many jobs sharing the same projects. Is project-per-job not a better approach? It also helps with resource leaks and the likes. Just a thought.
I'd argue this is actually uncovering a (dev) deployment bug; I should be able to deploy different versions of Kubernetes to the same project. I know you and I disagree (and probably always will) on the strategy here. :smile:
fwiw I think I agree with that if project-per-job avoids this issue and helps identify resource leaks, it's a pragmatic near-term improvement in the state of the world that would result in more benefit than the cost of changing all the everythings to support multi-job-in-project. Re: devs deploying different versions of k8s to the same project, P3 feature request?
I was just about to suggest the same thing. I will file a separate issue, and fix this in the short term an easier way. is out to make it easier to isolate a run to a project.
Also it seems like a reasonable thing (higher than P2) to want to deploy multiple clusters to the same project safely in prod, not just dev. Disagree?
Multiple clusters in prod using well-defined build versions such as v1.1.4 = P0. Multiple clusters in prod using our cooked-up test-building system = non-goal?
SGTM; this is prod code, though, not test code; it's used every time we call , regardless of whether it's a CI build or released version.
FYI: kubernetes-e2e-gce-reboot/ has one reboot test failed due to kube-proxy image change. But that build doesn't include pr yet.
looks like it. That run used version , but the first build with was .

${GCE_SLOW_TESTS[@]:+${GCE_SLOW_TESTS[@]}}  )"} : ${KUBE_GCE_INSTANCE_PREFIX:="e2e-gce-${NODE_NAME}-${EXECUTOR_NUMBER}"} <del> : ${KUBE_GCS_STAGING_PATH_SUFFIX:="-${NODE_NAME}-${EXECUTOR_NUMBER}"} </del> : ${PROJECT:="kubernetes-jenkins-pull"} : ${ENABLE_DEPLOYMENTS:=true} # Override GCE defaults


Question:
We seem to have an occasional consistency issue between the client and server versions on our Jenkins GCE e2e project. Build 6295 failed this morning because the client version was newer than the server version: The server version matches the build was was immediately running in 6294, which was aborted early. I wonder if (somehow) the cleanup didn't work properly? (Which is a bit worrying if so.) Here's the version from 6294: A short while later, build 6297 failed, this time with the client version older than the server version: This one makes a little bit more sense: there was a new build published between downloading the client/test tarballs and setting up the cluster. I'm guessing that somewhere in the cluster setup scripts the latest CI release file was read again, rather than using the version read for the client/test tarballs. (I haven't yet worked out where exactly it's read, but it should be an easier fix.) cc
A side issue: we should probably figure out why the build git client is being marked dirty. Since we're checking out the head version from GitHub, it should ideally be a clean client.
The -dirty bit is now fixed by .
This is caused by different Jenkins jobs uploading to the same GCS bucket and clobbering each other. In we see: Over in
This also explains the issues seen in ; the job is also running in the same project.
Ah, good sleuthing. We avoided this issue in the PR builds by setting (used ), but maybe we should always be setting to include the Jenkins job name (and optionally executor/node number).
The original idea was to run every e2e job in it's own project, to avoid these sorts of name clashes. But we seem to have regressed and now have many jobs sharing the same projects. Is project-per-job not a better approach? It also helps with resource leaks and the likes. Just a thought.
I'd argue this is actually uncovering a (dev) deployment bug; I should be able to deploy different versions of Kubernetes to the same project. I know you and I disagree (and probably always will) on the strategy here. :smile:
fwiw I think I agree with that if project-per-job avoids this issue and helps identify resource leaks, it's a pragmatic near-term improvement in the state of the world that would result in more benefit than the cost of changing all the everythings to support multi-job-in-project. Re: devs deploying different versions of k8s to the same project, P3 feature request?
I was just about to suggest the same thing. I will file a separate issue, and fix this in the short term an easier way. is out to make it easier to isolate a run to a project.
Also it seems like a reasonable thing (higher than P2) to want to deploy multiple clusters to the same project safely in prod, not just dev. Disagree?
Multiple clusters in prod using well-defined build versions such as v1.1.4 = P0. Multiple clusters in prod using our cooked-up test-building system = non-goal?
SGTM; this is prod code, though, not test code; it's used every time we call , regardless of whether it's a CI build or released version.
FYI: kubernetes-e2e-gce-reboot/ has one reboot test failed due to kube-proxy image change. But that build doesn't include pr yet.
looks like it. That run used version , but the first build with was .

export KUBE_GCE_ZONE=${E2E_ZONE} export KUBE_GCE_NETWORK=${E2E_NETWORK} export KUBE_GCE_INSTANCE_PREFIX=${KUBE_GCE_INSTANCE_PREFIX:-} <del> export KUBE_GCS_STAGING_PATH_SUFFIX=${KUBE_GCS_STAGING_PATH_SUFFIX:-} </del> export KUBE_GCE_NODE_PROJECT=${KUBE_GCE_NODE_PROJECT:-} export KUBE_GCE_NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-} export KUBE_OS_DISTRIBUTION=${KUBE_OS_DISTRIBUTION:-}


Question: We need to be squelching the auth variables on upgrade anyways, but this complaint gets issued (and it proceeds anyways): cc

KUBE_PROXY_TOKEN: $(yaml-quote ${KUBE_PROXY_TOKEN:-}) ADMISSION_CONTROL: $(yaml-quote ${ADMISSION_CONTROL:-}) MASTER_IP_RANGE: $(yaml-quote ${MASTER_IP_RANGE}) <del> CA_CERT: $(yaml-quote ${CA_CERT_BASE64}) </del> <ins> CA_CERT: $(yaml-quote ${CA_CERT_BASE64:-}) </ins> EOF if [[ "${master}" == "true" ]]; then


Question:
In case of cluster crash and status recovery, the persistentVolumeClaimBinder would not re-add available volumes to its internal index. Users would not be able to claim those volumes.
This is really bad. I think we should just nix Pending phase and always consider every Available volume for indexing - map lookups are cheap.

// available volumes await a claim case api.VolumeAvailable: <ins> // TODO:  remove api.VolumePending phase altogether _, exists, err := volumeIndex.Get(volume) if err != nil { return err } if !exists { volumeIndex.Add(volume) } </ins> if volume.Spec.ClaimRef != nil { _, err := binderClient.GetPersistentVolumeClaim(volume.Spec.ClaimRef.Namespace, volume.Spec.ClaimRef.Name) if err == nil {


Question:
In case of cluster crash and status recovery, the persistentVolumeClaimBinder would not re-add available volumes to its internal index. Users would not be able to claim those volumes.
This is really bad. I think we should just nix Pending phase and always consider every Available volume for indexing - map lookups are cheap.

} } <ins> func TestMissingFromIndex(t *testing.T) { api.ForTesting_ReferencesAllowBlankSelfLinks = true o := testclient.NewObjects(api.Scheme, api.Scheme) if err := testclient.AddObjectsFromPath("../../examples/persistent-volumes/claims/claim-01.yaml", o, api.Scheme); err != nil { t.Fatal(err) } if err := testclient.AddObjectsFromPath("../../examples/persistent-volumes/volumes/local-01.yaml", o, api.Scheme); err != nil { t.Fatal(err) } client := &testclient.Fake{ReactFn: testclient.ObjectReaction(o, latest.RESTMapper)} pv, err := client.PersistentVolumes().Get("any") if err != nil { t.Error("Unexpected error getting PV from client: %v", err) } claim, error := client.PersistentVolumeClaims("ns").Get("any") if error != nil { t.Errorf("Unexpected error getting PVC from client: %v", err) } volumeIndex := NewPersistentVolumeOrderedIndex() mockClient := &mockBinderClient{ volume: pv, claim:  claim, } // the default value of the PV is Pending. // if has previously been processed by the binder, it's status in etcd would be Available. // Only Pending volumes were being indexed and made ready for claims. pv.Status.Phase = api.VolumeAvailable // adds the volume to the index, making the volume available syncVolume(volumeIndex, mockClient, pv) if pv.Status.Phase != api.VolumeAvailable { t.Errorf("Expected phase %s but got %s", api.VolumeBound, pv.Status.Phase) } // an initial sync for a claim will bind it to an unbound volume, triggers state change err = syncClaim(volumeIndex, mockClient, claim) if err != nil { t.Fatalf("Expected Clam to be bound, instead got an error: %+vn", err) } // state change causes another syncClaim to update statuses syncClaim(volumeIndex, mockClient, claim) // claim updated volume's status, causing an update and syncVolume call syncVolume(volumeIndex, mockClient, pv) if pv.Spec.ClaimRef == nil { t.Errorf("Expected ClaimRef but got nil for pv.Status.ClaimRef: %+vn", pv) } if pv.Status.Phase != api.VolumeBound { t.Errorf("Expected phase %s but got %s", api.VolumeBound, pv.Status.Phase) } if claim.Status.Phase != api.ClaimBound { t.Errorf("Expected phase %s but got %s", api.ClaimBound, claim.Status.Phase) } if len(claim.Status.AccessModes) != len(pv.Spec.AccessModes) { t.Errorf("Expected phase %s but got %s", api.ClaimBound, claim.Status.Phase) } if claim.Status.AccessModes[0] != pv.Spec.AccessModes[0] { t.Errorf("Expected access mode %s but got %s", claim.Status.AccessModes[0], pv.Spec.AccessModes[0]) } // pretend the user deleted their claim mockClient.claim = nil syncVolume(volumeIndex, mockClient, pv) if pv.Status.Phase != api.VolumeReleased { t.Errorf("Expected phase %s but got %s", api.VolumeReleased, pv.Status.Phase) } } </ins> type mockBinderClient struct { volume *api.PersistentVolume claim  *api.PersistentVolumeClaim


Question:
The "" E2E test will sometimes fail because it fails to delete the PD used in the test after the test is finished. The error: Detailed test log: The disk that fails to delete still shows up in : My attempts to manually try to delete the PD fail with: (the work around for ) doesn't change anything here:
is the disk still mounted ? It looks like the detach call did not succeed for some reason
I'll have to get another repro and verify.
okay cool. If you get a repro check to see if the disk is still mounted. If so then the cleanup code is not being called. If not then the clean up started, detach was called but failed for some reason; look in the logs for something like "googleapi: some error about detaching". It can also just be a race condition. That is, detach was called but is taking too long; if the disk does not detach then you cannot delete it.

<ins> /* Copyright 2015 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package operationmanager import ( "fmt" "sync" ) // Operation Manager is a thread-safe interface for keeping track of multiple pending async operations. type OperationManager interface { // Called when the operation with the given ID has started. // Creates a new channel with specified buffer size tracked with the specified ID. // Returns a read-only version of the newly created channel. // Returns an error if an entry with the specified ID already exists (previous entry must be removed by calling Close). Start(id string, bufferSize uint) (<-chan interface{}, error) // Called when the operation with the given ID has terminated. // Closes and removes the channel associated with ID. // Returns an error if no associated channel exists. Close(id string) error // Attempts to send msg to the channel associated with ID. // Returns an error if no associated channel exists. Send(id string, msg interface{}) error } // Returns a new instance of a channel manager. func NewOperationManager() OperationManager { return &operationManager{ chanMap: make(map[string]chan interface{}), } } type operationManager struct { sync.RWMutex chanMap map[string]chan interface{} } // Called when the operation with the given ID has started. // Creates a new channel with specified buffer size tracked with the specified ID. // Returns a read-only version of the newly created channel. // Returns an error if an entry with the specified ID already exists (previous entry must be removed by calling Close). func (cm *operationManager) Start(id string, bufferSize uint) (<-chan interface{}, error) { cm.Lock() defer cm.Unlock() if _, exists := cm.chanMap[id]; exists { return nil, fmt.Errorf("id %q already exists", id) } cm.chanMap[id] = make(chan interface{}, bufferSize) return cm.chanMap[id], nil } // Called when the operation with the given ID has terminated. // Closes and removes the channel associated with ID. // Returns an error if no associated channel exists. func (cm *operationManager) Close(id string) error { cm.Lock() defer cm.Unlock() if _, exists := cm.chanMap[id]; !exists { return fmt.Errorf("id %q not found", id) } close(cm.chanMap[id]) delete(cm.chanMap, id) return nil } // Attempts to send msg to the channel associated with ID. // Returns an error if no associated channel exists. func (cm *operationManager) Send(id string, msg interface{}) error { cm.RLock() defer cm.RUnlock() if _, exists := cm.chanMap[id]; !exists { return fmt.Errorf("id %q not found", id) } cm.chanMap[id] <- msg return nil } </ins>


Question:
The "" E2E test will sometimes fail because it fails to delete the PD used in the test after the test is finished. The error: Detailed test log: The disk that fails to delete still shows up in : My attempts to manually try to delete the PD fail with: (the work around for ) doesn't change anything here:
is the disk still mounted ? It looks like the detach call did not succeed for some reason
I'll have to get another repro and verify.
okay cool. If you get a repro check to see if the disk is still mounted. If so then the cleanup code is not being called. If not then the clean up started, detach was called but failed for some reason; look in the logs for something like "googleapi: some error about detaching". It can also just be a race condition. That is, detach was called but is taking too long; if the disk does not detach then you cannot delete it.

<ins> /* Copyright 2015 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ // Channel Manager keeps track of multiple channels package operationmanager import ( "testing" ) func TestStart(t *testing.T) { // Arrange cm := NewOperationManager() chanId := "testChanId" testMsg := "test message" // Act ch, startErr := cm.Start(chanId, 1 /* bufferSize */) sigErr := cm.Send(chanId, testMsg) // Assert if startErr != nil { t.Fatalf("Unexpected error on Start. Expected: <no error> Actual: <%v>", startErr) } if sigErr != nil { t.Fatalf("Unexpected error on Send. Expected: <no error> Actual: <%v>", sigErr) } if actual := <-ch; actual != testMsg { t.Fatalf("Unexpected testMsg value. Expected: <%v> Actual: <%v>", testMsg, actual) } } func TestStartIdExists(t *testing.T) { // Arrange cm := NewOperationManager() chanId := "testChanId" // Act _, startErr1 := cm.Start(chanId, 1 /* bufferSize */) _, startErr2 := cm.Start(chanId, 1 /* bufferSize */) // Assert if startErr1 != nil { t.Fatalf("Unexpected error on Start1. Expected: <no error> Actual: <%v>", startErr1) } if startErr2 == nil { t.Fatalf("Expected error on Start2. Expected: <id already exists error> Actual: <no error>") } } func TestStartAndAdd2Chans(t *testing.T) { // Arrange cm := NewOperationManager() chanId1 := "testChanId1" chanId2 := "testChanId2" testMsg1 := "test message 1" testMsg2 := "test message 2" // Act ch1, startErr1 := cm.Start(chanId1, 1 /* bufferSize */) ch2, startErr2 := cm.Start(chanId2, 1 /* bufferSize */) sigErr1 := cm.Send(chanId1, testMsg1) sigErr2 := cm.Send(chanId2, testMsg2) // Assert if startErr1 != nil { t.Fatalf("Unexpected error on Start1. Expected: <no error> Actual: <%v>", startErr1) } if startErr2 != nil { t.Fatalf("Unexpected error on Start2. Expected: <no error> Actual: <%v>", startErr2) } if sigErr1 != nil { t.Fatalf("Unexpected error on Send1. Expected: <no error> Actual: <%v>", sigErr1) } if sigErr2 != nil { t.Fatalf("Unexpected error on Send2. Expected: <no error> Actual: <%v>", sigErr2) } if actual := <-ch1; actual != testMsg1 { t.Fatalf("Unexpected testMsg value. Expected: <%v> Actual: <%v>", testMsg1, actual) } if actual := <-ch2; actual != testMsg2 { t.Fatalf("Unexpected testMsg value. Expected: <%v> Actual: <%v>", testMsg2, actual) } } func TestStartAndAdd2ChansAndClose(t *testing.T) { // Arrange cm := NewOperationManager() chanId1 := "testChanId1" chanId2 := "testChanId2" testMsg1 := "test message 1" testMsg2 := "test message 2" // Act ch1, startErr1 := cm.Start(chanId1, 1 /* bufferSize */) ch2, startErr2 := cm.Start(chanId2, 1 /* bufferSize */) sigErr1 := cm.Send(chanId1, testMsg1) sigErr2 := cm.Send(chanId2, testMsg2) cm.Close(chanId1) sigErr3 := cm.Send(chanId1, testMsg1) // Assert if startErr1 != nil { t.Fatalf("Unexpected error on Start1. Expected: <no error> Actual: <%v>", startErr1) } if startErr2 != nil { t.Fatalf("Unexpected error on Start2. Expected: <no error> Actual: <%v>", startErr2) } if sigErr1 != nil { t.Fatalf("Unexpected error on Send1. Expected: <no error> Actual: <%v>", sigErr1) } if sigErr2 != nil { t.Fatalf("Unexpected error on Send2. Expected: <no error> Actual: <%v>", sigErr2) } if sigErr3 == nil { t.Fatalf("Expected error on Send3. Expected: <error> Actual: <no error>", sigErr2) } if actual := <-ch1; actual != testMsg1 { t.Fatalf("Unexpected testMsg value. Expected: <%v> Actual: <%v>", testMsg1, actual) } if actual := <-ch2; actual != testMsg2 { t.Fatalf("Unexpected testMsg value. Expected: <%v> Actual: <%v>", testMsg2, actual) } } </ins>


Question:
The "" E2E test will sometimes fail because it fails to delete the PD used in the test after the test is finished. The error: Detailed test log: The disk that fails to delete still shows up in : My attempts to manually try to delete the PD fail with: (the work around for ) doesn't change anything here:
is the disk still mounted ? It looks like the detach call did not succeed for some reason
I'll have to get another repro and verify.
okay cool. If you get a repro check to see if the disk is still mounted. If so then the cleanup code is not being called. If not then the clean up started, detach was called but failed for some reason; look in the logs for something like "googleapi: some error about detaching". It can also just be a race condition. That is, detach was called but is taking too long; if the disk does not detach then you cannot delete it.

package gce_pd import ( <del> "errors" </del> "fmt" "os" "path" <ins> "path/filepath" "strings" </ins> "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/cloudprovider" "github.com/GoogleCloudPlatform/kubernetes/pkg/cloudprovider/gce" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util/exec" "github.com/GoogleCloudPlatform/kubernetes/pkg/util/mount" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util/operationmanager" </ins> "github.com/golang/glog" ) <ins> const ( diskByIdPath         = "/dev/disk/by-id/" diskGooglePrefix     = "google-" diskScsiGooglePrefix = "scsi-0Google_PersistentDisk_" diskPartitionSuffix  = "-part" diskSDPath           = "/dev/sd" diskSDPattern        = "/dev/sd*" maxChecks            = 10 maxRetries           = 10 checkSleepDuration   = time.Second ) // Singleton operation manager for managing detach clean up go routines var detachCleanupManager = operationmanager.NewOperationManager() </ins> type GCEDiskUtil struct{} // Attaches a disk specified by a volume.GCEPersistentDisk to the current kubelet. // Mounts the disk to it's global path. <del> func (util *GCEDiskUtil) AttachAndMountDisk(pd *gcePersistentDisk, globalPDPath string) error { </del> <ins> func (diskUtil *GCEDiskUtil) AttachAndMountDisk(pd *gcePersistentDisk, globalPDPath string) error { glog.V(5).Infof("AttachAndMountDisk(pd, %q) where pd is %#vrn", globalPDPath, pd) // Terminate any in progress verify detach go routines, this will block until the goroutine is ready to exit because the channel is unbuffered detachCleanupManager.Send(pd.pdName, true) sdBefore, err := filepath.Glob(diskSDPattern) if err != nil { glog.Errorf("Error filepath.Glob("%s"): %vrn", diskSDPattern, err) } sdBeforeSet := util.NewStringSet(sdBefore...) </ins> gce, err := cloudprovider.GetCloudProvider("gce", nil) if err != nil { return err } if err := gce.(*gce_cloud.GCECloud).AttachDisk(pd.pdName, pd.readOnly); err != nil { return err } <del> devicePaths := []string{ path.Join("/dev/disk/by-id/", "google-"+pd.pdName), path.Join("/dev/disk/by-id/", "scsi-0Google_PersistentDisk_"+pd.pdName), } if pd.partition != "" { for i, path := range devicePaths { devicePaths[i] = path + "-part" + pd.partition } } //TODO(jonesdl) There should probably be better method than busy-waiting here. numTries := 0 devicePath := "" // Wait for the disk device to be created for { for _, path := range devicePaths { _, err := os.Stat(path) if err == nil { devicePath = path break } if err != nil && !os.IsNotExist(err) { return err } } if devicePath != "" { break } numTries++ if numTries == 10 { return errors.New("Could not attach disk: Timeout after 10s") } time.Sleep(time.Second) </del> <ins> devicePath, err := verifyAttached(pd, sdBeforeSet, gce) if err != nil { return err </ins> } // Only mount the PD globally once.


Question:
The "" E2E test will sometimes fail because it fails to delete the PD used in the test after the test is finished. The error: Detailed test log: The disk that fails to delete still shows up in : My attempts to manually try to delete the PD fail with: (the work around for ) doesn't change anything here:
is the disk still mounted ? It looks like the detach call did not succeed for some reason
I'll have to get another repro and verify.
okay cool. If you get a repro check to see if the disk is still mounted. If so then the cleanup code is not being called. If not then the clean up started, detach was called but failed for some reason; look in the logs for something like "googleapi: some error about detaching". It can also just be a race condition. That is, detach was called but is taking too long; if the disk does not detach then you cannot delete it.

func (util *GCEDiskUtil) DetachDisk(pd *gcePersistentDisk) error { // Unmount the global PD mount, which should be the only one. globalPDPath := makeGlobalPDName(pd.plugin.host, pd.pdName) <ins> glog.V(5).Infof("DetachDisk(pd) where pd is %#v and the globalPDPath is %qrn", pd, globalPDPath) // Terminate any in progress verify detach go routines, this will block until the goroutine is ready to exit because the channel is unbuffered detachCleanupManager.Send(pd.pdName, true) </ins> if err := pd.mounter.Unmount(globalPDPath); err != nil { return err }


Question:
The "" E2E test will sometimes fail because it fails to delete the PD used in the test after the test is finished. The error: Detailed test log: The disk that fails to delete still shows up in : My attempts to manually try to delete the PD fail with: (the work around for ) doesn't change anything here:
is the disk still mounted ? It looks like the detach call did not succeed for some reason
I'll have to get another repro and verify.
okay cool. If you get a repro check to see if the disk is still mounted. If so then the cleanup code is not being called. If not then the clean up started, detach was called but failed for some reason; look in the logs for something like "googleapi: some error about detaching". It can also just be a race condition. That is, detach was called but is taking too long; if the disk does not detach then you cannot delete it.

if err := gce.(*gce_cloud.GCECloud).DetachDisk(pd.pdName); err != nil { return err } <ins> // Verify disk detached, retry if needed. go verifyDetached(pd, gce) return nil } // Verifys the disk device to be created has been succesffully attached, and retries if it fails. func verifyAttached(pd *gcePersistentDisk, sdBeforeSet util.StringSet, gce cloudprovider.Interface) (string, error) { devicePaths := getDiskByIdPaths(pd) for numRetries := 0; numRetries < maxRetries; numRetries++ { for numChecks := 0; numChecks < maxChecks; numChecks++ { if err := udevadmChangeToNewDrives(sdBeforeSet); err != nil { // udevadm errors should not block disk attachment, log and continue glog.Errorf("%v", err) } for _, path := range devicePaths { if pathExists, err := pathExists(path); err != nil { return "", err } else if pathExists { // A device path has succesfully been created for the PD glog.V(5).Infof("Succesfully attached GCE PD %q.", pd.pdName) return path, nil } } // Sleep then check again glog.V(5).Infof("Waiting for GCE PD %q to attach.", pd.pdName) time.Sleep(checkSleepDuration) } // Try attaching the disk again glog.Warningf("Timed out waiting for GCE PD %q to attach. Retrying attach.", pd.pdName) if err := gce.(*gce_cloud.GCECloud).AttachDisk(pd.pdName, pd.readOnly); err != nil { return "", err } } return "", fmt.Errorf("Could not attach GCE PD %q. Timeout waiting for mount paths to be created.", pd.pdName) } // Veify the specified persistent disk device has been succesfully detached, and retries if it fails. // This function is intended to be called asynchronously as a go routine. func verifyDetached(pd *gcePersistentDisk, gce cloudprovider.Interface) { defer util.HandleCrash() // Setting bufferSize to 0 so that when senders send, they are blocked until we recieve. This avoids the need to have a separate exit check. ch, err := detachCleanupManager.Start(pd.pdName, 0 /* bufferSize */) if err != nil { glog.Errorf("Error adding %q to detachCleanupManager: %v", pd.pdName, err) return } defer detachCleanupManager.Close(pd.pdName) devicePaths := getDiskByIdPaths(pd) for numRetries := 0; numRetries < maxRetries; numRetries++ { for numChecks := 0; numChecks < maxChecks; numChecks++ { select { case <-ch: glog.Warningf("Terminating GCE PD %q detach verification. Another attach/detach call was made for this PD.", pd.pdName) return default: allPathsRemoved := true for _, path := range devicePaths { if err := udevadmChangeToDrive(path); err != nil { // udevadm errors should not block disk detachment, log and continue glog.Errorf("%v", err) } if exists, err := pathExists(path); err != nil { glog.Errorf("Error check path: %v", err) return } else { allPathsRemoved = allPathsRemoved && !exists } } if allPathsRemoved { // All paths to the PD have been succefully removed glog.V(5).Infof("Succesfully detached GCE PD %q.", pd.pdName) return } // Sleep then check again glog.V(5).Infof("Waiting for GCE PD %q to detach.", pd.pdName) time.Sleep(checkSleepDuration) } } // Try detaching disk again glog.Warningf("Timed out waiting for GCE PD %q to detach. Retrying detach.", pd.pdName) if err := gce.(*gce_cloud.GCECloud).DetachDisk(pd.pdName); err != nil { glog.Errorf("Error on retry detach PD %q: %v", pd.pdName, err) return } } glog.Errorf("Could not detach GCE PD %q. One or more mount paths was not removed.", pd.pdName) } // Returns list of all /dev/disk/by-id/* paths for given PD. func getDiskByIdPaths(pd *gcePersistentDisk) []string { devicePaths := []string{ path.Join(diskByIdPath, diskGooglePrefix+pd.pdName), path.Join(diskByIdPath, diskScsiGooglePrefix+pd.pdName), } if pd.partition != "" { for i, path := range devicePaths { devicePaths[i] = path + diskPartitionSuffix + pd.partition } } return devicePaths } // Checks if the specified path exists func pathExists(path string) (bool, error) { _, err := os.Stat(path) if err == nil { return true, nil } else if os.IsNotExist(err) { return false, nil } else { return false, err } } // Calls "udevadm trigger --action=change" for newly created "/dev/sd*" drives (exist only in after set). // This is workaround for Issue #7972. Once the underlying issue has been resolved, this may be removed. func udevadmChangeToNewDrives(sdBeforeSet util.StringSet) error { sdAfter, err := filepath.Glob(diskSDPattern) if err != nil { return fmt.Errorf("Error filepath.Glob("%s"): %vrn", diskSDPattern, err) } for _, sd := range sdAfter { if !sdBeforeSet.Has(sd) { return udevadmChangeToDrive(sd) } } return nil } // Calls "udevadm trigger --action=change" on the specified drive. // drivePath must be the the block device path to trigger on, in the format "/dev/sd*", or a symlink to it. // This is workaround for Issue #7972. Once the underlying issue has been resolved, this may be removed. func udevadmChangeToDrive(drivePath string) error { glog.V(5).Infof("udevadmChangeToDrive: drive=%q", drivePath) // Evaluate symlink, if any drive, err := filepath.EvalSymlinks(drivePath) if err != nil { return fmt.Errorf("udevadmChangeToDrive: filepath.EvalSymlinks(%q) failed with %v.", drivePath, err) } glog.V(5).Infof("udevadmChangeToDrive: symlink path is %q", drive) // Check to make sure input is "/dev/sd*" if !strings.Contains(drive, diskSDPath) { return fmt.Errorf("udevadmChangeToDrive: expected input in the form "%s" but drive is %q.", diskSDPattern, drive) } // Call "udevadm trigger --action=change --property-match=DEVNAME=/dev/sd..." _, err = exec.New().Command( "udevadm", "trigger", "--action=change", fmt.Sprintf("--property-match=DEVNAME=%s", drive)).CombinedOutput() if err != nil { return fmt.Errorf("udevadmChangeToDrive: udevadm trigger failed for drive %q with %v.", drive, err) } </ins> return nil }


Question:
If the container_probe test runs on a node without the nginx image, it sometimes takes 90s to get the pod in running. Most of this time goes in pulling the image, so I'm assuming it's registry related hiccup. Using might make it more reliable. Example failure: Note the following event in the logs:
a) we should not be pulling from docker in our tests - too hit-or-miss b) we should replace nginx (132 MB) with (4.5 MB) whenever possible. , Prashanth B wrote:

var _ = Describe("Probing container", func() { framework := Framework{BaseName: "container-probe"} var podClient client.PodInterface <del> probe := nginxProbeBuilder{} </del> <ins> probe := webserverProbeBuilder{} </ins> BeforeEach(func() { framework.beforeEach()


Question:
If the container_probe test runs on a node without the nginx image, it sometimes takes 90s to get the pod in running. Most of this time goes in pulling the image, so I'm assuming it's registry related hiccup. Using might make it more reliable. Example failure: Note the following event in the logs:
a) we should not be pulling from docker in our tests - too hit-or-miss b) we should replace nginx (132 MB) with (4.5 MB) whenever possible. , Prashanth B wrote:

expectNoError(err) startTime := time.Now() <del> expectNoError(wait.Poll(poll, 90*time.Second, func() (bool, error) { </del> <ins> Expect(wait.Poll(poll, 90*time.Second, func() (bool, error) { </ins> p, err := podClient.Get(p.Name) if err != nil { return false, err } <del> return api.IsPodReady(p), nil })) </del> <ins> ready := api.IsPodReady(p) if !ready { Logf("pod is not yet ready; pod has phase %q.", p.Status.Phase) return false, nil } return true, nil })).NotTo(HaveOccurred(), "pod never became ready") </ins> if time.Since(startTime) < 30*time.Second { Failf("Pod became ready before it's initial delay")


Question:
If the container_probe test runs on a node without the nginx image, it sometimes takes 90s to get the pod in running. Most of this time goes in pulling the image, so I'm assuming it's registry related hiccup. Using might make it more reliable. Example failure: Note the following event in the logs:
a) we should not be pulling from docker in our tests - too hit-or-miss b) we should replace nginx (132 MB) with (4.5 MB) whenever possible. , Prashanth B wrote:

isReady, err := podRunningReady(p) expectNoError(err) <del> Expect(isReady).To(BeTrue()) </del> <ins> Expect(isReady).To(BeTrue(), "pod should be ready") </ins> <del> Expect(getRestartCount(p) == 0).To(BeTrue()) </del> <ins> restartCount := getRestartCount(p) Expect(restartCount == 0).To(BeTrue(), "pod should have a restart count of 0 but got %v", restartCount) </ins> }) It("with readiness probe that fails should never be ready and never restart", func() {


Question:
If the container_probe test runs on a node without the nginx image, it sometimes takes 90s to get the pod in running. Most of this time goes in pulling the image, so I'm assuming it's registry related hiccup. Using might make it more reliable. Example failure: Note the following event in the logs:
a) we should not be pulling from docker in our tests - too hit-or-miss b) we should replace nginx (132 MB) with (4.5 MB) whenever possible. , Prashanth B wrote:

expectNoError(err) isReady, err := podRunningReady(p) <del> Expect(isReady).NotTo(BeTrue()) </del> <ins> Expect(isReady).NotTo(BeTrue(), "pod should be not ready") </ins> <del> Expect(getRestartCount(p) == 0).To(BeTrue()) </del> <ins> restartCount := getRestartCount(p) Expect(restartCount == 0).To(BeTrue(), "pod should have a restart count of 0 but got %v", restartCount) </ins> }) })


Question:
If the container_probe test runs on a node without the nginx image, it sometimes takes 90s to get the pod in running. Most of this time goes in pulling the image, so I'm assuming it's registry related hiccup. Using might make it more reliable. Example failure: Note the following event in the logs:
a) we should not be pulling from docker in our tests - too hit-or-miss b) we should replace nginx (132 MB) with (4.5 MB) whenever possible. , Prashanth B wrote:

func makePodSpec(readinessProbe, livenessProbe *api.Probe) *api.Pod { pod := &api.Pod{ <del> ObjectMeta: api.ObjectMeta{Name: "nginx-" + string(util.NewUUID())}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: "test-webserver-" + string(util.NewUUID())}, </ins> Spec: api.PodSpec{ Containers: []api.Container{ { <del> Name:           "nginx", Image:          "nginx", </del> <ins> Name:           "test-webserver", Image:          "gcr.io/google_containers/test-webserver", </ins> LivenessProbe:  livenessProbe, ReadinessProbe: readinessProbe, },


Question:
If the container_probe test runs on a node without the nginx image, it sometimes takes 90s to get the pod in running. Most of this time goes in pulling the image, so I'm assuming it's registry related hiccup. Using might make it more reliable. Example failure: Note the following event in the logs:
a) we should not be pulling from docker in our tests - too hit-or-miss b) we should replace nginx (132 MB) with (4.5 MB) whenever possible. , Prashanth B wrote:

return pod } <del> type nginxProbeBuilder struct { </del> <ins> type webserverProbeBuilder struct { </ins> failing      bool initialDelay bool } <del> func (b nginxProbeBuilder) withFailing() nginxProbeBuilder { </del> <ins> func (b webserverProbeBuilder) withFailing() webserverProbeBuilder { </ins> b.failing = true return b } <del> func (b nginxProbeBuilder) withInitialDelay() nginxProbeBuilder { </del> <ins> func (b webserverProbeBuilder) withInitialDelay() webserverProbeBuilder { </ins> b.initialDelay = true return b } <del> func (b nginxProbeBuilder) build() *api.Probe { </del> <ins> func (b webserverProbeBuilder) build() *api.Probe { </ins> probe := &api.Probe{ Handler: api.Handler{ HTTPGet: &api.HTTPGetAction{


Question:
I'm conducting PoC about distributed build using Kubernetes. Jenkins master distribute jenkins slave pods through Kubernetes API Server. When build is done, jenkins slave pod is also deleted by kubernetes API Server. But, kubernetes didn't delete container. (Pod is deleted well) After searching, I found kubelet’s containerGC option. So, I adjusted below options to kubelet. --minimum-container-ttl-duration=0s --maximum-dead-containers=0 After adjusting both 2 options, container was deleted, too. But, kubernetes containerGC didn’t delete host volume path. (/var/lib/docker/vfs/dir/..) Due to aufs’s low performance, I used data volume of host instead of container inside. So, data volume was accumulated and host's inodes was fulled, too. Is there any solution to clean the containers wreckage? My environment was.. Docker: v1.6.2 Kubernetes: v0.18.2 (container based multi clustering)
are you using the host volumes you created yourself? If this is the case, kubelet does not clean the volumes since they are external to kubelet. If not, kubelet should clean the volumes before container GC kicks in.
No.. volumes were created by kubenetes API server. LGTM. Will it be adjusted next release?
You said "I used data volume of host instead of container inside" which says you used a hostPath mount - correct? That is explicitly NOT managed by kubernetes - you've gone to the host, you get to manage it. But you also said "Due to aufs’s low performance" - an emptyDir volume is NOT stored on aufs, but it IS managed with the pod. , gghonor wrote:
The issue as i understand it is dangling volumes, that is volumes that are created via a VOLUME directive in Dockerfile with no corresponding kubernetes volume in spec which therefore has no host path & is not cleaned up by the kubelet. These volumes reside in /var/lib/docker/vfs/. This waste of disk space has the potential to cause out of space errors on the node & therefore make the node unstable.
As the volume dir you mentioned is I assume you didn't specify any volume in your Kubernetes pod spec but did use the directive in your Dockerfile? As mentions I guess this isn't a Kubernetes managed volume as that would be in the Kubernetes volume dir, right? That dir is used by Docker for volumes specified in Dockerfiles or a docker container that is run with with no specified host path. The latter doesn't happen with Kubernetes managed containers AFAIK.
This makes sense. Thanks for the explanation :) I see the point of removing the dangling volumes. Even though kubelet doesn't manage those volumes, it doesn't hurt to remove them.
Do we have a full understanding of their lifetime, though? We don't control them, I don't see how we can know when to delete them. Docker's volume implementation is just too under-defined to actually be useful. , Yu-Ju Hong wrote:

// Remove unidentified containers. for _, container := range unidentifiedContainers { glog.Infof("Removing unidentified dead container %q with ID %q", container.name, container.id) <del> err = cgc.dockerClient.RemoveContainer(docker.RemoveContainerOptions{ID: container.id}) </del> <ins> err = cgc.dockerClient.RemoveContainer(docker.RemoveContainerOptions{ID: container.id, RemoveVolumes: true}) </ins> if err != nil { glog.Warningf("Failed to remove unidentified dead container %q: %v", container.name, err) }


Question:
I'm conducting PoC about distributed build using Kubernetes. Jenkins master distribute jenkins slave pods through Kubernetes API Server. When build is done, jenkins slave pod is also deleted by kubernetes API Server. But, kubernetes didn't delete container. (Pod is deleted well) After searching, I found kubelet’s containerGC option. So, I adjusted below options to kubelet. --minimum-container-ttl-duration=0s --maximum-dead-containers=0 After adjusting both 2 options, container was deleted, too. But, kubernetes containerGC didn’t delete host volume path. (/var/lib/docker/vfs/dir/..) Due to aufs’s low performance, I used data volume of host instead of container inside. So, data volume was accumulated and host's inodes was fulled, too. Is there any solution to clean the containers wreckage? My environment was.. Docker: v1.6.2 Kubernetes: v0.18.2 (container based multi clustering)
are you using the host volumes you created yourself? If this is the case, kubelet does not clean the volumes since they are external to kubelet. If not, kubelet should clean the volumes before container GC kicks in.
No.. volumes were created by kubenetes API server. LGTM. Will it be adjusted next release?
You said "I used data volume of host instead of container inside" which says you used a hostPath mount - correct? That is explicitly NOT managed by kubernetes - you've gone to the host, you get to manage it. But you also said "Due to aufs’s low performance" - an emptyDir volume is NOT stored on aufs, but it IS managed with the pod. , gghonor wrote:
The issue as i understand it is dangling volumes, that is volumes that are created via a VOLUME directive in Dockerfile with no corresponding kubernetes volume in spec which therefore has no host path & is not cleaned up by the kubelet. These volumes reside in /var/lib/docker/vfs/. This waste of disk space has the potential to cause out of space errors on the node & therefore make the node unstable.
As the volume dir you mentioned is I assume you didn't specify any volume in your Kubernetes pod spec but did use the directive in your Dockerfile? As mentions I guess this isn't a Kubernetes managed volume as that would be in the Kubernetes volume dir, right? That dir is used by Docker for volumes specified in Dockerfiles or a docker container that is run with with no specified host path. The latter doesn't happen with Kubernetes managed containers AFAIK.
This makes sense. Thanks for the explanation :) I see the point of removing the dangling volumes. Even though kubelet doesn't manage those volumes, it doesn't hurt to remove them.
Do we have a full understanding of their lifetime, though? We don't control them, I don't see how we can know when to delete them. Docker's volume implementation is just too under-defined to actually be useful. , Yu-Ju Hong wrote:

// Remove from oldest to newest (last to first). numToKeep := len(containers) - toRemove for i := numToKeep; i < len(containers); i++ { <del> err := cgc.dockerClient.RemoveContainer(docker.RemoveContainerOptions{ID: containers[i].id}) </del> <ins> err := cgc.dockerClient.RemoveContainer(docker.RemoveContainerOptions{ID: containers[i].id, RemoveVolumes: true}) </ins> if err != nil { glog.Warningf("Failed to remove dead container %q: %v", containers[i].name, err) }


Question:
The file: has instructions for AWS, GCE, and VMware. I'd like to see documentation for OpenStack as well. I'd be happy to do the work for this as I don't think it would take too much time. I'll add some instructions to this issue if its something that people think would be helpful.
I would greatly appreciate any work you can do with this. I've been trying, unsuccessfully, to get the OpenStack integration working.
Ok, I created instructions in this gist: If it looks good to you I'll create a pull request and merge changes in the document after the GCE section (as it appears we're going in Alphabetical order? )
Sending a PR sounds good.

In one terminal, run `gcloud compute ssh master --ssh-flag="-L 8080:127.0.0.1:8080"` and in a second run `gcloud compute ssh master --ssh-flag="-R 8080:127.0.0.1:8080"`. <ins> ### OpenStack These instructions are for running on the command line.  Most of this you can also do through the Horizon dashboard. These instructions were tested on the Ice House release on a Metacloud distribution of OpenStack but should be similar if not the same across other versions/distributions of OpenStack. #### Make sure you can connect with OpenStack Make sure the environment variables are set for OpenStack such as: ```sh OS_TENANT_ID OS_PASSWORD OS_AUTH_URL OS_USERNAME OS_TENANT_NAME ``` Test this works with something like: ``` nova list ``` #### Get a Suitable CoreOS Image You'll need a [suitable version of CoreOS image for OpenStack] (https://coreos.com/os/docs/latest/booting-on-openstack.html) Once you download that, upload it to glance.  An example is shown below: ```sh glance image-create --name CoreOS723  --container-format bare --disk-format qcow2  --file coreos_production_openstack_image.img  --is-public True ``` #### Create security group ```sh nova secgroup-create kubernetes "Kubernetes Security Group" nova secgroup-add-rule kubernetes tcp 22 22   0.0.0.0/0 nova secgroup-add-rule kubernetes tcp 80 80   0.0.0.0/0 ``` #### Provision the Master ```sh nova boot  --image <image_name>  --key-name <my_key>  --flavor <flavor id>  --security-group kubernetes  --user-data files/master.yaml  kube-master ``` ```<image_name>``` is the CoreOS image name.  In our example we can use the image we created in the previous step and put in 'CoreOS723' ```<my_key>``` is the keypair name that you already generated to access the instance. ```<flavor_id>``` is the flavor ID you use to size the instance.  Run ```nova flavor-list``` to get the IDs.  3 on the system this was tested with gives the m1.large size. The important part is to ensure you have the files/master.yml as this is what will do all the post boot configuration. This path is relevant so we are assuming in this example that you are running the nova command in a directory where there is a subdirectory called files that has the master.yml file in it.  Absolute paths also work. Next, assign it a public IP address: ``` nova floating-ip-list ``` Get an IP address that's free and run: ``` nova floating-ip-associate kube-master <ip address> ``` where ```<ip address>``` is the IP address that was available from the ```nova floating-ip-list``` command. #### Provision Worker Nodes Edit ```node.yaml``` and replace all instances of ```<master-private-ip>``` with the private IP address of the master node.  You can get this by runnning ```nova show kube-master``` assuming you named your instance kube master.  This is not the floating IP address you just assigned it. ```sh nova boot  --image <image_name>  --key-name <my_key>  --flavor <flavor id>  --security-group kubernetes  --user-data files/node.yaml  minion01 ``` This is basically the same as the master nodes but with the node.yaml post-boot script instead of the master. </ins> ### VMware Fusion #### Create the master config-drive


Question: If is passed to the kubelet, container metrics are not included in the Prometheus metrics endpoint.

Manager: m, } <del> // Export the HTTP endpoint if a port was specified. if port > 0 { err = cadvisorClient.exportHTTP(port) if err != nil { return nil, err } </del> <ins> err = cadvisorClient.exportHTTP(port) if err != nil { return nil, err </ins> } return cadvisorClient, nil }


Question: If is passed to the kubelet, container metrics are not included in the Prometheus metrics endpoint.

} func (cc *cadvisorClient) exportHTTP(port uint) error { <ins> // Register the handlers regardless as this registers the prometheus // collector properly. </ins> mux := http.NewServeMux() err := cadvisorHttp.RegisterHandlers(mux, cc, "", "", "", "", "/metrics") if err != nil { return err } <del> serv := &http.Server{ Addr:    fmt.Sprintf(":%d", port), Handler: mux, } // TODO(vmarmol): Remove this when the cAdvisor port is once again free. // If export failed, retry in the background until we are able to bind. // This allows an existing cAdvisor to be killed before this one registers. go func() { defer util.HandleCrash() err := serv.ListenAndServe() for err != nil { glog.Infof("Failed to register cAdvisor on port %d, retrying. Error: %v", port, err) time.Sleep(time.Minute) err = serv.ListenAndServe() </del> <ins> // Only start the http server if port > 0 if port > 0 { serv := &http.Server{ Addr:    fmt.Sprintf(":%d", port), Handler: mux, </ins> } <del> }() </del> <ins> // TODO(vmarmol): Remove this when the cAdvisor port is once again free. // If export failed, retry in the background until we are able to bind. // This allows an existing cAdvisor to be killed before this one registers. go func() { defer util.HandleCrash() err := serv.ListenAndServe() for err != nil { glog.Infof("Failed to register cAdvisor on port %d, retrying. Error: %v", port, err) time.Sleep(time.Minute) err = serv.ListenAndServe() } }() } </ins> return nil }


Question:
The first line logged from the integration test is the last line from the service account token test is then the test panics with: I think the tests are just running too close the the timeout. On successful runs, the tests take ~115 seconds, which is ridiculously close to the threshold. For example: see
ok, do you want to send a PR to increase the timeout to 3 or 4 minutes?
Sure, I can do that tonight

# KUBE_TEST_API_VERSIONS=${KUBE_TEST_API_VERSIONS:-"v1,experimental/v1alpha1"} KUBE_TEST_API_VERSIONS=${KUBE_TEST_API_VERSIONS:-"v1,experimental/v1alpha1"} <ins> # Give integration tests longer to run KUBE_TIMEOUT=${KUBE_TIMEOUT:--timeout 240s} </ins> KUBE_INTEGRATION_TEST_MAX_CONCURRENCY=${KUBE_INTEGRATION_TEST_MAX_CONCURRENCY:-"-1"} LOG_LEVEL=${LOG_LEVEL:-2}


Question:
The first line logged from the integration test is the last line from the service account token test is then the test panics with: I think the tests are just running too close the the timeout. On successful runs, the tests take ~115 seconds, which is ridiculously close to the threshold. For example: see
ok, do you want to send a PR to increase the timeout to 3 or 4 minutes?
Sure, I can do that tonight

# KUBE_RACE="-race" KUBE_GOFLAGS="-tags 'integration no-docker' "  KUBE_RACE=""  <ins> KUBE_TIMEOUT="${KUBE_TIMEOUT}"  </ins> KUBE_TEST_API_VERSIONS="$1"  KUBE_API_VERSIONS="v1,experimental/v1alpha1"  "${KUBE_ROOT}/hack/test-go.sh" test/integration


Question:
Nodes Network when a minion node becomes unreachable [replication controller] recreates pods scheduled on the unreachable minion node AND allows scheduling of pods on a minion after it rejoins the cluster(Failed 17 times in the last 30 runs. Stability: 43 %)Failed with 0 retries
for triage
I'll look into this.
I talked to about this and he had a pretty good guess in about two seconds about what the problem is. ReplicationController sync method does this: but DaemonSet controller doesn't. (I guess when DaemonSet was copy-pasted from ReplicationController, this was accidentally removed.) Pod reflector and syncer are running in separate goroutines, and if pod reflector doesn't run then syncer will think there are no pods on the node.
To be clear: this is about the test Controller Manager should not create/delete replicas across restart which is the one that is actually failing.
I'm confused you're saying that this is about , but you're updating . I thought they are about two different tests...
Yes sorry, I put my comment in the wrong issue. Reassigning this one to you, and I'll copy my comment into the other one.
First failure: We check if all namespace are deleted. If other tests failed to delete their namespace this test will fail. I'm going to delete this logic as we already check if we have successfully deleted our namespace. See also for some more details.
Second failure: We disable network by running the following command: I checked and it seems that it results in ing master to hang. I suspect that Kubelet hangs on updating node status and depending when it breaks the connection (by timeout) it will fit in test timeouts or not.
What is the default timeout for connections using client libraries to apiserver? I'm checking the code, but can't find it.
I'm not sure if we set one, but we should. I guess I would expect to see that in the config, w/ default set by this function: , Filip Grzadkowski wrote:
So: let's add a timeout, and revert .
I think that adding a timeout and reverting are orthogonal. I we want to simulate network problems with node-master communication then using REJECT is better anyway. At the same time adding timeout makes sense for many other reasons.
OK then, let's add a timeout and test both DROP and REJECT-- no sort of network problem should hang a component.
I think it didn't hang. It just that the kubelet was retrying too slow and sometimes it fit test timeout, sometime it didn't.
Could we reduce the timeout to make it always fit the test timeout? , Filip Grzadkowski
I guess we can. I'll try tomorrow.
It seems we have a new type of failure: It seems that kubelet on a machine where network is blocked doesn't notice new pod. After reading logs I assume that kubelet doesn't notice that the watch is broken and doesn't try to relist everything. It notices new pod only after watch timeout.
Ugh, I hope we don't have to start sending watch heartbeats.
It seems that all recent failures are with the following error: which means the reason is here: - can you please take a look or delegate?
I did some more debugging of it. I was looking into this failure: There is something strange happening here: run performTemporaryNetworkFailure: succesfully block the load from kubelet: the succesfully wait for node to become not ready: some time nodecontroller successfully removes the pod: replication controller creates a new pod: here is something I don't undertand - because of some reason, the test doesn't observe that the pod actually disappeared. - any thoughts on it?
BTW - this is exactly the same situation in:
OK - I think this is related to default 30s deletion grace period.
Yes - it seems that this is exactly the case. I will send out PR fixing this out for review.
I'm reopening it for tracking. If there won't be any failures within next few days - I will move the test back from flaky and close this issue.
Thanks for fixing, .
It seems that the main issue is fixed. However, there was one failure in the last 30 runs: in which one pod simply didn't started (and it was "preparation" for the test, not the test itself).
During next 2 days there was exactly one more failure of this test: it is exactly the same as the previous one. However, those are duplicates of So I'm going to move this test out of flaky suite.

func rcByNameContainer(name string, replicas int, image string, labels map[string]string, c api.Container) *api.ReplicationController { // Add "name": name to the labels, overwriting if it exists. labels["name"] = name <ins> gracePeriod := int64(0) </ins> return &api.ReplicationController{ TypeMeta: unversioned.TypeMeta{ Kind:       "ReplicationController",


Question:
Nodes Network when a minion node becomes unreachable [replication controller] recreates pods scheduled on the unreachable minion node AND allows scheduling of pods on a minion after it rejoins the cluster(Failed 17 times in the last 30 runs. Stability: 43 %)Failed with 0 retries
for triage
I'll look into this.
I talked to about this and he had a pretty good guess in about two seconds about what the problem is. ReplicationController sync method does this: but DaemonSet controller doesn't. (I guess when DaemonSet was copy-pasted from ReplicationController, this was accidentally removed.) Pod reflector and syncer are running in separate goroutines, and if pod reflector doesn't run then syncer will think there are no pods on the node.
To be clear: this is about the test Controller Manager should not create/delete replicas across restart which is the one that is actually failing.
I'm confused you're saying that this is about , but you're updating . I thought they are about two different tests...
Yes sorry, I put my comment in the wrong issue. Reassigning this one to you, and I'll copy my comment into the other one.
First failure: We check if all namespace are deleted. If other tests failed to delete their namespace this test will fail. I'm going to delete this logic as we already check if we have successfully deleted our namespace. See also for some more details.
Second failure: We disable network by running the following command: I checked and it seems that it results in ing master to hang. I suspect that Kubelet hangs on updating node status and depending when it breaks the connection (by timeout) it will fit in test timeouts or not.
What is the default timeout for connections using client libraries to apiserver? I'm checking the code, but can't find it.
I'm not sure if we set one, but we should. I guess I would expect to see that in the config, w/ default set by this function: , Filip Grzadkowski wrote:
So: let's add a timeout, and revert .
I think that adding a timeout and reverting are orthogonal. I we want to simulate network problems with node-master communication then using REJECT is better anyway. At the same time adding timeout makes sense for many other reasons.
OK then, let's add a timeout and test both DROP and REJECT-- no sort of network problem should hang a component.
I think it didn't hang. It just that the kubelet was retrying too slow and sometimes it fit test timeout, sometime it didn't.
Could we reduce the timeout to make it always fit the test timeout? , Filip Grzadkowski
I guess we can. I'll try tomorrow.
It seems we have a new type of failure: It seems that kubelet on a machine where network is blocked doesn't notice new pod. After reading logs I assume that kubelet doesn't notice that the watch is broken and doesn't try to relist everything. It notices new pod only after watch timeout.
Ugh, I hope we don't have to start sending watch heartbeats.
It seems that all recent failures are with the following error: which means the reason is here: - can you please take a look or delegate?
I did some more debugging of it. I was looking into this failure: There is something strange happening here: run performTemporaryNetworkFailure: succesfully block the load from kubelet: the succesfully wait for node to become not ready: some time nodecontroller successfully removes the pod: replication controller creates a new pod: here is something I don't undertand - because of some reason, the test doesn't observe that the pod actually disappeared. - any thoughts on it?
BTW - this is exactly the same situation in:
OK - I think this is related to default 30s deletion grace period.
Yes - it seems that this is exactly the case. I will send out PR fixing this out for review.
I'm reopening it for tracking. If there won't be any failures within next few days - I will move the test back from flaky and close this issue.
Thanks for fixing, .
It seems that the main issue is fixed. However, there was one failure in the last 30 runs: in which one pod simply didn't started (and it was "preparation" for the test, not the test itself).
During next 2 days there was exactly one more failure of this test: it is exactly the same as the previous one. However, those are duplicates of So I'm going to move this test out of flaky suite.

Labels: labels, }, Spec: api.PodSpec{ <del> Containers: []api.Container{c}, </del> <ins> Containers:                    []api.Container{c}, TerminationGracePeriodSeconds: &gracePeriod, </ins> }, }, },


Question:
Nodes Network when a minion node becomes unreachable [replication controller] recreates pods scheduled on the unreachable minion node AND allows scheduling of pods on a minion after it rejoins the cluster(Failed 17 times in the last 30 runs. Stability: 43 %)Failed with 0 retries
for triage
I'll look into this.
I talked to about this and he had a pretty good guess in about two seconds about what the problem is. ReplicationController sync method does this: but DaemonSet controller doesn't. (I guess when DaemonSet was copy-pasted from ReplicationController, this was accidentally removed.) Pod reflector and syncer are running in separate goroutines, and if pod reflector doesn't run then syncer will think there are no pods on the node.
To be clear: this is about the test Controller Manager should not create/delete replicas across restart which is the one that is actually failing.
I'm confused you're saying that this is about , but you're updating . I thought they are about two different tests...
Yes sorry, I put my comment in the wrong issue. Reassigning this one to you, and I'll copy my comment into the other one.
First failure: We check if all namespace are deleted. If other tests failed to delete their namespace this test will fail. I'm going to delete this logic as we already check if we have successfully deleted our namespace. See also for some more details.
Second failure: We disable network by running the following command: I checked and it seems that it results in ing master to hang. I suspect that Kubelet hangs on updating node status and depending when it breaks the connection (by timeout) it will fit in test timeouts or not.
What is the default timeout for connections using client libraries to apiserver? I'm checking the code, but can't find it.
I'm not sure if we set one, but we should. I guess I would expect to see that in the config, w/ default set by this function: , Filip Grzadkowski wrote:
So: let's add a timeout, and revert .
I think that adding a timeout and reverting are orthogonal. I we want to simulate network problems with node-master communication then using REJECT is better anyway. At the same time adding timeout makes sense for many other reasons.
OK then, let's add a timeout and test both DROP and REJECT-- no sort of network problem should hang a component.
I think it didn't hang. It just that the kubelet was retrying too slow and sometimes it fit test timeout, sometime it didn't.
Could we reduce the timeout to make it always fit the test timeout? , Filip Grzadkowski
I guess we can. I'll try tomorrow.
It seems we have a new type of failure: It seems that kubelet on a machine where network is blocked doesn't notice new pod. After reading logs I assume that kubelet doesn't notice that the watch is broken and doesn't try to relist everything. It notices new pod only after watch timeout.
Ugh, I hope we don't have to start sending watch heartbeats.
It seems that all recent failures are with the following error: which means the reason is here: - can you please take a look or delegate?
I did some more debugging of it. I was looking into this failure: There is something strange happening here: run performTemporaryNetworkFailure: succesfully block the load from kubelet: the succesfully wait for node to become not ready: some time nodecontroller successfully removes the pod: replication controller creates a new pod: here is something I don't undertand - because of some reason, the test doesn't observe that the pod actually disappeared. - any thoughts on it?
BTW - this is exactly the same situation in:
OK - I think this is related to default 30s deletion grace period.
Yes - it seems that this is exactly the case. I will send out PR fixing this out for review.
I'm reopening it for tracking. If there won't be any failures within next few days - I will move the test back from flaky and close this issue.
Thanks for fixing, .
It seems that the main issue is fixed. However, there was one failure in the last 30 runs: in which one pod simply didn't started (and it was "preparation" for the test, not the test itself).
During next 2 days there was exactly one more failure of this test: it is exactly the same as the previous one. However, those are duplicates of So I'm going to move this test out of flaky suite.

// In case of failure or too long waiting time, an error is returned. func waitForRCPodToDisappear(c *client.Client, ns, rcName, podName string) error { label := labels.SelectorFromSet(labels.Set(map[string]string{"name": rcName})) <del> return waitForPodToDisappear(c, ns, podName, label, 20*time.Second, 5*time.Minute) </del> <ins> // NodeController evicts pod after 5 minutes, so we need timeout greater than that. // Additionally, there can be non-zero grace period, so we are setting 10 minutes // to be on the safe size. return waitForPodToDisappear(c, ns, podName, label, 20*time.Second, 10*time.Minute) </ins> } // waitForService waits until the service appears (exist == true), or disappears (exist == false)


Question:
runs foo and then waits for it to be ready or terminated. If the pod is terminated, it prints the log, and if it's not terminated, it attaches the output. However, it does not handle the case that the pod terminates between returning the ready status and attaching, as can be the case in the : Options to fix this: Fix kubectl: If attach fails, fall back to printing the logs If attach fails, check the status again and if it's terminated, print the logs Fix the flaky test: Prevent the container from exiting (e.g. `sh -c 'echo running in container && sleep 30') Prevent the container from entering ready state (e.g. )
Thanks for debugging. I think we should probably do both: If attach fails and the container is now terminated, fall back to printing logs Make the e2e more robust by not exiting for a while to ensure time to attach. I'm happy to review PRs

return fmt.Errorf("pod %s is not running and cannot be attached to; current phase is %s", p.PodName, pod.Status.Phase) } <del> containerName := p.ContainerName if len(containerName) == 0 { glog.V(4).Infof("defaulting container name to %s", pod.Spec.Containers[0].Name) containerName = pod.Spec.Containers[0].Name } </del> // TODO: refactor with terminal helpers from the edit utility once that is merged var stdin io.Reader tty := p.TTY


Question:
runs foo and then waits for it to be ready or terminated. If the pod is terminated, it prints the log, and if it's not terminated, it attaches the output. However, it does not handle the case that the pod terminates between returning the ready status and attaching, as can be the case in the : Options to fix this: Fix kubectl: If attach fails, fall back to printing the logs If attach fails, check the status again and if it's terminated, print the logs Fix the flaky test: Prevent the container from exiting (e.g. `sh -c 'echo running in container && sleep 30') Prevent the container from entering ready state (e.g. )
Thanks for debugging. I think we should probably do both: If attach fails and the container is now terminated, fall back to printing logs Make the e2e more robust by not exiting for a while to ensure time to attach. I'm happy to review PRs

Name(pod.Name). Namespace(pod.Namespace). SubResource("attach"). <del> Param("container", containerName) </del> <ins> Param("container", p.GetContainerName(pod)) </ins> return p.Attach.Attach(req, p.Config, stdin, p.Out, p.Err, tty) } <ins> // GetContainerName returns the name of the container to attach to, with a fallback. func (p *AttachOptions) GetContainerName(pod *api.Pod) string { if len(p.ContainerName) > 0 { return p.ContainerName } glog.V(4).Infof("defaulting container name to %s", pod.Spec.Containers[0].Name) return pod.Spec.Containers[0].Name } </ins>


Question:
runs foo and then waits for it to be ready or terminated. If the pod is terminated, it prints the log, and if it's not terminated, it attaches the output. However, it does not handle the case that the pod terminates between returning the ready status and attaching, as can be the case in the : Options to fix this: Fix kubectl: If attach fails, fall back to printing the logs If attach fails, check the status again and if it's terminated, print the logs Fix the flaky test: Prevent the container from exiting (e.g. `sh -c 'echo running in container && sleep 30') Prevent the container from entering ready state (e.g. )
Thanks for debugging. I think we should probably do both: If attach fails and the container is now terminated, fall back to printing logs Make the e2e more robust by not exiting for a while to ensure time to attach. I'm happy to review PRs

return err } if status == api.PodSucceeded || status == api.PodFailed { <del> return handleLog(c, pod.Namespace, pod.Name, &api.PodLogOptions{Container: pod.Spec.Containers[0].Name}, opts.Out) </del> <ins> return handleLog(c, pod.Namespace, pod.Name, &api.PodLogOptions{Container: opts.GetContainerName(pod)}, opts.Out) </ins> } opts.Client = c opts.PodName = pod.Name opts.Namespace = pod.Namespace <del> return opts.Run() </del> <ins> if err := opts.Run(); err != nil { fmt.Fprintf(opts.Out, "Error attaching, falling back to logs: %vn", err) return handleLog(c, pod.Namespace, pod.Name, &api.PodLogOptions{Container: opts.GetContainerName(pod)}, opts.Out) } return nil </ins> } func getRestartPolicy(cmd *cobra.Command, interactive bool) (api.RestartPolicy, error) {


Question:
I'm seeing a bug where the network plugin fails to configure networking, and yet the containers for this pod are still deployed (without networking configured). It looks like the following is occurring: 1) Kubelet attempts to create infra container in 2) Network plugin fails, returns error code. 3) Kubelet detects error and skips pod creation but does not tear down the infra container 4) On the next call, the kubelet detects that the infra container is running and deploys the other containers in the pod. The fix here is for the kubelet to clean up the infra container in when it receives the error from the network plugin and before it returns. Here are some logs illustrating the steps I described above:
Cleaning up infra container on network plugin error sounds reasonable/

if containerChanges.StartInfraContainer && (len(containerChanges.ContainersToStart) > 0) { glog.V(4).Infof("Creating pod infra container for %q", podFullName) podInfraContainerID, err = dm.createPodInfraContainer(pod) <ins> if err != nil { glog.Errorf("Failed to create pod infra container: %v; Skipping pod %q", err, podFullName) return err } </ins> // Call the networking plugin <del> if err == nil { err = dm.networkPlugin.SetUpPod(pod.Namespace, pod.Name, podInfraContainerID) } </del> <ins> err = dm.networkPlugin.SetUpPod(pod.Namespace, pod.Name, podInfraContainerID) </ins> if err != nil { glog.Errorf("Failed to create pod infra container: %v; Skipping pod %q", err, podFullName) <ins> // Delete infra container if delErr := dm.KillContainerInPod(kubecontainer.ContainerID{ ID:   string(podInfraContainerID), Type: "docker"}, nil, pod); delErr != nil { glog.Warningf("Clear infra container failed for pod %q: %v", podFullName, delErr) } </ins> return err }


Question:
Unit tests on our CI have apparently been failing for 2 weeks and no one noticed (oops!)... The PR that seems to have caused the failures is . The failing test is - . Full Test Log: I'm not sure why this is failing on our CI and not on shippable. Has anyone else seen something similar? That test only seems to fail on the second run through with . Our unit tests run in a container built specifically for testing k8s-mesos: The Dockerfile for the env: Is anyone else able to repro with the above env or their own?
/cc
that test was around a week ago, it's doesn't seem possible to me that it was failing since 2 weeks ago. Anyway, we have few duplications of this issue: this one And the PR where is working on: It's a little bit difficult to me to follow all the conversations in all the differente places about the same bug, could it be possible to unificate some? Since you provided the environment I will try and take a look to reproduce it locally. Thanks!
I think I've got a working patch in
Closing in favor of &

package flocker import ( <ins> "io/ioutil" "os" </ins> "testing" flockerClient "github.com/ClusterHQ/flocker-go"


Question:
Unit tests on our CI have apparently been failing for 2 weeks and no one noticed (oops!)... The PR that seems to have caused the failures is . The failing test is - . Full Test Log: I'm not sure why this is failing on our CI and not on shippable. Has anyone else seen something similar? That test only seems to fail on the second run through with . Our unit tests run in a container built specifically for testing k8s-mesos: The Dockerfile for the env: Is anyone else able to repro with the above env or their own?
/cc
that test was around a week ago, it's doesn't seem possible to me that it was failing since 2 weeks ago. Anyway, we have few duplications of this issue: this one And the PR where is working on: It's a little bit difficult to me to follow all the conversations in all the differente places about the same bug, could it be possible to unificate some? Since you provided the environment I will try and take a look to reproduce it locally. Thanks!
I think I've got a working patch in
Closing in favor of &

const pluginName = "kubernetes.io/flocker" <del> func newInitializedVolumePlugMgr() volume.VolumePluginMgr { </del> <ins> func newInitializedVolumePlugMgr(t *testing.T) (volume.VolumePluginMgr, string) { </ins> plugMgr := volume.VolumePluginMgr{} <del> plugMgr.InitPlugins(ProbeVolumePlugins(), volume.NewFakeVolumeHost("/foo/bar", nil, nil)) return plugMgr </del> <ins> dir, err := ioutil.TempDir("", "flocker") assert.NoError(t, err) plugMgr.InitPlugins(ProbeVolumePlugins(), volume.NewFakeVolumeHost(dir, nil, nil)) return plugMgr, dir </ins> } func TestGetByName(t *testing.T) { assert := assert.New(t) <del> plugMgr := newInitializedVolumePlugMgr() </del> <ins> plugMgr, _ := newInitializedVolumePlugMgr(t) </ins> plug, err := plugMgr.FindPluginByName(pluginName) assert.NotNil(plug, "Can't find the plugin by name")


Question:
Unit tests on our CI have apparently been failing for 2 weeks and no one noticed (oops!)... The PR that seems to have caused the failures is . The failing test is - . Full Test Log: I'm not sure why this is failing on our CI and not on shippable. Has anyone else seen something similar? That test only seems to fail on the second run through with . Our unit tests run in a container built specifically for testing k8s-mesos: The Dockerfile for the env: Is anyone else able to repro with the above env or their own?
/cc
that test was around a week ago, it's doesn't seem possible to me that it was failing since 2 weeks ago. Anyway, we have few duplications of this issue: this one And the PR where is working on: It's a little bit difficult to me to follow all the conversations in all the differente places about the same bug, could it be possible to unificate some? Since you provided the environment I will try and take a look to reproduce it locally. Thanks!
I think I've got a working patch in
Closing in favor of &

func TestCanSupport(t *testing.T) { assert := assert.New(t) <del> plugMgr := newInitializedVolumePlugMgr() </del> <ins> plugMgr, _ := newInitializedVolumePlugMgr(t) </ins> plug, err := plugMgr.FindPluginByName(pluginName) assert.NoError(err)


Question:
Unit tests on our CI have apparently been failing for 2 weeks and no one noticed (oops!)... The PR that seems to have caused the failures is . The failing test is - . Full Test Log: I'm not sure why this is failing on our CI and not on shippable. Has anyone else seen something similar? That test only seems to fail on the second run through with . Our unit tests run in a container built specifically for testing k8s-mesos: The Dockerfile for the env: Is anyone else able to repro with the above env or their own?
/cc
that test was around a week ago, it's doesn't seem possible to me that it was failing since 2 weeks ago. Anyway, we have few duplications of this issue: this one And the PR where is working on: It's a little bit difficult to me to follow all the conversations in all the differente places about the same bug, could it be possible to unificate some? Since you provided the environment I will try and take a look to reproduce it locally. Thanks!
I think I've got a working patch in
Closing in favor of &

func TestNewBuilder(t *testing.T) { assert := assert.New(t) <del> plugMgr := newInitializedVolumePlugMgr() </del> <ins> plugMgr, _ := newInitializedVolumePlugMgr(t) </ins> plug, err := plugMgr.FindPluginByName(pluginName) assert.NoError(err)


Question:
Unit tests on our CI have apparently been failing for 2 weeks and no one noticed (oops!)... The PR that seems to have caused the failures is . The failing test is - . Full Test Log: I'm not sure why this is failing on our CI and not on shippable. Has anyone else seen something similar? That test only seems to fail on the second run through with . Our unit tests run in a container built specifically for testing k8s-mesos: The Dockerfile for the env: Is anyone else able to repro with the above env or their own?
/cc
that test was around a week ago, it's doesn't seem possible to me that it was failing since 2 weeks ago. Anyway, we have few duplications of this issue: this one And the PR where is working on: It's a little bit difficult to me to follow all the conversations in all the differente places about the same bug, could it be possible to unificate some? Since you provided the environment I will try and take a look to reproduce it locally. Thanks!
I think I've got a working patch in
Closing in favor of &

assert := assert.New(t) <del> plugMgr := newInitializedVolumePlugMgr() </del> <ins> plugMgr, rootDir := newInitializedVolumePlugMgr(t) if rootDir != "" { defer os.RemoveAll(rootDir) } </ins> plug, err := plugMgr.FindPluginByName(flockerPluginName) assert.NoError(err)


Question:
When trying to edit multiple resources, for example: will panic. Ref as the possible cause cc
if was the cause, any fix needs to happen in 1.1 as well, was already cherry picked there
Also shows that we need a test for this case.
is the cause. When we edit multiple resources (i.e., list-type objects), the list's Accessor.annotations (type ) is nil and will panic when UpdateApplyAnnotation tries to SetAnnotations ().

kube::test::get_object_assert 'rc mock2' "{{${labels_field}.status}}" 'replaced' fi fi <ins> # Command: kubectl edit multiple resources temp_editor="${KUBE_TEMP}/tmp-editor.sh" echo -e '#!/bin/bashnsed -i "s/status: replaced/status: edited/g" $@' > "${temp_editor}" chmod +x "${temp_editor}" EDITOR="${temp_editor}" kubectl edit "${kube_flags[@]}" -f "${file}" # Post-condition: mock service (and mock2) and mock rc (and mock2) are edited if [ "$has_svc" = true ]; then kube::test::get_object_assert 'services mock' "{{${labels_field}.status}}" 'edited' if [ "$two_svcs" = true ]; then kube::test::get_object_assert 'services mock2' "{{${labels_field}.status}}" 'edited' fi fi if [ "$has_rc" = true ]; then kube::test::get_object_assert 'rc mock' "{{${labels_field}.status}}" 'edited' if [ "$two_rcs" = true ]; then kube::test::get_object_assert 'rc mock2' "{{${labels_field}.status}}" 'edited' fi fi # cleaning rm "${temp_editor}" </ins> # Command # We need to set --overwrite, because otherwise, if the first attempt to run "kubectl label" # fails on some, but not all, of the resources, retries will fail because it tries to modify


Question:
When trying to edit multiple resources, for example: will panic. Ref as the possible cause cc
if was the cause, any fix needs to happen in 1.1 as well, was already cherry picked there
Also shows that we need a test for this case.
is the cause. When we edit multiple resources (i.e., list-type objects), the list's Accessor.annotations (type ) is nil and will panic when UpdateApplyAnnotation tries to SetAnnotations ().

} func (a genericAccessor) SetAnnotations(annotations map[string]string) { <ins> if a.annotations == nil { emptyAnnotations := make(map[string]string) a.annotations = &emptyAnnotations } </ins> *a.annotations = annotations }