The test is failing in . https://k8s- https://k8s- https://k8s- The reason is that the test expects that all pause containers () should have . However, in many tests, we are starting test containers with . Those pause containers have . Ideally, containers started in previous test should be cleaned up, so in serial suite it should not affect later test. However, it is not always true when CRI is enabled. For example, in the oom message, we could see: There are 2 problems here: Why are containers not properly cleaned up when CRI enabled? I can't even find which test leaked pause container from the log, because there are many test using pause image, and some of them start a lot of containers, such as density test, restart test. The oom score adj test should not rely on process name . However, I can't find an alternative for now, because we don't report pause container status in our api, we don't even know the container id to inspect with docker or CRI. /cc
This test is still flaking in cri-serial. Some examples: https://k8s- https://k8s- Need to check the logs to see if the symptoms are the same.
It could be that with CRI, it takes kubelet longer to clean up. Or, even worse, that kubelet is not cleaning up properly. We need more information on this on failure. Dumping pods on the node Dumping containers () on the node. Accessing docker directly is not ideal, but it's okay because the test itself only applies to docker. Alternatively, we can get the current running containers from kubelet's /runningPods endpoint. The downside is that this may lag behind the actual state a little bit (seconds). We should also make the test suite run more frequently. It runs once a day, which makes the debugging round-trip overhead too high.
I can reproduce this process leak by running the "Network should recover from ip leaks" on my own node, without CRI enabled. The image used in the test was , and the docker version 1.11.2. The test creates 100 pods, then restarts the docker service by 6 times, with 20s interval in between. After the test ends, there could be multiple processes running left on the node. Those processes were reparented to pid 1, and showed no containers running. /cc
On the side note, the test should not be blindly checking all processes. It should, instead, take a snapshot of the system before the test and only check the new processes created by the tests.
I filed a new issue for the docker bug since it's not unique to CRI and this test is catching it by accident: Leaving this issue to fix the test by not blindly checking all processes:
The docker bug is not a regression, and is tracked in . The problem of the test is not necessarily to be fixed in v1.6, at least it's not release blocker. :)


Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:38:40Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: None Others: None What happened: When I create a deployment with vSphere cloud provider dynamic provisioning, it creates a volume spec with volume path - [datastore] kubevols/volume which contains a white space in it. In this case the cloud provider queries the getDeviceNameFromMount() to return the path of the volume mounted. Since getDeviceNameFromMount() queries the filesystem to get the mount references, it returns a volume path "[datastore]040kubevols/volume". Later the kubelet searches for this volume path in both the actual and desired states. Th actual and desired states contains volume with path "[datastore] kubevols/volume". So, it couldn't find such volume path and therefore kubernetes stalls unable to make any progress further similar to one described in . Also similar issue described in What you expected to happen: All the pods in the recreated deployment comes up. How to reproduce it (as minimally and precisely as possible): See above. Anything else do we need to know: None
cc


From etcd3: May need to retry poststarthook a few times to avoid bootstrapping failures like this.


adler when brute-forced turns out to break really fast (). There are a couple of places we use it in the repo: We should probably replace it with fnv. I tested fnv and it seems to be slightly more expensive but also a lot more stable than adler:
Just work out a patch to fix this, please take a look.
assigning this to you since the only place left to use adler is in and it's a legacy helper.
/assign
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. If the required changes are not made within 3 days, the issue will be moved out of the v1.8 milestone. kind: Must specify at most one of ['kind/bug', 'kind/feature', 'kind/cleanup']. priority: Must specify at most one of ['priority/critical-urgent', 'priority/important-soon', 'priority/important-longterm']. <detailsAdditional instructions available <a href=""here</a</details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This was removed in


Our scalability tests started failing on Thu last week: https://k8s- [250 run passed, 251 failed and since then everything failed] I started looking into it and it seems that something happened to how we are handling watches (i.e. I'm seeing a huge number of watches failing with "err too old" errors (which I didn't see before).
As an example in run 258 of kubemark-scale, I'm seeing this log in the test (after 5 minutes of running test); This is extremely strange and unexpected. I looked into apiserver logs, and there is expected number of POST and PUT operations on pods (and no DELETE and PATCH operations until then). Until then, there were alltogethere ~60.000 operations on pods. However, in 5000-node cluster, the size of watchcache of pods is supposed to be 250.000, which means that at that moment we should have all the history from the beginning in watch cache. And still we are observing those errors. Something must have changed around that recently...
As an example, logs from apiserver from a given node: This means something is really wrong: the first watch started at 45, then after it finished, we tried starting watch from rv= and it failed with "rv too old" the list, returned rv= from which we started the next watch. If the watch cache size for pods would be 250.000 (50*cluster size), there there is no way that with current rv=, the rv= would be outside history window...
I had a hypothesis that it might be the problem of breaking watch to etcd. But in that case there should be a warning "Terminating all watchers" log in apiserver, and there is no such log. It also seems that the size of cache is being set correctly...
So I tried running large kubemark locally and it seems that the problem is that the size of cache is too small. I few more logs, and here is what I'm seeing: So it seems that we are correctly initializing cachesizes, but then watch cache capacity is 100 (which explains why we are seeing all those "rv too old" errors. I will try to debug why the capacity is so small.
OK - I found it. contains the fix.


The description of says: However the implementation will interpret the name as a regex. So for example "..." will match any three-letter string. This is not what I understand by "exact match".
Will open an PR fix this comment


If in line line is executed also for step 2.2.1 if the tool failed during first run of upgrade. I guess we should also check for the step and compare it to version.


At the very least it should be clear when (timestamp) the migration started and finished.


https://k8s- do you know the statefulset owners?
cc
I did not see this happening until recently.
The controller misses that the first pod transitioned to unready and proceeds to create the second pod. The test uses a client to hit the api server and will scale up the statefulset as soon as the broken pod is observed but the controller uses caches so it may observe the scaled up statefulset quicker than the updated pod.
I will update the test to ensure that we wait until the unreadiness is observed on StatefulSet.Status.Replicas field prior to the scaling operation
SGTM


https://k8s- replica set controller maybe?
The rc seems to be updated almost a second after the failure. Maybe the test needs to retry
And the real log for the failure is
/assign flakes PRs at least once a day The can filter down to focus on a single test:
For when you have some time.
Reassign to you
Output of has been change in 1.6. That's why upgrade tests are failing. But I'm not sure why it is not failing 100% from the triage board. Do you know why? Output of 1.5 or earlier: Output in 1.6 or later:
The timeout is now. Is it reasonable to give it a little more time?
That timeout waits for the pods to run based on the API server. The rc controller is using its own cache so it may lag a bit before it actually updates the rc status. The test needs to retry for some time if the output we get back from describe does not match.
In other word, should also have its own retries.
SGTM. Going to send a fix today.
FYI, this test continues to flake PRs at least once per day:
should it show up in
Hm, this seems to fail in the upgrade tests. this failure seems to be related to an image name change as you already pointed out before. Mind finding what changed the test to use that image and fix the image name?
It is actually the key name has been changed in the key-value pair, not the value. As I pointed out in before expected key: now expected key:
Hm, this is unfortunate. These kubectl tests should really be unit tests.
the 24 hr test history is largely focused on failures right now, and this is highlighting flakes specifically (aka which tests are both passing and failing at the same commit the most often). Please follow the reports I'm sending to kubernetes-dev and related sigs. Also you can generate this info yourself using PS: also try out for finding failures (I'm sending out a separate email highlighting the biggest sources of failure). PPS: we are in the process of creating infrastructure to automate creating these reports, but automation never works as well as a fellow human asking for attention.
We still need to patch the release-1.4 and release-1.5 branch to make the upgrade tests passing. I will send PRs later this week.
We didn't fix
The change mentioned in was introduced in 1.6. The change covers describer and kubectl e2e test. After discussing with we think we are not going to backport the changes to 1.4 and 1.5, instead we are going to skip the tests in version skew cases. WDYT?
should fix this issue
looks like the PR is resolved, should we close?
Please wait until fix to be cherrypick into 1.4.
Sure! Which PR is this?
Is it possible to cherrypick to 1.4 by just changing the milestone to 1.4?
no you need to create a new PR with the target branch of 1.4
Ready to close?
Yup. /close


We need to gracefully handle the case when a Deployment specifies an absolute value of MaxUnavailable larger than its spec.replicas. I don't think we want to add validation for this, simply collapsing maxUnavailable to spec.replicas would suffice.
Are you working on this? I am glad to take this if not.
feel free to work on it


When patching, in the type decoded from ToUnstructured mismatch with the one decoded form json.Unmarshal for numbers. Example: port: 80 in original object has type int64 port: 80 in original object has type float64 This is the root cause of


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Kubernetes version (use ): master What happened: The functionality is hidden in a large function, and not explicit, testable. What you expected to happen: The logic of should read more clearly, with tests for individual filters.
has a PR for this .
Few more things include removing limits in entries like
yeah, im not sure why the thing is there, after all they are just appending to the slice and the limit wont have an effect , from what i can tell, will it? probably good idea to remove unless im missing something.
Its part of part of PR too.


Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): container-os Kernel (e.g. ): Linux gke-wordpress-cluster-default-pool--m764 4.4.21+ SMP Fri Feb 17 15:34:45 PST 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.60GHz GenuineIntel GNU/Linux Install tools: Others: What happened: ` I was able to bring up wordpress okay the first time, except GKE wasn't creating loadbalancer ips due to a quota issue, which I resolved, note at this point the mysql pod was up and had attached to it's volume. Upon deleting the wordpress deployment and creating it again I started getting the above errors. I deleted the mysql pod as well and brought it up again to see that it had the same issue. The volumes are backed by a gluster cluster on GCE. Looking at the brick logs on one of the gluster nodes I see I've tried restarting kubelet on the node. I can't find the kubelet log file on any of gke nodes. And I don't know how to get the kube-controller and apiserver logs from the master (GKE). I suspect it's a failing of the glusterfs client on the gke nodes? What you expected to happen: Deployment to mount the volumes successfully. How to reproduce it (as minimally and precisely as possible): Not sure how, but I've run into this intermittently Anything else we need to know: The volumes are backed by gluster
/cc
If you delete pod and create pods soon, it is possible that you could see some errors like "timeout expired waiting for volumes to attach/mount" temporally. Could you wait for a few more minutes to see whether the pods could be started correctly?
I actually tried that, deleted the pods, then waited 5 minutes and recreated them with the same error. Maybe it wasn't long enough
Can you post output of ? It may have some other events for failure..
I deleted the cluster this afternoon, but I checked constantly and those were the only errors, Timeout on mount and a failed to sync error
If you use level 4 log, it should also give the detailed messages of failed mount so that we can figure out why it failed.
Can you email me your project name, cluster name, and zone information? I can take a look at the master logs to see if there's anything useful there.
This is starting to happen more frequently..
could you provide more details about in what situation it happened frequently? Kubelet log could give us more information. If you use GKE cluster, run "journctlctl -u kubelet /tmp/log" to get the log. You could also share with us the log if it is ok.
This seems to be the relevant part of the log I re-created that and my pods sprang back up. Weird thing is I had created that, along with the pvcs and pvs but seems it's somehow getting deleted. There's almost certainly certain I'm doing wrong but I'm not sure what. ]
HI Jonathan, can you clarify what you had to recreate?
The glusterfs endpoints , Michelle Au <:
I have a very similar problem running OpenShift 3.4 in AWS EC2 doing the same exact thing. Brand new OSE install [root master]# oc version oc v3.4.1.12 kubernetes v1.4.0+ features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.31.40.126:8443 openshift v3.4.1.12 kubernetes v1.4.0+ POD in question is the docker-registry (kind of important for OSE to work) I've followed this to see if it would fix the issue: But no luck or change. Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] then: Unable to mount volumes for pod "docker-registry-3-afbbt_default(-1a45-11e7-abf8-)": timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] I have no idea what log(s) to look at, this was of no help when I deployed a new docker-registry: oc logs -f dc/docker-registry Any help or guidance would be greatly appreciated as this is a blocker issue for me. thanks and best wishes,
are you using glusterfs for registry-storage volume or other type of volume? Could you share us your kubelet log if possible over slack Thanks!
Hi Jing Xu, Would you know where this log would be? Or is the name kublet<somethingso I may find it for you? I just requested to join the slack channel, as soon as I'm signed in I will share the log. Probably tomorrow as I'm on the east coast USA Thank you! Joel
For you are using GKE with GCI image, you can use "journalctl -u kubelet " to get the log. Otherwise, it should be in " , shepp67 <:
I also experienced this. Kubernetes version:Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Install tools: custom That kubernetes worker vm actually had volumes attached. I saw it on OpenStack dashboard. But: and actually: Besides, this is a new worker vm. I have other worker vms already running, using cinder volumes and there were no such errors. I use Helm to deploy this Pod. I have deployed it earlier and removed it. But it was 2 days ago. I found the solution here: . On that worker vm I run: Edit: or upgrade Kubernetes to = 1.6.0-beta.3 I don't know if this is the solution to issue.
I'm seeing this same error on all my manually launched private Ubuntu based clusters (6), running 1.5.3 and 1.6.0. In my case the messages reference only stopped pods launched by CronJob Jobs. The messages indicate issues mounting every volume in the pod including PVs and node-local volumes. These pods, and future ones launched by these CronJobs, are fully successful when they run. It is only after they run to successful completion and exit that these messages show up. Glancing at the kubelet code, this appears to be coming out of some housekeeping/synch loop.
I've identified a bug and am working on a fix for when "Unable to mount volumes for pod..." + "Error syncing pod" happen indefinitely BUT the pod is actually running just fine, as observed by . For others, if you see this message but your pod is Running, you can safely ignore it. It's caused by a small limitation of the reconstruction of state (what volumes are mounted) so there is a chance you will see this only if you restart kubelet while your pods had volumes mounted. correction: you say they even Completed, they weren't just Running fine. So yeah, it's just misleading logging and not really an indication of anything breaking
Because the error message is very generic, I think there might be different problems mentioned in this thread. 's issue could be fixed by 's PR are you still have any issues with mount? could you also let us know whether you still have the issue? Thanks! Jing
Hi my issue was related to which helped me sort out. I was creating the gluster endpoints without a service and turns out the kube-controller deletes all such endpoints that are not attached to a service at regular intervals. So after every so often when a pod went down and tried to come up again and mount its respective pv it failed because it couldn't connect to gluster. The fix is to create a service for the gluster endpoints.
I also just ran into such timeout issue: After running a load test against my postgres backed web application, the database didn't respond anymore for still unknown reasons. According to kubernetes it was up and running The log showed a bunch of these messages. Not sure yet why but possibly it couldn't access the volume anymore: Short after that, the pod entered "error" state: The event log showed: This error disappeared after a few minutes (not sure how long exactly, but it took a while), but then I got the same error as described here: I couldn't figure out anything more on my own. Nothing in the syslog, nothing more than this in the kubelet log. Finally I rebooted the machine which fixed the issue.
cc do you know what error of the network not ready? It might be helpful if you could share the kubelet log with us to debug. Thanks!
AFAIK, the "network not ready error" is benign, kubelet will retry after this error () cc
I hope I didn't miss some secrets in there, quite a talky log. But here it is: The database didn't respond anymore because it ran OOM and was killed, as also shown in the log. Although it appears like it was OOM killed after it became unresponsive in the log.. Either way, a bunch of other things got killed too, so granted - "not ideal" situation :). Yet it shouldn't drop the volume and should re-attach it.
I think you are hitting the issue mentioned above. He has a PR to fix it. I will make sure it can merge soon. Thanks!
Great, thanks! Do you know roughly when this will be available in GCE?
you mean GKE or GCE? For GKE it will take a little bit longer. On GCE, when the PR gets merged, the fix should be available when the next release is out (I think should be soon)
Running on GKE. I'd rather not launch my product with this bug around. Is there a workaround for it or something I can monitor for? Or maybe it could get hot fixed on GKE? If there is no workaround, this effectively renders GKE useless for stateful stuff.
It's hard to say if volumes are actually being dropped/unmounted or if the FailedMount errors are wrong & misleading, so I wouldn't go so far as to say it's broken because of some volume issues. It could be something else entirely. If nothing else, the PR jing's referring to will clean up the event noise and help us debug the root cause. In the meantime, I wouldn't trust the FailedMount events to be accurate, so if your applications are unresponsive, there may be other causes worth considering
Well, I only can say that my volumes were dropped and it didn't recover on it's own (only after reboot). Dropped a seen by kubernetes, dunno if the volumes were still attached but they weren't usable by my scheduled database. To be clear: I'm not "complaining" and this was just test/R&D, but it would this is the scariest kind of issue for me to happen in prod, given that it's a managed service where I can't find a workaround on my own. Rebooting fixed it in my case, but that's a big hammer..
You don't need to reboot the node. Delete the pod and restart it should work for this issue. Is this workaround ok for your case? , discordianfish <:
Good to know, thanks! Will try this next time.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hi, I'm facing the same issue with k8s 1.9, installed by kops on AWS. volume created successfully on AWS and attached to the right node, but the pod is failed to start.
/remove-lifecycle stale
could you please provide more details about your issue? What are the steps to reproduce it? Thanks!
yeah can you open a new bug for EBS and cc me with details? I was hoping we fixed most of EBS problems in 1.9 - I will be more than happy to walk through and find what is causing volume to not mount. As I understand this is different from typical EBS problems where volume fails to attach. In your case, volume is being attached to right node but it is not being seen by kubelet.
I found the issue, and it was the fact that i used c5 instances with nvme, this is why the volume never been attached as k8s looking for /dev/xvdXX and with nvme that path is different. I did notice that in 1.9 k8s should support nvme natively but it seems that it's still not stable. thanks
oh weird. You should open a bug about NVMe support then. I was thinking it would just work.
cc Is there an issue related to nvme support? please let us know any issues you still have with 1.9. Thanks!
This is nvme on AWS, not related to the work is doing on gce.
Thanks Definitely related to NVMe support. Ran into this trying to mount a gp2 EBS volume on a node which was an m5 EC2 instance.
I think I am seeing like this with , but unsure of "why" at the moment. It would seem works perfectly fine.
did you guys had a chance to open a BUG for NVMe support on EBS? If you do - please paste output of and PersistentVolume spec. We can't fix it unless we know bit more. :(
sorry, we worked around this by reverting back to m4 class instances. It should be simple enough to reproduce however... create a k8s cluster w/ m5 EBS backed nodes and create a simple pod w/ a persistent volume. You'll notice that the pod will never launch and you should be able to debug further. If I do run into this again I'll be sure to get some more data for you.
I too have the error, error syncing pod. kubectl describe pod kafka1-kafka-0 .... .... .... Warning FailedSync 5m (x141 over 35m) kubelet, 10.186.. Error syncing pod Warning BackOff 57s (x164 over 35m) kubelet, 10.186.. Back-off restarting failed container kubectl logs kafka1-kafka-0 -p export KAFKABROKERID=kafka1-kafka-0 export KAFKAADVERTISEDLISTENERS=PLAINTEXT://172.30.*.:9092 /etc/confluent/docker/run sh: 3: /etc/confluent/docker/run: not found


Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.


In order to support running tests in parallel, we randomized NodePort selection. There appears to be some evidence that this is not sufficient and is causing e2e test flakes: https://k8s-
Hi, As a need contributor, I'm interesting of solving this issue. I already check the current code, and propose to retry the service creation until success or all possible ports (between FederatedSvcNodePortFirst and FederatedSvcNodePortLast) have been tried. If this approach is fine for you, I can submit a PR soon.
How do you plan to check if the NodePorts are available? In our presubmit test environment, tests run across many instances of federation control planes in parallel, but share the same underlying clusters. So this approach might be viable depending on how and where you plan to check for these used NodePorts.
my approach for getting an available port is a "Dummy" approach: create a list of possible available port, get one of them randomly, then try to create the Service. If Service creation failed. try to use another possible available port, until service creation success or all possible available ports have been tested.
Sounds reasonable to me. Go for it. A couple of caveats: need to create a real list, you could just use the range boundaries. careful about service creation failure. You need to check if the service shard creation failed in the underlying clusters and rollback all those shards before retrying.
thanks for the caveats, I will do soon a PR.
Hi, what is the state of this issue? Is working in this issues with the others issues related to e2e tests?
has a PR to fix this issue -


We copy in BUILD files from the saved directories. The BUILD files should be preserved. This was hit in


The web and instructions for [Installing and Setting Up kubectl] need a slight tuneup. The homebrew instructions should reflect a few different options for Bash on macOS. bash-completion is for Bash 3.2 (macOS default), and bash-completion is for Bash 4.1+ (upgraded GNU Bash) I lost an hour of my life to this; I hope this helps others. What happened: By following the homebrew install instructions on this page, then entering and tabbing, this is the output: What you expected to happen: By following the same path as above this should be the output: Anything else we need to know: Currently the instructions assume the default Bash on a vanilla macOS install; while I understand this being most-likely scenario, the default version of Bash is shamefully old. After reviewing to options in homebrew it appears there are 2: -completion is for Bash 3.2 and -completion is for Bash 4.1+ For those of us that cannot live without all the options, this is how we do: Install: Configure Then the needs to be updated to reflect this changes: TEST a new tab (the is sourced-in) and tab twice We then get the expected result. The requested details follow: Kubernetes version (use ): Environment: personal/dev Cloud provider or hardware configuration: NA OS (e.g. from /etc/os-release): macOS Sierra Kernel (e.g. ): Install tools: homebrew [Installing and Setting Up kubectl]
So were you running bash 3.2 or bash 4 prior to this? It looks like the instructions are for bash-completion which should be for the bash installed by default in macOS. /assign
I always install the latest Bash, currently 4.4.12(1)-release. Apple's is too old and I'm not going live in their cave; most people won't - that's why homebrew is a huge project. Correct, the instructions are for the stock version of Bash (presently 3.2.57(1)-release). My suggestion is to include instructions for both versions. Mostly to keep others from wasting the same time. I've already done the testing so folding in the new details should be pretty straight-forward.
I've opened to update the kubernetes documentation, and to potentially update how homebrew works to make this clearer.
Docs have been updated: /close


seen in https://k8s-
Should be fixed by
I believe this is fixed?
Yes, I missed this one.


https://k8s- Failed: TestServeExecInContainer {server}
Should be fixed by
https://k8s- Failed: TestServeExecInContainer {server}


On , will fail to evaluate a filter when the member does not exist: For instance, if I were to execute Where not all in that namespace had the annotation, this would fail with the error message: This is confusing as it leads me to think more than one item was selected, so while it is not technically incorrect the error message is misleading. I would furthermore expect to more gracefully handle this case -- a missing here means that the filter should fail, not error.
What do you mean by ? How is it different from ?
As in, don't add that item to the list of items on which the following access is made. When I do: And there is no annotation on , I just get nothing back -- no error, just nothing. Similarly, i would have expected the filtered expression to give me the for all of the pods that have an annotation with value . That, anyway is what I understand the JSONPath syntax to mean. When you write You are saying:
cc since you authored and I'm not sure how we want to fix this.
Taking a look now
it looks like you need to do this (note that the annotations portion is wrapped by ): I wrote a unit test and it worked as expected. PTAL!
I'm pretty sure that was a typo on my end but let me check -- do you have a reproducer script?
I see an entirely different error today... weird:
Ok so you have 1 object with annotations, and 1 without. Let me look into that case.
Let me add another annotation to
No difference if both have annotations:
Ok, I'll have to test using kubectl
I'm the Origin bundled but I don't think there are any patches on JSONPath there:
I'm able to reproduce in the unit test when 1 object has annotations and the other does not, if allowMissingKeys is true for the jsonpath parser.
I see. Let me know if I can be of more help.
I think I have a fix. PR coming soon.
Possible


Kubernetes version (use ): Kubernetes HEAD Environment: Cloud provider or hardware configuration: linux OS (e.g. from /etc/os-release): Kernel (e.g. ):NAME=Fedora25 What happened: apiserver panic What you expected to happen: No panic How to reproduce it (as minimally and precisely as possible): Testcase is going to be sent as a PR
Fixed by


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? NO Note: Please file issues for subcomponents under the appropriate repo What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubernetes, azure file, chinacloudapi Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Others: What happened: We try to mount azure file to pod, and get the following error That is because azure china's storage DNS suffix is rather than What you expected to happen: Kubernetes should provide some method to let user specify storage dns suffix, so we can use Azure file volume How to reproduce it (as minimally and precisely as possible): Any Azure cluster in china can reproduce it. Anything else we need to know: kubernetes has hard coded in shows a relative issue
could you help confirm this?
One possible way is configuring DNS suffix via annotations of PV. If thinks this way okay, I'd like to create a PR to fix it.
azure file was implemented without using resource group or azure cloud provider. The endpoint can be found at One fix is to use azure cloud provider and choose the file service endpoint from azure environment as I did in blob service . Let me know if you can help.
I'm not familiar with azure cloud provider and volume codes. I'd like to try to dig into it to see if I can fix as the way you suggested.


Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Kubernetes version (use ): kubernetes/master using CRI with docker What happened: Create a pod that uses a readiness probe that does an exec to command that doesnt exist See the following in logs: What you expected to happen: No error in log, expected the event to properly be thrown. How to reproduce it (as minimally and precisely as possible): See above. Anything else we need to know: Nope.
this is a regression.
debugging, will post a fix once found.
Other reports: ,
-- thanks for the other reports. at least now we have a reliable reproduction.
The problem appears to be the following: has a that correlates containers to a . It uses the type . event recorder library we use in the prober works with . If it doesn't get one of those, it expects the passed in object to implement List. As a result, we are passing a v1.ObjectReference the recorder was not expecting.
The bug may have been introduced in , which was there since v1.6.0.
Opened for a fix.
Keep this open until 1.6 is patched.
/remove-priority P0 /priority critical-urgent (I'm not actually sure this is the right priority, just trying to remove the old priority/PN labels)
Closing since has been merged.


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): selflink kubelet selflink default selflink manifest selflink Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): admin kubectl version Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.7", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-27T10:00:30Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.7", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-27T09:42:05Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: AWS OS: Debian GNU/Linux 8 (jessie) Kernel: Linux ip-10-150-109-115 4.4.41-k8s SMP Mon Jan 9 15:34:39 UTC 2017 x86_64 GNU/Linux Install tools: kops 1.5.3 Others: What happened: When running the selfLink for pods started from manifests were incorrect. Example: What you expected to happen: To receive a correct selfLink: Example: How to reproduce it (as minimally and precisely as possible): I believe simply using a pod manifest to start a pod will cause this issue as kubelet seems to rely on defaults when starting pods from manifests. Anything else we need to know: I believe the below linked line needs to be corrected:
/assign I reproduced it. Will fix. Thanks for reporting!
GitHub didn't allow me to assign the following users: xiangpengzhao. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details


When a node is deleting a pod, the NodeRestriction admission plugin ensures the pod is bound to the node. It first checks the server cache for the pod. In the case where a pod was created and immediately deleted, it may not yet exist in the server cache. Before forbidding, we should do a live lookup. https://k8s- cc


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): Note: Please file issues for subcomponents under the appropriate repo What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: Currently, the YAML template for both npd and fluentd service do not have toleration, this may cause the daemonset cannot be started on a node if it is already selected to run , it is better to add toleration for both npd and fluentd so as to make sure those daemonset can run on the nodes even if it is for . All other daemonset in kubernetes also have such tolerations. What you expected to happen: Should add toleration for both npd and fluentd. How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
You sent PRs to fix this, right? Can this be closed now?
the PR was not committed yet, can you help lgtm and approve it?


We have moved to non-global schema in a number of places in 1.6 and 1.7. It is easy to forget registering the unversioned types like , leading to encoding errors in error cases. The registration of the unversioned types should be automatic as for the types, i.e. in .
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
do we need to close out this issue? (since referenced PR has merged)
Fixed by


We should do this after 1.7. xref /sig testing cc
Is this the task that needs completed - Add support to Ubuntu e2e test-infra
That is different, this is about running GPU tests as part of PR pre-submit tests. Right now we skip them because they are "[Serial]".
I can work on this issue in conjunction with . Is this issue just a matter of adding GPU tests to a different suite or are there more subtasks?
Anymore progress on this?
Thanks for the reminder. I just submitted an internal request for quota to run these tests. Once that is approved, I will add a PR in test-infra to run GPU tests on all k/k PRs. It will be non-blocking initially and then become blocking later on.
Awesome, thanks!


xref: /sig node /area test
I am working on this.
Can you update this with the types of tests that you are adding? Is this similar to ?
No, this is for running the existing GPU e2e tests on GKE which are currently running on only GCE.
The referenced PRs appear to have merged, closing.


several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): local-up- Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: KVM OS (e.g. from /etc/os-release): fedora 25 Kernel (e.g. ): uname -a Linux k8sbuild-master 4.11.3-200.fc25.x8664 SMP Thu May 25 19:03:07 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others: What happened: local-up- fails to create kube-dns and StorageClass. What you expected to happen: local-up- is successfully executed. How to reproduce it (as minimally and precisely as possible): 1) Clone kubernetes from master branch 2) run local-up- $ export ENABLEHOSTPATHPROVISIONER="true"; ./hack/local-up- If you already have kubernetes , try this 1) make cleanmeta 2) run local-up- $ export ENABLEHOSTPATHPROVISIONER="true"; ./hack/local-up- Anything else we need to know: I think this problem is timing issue because sleeping 3 seconds before startkubedns can avoid this problem.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig testing /sig storage /sig network
/sig apps
I met the same issue we should add a function to check if all api-versions are up


Every time watch is reset, event exporter exports an entry about it to Stackdriver. To make it less irritating, this has to be done once per container run Related image PR: /cc


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If node is unreachable, NodeController only set , , to unknown; it's better to also set NetworkUnavailable & InodePressure to unknown.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
Isn't a known condition if node is unreachable?
is the status of container, e.g. CNI is not ready. If node is unreachable, it should be Unknown.
Thanks for explanation!
will not set NetworkUnavailable & InodePressure to unknown, refer to .


There's a problem b/c pods are created to run on all nodes, but run sequentially, effectively multiplying testing time


/kind bug What happened: A pod with an empty namespace field was mutated by a pod preset in a different namespace. What you expected to happen: Pod presets should only operate on pods in the same namespace. How to reproduce it (as minimally and precisely as possible): Submit a pod with an empty namespace field and observe pod preset mutation. Anything else we need to know?: This is a fix for an alpha non-default feature. Fix submitted in
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When trying to list object using field spec.SchedulerName, we are getting the below error message. Failed to list : field label not supported: spec.schedulerName What you expected to happen: To be able to fetch pods using the list and watch based on the field 'spec.schedulerName' How to reproduce it (as minimally and precisely as possible): Create a simple controller to with a list and watch method. Set the FieldSelector ( in the ListOptions structure ) in the ListFunc and the WatchFunc to 'spec.SchedulerName=custome-SchedulerName' Running this controller yields the above error. Anything else we need to know?: Description: Unable to fetch pods based on the field 'spec.schedulerName'. This was in 1.6 version, for multi scheduler support in kubernetes. We need to be able to fetch pods 1) Assigned to a particular custom scheduler. 2) Should be able to fetch the state transition of the pods scheduled by a custom scheduler using the fields selector. Whats the correct way to use 'schedulerName' fields for fetching objects? Or is there any other way for the custom scheduler to determine the above information from the kubernetes? This is an urgent issue, any quick help/suggestion will be of great help. Environment: Kubernetes version : Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-14T01:48:01Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Install tools: kubeadm
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
/cc
cc
SIG API Machinery doesn't own the contents of the API--we only need to be involved if the mechanism needs to be general.
Given the error message mentioned in the bug report at the top of this thread, do you have a suggestion of what we forgot to do when we the SchedulerName field to the PodSpec? i.e. to enable field selection on the schedulerName field?
/cc me


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig storage What happened: Offshoot of . Let's change the tests to use the programmatic API instead of gcloud CLI.
Just confirming that this issue is referring only to the 3 gcloud CLI calls in
yup
PR:


Seeing flakes on master: https://k8s- https://k8s- and on unrelated PRs: https://k8s-
I was able to reproduce this locally (although it's very hard to trigger). Initial thoughts are that it's different from ; in this case, the actual sync is instant (millseconds) but the CRD created in the test fails to appear in discovery calls over the course of repeated attempts every 5 seconds for the remainder of the test. Normally, the type shows up in discovery almost immediately after creation.
Looks like PR might be hitting a related flaky test. I get 6 failures in 60 runs from the command I'm super new around here so thought it prudent to add the information I have found before attempting to dig deeper. I followed instructions in to build from within directory If the test failure logs are of any use to anyone just message and I'll post one of them.
Thanks. I've also been able to reproduce locally and am debugging it today. I did have to hack up the integration test setup to get apiserver logs in this context. I've also opened to explore making the integration test script more configurable with regards to logging.
I was wrong about the discovery aspect; the CRD is coming back from discovery, but the restMapper fails to map it. I also notice we're using DiscoveryInterface.ServerPreferredResources which is not actually implemented with a cache in memCacheClient.
okay, now I'm getting somewhere. I made a which exercises discovery and REST mapping without the GC in the picture at all, and have been able to reproduce the flake 1 time out of 40 runs so far with .
First problem is pretty clear in hindsight; we reset the restmapper, then get deletable resources using the discovery client directly, then sync based on the resources from discovery. The direct discovery call doesn't use the client's internal cache, and so the restmapper might be out of sync with the subsequent discovery call, causing us to sync with resources not yet represented in the restmapper's cache. That explains the isolated flake in the test I made, but doesn't yet explain why the restmapper would continue to fail on subsequent sync calls (since the nest restmapper reset would usually bring it up to date with the resources returned from the direct discovery call).
I'm out tomorrow, so a quick update here in case somebody else looks into it... can reproduce the issue where DeferredDiscoveryRESTMapper sometimes becomes unable to map the CRD resource, even after the type is available from discovery. I've also ruled out the memCacheClient by replacing it with a non-caching implementation like: And using that in the setup like: The issue can still be reproduced that way. Looks like more debugging of DeferredDiscoveryRESTMapper itself is required.
Finally tracked this down. Custom resource kinds are always given a plural suffix of "s". If the singular name ends in "s", the assumes the plural form is an "es" suffix, which will never match the generated kind. This explains the intermittent nature of the failure (the randomly generated names often happen to match the assumptions of the rest mapper).
is the root cause
Good effort


In Cinder cloud provider openstack Inside of AttachDisk function, it issues DetachDisk if it find out the the volume is currently attached to a different instance. This is very dangerous because it is very likely the volume is still mounted to that instance. The correct behavior should be fail this AttachDisk call. K8s attachdetach controller should handle unmount and detach volume safely.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
cc


Making an API request for table output and specifying the parameter causes a server error in parameter conversion, and an inaccurate error message in response: server log: E0804 09:41:14. ] apiserver received an error that is not an metav1.Status: converting (url.Values).[]string to (v1alpha1.TableOptions).v1alpha1.IncludeObjectPolicy: couldn't copy '[]string' into 'v1alpha1.IncludeObjectPolicy'; didn't understand types
Ugh, typed strings.


Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Unit test error message match the conventional Go style: For example: if got, want := set.Status.Replicas, 3; got != want { t.Errorf("set.Status.Replicas = %v; want %v", got, want) }
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing
/assign
/reopen


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: e2e test panic in test What you expected to happen: no panic How to reproduce it (as minimally and precisely as possible): start " [Feature:ManualPerformance] should be able to handle 3 pods per node" e2e test on a 3-node cluster Anything else we need to know?: panic info : ! After some digging find why. Because the cluster size is 3 that d is 0 in sleepUpTo func which causes rand.Int63n panic happy to fix this. Environment: Kubernetes version (use ):1.7.3 Cloud provider or hardware configuration*: bare-metal OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): 16.04 Install tools: kubeadm Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig Testing
just add a sanity check is fine I think
It seems that the count of pods is 0.
pod count is not 0, is 9(33)
If totalPods less than throughput, in our case, totalPods is 9, throughput is 10, the creatingTime will be 0, which makes rand.Int63n panic
or throughput is getting from env LOADTESTTHROUGHPUT, safe to add a sanity check
If totalPods less than throughput, in our case, totalPods is 9, throughput is 10, the creatingTime will be 0, which makes rand.Int63n panic agree.


seems to belong to sig node PTAL and fix ASAP
/assign as it seems like having something to do with


We allow users to specify in the Service object. If the user later removes this from the Service Spec, the controller will fail to sync. For example: creates an LB for Service.Spec.LoadBalancerIP = "1.1.1.1" with a user-owned static IP address. updates the Service to remove the requested LoadBalancerIP tries to used by the existing forwarding rule (i.e., "1.1.1.1"). It encounters a conflict. Controller assumes it owns the IP and by name but fails. We can fix the controller to continue syncing and using the user-owned IP for the LB, or create a new address for the LB. The former seems less disruptive to me, but the downside is that if user will not be able to let other components use this IP (i.e., 1.1.1.1) unless they explicitly delete the Service. I think it's ok but would like more opinions. thoughts?
If a service is updated to remove a static address, then it seems like that would be a deliberate move on the part of the user. It seems like it should probably release the address and provision a dynamic one, even if this is disruptive. How do we handle the reverse? If a user has an already existing LoadBalancer service, and they add the after the fact? Would this release the dynamic address and recreate the forwarding rule with the reserved static address?
It seems like the creating a new address makes sense here... General comment: given the combinations (and the bugs we have been running into), I think this is a problem that all of the potential state transitions for the load balancer have not been fully spec'd out.
Creating a new address seems nice but the implementation is a nightmare. The service controller doesn't give you a previous Service spec, so you don't know what's changed (controller might not know either if spec was modified while offline). Given a desired service spec without a how do you decide to ditch your current IP and grab a new one? If we were to call to evaluate the situation, it's highly likely users will accidentally cause LBs to change IPs during typical usage. Imagine a user has a dynamically allocated IP for a service, but they want to promote it for safe keeping. The controller would think it needs to surrender the IP to the user's reservation. The only correct process would be to set before promoting the address; however, that will surely be a missed point in docs and result in unhappy users. I lean towards the LB continuing use of the IP. Besides the easy option of deleting & re-creating the service, they could change the to another static address then demote the address. I'm not too familiar with the ideological boundaries of controllers and spec, so let me know if this crosses the line. What if the service controller set to the created IP on first sync if the IP wasn't pre-specified? The controller could rely on loadBalancerIP to always indicate which IP to use. If the field is empty (blanked by user), then it's a signal to get a new IP.
By spec, I mean we should write down the careful reasoning that you just wrote in a permanent place :-)
would it then make sense for to be immutable on an already created service?
+1 +1 Was thinking the same. Unlike the many annotations we support, updates to this actual field could be validated and blocked at the apiserver... Changing IP is disruptive anyway. Could we simply ask the user to re-create the Service?
Of course, this could be an API-breaking change, and other cloud providers may actually be supporting this properly.
I just ran across this today, but in a more likely (IMO) case. 1) create the Service, type=LoadBalancer 2) let the IP be auto-allocated 3) manually promote that IP to static and give it a meaningful name 4) observe errors from service controller trying to allocate that IP Since my Service does not spec "loadBalancerIP", the question is, do we: a) use the IP, but leave it as static b) use the IP, but demote it back to ephemeral c) allocate and use a new IP C is completely hostile. B is subtly hostile. A seems correct. Can it be implemented? If we find that the IP exists but the name of it is not what we expect, we should treat it as user-provided and leave it alone. This is still nasty because it logs a failure in the GCP activities log for every loop, but I don't know how to avoid that without getting a list of IPs and scanning for the IP we want.
Filed for the looping.
+1 as long as we can rely on the "naming" of the address to decide whether the controller owns the address and can promote/demote it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Still relevant
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Still relevant
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
any update on this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.


When using IP aliases on GCE, use a secondary range rather than subnetwork to reserve cluster IPs.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals


Test Cases: [ ] Egress blocking based on cidr to a set of pods [ ] Egress blocking based on namespace to a set of pods [ ] Egress blocking based on a set of pods to another set of pods
Update: I'm working to try to enable these new features in a few CNI plug-ins so the tests can run locally.
Can we please move out of 1.8 milestone?
/label status/approved-for-milestone
/kind cleanup
/remove-kind bug
Those labels are not set on the issue: <detailsIn response to : /remove-kind bug Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind cleanup
/status approved-for-milestone /kind cleanup /remove-kind bug
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Adding tests, refactoring, fixing old bugs. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Can we please move out of 1.9 milestone? :)
this will be partially fixed with . I think it is safe to keep in the milestone.


bumped the vendored docker library and introduced a regression where pods with images that don't contain an explicit "" repo path result in failures. Results in the following error: Pod gets shows that the image has been pulled, though not by that name Changing the image to in the pod spec corrects the issue. I think the new code that is changing the behavior is somewhere in here is a new function brought in by the bump. If I checkout the commit before the bump, the problem does not occur. related:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
taking a look now
What docker version are you using?
Do you have any e2e tests that are failing? We run a lot of tests with image "busybox"
Didn't think about it before, but I guess 1.12 is the supported version for 1.8 eh? Let me try it on 1.12.
Hm, weird. We also have cluster e2e test coverage for 1.13. Many test are using directly from this function
I installed a completely fresh box with docker 1.12 and could still recreate. I have no explanation :confused: from kubelet built from upstream/release-1.8
This is one of our e2e test node log: You could find tons of . One thing I don't understand on your side is that, logs in our test are always: Why on your node it's ? Where does that image come from? What if you delete it?
does this mean that upgrades will fail if the local image names need to change?
Seems not. Based on the following code, it's fine to pull/inspect with new reference when we previously pulled with old reference. Could not reproduce this.
I could not reproduce with the code above either.
For any image which has no tag, kubelet will add a default tag , and default repo to it. So will change to and send to dockerd to pull. But dockerd knows and are the same thing, so should return right instead of not found. (as demo) Can you make sure your dockerd is the a version compatible with k8s 1.8?
Just ran into this. If you up the kubelet's log level to 4, you'll see a "not found" error from For example: If we look at the code in that helper function, we have this: Here is the result of inspecting the image on my system (only showing relevant fields): You'll notice that the RepoTag it has is for , not . I am guessing the fact that we're somewhere inserting "library" into the image's name is what's tripping this up.
If it wasn't clear, I believe dockershim is what's having the issue. Once the kubelet sends the protobuf message to the dockershim socket to pull the image, it goes from to to to and then into the helper function I listed above.
It appears to be this: to to to That's where the "library" is coming from.
Prior to the PR that bumped docker/distribution, used to have , which does not do any normalization (i.e. including "library").
OK, I see why when we bumped to the newer docker/distribution, we switch to using . It's because now requires that the input be in "canonical" format (in this case, has and , if applicable), which you get automatically by calling ...
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
could you give steps to reproduce this? We can't reproduce this and many of the node/cluster e2e tests pull images using unqualified name such as "buxybox"
What docker distribution are you using? I could not reproduce this myself. I don't understand how does the reference show up, we never pull image with it. Is this a problem specific to a docker distribution?
run the e2e tests on a centos/fedora/rhel image
Is there any signal in test-grid for centos/fedora/rhel? Would be nice to catch issues like this sooner than later.
run the e2e tests on a centos/fedora/rhel image there is not, and definitely should be, given that k8s releases rpms as part of each release. will be on the short list for the release retrospective.
I feel like this is a problem in docker distribution in centos/fedora/rhel. Docker should not return when we are using to pull image. Should we fix the corresponding docker distribution instead of blocking Kubernete release?
Since that would mean releasing a new version would regress existing users, we generally say that we'll do our best to tolerate it instead of releasing something known broken. We've done that with various versions of Docker, systemd, and iptables for years, and I'm sure we'll do our best to do that. I do agree that this looks like a bug in that docker distribution that should be fixed. I also agree that this should have been in the test grid and that the distro owner is responsible for ensuring the project has that signal.
Thanks! I agree that we should not break existing users. I'm fine with as long as we keep in mind we made this change, and clean it up in the future. I think it would be helpful if we could have test coverage in test-grid on more distros, so that we could have caught similar problems earlier.
Agree - I think anyone who causes a release slip is required to make sure they fix the problem so it can't happen in the future. And get better all around testing. On Sep 27, 2017, at 7:40 PM, Lantao Liu <: ! I agree that we should not break existing users. I'm fine with long as we keep in mind we made this change, and clean it up in the future. I think it would be helpful if we could have test coverage on more distros, so that we could have caught similar problems earlier. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
I have opened a bug for this
This has broken the CRI as well. Kubernetes should not normalize image names on behalf of the CRI runtime. It's up to implementors to decide how to handle an image name and since there's not a standard around Kube and Image names, I'd prefer not to break the CRI.
Are you saying it's currently broken, or the fix was ok? , Antonio Murdaca <:
The fix is ok but the original patch caused another bug
Which one?
caused a bug fixed by
Does this problem fixed?


(SEE UPDATE at ) Currently, if you're writing an initializer controller, you're most likely using it to initialize a and deploying the controller with . Therefore you're likely using the following trick to avoid the chicken and egg problem: This bypasses the initializers for the pods and lets your controller to be deployed even though your InitializerConfiguration is configured to capture all pods. Problem occurs when you apply an update to this Deployment with the same manifest containing empty pending list. You get error: I think this is a somewhat major UX issue. It requires users to deploy the initializer controller with the but later on when they need to update the controller with , they can't use the same manifest containing the empty list. /area api /area extensibility /sig api-machinery /kind bug /assign
/cc
We silently ignore some metadata updates, which is one option. Potentially we shouldn't require empty initializers pending, but simply empty initializers (which means - init directly), and only ignore that. Are there any examples of use of SMP for similar? I'm not sure this meets the bar for 1.8 but if we can find an example of something consistent already in the API I'd be willing to review.
I don't have any concrete examples, there's an example initializer at making use the trick, I use it myself, and I suspect many more initializers to come will use this, too. I'm not sure if this would work, but does setting provide a way to skip initialization (if possible at all)? If not, we might actually need a separate field to signal skipping initialization.
/cc
Currently the initializer plugin to on UPDATE, but that happens after it updating already initialized objects. We can simply change their order.
I'd like to amend the issue, it's more serious than I thought. (thanks for bringing it to my attention If you're deploying your Initializer with and if you have an initializerconfiguration for objects, it's impossible to deploy your initializer in the first place without deleting InitializerConfiguration first: is not inherited by the created on is not honored (this field is not propagated to the created Pod). I think (2) should be fixed somehow, otherwise there will be no way of writing an initializer for Pods and deploying the initializer itself using Deployment/ReplicaSet/RC without deleting the InitializerConfiguration first. Maybe what we need is an exclusion/exemption selector (or namespace(s)) as discussed in to allow deployment of Initializers themselves easier.
For 1, see I don't want to pursue that though, because i) we'll need to fix workload controllers one by one, and ii) we need to carefully consider permission issues. I think we should focus on adding the namespace selector.


/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


/kind e2e-test-failure GCE serial and GCI GCE serial started failing last evening https://k8s- https://k8s-
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/testing: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
https://k8s- is still green, I would suspect something ~last Friday afternoon gets merged
Here is what I found, after a while, the node lost their authorization to connect to API server. I believe this has the same root cause as
I am going to revert offending pr () to see if it helps. EDIT: Looks like is in the process of reverting it:


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG /kind bug What happened: Attempting to run e2e test on GCE without cluster started fails with stack trace What you expected to happen: A useful error message to be printed How to reproduce it (as minimally and precisely as possible): Without a cluster started: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig testing /assign
This happens because gcloud returns an error when a resource isn't found and ginkgo- sets errexit


It is possible to have multiple signals that point to the same type of resource, e.g., both SignalNodeFsAvailable and SignalAllocatableNodeFsAvailable refer to the same resource NodeFs. Change the map from map[v1.ResourceName]evictionapi.Signal to map[v1.ResourceName][]evictionapi.Signal
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/kind cleanup /priority important/soon taking a guess at labels to satisfy the milestone munger
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Looks like the PR for this is just awaiting merge


Currently, always returns with 0 exit code no matter the test fails or not. We should fix it to return non-zero exit code for test failure.


When pod is evicted, pod is terminated. After this change , attach/detach controller should be able to detach the pod correctly so that new pod which uses the same volume will start up. We need to have an e2e test for this scenario.
what environment is good for this e2e test? GCE or GKE or AWS? I see we have which has func, Can you comment on why is insufficient to cover this issue?
does the pod need to be evicted? Or would be ok to delete the pod and then check that the PD is detached. Then create a new pod using the same PD and test that it attaches?
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.9 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ping ^^
sorry for the delay. Here we want to test the detach behavior of evicted pods. So delete a pod directly is different from it. When Pod is evicted, the containers will be killed and the pod status is terminated. But the pod api object is not deleted from API server. Before PR , such evicted pods will not trigger detach operation because the pod api object is not deleted from api server. So the purpose of the test is to make sure detach operation is triggered for such evicted pods.
Thanks Do you have a suggestion for how to force a pod to be evicted?
You could have the pod do some memory intensive application
ah, like my use of in the io-streaming pr :-)
- Here's another option to cause eviction: .


/kind bug What happened: When a cluster is provisioned using a containerised kubelet using the following images: Pods with PVCs are not created because kubelet cannot mount the disks. If you look at events, you can see that it's missing some binaries: This executable is . This problem can be resolved by mounting paths from the host into the rkt container, specifically and . Is this expected? mentioned in , that mounting operations are moving out from the main process into containers. Is this part of that effort? This caught me quite unexpected, and I didn't find any docs or changelog entries which indicated additional host mounts were necessary. If it's deliberate, I can close this until we perhaps document the breaking change. Is there a good place where we can document this? If it's not, I'll leave this open as a tracking issue. What you expected to happen: The hyperkube image to work without lots of host mounts. How to reproduce it (as minimally and precisely as possible): Run kubelet with rkt in one of the above listed images (see CoreOS' ). Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OpenStack OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: tectonic-installer
/sig storage /sig node
Seems the base hyperkube image was changed: done in /cc
The pkg/util/mount/mount_linux source defaults to ext4 if a fstype is not selected. It makes sense to restore the e2fsprogs package within the hyperkube image.
cc cc cc This seems like something that should be in the 1.8 milestone. Can someone who owns the hyperkube image comment on that?
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Containerization of mount is not going to happen until 1.9 at the earliest. So if hyperkube is broken, it should be fixed before ship.
"modprobe" also went missing in 1.8 hyperkube images... Both kubelet and kube-proxy need it.
Are you noticing any buggy behaviour on those components without it? Would you mind creating an issue and/or PR to restore that dep
Done: And done: :)


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: my yaml: exec kubectl set resources command,just update first object: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind bug
/assign


We should introduce a buffer (similar to what we have in the webhook output backend) to the log file output backend. Audit events should be queued on a relatively large buffer chanel, and an output go routine should grab events off the queue and write them to disk. If an audit event is output when the channel is full, it should be dropped (this should be counted by a prometheus metric, and an e2e test should ensure that the count is below some threshold). This behavior should be optional (configurable) /cc is tracking the regression
Naturally os will cache , these cached data will be written to disk either by fsync or pdflush periodically. What do you think is the most time-costing procedure in audit?
The problem is not with the time consumed. The problem is that currently write is performed synchronously with processing the request, which slows the request down, no matter how well caching works
The problem is that currently write is performed synchronously with processing the request Sure synchronously write does cost time serializing. Introduce buffer mechanism can really reduce api request latency time.
It doesn't matter that much if it happens asynchronously Indeed
Yes, this is a good point to improve performance.
Now I totally comprehend this. I am pleased to implement this buffer.
I'll be happy to review that
audit log is important sensitive information which we should try to make sure it reaches disk at all cost. Does this effort compromises that in favor of optimizing for latency ?
I'll make sure it's configurable and if the security is critical, you can setup synchronous write in your deployment. Having said that, it's critical to be able to accommodate the case where having 100% calls written does not compensate for increased latencies. Does it make sense?
thanks my preference would be to keep the default as no buffering when making it configurable. I didnt understand how does one setup "synchronous write in your deployment" and how that will affect audit logs reaching disk. My worry is mainly around a crash where the buffered audit logs dont reach the disk and are lost
if use buffer, audit events lost is inevitable when crash. As like webhook backend, still exist this risk. But to improve apiserver qps, something should comprise. By the way, we still reserve the synchronous write mode, which is configurable.
To make it simple, when offered trade-off, people are more likely to give up audit logs reliability than performance. Makes sense?
i am not against this change. i think what would be best is to make the default as no buffering and enable buffering through a parameter. Also for Enterprises, the trade off is not correct, they would prefer to NOT give up audit logs
I understand what you're suggesting and I don't agree :) Switching to buffering will for sure increase the latency, but will only increase the probability of losing logs, which will never be zero. I'm pretty sure we need to have buffered mode by default. However, since this argument is about making a trade-off and since neither of us represents quorum, let's ask more people for their opinion What do you think?
The default is slightly less important here since audit logging as a whole is not enabled by default. If we do turn it on by default, it becomes more important. That said, my hunch is that we should default to the mode with lower performance impact, but I don't have strong feelings either way.
As long as there are options to turn it on of off (the synchronous write) I'm fine with either default. Still, I'd like to see due diligence towards ensuring low lost rate when using buffered write, and sufficient documentation around caveats of using one or the other. Audit, from my personal experience, is crucial for majority of enterprises and they won't tolerate not even single lost event.
That is not possible, there will always be a situation when an audit event can be lost: sudden power loss, disk/network failure, random bit flip, etc. The questions is whether 10^(-6) is low enough loss probability, or we need to get 10^(-14) by sacrificing performance (arbitrary numbers, just to illustrate my point). Note, that reduced performance can increase the number of timeouts and retries, increasing load and leading to failures and even more lost events.
I think this is an important bug that we're not testing audit logging at scale
Marking it as urgent, since it's blocking
If split into 3 prs, I must do it one by one . Now first one is up .
SGTM Status of this issue: ETA: 1/12/2017 Risks: Complicated fix required
has been reviewed for almost 2 months and has not done. If I raise 3 prs to replace it, I am still worried whether all of them can be merged. They act as a whole.
Now I'm going to devote my time to it, we'll do it
OK.
In progress, reviewed first PR, waiting for applying changes ETA: Dec 01
Per Exception discussion, this portion of the feature is being removed from the milestone. As such, changing the priority. /priority important-soon /remove-priority critical-urgent
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.9 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Creating a service with created a load balancer without a floating IP. What you expected to happen: Load balancer should have a floating IP assigned by default. How to reproduce it (as minimally and precisely as possible): Deploy this manifest on a cluster and check the resulting load balancer. No floating IP will be assigned. Anything else we need to know?: The default behavior changed from k8s 1.7.x. It appears that an annotation that will add a floating IP, but the default should not have changed without warning. This also breaks the workflow. For example: Environment: Kubernetes version (use ): v1.8.0-rc.1+coreos.0 Cloud provider or hardware configuration**: openstack OS (e.g. from /etc/os-release): Container Linux by CoreOS 1520.4.0 (Ladybug) Kernel (e.g. ): 4.13.3-coreos Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Oops, default should be external. /assign


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: What you expected to happen: Test should not flake /sig storage /cc
/assign
GitHub didn't allow me to assign the following users: vkamra. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: vkamra. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
- You won't be able to assign this to me (not a kubernetes member). Am taking a look.
volumemetrics.go_ could/should have more logging, either calling or more calls to .
The test appears to fail on this metric: "kubeletvolumestatsused_bytes". So samplePVC != pvcName and/or sampleNS != namespace. It would be nice if the sample name and ns were also logged. Also something is weird here: the framework's is called before the local . Note in the log:
- will add a couple of Logf() statements to log the samples - you're right that will help. Once the volume is attached, volume stats are grabbed every interval. The thing I noticed in the logs is that we issue a delete for the pod at but the volume only got attached/bound successfully at The test assumes blocks till the pod is running and PVC is attached but maybe that's not a valid assumption?
Nevermind - That log message has 2 timestamps in it - the volume was provisioned at .
Pushed The working theory being that volume stats were not collected within the wait earlier being used.
did you figure out if I was correct in thinking that the framework was invoked before the local ? That seems very, very unlikely but that is how it looked in the log.
- I believe you're right that the framework is being invoked before the local one. I observe the same in a test as well, so I don't think it's specific to this call. Based on the , nested blocks should be invoked inner-outer. It's not clear, however - whether the local is nested - or at the same level as the framework block when I look at


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CRD: Existing object: New object: says: If this is a problem with the new object definition rather than a bug with , I'd expect a better error message. (The new object looks fine to me, though; it should be identical to the old object.) What you expected to happen: I expected the object to be updated without an error. How to reproduce it (as minimally and precisely as possible): See description. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: GCE OS (e.g. from /etc/os-release): CentOS Linux 7.4 (1708) Kernel (e.g. ): Linux master-head-1 4.13.4- SMP Wed Sep 27 13:32:23 EDT 2017 x8664 x8664 x8664 GNU/Linux Install tools: Custom scripts Others:
/sig api-machinery
Are you able to reproduce this everytime? I am able to use apply normally with your example.
I can reliably reproduce it if the annotation exists: If I use to remove the annotation, then it works the first time but fails thereafter:
This is happening for other CRDs as well (Prometheus Operator ServiceMonitors). If I apply a ServiceMonitor CRD 3 times: If I remove the annotation or replace the object () then I can work around the issue.
Seems any CRD is affected; the CRD used by our are also not working: This has completely broken our CI/CD workflow... unfortunately we'd already completed the TPR-CRD migration for the 1.8 upgrade before noticing this problem, so we can't really downgrade. How safe is as an alternative to ? It still generates an update event for watchers, not a delete/recreate, right?
It looks like something going wrong during the patching. After using changes from , I get this: /cc /area third-party-resource
But uses json merge patch and not strategic merge patch for custom resources:
I was able to reproduce this in v1.8.0 with the original manifests at the top. As found, it only happens on the 3rd execution: It also seems to only happen when the ought to be a no-op. If I change something in the IPPool spec, works once more, then starts failing again. When the is a no-op, the PATCH request body generated by looks like this: When the is NOT a no-op, the PATCH request body looks like this:
There's a typo in the code generating the error messages for both and : They should be calling instead of . When I fix that, the error message becomes:
I can reproduce this without , just by doing a no-op twice:
The same issue affects v1.7.8 (tested on GKE)
Same issue with v1.8.1
This just fix the error message: But it doesn't fix the issue
Still investigating the issue but I noticed that when we do a no-op patch for native resources, we get a "not patched" response. However, for non-native resources, it still tries to patch it (and bumps the resourceVersion too).
I don't konwn whether this is a issue, I can't do strategic merge patch to CRD. When I run , --type=merge must be passed.
We should be able to apply strategic merge patch to CRDs. However, we can't apply strategic merge to CRs because we're missing the metadata about how to handle each field for the strategic merge patch. Unfortunately, this error is occuring even with . :(
/assign
do you know if the fix fir this will make it to 1.8.3
hopefully yes, this should be merged soon.
thanks! this has been a blocker for us to upgrade to 1.8
can you confirm this is completely fixed? The PR writes "partial fix".
Now we can't reproduce this issue with the method descripted above. I still need some I extra time to research it. Maybe we can close this one. If I can reproduce the issue in another way, I can open another issue with new situation.
If it's not fixed, let's keep it open.
Fun, a fork closing issues.
From Then I think we can officially close this.
(the fix is merged into release-1.8, but after v1.8.4-beta.0)
what stable release of k8s will this be in? 1.8.4 or 1.8.5?
Is there an update for this?
looks like it was merged here: but did not make it into 1.8.4, might make it into 1.8.5