/kind feature What happened: The DevicePlugins feature was introduced in 1.8 Kubernetes release as an alpha feature. Since then, we have seen good development work on stabling the feature as well as early adopters on using the framework to support device resources in Kubernetes. This issue is to track the work requirements of entering this feature into beta. FYI, here is list of requirements from the OSS Kubernetes feature stage doc: """ Beta level: Object Versioning: API version name contains beta (e.g. v2beta3) Availability: in official Kubernetes releases, and enabled by default Audience: users interested in providing feedback on features Completeness: all API operations, CLI commands, and UI support should be implemented; end-to-end tests complete; the API has had a thorough API review and is thought to be complete, though use during beta may frequently turn up API issues not thought of during review Upgradeability: the object schema and semantics may change in a later software release; when this happens, an upgrade path will be documented; in some cases, objects will be automatically converted to the new version; in other cases, a manual upgrade may be necessary; a manual upgrade may require downtime for anything relying on the new feature, and may require manual conversion of objects to the new version; when manual conversion is necessary, the project will provide documentation on the process Cluster Reliability: since the feature has e2e tests, enabling the feature via a flag should not create new bugs in unrelated features; because the feature is new, it may have minor bugs Support: the project commits to complete the feature, in some form, in a subsequent Stable version; typically this will happen within 3 months, but sometimes longer; releases should simultaneously support two consecutive versions (e.g. v1beta1 and v1beta2; or v1beta2 and v1) for at least one minor release cycle (typically 3 months) so that users have enough time to upgrade and migrate objects Recommended Use Cases: in short-lived testing clusters; in production clusters as part of a short-lived evaluation of the feature in order to provide feedback """ I think we met most of the requirements here, with the commit that if we need to make incompatible API change in the future, we will document the upgrade path and if necessary, may support multiple API versions in a couple of releases. Things that I think would be nice to have to enter beta (entering beta may not necessary depends on their completeness) Regarding to the thorough API review requirement, I think we had good discussions during the design proposal but there have been some recent discussions on whether we should use registration vs probe model for the general Kubelet plugin communication. I think we should have a general discussion on this topic with larger audience like people working on CSI plugins. I will create a FR issue to track this discussion soon. We would like to see multiple device plugin implementations. We know some people have started working on device plugin implementations for Solarflare, FPGA, and Infiniband, and probably want to keep closer track on their progresses. We would like to target this FR to 1.10. Please let us know your thoughts or add things that you think are critical to finish. Thanks! What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig node
I'm a fan of probe model. I do feel it unfriendly to plugin developers to let them care about the re-registration after kubelet's restart. And have received complaint from my colleague.
Hi, for cluster reliability, I think we need more test cases, including scale and soak tests, to simulate more production scenarios.
can we consolidate this with ?
Sure. Will do this with Vish after KubeCon.
Sorry to bring up this topic a bit late (feature freeze is the 22nd of Jan). Here are the features we (NVIDIA) consider important or blocker before entering beta: [BLOCKER] Per container allocation Add labels during device plugin registration (e.g: gpu-family: tesla) Add annotations to a container for beta and add an option to CRI for GA to support OCI hooks Things that are not explicitly related to the device plugin: We're also wondering if we should tie up
Thanks a lot for your input, For per container allocation, I think we should be able to implement it in 1.10. I think we should be able to support ER quota in 1.10 as well. I think allowing device plugins to export device properties is an important feature, especially when we introduce ResourceClass, but I don't think it is a beta blocker. For "Add annotations to a container for beta and add an option to CRI for GA to support OCI hooks", this seems outside the scope of device plugin. There are different ways to implement them (if they have to be implemented) on Kubernetes. We haven't seen strong use cases indicating that they have to go through device plugin.
Sorry, I meant from the device plugin. i.e: the device plugin's returns container annotations for beta Product wise this would allow us to have the nvidia-runtime be supported by CRI-O (which allows hooks through container annotations) as well as docker + nvidia as the default runtime. BTW That would allow a convergence towards a single device plugin in beta. As for GA this would be a different discussion but it would be part of converging towards using any runtime and have them support the nvidia-hooks.
Not saying it's a beta blocker but it would be useful to have labels on homogeneous nodes about what devices are present. Especially: GPU memory GPU compute capability ... It's also not a big change, it just requires consensus
As a heads up, I am going to send a PR soon to change DevicePlugins feature to beta. After introducing the feature in 1.8, we have seen good development work on improving the feature and received many valuable feedbacks from early adopters on how we may extend the framework to support various kinds of resources on Kubernetes. To us, entering beta means: a) We think we have done enough early experiments during alpha phase to get the confidence that this a useful feature that we are willing to carry on to beta and to GA. b) We think the current DevicePlugins is ready to be tried on production. It has good e2e and e2enode test coverage. Some early adopters have tried it on production and didn't report any major breakage. Very likely we will keep seeing new feature requests coming in after beta, as more people try to use DevicePlugins for their specific use cases. Some of these features may even require incompatible API changes. When this happens, the required complexity to support multiple device plugin API versions on Kubelet side should be manageable and mostly local to the devicemanager component. It should also be manageable for a device plugin to support multiple API versions. As an example, in , I extended gke gpu device plugin to support both v1alpha and v1beta1 versions. For reference, here is list of changes we made on device plugin during 1.10: Support resource quota on extended resources (Issue ). Owner: Added PreStartContainer grpc function to allow device plugins to perform per container start operations. Owner: Add annotations to the device plugin API. Owner: Track unhealthy devices in devicemanager, and have node status capacity to reflect raw capacity but allocatable to reflect healthy device capacity. Owner: Stub device plugin based e2enode test Owner: There are a few pending PRs that would be nice to get into 1.10 but we don't consider as beta blocker: Determine and implement a general Kubelet plugin communication establish model and use it in device plugin. Owner: Uses common checkpointing mechanism in device plugin (PR Issue ) Owner: Device plugin stress test. Owner: Please let us know if anyone has objections on entering the feature to beta in 1.10. Thanks!
Hi, for stress test, here is a PR :)
/cc
Should we reopen this tracing issue?
I think this issue can be closed with We can open separate issues to track future device plugin changes.


Internal volume plugin for CSI creates VolumeAttachment with (= the plugin name). It should be CSI driver name instead, as the external attacher must know which CSI driver to call. /sig storage /kind bug please prioritize PTAL. It should be one-line change in
[MILESTONENOTIFIER] Milestone Issue Current Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ACK. In progress


BUG REPORT While configuring the kubelet with: And having the sock file on that location (default location from cri), the following error occurs: Later line: Uses the correct socket. Result: Disruptive in understanding other failures. Containers still start. Environment: Kubernetes version: 1.8.0-5, 1.9.0-beta.2 Cloud provider: gce OS: Ubuntu 16.04 Kernel (e.g. ): Linux kubeworker01 4.10.0-42-generic ~16.04.1-Ubuntu SMP Mon Dec 4 15:57:59 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: The hard way Others: Full kubelet config (as of now): Internal log reference: syslog.v6.3
/area [kubelet] /sig [node]
/area kubelet /sig node /assign
This is because uses , while is using as default. We should make them consistent. Fixed in . PTAL.
What are the consequences? Is it more than cosmetical? Does it affect the usage of CRI-O in 1.9.0 or how does it affect users?
/assign Is this relevant for CRI-O in 1.9.0?
It (at this moment) mostly looks cosmetical, however when I add a cloud provider (--cloud-provider=gce in this case) to the kubelet config, the whole thing breaks consistently on PLEG. I am trying to figure out if there is a relation between that and the
Can you weigh in on whether this is a release-blocker for 1.9.0?
Tentatively adding milestone so we can track getting a judgement from sig-node.
-- i think this is fixed in the packaging, let me close with and
fixed in packaging yeah, from 1.8 we moved to "" so you need to start kubelet with that socket path
Discussed at sig-node earlier today, and this is not a release blocker. Also like mentioned above, the issue was fixed in the packaging.
There is a reference that needs to be fixed for the new socket location. The referenced pr is required to ensure stats summary endpoint understands the right location, and this keeps it consistent with cadvisor after its rebase in 1.9.
/priority critical-urgent /kind bug /sig node /status approved-for-milestone
/status in-progress
[MILESTONENOTIFIER] Milestone Issue Current Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Hi, I am hitting the same error in this thread on my Openshift4.2 cluster which has configured CRIO on the kubelet. I would like to hit to get inventory of the containers. Any body has any insights why I am hitting this error when I make curl calls from the k8s node which has configured for the CRIO? [root /]# curl -v --unix-socket http://localhost/info Trying Immediate connect fail for No such file or directory Closing connection 0 curl: (7) Couldn't connect to server [root /]# sshadmin kubectl version Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-01-14T21:04:32Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6+", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-01-17T17:43:42Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: If a 1.8.x kubernetes master with a single controller-manager is configured with --leader-elect=false, deleting a deployment through the API with v1meta.DeletePropagationForeground doesn't work. The deployment has its deletionTimestamp set and a foregroundDeletion finalizer is , but the pod, rc and deployment are not deleted. What you expected to happen: The pod, rc and deployment should be deleted when the controller-manager uses --leader-elect=false. How to reproduce it (as minimally and precisely as possible): I used kubeadm with calico networking, and modified the controller-manager configuration via sudo sed -i /etc/kubernetes/manifests/kube-controller- -e '/leader-elect/ s/true/false/' and then ran to create and delete deployments. Anything else we need to know?: I suspect the problem is related to this change: It appears to require leader election before the GC controller will finish initialization. Looking at controller-manager logs with leader-elect=false, I saw this log entry from "GraphBuilder running" I did not see the "started %d new monitors, %d currently running" log entry at the end of the startMonitors() function. That seems consistent with the GC controller waiting for "<-gb.informersStarted" Environment: Kubernetes version (use ): tested with 1.8.2 and 1.8.4 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): vagrant virtualbox with the ubuntu/xenial64 box Kernel (e.g. ): Install tools: kubeadm Others:
cc Have you seen this in Openshift?
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think this problem occurs because initialiuzes the stop channel top nil when leader-elect=false. This causes garbage collector graph_builder to exit early. The following release-1.8 patch seems to fix the problem. creates a dummy channel which will never be closed and allows the garbage collector graph builder to work like it does when leader-elect is true. [Edit - just noticed that my initial patch had some unexpected differences]
do you really need to remove the following code?
That change was not intended. I also noticed that after I saved my change. I redid the patch and edited my previous comment to reflect that.
Agree with the diagnosis and fix, are you going to open a PR?
Yes, I can do that. Reading your PR guidelines and ours.
Will I need to go through to get this into the release-1.8 and release-1.9 branches?


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When enable cpumanager by , but not set and , kubelet will get panic error in log What you expected to happen: Should graceful exit with error not panic. How to reproduce it (as minimally and precisely as possible): set and but not set and Anything else we need to know?:
/sig node
Great that it's not panicing anymore, but I'd also love to get the "requires + to be greater than zero" hint in this particular case. It looks like even with this doesn't seem to be the case (since it errors out with the non-actionable "unable to determine reserved CPU resources for static policy". Should I open another issue for this?


ref: and An e2e test should be , and the default should be updated. /assign cc
/assign
Thanks I can take this on.
Could someone point me to where the manifests are for the e2e test suites, I am happy to submit the patches?
Are there API server e2e tests currently?
there are not HA API server e2e tests that I'm aware of
I opened a PR for the default changes. Are there suggestions on a good E2E test to write if we don't have HA API server tests?
what is needed for this issue to complete? Do you think you'll get that by Monday, which is Code Freeze?
We need some E2E testing or add an integration test as defined . It will not be done today.
so are you planning to move it to 1.11, then? It's about to be automatically kicked out of 1.10 by automation.
I do not have permission to change the milestone. If you could kindly change it, that would be awesome!
[MILESTONENOTIFIER] Milestone Removed From Issue Important: This issue was missing the label for more than 7 days. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Reopening as this was reverted in


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug for lack of better idea hack/make-rules/test-cmd-, line 1443 is failing, even when the debug output shows the expected text: from the test log: What happened: See for example https://k8s-
This should be owned by whoever owns the "test-cmd runkubectlget_tests". This doesn't look like a failure in the testing framework or the infrastructure. I believe this is
the linked test is actually failing on the next test, not the .missing test... from the test log: this is very odd:
the issue appears to be with
potential fix in
seems like the same issue for runpluginstests:
same fix will hopefully resolve the issue


Happy New Year! Now when I run make I get a diff like this in every zz_generated file: Not sure what a smart solution is, but we could merge something like in the meantime.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: call delete endpoint when service is not found. What you expected to happen: It is no need to do delete in controller. Because in service restStorage, endpoint is deleted when delete service. See How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/area controller-manager /sig apps
/kind bug


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The OpenStack cloud provider currently uses the name of the kubelet node to register with the API server. However the name is not guaranteed to conform to the DNS-1123 subdomain spec. Given a node name of , the following message appears in the kubelet logs: What you expected to happen: The OpenStack Cloud provider uses the instance id instead of the instance name. This will guarantee that the id conforms the DNS-1123 spec without making any assumptions on what the instance name may be. How to reproduce it (as minimally and precisely as possible): In the OpenStack console, rename a kubelet node to and attempt to register the kubelet node to the api server. Anything else we need to know?: Environment: OpenStack Newton Kubernetes version (use ):Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 14.04.5 Kernel (e.g. ): Linux -cab9-44ff-9c99- 4.4.0-103-generic ~14.04.1-Ubuntu SMP Mon Dec 4 19:33:04 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
/sig openstack
/assign I prefer to conform hostname to DNS-1123 subdomain spec. Already submitted a PR on this.
I think the issue may be more than just the hostname. The hostname on my vm is set to Since the OpenStack cloud provider uses the instance name to register with the api server, I also get error messages like the following: It seems like for OpenStack specifically, it sets it's node name to the name returned from the OpenStack metadata server () which is not restricted to being DNS-1123 compliant.
The node name must match ([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9], not contain '/'.
I can confirm that Openstack can boot up instances with the name like . But this will break kubelet currently as stated above. For such cloud provider based nodes, the nodeName registered is retrieved from the provider, which should not be allowed when registering nodes for kubelet. Agree. +1. I've an extra check in PR . PTAL. Thanks.
I think the question here is: Why did you choose to use as the nodename? This is the display name of the server, a property made for humans by humans. Essentially it is a field that everybody should be able to fill in whatever they think their server should be called. This doesn't need to be unique and has no guarantees about format or length or character format. We've had a number of people putting in unicode characters in the display name, for example. Furthermore, it can be changed during runtime of the server, without rebooting. Other possibilities for the nodename are: instance UUID, which is a string in a defined format, guaranteed to be unique and is only separated by dashes. Found in If you would like to have some way to associate your nodename with whatever is shown in the Horizon dashboard as server display name, there is . OpenStack already takes care to sanitize any weird display name. E.g. some weird display name like gets a reasonable hostname: <img width="745" alt="screen shot 2018-01-10 at 08 52 49" src="https://user-"What do you think about changing the source for selecting the nodename?
cc
Once upon a time, the cloud provider's (only!) role was to discover a dynamic list of nodes by regex. In that world, using the nodename made perfect sense because the admin could then configure it to be something like , with the obvious use of regex patterns to group the relevant nodes. Fast forward a few years, and this style of node registration is long gone, and there is no need to make the node name human readable, and in fact numerous reasons to choose other unique identifiers. There is still to removing "human-friendly" node names, however. In the end, I didn't see a good alternative that would please everyone (and meet the required technical character set/length constraints). I stuck with because at least it was under the control of the cluster admin, and so it could be obviously contrived to match our imposed k8s node name == openstack instance name constraint (once that constraint was communicated).


/kind feature What happened: When calling , there is no way to override default flag values for the different components of the controlplane. This is surprising, as in the config structure (and in controlplane phase codebase itself), there are explicit settings for doing that (APIServerExtraArgs, ControllerManagerExtraArgs, SchedulerExtraArgs). What you expected to happen: Those config fields should be exposed as flags on and How to reproduce it (as minimally and precisely as possible): Try to deploy K8s using Kubeadm with a custom Admission Control setting (I'll follow up on this issue with a PR)
btw, I am not sure which sig owns kubeadm.
it's sig-cluster-lifecycle.
/sig sig-cluster-lifecycle
/sig cluster-lifecycle


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Currently, FC plugin returns volume name and empty FCVolumeSource during ConstrutVolumeSpec during . However, in order to recreate newMounter from volumeSpec, volume information are lacking. As a result, filesystem volume's reconstruction always fails. What you expected to happen: Filesystem volume's reconstruction works properly, then a volume is unmounted and detached from kubelet node. How to reproduce it (as minimally and precisely as possible): (1) Create filesystem volume using FC plugin (2) Restart kubelet service which a pod with FC volume is running Anything else we need to know?: cc Environment: Kubernetes version (use ):Install tools: local-cluster-
/assign
/sig storage


If I run first time it goes fine (after reboot). For subsequent runs I get nodes NotReady. I've tried restarting docker but that won't help. Before christams this used to work all the time. Running Arch Linux if it helps. In I see
/sig node
ditto for me: Looking for a work around I noticed that HugePages recently moved to beta. I can get pods to start again by disabling the HugePages feature gate .
/assign
cc
default Arch kernel doesn't have a hugetlb cgroup controller compiled in(been there done that, I use Arch as well), so you have to prepare your own kernel to run it.
If it were a kernel problem I would have expected the kubelet to fail the first time it was started rather than the second time. This isn't confined to Arch, though. I'm having the same problem with the default Debian kernel:
can you check if your Kernel was compiled with flag? You can do it by running: I bet we have to add documentation, describing this behavior. On Ubuntu 16.04 machine:
Sure, looks like it was not: Documentation (especially in the error message) would be great, but if the kubelet isn't going to survive a restart, I would prefer that it not start the first time. Better to fail immediately than eventually.
can you please check if works (as in fails!) for you?
No, this doesn't resolve the issue. Logs from the first start: Logs from the second start:
Do you have the directory? in that case what i have is insufficient :( If you don't have that directory, then we should probably look for a better spot to put that check? any advice?
No, that directory is missing on my system:
I'm seeing this issue as well.
I am preparing a fix to not fail when hugetlb mount is not present. Will post a fix shortly.
I posted a fix here:


/kind bug What happened: When I want to enable both plugins by passing below flag, as they both need config file, and I find I donot know what to set If I set , then I get an error in apiserver
/kind bug /sig auth
/cc
/cc
first, you can't put comments in json. the error was complaining about your embedded comments. this would let the server start: second, the docs are really, really, not right (opened with fixes). if you want to specify config for multiple plugins, should point to a config that references individual config files for each plugin (which there's a bug in loading... fixed in ): example
/assign


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: We face But there is no way to generate default config file How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version 1.9.2
I'll take a look it /sig scheduling
/assign
Hi, same issue with controller-manager
Any updates here? , and don't have this flag. Is it even will be supported?
/cc
if not set, component should be use default value; why we'd like to dump all default config?
Because the documentation on the config file format is either nonexistent or difficult to find. If the program itself can generate a template, that fulfills (most of) the need for documentation of the format.
+1 for this, from my expirence, add an option to write the default config is really helpful.
/assign
also in 1.10
Deprecating all flags, prompting folks to use config files in advance is great, except when there's no documentation nor example anywhere. That means that we can't actually migrate to the new settings, and I can't seem to find clear alpha/beta/stable status for the new config file spec, nor timeline for the removal. It is also pretty confusing that the scheduler expects us to use a config file, but the controller-manager does not?
This issue indeed applies to kube-controller-manager. And there should already had deprecation policy.
According to , the flag will be available in . Since I can't find any reference online to an example config file, perhaps one could be provided or linked to here?
will you be able to pick my patch and generate a config file? I do not get my devbox handy so can only post here until Monday.


Is this a BUG REPORT or FEATURE REQUEST?: Bug Report /kind bug What happened: Applying a file that has no changes compared to current will output something like , which muddles the output when using What you expected to happen: Using the -o flag should suppress the unchanged messages in stdout to prevent output from being muddled How to reproduce it (as minimally and precisely as possible): Given the file Will result in proper yaml being generated as expected Run again, and is outputted to stdout, and is invalid yaml Anything else we need to know?: I do need to check whether this still affects the most recent version kubectl Environment: Kubernetes version (use ): 1.8.6 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Darwin Install tools: Others:
confirmed on latest stable version (1.9.2)
Reiterating the mentions to trigger a notification: <detailsIn response to : confirmed on latest stable version (1.9.2) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have a solution to this, still waiting on my company to look over the agreement


Currently as authentication and authorization are both standard config for all clusters, the kubeapiserver insecure http handler is unsecure, which means the cluster is vulnerable to the attack. So it should be deprecated, even removed totally.
/sig api-machinery
I think removing the insecure option is a good idea. Can you send an email the sig-apimachinery mailing list with a proposed timeline? I think this should be like other GA features and have a one year clock.
Sure, I will send it ASAP.
:+1:


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Today, . This seems unnecessary and should be done once for all the nodes. Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement /sig scheduling cc:
/assign Thanks, will send fix per slack discussion.
/cc


https://k8s- https://k8s- cc: /priority failing-test
Sorry, but I will fix it ASAP.
/assign
PR is already sent. Wating for CI report...


/kind bug What happened: kubectl became very slow, taking 30s to list pods etc. After going deep down the rabbit hole of debugging the controller components, I ran kubectl -v=8 which showed: What you expected to happen: Since caching is required for kubectl to respond in reasonable time and the issue isn't obvious, kubectl should at least print warnings in default log level. How to reproduce it (as minimally and precisely as possible): I got into the situation by running kubectl with sudo, but just chown'ing should be enough to reproduce this. Anything else we need to know?: Environment: kubectl v1.8.3
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle freeze


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Fix the following : The basic logic in of using the is continuing reducing the nodes slices through , , and . It needs 4 slices to do this. This is my thought to reuse the slices: Use 2 nodes slices, such as , The reducing process becomes: -Clear -Clear - please give some suggestion, thanks :smile:
/sig scheduling
The challenge is that clearing a slice and reusing it still needs GC clean up, so it may not be effective. The size of slices shrink as we proceed through the function. So, having two slices and over-writing the elements in them can help reduce GC time, but this approach needs some additional checks to ensure we are not using stale elements.
To avoid clearing the slices, this is my thought: Declare to indicate the actual length of the 2 node slices, instead of . Appending a node to a node slice will add 1 to . Clearing a node slice will set to 0. WDYT
Yes, I can't think of any better solution either, but remember that you should append only to the slice in the first for loop of the function. The second slice should be initialized with the same number of elements as the slice used in the first for loop. Setting elements of the slices should be done by indexing them (not by append) in the second, third, and forth loop.
OK, I will submit a PR to implement it.


http://k8s- /priority failing-test /kind bug /status approved-for-milestone /sig cluster-lifecycle the job is on the master-blocking dashboard and the e2e test has been failing, please take a look cc
testgrid change suggests
(author of the change in that range) -- any idea whether might have broken reboot tests?
I reverted the change as it may lead to this regression. The other change is really small () and unlikely related to this. Still the continues to say otherwise, I think we can reopen this issue.
The reboot test is still failing.
Just opened to achieve the revert.
Is the test passing now? Can this issue be closed?
looks green now


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: A Pod is run with local volume mounted in block mode. When that pod is deleted, the tear down process fails because device unmap fails. The device path appears to be empty, causing the unmap to fail. What you expected to happen: The teardown should have succeeded How to reproduce it (as minimally and precisely as possible): Create a pod with a PV for local volume in block mode. Delete the pod. Look at kubelet log files on the minion node to observe the error messages. Anything else we need to know?: Log messages: Feb 18 09:21:43 kubernetes-minion-group-crj1 kubelet[1510]: E0218 09:21:43. 1510 ] Operation for """ ("-148c-11e8-b315-")" failed. No retries permitted until 2018-02-18 09:21:44. +0000 UTC m=+. (durationBeforeRetry 500ms). Error: "UnmapVolume.UnmapDevice on global map path failed for volume "blkdata" (UniqueName: "") pod "-148c-11e8-b315-" (UID: "-148c-11e8-b315-") : Failed to unmap device from map path. mapPath is empty" Feb 18 09:21:43 kubernetes-minion-group-crj1 kubelet[1510]: I0218 09:21:43. 1510 ] operationExecutor.UnmountVolume started for volume "blkdata" (UniqueName: "") pod "-148c-11e8-b315-" (UID: "-148c-11e8-b315-") Environment: Kubernetes version (use ): 1.10+ Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): "Container-Optimized OS from Google" Kernel (e.g. ): Linux 4.4.86+ Install tools: Others:
/sig storage
This error is returned in this . It same that we set the global mount path to only if the volume plugin is attachable(see ). However, local volume plugin is not attachable, so will be empty and the operation will fail. I think we should extract the operation from the block, and execute this operation just before we try to map the device to global map path, then this global map path can be cached in for both attachable and non attachable volume plugins. Any comments? If not, I can help to fix this.
Thank you for the investigation. We've discussed this problem by offline yesterday. I proposed same fix which you mentioned above, then tested the fix works correctly with local volume plugin. I'll push a PR soon.
The device path appears to be empty, causing the unmap to fail. Please note, which was stored in is empty. device path isn't related to this problem.
Sorry about that, this is a typo, what I mean is global mount path.
No, no. My above comment is just note for the section. Your investigation is correct.


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The deployment controller for a deployment with a single replica failed to create a new ReplicaSet when the old ReplicaSet included an Evicted pod. Typically, the eviction was caused by "node low on resource: nodefs". When we manually deleted the evicted pod, the new RS and accompanying pod were created immediately. We have seen this happen at least a dozen times. Here are the audit logs for a recent case: ! What you expected to happen: The deployment controller ignores the evicted pod and creates the updated replicaset/pod. How to reproduce it (as minimally and precisely as possible): To reproduce, you would need a way to trigger a kubelet-driven eviction on a node that is currently running a single-replica Deployment with . Wait for the pod to be evicted and the new pod to be created, confirm that the evicted pod still exists, and then make a trivial change to the deployment spec. The running pod will be deleted, and no new replicaset/pod will be created. This issue does not seem to occur for manual evictions (via ); pods I evict that way disappear from the API server immediately. Anything else we need to know?: Possibly related to and/or Environment: Kubernetes version (use ): v1.8.5-gke.0 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Container-Optimized OS
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for the detailed bug report! From Here's what happened: Deployment was updated by the user, so it scaled all old ReplicaSets to 0 old ReplicaSet was scaled to 0 (ReplicaSet's == 0), but still owned the evicted pod, which wouldn't be killed automatically because of the above comment (ReplicaSet's 0) Deployment found that there were still old Pods, and kept waiting for them to be gone ~(wait for old ReplicaSet's to be 0)~ (Edit: Deployment also checks those old Pods by listing them to make sure there are none, not just by checking ReplicaSet's status) before starting the new rollout. We can either modify Deployment controller (to ignore evicted old pods) or ReplicaSet controller (to kill evicted pods ~when it gets scaled down~), if this is not the intended behavior. WDYT?
I am actually debugging likely the same production issue with Recreate strategy in two clusters right now. (We run on patched 1.9.1.) I have traced it down to broken RS as well. in 1 cluster RS leaves around pod failed matching the selector in the other one those are evicted pods. all replicas are scaled down to 0 and report 0 in status but pods with valid controllerRefs to them are still present. Same is 0 for me but pod is still there I think the correct behaviour is for replicaSet scaled to 0 to delete all of its pod, not to leave them laying around
xref - this should fix it


There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
probably caused by
Could be a clash between these two?
cc
I still see the issue with local-cluster on the latest master: F0225 17:06:49. ] error: v1alpha1.KubeProxyConfiguration: FeatureGates: ReadMapCB: expect { or n, but found ", error found in byte of ......
looks like that's another issue, can you please file a new bug? with more logs? this one was about none of the hyperkube based processes starting up.
will do as it is also broke container kube-proxy as well. Same type of error.


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The field passed to is never set. In where is a object that is initialized in the method in Notice the field is never set. What you expected to happen: The field should be set to How to reproduce it (as minimally and precisely as possible): Create a CSI volume and set to true. Check the value of in the CSI call. It should be true, but is false. CC Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I think is only used for mounting pod volume path to container, here(for ), we can set to , just like other volume plugins. I can help to fix this.
What about the case when a user wants to mount a volume for one of containers in a POD as read only, but the actual volume has ReadOnly False? I would like to suggest an alternative solution which passes the value of ReadOnly attribute, taken from POD's spec.
Thank you and for jumping on this! I feel like the other volume plugins have the wrong behavior here. The purpose of the call is to mount the volume. Therefore, the should receive the pod volume mount readonly field. That said, as mentioned , I think the fact that " is used for mounting pod volume path to container" might be hiding a bug: that is never set correctly. I haven't had a chance to validate this but my reading of the code suggests that the is never set. But because the container mount uses , even though the plugin did not get the correct readonly value, the mount ends up getting mounted with the correctly readonly value. Thoughts?
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
We discussed the proposed fixes in the Kubernetes CSI implementation meeting today (see ). We decided it would be better for CSI to be consistent with the other plugins (and still broken), and have CSI fixed, along with all the other plugins, whenever we fix the issue (likely post-1.10). Therefore, I will accept fix and we will close Then we can remove the 1.10 tag from this issue and track it independent of the 1.10 release.
Spoke with offline. Details here: Based on that conversation, I don't think there is anything else we need to do here.


/kind bug /sig cli /area kubectl Hello. This is my first interaction with the Kubernetes community, so please forgive me if i have left out any crucial details, or if, perhaps, this is PEBKAC after all! I am keen to have a working feature, as described in and I am running K8s v1.9.3 and realise that this feature is in alpha, but i thought i would ask anyway. We're not on v1.9.4 yet, but the changelog doesn't mention the feature. What you expected to happen: I have some services deployed to Kubernetes with YAML files, that has been working reliably. We always use to deploy things. When a service is already deployed, and i make a minor change in the config file, for example the service port, i expect to see that in the diff. Instead, gives no output and exits with a . How to reproduce it (as minimally and precisely as possible): Consider the file : I make sure that it is correctly running on the cluster: Great, looking good. Now, i change the value to use Docker image tag and save. Now, i run: The stderr/stdout output is empty, and the exit code is . Anything else we need to know?: Environment: Kubernetes version: Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:50:44Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-10T01:42:55Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
I have this same issue - if I change, say, to in my , running returns empty. If I try tweaking the version I'm diffing (which I don't understand) - e.g. - then I see a bunch of differences, but not actually the change I was expecting in the number of replicas.
I'll take this. Sorry for the inconvenience.


Anything else we need to know?: I notice there are TODOs in apimachinerypkgapimeta,return bool instead of error func CommonAccessor, ListAccessor, Accessor. If is it necessary to do,I push a PR for it. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
cc
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


IMO it's surprising that does not force delete a resource. only takes effect when . The description of reads this: We should make it more clear what does and even print warnings when is specified without .
Changing the meaning of force to actually break cluster consistency be default could lead to data loss, which is why we haven’t changed it so far. agree we should print more info and clarfiy


What happened: Deployed a webhook backend and service as follows: Please note that there is only 8443 port on the service. Register the webhook as follows: Got the following error when creating pods: What you expected to happen: Based on , should be called. Not the 443 port shown in the error message. How to reproduce it (as minimally and precisely as possible): Just deploy the same setup and register webhook. Anything else we need to know?: Environment: Kubernetes version (use ): HEAD Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.1.97+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2018-01-22T20:00:24Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-alpha.0.470+", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-12T19:36:42Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): COS Kernel (e.g. ): COS
thanks for the detailed bug report!
I actually think that the API specification (or documentation) must change. Not all API servers will have access to look up the Service object and see what ports it defines. If we want to support ports other than 443, the port information must be specified in the webhook object, not discovered by direct Service object lookup
/assign
Looks like it will always use 443 no matter what ports are available, right? Should our documentation reflect this or should it behave differently? (also just sets 443 in the custom dial wrapper thing )
As a first step, I would adjust the documentation. As noted in , if we want to support ports other than 443, that information needs to be to the webhook config object


The new test (introduced by ) seems to be quite flaky. Example failures: https://k8s- https://k8s- https://k8s- Should a "performance" test even be run in the pre-submit PR runs? /cc
It does connect to etcd and does a fair amount of node additions and deletions. Most of the times I've looked at the error, it seems to be at the startup when connecting to etcd. Is there a way to retain this capability so we can continue to check on demand but not necessarily run every time?
The error in has already been fixed, if you notice , we now have a lock and it should not be happening now. For the rest, the error is in the startup itself, namely, , which is starting the framework connecting to etcd. Any idea's on why this should be causing flakiness? This was the same pattern used in schedulerperf -
Can we please disable this test for the moment? It's causing significant flakiness and blocking other PRs from being merged...
I'd suggest disable the tests first to stop the bleeding, and debug the issue at your own pace. It's probably not a good idea to run performance tests in this environment anyway since the tests are run in parallel (IIRC). Maybe you can set up a separate post-submit test job for these special-purpose tests.
can you take a look at PR ? It adds skip for short tests, which should avoid running this for very PR?
https://k8s- shows that ipamperf tests are now skipped during integration testing. All checks have passed and has given /lgtm. Waiting for auto-merge. can you close this issue once verified? thanks!
the bot will do that automatically since your PR description says it fixes this issue.


This is continuation of and . What happened: When StorageClass refers to a key that is not accessible to Kubernetes (i.e. AWS API returns AccessDeniedException), AWS plugin still tries to provision a volume under assumption that while Kubernetes can't access the key, the other components may access it and create the volume and attach it to a node later. Related code is - checks only for . Other errors are just logged and provisioning continues. Problem is that when AWS really can't access the key, it will provision some volume (ec2.CreateVolume returns success + valid AWS EBS), but this AWS EBS disappears after few seconds. Therefore Kubernetes does not know that something went wrong and creates a PV for the returned EBS. Kubernetes later fails to run a pod that refers to such PV, because the PV does not exist. What you expected to happen: Kubernetes must not create PV in this case and must report error during provisioning of such volume. Ideas: According to , the only signal that AWS produces is an event in Amazon CloudWatch Events that a key could not be used for encryption. Kubernetes AWS cloud provider is not subscribed to these events and does not get this notification. We would need new permission for controller-manager to get the events and some new code to filter them and report back to user. Kubernetes AWS cloud provider could wait for X seconds and re-check that created volume still exists and did not disappear because of inaccessible key. What would be reliable value of X? Kubernetes volume plugin for AWS could return both a warning "I failed to check encryption key, created volume may not work" that would be propagated to user and the suspicious volume at the same time, but the API between volume plugins and Kubernetes allows only one - either error or volume. It could be extended, still the user experience would be suboptimal. Kubernetes could send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions. Again, UX is quite bad here. /kind bug /sig aws
Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. If the main source of errors are sent via CloudWatch Events, then best practice would seem to be to subscribe via SNS to receive those events. Possibly fall back to vague "send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions," where a SNS topic isn't provided, but otherwise use CloudWatch events to enrich state/messages. Would then also need to change docs to reflect needing to set up CloudWatch Events -SNS topic. Big drawback of going down this route is a fair few downstream cluster bootstrap projects would likely need updating to support it in their CloudFormation / Terraform bits.
According to it takes quite a long time until an event is available in CloudWatch events. what was the average / maximum delay?
I haven't calculated the metrics accurately but the cloudwatch events arrival vary(and standard deviation was high). Using cloudwatch events was also considered for maintaining instance cache and as well concluded that - they are not realtime (AWS itself documents them as "near realtime"). It may be - we need to measure them again and find out if we can use them for the purpose here. Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. from jan's analysis above it appears that - if IAM does not have permission to DescribeKey then EBS volume gets created but then it disappears soon. So net result still is - if "current user" can't describekey, she can't provision the volume. Is there any documentation that indicates otherwise? cc
Testing this for real now. User creates volume with no access to underlying CMK, or insufficient permissions does not seem to generate events with sufficient information as to what went wrong, i.e. don't appear to receive events for create volume failure, only deletion. E.g.: This policy is sufficient to allow the creation of volumes - isn't necessary for the volume to be created. It isn't sufficient to attach the volume however, creating the following error, The permission's required to fix this, to delegate access to EC2 itself: Bit of experimentation shows that only is required for attaching - the ones for creating the volume aren't needed at the attachment stage. The docs at is therefore correct for volume creation, but doesn't describe the volume attachment scenario.
So, what is our solution then? From the above it seems Kubernetes can't reliably check if a key is usable for encryption of a PV. And it can't get an usable event either.
Possibly one of the following: change. Live with the pain. "GenerateDataKey" and "Decrypt" on the generated ciphertext blob prior to volume creation as a preflight check. The advantage being that we can throw an error that this isn't going to work beforehand, but with the disadvantage of doubling up API calls. This is probably fine, and I think if you're creating encrypted volumes at a rate that's likely to run into API call limits or get super expensive, you might have other problems? We then also do the CreateGrant check during volume attachment (as an optional extra). the same check, but only after an error. Problem being that we get much less in the way of guaranteeing what the root cause is, but we don't add to the API calls in the happy path. In any case, is this bug biting sufficiently to fix here, or should we start thinking about the CSI implementation?
thanks a lot for detailed investigation. I'll try to get some time to implement 2. even before CSI. Users are very innovative in setting up their cluster permissions wrong and the messaging to user is very confusing in this particular issue.
/assign
2. does not work. Kubernetes does not need to have GenerateDataKey and Decrypt permissions to create encrypted volume. My AWS account is severely limited (I am using shared company account) and I can't call GenerateDataKey on a key: But I can create encrypted EBS using the same key: The volume gets created successfully. It's not the user who runs Kubernetes who needs permissions for GenerateDataKey, it's AWS itself.
And the volume stayed created and didn't get reaped after a few seconds? I was confused about this part, it didn't seem to follow how the APIs should work. Has anyone ever created a volume with that key, or is it a new one? I'll try what I did previously in a bit across a few regions.
Yes, the volume stayed around for several minutes, then I deleted it manually. The key is our default encryption key, so when I create an encrypted volume with no key, I get this one. No zones, no regions involved.
I got an idea, the volume is "creating" and then it gets "available". I could in a loop until newly created volume either gets "Available" or deleted. This will abuse the API quota ( is cheap, right?) + it will slow down provisioning a bit - all volumes I tried were Available in a second or two. Will it work reliably enough?
Might be the difference then, as I don't think the same perms as I described above are needed for the default key as I assume that it gets automatically delegated to the EC2 service. I only tested & documented the explicit, alternate key path above. Your approach should work, but I don't know when EC2 API limits will kick in and how soon users would see that happen. Worth a shot though?
It takes ~4 seconds for a volume to reach "available", it's quite a long time. This will affect encrypted volumes with explicit keys only.
Fix:


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: pull-kubernetes-e2e-gce and pull-kubernetes-kubemark-e2e-gce both start kube-apiserver with admission and but without enabling priorityclasses and podpresets resources. We can see many error logs from
/sig test /sig api-machinery
/kind bug
/sig testing


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: This was introduced by following code change introduced in v1.10.0: candidate fix is here(only works on Linux, not a good solution): This issue is caused by PR: Environment: Kubernetes version (use ): v1.10 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
Update: above fix does not work on Windows, I am still investigating a better solution. This issue is caused by updated after , related code is here: while should not be changed from LUN to real device path in azuredd since AttachDisk happens on master, is a LUN(Logical Unit Number) number after AttachDisk process in the beginning happens on agent node: on Linux node, could be ; while on Windows node, would be a disk number, when is recalled, it could not tell whether it's a LUN or a disk number (this is the key point).
thanks for issuing this problem. Could you submit a PR with your fix so that we can review it? Thanks!
finally I realized should not be used in func any more by referencing the gcepd code, PTAL , awsebs code should have same issue, I filed a aws_ebs issue here:


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When one pod is running and has two Containers. Container 1 exits with success and container 2 is running. gives correct info - the pod status is . However, , the pod status is . What you expected to happen: , the pod status is . How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.9.3 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig api-machinery
i can not see your pod named nginx in your above code
, updated the typo.
/sig node /remove-sig api-machinery


Is this a BUG REPORT or FEATURE REQUEST?: BUG /kind bug What happened: I am trying to get metrics working. According this there should be support after 1.8 kube. However, I am now using kube 1.10.1 and still I cannot see anything in ? Should these be there or not? What you expected to happen: I except that kubelet should expose prometheus metrics to /metrics path as well. Then it is easy to read with prometheus. How to reproduce it (as minimally and precisely as possible): Create new cinder volume, attach it to pod and try to query result from in kubelet Environment: Kubernetes version (use ): 1.10.1 Cloud provider or hardware configuration: openstack OS (e.g. from /etc/os-release): centos 7 Kernel (e.g. ): Install tools: Others:
cc
/sig storage
Prometheus metrics should be available on default port of Kubelet. Can you try ? If you are on a RBAC enabled cluster (such as openshift) you may have to use token or key to authenticate for fetching metrics.
returns many metrics but there is not things. There is one pod which have cinder volume attached running in that node. So there should be some metrics right?
It appears that Cinder volume plugin does not implement metric interface necessary for getting those stat metrics - This seems like an oversight and is fairly easy to fix. I can help getting the PR merged if you can open it.
/sig openstack
Kubernetes v1.19.16 there is no kubelet_volume* metrics in //metrics how to fix it???


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Scheduler's pod informer has a selector that . This is incorrect. Scheduler must watch all the pods of the cluster, including those scheduled by other schedulers in order to update its cache of the state of the cluster. Otherwise, it will miss important events and the amount of available resources, pods on nodes, ... which are needed for scheduler to work correctly, will not be updated properly. /sig scheduling /assign
cc
For updating it cache, shouldn't the default scheduler rely on node info/status? Also its not very clear, what information the default scheduler needs from other pods (not to be scheduled by it) that it makes decision on? Just trying to understand the issue more clearly.
Scheduler watches informer events to update its cache. Node status is not enough to update all the information that scheduler caches. Scheduler must know what pods are placed on what nodes. For example, in order to check inter-pod affinity/anti-affinity it needs to know the placement of all the pods, including those scheduled by other schedulers.
But this information is present here: So it seems that your PR will now make sure to get the scheduled pods' information directly from the pod informer.
So when I check the history, I see it was here:
have we seen any specific issues due to this change (like between default scheduler and DS controller)?
Yes, but scheduler relies on its pod informers to keep this location up to date. That's exactly why it is important for the scheduler to receive updates for all the pods in the cluster. Yes. We missed the problem at the time. This is problematic only in clusters that use multiple schedulers. I initially thought that DS controller could cause issues as well, but DS controller does not set , so it must not cause an issue. We have seen instances of scheduler's missing certain events in cluster and as a result scheduler's cache getting stale, but I doubt if it is related to this particular issue. This only affects clusters with more than one scheduler. If you run the test that I have written in with the head, it will fail. That tests simulate a multi-scheduler scenario.


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: endpoint of different k8s components expose Prometheus metrics that are not for them. For example, exposes metrics: or : What you expected to happen: Prometheus metrics only for to appear on kubelet's endpoint. How to reproduce it (as minimally and precisely as possible): Access the endpoint of all Kubernetes components Anything else we need to know?: This problem comes from usage of functions in different packages that automatically register Prometheus metrics. Just importing that package or some package that depends on it can cause those metrics to appear in unexpected places. In the list bellow you can find the different packages that have init functions and affected binaries: [ ] () cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] () cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubectl kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubelet [ ] hyperkube kube-apiserver kube-controller-manager [ ] hyperkube kube-apiserver [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubelet Environment: Kubernetes version (use ):Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A /sig instrumentation
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When use an existing azure disk(also called ) in pod, if that disk has partitions, the disk will be formatted in the pod mounting. What you expected to happen: How to reproduce it (as minimally and precisely as possible): create a disk with source from an os disk, e.g. that disk is already formatted before pod could access it Anything else we need to know?: Root cause: current diskLooksUnformatted func is using command to check whether disk is formatted or not while common k8s code() is using to detect disk format, could detect multiple partitions so if we use common k8s code(), following error will be returned for mulitple partition disks, which is expected: Environment: Kubernetes version (use ): v1.7 - v1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
cc
iirc we don't support partitions on Azure volumes. There was a specific hang long ago. in all cases i think should be safe to based check now via the underlaying api
thanks for the info, I am writing this PR, and also testing carefully. And this issue depends on , could you review it first? Thanks.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When trying to mount an Azure File mount it fails, it looks as though its putting a new line between the name of the Azure Storage Account and the Azure storage domain. There are no new line characters in the secret that holds the account name. The following is the secret: The following is the declaration within the section of the Kubernetes yaml: The following is the error that's shown in `kubectl describe po/pod_name: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS Kernel (e.g. ): 4.13.0-1011-azure Install tools: Others:
/sig storage
/sig azure
It seems to me that does decode to , so it seems to be behaving correctly. Maybe it's a good idea to trim the return char for and fields after ?
is specified by user, so trim the is not a good idea in the code. Another simple way is to use kubectl create secret to create azure-secret, the accountname and key won't require base64 encoded, so you won't mess up with let me know if you have any question, thanks.
Thanks !
Bumped into this issue today too on our cluster. Can confirm command worked. Is there an easy way to time the /n off the end of the echo for before using base64? Thanks.
yes it's
Of course it is! Doh!
i think would've made better sense to trim the azurestorageaccountname value when receiving it... anyways the storage account name wont have n in them right? I think that just made better developer experience...
is not allowed in azure storage account name
so if azure storage account name does not allow in the name.. then why not just trim the name instead of error... Because devs would just and that would already have the n in the resulting base64-encoded string... right?
sounds reasonable /reopen
Reopened this issue. <detailsIn response to : sounds reasonable /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
awesome. Thanks
so then this goes in to kube version 1.14.6 in AKS ? or how does this work?
It would only go into k8s v1.16.0 since it's not a critical bug, we don't want to do hot fix to the old release.
got it. thanks Andy.


the said: However the are different, it does not allow to set spec and metadata, but other changes are not ignored. And updates to other parts are not validated.
/cc for some suggestions.
/assign
Made a pull request:
Is the documentation wrong or the implementation? I think there was a longer discussion about the correct behaviour in the PR, but haven't found it yet.
It strange that other parts rather than metadata and spec can be modified and they are not validated. So I think the implementation is wrong, at least.
We should follow the design doc, I think.
/sig api-machinery
/priority important-soon
I agree the current behaviour is wrong. Added some comments to Especially, worries me whether we really have to right /status behaviour. Not really related to the given issue. But as we are touching the /status behaviour, we should clarify that as well.
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 3 days, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: On Windows Server with sandbox support the CNI gets called for all the containers within the pod including the sandbox container itself. What you expected to happen: The CNI should be called only for the sandbox container since they share the same network details. How to reproduce it (as minimally and precisely as possible): kubelet Pod at the parameters that get passed when the CNI is called Anything else we need to know?: This applies to Windows Server with sandbox support. The CNI should be called only for the infra container. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig windows /sig network /kind bug /assign /cc
Reopen this because of
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
has been merged now. /close
Closing this issue. <detailsIn response to : has been merged now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
has this issue been fixed? Is the CNI getting called only for the infra container? From what I see, the CNI still gets called for every single container from within a pod:
See comments It is required for windows containers.
The way I understand it, it's that the DNS issue will be fixed in Windows RS5 and there will not be required additional calls. The workaround at the moment is to call the CNI multiple times until the DNS issue gets fixed in Windows. As far as I know, that is the only reason why the additional calls to the CNI are required.
Is this true? Do we still need to invoke cni for each container in RS5? And even with new cni dns capabilities ()?
the PR that you linked it's unrelated to this issue. The CNI should be called for the sandbox pod which is the infra container. All the other containers created within the same pod are sharing all network details, excepting DNS. This is an issue in 1709 and 1803 where not everything gets replicated properly to the other containers. As a workaround for this, you need to call AttachEndpoint from HCS on the newly created container network interface only to replicate the DNS as well. You can also see more details about this issue here:
Can you re-open the issue in the meanwhile since this issue still exists?
I still think this isn't a issue, but a requirement for windows containers. Even after RS5, we should probably support old windows versions. WDYH
Why would the spec of the CNI should be different on Windows? On Linux you know that the sandbox ID that will get passed to the CNI is always the sandbox container. On Windows the sandbox ID won't be only that, it will also be an ID of a container from within the same pod. Does this sound like normal behavior to you? For now I agree, the workaround is to call the CNI for every single container because of the Windows issue with DNS. But this doesn't mean we have to stick with this for future Windows releases which will not require calling the CNI ADD multiple times. Also, this is a hack for now () which needs to be fixed. Please re-open the issue until we get feedback from others as well, thanks.
OK. Reopened the issue. Let's wait a while for feedbacks from other folks.
Apologize for the delay. Windows doenst have the namespace support native on the platform. we this support as part of RS5 Platform release. We are working on making runtime spec changes and CRI/ContainerD changes to use this model. here is a runtime spec PR that got merged. This is orthogonal to the DNS issue. Once we are done with the CRI/ContainerD changes then if u use CRI/containerD as runtime and if your OS version is RS5 and above then you don't need this second CNI add call. But as Pengfei said, if you are on a downlevel version or using dockershim as Runtime then you will have to make the second CNI call. So in my opinion this code will stay as workaround forever until we remove the support for downlevel OS. Hope this helps. I would recommend closing this issue.
thanks for the confirmation. In my opinion this should stay open until it gets fixed at least for future Windows releases, because the way it is at the moment, the CNI will still be called multiple times regardless of the Windows version. Why would you recommend closing this issue? Shouldn't this be fixed for futures Windows releases? The other calls will be useless when Windows gets proper namespace support. Also, I recommend watching this presentation about the CNI which had place at the KubeCon from May I don't see why we shouldn't have this open until the issue gets fixed for future releases and we stick with the official SPEC of the CNI.
and I had talked before about end of life of downlevel OS like 1803 and 1709. I think we should keep this open or open a new Issue to track updating the code so that the CNI is not called multiple times
This code would exists as far as we support 1709, 1803. The new workflow to use the windows namespace is getting implemented in containerd/cri. We can move kubernetes to use this new flow only for RS5 and above, if we plan to do it here as well. There isn't a good way to switch between 2 workflows of cni callflow wrt the Windows version hence we are not planning to change here till 1709 & 1803 exists, without adding some ugly code.
thanks for the explanation
I totally agree that this has to be fixed for future release and we are actively working on it. I was recommending to close this issue and open a feature to track this request. We should be able to raise a PR in another 2 weeks to support native network namespace support. This code will stay as long as we support down-level OS'es. I am fine either way. I am trying to close any open issues that is not a bug on Windows Networking(Platform).
/kind feature Although from a kubernetes standpoint, this isn't a new feature. It doesn't need a new KEP and won't change any API. It will just be a future Windows network code cleanup.
/kind cleanup
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
- can you clarify a few things so we know what's required to fix this? 's the minimum OS version needed? Windows Server 2019, or a future release? this be possible to fix for dockershim, or just CRI-containerD? Thanks!
What's the minimum OS version needed? Windows Server 2019, or a future release? [Dinesh]: Minimum OS version needed is 2019 Will this be possible to fix for dockershim, or just CRI-containerD? [Dinesh]: yes its possible to fix for Dockershim as well. but we are starting it with ContainerD. ContainerD is already done and available for testing.
excellent. thank you Dinesh!
ping - you have logs coming in email for this. Can you help us determine if it's needed for 1.14 vs 1.15?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Ping
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Status on this?


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: I am suffering from mysterious timeouts. See for the evidence from my latest investigation. It includes this line from the function in . The relevant line is this: I still do not know the type of the error in question or where it is coming from. It would be helpful if the log message included the type of the error object --- such as with or, if that does not include the type's package name, perhaps something that would really nail the type. What you expected to happen: I would expect more information about an unexpected type of error. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.3 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I think this timeout is from resourcequota admission. Did you enable admission plugin? will create a pr to make ResourceQuota return err.
thanks for the suggestion, I will respond in the original issue.
I appreciate removing a source of errors of unexpected type, but this issue is about what to do whenever an unexpected type of error does arrive. I suggest formatting it with instead of .
/reopen That PR did not really fix this issue.


/kind bug What happened: Wrote a that returned a faulty json patch which specified an our of range index (e.G. when there is only one container). This resulted in the apiserver crashing. Stacktrace: What you expected to happen: No crash /sig-api-machinery
/sig api-machinery
This maybe a bug in . I will follow this question.
This bug had been fixed in PR: you can checkout it.
Thanks for your comment but I don't think the linked PR fixed the issue, as it got merged almost a year ago and I am running Kube 1.10.1 which does contain that patch but still panics. Just from quickly looking at the code I the linked PR I believe there are two things wrong: The check if the index is valid only happens when a negative index was given should be
Once the PR of is merged I will update kubernetes.
/kind bug


/kind bug /priority critical-urgent What happened: In I'm removing all the reapers, unfortunately when invoking foreground deletion on a DaemonSet the delete operation hangs and controller indefinitely tries to manage a removed DS. What you expected to happen: A foreground deletion properly removes DaemonSet and all its dependents. How to reproduce it (as minimally and precisely as possible): Invoke a delete operation against kube API (without using command). Anything else we need to know?: This behavior makes it so that none of the consumers of the API is not able to remove the DS controller due to present in the DS reaper. A working fix that passed the tests with the aforementioned PR is in but I'm not 100% it's correct. This is an issue that needs fixing before 1.11 goes out. /assign /sig apps /cc
fyi
I filed to fix it. As mentioned in the PR, this is because the DaemonSet controller races with garbage collector when it tries to update DaemonSet status of the DaemonSet being deleted. Here's what happens when hangs: /something performs a foreground deletion on a DaemonSet finalizer and DeletionTimestamp are both set history objects (ControllerRevisions) and pods are being deleted by garbage collector; meanwhile, DaemonSet controller tries to update DaemonSet status. Updating DaemonSet status requires constructing DaemonSet history objects, to figure out current revision and which pods do/don't belong to current revision updating DaemonSet status, DaemonSet controller tries to create a DaemonSet history object that matches current DaemonSet spec collector then tries to delete that DaemonSet history object. And repeat. Because we can't make DaemonSet pods be deleted before DaemonSet history objects (DaemonSet history objects don't own DaemonSet pods!), we cannot reliably calculate DaemonSet status without history objects anyways. Therefore, we don't update DaemonSet status for DaemonSet being deleted. Note that the reason why the kubectl delete hack works is because it forces DaemonSet pods to be removed before history objects.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: After joining a new ppc64le node to the k8s cluster, the kube-proxy DaemonSet fails on the new node. What you expected to happen: It was expected that the kube-proxy DS would "just work" as it does when adding an amd64 node to the cluster. How to reproduce it (as minimally and precisely as possible): Set up an amd64 based k8s cluster, then join a ppc64le node (or any non-amd64 arch) to the cluster. See the pod fail via 'kubectl -n kube-system get pods' because it's pulling the amd64 kube-proxy docker image instead of the ppc64le image. Anything else we need to know?: The yaml for the DaemonSet specifies the image: <versionIt would be preferable to have a manifest so that the image in the DS could be specified as <versionand could "just work" on any platform. See: Environment: Kubernetes version (use ): v1.10.3 Cloud provider or hardware configuration: self - cluster installed via kubeadm OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.3 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.3 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTU_CODENAME=xenial Kernel (e.g. ): N/A Install tools: N/A Others: N/A
if you are running multi-architecture nodes, your daemonsets should specify a node selector to limit their arch-specific image to run on matching nodes
/close
I am well aware that a node selector can be used. That is what we are doing now. But it is a hack. The default should "just work". Adding a new non-amd64 node should not result in kube-proxy immediately failing on it because the default daemon set is amd64 arch specific. If kube-proxy is meant to "just work" clusterwide, a node selector should not be needed. Please use the manifest-tool so that the default "just works". Please see the weave-net plugin. They are using the manifest-tool and it "just works" on amd64 and ppc64le. Their daemon set does not require/use a nodeSelector.
The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this
Reiterating the mentions to trigger a notification: <detailsIn response to : The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this </details
ah, this was a comment about the kube-proxy image, not the daemonset functionality /reopen
the kube-proxy addon daemonset is probably the multi-arch kube-proxy image to support that daemonset might be
Looks like it's coming from here: /assign /sig cluster-lifecycle Lucas, Do we want to avoid the references?
A dirty workaround for this is to pull and rename to . The kube-proxy is running as daemonset, created during . That means the kube-proxy image arch keeps consistent with your master node. Currently we don't have a multi-arch kube-proxy image yet. We have to explicitly specify the right name, like .
Hi .. Yes, we just created a separate daemonset with a ppc64le node slector that refers to the ppc64le image directly. But it's a bit of a hack. That is why I opened this ticket looking for a multi-arch image. Thanks!
I don't know that mixed-ISA clusters are supported. cc
The requirement is: only use "" in DaemonSet yaml file, and the k8s will download the correct image according to the ARCH info. It's more about image registry's feature; and it seems some registries had supported that: .
Yeah, has already supported multi-arch. But actually all the images now are architecture specific, like and . We've not built any multi-arch images right now.
for the test images we have the ball rolling here - - so if you can do the same (or similar) for other images, that would be great!
I have pending PR in releases project for creating manifest for those -
The manifest images created in kubernetes/release should work for future released versions, but will they be created during CI testing?
if you see one of the jobs that pushes images say you can grep for that's basically from the same files that got changed in so i fully expect the images pushed to CI will reflect the changes made in 516.
please confirm that i am reading this right ^ :)
I was more concerned with ci jobs that do not use cross builds, such as https://k8s-
please open an issue about pushing manifests in CI as well


BUG REPORT: Portworx volume plugin when a PVC is created using a StorageClass with empty parameters /kind bug What happened: Create a StorageClass using empty parameters and then create a PVC using that storage class. The PVC will remain in pending state. On inspecting the logs of the controller manager, we find that the portworx volume plugin is panicing. What you expected to happen: PVC should have been created successfully. How to reproduce it (as minimally and precisely as possible): Create following storage class and PVC Anything else we need to know?: Environment: N/A Kubernetes version (use ): 1.9.7, 1.10.1, 1.8.8 Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A
/sig storage


The mount code () is reasonably well tested but brings in the entire Kubernetes source tree when CSI plugins have to depend on it. With a few exceptions, most of the mount code could be refactored to be a minimal set of utilities that can be layered and make it easier to develop CSI plugins. I noticed that quite a few CSI plugins take a dependency on k/k just to get that one package. Suggestions: and split into a minimal dependency library and move it out of k/k (perhaps into staging and have it published out) stop depending on k/k and start depending on that library dependency on so the library is easily usable. In general, there is a lot of boiler plate for CSI, and pkg/util/mount is a significant portion of reuse.
If there no one was working on this, I can help do it. :-)
/assign
is this something you are still looking at? It's come up in sig-storage, and is on the planning spreadsheet for 1.12. It's something I'd like to work on as well, so want to see where you are at with it. While it didn't get any traction, I will point out that when me and some coworkers wrote several CSI drivers to the 0.1 version of the spec, we saw this problem, and factored this code out into a project that eventually landed at It looks quite a bit different now, because we found the k/k code didn't handle bind-mounts of block devices correctly, which was something new to CSI and K8s at the time.
did some work recently in pkg/util/mount to detect bind mounts better. What I would like to see is this handled in at least 3 steps: Mounter interface to separate out purely mounting helper functions and Kubernetes utilities the refactored mount library into an external repo, maybe under kubernetes-sigs mount library from Kubernetes and vendor the external repo
Awesome. I plan to start working on this now, and i agree with the refactor approach. I do wonder where the resulting library should end up? Would kubernetes/utils be the right place?, rather than kubernetes/kubernetes-sigs? k/utils seems odd to me, though, as one goal is to not have to vendor a bunch of code you don't care about. but, k/utils has the intent of holding many different projects in it. So if you want one thing out of it, you have to vendor all of k/utils. I like seeing things in separate repos. =) Location doesn't hold up the refactoring, however, so I'll work on that!
Can it just be a standalone package, like kubernetes/utils/mount?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-kind bug /kind cleanup
Hi Now that is merged, I wanted to confirm that the next step here would be to move to be a package in .
Yup!
let me know if you need a hand with git surgery
The git-foo wasn't too bad. Just slow to filter on the whole k/k repo. =) I've opened the PR at , feel free to take a look. Once that is in, we'll need to be cognizant to not letting any fixes come into k/k/pkg/util/mount, otherwise they might get lost. Once the package goes into , I can open the PR on k/k to make use of it!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Currently, Calico CNI spec is specified in calico-node- under cluster/addons/calico-policy-controller without MTU specified (i.e., default 1500 is used). Given this is shared across many environments, we need a way to customize its MTU, say 1460 on GCP. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig network
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details