/kind bug What happened: When creating a node port application (in this case the example guestbook) on a host server that is dual stacked and has AAAA and A DNS entries, the node port on the v6 address is open to kube-proxy and hangs. What you expected to happen: The v6 address would refuse the connection and the client would fall back onto the v4 address which would connect correctly. How to reproduce it (as minimally and precisely as possible): kubeadm init with the above, and join in some workers. Then install the guestbook application Obtain the node port with and telnet to one of the servers on that port from a dual stacked client. The client will connect and refuse to respond to 'GET /' telnet directly to the IPv4 address of the server on the port and all works as expected. Anything else we need to know?: kube-proxy is noted as listening on the port. Environment: Kubernetes version (use ): ubuntu kubectl version Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Brightbox OS (e.g. from /etc/os-release): Ubuntu 18.04.1 LTS Kernel (e.g. ): Linux srv-jidtj 4.15.0-29-generic -Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm 1.11 Others:
/sig network
Looks like kube-proxy in ipvs mode listens on the nodeport ports to ensure they're not in use by something else. It just binds and listens, it doesn't serve requests. It binds on both ipv4 and ipv6 but never expects to receive connections because the iptables/ipvs rules intercept them, but on ipv4 clusters there is nothing stopping ipv6 connections coming in. The comment there does admit that only a bind is necessary to do this, but doing a full listen just made e2e tests easier. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen This now causes conformance failure on e2e tests due to the promotion of the Nodeport test in
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Still an issue on 1.16 when running in standard IPv4 mode
As I understand this thread, I have the same problem. I thought that it is a node configuration or some kubernetes flag that I didn't set but it seems it ain't. I ask already on SO: Environment Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-10-02T16:51:36Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: On premise, on virtual machines using hyper-v OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="18.04.3 LTS (Bionic Beaver)" Kernel (e.g. uname -a): Linux kubernetes 5.0.0-29-generic # 31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:34:01Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
/lifecycle frozen
/assign bowei
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Looking through the comments, this looks like a bug that could have a targeted fix. /remove-triage unresolved /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Looking through the comments, this looks like a bug that could have a targeted fix. /good-first-issue /help-wanted </details
I will attempt to solve this issue.
any success or progress? If you left this issue I would like to give it a go as a first time contributor. Thanks, Yuval
I'm still working on this as my first issue, and planning to produce a solution by 20th.
I wasn't able to tackle the issue and will not continue trying, feel free to take it.
/assign yuvalsteuer
Still, open -- anyone actively working on this? I would like to attempt if not. :) /assign rkc007
OK, again still open, doesn't look like anyone is working on this. I'd like to try it. The fix that occurs to me is simply to only listen to IPv6 addresses if the cluster is IPv6. If the e2e tests fail in this case (from looking at I don't think they will) then maybe they need to be updated. Does that make sense? What I'm not sure about is if we should do nothing with IPv6 ports or bind (to claim the port) but not listen.
/assign michaelbeaumont


/kind bug Hello, actually this is a bug about not about k8s. I use to observe request headers and found if the request have multiple same headers echoserver will give a wrong result: result: Expected result:
/sig testing
any chance you know some lua?
I never used lua before, but I'm willing to try to fix this issue.


Is this a BUG REPORT or FEATURE REQUEST?: What happened: Azure will not update the DNS label of the public IP, when update the service only changing the annotation of the DNS label from empty. annotations: -annotations: "to-update" What you expected to happen: It should update the dns label of the assigned public IP How to reproduce it (as minimally and precisely as possible): It probably caused when applying function 'getPublicIPAddress' in This function returns a bool 'existsPip' to judge the public ip existence. When only the annotations changed, the existsPip still returns true. Then, the reference in line 553 and line 446 step into this bug. pip, existsPip, err := az.getPublicIPAddress(pipResourceGroup, pipName) if err != nil { return false, err } if !existsPip { return true, nil } Firstly, in line 554, it will make function 'isFrontendIPChanged' return false, and then will not run 'ensurePublicIP' in line 691. Secondly, even if 'ensurePublicIP' in line 691 runs, it will detect the public ip exists and make no update to the DNS label in line 446_. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Azure
/sig azure
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
-- what's the status on this?
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/open I will take a look at this
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Still not fixed. /reopen /kind bug /priority important-soon /milestone v1.16
Reopened this issue. <detailsIn response to : Still not fixed. /kind bug /priority important-soon /milestone v1.16 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
i'm working on it.
/assign


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: PR() fixed pull image error from a cross-subscription azure container registry by service principal, while for MSI, this issue is not fixed. Need to find a way how to fix this issue for MSI, related code is here: Here the difficulty is could not list all ACR since it's based on one subscriptionID: So if we could not get in a different subscription, how could we set the . What you expected to happen: How to reproduce it (as minimally and precisely as possible): error msg would be like following when pull a repository from an ACR in another subscription: Anything else we need to know?: Environment: Kubernetes version (use ): v.9 - v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
By further investigation, I think we should implement LazyProvider for azure acrProvider: Related code:
Update: It's hard to fix this since in the beginning we cannot list ACR in another subscription, will always list acr under current subscription, and without target loginServer address of ACR, we cannot get : And in func, we cannot get target repo, it's not passed on
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
-- is this still an issue? If not, let's it out.
it's still an issue, and currently it should be low priority, let's keep it
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?


We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: CSI Plugin calls to get node ID information (see ). If that call is not implemented by driver, registration with kubelet will fail. What you expected to happen: Per CSI Spec, the deprecated call should be supported until CSI 1.0 is released (see ). Therefore if a by Kubernetes fails (because it is not implemented by the driver), we should fall back and try call instead. How to reproduce it (as minimally and precisely as possible): Use any CSI driver that implements but not -- says OpenSDS driver is currently hitting this, and may be used for repro. Anything else we need to know?: When CSI moves to 1.0 we can drop this fall back logic. If a fix is available within , we can include it, but 1.12 release should NOT block on this. /sig storage Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
please verify this is the issue you hit. Specifically see if you can find in your kubelet logs.
CC who volunteered to take a look
I don't see "error during CSI NodeGetInfo" in my kubelet logs. I don't see glog.Infof(log("Register new plugin with name: %s at endpoint: %s", pluginName, endpoint)) so it seems that RegisterPlugin is not called. I do see a message: "Plugin Watcher Start at /var/lib/kubelet/plugins". I don't know why RegisterPlugin is not called. I'll continue to check.
Here's an update. I implemented NodeGetInfo. I followed documentation about the Kubelet Plugin Watcher: https://kubernetes- to get my plugin registered. After that I don’t see the “driver name csi-opensdsplugin not found in the list of registred CSI drivers” any more. I see that now RegisterPlugin gets called and I see the following message in I0914 14:44:50. ] Register new plugin with name: csi-opensdsplugin at endpoint: /var/lib/kubelet/plugins/csi- So this confirmed that missing NodeGetInfo is the problem. I still have problems but I think that’s a config issue on my side. I the following as cluster role rule for attacher and node: apiGroups: [""] resources: ["csidrivers"] verbs: ["get", "list", "watch", "update"] However I’m still getting the following errors in the logs: I0914 16:12:31. ] Listing and watching v1alpha1.CSIDriver from E0914 16:12:31. ] Failed to list v1alpha1.CSIDriver: is forbidden: User "system:serviceaccount:kube-system:attachdetach-controller" cannot list resource "csidrivers" in API group "" at the cluster scope kube-controller- I0914 16:11:51. ] Listing and watching v1alpha1.CSIDriver from E0914 16:11:51. ] Failed to list v1alpha1.CSIDriver: is forbidden: User "system:node:127.0.0.1" cannot list resource "csidrivers" in API group "" at the cluster scope
Did more investigation on this. Here's an update. I set the following feature gate to true on my local cluster: alpha: CSICRDAutoInstall CSISkipAttach CSIPodInfo I got some new errors in and kube-controller- I0916 07:22:06. ] Listing and watching v1alpha1.CSIDriver from E0916 07:22:06. ] Failed to list v1alpha1.CSIDriver: the server could not find the requested resource (get ) I0916 07:25:24. ] Listing and watching v1alpha1.CSIDriver from E0916 07:25:24. ] Failed to list v1alpha1.CSIDriver: the server could not find the requested resource (get ) In addition, I found the following in kube-controller- E0916 07:22:01. ] failed to create CSIDrivers CRD: &v1beta1.CustomResourceDefinition{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.CustomResourceDefinitionSpec{Group:"", Version:"", Names:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, Scope:"", Validation:(v1beta1.CustomResourceValidation)(nil), Subresources:(v1beta1.CustomResourceSubresources)(nil), Versions:[]v1beta1.CustomResourceDefinitionVersion(nil), AdditionalPrinterColumns:[]v1beta1.CustomResourceColumnDefinition(nil)}, Status:v1beta1.CustomResourceDefinitionStatus{Conditions:[]v1beta1.CustomResourceDefinitionCondition(nil), AcceptedNames:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, StoredVersions:[]string(nil)}}, err: &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:""}, Status:"Failure", Message:" is forbidden: User "system:serviceaccount:kube-system:attachdetach-controller" cannot create resource "customresourcedefinitions" in API group "" at the cluster scope", Reason:"Forbidden", Details:(*v1.StatusDetails)(0xc4212af560), Code:403}} So I made this change to add RBAC rule for customresourcedefinitions: After that, my CSI plugin is functional. I can create/attach/mount volumes successfully now.
/cc
(As we discussed offline) the latest code in master no longer has CSICRDAutoInstall, CSISkipAttach, CSIPodInfo flags. New CSI features are now flag gated under CSINodeInfo and CSIDriverRegistry flags (both alpha and disabled by default). KubeletPluginsWatcher must be enabled as well, but that's in beta now so it's enabled by default. Since this PR merged: , CRDs must be installed manually, and you can find an example of it here: (note they have to be created with --validation=false kubectl flag for the time being). This PR has not been cherry-picked to 1.12.0-rc.0 yet.
Thanks Tested latest code. If NodeGetInfo is implemented, we just need to make necessary changes when deploying driver-registrar sidecar container so KubeletPluginsWatcher can work. NodeGetInfo is needed for kubelet to register CSI plugins. Also tested with and without "CSINodeInfo=true,CSIDriverRegistry=true" and manually installed CRDs using when they are enabled.
I have the same problem when deploying my CSI plugin based on CSI v0.2.0 on Kubernetes v.1.12.0. Daemonset pods of CSI plugin cannot launch successfully. Kubelet logs:
Considering Kubernetes v1.12 is already out, and we planned to deprecate the old call in CSI 1.0/Kubernetes 1.13 anyway, we'll pull of the bandaid and deprecate it now. If you are running Kubernetes 1.13, make sure your CSI driver implements the .
looks like you might be running in to a different issue: Kubernetes v1.13 also enables Kubelet device plugin registration by default. Before upgrading to v1.13, ensure the CSI sidecar container for your CSI driver is configured to handle plugin registration (set the parameter on to expose a new unix domain socket to handle Kubelet Plugin Registration). From
Hi Because goes Beta and is enabled by default in Kubernetes v1.12. After disabling this feater gate, I can run CSI plugin based on CSI spec v0.2.0 in Kubernetes v1.12.2. When Kubelet enable , Kubelet would call gRPC call to register CSI plugin. gRPC call was first proposed at CSI spec v0.3.0. Therefore, user should disable feature gate when running CSI plugin based on CSI spec v0.2.0 in Kubernetes v1.12.
I have implement the NodeGetInfo call. But it still has this issue. My kubernetes version is 1.14.3.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: After running (on master branch), a new file named left as a surprise. and the file is not matched by git ignore so it's tracked by git which is really disturbing. I tried to find out where it's created but failed. Any one has a context w/ it? Environment: OS (e.g. from /etc/os-release): Darwin Kernel Version 17.6.0: Tue May 8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASEX8664 x86_64
/sig instrumentation
the only piece of code in local-up-cluster that touches the file is
can't reproduce this. Two questions: Can you see this everytime? What is the SHA of the commit on master when you tried this? Can check with that too.
finally i realize that the cause is command is built receiving different flags between Linux and OSX. on Linux, you'll find that for linux sed the is explained as: '--in-place[=SUFFIX]' This option specifies that files are to be edited in-place. GNU 'sed' does this by creating a temporary file and sending output to this file rather than to the standard output.(1). so the is optional. but for OSX, it is: Edit files in-place, saving backups with the specified extension. If a zero-length extension is given, no backup will be saved. It is not recommended to give a zero- length extension when in-place editing files, as you risk corruption or partial content in situations where disk space is exhausted, etc. the difference is that on OSX, flag is requiring a variable , so when we're using on OSX in the script: the which is supposed to be received as a flag will be a suffix for the backup file. then the magic happened.will raise a fix for this. /assign
linking the difference between and


/kind bug What happened: After upgrading 1.11.1 -1.12.0 , when a node gets terminated in AWS, the controller-manager doesn't delete the node in the API. This has lead to several side effect for us : pods don't get GC Prometheus alerts firing because 100 of nodes are in the NotReady state for a long period What you expected to happen: kube-controller-manager deletes the node when it stops reporting, after checking with the cloud provider. How to reproduce it (as minimally and precisely as possible): Looking at the code, I believe this is what introduced the behaviour change : - The node gets tainted with because it's in a Terminated state. But it doesn't get removed because it still "exists" in the cloud provider. The behaviour is probably correct for a shutdown node, but in AWS a terminated node can't be restarted so there's no good reason to keep it there ? Environment: Kubernetes version : 1.2.0 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS (sorry for the all the sigs, not sure who it belongs to) /sig aws /sig cloud-provider /sig cluster-lifecycle
as I see it should delete node if instance is in terminated state have you tested that change in that row? I do not see what could be the issue there. Or could it be that the is not in place for all binaries in 1.12.0?
I might be wrong, but I think that part of the code never gets executed once the node is tainted :
good finding. So we should remove terminated state from this row. Then it does not add shutdown taint if node is terminated - instead it will delete node. also this should return false.. because if it returns true node is not deleted
I had the same issue. After I scale down my worker node instance number from 3 to 1. One node is stuck in status and never got deleted. However, one interesting thing is one of the other node is removed successfully. do you see similar behavior too?
- can we make sure this gets cherry picked for 1.12 releases ? I think we already missed the cut for 1.12.2


Is this a BUG REPORT or FEATURE REQUEST?: What happened: After upgrade of Kubernetes 1.11.3 to 1.12.1 via kubeadm in bare-metal setup I see these errors in kubelet logs: What you expected to happen: Since this is a bare-metal setup I expect no messages concerning Azure cloud How to reproduce it (as minimally and precisely as possible): Do an upgrade from 1.11.3 to 1.12.1 Anything else we need to know?: kubelet-1.12.1- was downloaded from Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Oracle Linux Server 7.4 Kernel (e.g. ): 62 Install tools: Others:
Lines from kubelet log which say that no cloud provider was specified (that is a bare-metal setup): ..
/sig azure
have the same issue related to the azure.. also a bare-metal cluster.. these messages are flooding my centralized logging elasticsearch (though they aren't really problematic more than being an annoyance). i also upgraded from 1.11.3 to 1.12.1 on a 3-master HA set up. i also noted (perhaps worthy of a separate issue) that the two of my three control-planes that are showing the azure error also previously had been having cert authority errors from the api-server container, when having used kubeadm upgrade.. (on the second two control plane systems, with the issue) (and on the first control without the issue) (and i needed to kubeadm reset and then kubeadm init all over again with 1.12.1 on the 2nd two control-plane nodes).. the first control-plane node that didn't need to do a reset/init didn't have the azure errors (nor the cert errors). so i kind of suspect it might not be an kubeadm upgrade issue but a general issue going from 1.11.3 to 1.12.1 ?
Same problems +1
Same problem on brand new bm 1.12.1 cluster built with kubeadm and latest 1alpha3 config based off of customized kubeadm config print-default example
/assign
Same problems +1
Oct 9 11:11:32 localhost kubelet: E1009 11:11:32. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:11:42 localhost kubelet: E1009 11:11:42. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:11:52 localhost kubelet: E1009 11:11:52. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:02 localhost kubelet: E1009 11:12:02. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:12 localhost kubelet: E1009 11:12:12. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:23 localhost kubelet: E1009 11:12:23. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:33 localhost kubelet: E1009 11:12:33. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:43 localhost kubelet: E1009 11:12:43. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:53 localhost kubelet: E1009 11:12:53. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:13:03 localhost kubelet: E1009 11:13:03. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1
how often did you hit this error log: I am planning to make this log as , is that ok for you? Usually V(2) is the default info(see ) log for k8s on azure I am not sure what the default info log by kubeadmin installation.
It is logged every 10 seconds. I disagree. At least previous incarnation of GetVolumeLimits in version 1.11.3 : would have logged an error "No cloudprovider present". Which I didn't see in kubelet log. In fact, I didn't see any message in kubelet logs with 'azure' substring at all. I think it means that AzureDataDisk plugin was not instantiated at all in 1.11.3. So the question is: why azuredd is instantiated now in 1.12.1 in bare-metal setup (that is without cloud provider)? Do you mind checking this idea? (Besides your method of lowering message priority won't work as I said this is a bare-metal setup where kubelet systemd unit file has no --v option at all )
That's because feature reaches beta in v1.12, in v1.11.3, even the code is there, it won't be invoked since feature is still alpha in v1.11
And in v1.12, aws, gce, and azure disk plugins are all loaded, take aws ebs for example, it will return error directly: And logging as V(4): I suppose V(4) logs won't display in logs. Let me make the same change as aws and gce.
also tries to address this error.
The same mistake, I also encountered NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME 172.16.16.13 Ready <none5m27s v1.12.1 172.16.16.13 <noneCentOS Linux 7 (Core) 4.18.12- docker://18.6.0 failed to get azure cloud in GetVolumeLimits, 172.16.16.13 failed to get azure cloud in GetVolumeLimits, 172.16.16.13 failed to get azure cloud in GetVolumeLimits, 172.16.16.13
I have cherry picked the fix to v1.12
I understand that the feature Node-specific Volume Limits has now reached beta in 1.12, but for those of us running Kubernetes on-premises, this log error is pretty useless. In fact, I would even want to disable this plugin alltogether, since it will never be used on-prem. Is there a way to stop the plugin from loading, or in the very least make sure the log does not appear? Right now, it is polluting the system log file quite a lot.
I second that. That was the key message of my bug report. That is we must not even instantiate cloud-specific plugins if no cloud provider is configured.
I'm having exactly the same problem, doing an installation from scratch on DigitalOcean
+1
+1
Hi I have the same issue. It would be really nice to know how to disable this error message that floods the logs for no reason.
For those using logstash it was sufficient to just temporarily add this to your syslog filter: if [message] =~ "" { drop { } }
After a fresh installation of Raspbian and kubernetes, same issue is occurring in my 4 node cluster. Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/arm"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/arm"} flood of this: epsilon kubelet[2412]: E1024 12:25:58. 2412 ] failed to get azure cloud in GetVolumeLimits, epsilon
disable in controller plane should work: "--feature-gates=AttachVolumeLimit=false"
I have tried "--feature-gates=AttachVolumeLimit=false" and it's not working. I still see messages kubelet[]: E1024 22:51:02. ] failed to get azure cloud in GetVolumeLimits,
I have the same problem here.., ] failed to get azure cloud in GetVolumeLimits,
+1 same case here, also tried with --feature-gates=AttachVolumeLimit=false
Will this be disabled by default in the next release?
It's already fixed in v1.12.2 and v1.13.0. do you know how to disable in kubelet in v1.12? Would add in kubelet command arguments work?
I haven't updated to 1.12.2 yet (still on 1.12.1) but just to confirm, disabling that feature works. For those that are also running on CentOS and/or RHEL, just to the following: Edit file on your master server (control plane), and add the following: If you already have extra arguments in there, just add this one at the end of the line. Then restart kubelet on your master server:
Yes that will work.
--feature-gates=AttachVolumeLimit property doesn't exist in the documentation
Update tho 1.12.2 resolve for me Em qui, 1 de nov de 2018 9:22 PM, Abhilash Pallerlamudi < escreveu:
add follow line to "your kubelet.service" file. like this: [root ~]# systemctl daemon-reload [root ~]# sudo systemctl restart kubelet then it's ok
So to get rid of a bug we should disable an entire feature? Says it is supported by Azure-Disk. Why isn't node reporting the limit? It is getting instance metadata okay as all my nodes are tagged with SKU. Being that SKU determines attach limit as well, I do not understand why this cannot be made working? I am on 1.14 in Azure on Ubuntu.
how do you deploy k8s on Azure? To get volume limit, kubelet on agent node needs azure identity, e.g. service principal, usually it's under file
I deployed with kubespray. I have cloudconfig set as such: KUBELETCLOUDPROVIDER="--cloud-provider=azure --cloud-config=/etc/kubernetes/cloudconfig" Service principle is set in cloudconfig file.
I triple checked and location was misspelled on some of the nodes for the cloud provider. Thanks!
the get vm size list call depends on location, so now it works?
Yes it does. Thanks!


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "[sig-network] Network [It] should set TCP CLOSEWAIT timeout" requires the test target OS supports /proc/net/nfconntrack, but Ubuntu 16.04 doesn't support it. Then the test fails today like the following: What you expected to happen: The test succeeded or was skipped. How to reproduce it (as minimally and precisely as possible): Just kick Anything else we need to know?: The existence of /proc/net/nfconntrack depends on the Linux kernel config CONFIGNFCONNTRACKPROCFS, but that is disabled on Ubuntu 16.04. In addition, the test is skipped if the node number is less than 2. So I guess that is one of reasons this issue was not present until today. Environment: Kubernetes version (use ): v1.11.3 Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.4 LTS Kernel (e.g. ): 4.4.0-134-generic Install tools: kubeadm Others:
/sig network
Some kube-proxy functionalities depends on conntrack, is it possible to run a conformance cluster without having conntrack enable?


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the for a CRDs Established condition is always null What you expected to happen: it should have a timestamp How to reproduce it (as minimally and precisely as possible): create a CRD and then examine its yaml/json. For example: Anything else we need to know?: Environment: community master
/assign
/area custom-resource
/sig api-machinery /area custom-resources
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
edit: actually, that issue is about kubectl display and clock skew. this issue is about the condition timestamp updating, which still requires correction
Closing this issue. <detailsIn response to : duplicate of Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now), which means that this PR must merged by then. Do we still target this issue to be fixed for 1.16?
yes, will take a look at it tomorrow /assign


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Here's an example from an e2e test: What you expected to happen: leader election success logs to be at a higher log level.
Leader election library is not a part of the scheduler. The scheduler is only a user of the library. It is located in client-go. It is currently . I am not sure if it is a good idea to reduce verbosity further. The log messages seem to be excessive in the scheduler logs when the scheduler is idle in our tests.
I think this is a common problem for all components, not just for scheduler.
In that case, we should consider reducing its verbosity to 5 or 6.
For some reason I thought that this was something new but I went back to 1.9 tests and checked the logs and I also see it printed every 2 seconds. So maybe this is fine for test environments.
This is certainly not new, but I am still in favor of lowering it verbosity unless there is a good reason for having it in tests by default.
I'm ok with changing the verbosity but the code now lives in the api-machinery land.
I'd like to pick this up. Could I do that?
The log level seems correct following these guidelines: I am not sold on the value of glog's infinite log levels feature, however I could be convinced that our logging guidelines need a TRACE level.
Thank you for your response. I'm on your side, How about ?


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: In PR we tightened the --provider parameter checking of the binary such that it only accepted known providers. This turned out to be problematic () because people had started to depend on the previous behavior, which was to accept anything and merely warn about it. For the sake of giving people time to adapt, PR reverted to the older behavior. What you expected to happen: In 1.14 we want to allow only known providers. While at it, the error handling for unknown providers should be enhanced (print proper error message instead of calling Failf, which panics). /assign /sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
kubeadm was one of the tools which got broken by the tightening of the parameter checking. Does kubeadm still rely on the acceptance of unknown provider parameters or can the current warning be turned into an error now? /remove-lifecycle stale
yes the kubernetes-anywhere deployer is still bound to provider=kubernetes-anywhere (unknown) in test infra jobs. we went back and forth on this topic, but if you wish to replace this support for unknown deployers i can switch those jobs to . LMK.
given one release has passed with the deprecated message on the CLI, it's probably safe to remove this functionality now. :)
"Lubomir I. Ivanov" <: Yes, please do. Once that is complete, I'll turn the warning into an error.
done:


<!-- Please only use this template for submitting enhancement requests --What would you like to be : a new addon to addon manager "that automatically installs CSI CRDs if CSIDriverRegistry or CSINodeInfo feature gates are true". But this is either broken or this addon manager isn't active in a cluster brough up with: This should work automatically also for such local clusters. Why is this needed: The CSI E2E tests are now expecting that the cluster has the CRDs and fail when that isn't the case. /kind feature (or perhaps a bug, not sure) /cc
/assign
Is this expected to be installed in other clusters? A lot of clusters are not going to be using cluster/* scripts.


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-new-master-upgrade-cluster-new-parallel Which test(s) are failing: Varies, but all subpath failures, including: [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath with backstepping is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] ... and pretty much every other subpath test, but never all of them at once. There's also a few other storage tests failing, such as: [sig-storage] Volume expand [Slow] Verify if editing PVC allows resize [sig-storage] Detaching volumes should not work when mount is in progress Since when has it been failing: 11/22 Testgrid link: https://k8s- Reason for failure: These flakes started around the time that merged, but doesn't match up with the exact merge stamp, so it's probably coincidental. The subpath test failures seem to be mostly timeouts: ... so possibly this is just GCE fail. Anything else we need to know: This test job has always been flaky, with around a 40% failure rate. <!-- DO NOT EDIT BELOW THIS LINE --/kind flake /sig storage /priority important-soon cc
can you please help investigate this. It looks to me like the tests are flaking for and . I wonder if the v0 tests are racing with the non-v0 tests for something in this test?
The other tests including for the other CSI driver and non-CSI in-tree plugins are solid green which also makes me think this is a issue with the hostpath tests or driver.
(just adding additional similar flakes here for tracking) Similar flakes in :
adding to the milestone for triage since the flakes look new
These all 4 subpath tests call and it fails in finding a specific failed event by using . As uses to check if the specific error event happens, and , I guess that tests may flake if other failed event happens before the expected one. Also, already wait for the event, we might be able to delete  in . I will create a PR to fix above.
/priority critical-urgent /remove-priority important-soon /milestone v1.13
while we should update the test with your PR, I'm not sure if it's going to fix the failure. Looking at one failing test log, these are the events I see on the failing pod: So it seems to be failing before even reaching the subpath logic.
Timeline of events: Test pod started getting processed by kubelet: It took a minute after the test pod was created for the csi driver to finish registering and mount to succeed, and then subpath event to trigger: So should address the issue. I was initially confused because the test logs didn't show the subpath event for some reason
I opened up to investigate if 1 minute for csi plugin registration is expected
I've a tracker issue for the complete list of storage flakes: This is so that we can track whether we're addressing most of the flakes with the subpath fixes, or if additional fixes are required.
keeping open until flakes are verified cleaned up
/reopen Reopening this issue until we can verify the fix in CI
As of this morning we are seeing CSI "subpath should fail" flakes in the following job Multiple "subpath should fail" flakes in this slow job alone Multiple "subpath should fail" flakes in the 1.13 slow job, similar to master. The following 2 tests are flaky only in this upgrade job can you help us determine if these are cases are flaky tests that you are fixing or is there an underlying CSI bug that needs addressing?
root cause of this is a delay in an external sidecar containers and the tests not waiting long enough to accommodate the delay. There are no changes required in (outside of test) to handle this. These should not be 1.13 blocking. We'll continue to look in to getting them non-flaky today. In the meantime, we can mark these tests as flaky to reduce the noise, if you'd like. looks like the same issue () is still being hit. See : And logs from (): We need to make the tests handle this condition more gracefully. The suggestion here is one possible solution.
Would be good to capture this as a known issue in release notes.
to followup on capturing this in Release notes
per , can you create a targeted fix increasing the timeout? Looks like marking these tests as flaky will be difficult for 1.13, but if that doesn't fix it we will need to do that.
Short term fix to bump the test timeout:
As per discussion in release burndown meeting, this is not a 1.13 blocker. please move this issue to 1.14 if there are no pending issues you are concerned about
Ok looks like the test still flaked. Looking at , the mount succeeded after a minute and proceeded to subpath processing, which (correctly) failed. However, for some resaon, the test is not seeing the subpath event and still failed after 4 minutes. On kubelet, subpath processing failed after a minute and generated the event: But the test doesn't see it for some reason, and still times out after 4 more minutes:
I saw the same on CSI volume mount succeeded at: Kubelet correctly failed the pod because of subpath But test still failed: Because the test failed to find the pod event. This looks like a test issue. Not a 1.13 blocker.
can you PTAL at this
Is there a way to check if events are overloaded and not able to handle new events? The MountVolume retries while the driver is not registered is very frequent and could have overloaded the event server.
Checking for the existence of an event seems like the wrong way to test this. Would checking pod status alone be insufficient? Ultimately that is what we really care about.
Checking for the specific event guarantees that we are failing with the correct error. But perhaps events are not reliable enough to depend on. Checking if the Pod never came up running would probably reduce the flakes but could potentially mask false negatives. cc for any thoughts.
Tests are still flaking and actually it seems like in general storage flakes in slow are flaking more since 11-21 or so: https://k8s- My theory is that the driver registration delay is causing a flood of events, which causes tests that check for events to fail. I think this is revealing multiple issues in both the tests and product, and we should work on fixing them all, although any one of the fixes should mitigate the test flakes: Driver registrar can sometimes take a minute to come up: Volume manager operation executor doesn't seem to be throttling retries or backing off: Tests should not rely on events: Tests should wait for CSI driver to be registered on all nodes before creating test pods:
and have been merged and will address the biggest issues. has been backported to 1.13 already, backport is still pending.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


What happened: POST or PUT to a custom resource with a JSON body missing "kind" results in a 500 error: A missing "apiVersion" also results in a 500 error: Then with both "kind" and "apiVersion" defined we get the normal 4XX return codes. What you expected to happen: All bad inputs for create and update requests should be 4XX return codes. E.g. Creating a with body of results in the 422 error: How to reproduce it: Create a CRD or use a cluster that already has a one prepared (GKE 1.11.0+ clusters have the CRD installed which you could test against). HTTP to that custom resource with a JSON body of . E.g. Environment: Kubernetes version (use ): 1.11.2 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): /sig api-machinery <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
This is fixed on master, but I can reproduce it on 1.11. Posting a custom resource with a JSON body of : on master: on 1.11:
we are transforming all non-status error into 500 status error. in this case, it's the decoder returning a which is non-status. a quick fix by . (i like this number)
instead of fixed, it's rather been shadowed by i suppose. now when that empty unstructured obj going thru the decoder, the gvk is explicitly coerced, so this case is not reproducable on master for now.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
I get the same response when trying to update the custom resource's 'status' sub-resource. On k8s v1.11.4
/assign
submitting the specified object on master returns a 400 badrequest, as desired /close
Closing this issue. <detailsIn response to : submitting the specified object on master returns a 400 badrequest, as desired Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: If kubelet plugin registration fails during the registration process, the kubelet never attempts to register the socket again (assuming the error was transient), unless the socket for the driver is deleted & recreated to trigger a new registration. What you expected to happen: Maybe retry with exponential backoff when kubelet plugin registration fails instead of just logging and waiting for another filesystem event like it does today. Of course, if a socket returns a terminal error (does not implement the kubelet plugin registration service, for example), we should not retry. Follow up on /milestone v1.14 /sig storage How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'd like to work on this issue
/assign
Re opening. was a temporary work around for this. is the permanent fix.
Hello! We are starting Bug Freeze for 1.15. Is this issue still planned on being incorporated into 1.15?
the dependent PR is not yet merged. can we get this in for 1.15?
said that is the "permanent fix." and that has merged. So let's close this out /close
Closing this issue. <detailsIn response to : said that is the "permanent fix." and that has merged. So let's close this out Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In the test failed due to namespace leakage: we created e2e-tests-load-30-nodepods-1-pfn9m, but never got response from api server (got 0-length response instead) we retried and created e2e-tests-load-30-nodepods-1-m9tq5 e2e-tests-load-30-nodepods-1-m9tq5 to f.namespacesToDelete and leaked -pfn9m next test case failed because e2e-tests-load-30-nodepods-1-pfn9m wasn't deleted. What you expected to happen: While it's unclear why we got 0-length response instead ( investigates that), we always can get error while the namespace was created (e.g. connection reset, timeout, io error). We should handle this case in retrying correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.14.0-alpha.0.793+ Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


What happened: removed reporting of the OutOfDisk condition in the kubelet. A 1.12 kubelet upgraded to a 1.13 kubelet leaves the previously reported OutOfDisk condition in place, and it eventually gets switched to Unknown status. What you expected to happen: If this condition is no longer reported, it should be removed by the kubelet. How to reproduce it (as minimally and precisely as possible): a 1.12 kubelet, let it report OutOfDisk condition status. to 1.13 the condition is no longer updated a while, the node controller flips the status to Unknown Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig node
The fix for this should be picked to 1.13
/assign happened to have a fix handy, if no one is working on it.
after upgrade kubelet should ignore this condition . or change it to 'DiskPressure'
Better not. What if both condition and exist?
should be picked to 1.13 also, I think.
13.2 doesn't have this patch; would it be possible to make it for 1.13.3?
Unfortunately this was not mentioned in the release notes (or I missed it), got hit by this too. 13.2 is affected too, would be good for me to see this getting fixed in 1.13.3


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When using the function from Python's Kuberneets API, we noticed that Pods are terminated bythe cluster without respecting their preStop hooks or terminationGracePeriodSeconds seconds. What you expected to happen: When using the eviction API, just as when using delete, Pods should be allowed to run their preStop hooks, as well as be given terminationGracePeriodSeconds before being terminated with SIGKILL. How to reproduce it (as minimally and precisely as possible): the following pod: the pod logs in another window to evict the pod. Here is a sample curl equivalent of what (1.13) is doing - happy path: Here is a sample curl equivalent of what Python's is doing (when not passed ) - broken path: The log output generated by the guinea pod on the happy path looks like this: The "Got signal 15" message shows up only after the preStop hook is finished, so ~10 seconds after the evict call is made What happens on the broken path is: We seem to be getting 2x SIGTERM, there are no messages indicating continued processing after the signal, and the signal itself is registered immediately after the eviction call is made, suggesting that preStop didn't run. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 Kernel (e.g. ): 3.10.0-862.14.4.el7.x8664 SMP Wed Sep 26 15:12:11 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: in-house Others:
/sig api-machinery
this is because of this code: invoked from here: it is passing in the eviction delete options field as-is, meaning nil gets propagated and defaulted to mean delete immediately. when calling delete APIs normally, delete options defaults to a non-nil empty set of options:
Fixed in
patch looks legit. Thanks for the quick reaction and for the pointers!
I've a elasticsearch stateful set which have a preHook which never completes and the pod is never killed despite terminationGracePeriodSeconds being set. Is it related ? If so can this be considered to be cherry-picked on v1.13.x release branch ?


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: I'm running a K8s 1.13 cluster and a partial netsplit happened due to a defective linecard dropping only TCP packets above a certain sizes between two masters, but the third could still communicate with both (so the fault made the network non-transitive). etcd was run on the physical network, but Kubernetes traffic runs on a Wireguard overlay for security and routing is provided by OSPF (so internal communication was not affected since it was tunelled over UDP). Etcd experienced some temporary issues due to the connections dropping only certain packets (above ~1000 bytes), but quickly dropped one host out of the quorum, at which point it was degraded but all the remaining hosts could communicate freely. Each master node has an etcd instance and an apiserver. All apiservers are anycasted inside the overlay to balance the load and provide redundancy and the announcement is coupled to the health check, so in case any apiserver fails no traffic will be routed to it. The problem was that the apiserver did not fail its healthcheck even though all etcd write requests timed out. The non-working apiserver was still accessible and got routed its share of the anycast traffic since that was routed over the UDP overlay. Thus ~1/3 of all workloads and nodes were stuck on a non-working apiserver until I nuked the non-working apiserver. TLDR: One etcd server started timing out all writes due to a non-transitive netsplit and the Kubernetes apiserver healthcheck did not fail, causing 1/3 of K8s clients to not have a working control plane until manual intervention. What you expected to happen: The health check should've failed (in my opinion, I'm happy to discuss if you think otherwise). How to reproduce it (as minimally and precisely as possible): Set up 3 etcds with corresponding apiservers, introduce a split between two (shouldn't have to be a packet size limit, a full disconnect should reproduce cleanly), observe that all three report healthy even though at least one of them no longer works. Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.0 Cloud provider or hardware configuration: Physical x86_64 servers OS (e.g. from /etc/os-release): Custom Kernel (e.g. ): Custom, based on 4.19. No patches affect this functionality. Install tools: Custom Others:
/sig api-machinery (maybe cluster-ops?)
cc
Looks like the etcdctl endpoint health command does a random read after fetching the list of members - - we should probably do the same?
Looks sane and would have prevented my issue. Should I PR this?
sure. thanks
/assign


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: azure file remount on Windows in same node would fail What you expected to happen: If user delete a pod with azure file mount in deployment and it would probably schedule a pod on same node, azure file mount will fail since command would fail if file share is already mounted on the node: It's related to bug: Error logs would be like following: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.9-v1.12, master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign
/sig windows
/sig azure
/milestone v1.14 cc This may be good to backport to 1.12 or 1.13 too if customers are using those versions.


We need to add Windows-specific DNS tests to cover the following fields: Test cases needed for : Default, ClusterFirst, None Test cases needed for Test cases needed for Test cases needed for These need to run on Windows containers because the domain resolution policy is different, so different results need to be checked. Windows assumes that any name with a in it is a FQDN, not a PQDN. There is an to cover simple PQDN resolution, but they don't cover the Pod DNS configurations above. /sig windows /milestone v1.14
FYI I'm working on this. I plan to start out by porting over to Windows.
I just moved it into the in progress column. THanks!
What test cases are needed for , , and ? My thoughts were that covers and . I'm adding test cases for Default and ClusterFirst. Does cover the test cases needed for and or do I need more? Or should I just be porting over the entire file?
For the first one - yes. The main differences will be checking the output of instead of , and maybe using instead of but the flow is the same Checking second one now
Agree on second one. I think that will cover it. It may make sense to simplify it a bit because the common methods used there are all based on running bash + dig. If it works with busybox, that may be feasible. The description at matches those test cases
If at all possible, could we use native Windows cmdlets such as instead of relying on Linux cmdlets that have been ported to Windows like ? Am not 100% confident if they are going to behave consistently.
yeah, I think what David's suggesting makes sense if the test cases are forked for Windows.
Opened PR for enabling custom DNS on Azure CNI:
/milestone clear This will need to merge after 1.14 as we're waiting on CNI updates for these tests to pass. I this status note to the Windows kep
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In an otherwise-stable bare-metal setup, with only LocalVolume and NFS storage, and without any volume mount activity, I get 80,000+ of these warning messages per day: What you expected to happen: No warnings; and if there is a reason to generate these warnings, more info about what operation is being attempted should be to the warnings message. How to reproduce it (as minimally and precisely as possible): I don't know. This issue emerged after I about 40 services and 300 volumes on a 3-worker cluster. Anything else we need to know?: Environment: Kubernetes version (use ): 1.13.0Cloud provider or hardware configuration: Bare-metal OS (e.g. from /etc/os-release): Ubuntu 18.04.1 bionic Kernel (e.g. ): 4.15.0-42-generic Install tools: kubeadm Others:
/sig storage


Which jobs are failing: Which test(s) are failing: Job Reason for failure --- --- Since when has it been failing: 2019-02-11 Anything else we need to know: started failing with commit started failing with commit cc /kind failing-test /priority important-soon /sig api-machinery


What happened: The preparation step of the e2e tests were failed sometimes on my local environment like What you expected to happen: These tests can be passed without any errors. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.1 Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): 16.04.4 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-138-generic Install tools: kubeadm Others:
/sig storage
After discussing on the PR I have known I need to enable StorageObjectInUseProtection plugin of admission controller which is disabled by default. The corresponding e2e tests require the plugin is enabled, so I think it is better to enable the plugin by default for most users.
Investigation of StorageObjectInUseProtection plugin of admission controller under kubernetes/kubernetes: cluster/centos/config- Enabled cluster/gce/config- Enabled cluster/gce/config- Enabled cluster/kubemark/gce/config- Enabled hack/local-up- Enabled cluster/juju/layers/kubernetes- Disabled This is the reason why we don't face this issue on the upstream CI system, and I it on my local environment which is deployed with kubeadm.


I have been deploying master-branch kubernetes on Windows using flannel/host-gateway networking, with one master, one linux worker, and three windows worker. Recently, kube-proxy stopped running correctly. It exits with an error about not finding the HNS Network. I have confirmed that the HNS network exists on the host with the correct name (cbr0), and have identified the last commit where kube-proxy works. What happened: Starting with the commit merging , kube-proxy outputs the following error message: This is confusing because does show : (nat network removed from above output) What you expected to happen: kube-proxy to work in host-gateway mode. How to reproduce it (as minimally and precisely as possible): Build kubernetes at or after . Start flanneld, kubelet, and kube-proxy on a windows 2019 vm. See kube-proxy fail despite flanneld having created the network. Anything else we need to know?: Environment: Kubernetes version (use ): kube-proxy breaks on commit and later; works on Cloud provider or hardware configuration: vsphere OS (e.g: ): windows server 2019, with up-to-date patches Kernel (e.g. ): windows nodes are , linux master/worker are Install tools: deployed using /sig windows cc
I'm aware of this issue. The fix will be out within an hour.


Which jobs are failing: All new tests introduced for api change Which test(s) are failing: New alpha subpathExpr tests [Feature:VolumeSubpathEnvExpansion][NodeAlphaFeature:VolumeSubpathEnvExpansion] Existing tests are working fine and the whole suite works correctly in the gce suite https://k8s- Since when has it been failing: 21st February 2019 (since merge of PR) Testgrid link: https://k8s- Reason for failure: Investigation shows that node e2e tests use a different framework pod creation pattern Doing some comparisons of tests that work and those that don't reveals differences in the standard client set as and the existing ones which do work go through the intermediate which uses The version has an additional function specifically for node tests
/sig storage
/sig node /sig test
/assign
/cc
/sig test


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: in Which test(s) are failing: Since when has it been failing: 2019-02-27 () -- since it was Testgrid link: Reason for failure: Timeout: Anything else we need to know: /kind failing-test /sig storage /priority important-soon /milestone v1.14 cc
cc Could you help with this please?
/assign
Neither of the failing tests mentioned in this issue are present in testgrid anymore. is all clear


What happened: Currently starting Kubectl failing on Azure Stack. What you expected to happen: Kubectl should start on Azure Stack environment. How to reproduce it (as minimally and precisely as possible): Try to deploy Kubernets on Azure Stack environment.. Anything else we need to know?: Environment: Kubernetes version (use kubectl version): master Cloud provider or hardware configuration: OS (e.g: cat /etc/os-release): Kernel (e.g. uname -a): Install tools: Others: /sig azure /assign
is it ? is a tool ...
my bad its kubelet. I update the title.
then could you provide the kubelet log? I am not sure kubelet on Azure stack is running as a container or process, you could get the logs by: And how do you deploy k8s on azure stack? cc


<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When referencing init container's env value from its fields, kubelet complains What you expected to happen: Init container should be supported to get env values from its own fields, as to How to reproduce it (as minimally and precisely as possible): a pod with init container refering its fields value as env value kubectl create -f test-pod-1.yaml cannot be created pod events Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The err happens at this line , it seems that we should search for init containers as well


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: Since when has it been failing: Ever Reason for failure: Vm on Azure with no public IP assigned aren't able to perform a command to the internet. Anything else we need to know: The test (see ) tries to test the internet connection for a pod by pinging 8.8.8.8. But azure does not provide ICMP support for VM without a public IP therefore this test cannot succeed.(see ) A fix would be to change the test method to a tcp request instead of a ping request (the test already performs a http get to in its BeforeSuite (see ))
/sig network /sig testing /sig azure
you may refer to (Pods don't have connectivity to external networks )
cc WDYT about the proposal to change this from an ICMP call? We probably need input from sig-architecture as well since this as well /sig architecture
/triage unresolved
An alternative is to issue a TCP/UDP request to 8.8.8.8 on port 53.
ICMP is dropped by Azure for security reasons. This is intended. Can use curl as a replacement. I will fix the corresponding test.
SGTM
Can you please review the related PR? I'll in the PR again. Thanks!


What would you like to be : Remove any logic in kube-proxy that tries to automatically clean up proxy rules based on the proxy mode. Only attempt to clean up if is set. Why is this needed: This is a continuation of a discussion from There's a lot of complexity that comes with switching the proxy mode on kube-proxy because it tries to delete proxy rules created from other proxy modes. This has a lot of side-effects and we should be encouraging users to reboot thier nodes along with switching proxy modes. One example of this can be found
/sig network
I can take this on - does this issue have/need a KEP?
I think not but will defer to
I think a micro-KEP is appropriate. Specifically to talk about compatibility with regards to flags and to talk about docs (not sure which, but we have to write this down :)
also to talk about --cleanup (I think it should probably clean up its own mode, not the others -- or maybe I am wrong, please think about the right thing here :)
/assign (Yay, I can do this now!)
/priority important-soon


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: LoopbackHostPort code is calling the which depends on the machine on which this test is running. Especially in this particular testase it looks for the ipv6 ip address which does not exist because the machine is not configured for ipv6. Because of the LoopbackHostPort function bahaves based on the interfaces configured, test will fail if machine is not really configured with ipv6. Need to rewrite the testcase to handle such situation. What you expected to happen: Testcase should pass
/sig network
/cc
/triage unresolved I am a bot run by
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved


What would you like to be : Change excludeCIDRs from []string to []*IPNet in Proxier and perform the parsing at construction of Proxier. Why is this needed: This was proposed by over comments in Pr This improves the performance of isIPInExcludeCIDRs since the cost of parsing becomes one time.
/sig network /kind cleanup
/remove-kind feature
/triage unresolved I am a bot run by
ping


What happened: We're running pod behind and sending logs via UDP to it. It is k8s v1.13.3 with ipvs services. When pod gets restarted logs getting blackholed. Simplified example on a node: So, old pod had IP 192.168.129.116, new pod - 192.168.128.193. Weight correctly forwards all new traffic to new pod, but there are some which are still sending traffic to non-existing IP: Minimal value for UDP connection expiration is 1s: But logs are being sent at much faster rates, so these broken connections are never expire. Issue introduced by Looks like this supposed to keep old pod running, while there are still connections to it But is there a reason to still keep these realServer entries, while old pod already cleaned up and no such IP exist? What you expected to happen: would cleanup all non-existing realServers from ipvs entries. How to reproduce it (as minimally and precisely as possible): Repro steps are same as for host-port issue: Anything else we need to know?: sysctls are default: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: on-premise OS (e.g: ): Debian 9 Kernel (e.g. ): 4.9.0-8-amd64 Install tools: Others:
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by
cc
We observe the same behavior in k8s v1.12.7 and it hits us hard. In our case Nginx in a pod uses coredns service IP as DNS resolver address and every time coredns pods get new addresses DNS queries are blackholed. Query rate is higher than ipvs UDP connection timeout (30s in our case), so the connection to a non-existent ip address stays forever. Kube-proxy keeps non-existent real server entry in ipvs service and conntrack entry is never getting removed either. Kube-proxy logs: ipvsadm -ln:
I've been putting some thoughts on the issue (we actually have the same one). Currently, we handle graceful termination the same way for udp as we do for tcp. However on heavily loaded UDP services, (DNS for instance) udp ports are reused quickly and old backends are not deleted fast enough. There are two possible solutions: use more aggressive timeouts (30s instead of the default 300s). Not perfect but better immediately delete real servers and associated connections when a backend is not ready anymore (very good for DNS queries but this means no graceful termination for udp) / what do you think? I'm not sure what the exact behavior with iptables I can create a detailed issue with options to have a wider discussion. Or even directly send an email to the sig-network group
I like the idea. BTW, iptables also does not support graceful termination for UDP connections.
this is very easy to implement: change the test here: make sure is set (which we do : )
If we agree this is the way to go, I can implement it (if iptables works this way this is the best solution, I agree) We could still tweak timeouts for TCP (defaults are very long: 900s, which can lead to complete blackholing for 15mn on a crash of a target because the connection is never closed)
I agree.
/area ipvs
If you have a test environment I have a test image with the fix discussed just above: Commit here: This is not perfect, but I believe it is much better. What happens is the following: udp target pod becomes NotReady IP is removed from RealServers immediately Because of any new packet reusing the port will generate a write error and the connection will be removed from the ipvs connection table (if the application retries you are fine) I think in addition to that we should set much more aggressive UDP timeouts (5s? instead of the default: 300s) and make it configurable with a flag for specific use-case (I haven't seen a use case for long lived udp "connections" with very low amount of traffic)
Sorry, i was unable to quick test this on local dev cluster. Tried to replace image with but it seems it does not work with k8s 1.13. It then generates a lot of the same messages, and no any ipvs VIPs are created. I've also tried on fresh new cluster with the same result. Then tried but was unable to even start the cluster (with modified config from our 1.13) Can you please cherry-pick this to 1.13 or 1.14 or am I doing something wrong?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
We tested this patch to kubernetes v1.12.7 and it works well in production. BTW, I don't know much about SCTP. But question is do we need to threat it as UDP or TCP in this case? But it seems more close to UDP.
Thanks for testing! The PR is limiting graceful termination to TCP which seems the safest. I don't know SCTP but making it behave like TCP would be an easy change.
This feels like a fundamental problem w/ endpoints... ideally the proxier would remove the realserver when the pod is nonexistent rather than relying on arbitrary timeouts. Otherwise, there is no way to support graceful termination with UDP based protocols. Not a big deal for most UDP things, but I imagine e.g. QUIC users would be affected. Immediately terminating udp connections is in line with the iptables behavior, which seems to be okay for most people, so the change seems reasonable until there is a better way to determine pod state.
The PR that was just merged will improve things a lot for UDP but I completely agree with you on the fundamental problem with endpoints. I think we should have 3 different states: Ready (accepting traffic), in which case we should route traffic to the endpoint NotReady (terminating for instance), in which case we should stop routing new traffic to the endpoint but should keep existing connections "Gone" (no in the endpoint object anymore), in which case we should remove the endpoint altogether Today the endpoint object is too simple to handle these differences. I hope the new EndpointCondition status in the EndpointSlice API will help address this problem. More than happy to integrate this in IPVS when it is released (cc / )


we need to add such a file under: test/e2e/framework see this file as an example: xref: /help /good-first-issue /area e2e-testing-framework /sig testing /kind cleanup
/remove-good-first-issue /remove-kind feature
/priority important-longterm /area e2e-test-framework
/assign
i think it might be a good idea to create the dep graph as JPG and have a chat about that during a meeting.
What are the restrictions that we are trying to place in the e2e framework? In the example given it looks like the only thing blacklisted is provider-specific implementations or the internal client. I know one of the recent changes was ensuring we dont use the internal client, is that all we want to blacklist for now? Or also all specific provider implementations? Some other dep we want to prevent?
I'd like to help on this issue, this would be my first issue so I'd need some help as well if that's OK.
/cc


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Make release of v1.14.2 failed. After facing the error () we did 4 : eaafa5952f8073f51b13dbeafe406019995c8c0ba commit in 's opening PR () Then the build process failed again: What we did was: we ran two commands called , after that, we did . And right after this command, the error happened. Environment: Kubernetes version (use ): v1.14.2 OS (e.g: ): ubuntu 16.04, bash version 4.3
/sig release


<!-- Please only use this template for submitting enhancement requests --What would you like to be : Introduce a mechanism that allows specifying the IP protocol version used in the e2e testing. Why is this needed: This is a followup on , my understanding is that in order to discriminate between IPv4 or IPv6 we have to use the test labels and this adds complexity to the testing automation and doesn't allow to send this information to the context of the tests. Having a mechanism to specify the IP protocol will simplify the testing for IPv4 and IPv6 and will allow reusing most of the code.
/sig testing /cc /assign Does this make sense? If affirmative, what should be the best way to add this functionality? Checking the code I think that or can provide this, don't know if checking some of the cluster config parameters, ...
the testing-commons group (sig-testing) maintains the framework, but ideally the decision on how to proceed in terms of adding ipv6 detection should come from sig-network. possibly having this as agenda for a sig-network meeting is a good start, then discussing it in a testing-commons meeting. /sig network the linked issue is a big one, so i don't have all the context here. but IMO existing tests should just work for ipv6 without the requirement for a new "IPV6" tag in test names. dual-stack or protocol specific tests should have their descriptions contain the details properly (i.e. what is this test doing) we might want to add an utility function under: (there is a PR to move it BTW ) then such an utility might be useful only to tests that need to detect the protocol. one of the goals is to refactor the code in testcontext / framework, so adding this logic there is less preferred.
/area e2e-test-framework
I think that we can add an option to test_context to specify the cluster IP family the same way that exists one for the cloud provider this is simpler because when setting up the CI jobs you will know beforehand what IP family you want to test, it also will allow reusing all the e2e code for IPv6 and Dual Stack (cc: and by default will be IPv4, so it´s backward compatible.
Being able to override and specify exactly what you want the test to beleive about your cluster is nice, but ideally it should generally work without specifying more flags. Surely we can detect this?
I was thinking during some time about this, detecting the IP family is very simple having IPv4 or IPv6 only clusters, but with the introduction of dual stack I´m afraid about the supportability and reliability of the autodetecting mechanisms in the long term. If you see the dual stack PRs there are still a lot of unknowns about the cluster behavior. My point is, let´s have something first stable that works and is simple, and once dual stack is implemented, we can replace the flag with a function in the framework . I guess that migrating from a flag variable to a function is straight forward, is just replacing by , what do you think?
in is as part of the context. do you mean adding as a function which will auto-detecting the ip family. less flags for the framework binary is preferred.
once we have this I will open a new issue to start cleaning up and , we no longer need to discriminate based on tags


What happened: The externalName of a Service is not allowed to end with a trailing dot. What you expected to happen: The externalName of a Service should allow a trailing dot. This is very useful to denote a fully qualified value of the CNAME record created by the service and prevent DNS-trial-and-error by appending different (cluster-local) search domains. How to reproduce it (as minimally and precisely as possible): Try to set as a externalName for a Service of type ExternalName. More details The reference reads: RFC-1123 section 6.1.4.3 also states: Also in the errata of RFC-1123 a reference to RFC-1034 is to reiterate on what a legal hostname is. Section 3.1 of RFC-1034 explains the use and validity of a trailing dot. So while one could argue that a valid RFC-1123 subdomain may have a trailing dot, it might still be best to not change the which is used in a lot of places, but instead change the validation locally. Anything else we need to know?: I'll send a PR. Environment: Not much applicable.
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


What happened: Kops runs from a pod manifest stored on the node, having the annotation. We've experienced the case where the cluster autoscaler turned on a new node and, once the node has been to the cluster, application pods have been admitted before . Once it was the turn of , all system resources were utilized, so the pod has not been regularly admitted. However, since it's a critical pod, the executed the , which has tried to kill () a pod whose container wasn't started yet (it was downloading the Docker image), which has caused to timeout and return the error . The admission of failed with the following error: The pod execution has not been retried, so we've got a node without running and its networking broken. What you expected to happen: The should be eventually admitted. Anything else we need to know?: A minimal log extraction for comprehension: From the "Failed to admit pod" log I have been able to trace back the core code involved: () Logs: "Failed to admit pod [POD] - [ERROR]" () Error message: "preemption: pod [POD] failed to evict [ERROR]" () -() Error message: "timeout waiting to kill pod" Environment: Kubernetes version: Cloud provider: AWS OS: ubuntu 16.04.6 LTS (Xenial Xerus) Kernel: Install tools: kops Network plugin and version: kubenet
/sig node
I think the following change may alleviate the situation you encountered:
Thanks for your reply. To my understanding, when the is called it will return the minimum set of pods required to free up resources, so even if only 1 pod eviction fails it's likely that at the end of the the will not have enough resources anyway to admit the critical pod. In your opinion / experience: be an option introducing retries in the in case the eviction of 1 pod fails? yes, do you have any suggestion of a similar design adopted somewhere else in the ?
Looking at getPodsToPreempt, I am not sure it returns the minimum set of Pods. For , retrying may not help much since the pod to be killed may not respond to the subsequent kill command.
Sorry to reiterate on this, but I double checked the code and - if I'm not missing anything - returns the minimum set of pods. Let's look at logic: pods by QOS class using that the total sum of evictable pods resources is enough to satify the requirements (resources to evict for the critical pod) the list of pods to evict, as the merge of three sublists: : the minimum set of pods to evict assuming we've already evicted all and all : the minimum set of pods to evict assuming we've already evicted all and the required : the minimum set of pods to evict to satify the requirements left Am I missing anything? A possible different approach to solve the issue is to refactor to return all input pods sorted by "should be evicted first". Then iterates over the full sorted list of potential pods to evict until the requirements are not satisfied (skipping to subtract pod resources from requirements if a pod fails on ) and breaking the loop as soon as the requirements are satisfied. What's your take on this? /cc who originally implemented this
w.r.t. approach outlined above, it makes sense.
Whether this evicts the "minimum set" depends on how you define the "minimum set". At the time this was implemented (k8s v1.5.0 IIRC), we didn't have PodPriority, so we used QoS to mirror the kubelet's eviction algorithm. It should probably prioritize low-priority pods for eviction today given the recent changes to the eviction algorithm to incorporate PodPriority. The current approach minimizes the eviction of guaranteed pods, and prioritizes evicting few larger pods over many smaller ones. Since eviction marks the evicted pod as failed, it should suffice to just admit the pod regardless of what is returned by the evictPod function. In future syncPod loops, the kubelet will retry the pod deletion steps that it was stuck on. It would eliminate this class of bugs entirely, which was the purpose of the feature in the first place.
See if the following aligns with your suggestion:
Something like that, yeah. I'm not sure the "Failed to admit pod" log warning is still required, but we can iron those things out during code review.
Doing like this it means we do always admit pods, while we should do it only for critical pods, correct?
Please see this The change is for critical pod(s).


What happened: logs a nil error on success, EG: . What you expected to happen: kube-proxy shouldn't log an error on success. Either a nil error should be interpreted as success, or an explicit exit boolean should be returned & evaluated. How to reproduce it (as minimally and precisely as possible): Run in master/1.15, or 1.14.2+. Anything else we need to know?: Code indicates that this is a problem ever since exiting was introduced (kube-proxy previously hung). It calls log.Fatal() on the return result, under the now-false assumption that an exit -a crash. Environment: Hyperkube, 1.14.2
/sig network /priority important-longterm /cc
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
GitHub didn't allow me to assign the following users: johscheuer. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would start looking into this but it seems I can't assign myself
the bot only works on org members
GitHub doesn't allow assigning people unless they are either members of the organization or repo "collaborators". There's no way around this. Kubernetes doesn't use the latter mechanism in favor of the org membership process. (some notes / links in the bot comment). You can certainly take a stab at this without assigning it though


<!-- Please only use this template for submitting enhancement requests --The built-in and cluster roles here are deprecated in 1.13 when this issue was resolved: /sig storage /milestone v1.17 /priorit important-longterm
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please only use this template for submitting enhancement requests -- /milestone v1.17 /priorit important-longterm Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /priority important-longterm
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /assign
Bug triage for 1.17 here with a gentle reminder that code freeze for this release is on November 14. Is this issue still intended for 1.17?