After the cluster was updated to enable PD CSI Driver, there was a race condition where AD Controller (kube-controller-manager) started to BEFORE the PD CSI Driver was up, thus AD Controller gave us the error Then AD Controller triggered the disk detach because the disk/volume was not populated in the desired state of world. Unfortunately the disk was in use during this detach and got corrupted. Since cluster upgrades can cause the controller manager and CSI drivers to start at different times, the AD controller needs to be more conservative when detaching We have not been able to reproduce this, it came up from a customer. /sig storage /assign <detailsk8s 1.23, CSI migration enabled </details<detailsGKE </details<details</details<details</details<details</details<details</details
/triage accepted
I think there's potentially two changes we should do: 1) for the initial DSW population, we should skip the check that if the csi driver is not installed. 2) in general, we should not skip volumes during dsw initial population if an error is returned. If we had some temporary api server error then that would cause volumes to not get to dsw and similar behavior happen. cc
for a repro, maybe you can try something like: a cluster with csi migration enabled (control plane + nodes) a pod with intree-volume and get it running the csi driver node daemonset. verify the driver is removed from CSINode kube-controller-manager
cc
/cc was trying to repro this, but on GKE with the managed driver which made it complicated.
Yeah maybe gce kube-up would be easier for a repro
If CSINode says that it has migrated GCP PD volume plugin to CSI, then IMO A/D controller should always attach GCP PD volumes to it using CSI, regardless if the CSI driver is present or not. In-tree volume plugin in the kubelet is disabled and can't mount any in-tree volumes anyway. To flip the migration feature gate / upgrade / downgrade kubelet, the cluster admin must drain the node first to make sure everything is detached. A/D controller should not see change of CSINode migration annotations with any volume attached to it.
/priority important-soon
Another option could be to process these volumes anyways but add them to DSOW with migration status as "uncertain", so as they can be re-processed later.


But surely it looks like might be easiest unless it has any downsides.
There wasn't a node update in this case---there was a control plane update which restarted the PD CSI driver, and it looks like the CSIDriver resource was updated or changed in some way which raced with the kube-controller-manager restart.
Is there anyway we can know if the csi driver is up and running(like a health check ping) before populating DSW?
We use the CSINode registration to determine that. The discussion here I think is concluding that we should not use that as a criteria for adding the volume to DSW. We only need to check that CSI migration is enabled on the node.
+1 we need to be very resilient against races, I think we have to assume a driver is in process of being installed at any point in the AD controller reconcilliation.
/assign sunnylovestiramisu
Reproduced with the steps: In /var/log/kube-controller-
Tested with the and doing the reproduce steps above. The error of does not show up anymore. The error message still get logged in the kube-controller- for debugging purpose: Ran Ran On the GCP console, the disk is used by node: Ran I redeploy the pd-csi-driver and then make sure it is up and running. Ran


We need to fix this potentially flaky test and add it back to conformance suite. Please see discussion in Also waiting on:
cc /sig testing
/area conformance
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
can we convert it to conformance now??
Once 1.15 opens up


Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestPreemptionRaces Testgrid link: /sig scheduling /help
/reopen I just saw this again
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Seems to be green: https://k8s-
It's still flaky on both master and PR testing: master: https://k8s- () PR testing: (). /reopen
Reopened this issue. <detailsIn response to : It's still flaky on both master and PR testing: PR testing: (). </details
From the above two logs, I'm feeling that there is a "phantom" event (very likely of type Update by enforced resync) triggering the pod in the completed N<subth</subtest, to be still alive in the N+1<subth</subtest.
/triage accepted working on a fix.
It would be great if we assign a different name to the Nth round test's preemptor pod. Currently, whatever this is Nth test, the preemptor pod is always called 'preemptor-pod'.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT What happened: When deleting/recreating pods backing a UDP service, traffic from external senders stops flowing. What you expected to happen: Traffic would continue to flow to new pods. How to reproduce it (as minimally and precisely as possible): Based on reading of similar issues and commit messages it seems like this would be most noticable on high-volume UDP applications, statsd and syslog are good examples. a deployment and service for a UDP app. In my test case, this would be the container. an external node, create a small test script utilizing a statsd library such as pystatsd. The most important part is that the sender uses the same source-port for every packet sent. Ex: the sender. Install tcpdump into the pod and watch for incoming packets, example from the test case: the pod tcpdump in the newly created pod. Observe that no new matching packets are flowing. existing conntrack rules on all nodes in the cluster. You should see traffic start flowing to the pod again. example command for clearing conntrack rules on the nodes: Anything else we need to know?: I have tested this only on GKE-managed clusters that utilize google forwarding-rules for external LBs, but I believe the issue may be present on any platform where iptables mode for kube-proxy is used. Tested and verified on 1.8.6-gke.0 and 1.7.11-gke.1. Also tested with both Google Container-optimized OS and the Ubuntu-based node images. I have only test this on GKE It appears this was a known issue and should have been cleared up by: From my reading of , it seems as if the func should exec() to clear up UDP entries on service endpoint change, but the only place I see that func called is from syncProxyRules(). That func looks complex and possibly like it is only run occasionally and not on service/endpoint changes. Environment: Kubernetes version (use ): (Also tested on 1.7.11-gke.1) Cloud provider or hardware configuration: GKE-managed clusters, google cloud. OS (e.g.


from /etc/os-release): tried both: and Kernel (e.g. ): and Install tools: Others:
/sig network
We also ran into this in kubernetes v1.7.6 using OpenShift 3.7:
~I believe this is the same as ~If so, it is fixed in 1.10: These are about HostPort, not NodePort, sorry for the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any other known workaround for this besides updating conntrack rules on the kube nodes? I tried deleting and recreating the service, but this didn't update the rules. When I restart the stream on the client side data flows correctly again (perhaps it's perceived as a new connection), but I have to restart the stream for each sender.
/remove-lifecycle stale
FWIW, this is not fixed in 1.10 - that fix was for HostPort, and this is still a bug for NodePort.
I am seeing this issue in 1.11. I am using LoadBalancer type.
has been working on a fix for this, FYI.
This issue is extremely impactful for telephony protocols that rely on static source ip and source port. Any update on a possible workaround? I was thinking a potential workaround might be to reduce the UDP timeout to something really short so the stale conntrack entries go away faster. Or maybe there is a way to flush conntrack on the event of pod deletion? Edit: Removed stupid idea of a workaround to disable conntrack for UDP. Can't disable conntrack for UDP because it's used for DNAT from node to container.
The workaround is to not use NodePorts. Other kinds of services are already correct.
not sure what you mean. I am seeing the issue with a LoadBalancer type so it does not affect NodePorts only. As a workaround I have created a small python script that watches the K8s API for pod deletions and clears the related conntrack entries when that happens. I can share it when I get to work on Monday if you want to try it out.


I've my script . Note that it's not tested much so don't use it in a production environment. Also it cleans up all conntrack entries for a deleted pod, not only the UDP related ones.
We are also seeing this issue with type loadbalancer. We are on 1.11. Our use-case is the following: multiple devices are sending to an aggreagator machine. This machine forwards the UDP packages to a loadBalancer in gcp. When debugging the service, I noticed the following: The traffic is assigned to one pod, and the packages are not loadbalanced (client and server are the same cfr use-case) When redeploying a new cluster, Conntrack is cleared on termination signal of the old pod. (client receives connection refused) red/black and highlander deployment strategies have the same effect. How would I expect this to be working then? New cluster is . Conntrack is cleared Data is routed to NEW pods old pods are terminated Best Regards, Tim
I am seeing this exact same issue on our cluster which has k8s v1.14.5 . We have endpoint nginx pods backing a UDP service with nodeport. Suddenly one day we see traffic not hitting into nginx pod but reaching the node hosting the nginx pod. We could simulate the problem by restarting the nginx pods. I am assuming with this issue marked as closed; and having made its way into v1.14; we should not be seeing this. Would also like to point out; the one thing, in addition we have in the service is ExternalTrafficPolicy set to Local. Can anyone please guide/help on why we are seeing this in v.1.14.5 of our cluster.
Seeing this with k8s v1.16.5


What happened: Visited It shows the release notes for v1.17 What you expected to happen: See the release notes for v1.18 How to reproduce it (as minimally and precisely as possible): Open in any browser or with curl
/sig docs
/assign


The test is in flaky suite and no one seems to care. We need an owner for this test. cc Last failure log:
do you think you can take a look at this?
This is happening because of a bad assumption. The assumption is: if the reflector's store is consistent with etcd, the store given back to clients of the framework.Controller is too. I'd really like a general fix applicable to all controllers for this problem, but fixing it just for the RCs is pretty easy. In more detail: This function reports when the reflector has listed from apiserver. But the store given to the reflector is the fifo of the framework.Controller. The store given back to the caller is still not consistent, and will not be till the framework.Controller puts all the objects from the fifo into the client state store (). The rc specific fix: Before starting the rc manager, seed the expectations store with an expectation like {rc:foo, adds: foo.Status.Replicas, deletes:0}. This will wait for status.Replicas number of ADD events till a timeout. The general fix: Somehow expose this through framework.Controller.HasSynced (clients don't care about the fifo anyway, shouldn't it just be an implementation detail?) ideas? cc'ing people who have written controllers that might be interested I don't think this (and generally anything flaky for 1 release) is a p0, I can fix it if you don't mind dealing with the flakes till I've cleared things off my plate. I'd be more than happy to review a fix.
Well, it's a P0 because test is flaky. E.g last run:
- did you have chance to look into it?
The "DaemonRestart" tests are still labeled at . I will send a PR to promote them.
are you still planning to send aPR to promote the DaemonRestart tests out of flaky? I think this is the right thing to do.
Ack; sorry, I lost this in the muck.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: flag is not supported on Azure Stack, should switch to use original way to remove a disk from data disk list, refer to for more details What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


only check that the output includes the correct keys - they don't confirm that it has appropriate values. Doing full validation on the version data, or at least enough to catch null-string values, would guard against failures like the bad functionality seen in OpenSUSE's binaries, which have been shipping with many of the fields left completely blank (as I discovered ).
/sig testing
/remove-sig testing /sig cli Edit: testing owns the framework, binaries, infrastructure etc. for testing. SIG CLI is nominally responsible for their test cases.
If SIGs are responsible for their own testcases, then this touches test cases for api-machinery as well (I'm not sure what the non-CLI version of this is, but there's definitely a schema adherence test that should fail at the API level). /sig api-machinery
/triage accepted /assign
GitHub didn't allow me to assign the following users: SaiHarshaK. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /triage accepted /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


After KCM restart, any Pod that reuses a volume that failed to detach gets stuck ContainerCreating with message: User "system:node:ip-10-0-243-" cannot get resource "volumeattachments" in API group "" at the cluster scope: no relationship found between node 'ip-10-0-243-' and this object See steps to reproduce below, it's quite a complex condition. It happens when a pod is restarted, a CSI driver "busy" (i.e. times out) + KCM restarted at a wrong time. Pods start just fine. a Pod that uses an attachable CSI volume. the Pod. Make sure it unmounts successfully, but any fails with a transient error, such as timeout. The external attacher continues to detach the volume at this point. kube-controller-manager (KCM) and start a new one. Wait for any leader election etc - it's important for the new KCM to populate A/D controller with no pod using the detaching volume. Let the A/D controller to observe detach error. a new Pod on the same node that uses the same volume. the CSI driver to detach the volume (i.e. stop injecting any transient errors). At this point: The external attacher succeeds detaching the volume, VolumeAttachment is deleted. The volume is marked as attached in node status (!!!) despite no VolumeAttachment exists. A/D controller does not even try to create a new VolumeAttachment, the volume is marked as attached in its Actual State of World (ASW). Kubelet tries to mount the volume, but WaitForAttach fails with: , basically forever. I created . The script expects that the cluster is created by , because it restarts KCM in the middle. It uses my hacked hostpath CSI driver to inject detach errors. Sources are at / the last commit(s) there. There are two issues that together lead to this situation: fails, , i.e. puts it back to . With the volume in , kubelet may try to mount it, while the external attacher tries to detach it. It is not as bad as it sounds, because kubelet can't read VolumeAttachment with the deletion timestamp and it will not call / of un-attached volume.


KCM starts, all VolumeAttachments that don't have a Pod (that's good), but also (that's probably bad). When the newly started KCM re-tries to detach the volume and gets an error again, it marks the volume as fully attached (same as the previous point) and adds it to . Now the volume is both mounted + attached in ASW and A/D controller actually stops here and waits forever for kubelet to unmount the volume. <detailsToday's k/k master, </details<detailsNone </details<detailsN/A, it's KCM issue. </details<details, but seen in a downstream Kubernetes distro too. </details<details</details<details</details
/sig storage
/assign
/triage accepted
I have the same problem: Seems restart of Kubelet solves this problem. Kubernetes v1.25.13


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In the test failed due to namespace leakage: we created e2e-tests-load-30-nodepods-1-pfn9m, but never got response from api server (got 0-length response instead) we retried and created e2e-tests-load-30-nodepods-1-m9tq5 e2e-tests-load-30-nodepods-1-m9tq5 to f.namespacesToDelete and leaked -pfn9m next test case failed because e2e-tests-load-30-nodepods-1-pfn9m wasn't deleted. What you expected to happen: While it's unclear why we got 0-length response instead ( investigates that), we always can get error while the namespace was created (e.g. connection reset, timeout, io error). We should handle this case in retrying correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.14.0-alpha.0.793+ Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Hello, the SRC folks asked that I file this publicly: In the client-go repo, , function creates a private key and stores it into a file with mode : Typically keys are created with more restrictive modes to prevent other users or services on the system from being able to copy them before the user fixes the permissions. Of course, if a more restrictive umask is in place, this may not be a big deal, but it's an extremely fickle piece of the environment to rely upon for security. It's better to be explicit. Thanks
/kind bug Not sure if this is /sig api-machinery or /sig auth
is this related to us?
Hi IMO, mode make sense for the key file. I would like to work on this, waiting for Thanks!!
/triage accepted
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/lifecycle frozen /good-first-issue I think changing the code to use should be fine. Anyone is free to pick this up. Would be good to include a unit test.
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /lifecycle frozen /good-first-issue Instructions for interacting with me using PR comments are available .


If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Hello, this is my first ever open source contribution. I am trying to understand and get this fixed. Any suggestions would be helpful!
/assign
/assign


=== RUN TestStructuredAuthenticationConfigReload/oldinvalidconfigtonewvalidconfig first seen in 3/11 No response /sig api-machinery instrumentation
/sig auth /remove-sig api-machinery instrumentation /triage accepted /assign


attemting to run throws and error with ERROR: Error: - does not specify a valid ReplicationController
The documentation is a little ambiguous, as the second example suggests that - should work but the Synopsis section does not. Could you either fix the documentation or the command?
Calvin, can you post the contents of , and the schema of the original rc? There are some for the schema when providing a target from file, which may not be met.
running works, I was interpreting the error message that it though - was a rc
will work for any object schema file. I.e, that is the same command you use to create a podreplicationcontroller from file. The type being created is determined by the contents of the file. Looking back at the code, the error you saw indicates that does not contain a replicationcontroller spec. If you can create it, that must mean it's something else; can you post/link the contents?
yes I need to clean it up first
That looks like a valid rc spec, but the labels may be insufficient for rollingupdate. In order to do a rollingupdate you need to have a label to control the update. Typically this would be a "version" label. Example Basically the selector for the old and new rc must share at least one label key with a differing value. By the way, rollingupdate is currently getting a simplified version, which generates the new rc spec for you. This is wip that will be available in next week's GKE release. See
ah I see, I actually have a dynamic version in there so that will be easy to do thanks Jeff Lowdermilk wrote:
so with an updated config still does not work while does work with the first one still erroring out with so likely my missing selector issue was an unrelated issue
From your last comment, seems this is resolved. If you are still having trouble, lmk and I will reopen.
still not fixed because I can't do
I'm pretty sure that's a piping issue with the wrapper command that likely doesn't occur if you use kubectl directly. Try Note that steps 1,2 will be unnecessary with the next gcloud release; you will be able to use kubectl directly after creating a cluster.
With cloudsdk 0.9.60, standalone kubectl works by default with GKE clusters.


The wrapper command is no longer needed and will be removed in a future release. Let me know if you have problems running against a GKE cluster.
I am running across this issue with a recent build of .
is still throwing errors with version 0.17 used directly (without wrapping)


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Currently when using it adds as one of the limiters. This does not work as intended because there is no immediate sleep following the call to the limiter as seen in which does work correctly. The issue with the in its current usage is there is no forget nor a sleep immediately following the call so if is called thousands of times in a row, the timeout grows very quickly with no max. For example, if a clusters has 10k secrets and those are all enqueued using the the will end up with a 16 min wait before the tail end of objects can be enqueued and processed. I would expect some max backoff to take affect. This can be seen at
/sig api-machinery
/cc /triage accepted
it possible to set maxWaitingTime for BucketRateLimiter? when r.Limiter.Reserve().Delay() maxWaitingTime, the return of BucketRateLimiter When() is maxWaitingTime. defined is 1000ms / r.limit * MAXQPS(5000) this issue example (r.tokens = 100, r.limit = 10), if call When() times, the tail end of the object is enqueued waiting for 8.33 min. i'm not sure how to set the value of MAXQPS please help review it
The fix merged to close this issue does not fix the underlying issue with the DefaultControllerRateLimiter. The bucket is not being used correctly so this does not work as intended, please read my issue description again as to what is actually happening.
is another fix needed to solve this issue?
A token bucket rate limiter will only refill at the defined rate. If enough requeues are , it will take a long time for the refill rate to clear the queue. If a max wait is introduced, then requeues will be inserted into the queue faster than the refill rate of the token bucket rate limiter. A max wait will violate the token bucket rate limiter and we want token bucket rate limiting behavior.


Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
this appears to have always been broken, I'm going to disable the job for now. https://k8s-
Failed: https://k8s- Run so broken it didn't make JUnit output!


Aggregated discovery v2beta1 fixtures should be removed. /sig api-machinery /triage accepted


What happened: After creating Kubernetes service on a Kubernetes cluster built on Azure IaaS (not AKS), nodes which are not master nodes but are also not worker nodes and which are labeled with are included in the backend pool for the load balancer. What you expected to happen: All nodes labeled with should be excluded from the load balancer's backend pool. How to reproduce it (as minimally and precisely as possible): Create a node with a type other than and label it with Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-05-20T14:00:52Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-05-20T13:51:56Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Azure OS (e.g: ): CentOS Linux release 7.8.2003 (Core) Kernel (e.g. ): Linux master- 3.10.0-1127.8.2.el7.x8664 SMP Tue May 12 16:57:42 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Custom terraform stack Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/priority important-soon /sig cloud-provider /area provider/azure
/priority critical-urgent


I followed the document to create an e2e cluster However, doesn't work after the cluster is up. I got the following error: /cc
/cc
where are you trying to ssh from? Can you ssh to a Linux instance in GCE? If you run do you see ssh keys there? Do any of them match a public key that you have under ? There could be a delay the very first time you try to ssh; we noted this in , but we could add it to the README too. If you try a second time after a couple minutes does it still fail?
Yeah I can ssh to a linux instance in the same cluster. It still fails after a long. I've removed all my ssh keys from cloud metadata, and recreating a cluster. Let me see how it goes this time.
I still got the same error after recreating the cluster. I can ssh into linux nodes in the same cluster just fine.


Keep running but not crashing Have a long volume attach path This corresponds to these lines We are using k0s <details</detailsbare metal <details</details<details</details<details</details<details</details
I turned to use the JSON logger, it is gorgeous no wonder it failed
related
/sig node storage
/cc
VolumeToAttach contains Pod pointers, so yes, I think we should not log the entire struct, and only log important parts to it. /triage accepted /help-wanted
/priority important-longterm /assign


Every time watch is reset, event exporter exports an entry about it to Stackdriver. To make it less irritating, this has to be done once per container run Related image PR: /cc


As planned for 1.12 in kubernetes/features: After , the next step is to add the logic to the reconciler to detach the volumes when the nodeShutdown taint is applied. /assign /sig storage /kind feature looping
How do you know that node is Shutdown? is it a planned shutdown or a unplanned shutdown?
AFAIK regardless of whether it's planned or not, it's detected by the cloud provider. In case of bare-metal node it doesn't happen automatically, as there is no universal way to reliably determine if a node is powered off, so the node must be tainted as by some external source.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
any update on this issue?
, I am currently working with sig-node team on handling node shut down. I will make a proposal and discuss in sig-node meeting next Tuesday. Will post the update soon.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /kind bug /priority critical-urgent /assign /sig node
Those labels are not set on the issue: <detailsIn response to : /remove-kind feature Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
any update on this? is it going to be part of 1.15?
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .


Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/priority important-longterm
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/triage accepted /kind feature
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this can be closed now as we have non-graceful node shutdown.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: https://k8s- Which test(s) are failing: should unmount if pod is force deleted while kubelet is down [Disruptive][Slow] Since when has it been failing: 11/07/18 15:02 Testgrid link: https://k8s- Reason for failure: The test case stops and starts kubelet, and it seems like kubelet is unable to start again Anything else we need to know: <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
cc
Thanks for opening this Looking at , looks like could be the cause? /assign can you please look into this as priority. This is failing critical release blocking tests. /cc /sig cli /priority critical-urgent /milestone v1.13
It looks like kubelet keeps restarting. The error right before restarting is:
/assign
This bug could have been exposed by the newly tests in
Hmm, so I see two errors: Which is a Fatal error and would prevent Kubelet from starting anyways. In which case this error is handled correctly by the pluginwatcher, but I'm not sure preventing Kubelet from starting is the desired behavior. Should we log this error and skip it? There also seem to be a cleanup issue here on that file is this expected? This one is clearly a bug and not really handled correctly. It's due to the fact that we encounter an error at start, cleanup but don't wait for the gorountines to finish before cleaning up. I can send in a fix pretty easily for the second issue, basically add a waitgroup in the . For the first error, what do we think is the expected behavior?
This was mentioned in as well. /cc
Is this related to the issue where plugin watcher is recursively scanning for sockets?
Also, it should be noted that the path it's erroring on is not a socket type.
That's a bit different, as it is intended to scan recursively. Also note that the error here is produced by . What should be our behavior in such a case? Ignore that error?
I'm not sure if we want to be recursively scanning. The volume directories are mount points, which means there could be lots of files/directories existing in those volumes.


maybe we can put a limit on depth of plugin directory for on the same PR:
Ideally the directory that the plugin sockets are dropped in should be different from the directory where mounts happens. Currently the plugin socket directory is: ) The mounts directory is (so far so good). Problem is the "global mount" directory is also :( So options are: recursively scanning like suggested, and say plugins will can only be directly under This will break existing plugins, for example, the GCP plugin is at (see ). recursively scanning but change the plugin socket directory to ? This will also break existing plugins, like the GCP one pointed out above. the plugin dir the same, and leave recursion in, but put a limit on depth of plugin directory as suggested by This is ugly but may be the cleanest solution.
This conversation is completely different from the bug surfaced by this issue. Let's move that to the corresponding issue: I have a fix ready to go out for the panic, we should figure out what behavior we expect Kubelet to have when encountering this error from filepath.Walk: Should we ignore it?
Ignoring the error (with a warning maybe) instead of failing all of kubelet seems reasonable to me. If the plugin did not setup their registration socket correctly, then the end result should be that the plugin fails to register, but all other plugins should continue to work correctly.
the latest runs of this job no longer timesout - Instead I am seeing a new CSI related failure Is this expected. Do we want to track that failure in this bug now or do we have a separate issue?
the latest test logs indicate some test-infra issue:
the same shows up as a timeout on that particular test. Lets wait for the next run. but somehow the timeout seems to have gone. Are we still planning to merge and associated with this issue?
should be merged as it fixes a bug. can you help with approval? Thanks!
are you missing the right PR# in the comment above?
this is the correct PR -
/reopen Keeping it open until we can see the job pass a couple of rounds
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.


</details
I see just one CSI test timing out in serial job Is that something the fixes here attempted to resolve? if not I can open a new issue if it persists and close this one out. Thanks.
The test mentioned in my comment above was a flake.. closing this issue now. will open a new issue for the flake
I'm getting the following related error on a busy Node: In the above case it's a full filesystem, but I imagine the same error would pop up with a RO filesystem as well. Looking at the patch, I'd expect kubelet would still be failing to start when it tries to watch it. (above was from a v1.12.3 cluster) , what do you reckon?
plugin watcher fixes for 1.12 are expected to be released with 1.12.5:
, I understand that I'm missing that patch. The patch code, however, would not make a difference in this case, if I read it correctly. It would not prevent the recursive dive into mounted volumes, as the errors happening at the deeper levels of that dive.
The patch blacklists the plugin watcher from searching the directory
OK, I was looking at the wrong patch after all (). Thanks for the clarifications.


master-informing: capz-windows-master ! 07-10 17:46 IST No response /sig cli cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test
Looking at the failure and tracing the recent code changes, the test was recently modified for the regex pattern match as part of this PR which comes with deprecation of the golang-based short version information in favor of making the --short the new default. But the output of kubectl version is still reflecting the older long format which is causing this test to fail.
/assign


Is this a BUG REPORT or FEATURE REQUEST?: BUG /kind bug What happened: If you are using a mac and GCE and you specify NODELOCALSSDS=0 when using GCE via kube-up or e2e --up, then you end up getting 2 local SSDs. What you expected to happen: I would expect to get 0 local SSDs by default and when I specify NODELOCALSSDS=0 explicitly. How to reproduce it (as minimally and precisely as possible): On mac execute: NODELOCALSSDS=0 go run -- -v --up Anything else we need to know?: I've discovered that this issue is due to If NODELOCALSSDS=0, this evaluates to $(seq 0) From man seq on mac osx This was run on mac with the seq manpage indicating it comes from BSD Feb 19 2010. Note: some unix/linux variants, e.g. a test on ubuntu with coreutils 8.25 and a test on centos-7 with coreutils 8.22, had a different behavior that does not demonstrate this bug. Environment: Cloud provider or hardware configuration**: GCE OS (e.g. from /etc/os-release): mac osx
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage


If the namespace controller gets an error when doing a sync and the error is not a error the error will not be handled. Ran in to this when we found When deleting a namespace it received a error that wasn't reported anywhere and the namespace stuck in because a finalizer was left.
Other things noticed in the namespace controller: need a crash handler in sweep for all error return cases and determine whether should be returned determine what to do with non- errors at ... I think that currently it just hangs termination of that namespace until a resync
I will take a look. Thanks for the heads up.


reported a variety of failure modes in .
+1
I figured I'd add to this issue rather than create a new one. I attempted to create a vagrant cluster today (dev tree is up to date) with 1 Master and 1 Minion and the cluster creation is stuck in the 'Validating master' loop ( for over 10 minutes in the validating loop ). salt-master and salt-minion logs seem to be empty ( they don't appear to be running, in fact ).
I still have the cluster running in case anybody wants additional information.
/cc please let me know if there is any other information about my vagrant setup that I need to post here.
I have been working to get more stability in this environment over last few days. Can you try again from HEAD after this merges:
Can you also verify that you built a release prior to running cluster/Kube-up?
yes, I started from scratch and built a release before running cluster/kube- - cluster/kube- seems to fail fairly early in case there is no release build available.
It looks like got merged. I'll try again and get back to you.
The vagrant cluster seems to come up ( completes master and minion validation ). However, there is new credentials related error that is showing up at the end of running cluster/kube- :
I'll add one more problem I've encountered. Runing from head:
New to me. Will investigate more tomorrow. Sent from my iPhone
Also new to me :-) - will see if I can reproduce either. Sent from my iPhone
Your credendials and certificates are not working with new environment. Just delete your .kubernetesvagrant* files and recreate the cluster. It would be nice to automate this task, if you tear down the cluster.
I am able to reproduce the 502 Bad Gateway error and am working on determining what changed in order to provide a resolution over the last couple days. Please assign this issue to me.
The issue appears to me that the kube-apiserver is not getting provisioned by salt correctly. Parsing Salt logs to determine why.
Looks like the file is not getting laid down by salt for some reason ;-)
The Jinja template was changed as part of identifying the cluster. Fixing up now, and will send a PR to close this issue.
that was it, thanks!


What happened: In the PMEM-CSI CI, we rapidly deploy and remove a custom resources which then owns all objects created for that resource. Deletion of those objects is handled through ownership references, i.e. deleting the CR is expected to cascade to all objects. This worked fine up to Kubernetes 1.17. When starting to test with 1.18.2, we pretty reliably get into a state where the CR and the service objects owned by it are marked for deletion, but don't go away. Debugging with the garbage collector's dependency graph (see ) showed that endpointslices were blocking the deletion of the services and thus the CR. In the graph dump (= a snaphot in time), they were not even marked for deletion. A closer look showed that endpointslices for the services were getting created and deleted continuously for over 15 minutes. Eventually, the garbage collector won and managed to delete endpointslices and services before new endpointslices were created. What you expected to happen: There seems to be a missing check in if the service is marked for deletion (i.e. has a deletionTimestamp), then the controller should not create new endpointslices. How to reproduce it (as minimally and precisely as possible): I tried to boil it down to a script, but it just ran fine. The "easiest" way to reproduce it is to run PMEM-CSI's E2E testing, which is probably not easy enough :disappointed: Anything else we need to know?: Disabling the EndpointSlice feature makes the problem go away. Environment: Kubernetes version (use ): 1.18.2 Cloud provider or hardware configuration: QEMU virtual machines OS (e.g: ): Fedora Install tools: kubeadm /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
cc for endpointslice
Thanks for the detailed bug report! Will work on getting a fix in soon. /assign
/remove-triage unresolved
Can you describe the general process to reproduce this bug? Would it involve hundreds of EndpointSlices for a Service? I've tried to recreate this in a , but unfortunately I haven't been able to get anything to break. Any ideas for what I could try?
There's one endpointslice per service. Our operator creates two services.


One peculiarity may be that we don't actually want the pods to start during these tests because we only want to verify object creation, not the usage of the result, and therefore all pods reference non-existent images. All objects immediately get deleted once the test has verified that they exist, so the timing is probably very different from "normal" usage patterns. I suspect that it is more important to create and delete services quickly than it is to have many endpointslices per service.
I had tried to come up with a script that triggers the problem, without success. That was before I realized that endpointslices are involved. On Monday I can have another look.
That was because I thought that (the default) enables foreground deletion. It doesn't, and there's also no option to select it. After explicitly selecting it via curl, the following script failed for me eventually: Output: Note that for the endpointslice is greater than the of its service.


What happened: One pod stuck at CreateContainerError and can't get recovered automatically. Kubelet log shows that the requested name (let's say Container2) is already in use. shows the Container2 Running, and it's created before Container1 This should never happen. How to reproduce it (as minimally and precisely as possible): I wish i knew. Anything else we need to know?: I know this can be caused by time shifting, but I didn't find any time shifting sign in system log and chrony log, and there is no rebooting between the create timestamp of Container1 and Container2, so this should be caused by something else. This happens intermittently in our env, like 3 times in half a year, and I didn't find any ime shifting sign each time. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-13T11:20:18Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-08-28T11:18:26Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): CentOS Linux release 7.5.1804 (Core) Kernel (e.g. ): Linux node-1 3.10.0-957.21.3.el7.x8664 SMP Wed Jul 17 06:57:50 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
/sig node
Any update on this issue, we are also facing the same issue. EKS version : 1.14.9 Docker version: 19.03.6-ce AWS CNI: 1.6.2 Intermittently we are also facing this , what might the issue, could please guide us here? Below are the kubelet logs. )"


with CreatePodSandboxError: "CreatePodSandbox for pod "-f0fe-11ea-9872-)" failed: rpc error: code = Unknown desc = failed to create a sandbox for pod "aiqx-server-dep--b9lmv": operation timeout: context deadline exceeded" Sep 07 11:40:40 kubelet[2899]: W0907 11:40:40. 2899 ] No ref for container "docker://7b9c86b529530f15d941f8e68973a202c8f348ab5db864a8371e8c82ddbf9dc4" (r(-e845-11ea-bd96-) Sep 07 11:40:43 kubelet[2899]: W0907 11:40:43. 2899 ] Unable to create pod sandbox due to conflict. Attempting to remove sandbox "a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8" Sep 07 11:40:43 kubelet[2899]: E0907 11:40:43. 2899 ] Failed to remove the conflicting container "a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8": Error: No such container: a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


/lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


This issue is here to satisfy PR requirements of issue association. A lot of times we need to do version bumps for minor bug fixes that don't deserve a separate issue in the kubernetes/kubernetes repo. /kind bug /sig node
/lifecycle frozen
all the linked PRs are emrged and no activity since more than 3 years /close
Closing this issue. <detailsIn response to : all the linked PRs are emrged and no activity since more than 3 years /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


committed a changed version of go into godeps. Was this deliberate? I don't see any discussion of it ( I've almost done this by mistake as well, so I'm inclined to think it was just an oversight. (I think this is giving me errors building, but I'm not yet 100% sure this is the cause)
Hmm, this is a good point. It does look to be accidental. I don't know. I think and would have context though I believe they are out— do you know offhand whether this is probably an accident? I guess were talking specifically about in the .
I believe it's an accident.
Thanks, The linked PR should fix this (assuming I don't have to do anything else with Godeps).
This was almost certainly an accident. Whenever I've updated godeps it's changed automatically to 1.3 and I've had to adjust it back manually.
Actually, jeff lowdermilk was super helpful and showed me how to update the deps on this PR. it was his commit which I merged into my PR, so not sure it it was accidental or not. Fortunately, now that k8petstore is in the go build, we will know if it breaks anything when you revert the version back. Thanks for noticing this. , Alex Robinson wrote:


If I have probe and then all good. Previous version of the pod is terminated after startup probe success, but if wasn't specified the pod becomes ready immediately. It would be nice to add new state (for example starting) which would have pods until a is successful. Also, it would mean that container not ready for connection. Thank you for attention!
/sig cluster-lifecycle
/remove-sig cluster-lifecycle /sig apps try asking in #kubernetes-users on k8s slack.
they are silent in the chat. Who is involved in the development of startup probe?
i think this topic better fits SIG Node. try asking in their channel on slack , but be patient ;) /sig node /remove-sig apps
Those labels are not set on the issue: <detailsIn response to : i think this topic better fits SIG Node. /sig node /remove-sig apps </details
I think this is a feature request, not support, so I changed labels. On its face it SEEMS reasonable, but I am not close enough to the details to say. This is definitely sig-node.
Adding a new pod lifecycle state has been determined to be intractable in the past. Is the proposed change keeping the pod in !Ready until the first success from ?
Where do you usually discuss this? Just wondering. Yes, I want that pod considered not ready until will not be successful. If readiness probe doesn't present then a pod can become ready. How can a pod be ready if it is not running?
how long wait?
Looking at this issue, I can confirm it, at least superficially - a pod with a perma-failing is "ready" if there is no readinessGate defined and "not ready" if it is defined. The question I have is whether we think that is "correct" or not? Initially I thought it was correct, but the more I think about it the less convinced I am. If I have defined a startupProbe and it is not passing, my app should not be ready. Does anyone disagree?
Also, the app should not be ready in status. It's funny, but without a readiness probe, pods in the final state have the status ready and accept requests. This is unexpected.
Whether a pod is ready while terminating is an entirely different question, IMO.


Apps that server network traffic probably need to "drain" and to do that they need to stay "ready" for a while as upstream LBs de-program. Let's not conflate issues, please.
The fix seems simple - lets see what CI says.
Sorry, I don't spend much time on slack... you could have checked the release notes in 1.18 ;-) Tim's fix seems good, thanks!
/reopen Until we merge all related PRs...
Reopened this issue. <detailsIn response to : /reopen Until we merge all related PRs... Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
All merged. /close
Closing this issue. <detailsIn response to : All merged. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When service is set with ILB annotation and empty string subnet, e.g. The service would be pending forever with following error: What you expected to happen: ILB should be provisioned successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /kind bug /assign
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
Thanks


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When attempting to upgrade kubernetes using kubeadm upgrade apply, kubeadm fails to parse an etcd imagetag containing build metadata in the form of as a valid semver. An example upgrade produces an error similar to: Note this will occur even if the etcd version is unchanged. The +` character. This pattern is used to support custom kubernetes images, for instance kube-apiserver:v1.20.4+patch.0 will be converted to imagetag v1.20.4_patch.0 when running make quick-release-images. Add the following to kubeadm- Install kubernetes. Then upgrade kubernetes with kubeadm upgrade apply. I have tested a fix for this issue and will PR shortly. Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: kubeadm Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig cluster-lifecycle
/area kubeadm
/triage accepted


In asked the following question. While the user-facing documentation answers this question clearly enough, John was reading the HELP string in the Prometheus scrapes. That thing only says the following. The metric HELP strings should be clear enough. Scrape or read the code (). No response <detailsThis problem started when we generalized to allow width other than 1 and got worse when we introduced borrowing. </details<detailsN/A </details<detailsN/A </details<details</details<details</details<details</details
/assign
Actually, this metric should be removed. It is a duplicate of but with an inferior name.
Looks good (Sorry I never got around to logging this issue myself. Have been super busy. And your wording is better anyway :-) )
/triage accepted Thank you.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I visited the Kubernetes 1.11 release blog post and clicked on the link to the article on resizing persistent volumes. The server returned a 404. What you expected to happen: I expected to navigate to the article on resizing persistent volumes in Kubernetes. How to reproduce it (as minimally and precisely as possible): Visit the release announcement blog post (), scroll down to the "4 Day Features Blog Series" section, and click on the link to the article on resizing persistent volumes. Anything else we need to know?: The link currently points to: The correct link is: Environment: Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A
Reiterating the mentions to trigger a notification: <detailsIn response to : <!-- This form is for bug reports and feature requests ONLY! --/kind bug /kind feature What happened: I visited the Kubernetes 1.11 release blog post and clicked on the link to the article on resizing persistent volumes. The server returned a 404. I expected to navigate to the article on resizing persistent volumes in Kubernetes. Visit the release announcement blog post (), scroll down to the "4 Day Features Blog Series" section, and click on the link to the article on resizing persistent volumes. The link currently points to: The correct link is: Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It should be reported to kubernetes/website, but doesn't matter.


is failing with: Examples:
This is the failing line of code...
PASS: FAIL: Seems like this disappeared: Also this is flipped: we run the following
my leading theory is this is something in
The issue is that the above PR changed a bunch of to be and thus we no longer wait for getArchive() to complete before starting tests. And unfortunately the previous PR cannot be reverted
After merging, kubelet-gce-e2e-ci still failed with the same issue. reopen the issue. Currently the failure blocked the merge-bot.
did stop the bleeding. Now kubelet-gce-e2e-ci is green: and merge-bot is unblocked.
I think the "go arc" piece was a red herring. I suspect the issue is that the same PR tries to copy the binary into the archive directory, but PR stops building ginkgo, so the copy fails. That is why this line then fails...


Intermittent failures on Travis.
Hrm, this passes 100/100 locally for me. I'll look at the code and see if I see anything. --brendan


We should drop the bit once in favor of a once gcp has a replacement. xref: /assign /milestone v1.24 /triage accepted /sig auth
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten /lifecycle frozen
I'm in the release 1.24 bug triage shadow. Does this issue still target release 1.24? If yes, would you please update the kind and priority fields for us to track it better?
Yes, definitely targeting 1.24 /priority important-soon /kind cleanup deprecation
would you please link the PR that solves this issue and update the status?
There is none yet. Waiting on GKE folks.
addresses this issue.


Should be able to tell from logs if a minion isn't present because cloud provider didn't tell us about it, or if it's failing health checks.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --On Windows 20H2 we are observing crashloops with fluent-bit due to a malformed Windows Event Log entry in the Security log. Fluent-bit to not crash loop. Bring up a GCE Windows cluster on Windows 20H2. You may see a loss of logs. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted


Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: It seems kubelet did not update inode pressure; and only test codes are using . So prefer to deprecate or remove it directly. (seems no node-misc group :). )


Problem Running the e2e density tests on a local cluster or any other cluster where ssh to master doesn't work fails, even though enough pods are created, because of SSH attempts. Specifically "should allow starting 30" is the argument to reproduce. Solution Allow density tests to pass even if some meta test operations fail after the fact. cc


In my CSI controller I have enabled the feature gate HonorPVReclaimPolicy for csi-provisioner. Based on the documentation Finalizers can be on a PersistentVolume to ensure that PersistentVolumes having Delete reclaim policy are deleted only after the backing storage are deleted. The newly introduced finalizers and external- are only to dynamically provisioned volumes. I can see the finalizer set on the PV while creating static Persistent Volume. PV deletion is stuck due to this finalizer apiVersion: v1 kind: PersistentVolume metadata: creationTimestamp: "2023-11-08T11:39:24Z" deletionGracePeriodSeconds: 0 deletionTimestamp: "2023-11-08T11:54:31Z" finalizers: external- name: csi-infoscale-pv PV should be deleted when PVC is deleted. Persistent Volume yaml apiVersion: v1 kind: PersistentVolume metadata: name: csi-infoscale-pv spec: storageClassName: csi-infoscale-sc-prod persistentVolumeReclaimPolicy: Delete capacity: storage: 2Gi accessModes: ReadWriteOnce csi: driver: org.veritas.infoscale # Please provide pre-provisioned Infoscale volume name. volumeHandle: clust21432/vrtskube_dg-/vol1 fsType: vxfs Persistent Volume Claim apiVersion: v1 kind: PersistentVolumeClaim metadata: name: csi-infoscale-pvc-static spec: accessModes: ReadWriteOnce resources: requests: storage: 2Gi storageClassName: csi-infoscale-sc-prod volumeName: csi-infoscale-pv Same thing works fine on OCP. WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.


Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-08-24T00:48:26Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v5.0.1 Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-08-24T00:42:11Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"} Vanilla K8s setup <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage maybe similar issues should go in
Opened similar issue in external-provisioner repo also There is no reply on that as well. Can you please help understand the behavior of the feature HonorPVReclaimPolicy.
If you want Persistent Volume (PV) to be deleted when the associated Persistent Volume Claim (PVC) is deleted, you donot need to use finalizers explicitly. Kubernetes handles this behavior by default. It is ok that if you use to get the desire output using Finalizer we can do it.
As per my understanding Please follow the steps : Add finalizer in metadata of as you did above need to make change in the controller which is handling PV deletion in .go file. See whether the controller administrator is working or not. I think it will work.
Finalizer is because the feature gate HonorPVReclaimPolicy for csi-provisioner is enabled. We are not adding the finalizer explicitly. From the documentation, "The newly introduced finalizers and external- are only to dynamically provisioned volumes." Not sure why is the finalizer getting set.


Also, for deleting PV it requires the annotation "" but this annotation is set in case of dynamic PVs. Should it be set explicitly for static PV?
This feature is alpha so the feature gate is not enabled by default. Did you enable the feature gate on kube-controller-manager?
Yes, i have enabled this feature.
external- is also applied to statically provisioned volumes. FYI
Hi Which version of the external provisioner is the driver using? Does the driver meet the requirements of the CSI spec on the DeleteVolume PRC? FYI: Is this problem occasional or permanently reproducible? Can you provide the provisioner logs at that time?
/triage needs-information
cc
cc
External provisioner used is : "csi-provisioner:v3.5.0" This problem is always reproducible Provisioner logs: I1123 12:22:38. 1 ] handleProtectionFinalizer Volume : &PersistentVolume{ObjectMeta:{csi-infoscale-pv -420a-4e9e-be82- 0 2023-11-23 12:21:18 +0000 UTC <nil<nilmap[] map[{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"csi-infoscale-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"2Gi"},"csi":{"driver":"org.veritas.infoscale","fsType":"vxfs","volumeHandle":"clust40559/vrtskubedg-/vol1"},"persistentVolumeReclaimPolicy":"Delete","storageClassName":"csi-infoscale-sc-prod"}} ] [] [ external-] [{kubectl-client-side-apply Update v1 2023-11-23 12:21:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"":{}}},"f:spec":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:csi":{".


":{},"f:driver":{},"f:fsType":{},"f:volumeHandle":{}},"f:persistentVolumeReclaimPolicy":{},"f:storageClassName":{},"f:volumeMode":{}}} } {csi-provisioner Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{"v:"external-"":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"":{}}},"f:spec":{"f:claimRef":{".


":{},"f:apiVersion":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:22:38 +0000 UTC FieldsV1 {"f:status":{"f:phase":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{ 0} {<nil} 2Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:org.veritas.infoscale,VolumeHandle:clust40559/vrtskubedg-/vol1,ReadOnly:false,FSType:vxfs,VolumeAttributes:map[string]string{},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:csi-infoscale-pvc-static,UID:-59a5-4f02-83ac-,APIVersion:v1,ResourceVersion:,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:csi-infoscale-sc-prod,MountOptions:[],VolumeMode:Filesystem,NodeAffinity:nil,},Status:PersistentVolumeStatus{Phase:Released,Message:,Reason:,},} I1123 12:22:38. 1 ] delete "csi-infoscale-pv": started I1123 12:22:38. 1 ] GRPC call: /csi.v1.Controller/DeleteVolume I1123 12:22:38. 1 ] GRPC request: {"volumeid":"clust40559/vrtskubedg-/vol1"} I1123 12:22:39.


1 ] GRPC response: {} I1123 12:22:39. 1 ] GRPC error: <nilI1123 12:22:39. 1 ] delete "csi-infoscale-pv": volume deleted I1123 12:22:39. 1 ] delete "csi-infoscale-pv": failed to remove finalizer for persistentvolume: Operation cannot be fulfilled on persistentvolumes "csi-infoscale-pv": the object has been modified; please apply your changes to the latest version and try again W1123 12:22:39. 1 ] Retrying syncing volume "csi-infoscale-pv", failure 0 E1123 12:22:39. 1 ] error syncing volume "csi-infoscale-pv": Operation cannot be fulfilled on persistentvolumes "csi-infoscale-pv": the object has been modified; please apply your changes to the latest version and try again I1123 12:22:39. 1 ] handleProtectionFinalizer Volume : &PersistentVolume{ObjectMeta:{csi-infoscale-pv -420a-4e9e-be82- 0 2023-11-23 12:21:18 +0000 UTC 2023-11-23 12:22:38 +0000 UTC 0xc0007f63d0 map[] map[{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"csi-infoscale-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"2Gi"},"csi":{"driver":"org.veritas.infoscale","fsType":"vxfs","volumeHandle":"clust40559/vrtskubedg-/vol1"},"persistentVolumeReclaimPolicy":"Delete","storageClassName":"csi-infoscale-sc-prod"}} ] [] [external-] [{kubectl-client-side-apply Update v1 2023-11-23 12:21:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"":{}}},"f:spec":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:csi":{".


":{},"f:apiVersion":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:22:38 +0000 UTC FieldsV1 {"f:status":{"f:message":{},"f:phase":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{ 0} {<nil} 2Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:org.veritas.infoscale,VolumeHandle:clust40559/vrtskubedg-/vol1,ReadOnly:false,FSType:vxfs,VolumeAttributes:map[string]string{},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:csi-infoscale-pvc-static,UID:-59a5-4f02-83ac-,APIVersion:v1,ResourceVersion:,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:csi-infoscale-sc-prod,MountOptions:[],VolumeMode:Filesystem,NodeAffinity:nil,},Status:PersistentVolumeStatus{Phase:Failed,Message:error getting deleter volume plugin for volume "csi-infoscale-pv": no deletable volume plugin matched,Reason:,},} I1123 12:22:40.


1 ] handleProtectionFinalizer Volume : &PersistentVolume{ObjectMeta:{csi-infoscale-pv -420a-4e9e-be82- 0 2023-11-23 12:21:18 +0000 UTC 2023-11-23 12:22:38 +0000 UTC 0xc0007f63d0 map[] map[{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"csi-infoscale-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"2Gi"},"csi":{"driver":"org.veritas.infoscale","fsType":"vxfs","volumeHandle":"clust40559/vrtskubedg-/vol1"},"persistentVolumeReclaimPolicy":"Delete","storageClassName":"csi-infoscale-sc-prod"}} ] [] [external-] [{kubectl-client-side-apply Update v1 2023-11-23 12:21:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"":{}}},"f:spec":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:csi":{".":{},"f:driver":{},"f:fsType":{},"f:volumeHandle":{}},"f:persistentVolumeReclaimPolicy":{},"f:storageClassName":{},"f:volumeMode":{}}} } {csi-provisioner Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{"v:"external-"":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"":{}}},"f:spec":{"f:claimRef":{".


":{},"f:apiVersion":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:22:38 +0000 UTC FieldsV1 {"f:status":{"f:message":{},"f:phase":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{ 0} {<nil} 2Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:org.veritas.infoscale,VolumeHandle:clust40559/vrtskubedg-/vol1,ReadOnly:false,FSType:vxfs,VolumeAttributes:map[string]string{},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:csi-infoscale-pvc-static,UID:-59a5-4f02-83ac-,APIVersion:v1,ResourceVersion:,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:csi-infoscale-sc-prod,MountOptions:[],VolumeMode:*Filesystem,NodeAffinity:nil,},Status:PersistentVolumeStatus{Phase:Failed,Message:error getting deleter volume plugin for volume "csi-infoscale-pv": no deletable volume plugin matched,Reason:,},}
Thank you I will try to fix it. /remove-triage needs-information /assign


<!-- Please only use this template for submitting enhancement requests --What would you like to be : and functions need unit tests Why is this needed: /kind test /sig cloud-provider /area provider/azure /assign
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What would you like to be : /sig cloud-provider /area provider/azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
related to
cc
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi I'd like to give this one a go, if no one is currently working on it.
sure, go ahead, thanks
I pushed some first tests for


This was the root cause of rancher/rancher. In , when is called, it is given the oldVersion of an object and the newVersion of the object. It then uses the indexers to calculate the old values (from the old version) and the new updated values. In cases where are size of 1, are of size greater than 1, and contain the sole value from , the key will be deleted for the old value will be deleted and won't be re-. This is due to the two for loops at the end of the method which are used for optimization. When the first for loop is processed, . Because of this, the key is deleted from the index for this value. In the second for loop, the , and for at least one of the values . The continue statement causes the key to not be to the index for this value (and since the value was deleted in the prior for loop, it no longer exists in the index for this value), despite the fact that the indicated that this is the desired state. Since specified that this value was still valid for this object, the key should not be deleted from the index for this value, or it should be re- when it is re-discovered in . Either way, the end state of the index should include this key in the set for the specified value. I've created a which demonstrates the issue (branch is ). To use it (assuming that you have a running kubernetes cluster to use with): the controller: . the foo crd: the controller (assumes that you have $KUBECONFIG set to the file path of your kubeconfig file). an example foo with only one label: . Observe, in the logs of the running sample-controller, that the indexer was able to locate this foo using the indexer. the foo to include a second label: . Observe, in the logs of the running sample-controller, that the indexer was not able to locate the foo using the first key any longer, despite the fact that the label was still on the object. The sample-controller fork adds an indexer for foo objects which uses the indexerFunction of . This function looks for labels that have a key with a prefix of and uses their values as index keys for the foo objects. For example, if I have a with a label of , the indexer will allow you to retrieve using .


The controller's method has been altered to just ensure that the foos can be retrieved using the indexer for all valid labels. If the controller can't retrieve foo by using the value from a valid label, then it returns an error (it tries for all valid labels). This was originally discovered in a fork of client-go that we maintain. After we reverted the changes introduced by , our issues stopped occurring. N/A - This was introduced in and appears to be in all versions of the client v0.23+. N/A - issue is with client-go No response No response No response No response
/sig api-machinery
I think that you have background in this issue/area of the code. Would you mind taking a look at this issue and the PR that I have up to fix it? Please let me know if I've misunderstood anything.
/assign Would you mind to take a look? Thank you /triage accepted
Thanks for reporting the issue. will address this issue
Reopening - since we need to cherrypick back to 1.23
Cherrypick is merged - closing.


What happened: node-local-dns caches negative queries, causing things that do polling for orchestration to fail (like operators with StatefulSets). What you expected to happen: Have a minimal TTL for negative responses (alternatively just disable caching by setting the cache size to 0) that follows what was done for kube-dns: How to reproduce it (as minimally and precisely as possible): With node-local-dns cache deployed, in my cluster it was the that failed to create clusters because the cached entry didn't expire before it had exhausted all of it's retries. Environment: Kubernetes version (use ): 1.11.6 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): any Kernel (e.g. ): any
/sig network


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --E2E test fails. The test is implemented at: I would expect the test pass with these objects in place: AFAIK this fulfills the general requirement of the test that there's a corresponding endpointslice object with same addressing information as is on the endpoint object. IMO the test should rather look for epslice which is owned by the endpoint rather than relying on a hardcoded name. Add flag to kube-apiserver Manually create the endpoint with proper addressing for the api-server epslice "mirroring" create new endpointslice with name like --The test fails as it's hardcoded to look for endpointslice named I'm working on a PR to provide a fix proposal Kubernetes version (use ): 1.22.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: k0s Network plugin and version (if this is a network-related bug): kube-router 1.3.1 Others:
/sig testing
/remove-sig testing /sig network EndpointSlices are owned by sig network.
/kind failing-test
/remove kind-bug /remove kind-failing-test The test is correct, apiserver must generates both endpoint and enpointslices with the special name "kubernetes" , that are the backends for the "kubernetes.default" service with the IP addresses of the apiservers. This special service "kubernetes.default" and its endpoints are used by clients with In-Cluster auto discovery. I think that whatever cluster that is not generating the endpoint and endpointslices for the "kubernetes.default" service should not meet Conformance, at least I can't see how automatic in-cluster configuration will work without that ... I only see option used for webhook tests, the other 2 reconcilers always create the endpoints and slices for "kubernetes.default" should we clarify the goal of this "none" option?
Yes, I do understand this. And in our case this is true, there is a svc called as there is also endpoints object called . This makes the in-cluster discovery work.


Otherwise we'd get much more errors in the conformance instead of this only one... :) In our case we're relying on the endpointslice mirroring to take care of creating the endpointslice object, which makes this test fail as it uses generated "suffix" on the object name. I have not dived too deep (yet) into the internals of endpointslices but I fail to see how the endpointslice object name really matters when both svc and endpoint object does have proper name.
oh, this seems interesting then ... who is creating the corresponding endpoint? can you paste a kubectl get endpoints -o yaml?
this is how it looks like in my cluster
In our case (k0s distro) it's our custom controller that is triggered when the controlplane is configured in HA mode. So this basically sets the api to mode . When our controller runs it has a configured LB address which it resolves to IP address(es) and creates the corresponding EP object. We do NOT create the endpointslice as we rely on the mirroring functionality to let it create and sync it. Here's the object details: Reading the code for API built-in endpoint reconcilers I see they set and create the endpointslice object by themselves. Yes, we could do that too in k0s but I fail to see why the name of the endpointslice object really matters at all.
I agree the conformance test should not be looking up endpointslice by name. It should be using a label selector of to locate one or more endpointslices and verifying the ports and endpoints across them.
Fair enough, on the good side this also will simplify the KEP from to dual-stack the apiserver you are the expert in this area, can you PTAL at this discussion? /kind failing-test
hmm the test description explicitly says the name of the object has to be "kubernetes", if the name is mandatory the test is correct, otherwise we need to rewrite the test description and talk with conformance people /area conformance
the description should be rewritten... the test is enforcing stricter requirements on endpointslices than required/expected
/triage accepted


Once e2e tests are running, the tests keep running even if I press and exit the process that controls the TTY. But the tests actually keep running in the background and they keep printing stuff on a shell prompt which has already returned and thereby clobbering the STDOUT
/assign
GitHub didn't allow me to assign the following users: oomichi. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Solution: I will add one.
/cc
xref
Thanks for sending a PR so quickly!
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code slush, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: If this issue is not resolved or labeled as by Wed, Nov 22 it will be moved out of the v1.9 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


During OPNFV test execution we noticed the folder selinux context changed. e2e test log How the folder changed after the test executed. I expected the TC won't change the selinux context. The question here, should this sigstorageserial TC set work on an selinux enabled K8S cluster or not? This question valid for the another sigstorage TC set with several test case, where selinux block a lot of TC too. See above. No response_ <details</details<detailsOpenStack </details<details</details<details</details<details</details<details</details
/sig testing
/sig storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/assign
Reproducer with simple kubernetes/kubernetes artifacts:
/triage accepted


Forking from To summarize, here's what we observed: daemonset calls coming every minute from both fluentd-scaler and addon-manager (verified this by individually turning them on/off). Things we need to understand here: Is the daemonset object continuously toggling b/w 2 states? (we know here that it's RV is increasing continuously) If yes, what field(s) in the object are changing? IIRC value of some label/annotation (I think 'UpdatedPodsScheduled') is changing (probably related to 2 below) Also, why/should the fluentd-scaler send any api request if the resources are already set to the right value? pods are getting deleted and recreated by daemonset-controller when the scaler is enabled (as was also seen in ). Why this is happening? One thing to note here - all those delete calls are preceded by calls from respective kubelets (but maybe that's expected). cc /priority critical-urgent /assign
Marked this as release-blocker as at least some part of it seems like a correctness issue (i.e fluentd pods dying and getting recreated). Note also that this is causing increased mem-usage with daemonset-controller (depicted here - )
Thanks for filing this! The only thing that was changing (apart from ds/fluentd-gcp status) was the resource version.
I modified fluentd scaler to use --dry-run in to check whether there would be a change, but this still keeps on updating, even though there is no change. Apparently, the logic inside kubectl is broken...
Btw, addon manager sends tons of PATCH requests for all resources every minute anyway, so even though I will fix fluent-scaler not to send them, it won't solve everything.
Should this issue actually be closed? It was auto-closed by the bot, but nobody has confirmed that the actual issue is fixed.
I'll let to confirm (looks like he'll be back on Monday)
/status in-progress
Yes, we should keep this issue open until we fully understand what's going wrong and fix the root-cause. I believe much of the discussion wrt this is happening under related issue
ACK. In progress ETA: 20/03/2018 Risks: May delay the release if not satisfactorily resolved
Need a number of successive green runs, then close.


So I took a look and seems like the fluentd-scaler version bump (which was supposed to fix) was merged b/w runs and of our . However, I don't see any visible improvement in controller-manager memory usage around that time: !
Also, I'm noticing such lines occasionally (once every 5-10 secs in the start and once every ~1min later on) from of latest run: And back when the fluentd-scaler wasn't doing anything useful (i.e before ), the had such lines only sparingly.
Now it isn't even sending requests (other than GET) to the apiserver. How do you generate that graph? Is it 99th percentile of memory usage over time? Or what exactly? One difference after introducing fluentd-gcp-scaler is that now there is a change occurring on the DaemonSet (once, but still). Also, what test is failing because of this? I'd like to take a look into the logs.
Answering your questions: I fetched the ResourceUsageSummaries for density test from historical runs (e.g ) and then obtained the 99% mem-usage for controller-manager for each run. Yes. More precisely, we're collecting mem-usage from controller-manager once every 1 min (or sth like that) and then taking 99th %ile of the resulting sample. Density and load tests. Where we're checking that mem usage of controller-manager is less than the given threshold (i.e 500MB). Ref: The tests are occasionally failing due to just crossing the threshold. Overall, the mem-usage had seen an increase back during the scaler change (see graph above). The job is this - https://k8s- Inside , you'll find all necessary logs (also a file named which has the resource-usage data).
The fluentd-scaler update eliminated fluentd restarts problem. Verified from controller-manager logs that we no longer see such logs that we were : This is great, and I believe that moves this out of release-blocker as we no longer face problems I mentioned in
That said, and me will continue looking into increase in mem-usage (which is not that serious). We'll start by verifying that turning the scaler off improves mem-usage.
Feel free to override me if you have a good reason :)
I agree that this doesn't sounds to be release-blocker.


But I would like to see follow-ups at least for issues like: does this memory increase come from? what was claimed to be "no-op" changes for daemon set were causing problems [is that issues in kubectl, apiserver, somewhere else?] Something like that may hit us again in the (near?) future if we won't understand it now. The current solution is really volatile [if there is a bug somewhere we shouldn't workaround it on our side, but instead have this fixed] at the cluster start (maybe it's fine to leave as is, but I would like to see a discussion of that and a conscious decision being made here)
Definitely agree those are questions we need to answer.
Yup. 1) is tested in . 2) I believe apiserver sends watch events even when only resource version changes, which causes the daemonset controller to recreate the pods. So either: resource version shouldn't change when there is no change anywhere else in the resource apiserver shouldn't send watch events when only resource version is changing daemonset should check whether watch event it got is actually changing anything 3) Isn't that an inherent part of the way addon manager works? It creates all sorts of resources, some dependent on the other (e.g. cluster roles and cluster role bindings), all at once. If some dependency is not satisfied, it will get fixed during the next run.
This is the code that is supposed to protect us from situation that something must change when we write object to etcd (resource version is not stored in etcd): So there should never be a situation that only resource version is changing. If it is, there is bug and need to be fixed. - FYI Reconciliation is a different thing that creating something in one state only to change that to a different state by a different component. I'm not saying this is for sure wrong approach. But I would like to see a conscious decision process discussing pros-and-cons of that.
That's what I'd expect as well. This I'm not sure if should be correct behaviour as IIUC watchers need to receive every RV update for correctness reasons ( ) Maybe. But are we sure if RV is just increasing without any object change? For creation of addons, yes (that race will likely last only for starting few seconds).