/kind bug Using , if a block device is symlinked to the discovery directory e.g. , then MapVolume will fail with since the containerized kubelet cannot see the path on host. is mounted by containerized kubelet, so if before we try to map the device we set devicePath to the evaluation of symlink and end up with a devicePath of , containerized kubelet will see the device and successfully map the device. Are there any issues with this solution? This is on Kubernetes 1.10
/sig storage
ah yes containerized kubelet has many issues when dealing with symlinks. Check out Mounter.GetMode(), it calls a nsenter EvalSymlinks which should be able to do what you need. But actually, one thing to consider, is do you need to do the losetup on the host, or does inside the container work?
Yes it appears to work from inside the container, on the host I run which disappears after I delete the pod using the device. My understanding of running losetup in MapVolume is not that great but I suppose that as long as that /dev/loop0 is created we are fine. Thanks for the quick response, I'll create a PR using that function and tag you
Does it create it on the host or in the containerized kubelet? What happens when containerized kubelet is restarted? The purpose of losetup is to do "refcounting" on the block device to prevent possible naming collisions when devices are attached/detached under unhealthy conditions. Ie, if a device showed up as /dev/sdb and you gave that to a pod, and then for some reason it become unhealthy and disappeared (while the pod is still running), another device (maybe even another volume type) could get attached to the node and reuse /dev/sdb. Now the pod has access to a device it's not supposed to. With filesystem volumes, the refcounting is done with mount points, but for block device, loopback is the equivalent.
It is created on the host (and also visible in the container), and stays around through restart. I understand now, thank you.
Actually after I restart it becomes /vdb instead of /dev/vdb in the losetup output, idk what that is about.


There is no such thing as /vdb and /dev/vdb is still intact of course. Seems inconsequential since it is disappearing after all pods using the device are deleted, i.e. it's still serving as a refcounter


ci-kubernetes-e2e-gce-scale-correctness Kubernetes e2e suite: [sig-node] SSH should SSH to all nodes and run commands kubetest.Test Jul 10 14:01:51 UTC+0200 Kubernetes e2e suite: [sig-node] SSH should SSH to all nodes and run commands: more . Test /sig node
/cc
/triage accepted
Uhhh, this may help here
/milestone v1.22
/sig scalability FYI
I got annoyed that I couldn't click through to see the other logs, so now you can, ref: (spyglass doesn't render build- because it's 500MB, and the limit for rendering is 100MB)
/assign SIG Scalability chairs/TLs, please find someone to address or call this a non-release-blocking failure
Assigning to current scalability oncall (): /assign
This is sig-node issue - should be addres by SIG-node folks. /sig node
is being discussed as a fix for this and I'm not sure it's correct
is it sig scalability's intent that 5k node clusters should support SSH?
The SSH test in question passes in the "default" job: So the fact that this failure is unique to a scalability job means to me that scalability needs to own at least some part of this
I did a little bit of digging and ci-kubernetes-e2e-gce-scale-correctness uses option in kubetest, which according to seems to be setting KUBESSHBASTION env. So I believe this is main difference between our scalability job and "default".
Debugging the test in and providing a fix there. The crictl host PR () is doing something different.


What happened: When the duration is exactly 2 years or 3 years, human readable duration is returned as or . But when the duration is exactly 4 years, 5 years, etc, the human readable duration is returned as , , etc. What you expected to happen: The format of an exact year should be consistent. Remove when duration is an exact year: should be should be Check other years... 4, 5, 6, 7... not sure if they also have the problem. How to reproduce it (as minimally and precisely as possible): Because this is time-dependent, it can be difficult to reproduce via the CLI, but you can see the problem illustrated in which shows the current behavior: The and test cases should be changed to want and , and the code should be changed to make that pass. Anything else we need to know?: Note, this is not a problem when the duration is exactly 1 year because the human readable duration is still shown in days. So nothing needs to change to handle the case of exactly 1 year. Environment: Kubernetes version (use ): 1.18 Cloud provider or hardware configuration: n/a OS (e.g: ): n/a Kernel (e.g. ): n/a Install tools: n/a Network plugin and version (if this is a network-related bug): n/a Others: n/a
/sig api-machinery
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Could I pick this up as my first issue?
Absolutely. Just use to assign it to yourself.
Also I just updated the issue description. Check the other exact years 4, 5, 6, 7 because they might also have the problem, I can't remember. I know 8 and beyond don't have the problem because it just stops showing days after 8 years.
/assign bitscuit
might recall a related PR?


(also, I have to leave the pedantic note: a year isn't always 365 days, so it's surprising to me to see the abbreviation exists at all)
I can't find it. Maybe I was remembering
Yeah it was probably that one. At first I had the tests and the fix. But it sat for a while. I changed it to just the tests of existing behavior so it could be approved more easily. Then I made this issue for the fix but decided it would probably be a good first time issue for someone instead of me doing it.
so should leap years be considered by the code?
There's no way to know how long a year is without knowing the date, so it's not a good unit--in the abstract, there's no way to know if it's a leap year or not. The go language deliberately omits it IIRC. Henry Li <:
What's the intention of HumanDuration? From what I've seen, it is just an approximation of a timestamp solely for humans to understand, so assuming 365 days per year sounds about right to me. This issue is about consistency though, so it could be solved the other way. Instead of abbreviating when it's an exact year (2y, 3y, etc.), exact years could just all have 0d (2y0d, 3y0d, etc.)
I think it is just used for a ballpark duration. It loses accuracy as the duration increases anyway because it changes the units. I guess my intention for opening the issue was to make it consistent with itself. For example it doesn't say 2d0h or 2h0m, so it should not say 2y0d.
(FTR I agree that Xy0d looks worse than Xy.) Brian Pursley <:
Hi I have made the required changes. Shall I raise the PR if you haven't started working in it ?
already assigned it, so I think it would be good to let him submit the PR, unless he unassigns or does not submit a PR in a reasonable amount of time.
Thanks I was planning on submitting the PR over the weekend
Hello, I am a first timer in open source contribution community. I think I made the required changes for that issue. However, I don't know how assignment rules work.


/kind bug What happened: Wrote a that returned a faulty json patch which specified an our of range index (e.G. when there is only one container). This resulted in the apiserver crashing. Stacktrace: What you expected to happen: No crash /sig-api-machinery
/sig api-machinery
This maybe a bug in . I will follow this question.
This bug had been fixed in PR: you can checkout it.
Thanks for your comment but I don't think the linked PR fixed the issue, as it got merged almost a year ago and I am running Kube 1.10.1 which does contain that patch but still panics. Just from quickly looking at the code I the linked PR I believe there are two things wrong: The check if the index is valid only happens when a negative index was given should be
Once the PR of is merged I will update kubernetes.
/kind bug


release-master-informing: capz-windows-containerd-masterKubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] Kubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] Both the tests have been failing during most of the runs since 20th October Kubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] | 1m6s -- | -- {Oct 31 18:42:02.425: Timed out after 30.001s. Expected <errors.errorString | 0xc0015283b0: { s: "failed to match regexp "GET /echo?msg=prestop" in output "I1031 18:41:06. 8988 ] Started HTTP server on port 9090nI1031 18:41:06. 8988 ] Started UDP server on port 9091n"", } to be nil failed (0xc000f1d000) +0x469 () +0x335} Kubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] | 5m11s -- | -- {Oct 31 19:05:50.205: timed out while waiting for pod container-lifecycle-hook-6892/pod-with-poststart-https-hook to be running and ready failed (PodClient).CreateSync(0xc000eb5920, 0x22?) +0x94 (0xc0045e8c00) +0x73 () +0x325} No response_ /sig windows cc
/triage accepted /cc
These failures happen also with Kubernetes master + containerd master with Flannel CNI. Testgrid link:
I figured out why these new failures appear.


I've submitted the fix via PR
This issue is also affecting 1.25 tests: If possible, we should cherry-pick to the release-1.25 branch once it's merged on master.


opencontainers/runc has fixed three CVEs: Latest runc is pulled by k/k Check runc version in file No response n/a n/a <details</details<details</details<details</details<details</details
/assign /sig security architecture release
/triage accepted


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? controller /etc/ssl/certs kubeadm /etc/ssl/certs Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 5 custom built master I also use a custom built hyperkube image instead of the single app images for apiserver, controller, ... Environment: Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): CentOS 7.2 Kernel (e.g. ): Linux ma-kub8ms0 3.10.0-327.36.3.el7.x8664 SMP Mon Oct 24 16:09:20 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: deployed with Azure RM template and ansible What happened: controller is not able to do https, including calls to the Azure RM API What you expected to happen: https should work as usual How to reproduce it (as minimally and precisely as possible): Use kubeadm to install kubernetes on CentOS exec into the controller and try "curl " Anything else do we need to know: It looks like caused this issue. On CentOS, I see these files on the host OS: I suspect the symlinks to /etc/pki/... being the problem here, as they are not available inside the hyperkube Debian image. I also tried to verify this with It seems to not be based on Debian and thus gives a different error: I see 2 possible solutions: of host mounting /etc/ssl/certs into the controller pod, the ca certificates could be installed into the image.


This is already done for hyperkube and would work out of the box when the host mount would be omitted. could try to figure out if /etc/pki should also be host mounted and add this to the controller manifest if required. I'd be happy to provide a fix by myself, I just need to know which solution you would prefer.
Same problem seems to apply to the apiserver. Not sure however if it actually does https calls.
implements solution 2.
This issue was moved to kubernetes/kubeadm


Right now, we have a mixture of user documentation and developer documentation in the top-level directory. We should make it easier to find a just user documentation. Step 1 would be to move developer documentation into a subdir like docs/development or something like that. User docs would stay in the top-level docs dir.
move to


When a --resolv-conf (DNS resolver config) is passed to kubelet on a Windows node, the config content is used as part of sandbox initialization (through getHostDNSConfig()). Now with the recent changes (), it seems now to fail with an error like: E0315 10:22:16. 4764 ] "Error syncing pod, skipping" err="failed to "CreatePodSandbox" for "img-pull-(-b40a-459d-9033-)" with CreatePodSandboxError: "Failed to generate sandbox config for pod "img-pull-(-b40a-459d-9033-)": Unexpected resolver config value: "C:etckubernetescni". Expected "" or "Host"."" pod="img-puller-3842/img-pull-" podUID=-b40a-459d-9033- This is showing on all the runs of GCE testgrid (latest): Reading/using the passed resolv config on Windows Run a Windows pool with DNS resolv config being set as part of kubelet params. No response_ 27 alpha.3 1.27 beta.0 master <detailsGCE </details<details</details<details</details<details</details<details</details
/triage accepted /milestone v1.27 /sig node /sig network
/sig windows /cc
/priority criticial-urgent as this is likely a release blocking
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority criticial-urgent Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
Hmm, so you're setting the [1]. This PR introduced the capability to read the DNS config options from the Windows host registries while removing the capability to read from a file, introducing a regression. I've sent a PR addressing this issue: [1]
Is the file you are pointing to a UNIX-style or something else?
Was the file changing the behavior of the node?


Yeah, what used on GCE/GKE, seems to be a Linux-like conf, eg: nameserver 10.128.0.1 search c.<project_name.internal google.internal ` We haven't tried to remove from kubelet to see the behavior.
This Issue/PR has been marked as release blocker. Please have the fix merged by eod today otherwise the feature has to be reverted. For the Issue/PR owner: please have a revert/feature disable PR open today in case the fix failed to get in. Thank you! See the -- Kubernetes 1.27 release manager
I opened up a revert PR in case this PR does not merge by code freeze.
Thanks for the resolution! both 2019 and 2022 grids seems to be back to green now!
Thanks for confirming! I opened to ensure gets updated too


When setting for other container runtime, won't start kubelet correctly because cri is disabled : Is there any reason of disabling cri at ? cc
no, that wasn't supposed to go in. I was testing something and it slipped in accidentally. I'll open a PR to fix.
Ah I see your PR for it. Thanks!


New tests for subpath consistently fail in ci-kubernetes-e2e-gce-multizone: https://k8s- /kind bug /sig storage
Is this a blocker for 1.10?
I would expect so, yes. /milestone v1.10
I'm on it /assign
The problem is likely due to inline GCE PD volumes not having enough zone information. Considering changing volumes to PVs instead.
fix was in progress in to change to a PV
/unassign
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


From etcd3: May need to retry poststarthook a few times to avoid bootstrapping failures like this.


What would you like to be : Currently Mutex lock is acquired in for accessing different maps. Some functions only read map entry: We can utilize RWMutex where the above function acquires read lock. Why is this needed: For concurrent calls to IsFitPredicateRegistered / IsPriorityFunctionRegistered, read lock would lower contention.
/sig scheduling
functions getFitPredicateFunctions, getPriorityFunctionConfigs, ListRegisteredFitPredicates, ListRegisteredPriorityFunctions and GetAlgorithmProvider can also use read lock.
I happened to raise earlier today. Anyone can finish resting ones.
Good to know,
Thanks for raising this ! My PR covered and . Feel free to raise another PR to cover others.
Once is merged, I will send my PR thanks
it's merged now.


The namespace seems to be hard coded :
cc:
Thanks. We can improve it after v1 release.


is a library yet is is instead of release tag. Please consider vendoring by tag whenever possible. Thanks.
We vendored it that way because the commit that fixed a bug (which we needed to be fixed and vendored back in) was This was merged on June 13. The release 1.1.4 () was released on July 1. If we want to update it to 1.1.4, I believe it should be fine. Between the commit we have vendored right now and the latest release, there are only three commits: adds tests merges (?) the case-sensitive fix - I'm not really sure what this commit is doing. fixes renaming when naming strategy is set (looks fine). Also, there are also no licensing changes. opinions?
/sig api-machinery
+1 to what said, it was a fire drill and 1.11.0 release was in trouble. Though in hind sight, we should have opened a issue to track that. something better to do next time. cc
I'm indifferent to updating this. I'm okay with moving up if it doesn't break anything, or I'm fine with leaving things as is. We don't require deps to match semver. We can version out any commit in a deps tree, as we rely on our vendored copy.
It is always a good idea to match semver and it should be a default policy. Even if it doesn't matter to you, it should, because it is a best practice. Moreover downstream it matters much more as dependencies may be un-vendored in favour of tested re-usable packages.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Someone please take care of this or bot will eventually close this valid issue without action...
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


/lifecycle rotten
/lifecycle rotten
/remove-lifecycle rotten PR open (thanks,
oops! sorry, I meant to remove it. thanks!
Thanks.
please do feel free to open issues/prs of this sort early in the release cycle, so we can work through things needed well before the milestones/freeze etc. we are getting some heat about updating dependencies and we kind of settled on some guidelines here: if you think that needs updating, we could definitely debate that as well :) (on a PR against that doc)
it have nothing to do with release cycle or justification for upgrade if/when dependency has been vendored incorrectly. It is basically a vendoring bug. No debate is necessary although that kind of issues usually meet some resistance and even hostility here... Bugs are filed when they are discovered, regardless of release cycle. It would be wrong to sit on the bug without reporting it in order to synchronise with release cycle.


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority important-soon /sig network
Ref:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
my fu points me to this commit [] Fix multiple conversion tests that belongs to this PR , that seems related can you confirm if that PR is causing this test to fail? (should we gate on this test?)
Yes, that PR is related to the failure. The test utility is trying to convert extensions/v1beta1 data read from various files directly to a go type. This was never a valid thing to do, and happened to silently succeed because the two structs had an identical structure. This is exactly the type of bug was designed to flush out
the correct fix is to update the test/e2e/testing- files to
/assign
It was my PR that broke this - I can also fix that.
Thanks, o it was the commit that was good but the test that was wrong :upsidedownface: , At least I got right the commit :sweat_smile:
is out for review (talked with offline, he was aware that I'll send that out)


pointed out that for scheduling and kubelet admission, we should consider resources of terminated-but-not-yet-deleted pods to still be in use, because it may take an arbitrary amount of time for the system to deallocate resources. We didn't check to see if this is currently how it is implemented or not. cc/ /sig scheduling
cc/
For terminated-but-not-yet-deleted & phase is case, the Pod is still in scheduler's cache, so it's handled well in scheduler; the Pod will be removed from cache when it's deleted. But in kubelet, it filtered terminated-but-not-yet-deleted out () when using scheduler.predicates. The option in my mind is to ~pass all Pods (including terminated-but-not-yet-deleted) to each admin, and let each admin decide whether filter terminated-but-not-yet-deleted out~ only filter out finished (Failed or Succeed) pods for Admits.
In -, it already consider by . So both scheduler & kubelet handles terminated-but-not-yet-deleted pods well.
Great, thanks for looking into it!
should this issue be closed then?
I think so.
Just to clarify: Kubelet correctly accounts the resources used by terminated-but-not-yet-deleted pods, i.e, kubelet does not count the resource usage of those pods towards total usage on the node. Scheduler, however, does not do the same. Scheduler waits for a pod to be deleted, and then it deducts its resource request from the associated node total usage. Upon deletion of a pod, scheduler is notified and acts: So, scheduler and kubelet behaviors are not exactly the same. I am not sure if scheduler can receive an event when a pod's container status changes. If it could, we could make the logic the same as that of Kubelet.
I guess the current behavior is OK because it is conservative (scheduler considers the resources to be in use even though kubelet considers them free). And actually it seems that scheduler is more "correct" than kubelet, since the resources really aren't usable until the pod is deleted, right? (IIUC that was Vish's original point)
I agree. The current behavior is correct, but a bit conservative. Basically, the scheduler side is more conservative and waits until a pod is deleted from the cache before considering its resources released.


That said, I believe resources of terminated-but-not-deleted pods are usable on nodes (should be verified). So, the current behavior is a bit less than ideal in terms of efficiency, but AFAIK the duration between termination and deletion is only a few seconds. So, it may not be worth optimizing.
I thought was saying the opposite (see my first entry in this issue). Maybe he can clarify.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: controller-manager is stuck in deleting an incorrectly configured LB type service. it cannot be recovered until restarted. What you expected to happen: controller-manager should recover. At least it should allow us to create the service without restarting it. How to reproduce it (as minimally and precisely as possible): put a random IP in above yaml and make sure the resource group doen't exist. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.4 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig azure
I also want to add that the above repro is stuck in 403. It's possible for it to be stuck in 404 as well when the SP has the right permission.
Can you share the log for 404?
Thanks for reporting the issue. /assign Do you mean other services couldn't be configured when this happens?
correct.
FYI, we had the same issue when the Idle Timeout annotation was set to "3" which isn't a supported case. Will this PR also cover this edge case ?
Unfortunately , this is still not fixed. The idle timeout is not 403 or 404. We need another fix for this (and possibly more for other annotations)


We had rather strange case today when after rolling update one of the kubelet's started wrong container. For this specific replicationcontroller we are using What happened was that after rolling-update when i started to debug the issue i discovered that one of the kubelet's did not pull the image: When looking into what image it actually started i saw: When checking from the actual node i saw this It seems that instead of starting myimage with tag 2208 kubelet started image tagged as 1393 since the docker image id starts with 2208.


TestRunOneVolumeDetachFailNodeWithReadWriteOnce 4 times since 03-19 23:00 PDT, intermittently ! https://k8s- Still investigating whether this is a tolerable flake or not with more history than testgrid view, or otherwise would appreciate input from anyone with expertise in this test, especially as regards to whether this should be a blocker for tomorrows beta cut. maybe: /sig storage /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is already open for this, looks like it is attempting to fix it
thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. /close
Closing this issue. <detailsIn response to : thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


pull-kubernetes-unit TestRunOneVolumeDetachFailNodeWithReadWriteOnce Jan 7th, 2022 Check volume <fake-plugin/volume-nameis reported as attached to node <fail-detach-node, got false, expected true No response_ /sig storage
/cc
this is the top unit test flake according to
cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
I think this test is no longer flaky by checking the test grid, close it now.
Unfortunately this does still seem to be flaky: Here was the output from : /reopen
Reopened this issue. <detailsIn response to : Unfortunately this does still seem to be flaky: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
still the case
How about adding some logging to help troubleshoot this? I looked at it a little, but it is hard to tell which part of the test is failing. PR
/
This is still happening
/triage accepted
From the logs, in the OneVolumeDetachFailNodeWithReadWriteOnce test, succeeded, but the next verification failed. In verifyVolumeAttachedToNode, the attachState is from calling and within asw, it calls . In verifyVolumeReportedAsAttachedToNode, the volumes are obtained from calling , it seems that the node is not in asw.nodesToUpdateStatusFor at this step. Cannot reproduce locally. I tried to run the test 100 times locally with debug.Stack()... 3 * 100 times all passed.
Tried to run stress test on my linux machine as well but still cannot reproduce. Command: Result: Also tried running n process in parallel. Command: Result:
It seems like it happens more frequently.


But I may have no time to dig into the cause. Let's see.
cc as the test author.
The problem is that this test has a lot of dependency on the sleeps that are hardcoded with a very narrow scope, but our CI seems to be able to stress it much more. As soon as you slightly change one of the sleep duration it starts to flake locally The test can not make an assumption that timing is perfect, even more in the sub second range can someone in sig-storage review this test and make it less dependant on those tight sleeps?
According to most failures, it failed at Line 660. Let me run some stress locally. with 250ms sleep with 100ms sleep
I agree this test case is too brittle due to its specific timing dependencies. To fix this, I suggest reworking the test case to: 1) Add retry with timeout to wait for final conditions 2) Remove the intermediate timing-based state transition checks. Instead, we can keep track of state transitions in the Fakes and validate those stats. wdyt?
Friendly ping from 1.27 CI Signal that this test is still a noisy flake on the master-blocking dashboard as CI Signal has flagged it in and


introduced somethng that really should be e2e tested.
Would be nice to get this in in the 2 weeks we have after 1.2 branch cut, if no more flakes crop up (but I'm not too optimistic). Bumping to next-release.
/sig network I'm assuming sig-network owns kube-proxy
I thought I noted a change related to this recently, but I may be mistaken.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/lifecycle frozen /remove-lifecycle stale
/remove-lifecycle rotten
Going to try to take this on.
thanks!
I've had some trouble constructing a test for this. should I determine the timeout? There doesn't appear to be an existing mechanism for waiting for service creation. seem to time out rather than be refused - I'm confused by this, as it seems to be the opposite of the intended behavior.
can look at the endpoints + wait for longer than kube-proxy sync time. refused means packets got to their destination but there was nothing listening there. If traffic is blocked, then you will just get a timeout.
I created a service with no matching pods to test manually outside of the e2e suite. # curl test-api curl: (7) Failed to connect to athena-api port 80: Connection timed out # curl --connect-timeout 8 test-api curl: (28) Connection timed out after 8000 milliseconds Looking back at original example in I see that I'm getting the "before" behaviour. An iptables REJECT should send a denial, not a timeout. Am I looking at the wrong thing here? To me this looks like a regression, but I don't think that's likely.


I have a cluster from HEAD as of 2 weeks ago: So indeed it seems to not work. If I hit that same service IP from the VM itself I immediately get curl: (7) Failed to connect to 10.0.199.54 port 80: Connection refused. If I insert an iptables jump to KUBE-SERVICES at the front of FORWARD, it correctly rejects. This seems in accordance with the iptables flowchart. I'll do some spelunking to see if there's a reason NOT to reject in this case. Vallery Lancey <:
Can you check IPVS mode?
This is why we need tests. Sigh.
Proposed fix is - still needs your e2e :)
Thanks Tim, if I have time I’ll get a PR up tonight.
Sure will check IPVS mode.
Base on my test, IPVS has no such issue.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --/kind bug /sig cloud-provider /area provider/azure Azure in-tree CCM fails to delete a load balancer: The corresponding logs: The clouprovider config is: The Nodes are actually Azure VMs that belong to a VMO (aka vmss with flexible orchestration mode) - regular VMs and managed independently of the vmss. So those VMs are managed on Azure as . LoadBalancer deletion works with the very same setup with v1.21.1 of the in-tree Azure CCM. After recent cherry-picks, with v1.21.4 we observe the failure above. a cluster using VMs that belong to a VMO a Service of type LoadBalancer the Service of type LoadBalancer sure that the deletion fails with: Kubernetes version (use ): v1.21.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Gardener Network plugin and version (if this is a network-related bug): Others:
/assign /triage accepted


The recently merged PR a version cache in dockertools, which makes the test  flaky. The reason is that injected error into fakedockerclient and then expected to see the error from right after that (See ). However, the error was consumed by the goroutine which periodically updates the version cache (See ). The docker daemon version is rarely updated, and we only need version once for each container creating, during which we also do container create, start, inspect #etc. 1) Do we care about the cost of the call so much that we have to introduce the complexity? 2) Kubelet periodically calls , can we rely on that to update the cache? Anyhow, at least, we should disable the cache or have a fake one in the unit test. :) /cc
I once considered move this cache to runtime level, while thus may need to change runtime interface. What if only cache apiversion?
We can only get via , so there will still be the data race. :) One option is to update the version cache in , now kubelet periodically call to check the runtime status. Another better option is to cache the version in layer, now is under our control, we can refactor it with an additional argument indicating whether to use cached value or get directly from the daemon. If we cache the version only for performance, I prefer the second option which is cleaner and more efficient. :)
OK, will look into this tonight
The unit test itself should not start a goroutine to update the cache (unless you're actually testing the goroutine). We should disable it anyway, as we often do in our unit tests. It also sounds like we don't need a goroutine dedicated for this. If we can tolerate the additional overhead from time to time, we can set a expiration time (e.g., 1 minute), and only call the runtime if the entry expires.
Totally agree. :) Maybe a new function in wrapping the update logic, or wrapping the logic in the .
expiration time sounds good in this case
This is happening quite frequently - increasing priority to P0.


The compute project and zone in the prerequisites check are incorrectly calculated: !!! Error in hack/make-rules/test-e2e- Error in hack/make-rules/test-e2e- 'tee -i "${artifacts}/build-"' exited with status 1 0 Call stack: 1: hack/make-rules/test-e2e- main(...) Exiting with status 1 blocked by the following errors in each case: 1- Could not find gcloud compute/zone when running: .... 2- Could not find gcloud project when running: ..... 1- gcloud config unset compute/zone 2- gcloud config unset project 3- make test-e2e-node REMOTE=true LIST_IMAGES=true gcloud --version Google Cloud SDK 453.0.0 alpha 2023.10.27 beta 2023.10.27 bq 2.0.98 core 2023.10.27 gcloud-crc32c 1.0.0 <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /assign


Currently, it could not delete multiple pods at once, need to delete them one by one, it's very inconvenient if I have tons of pods to delete. Unlike docker, docker can multiple stop/rm containers/images at once.
This is possible using label selectors as of , which is better if you have tons of pods.
I agree that using label selector is better to delete all of pods under the label. But to delete multile pods at once is still usable: I just want to delete 1/2 pods under the label, label selector will not work selector could not span over other selectors or pods Another reason is docker rm/stop x y z works, I will wonder why kubectl cannot. This is not urgent, but I hope it can be implemented.
I agree. Thanks for the suggestion.
Would this be a suitable task for someone new to k8s development? If so, I'd be willing to take a look at this, if someone could provide a couple of hints where to start.
Further to my last message, I have a functional work in progress I would like to share, if anyone is interested?
PRs are welcome. See
Yes, this is a reasonable place to start. Since its in the client, you won't get much orientation to the core server code, but it will be a good place to orient to the client side API. There us also prior art that you can model off of. The kubectl create ... command already supports multiple resources. A good command to adapter would be the 'label' command. You can find it in Many thanks in advance! Brendan On Feb 19, 2015 8:21 AM, "Eric Tune" wrote:


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-new-master-upgrade-cluster-new-parallel Which test(s) are failing: Varies, but all subpath failures, including: [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath with backstepping is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] ... and pretty much every other subpath test, but never all of them at once. There's also a few other storage tests failing, such as: [sig-storage] Volume expand [Slow] Verify if editing PVC allows resize [sig-storage] Detaching volumes should not work when mount is in progress Since when has it been failing: 11/22 Testgrid link: https://k8s- Reason for failure: These flakes started around the time that merged, but doesn't match up with the exact merge stamp, so it's probably coincidental. The subpath test failures seem to be mostly timeouts: ... so possibly this is just GCE fail. Anything else we need to know: This test job has always been flaky, with around a 40% failure rate. <!-- DO NOT EDIT BELOW THIS LINE --/kind flake /sig storage /priority important-soon cc
can you please help investigate this. It looks to me like the tests are flaking for and . I wonder if the v0 tests are racing with the non-v0 tests for something in this test?
The other tests including for the other CSI driver and non-CSI in-tree plugins are solid green which also makes me think this is a issue with the hostpath tests or driver.


(just adding additional similar flakes here for tracking) Similar flakes in :
adding to the milestone for triage since the flakes look new
These all 4 subpath tests call and it fails in finding a specific failed event by using . As uses to check if the specific error event happens, and , I guess that tests may flake if other failed event happens before the expected one. Also, already wait for the event, we might be able to delete  in . I will create a PR to fix above.
/priority critical-urgent /remove-priority important-soon /milestone v1.13
while we should update the test with your PR, I'm not sure if it's going to fix the failure. Looking at one failing test log, these are the events I see on the failing pod: So it seems to be failing before even reaching the subpath logic.
Timeline of events: Test pod started getting processed by kubelet: It took a minute after the test pod was created for the csi driver to finish registering and mount to succeed, and then subpath event to trigger: So should address the issue. I was initially confused because the test logs didn't show the subpath event for some reason
I opened up to investigate if 1 minute for csi plugin registration is expected
I've a tracker issue for the complete list of storage flakes: This is so that we can track whether we're addressing most of the flakes with the subpath fixes, or if additional fixes are required.
keeping open until flakes are verified cleaned up
/reopen Reopening this issue until we can verify the fix in CI
As of this morning we are seeing CSI "subpath should fail" flakes in the following job Multiple "subpath should fail" flakes in this slow job alone Multiple "subpath should fail" flakes in the 1.13 slow job, similar to master. The following 2 tests are flaky only in this upgrade job can you help us determine if these are cases are flaky tests that you are fixing or is there an underlying CSI bug that needs addressing?
root cause of this is a delay in an external sidecar containers and the tests not waiting long enough to accommodate the delay. There are no changes required in (outside of test) to handle this. These should not be 1.13 blocking. We'll continue to look in to getting them non-flaky today.


In the meantime, we can mark these tests as flaky to reduce the noise, if you'd like. looks like the same issue () is still being hit. See : And logs from (): We need to make the tests handle this condition more gracefully. The suggestion here is one possible solution.
Would be good to capture this as a known issue in release notes.
to followup on capturing this in Release notes
per , can you create a targeted fix increasing the timeout? Looks like marking these tests as flaky will be difficult for 1.13, but if that doesn't fix it we will need to do that.
Short term fix to bump the test timeout:
As per discussion in release burndown meeting, this is not a 1.13 blocker. please move this issue to 1.14 if there are no pending issues you are concerned about
Ok looks like the test still flaked. Looking at , the mount succeeded after a minute and proceeded to subpath processing, which (correctly) failed. However, for some resaon, the test is not seeing the subpath event and still failed after 4 minutes. On kubelet, subpath processing failed after a minute and generated the event: But the test doesn't see it for some reason, and still times out after 4 more minutes:
I saw the same on CSI volume mount succeeded at: Kubelet correctly failed the pod because of subpath But test still failed: Because the test failed to find the pod event. This looks like a test issue. Not a 1.13 blocker.
can you PTAL at this
Is there a way to check if events are overloaded and not able to handle new events? The MountVolume retries while the driver is not registered is very frequent and could have overloaded the event server.
Checking for the existence of an event seems like the wrong way to test this. Would checking pod status alone be insufficient? Ultimately that is what we really care about.
Checking for the specific event guarantees that we are failing with the correct error. But perhaps events are not reliable enough to depend on. Checking if the Pod never came up running would probably reduce the flakes but could potentially mask false negatives. cc for any thoughts.


Tests are still flaking and actually it seems like in general storage flakes in slow are flaking more since 11-21 or so: https://k8s- My theory is that the driver registration delay is causing a flood of events, which causes tests that check for events to fail. I think this is revealing multiple issues in both the tests and product, and we should work on fixing them all, although any one of the fixes should mitigate the test flakes: Driver registrar can sometimes take a minute to come up: Volume manager operation executor doesn't seem to be throttling retries or backing off: Tests should not rely on events: Tests should wait for CSI driver to be registered on all nodes before creating test pods:
and have been merged and will address the biggest issues. has been backported to 1.13 already, backport is still pending.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Example: kubectl cluster-info shows a redundant line: The cluster is already up, so the phrase "Starting cluster" is confusing in context of the command. (My kubernetes test cluster is running on AWS i.e. KUBERNETES_PROVIDER=aws. I had used kube- to create a test kubernetes cluster on aws). Log of my shell session:
Thanks for the bug report I've filed a PR to fix it.


Is this a BUG REPORT or FEATURE REQUEST?: What happened: use kubectl search log with follow option get api latency (apiserverrequestlatenciessummary{quantile="0.99",resource="pods",verb="GET"} )high to give a alert in my cluster. the apiserver log is below and we can see it costs 1m25s. What you expected to happen: do not make this api latency account in getting pod request api latency How to reproduce it (as minimally and precisely as possible): use kubectl to search a pod log with follow option long enough and you will get this metric apiserverrequestlatenciessummary{quantile="0.99",resource="pods",verb="GET"} higher than the ordinary case. Anything else we need to know?: I am happy to fix it. Environment: Kubernetes version (use ): 1.6 Cloud provider or hardware configuration**: bare-metal OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): 4.4 Install tools: kubeadm Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig API Machinery
/area apiserver
/sig api-machinery
/assign Thanks... looking forward to your PR!
fixed with PTAL


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the pause-amd64:3.0 image on has an empty schema2 manifest, which prevents is from being pulled by proxy/caches which only support V2 registries. What you expected to happen: Being able to pull ` using a V2 manifest compatible client. Anything else we need to know?: The proxy/cahce is requesting this
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After talking with , he mentionned the following
AFAIK, does not support well. After inspecting the image, you can find is still using manifest v1.
/cc FYI
Im not sure which exact version x schéma X is needed. What I know is that we can download through the proxy/cache all other Kubernetes kube-system images, only the pause image is giving us trouble. Therefore, I would say that having the image available using the same format as the other kube-system images would be the request here.
Yeah, we haven't updated the pause image in a while, since it doesn't really ever change. I'm not sure whether rebuilding + pushing over the existing tag is safe, or if we should bump to or something. thoughts?
If we do rebuilding/updating the image, that would help and a bunch of similar issues.
it's better/save to go with new tag
I didn't realize there were changes to the pause binary that were never actually deployed. It seems like we should probably just do that (and bump the tag).
that sounds great. Any idea when the existing 3.0 tag could be pushed again?
We shouldn't push over the 3.0 tag; we should push a 3.1 or 4.0 tag instead. I'm not sure who owns the pause container who should drive this. Maybe sig-node and/or
It seems like it should be sig-node. I'll look into it.
The new container has been built, can you test with
Its working! thanks! I was able to pull it through our artifactory proxy , Lee Verberne <:
excellent, thanks for checking. /assign


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: pause does not reap zombie processes using the default pause container provided by What you expected to happen: should include the up to date pause code to reap zombie processes when using docker = 1.13.1 found here How to reproduce it (as minimally and precisely as possible): a.out will leave a zombie process that is not cleaned up Anything else we need to know?: Building the pause container from master solves this issue Related: Environment: Kubernetes version (use ): 1.7.0+ Cloud provider or hardware configuration**: N/A OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 4.10.13+ Install tools: Others: docker = 1.13.1
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
i get a simliar problem when i run crond service in pod container, every day it makes some [anacron] zombie progresses UID PID PPID C STIME TTY TIME CMD root 1 0 0 Nov14 ? 00:00:00 /pause root 7 0 0 Nov14 ? 00:00:00 /usr/local/bin/dumb-init /usr/lo root 13 7 0 Nov14 ? 00:00:00 nginx: master process /usr/local nobody 21 13 0 Nov14 ? 00:00:01 nginx: worker process nobody 22 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 23 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 24 13 0 Nov14 ? 00:00:05 nginx: worker process root 77 1 0 Nov14 ? 00:00:00 [anacron] root 1 0 00:01 ? 00:00:00 [anacron] <defunctroot 1 0 01:01 ? 00:00:00 [anacron] <defunctroot 1 0 02:01 ?


00:00:00 [anacron] <defunctroot 1 0 03:01 ? 00:00:00 [anacron] <defunctroot 0 0 11:49 ? 00:00:00 /bin/bash root 0 1 19:15 ? 00:00:00 /bin/bash root 0 19:15 ? 00:00:00 ps -ef !
Do we know why this was never resolved / was never deployed?
I just fought through this issue with a rabbitmq helm chart. I had to custom build a pause container from master to fix it. The rabbitmq helm chart uses for its liveness checks. It's a thin wrapper around a call to the rabbitmq-server daemon, which is not PID 1 (/pause is PID 1). The rabbitmq-server process forks some children to do the work and then they exit. Since the pause-amd64:3.0 is not reaping, those processes stick around as zombies. The liveness check runs every 5 seconds. After about 4 hours, there are about 27K zombies and the node begins failing to launch new containers because it's starved for resources. The pause container built from current master fixes that behavior, no zombies at all in the container.
The plan is to build a new version of pause-amd64 and then update the image referenced by Kubernetes. This means the default image will change in the next Kubernetes release. We can make the new image available once the freeze lifts and we can merge .
Looks like this made it into 1.10


Though I have successfully installed kubernetes via :smiley: , still want to point out this doc missing the step to install . Thus, if user just follow the guide in the doc without having much knowledge about kubernetes, he/she would get error "It looks as if you don't have a compiled kubectl binary" during the step of . Manually install kubectl on at least master node should be necessary (suggest to let user also install on minion node for convenience), any way, the installation step is .
I forget to mention that.I will update the doc this tuned
I update the scripts in pr , you can try it now. I copied the kubectl binary from the kubernetes release and put it in , so there is no extra step for user to manually download . Actually, in the first version of ubuntu k8s deployment, I copied the to every node, but I think it is an overhead, so I removed it.I think this is just enough.
Doc update : Move the kubectl into the PATH, it may need sudo privilege so I decide not do for the user.
Have the same issue, please update the doc ASAP.
Great job :+1:
Also ran into this bug after recklessly running ( instead of ) :)


If one target in multipath volume gets down and a pod that uses the volume is deleted and quickly recreated (e.g. by a Deployment), the new pod never starts. How to reproduce it (as minimally and precisely as possible): Deployment with single replica with one multipath iSCSI volume one target (e.g. switch the machine off). Multipath will mark appropriate path as failed: the deployment pod. will quickly start a new pod for the Deployment. new pod will be (or ) forever, What happened: Kubelet tries to WaitForAttach / SetUp volume for the new pod. The volume is still attached. Kubelet goes through all targets and tries to find out if it's logged in or not. Eventually it tries to read of the failed path and that returns error: What you expected to happen: Kubelet assumes that the path is failed / not logged in and tries to recover it. If it fails it continues with multipath with 2 available devices and 1 failed one. IMO, simple instead of will do its job just fine. /assign /sig storage cc for iSCSI expertise
Fix:


reported that conformance testing with the latest version of sonobuoy against a version 1.29.0-alpha.3 cluster fails on the support the 1.17 Sample API Server using the current Aggregator test: , related to I found the reason why the test is working fine in Kubernetes CI jobs but fails with sonobuoy is because the service account used by sonobuoy "sonobuoy-serviceaccount" doesn't have read permission of the non-resource url "/", which is used by the test to confirm the removal of group path: The permission of "sonobuoy-serviceaccount" is as below: There may be two ways to fix it: the RBAC rule of sonobuoy to include "get" of "/". However, could other test runners encounter this issue? Is there a doc listing the required permissions for running e2e tests? the e2e to skip checking the removal of the group path if the service account doesn't have permission to access the root path. Which way do we usually prefer? Running e2e with sonobuoy should succeed. a v1.29.0-alpha.3 cluster. No response <details</details<details</details<details</details<details</details<details</details<details</details
implemented this solution.
/sig api-machinery
thank you for opening this issue and working on a solution. I'm probably not the right person to decide on which solution is preferred. From my perspective, either solution is fine.
Note: Sonobuoy is not owned by any Kubernetes SIG. At most it is between SIG Architecture and the CNCF to choose whether to recommend it for running tests. SIG Testing owns the e2e image that sonobuoy can be used to schedule but not sonobuoy. IMHO this is a bug in running the tests via a pod, which is not how we develop the tests (they're run outside the cluster, by a cluster admin account typically). I would say 1) is the correct answer. xref: /sig architecture /sig testing /triage accepted cc
thanks for your comment. I guess 1) means non of the existing sonobuoy versions can pass K8s conformance test against a cluster 1.29.0-alpha3, as the RBAC rule is hardcoded.


I will file an issue to sonobuoy first.
question here is what are the set of client permissions required to run the conformance suite? perhaps a level of access or a more granular recommendation should be documented. what is your take on the root path requirement as approver for: EDIT: slack thread:
I agree, we already require e.g. privileged pods for a long time in conformance tests and we should make it clear that we expect a highly privileged client.
+1 for what is saying, that we should expect that the user running the tests has required permissions. These tests are not built for being run unprivileged.
i wanted to PR the admin note in our conformance docs under sig-arch, but then i saw this: so broke this guarantee. sounds like SIG Arch needs to comment.
This statement surprises me. Root discovery is one of the least-privileged permissions you can have on a cluster. For years this was even open for unauthorized users on the internet (if your api endpoint has been exposed). In other words, just add the permission. While thinking about it, it is surprising that the user in question does not have this access already. Don't we allow that for every authorized user?
In contrast, I hope that lots of privileged operations are tested by conformance today. If we don't test things like admission webhooks for example, we have much bigger problems. And admission webhooks are basically equivalent to cluster-admin.
that is outdated, we use hostNetwork pods one some of the tests IIRC
looks like we don't. before this ticket, i was under the impression that the conformance suite already requires admin and that all testers (like Sonobuoy) use admin level credentials. it is non-privileged (e.g., does not require root on nodes, access to raw network interfaces, or cluster admin permissions) looks like we may have to change that sentence completely.
Would have expected it here: The PR in question explicitly fixes the result returned by . If that is part of conformance (IMO it should), every conformance test needs access it obviously. Whether we require admin, or we add it manually, either would work.


adding to milestone and making release blocking ... we should not release 1.29 in a state that the most commonly used conformance tool will fail
IMO we should clarify the state and go fix the tool, but we should keep the test and we should NOT add skip. Conformance docs are not a release artifact, and sonobuoy release lifecycle is not a conformance artifact. If we can't get sonobuoy working in a reasonable time, we already know the tests can be run by scheduling the image (which is ours) directly with kubectl instead and we can document that in the upstream conformance docs. We need to fix the for this image: ,%20master%20(dev) And it should probably be to release-blocking in the future.
it looks like the e2e change made in was not consciously made part of conformance, it was just trying to functionally test the bugfix being made. For 1.29, I think we should revert the change to and make an integration test that validates the fix instead. We can revisit whether content of is part of conformance or not for 1.30
I opened to revert the e2e change and move the functional test to an integration test
Filed to try to get the image job passing again.
We probably need another issue to track 1.30 follow-up?
If no one has done it, I could create another issue and follow up with the items that have been discussed here: outdated privilege requirement in commonly used conformance tools like sonobuoy aware of the required priviledge to avoid similar situations in the future. permission of GET "/" to authorized users via the role for root discovery.


The latest release of CNI plugins is , but we still have . I'll attempt to create a PR for this based on the .
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
- assigning to you :)


I started the federation-apiserver and went to and got the following output: Expected: I should have got an empty v1.ServiceList Any idea on what we are doing wrong? Have you guys seen this error?
I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote:
Deep copy generation was slightly broken as well - you may want to double check that they are registered. On May 23, 2016, at 5:50 PM, Daniel Smith wrote: I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote: — You are receiving this because you are on a team that was mentioned. Reply to this email directly or view it on GitHub
I got the same error with latest branch, as I was swamping on service controller, so I use a previous version and worked around it.
deep copies were indeed missing, but adding them hasnt fixed the issue. Will debug more. is my PR to add the deep copies cc Please feel free to file issues for any such problem you face. We need to fix all these issues. Its good if we know about them early. Thanks!
fixes this


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /assign What happened: The tests are failing again because the scheduler messages have changed yet again. I'm going to remove the check for the exact messages because this is at least the 3rd time this has happened. Instead depend on unit/integration tests to check for specific failures. Also there's no need for these tests to run across all types of local volumes because it's just testing the scheduler part.


Typically, the dns and dashboard is in crash loop. Ref cc
if you check the file, does it contain lines like ?
Previous comment is ref: from I'll copy that to container-linux as well.


What happened: When debugging timeouts related to the stopping of cgroups, the current logs from -tryKillingCgroupProcesses do not provide any correlation between the PIDs which were killed and the cgroup they belonged to. What you expected to happen: Log the cgroup name along with the PID How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: N/A Environment: Kubernetes version (use ): master
/sig node


The metric used to report acutal pod in the label , but now I see the podname prefix only. The full pod name in the label, so I can trace the metric to the failing pod. Check raw api and verify the actual pod name with uids. <details</detailsAzure AKS <details</detailsNo response No response No response_
/sig node
/triage accepted /priority important-soon /cc As discussed this is a change in functionaloty that wasn't highlighted in release notes. Need to decide on what do do with it. I understand it's a change of functionality. what is the side effect of this for you? Does the label without last part not informational enough?
I've been using the metric in my custom grafana dashboard that shows CPU, RAM, IO usage of pods with graph of increases in probes. If the apps are stable, the probe graph looks steadily. I can see if some irregularities in system metrics are influencing probes of respective pod. Also, if readiness probe is failing, the pod is removed from service, but is not restarted so I must have knowledge of pods with failing readiness probes. I attach the screenshot of my graph. ! So I'd welcome the possibility to pair the probes with exact pod and thank you for looking into it.
are we thinking that this resulted in an undesirable change? Spologies for being a bit late to reply, I'm returning from a brief paternity leave staycation.
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
Well this is awkward... I still believe the actual behavior does not respect the intended idea...
This issue makes it hard to link probe metrics with right pods. The value of pod label in and are also inconsistent with how it is used in other kubelet metrics, see: vs Definitely a bug, regression since 1.25.
I can't get anyone to look at the fix/revert: you may have better luck
/milestone v1.28
discussion in
You've pushed it through, congrats!


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kube-proxy doesn't validate the configurable IPVS tcp, tcpfin and udp timeouts. For example, when kube-proxy starts with , it will always log a warning: . What you expected to happen: kube-proxy should validate --ipvs-tcp-timeout, --ipvs-tcpfin-timeout, --ipvs-udp-timeout flag values at start. How to reproduce it (as minimally and precisely as possible): add flags to kube-proxy Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network /area ipvs
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Node external IP of azure VMSS cannot be obtained when cloud-controller-manager is enabled. Corresponding logs are shown as follows:
/sig azure
/milestone v1.16


In 1.1, if you specify "current-context: i-do-not-exist" in a kubeconfig, and there's no matching context by that name in the contexts, kubelet silently uses what I assume is either an empty or a default configuration. This can result in hilarity like wondering why your kubelet isn't using the TLS settings you provided, and spending several hours debugging the whole stack only to discover that you typo'd current-context :). It would be better if kubelet and other users of errored out on such a misconfiguration, to make the failure obvious and trivial.
cc
I think kubelet silently uses an empty configuration. Guess we should give a waring here.
Did you start to work on this?
Below is related code: can you confirm this? Besides if we want to log a waring in kubelet, we may need to get in kubelet.
cc
cc
I believe kubelet already uses . Adding a check in that current context exists if specified sounds like the right approach.
Are there any backwards compatibility implications here? would this break users who have existing configs (that fallback to default namespace)? , Jeff Lowdermilk wrote:
I think kubelet directly use not . Also already checks whether the user-specified current context exists, the problem is that if not exist, we just skip. We need to do something if the current context not exist, like log a warning.
Erroring on a specified and bad seems reasonable. If it is not specified and there's only one option, I don't think we should fail.
yeap, the case is that user specify the but not exist, in such case we should just fail.
If no one is working on this I can take it.
I think this is the root cause. Basically we should error on this not exist context. If you have extra bandwidth, please go ahead :)
This is related to - the defaulting is what allows us to continue. When that merges, only kubectl will have the "use localhost:8080" as defaulting.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the example in , I created a pod with 3 sysctls specified in the pod's security context. Setting the sysctl resulted in the pod failing to start. What you expected to happen: I expected the pod to start with no issues. If there is something missing in the docs, I appreciate updating There is also a discussion at that suggests that kernel does not namespace certain net.* sysctls. If that's the case, can we document which sysctls are supported? I assumed is a supported sysctl because it's in the official k8s docs. How to reproduce it (as minimally and precisely as possible): Following the pod creation, the pod doesn't start: Once I delete the following two lines from the pod securityContext, the pod starts fine: Anything else we need to know?: You can see that I've enabled the unsafe sysctls with kubelet: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: VMware Fusion OS (e.g: ):Kernel (e.g. ):Install tools: A single-node k8s cluster created with kubeadm Network plugin and version (if this is a network-related bug): Calico Others:
/sig node
Thank you for the incredibly detailed issue report :) I did some poking around, and if is accurate and still current, then I believe the issue is that is not namespaced (i.e. cannot be set on a per pod/container level). If I'm understanding correctly, we'd need to make changes to the Linux kernel in order for it to be namespaced. As a disclaimer, I'm not positive if this parameter became namespaced in new kernels... it could be interesting to investigate if this still happens on a machine with the latest kernel. W.r.t. fixes... I would agree with you that short term, we need to update this documentation so that it doesn't include . do you have any interest in doing so? If not, I'm happy to.


Long term, I think it would be great to have better documentation re which sysctl parameters are namespaced (and thus supported) and which ones are not, and also a more descriptive error message if you try and utilize a non-supported parameter.
This historic issue also seems relevant: Perhaps we were premature to close it.
Matt, thanks for confirming the problem! I found the following comments very helpful: Looking at , there doesn't seem to be any difference between Linux 4.15 and 5.1. I'm happy to remove the references to in the docs, but before doing so, should we confirm sysctl worked at some point in the past? As you said, we should enhance the error message and the docs with the supported sysctls. I understand documenting the namespaced sysctls can be a moving target, but not everyone likes to check the Linux source code to see what's namespaced or not. :)
It's probably better to change the docs to set a namespaced sysctl like or . A good way to figure out what network sysctls are namespaced is to pass to kubelet and see all the sysctls listed under inside the pod.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Agree with all your statements here :) I'm also surprised that ever worked, but perhaps we could sync with the original author of the docs? +1 to in the meantime changing the docs to a namespaced sysctl.
I don't think ever worked as it's hardly to believe a sysctl got unnamespacified in the kernel (it's always the other way around). is a good example of namespaced networking sysctl. However, the above suggested is not in some cases - it's unnamespaced in kernel 4.14 and became namespaced in 4.15. In general, this is hard to document because there are hundreds of parameters and some of them is kernel version dependent and you always need to lookup the kernel code for answers. I think what we should do now is to with in the example in the doc. in the doc to something like . I think this should be very close ( is an exception) - I couldn't verify all of them as there are so many sysctls and they can also be by loading kernel modules.
SGTM, can you drive that?


Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): HEAD Since v1.7.0-alpha.3, control plane images are only pushed for amd64. Compare: to This is a release-blocker and should be fixed ASAP. I guess the actual regression code-wise happened in test infra. cc
/assign
also cc
/remove-sig testing
Still looking, but "Only tag release containers if the name is different." makes me suspicious.
oh, this is a bug introduced by . fix should be easy.
I can open a new issue as well to track what I sig-testing for but I want us to set up the required testing to make sure this never happens again
the removal was under the guise of "sig testing doesn't control the cutting of bits", if there are tests that need to be put in place to ensure all the right things are cut/released, I would expect sig-release to track that eg: if there's code that needs to be changed in kubernetes/release, it's a sig-release issue; if there's code that needs to be changed in kubernetes/test-infra, it might be a sig-testing issue (or we can triage to the right group)
Sounds good
Details about the exact bug and the fix are in . The reason we didn't notice this sooner is that anago only if the required image doesn't exist - it does so because we don't support all arches for all release versions, and we didn't want to hard-code a list of expected versions into the release repo. (Also, anago automatically , so we were still pushing the legacy versions - we were only dropping the non-amd64 images with this bug.)
Can we add a after the warns? Otherwise even is the user of anago sees the warning, there's nothing he can do about it.
Anago is moving toward being less interactive. If any checks need to be done, they should be done at build time. 1) Determine what should be built 2) Stop right there if the build fails for whatever reason. Adding redundant checks comes with the cost of maintaining hardcoded lists in multiple places. The push step (and release infra in general) should be agnostic and just push what was built.
But in this issue, the build was successful, but some images were removed due to the bug.


I'm worried that when building beta.2 this Wednesday, if a similar problem happens, i won't be able to stop anago from pushing the incomplete image sets.
I'm reopening this unless we are really sure that it actually works...
How can we create the testing required to stop this from happening again?
I'm doing a dry run today.
anago could have some sort of sanity checking, though I'm not sure how we'd do that without maintaining hardcoded lists, as notes. I guess the "real" fix is having some automated testing somewhere to consume the non-x86 docker images.
After anago dry run, i saw these images on my machine, so it looks fixed.
Thanks and :tada:!


When user creates a static PV using folder name in path: on vSAN backed datastore then detach disk does not work because check fails - The code expects folder name to be UUID but it does not and hence it determines disk is not attached to the node and hence no detach is ever performed. /sig storage cc


The replicas is 3 but just create 1 pod 'web-0' and one pvc, then kube-controller-manager panic happend. Should create 3 pod and 3 pvc add '--feature-gates=StatefulSetAutoDeletePVC=true' arg for kube-controller-manager, then create statefulset with 2 replicas and volumeClaimTemplates. No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig apps
I can reproduce the issue and want to fix it. It seems like variables cause . /assign
/triage accepted
Great find! Does this only happen on an upgraded cluster? I'll spin up a cluster and see if the e2e test catches this problem.
The e2e tests are still passing on a 1.24 cluster, so I suspect this is an upgrade problem. I wonder if there are other holes here.
I installed the Kubernetes 1.25 cluster directly using and still reproduced this issue, so it was probably not caused by upgrading the cluster.
Interesting. I thought that would set the policy to be non-nil. Let's keep this bug open after you submit your PR as I think something else is going on here---I don't think we should be seeing the null policy in the first place.
I suspect that somewhere accidentally set it to nil, and I'll keep looking at this code.
I tried the following (and also with parallel pod management), and it didn't seem to cause the panic. This was a 1.24 cluster, maybe this is only happening on head versions?
I just tried this on a kind cluster running on k8s head, and the problem didn't seem to reproduce. But this was setting the feature flag default in I wonder----does the api server need to have the feature gate as well? That would explain why defaulting wasn't happening.
As far as I'm concerned,I just set the in kube-controller-manager manifests to reproduce this issue. I repeat the yaml used as follows: I tested that in statefulset yaml, setting or not would cause the component to panic. The panic logs are the same as above, so ignore them here.


statefulset with : The environment
I believe you---this doesn't seem to be well-documented, but it's my understanding that you need to set the feature gate on all of the system components. Obviously failing to do so shouldn't cause a panic, so thank you again for finding & fixing this bug! Let me ask around to see what the expected behavior is. I know there may be skew between kubelets and master versions, but I didn't expect to have to deal with skew between master components.
I read the code, and now I understand what you mean. This feature gate must be enabled in both the and components. We need to describe it in the to prevent someone from ignoring the component, which also needs to enable the feature gate.
I don't think we need all the system components, like the component, right? I read the original KEP documentation. This feature gate only involves components and :
I had a bit of discussion about this with others. Because upgrades don't occur atomically, any feature gate has to work regardless of how the other system components are configured (things shouldn't panic!). I suppose if the cluster administrator wants to track what needs to be enabled on each feature they can try to sequence which component has which feature gate. If it were me I'd try to make all the components consistent though.
This is certainly a method, but it is a bit more cumbersome for the administrator. It is best to have a place for the document to be queried by the administrator and avoid confusion. I don't think everyone is very familiar with the k8s mechanism. I suggest adding a partial description to this section of the : Before: After: I can submit this PR on .
This is going to be true of any api change. When you tried to use it with only adding the controller manager feature gate, I think you would be able to set the retain policy, but there wouldn't be any defaulting or validation (?).
I've seen other feature gates that say what components are needed to enable it to use the new feature. For example: If it's not necessary, then I close .
only adding the controller manager feature gate, I think you would be able to set the retain policy, but there wouldn't be any defaulting or validation (?). I remember just enabling it in kube-controller-manager.


it using the retain policy, but the statefulset does not have the policy from output of (It should be the apiserver authentication forbids it)
Yeah I think that makes sense. Thanks for finding this issue! After your PR with the panic fix is submitted, I think we can close this?
Yes. I agree. /close
Closing this issue. <detailsIn response to : Yes. I agree. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Follow up from: See details there. When is , there is no need to discover the NUMA topology. /sig node /priority backlog /kind feature /help-wanted
/cc
/triage accepted
I would like to take this up also can I get more brief information about it? code review comments from asking that inside the NewManager() implementation, we need to skip NewPolicyOptions and NewNUMAInfo but when the NumaTopology comes kubelet issue also comes. I need some more information to start with this!
/cc
When there is no topology manger policy specified, NewManager should be a no-op. Just skip all checks.
Exactly. Just to add: we may need few iterations over the change to get a clean flow in the call.
/assign


Event for a pod: The cni plugins do exist at /opt/cni/bin. They cannot be executed and bash is complaining:
cc
I'm guessing this is a shared lib problem, since /usr/grte/v4/lib64/ld-linux-x86- doesn't exist on my gce vm. We should update docs to make it clear how someone would install manually (if they run into such problems again) and statically link (if the problem is indeed the shared lib) with something like Unsure how we'd support different GOOS (vend different tarballs vs checking in and building the plugin inline with our make scripts)
uggh. These binaries need to be ultra-static or else we need to containerize them.
Essentially doing would've prevented this bug :)
And see for the CNI_ARGS problems
also , re ... however, I'm not too sure if your comment was mean to be on this ticket.
cc
I have a working version. I uploaded my cni binary with bug fix on GCS. change source to change sourcehash to Welcome to give it a try.
Also see problem:


Failed here /retest Originally posted by in This fails because the pod tries to bind port 80, that is a very demanded port, but there are two tests doing that, that increase the chance to collide
/sig network
Close when merges
/triage accepted


Failed due to an apparently unrelated PR: https://travis-
Problem here is that I left a small chance of a double-call in my parallelized cache. Didn't realize this was being tested. Will fix.


Golang 1.13 uses TLS 1.13 by default (opt-out), which introduces RSA-PSS. RSA-PSS does not work with keys that are too small. The issue was uncovered during Fedora mass rebuild which uses Golang 1.13: See /sig api-machinery
What is the failure mode if a smaller key is used?
/sig auth /sig scalability
/sub


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: kubernetes/node-problem-detector/node-problem-detector-ci: ci-npd-e2e-node See the Which test(s) are failing: [] NodeProblemDetector [NodeFeature:NodeProblemDetector] [] SystemLogMonitor should generate node condition and events for corresponding errors Since when has it been failing: Dec 03. was at 19:50 PST, was at 21:51 PST. Testgrid link: https://k8s- Reason for failure: Apparently is the culprit. Anything else we need to know: This is likely caused by some library upgrade. Nothing big.
/sig node
/assign
Seems like the failure is caused by . /cc /cc /cc /cc
/priority critical-urgent
The particular bug I care about has been fixed in . I'm gonna still work on submitting , as the same error should also be affecting a few other tests.


PR introduced an annotation for fluentd migration from static pod to the daemonset to avoid two instances of fluentd on a single node during the transition stage. I didn't read the pr carefully, but based on the code at , every node object has one label be default: "". I think this is wrong. 1) Not everyone use fluentd as the logging agent. 2) It is hard-coded in Kubelet code as part of kubeletnodestatus. No matter running fluentd as a static pod or a daemonset on the node, Kubelet shouldn't treat it different from other pods. The right approach is introducing an external controller to apply this cloud-provider / kubernetes vendor (i.e. GKE) specific label to the node. cc/
The string "fluentd" shouldn't appear anywhere in our code. Please remove it ASAP.
Absolutely agree with you, thanks for bringing this up! Is it possible to apply labels to the node as part of the node startup script/salt?
To clarify, I'm thinking about possible ways to provide a fix for that ASAP. Writing external controller is a best solution, but here we have a release coming and this label is intended to be only a temporary measure to perform upgrade from using fluentd as static pods to using daemon set.
There are many ways to achieve this outside the core system. But I believe GKE already has an existing mechanism to add this kind of labels to the node, and the consistency is important for GKE management. cc/
OK, thanks! However, I believe we should be sure not to break everyone's upgrade process, i.e. after master upgrade to 1.6 on any platform, there should be only one fluentd agent on each node. I propose changing the configure scripts and salt to set up needed labels in kubelet's flag (). WDYT?
It seems to me like upgrades could survive without the explicit dependency on fluentd: I'm not sure if upgrades call initialNode or not, my own ignorance ? If they do, and this is removed, can't an external controller just do the work of re--adding that label if its dropped?
thanks for raising this. Let's sync f2f
cc
WDYT about exposing kubelet version as a label and have a selector in fluentd ds config targeting version 1.6+?


and I talked about this issue a bit for migrating kube-proxy. One idea we came up with was that of running a controller that will identify kubelet versions and apply taints to nodes running static pods. The daemonset version would not tolerate that taint and that way we keep daemon set pods off of nodes running static pods. Have you considered that approach?
It doesn't differ from a controller applying labels. I'd like to avoid necessity of introducing such controller, since it's not clear what is it's responsibility (generic controller for applying labels on nodes based on some rules? a custom controller just for fluentd purposes). In GKE we can do it on GKE control plane side, but we need a story for open source.
Move applying labels to startup scripts/salt seems to be the least intrusive and easiest to implement way.
Actually what I proposed in is not an option since nodeSelector in PodSpec is old style selector.
I'm not a big fan of approach with salt, but it seems there is no better option.
Re: Instead of exposing Kubelet version as a label, why not using field selector. Kubelet version is part of NodeStatus already. I just simply don't want to have a duplicate info for the same object.
is it possible to schedule based on fieldselector? I'm only aware of using field selector while listing objects.
Also node-labels option is in alpha
But it's present since 1.2 What is the status of this flag? Can we use it for fluentd purposes or better avoid this?
Maybe it does make sense to implement a temporary ad-hoc controller? It's the easiest to deploy solution, just another in fluentd addon.
I know. It totally doesn't make sense to introduce anything custom. I'll use just the flag. It's rather hacky but unfortunately I can't see anything better.
Cool thing about controller is that you can deploy it into any cluster and it will work, with CA also. You don't have to set up your node labels manually then or restart the cluster.
There is a number of open question with having a new controller: it run on the master or a node? we guarantee that it will be always running? it have any resource requirements? I'd like to avoid discussing it right before the release.
Having a controller will help with kube-proxy transition too in the future.


, Piotr Szczesniak <:
is this a 1.6 release blocker?
I think yes. The PR is already sent and approved. I need to fix e2e tests because it doesn't work well with the test infra (e2e tests passes locally). EDIT: I'll do it hopefully today.
The PR is ready to be merged and lgtmed. Waiting for approvals.


Files storing latest versions for minors are not updated since infix was to the version string. Example Similar issues can be found in 1.25 and 1.24 builds (last 1.23 build infix so it has not happened there yet): I expect the files to be overwritten with newer version of the build. Manually run build job for one of the minors. No response Not related. Not related. No response No response No response No response
I believe this may be also responsible for breaking kubemark tests: since they fail due to version mismatch
/sig release
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority high Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for raising the issue! /cc who is working on before
I assume this will be fixed automatically with the next patches, because then we would compare with . Anyway, I think we should fix that intermediately by manually updating the version markers to match their correct version. I did that now for the bucket
We had to update the versions markers again, because those were pointing to two commits earlier and didn't include the Go updates. The latest version markers are: v1.24.12-35+ v1.25.8-37+ v1.26.3-41+
latest-1.27 has been affected by this as well. I updated it manually to v1.27.0-9+. See this thread for more details:
/triage accepted
This should be fixed now, closing it. Feel free to reopen if we missed anything.
This issue is still present, see this Slack thread for more details: /reopen
Reopened this issue. <detailsIn response to : This issue is still present, see this Slack thread for more details: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a similar situation with latest-1.28 which is currently pointing to: . This is one commit before the release commit: ! It's mostly like that the release step didn't update version markers, we should look into this.


As of now, I'm going to manually update this version marker to unblock 1.28.1 patch release. I'll also take a look into this. /assign
The PR that fixed this issue got reverted so this is still relevant: /reopen
Reopened this issue. <detailsIn response to : The PR that fixed this issue got reverted so this is still relevant: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're running into this issue again with stable-1.29 marker. It's currently pointing to which is It should instead point to I'm going to manually change it following the same procedure as the last time. Version marker updated to
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted
Same issue for 1.30, where I had to update the version marker manually again.


is a bash script linter which could help catch common problems in our (numerous) bash scripts. I know we'd like to reduce our use of bash scripts, but until then I think being stricter about script quality would be helpful. /cc
cc you were talking about this earlier. It has to only check modified lines.
False positives are a problem. For instance, not being able to follow a call through an environment variable causes an error. I would be much more comfortable with a non-blocking way to point out these issues, and leave to it human judgement.
Yeah, I've been interested in doing this for a while, but can be pretty persnickety and I haven't yet taken the time to configure it to be less noisy (or figure out if it can be). It's good, though.
Yeah, it would definitely need to be configured to ignore a lot of types of errors, but I still think it could be used to check for some types of errors. We could start with a pretty minimal set.
I would settle for something that just produced a quick-fix format artifact in the GCS bucket, and then the PR munger reading from it and commenting on the PR. We could re-use that for other linters too.
has been pretty accurate historically.
shellcheck, not spellcheck
reading too many issues in a on
/sig testing /priority awaiting-more-evidence Might be more noise than it's worth, but we do pylint our .py scripts now
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .


Send feedback to sig-testing, kubernetes/test-infra and/or . /close
I'd like to reopen this, and eventually implement (first we need to deal with the failures ...) I've easily found a lot of bugs in test-infra and kubernetes/kubernetes bash with shellcheck. See also /assign
/reopen /remove-lifecycle rotten /priority important-longterm
Reopening this issue. <detailsIn response to : /reopen /remove-lifecycle rotten /priority important-longterm Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We also have non-blocking and optionally-run presubmits, even just having an easy way for reviewers to request a shellcheck result for changed files seems like something we should do. I also think we should fix the existing failures. Glancing through some k/k bash with shellcheck in my editor, these generally seem to be things that are actually wrong.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten coming back to this hopefully
This is done now. Only took two years :^)


We are getting the following error when the apiserver connects the webhook on localhost (configured via URL). We should only use the SSL tunnel for the connections to nodes when the webhooks are running as services. /sig api-machinery /kind bug /assign
/close