<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: is consistently failing because of PodPreset feature tests, which require settings/v1alpha1, which is no longer enabled by default What you expected to happen: . How to reproduce it (as minimally and precisely as possible): . Anything else we need to know?: This was caused by and kubernetes/test-infra Environment: . Kubernetes version (use ): . Cloud provider or hardware configuration**: . OS (e.g. from /etc/os-release): . Kernel (e.g. ): . Install tools: . Others: .
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig service-catalog
does the alpha CI job enable alpha API versions?
file has "AllAlpha=true" Maybe would be better?


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl version --client=yes command is giving the following error message. Error message is repeated at the start and the end and also containing programming jargon Error: invalid argument "yes" for "--client" flag: strconv.ParseBool: parsing "yes": invalid syntax Examples: Print the client and server versions for the current context kubectl version Options: --client=false: Client version only (no server required). -o, --output='': One of 'yaml' or 'json'. --short=false: Print just the version number. Usage: kubectl version [flags] [options] Use "kubectl options" for a list of global command-line options (applies to all commands). invalid argument "yes" for "--client" flag: strconv.ParseBool: parsing "yes": invalid syntax What you expected to happen: The error message can be improved with one of the two variants invalid argument "yes" for "--client" flag Examples: Print the client and server versions for the current context kubectl version Options: --client=false: Client version only (no server required). -o, --output='': One of 'yaml' or 'json'. --short=false: Print just the version number. Usage: kubectl version [flags] [options] Use "kubectl options" for a list of global command-line options (applies to all commands). (or) Invalid argument "yes" for "--client" flag How to reproduce it (as minimally and precisely as possible): Run the command kubectl version --client=yes Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: AWS OS (e.g: ): Amazon Linux Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
Hi Thanks for posting this issue! It turns out that this duplicate output has been in kubectl, and all the other cli cmds, since . This should be fixed by , and hopefully we can cherry pick it back to at least .


IMO it feels like a bad practice to have mutating resource handlers. The deletePod handler is used for queueing a Recreate deployment once it has no pods running. It should merely look at the ReplicaSets that have owner refs as opposed to using dc.getReplicaSetsForDeployment. The orphaning/adoption dance is already happening inside the reconcilation loop.
I'd like to work on this
feel free to grab it. I can't assign the issue to you so I'll add myself; ping me on the PR when you have one.


There is a funny subtle behvaiour in get namespaces, in which if i call more than one time, it seems that the removal from Terminating state is accelarated. It seems this might indicate a bug, or maybe its not a bug and the eventual semantics of is a little funny sometimes. in any case, here's what we are seeing : 1) when i delete a namespace the first time usning kubectl, we get a state change to . that is correct, of course. 2) but ... if make the call a second time, then the same status line is immediately removed from . Expected behaviour : A second call to "kubectl delete ..." shouldn't have any effect on the timing of removal of a namespace from 'Terminating' state*... So i guess there is some "eventual consistency" subtle bug that is resolved by repeated calls to
suggests this may not be e bug, maybe its just the etcd eventual deleter thingy.
Looks to definitely be a bug. The 2nd delete seems to have interrupted the delete process and artifacts (like events) from the namespace still exist in etcd.
investigating...
oops, see the error, will submit a fix.
I have a fix, but need a test case, will submit PR tomorrow AM


The test image currently uses as its BASEIMAGE: Per , v3.12 reached end of support on 2022-05-01 and is no longer receiving regular bugfixes or security fixes. Current versions of the image appear vulnerable to known CVEs as a result. I don't know if a fix is quite as simple as just The image's base should be an in-support version which is maintained with at least security fixes. No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig testing
is like that, but then there are 2 additional steps the changes the image that has been built with the postsubmit job the repo to use the new image More detail in If you want to try, please make sure you are going to follow up with all the steps, you can see other promotions PRs too /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : is like that, but then there are 2 additional steps promote the image that has been built with the postsubmit job update the repo to use the new image </details
Thank you for this information! Is the manual and of images described in that same document something that you would consider a prerequisite for a proposed change like this? Or is it safe to assume that the automated PR test bucket(s) would provide sufficient test coverage?
I always trust the CI system
Good enough for me! Leading off with
/triage accepted


We are using (and ) to interact with Kubernetes clusters behind a couple of proxy layers. One of these proxies completes the headers to include character encoding details. This was working fine until recently, when client-go/kubectl have started to fail when interacting with the discovery API. It seems that a recent change has made kubectl more restrictive of the content-type that it expects. I suspect that it is related to this commit: For example, this is an error from a kubectl command: We can of course try to fix it on our end, removing that charset information. However, it is a valid content-type and the way kubectl is handling the content-type there seems like it is relying on implementation details of kube-apiserver (that it does not add metadata to the content-type). I propose to fix this issue in , by properly parsing and handling the content-type.
cc /kind regression /kind bug /sig api-machinery /priority important-soon looks like the switch statement in downloadLegacy() needs to be more tolerant of the variety of content-types it can get from a server (including text/plain or no content-type if old servers could have returned that...)
/transfer kubernetes/kubernetes
Something went wrong or the destination repo kubernetes/kubernetes/kubernetes does not exist. <detailsIn response to : /transfer kubernetes/kubernetes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes
Yep, the string comparison for the content-type is flimsy, let's address that. Thanks for the report!
/assign
/triage accepted
/cc
The fix is being cherry-picked back to: : :


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If computer name prefix does not match VMSS name, then Azure cloud provider will use VMSS API instead of using metadata service data. This leads to throttling, because every affected kubelet is making API requests. What you expected to happen: Azure cloud provider should use metadata service data instead of calling VMSS API when instance is current. How to reproduce it (as minimally and precisely as possible): Set computer name prefix of the VMSS to something different than default. You should observe Azure provider making requests to the VMSS API rather than using metadata service, which leads to throttling if you have multiple kubelets with this bug. Anything else we need to know?: This is bug was introduced by - returns false even if instance is current, because it expects node name to match VMSS name + instance id, however node name is derived from the computer-name-prefix which can be different than VMSS name. Node name should be matched against , not against . Environment: Kubernetes version (use ): 1.18.x and higher Cloud provider or hardware configuration: azure /sig cloud-provider
/sig cloud-provider
/area provider/azure
CC
/remove-sig cloud-provider /sig cloud-provider (trying to remove the label)
/sig cloud-provider
Thanks for the reporting. Unfortunately, computer name prefix is not included in IMDS, so it can't be used without VMSS API for constructing the NodeName. Would check what could we do to workaround the issue.
Can you memorize the computer name prefix in the first request to the Azure API and then use it for the isCurrentInstance check?
opened to fix the issue. Would you like to have a look?
I can confirm that your change should fix the problem I reported. Do you know if your fix will be to 1.18.x?
is working on the cherry pick
Thank you for quick fix!


What happened: It's most visible in the test <img width="323" alt="Screen Shot 2020-05-20 at 6 09 10 PM" src="https://user-"It shows up in kind, kube- and kops jobs The regression was introduced somewhere within , possible culprits are: (FYI (FYI (FYI There are more tests affected, just not as prominently (visually inspecting ) [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data I wasn't sure whether to tag this as kind/regression or not, I'll leave that up to or someone /sig node /sig api-machimery /sig scheduling
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: It's most visible in the test (FYI (FYI (FYI StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data /sig api-machimery /sig scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.


</details
/cc
will increase the time for deletion of pod and it should be expected because currently pod is deleted from apiserver before removing the sandbox container but after this fix pod will be deleted from the apiserver only after deleting the sandbox container.
/remove-sig scheduling doesn't affect scheduling running time at all.
Yeah, is almost definitely the culprit. Last time we changed that code path, it also caused a regression. The degradation in performance is expected, but a 5x slowdown is definitely larger than it should be.
do we know if that's because the status used by the path is stale, or if removing sandboxes is actually that slow?
if there's not a quick fix, should we revert and bring it back in once the root issue is understood?
I think a revert is appropriate in this case.
ok, I'll open it
open at
I agree with that 5x is something which shouldn't happen and i can take a look on why thats happening and that might require changing the way how the sandbox is currently deleted which is only during GC.


Is this a BUG REPORT or FEATURE REQUEST?: What happened: shows: The use of "REDACTED" for a CA here is misleading, since it's a public certificate that is not a secret. What you expected to happen: I expected to see the full CA in pem format (like when you cat ) or something like "SUPPRESSED" or "TRUNCATED" How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Nope Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g. from /etc/os-release):Kernel (e.g. ):
(for thoughts)
Reiterating the mentions to trigger a notification: <detailsIn response to : (for thoughts) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
use you can see the complete certificate authority data
Hey Thank you - I have tracking that . The point of this issue is that at best it's a bad UX; at worst it's a miseducation on pki. "REDACTED" isn't the right word here
If there are multiple clusters displaying certificates, the configuration will be too long, so this switch is turned off by default. But we can add an example to help use. thanks
Hey I don't personally view truncation the responsibility of the CLI... users can pipe things to a pager if they really care. That being said, I'm not concerned about the length. The issue is purely about the word choice of "REDACTED". The term means: The use of the term "REDACTED" in the output implies that a CA public key is remove for security when (as you've pointed out) it is removed for truncation. Truncation and redaction are fundamentally different things, and k8s is misusing the word redacted for this use case.
Not quite a bug, but I think you're right a better term could be used there as it is a public key, not a private key. /remove-kind bug
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale I'm not sure what I need to do here. This was marked as a feature. Do you need something more from me?
We are waiting for someone to fix this.
/remove-good-first-issue I don't think this fits our definition of a :
So I have to keep replying to the comment bot every 30 days so it doesn't get auto-closed? That feels... odd
thanks for the education :) unfortunately other issues are higher priority and no one has stepped up to fix this. The autocloser is a form of bug bankruptcy that keeps our issue load in check. The contribex special interest group is definitely open to feedback on the bot.
The biggest hurdle in this is that Kubernetes is using some base64 trickery to display the string (basically setting the certificate data to the decoded bytes from ). We can change this to or but not (it has to be a valid base64 encoded string). I tried exploring multiple options for the text, but I couldn't find a suitable one. Given that we already print the user tokens and other credentials this is a bit weird. Should we redact those too? Is it okay if we print the CA certificate instead of truncating it?
I can't help but laugh at: I'm not wed to any particular name. I also have no problem displaying the entire certificate. The crux of the issue is that a CA public key is not secret or sensitive. It's a public key
I agree. The issue is with the constraints imposed by how this is implemented currently. I tried to come up with a suitable string (given the constraint) and couldn't find anything acceptable... The only suitable option that worked for me is . I personally prefer to print the CA certificate since we are already printing user tokens and client certs that are actual secrets.
I agree. Who do we need to get in agreement that printing the full CA is the best path forward?
the data is not omitted for security purposes, but to make the output more readable. see REDACTED is misleading... it should probably be "elided" or "..."
you can run to view full content
that's the crux of this issue - REDACTED is misleading.


However, pointed out that other "long" fields are still completely printed in plain text such as client certs, so I think we need to revisit the "readability" argument. What do you think?
they are not... client cert/key are elided in the same way for the same reason: something like "++data+omitted++" could be better If the intent of config view without --raw specified is to be readable (which it currently is), printing many extremely long fields isn't great
Hmm what about: I'm pretty sure that would give us a valid b64 string (8+12=20%4=0) and it would subtly hint that the data isn't a secret.
Only valid characters are . And adding a prefix from the certificate only complicates things. is a suitable usable option. I'll submit a PR to continue the discussion there.


Alpha environment tests are failing - The root cause of the tests failing in alpha environment appears to be: So basically kubelet is not allowed to update PVC. Which is weird. Need to investigate further. Also for multizone environment - This line looks weird from e2e: So the deployment is straight up not getting started. /sig storage /assign
For multizone failure it looks like the pod has a node selector that conflicts with the zone where the PV was provisioned
I think failures on alpha cluster is related to generated api code: Obviously none of the PVC patch code sets this stuff and yet is showing up in object diff. I am wondering why. cc
/wg apply /cc
It looks like we may have to reset field before comparing PVCs for admitting them as well.
Yeah, server side apply has an alpha feature to kube-apiserver that allows to track ownership of fields. This is what you see in the managedFields field. It’s automatically when objects are modified. If you want to compare your object against a staticly compiled expected object, you should be emptying/setting the managedField to nil.
this is the code in question: Thing is, we don't actually want the kubelet taking ownership of any fields other than the one it changed. If we nil out the managed field for the comparison, that allows the kubelet to mess with managedFields in any way it wants to
can you pick this up to fix failure in alpha clusters? Reading last comment - it doesn't seem like a storage issue.
/unassign /assign
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
We don't have a good way to do what is suggesting now, so I suspect we will have to default to ignore this field specifically when comparing (by setting it to null on both sides), at least for now.
will you be able to submit a PR to fix this?
As per /remove-sig storage /sig api-machinery
Sure, I'll try to send a pr this morning


from https://k8s-
Really suspect this is
This is putting us into test-go post-commit run is broken
cc:
marked this p0 - if a fix isn't easy let's try to identify what introduced this and revert.
This flake on: 2016-10-30 07:19 PDT First instance: : It is flaking frequently, blocking PRs from getting merged, and slowing the submit queue ( at 75%). As on-call, I'm going to try and identify the cluprit and revert. Can use all help in doing so.
can you investigate whether your additional tests could cause this (by breaking the deployment of those pods, for instance) , Saad Ali wrote:
PR looks most suspect to me. It mucked with , and the errors we are seeing in each of the failures is related to : That PR was merged at Oct 30, 2016, 6:18 AM PDT and the first flake in was at Oct 30, 2016, 7:19 AM PDT. Thoughts? CC
I'm going to initiate a revert of to see if that fixes the issue
IMO we can just change the flaky test to not get pods but some other resources, such as deployments. We can't really expect to clean up pods immediately.
I'm going to proceed with revert (). Please feel free to send a follow up PR to handle this more gracefully.
Going to leave this opened until we have a few hours of runs with the revert. Want to verify, with some degree of certainty, that the revert took care of this flake.
We had 15 green runs of in a row. Considering this flake fixed.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I create a job with initcontainer with restartPolicy : when the initContainer exec failed and it will be restarted, the pod Phase will be pending all the time. but in JobController, the only care the running pod. So this pod will be always restarted. What you expected to happen: when restart count to backoffLimit, the job failed. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig apps
/assign


xref: When running the GCE PD CSI Driver against all dynamic provisioning tests from this PR: We encounter the following test failures that need to be resolved: /kind bug /sig storage /cc
So it seems that the flag is getting passed all the way through to the driver and it is behaving as we would expect it to in terms of mounting. All the correct commands are being run: When exec-ing into the gce-pd-driver container we can observe that the mounted point is indeed read only. However, when viewed from the host node's perspective the mount is still read write. And therefore since the subpath folder is created by the host, the folder is still being created successfully which fails the test. This behavior suggests that is either not properly propagating the read only option on remount, or not properly remounting at all. I can't figure out how to "view mount events" if that's even possible, some help would be appreciated if anyone is aware of how to do that. That would help me narrrow down which of those two options is happening. I have also attempted (from inside the gce pd driver container) to unmount the point (successful) and bind mount it back with the readonly option directly (not using remount trick, still unsuccessful). Suggesting that maybe it is the that is not being mount propagated and not a problem with in particular. /cc
/assign
/milestone v1.13 /priority important-soon


From shippable, I noticed there are several failed cases, for example, E0413 17:34:38. ] watch of *api.Pod ended with: Stop requested --- FAIL: Example (0.61s) got: want: a-hello b-controller c-framework FAIL coverage: 85.9% of statements FAIL 7.652s cc/
another example:
ah-- this is a different thing. I will fix this, too.
I'm seeing this from an unrelated change
Yeah I think that's the one I expected to see.


Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): describe psp Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): Environment: Doesn't matter. What happened: doesn't show PSP's fields: What you expected to happen: All PSP fields should be shown. How to reproduce it (as minimally and precisely as possible):
If you're agree with that than I (or :) ) could implement it.
yes, improve the describer. Look at the scc describer for formatting.
any ideas which SIG should own PSP?
In this case it should be sig-cli /sig cli
yeah, this is cli related. I'd go with sig-auth for actual behavior.


Up to 1.21, that field is immutable. That means in practice, a CSIDriver object has to be deleted and then re-created with a different value instead. Mutation via delete/created is unnecessarily complex and as the only consumer of that field (the kube-scheduler) can handle mutation just fine (it'll always ask the informer cache for the current value), it's better to allow mutation.
/sig storage /assign
This is independent of allowing other fields to become mutable, but that might also make sense.
/triage accepted


Resource quota example is missing readme. Was it once there and was accidentally deleted?
I wrote something up in Origin repo I need to repurpose for Kubernetes. Sent from my iPhone


Which jobs are flaking: periodics-kubernetes-bazel-test Which test(s) are flaking: //pkg/kubelet/cri/remote TestVersion Testgrid link: Reason for failure: Error: Received unexpected error: rpc error: code = Unavailable desc = transport is closing panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference Anything else we need to know: To reproduce with stress: go test -race -c ./pkg/kubelet/cri/remote stress - TestVersion Detailed logs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
I 'll give it a see and fix it :)
/assign
/remove-sig testing SIG node owns their unit tests. SIG testing owns the framework, infrastructure, etc.
/sig node


A V(2) log line that includes preemption details (preemptee and preemptor), somewhere here We used to have most of this information as part of the logging for Events. But we removed the preemptor from it as a security measure Logs can have pod names.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/help-remove


It's at line 111 When the server stopped before timeout, the first goroutine will be block on because the receiver in the second goroutine is not exist. Changing to a buffered one may fix it. no goroutine leaks run related test with leak checker No response <detailsdev </details<detailsunknown </details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
/cc
It should close the channel rather than write to it, that's the easiest fix


The comments in says that : However there are several cases where this is false : When using a Service with a targetPort that references a name in the ports arrays of the pod. Some software (ab)uses this API to get which ports are used (See Although unfixable, my goal is to fix the documentation so other people don't have the same problem). The Azure Load Balancer implementation does this for instance. Some other network/security software might use this information. Also, is it possible to document/reference the above issue in the comment ? Since it is described as informational I expected for the network to work without this information. But it's definitely not the case. Not needed since it's mostly a doc issue. I'm interested in contributing this patch but would like some feedbacks before doing it. :) <details</details<details</details<details</details<details</details<details</details<details</details
/sig network
/triage accepted -- thoughts? I believe the doc are due for a refresh.
I'd take this one. /assign
I didn't see that you wanted to do this patch, I am very sorry and now I unassign it. /unassign
No worries But I'm still not sure where / what to add. Should I only fix the part saying it is only informational ? Is there anything else ? (cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


In , we merged a work around for heapster panic We should get rid of the work around after heapster version is updated. /cc /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


What happened: Latest kubelet + Windows Docker version 1.14.0-dev refused to start any containers, error output: "invalid option: Windows does not support MemorySwap" How to reproduce it (as minimally and precisely as possible): It's always reproducible with Windows Docker 1.14.0-dev + latest kubelet code base. Anything else do we need to know: It didn't happen for Windows Docker 1.12.2-cs2-ws-beta. The reason behind the issue was because new code was for Windows Docker to check Resources parameters, it doesn't accept a non-zero value for MemorySwap: But kubelet was default to set an unlimited (-1) value. The issue was gone in my setup after applied a private fix to disable MemorySwap by set it to zero for Windows.


During the startup process of kubelet, we have Since we have following Code segment 1 and 2 seems dup to me. Am i getting this wrong?


ci-kubernetes-e2e-gce-device-plugin-gpu.Overall ci-kubernetes-e2e-gce-device-plugin-gpu.Pod kubetest.Up Intermittently since 02/23 No response No response /sig testing /sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /sig node BTW, would you help to confirm if this is a release cut blocker? We plan to release 1.30.0-alpha.3 on Tuesday 27th February 2024.
It passed! Maybe it was some infra issues?
Failed again. /sig k8s-infra
Many recreating of nodes in log.
This is caused by infra flaky.
The problem is
This job only runs one test and GPU instances are hard to find we don’t need many of those instances for this job, can we please change this job to only use 2 workers?
/cc
tests failed in a recent run. It failed the first time. The Job failure now changed to test case failing today. It failed 4 times in a row.
/assign
I launched a test run, logged into the master node and took the kubeconfig and found the following: Relevant bit
Link some related updates recently on this job.
The error above in is weird to me. It said No firmware image found . In successful run before, like (02-27), there are no such failure logs. failure in only kubelet log shows the pod crashed and restarted for many times. No pod log or containerd log.
After , the CI is not always failing now and 4 in a row.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen as keeps flaking
Reopened this issue. <detailsIn response to : /reopen as keeps flaking Instructions for interacting with me using PR comments are available .


If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The problem is different with the failure after the revert. We(release signal team) plan to open an issue for the flake later. (Probably, we should open that one and then close on behalf of that new issue later.)
Feel free to close it then. However, it failed 3 times out of last 4 runs. It's more a failure than a flake in my opinion. If doesn't help/ not appropriate, I'd suggest to revert installer and image changes
comment is clear about the root cause, we have a problem installing the drivers, can you redirect this issue if you think is a bug?
After was merged, there are 6 failure in last 24 run. Tab stats: 8 of 10 (80.0%) recent columns passed (258 of 262 or 98.5% cells) The Flaky rate is lower.


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: Introduced in /cc /sig node /assign /milestone v1.20
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. Main tracking issue: The method of is a simple wrapper around . It should be converted into a simple closure instead of being a method. i.e. we should have, (code is in ) /sig scheduling /kind cleanup /help
I'd like to take this task. /assign /remove-help
Thanks, go for it! :smile:


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I0726 15:05:12. ] UnmountDevice succeeded for volume "volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" ) Here at the same time, simple, detail information is returned and formatting problems are caused. No format error. For example, only datail is returned. Found from running Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --#### What happened: I0726 15:05:12. ] UnmountDevice succeeded for volume "volume-name" %!(EXTRA string=UnmountDevice succeeded for volume "volume-name" (UniqueName: "fake-plugin/fake-device1") on node "mynodename" ) #### What you expected to happen: No format error. For example, only datail is returned. #### How to reproduce it (as minimally and precisely as possible): Found from running Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted /sig instrumentation /wg structured-logging
Seems like caused by , this line still not in structured logging, we need to handle these messages.


I don’t know if this package will be accepted for structured log migration. It seems that only kubelet is required now.
/assign
I think it‘s only kubelet required too.
Hi, sorry about that I didn't realize that you assigned yourself to this issue. Hope is helpful for this issue.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/cc
Interesting. I'm running the same test locally: And I'm not finding the same log output issue there. Can anyone else see it on (master)?
it's quite likely that the source code line changed. Search for "EXTRA string" instead.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue.


<detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


In We should be using ConteaxtEval for evaluation. Use ContextEval for cel evaluation n/a No response <details</details<details</details<details</details<details</details<details</details<details</details
/assign /triage accepted /sig api-machinery


E2eNode Suite.[sig-node] Node Performance Testing [Serial] [Slow] Run node performance testing with pre-defined workloads TensorFlow workload From the testgrid, it has been 1/2 flaky since the start of history (03/20/2022) and has been failing constantly (0/2) since 04/01/2022 No response /sig node
/triage accepted /priority important-soon
The failing test seems to be expecting GPU on machine and the image seems to be broken. I tried the image manually: Logs:
/assign as discussed at SIG Node CI meeting, can you please take a look? I see that pod is now running for couple minutes and then stops. I wonder if this is something environment related - maybe pod was evicted or something. I doesn't seem to be able to repro the failure of the pod locally
GitHub didn't allow me to assign the following users: xmcqueen. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is no longer a flake. It has failed every run since the start of June. Locally this test runs perfectly with a legit correct local run of the test suite code, doing the full tensorflow run of 300 epochs. The failure in the testgrid job seems to be fast (173 sec to build and fail, locally its 160 sec before it starts the tensorflow pod). I wonder if there's a blocked port or something from the data initializing python program. It needs access to: . Here's the command it runs: Is it feasible to add debugging code, like a pre-check for that http port? It would be good to at least force it to fail with a meaningful error message.
Note that HEAD is now at image version 1.2, and it is still failing per . The successful local runs of the test mentioned above used image version 1.2.


Locally image version 1.1 does produce the cuda/GPU error shown above by :
More info is needed to find the RC for this test failure. I've put in a PR to get the test framework to preserve the container logs when the pod fails.
The pod logs show that it is timing out on an http connection, and it looks like it's the one where it tries to download the test data set for the tensorflow run. You can see it . Search down to . The data_download script in the image attempts to pull down a dataset from: . That url is live and healthy, so they might be blocking the gcloud.
This test depending on that external data doesn't look quite right :/
I'm discussing this test with sig-testing. I'll post an update later.
Talked to from sig-testing: he's contacting the remote to see if they are blocking us, because it looks like they are is pulling the script from a that is So I see three choices: upgrade this thing to a more recent edition delete it grab everything that's needed to run this and keep it in-house <- clear winner here what do you think: Maybe the folks over at sig-testing have a place to store the artifacts in-house? They are not massive (about 10MB see below) Size of data about 10MB: the tools and libs for the tensorflow run: the training and eval files for the testrun:
Here's a relevant issue in a related k8s repo about using/not using docker registries: Registries used in k/k CI should be on Kubernetes Community infra
looks good for the subject of putting up a new docker image containing the deps. This seems feasible and effective. I'd prefer it to deleting the test. I'll discuss it a bit more with a few others, but this seems like the way to go forward with this. TBS, I plan to take the above listed test requirements, and build a docker image containing all that is needed, and reset the test to pull the new docker image.
I've got some Dockerfile changes to fix this. I've tested it locally with no network and it does the full 300 epochs. PR coming soon.
Is it worth pushing up 182 files to get this green? It is about 6M of python, and 5M of a test dataset.


Its all stashed away in the to get the docker image to contain the prereqs. It is selected subdirectories of the quite old archived version (v1.9.0) of the code from here: current version v2.7.2 and two small test data sets from here:
OK I got it down to 43 files, including the 2 datasets. That's the drivers and the datasets and the libs they need. It all gets packed into the docker image.
quick question - is there another repository that is under sig-node that can host this image?
To be clear, the test's docker image is currently in a repo, the : This PR adds artifacts to an existing docker image in the above repo.
I've cancelled the above PR. I'll try to get something else going.
There has been a decent amount of discussion in slack relating to handling the test's required artifacts. The solution that seems best is to get some practical version of the test requirements into locally owned and controlled storage, so the endpoints that provide access to the assets will at least be under internal control. This is not code, and is not under development. It is a static dataset, that is now stored in an offsite archive, and will not require maintenance. The datasets are published with the "": (see bottom right) "This allows for the sharing and adaptation of the datasets for any purpose, provided that the appropriate credit is given." Here are some links to the dataset and where its hosted: https://archive-
I'd like to get the dataset that holds the test data (about 650k) hosted in GCS where I can get it via http. With that change it is expected that the test will go green. The failures show it is failing when it tries to download the artifact from the uncontrolled remote endpoint. Somebody, please help me get this artifact into an internally controlled server. Thanks
Was the idea of downloading this only on image creation in dockerfile rejected?
Thanks for the reminder I did not fairly evaluate that proposal. I just jumped to my earlier proposal. Thinking about this proposal I see that it would mean downloading the test data for inclusion into the docker image, while the image's test driver tooling is built to pull down the dataset itself from a remote.


That is a simple change to handle, while simply re-hosting the artifact on GCS would require only changing the endpoint. Also building the docker would probably still require network access to the artifact, which would probably require us to put it somewhere anyway. I assume, putting it into the docker image would best be done by downloading the test data artifact from somewhere. Would we then be storing the artifact twice - once in the docker image, and once in a locally controlled endpoint? If the docker build server has unrestricted access to the artifact, this proposal would be good. I don't have access to the servers to run tests. However, I could put up an RB for the Dockerfile to try it out. This is probably worth the effort. I'll get an PR going for trying to pull down the remote test-data artifact from the image build server, and bunde it into the docker image, and I'll link it to this ticket.
thank you. We do not change the image for a long time, so downloading it once would likely be enough. Once it is self-contained, it is even better for the continuous perf runs in tests.
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Currently the image builds for the first arch (amd64) but fails when it enters the cross compiling phase (arm64) and specifically when it tries to do apt installs from within the qemu-*-static emulator: The other images in node-perf fail to build in the same way. The jessie-dnsutils image build fails to build in the same way. I have tried pulling in newer base images up to buster, but it did not fix the problem. -Start focusing on the version of qemu. - qemu is version locked in the Makefile and has not changed in two years. There was an and the images were building at that time.


Try to find clues over there.
I tried pinning many relevant versions of from but none helped. Here's a clue: This ugly hack gets it past the observed error:
Note the previous PR was missing an important change, but I do not see any feasible way that could explain the current problem with failing, so I have not felt any urgency to put up a PR for it. FTR I need to bump the VERSION. I did not bump the version in I thought maybe the CI would bump it automatically.
There's an image version in . Its still pulling the old version and failing. Bump it from 1.2 to 1.3. So one more commit is needed.
The new image is present and runs perfectly. Tested it on local linux and in google cloud: It ran to completion. Note the new image URL. Its NOT the same as the old one:
podspec references the image Here's the . tf-wide-deep's url is built The above has staging in it. . Its already built and pushed to staging. It must be promoted when this new version number goes out.
FTR: My "testing" above did not catch the current problem because I did not run the full e2e_test binary. I only tested the image via a . AND I should have read way back at the beginning.
Local run of : has to change too.
Local test using the published 1.3 image run the standard way was successful with the above: ran the test successfully. So promote the image per bump the image version in the e2e test manifest change the
After the image promotion is merged, there's STILL one more commit coming: bump the image version in the e2e test manifest change the
its promoted and is found at the expected url local containerd cleared out local test was rerun pulling from the new prod registry local test using that new registry passes
its fixed. the test is green again.


What happened: A deployment is created that uses a after the cluster master is created but before the cluster worker nodes are fully initialized. During the initialization of the worker nodes, the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector. Depending on how long the nodes are available for scheduling without the necessary node labels for the deployment, the scheduler will start to spam with cluster with pods in status. This spamming stops once the worker nodes are fully initialized and the pods are scheduled successful. The exact timing of all of this is TBD. What you expected to happen: Pods in status are annoying and require manual cleanup. It would be preferred that the scheduler avoid this an simply issue warning events. How to reproduce it (as minimally and precisely as possible): See issue description. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.18.3 Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 4.15.0-96-generic Install tools: Managed service install Network plugin and version (if this is a network-related bug): N/A Others: N/A
Related to and
/sig scheduling
by spamming are you referring to scheduler logs? also, why are the nodes not labelled from the get go?
Just to confirm, all the pods in question are still in Pending state, yes? If the pods are in pending state, all of them will get scheduled eventually once the nodes get their labels.
The pods are not in state. They are actually in state. The pods eventually get scheduled but we are left with anywhere from a few to dozens of these state pods. Also, we are looking at labeling the nodes via kubelet flags to see if that helps.
/remove-kind bug
After we add some retry logic and wait time in this problem can be solved: As you can see this function returns initial node as the first request failed.
I also get hit by this, when single controller node gets rebooted and scheduler starts as static container before kubelet becomes ready.


Can be cleaned up with:
/sig node
the spam cycle is this loop: "pod create, pod schedule, worker reject, pod failed" -- we don't have a lot of protection in higher level workload controllers to handle pods that fail at kubelet admission time versus pods that just dont create due to quota/admission. in general, is the spam causing any other stability issues? is pod gc pruning the pods?
I wonder if we can do a trick in node lister to mitigate any race condition here...
we could maybe do something similar to to ensure node lister has synced...
hacking on option to ensure if kubelet has a node lister with a valid kube client, we wait for it to sync at least once here which should mitigate this issue. see:
We noticed the problem on the OLM pod which is managed by the OLM operator. Right or wrong, the operator does not like seeing the pod go to status and no longer reconciles the pod. The end result is that manual intervention (i.e. pod deletion) is required to fix the problem.
sure... i would have thought the pod is failed in that instance and reached a terminal state. either way, i am looking at what i can do to mitigate this.
The problem persists in our environment as well, and the workaround cited above will not do anymore. Is there something in the works for this one?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Hey Guys , I try to use the option --override with kubectl expose . First i did this and it work : Now i want to add another service with kubectl expose and this command does not work I have this error If i remove --overrides the command work . `
/sig cli
expose tries to convert version from "v1" to "_internal" which is not registered. This is the reason that the function doesn't works. == "" 316 if inline := cmdutil.GetFlagString(cmd, "overrides"); len(inline) 0 { 317 object, err = cmdutil.Merge(scheme.Codecs.LegacyCodec(scheme.Scheme.PrioritizedVersionsAllGroups()...), object, inline) 318 if err != nil { 319 return err 320 } 321 } === 156 // LegacyCodec encodes output to a given API versions, and decodes output into the internal form from 157 // any recognized source. The returned codec will always encode output to JSON. If a type is not 158 // found in the list of versions an error will be returned. 159 // 160 // This method is deprecated - clients and servers should negotiate a serializer by mime-type and 161 // invoke CodecForVersions. Callers that need only to read data should use UniversalDecoder(). 162 // 163 // TODO: make this call exist only in pkg/api, and initialize it with the set of default versions. 164 // All other callers will be forced to request a Codec directly. 165 func (f CodecFactory) LegacyCodec(version ) runtime.Codec { 166 return versioning.NewDefaultingCodecForScheme(f.scheme, f.legacySerializer, f.universal, schema.GroupVersions(version), runtime.InternalGroupVersioner) 167 }
kubectl help expose Expose a resource as a new Kubernetes service. Possible resources include (case insensitive): pod (po), service (svc), replicationcontroller (rc), deployment (deploy), replicaset (rs) In order to control the scope, I only register them as un-versioned type in the fix.


xref I think this may be one of the few cases where we can actually change the behavior in a "breaking" way. User creates service with , is empty Controller or user updates the status subresource, sets (not wiped or validated, should be) User updates in any way, is wiped It's clear that the is not meant to be set when is not "LoadBalancer". The question is: 1) Fail validation? 2) Silently discard the value? 3) Warn and then discard the value? 4) Just warn and leave it. Because we allowed it in the past, but it is so volatile, I lean towards 3. and - the people most familair with this type. Edit: PrepareForUpdate() runs before warnings. So: (1) is easy (2) is easy (4) is slightly less easy than (2) (3) is slightly less easy than (4)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
OMG change it to clusterIP as expected Status is wiped update status as ClusterIP ... and if I edit the service back to loadbalancer it keeps the status??
/kind bug
/retitle Service can be set when !=
We talked about this one at the SIG Network meeting today. suggested we use the feature gate for deprecation option to phase the current (problematic) functionality out.
/triage accept
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Subpath unix socket tests can flake because of long socket path. On Linux the path limit should be 108 chars or less. /sig storage


A CSI plugin that the 1.10 controller manager can't detect (which could occur in a number of scenarios, such as when a rollout is being performed or a node is evacuated) seems to retry without backoff. In this case, I have a pod referencing a PVC -PV that references my CSI plugin, but no controller plugin running. The controller is probing every 100ms continuously for a single volume attachment. The controller should backoff. Have not tested with higher numbers of pods, but in a busy system this could cause the controller manager to burn CPU. Also, there isn't actually a great error telling the user why the probing is happening (I knew in this case, but an admin likely has no idea).
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /milestone v1.14 adding it to next milestone as per if it doesn't make it then it can be further moved
as KEP has been moved, unless an exception is made, /milestone v1.15
the linked kep above is unrelated to this issue. This is a bug and I don't think it needs an associated enhancement
sorry, got diverted by the comments on that KEP. /milestone v1.14
Hello from Bug Triage Shadow . I'd like to remind that code freeze is coming in 3 days (Friday, 8th March). Is this still planned to be fixed for 1.14?
We have not been tracking this for 1.14. /assign can you help triage and see if this is still an issue and if there is a quick fix?
I can't reproduce the issue, do you remember what you did? From the logs it seems that Watch channel consistently returns so new is called recursively (beware, 1.10 code, but it's very similar in master): Can it produce fairly stable 100ms delay between calls? Anyway, this is not usual code path, watcher does not return in most cases.


if I read the code correctly, such recursive must end up with timeout - caller does not care about return value and caller's watch probably won't return anything useful. Is there a reason why we don't return error? This will kick in exponential backoff in A/D controller.
clearing the milestone for tracking purposes to let the fix land in its own time /milestone clear
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I found the error would happen when a pv is attahing(volumeattachment has created) and the csi-attacher is deleted at the same time. Then the log would print so frequently (because of recursion).
is this fixed by ?
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: ! What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.20.1 Cloud provider or hardware configuration: OS (e.g: ): 7 Kernel (e.g. ): Linux zsm-test-2 3.10.0-1160.11.1.el7.x8664 SMP Fri Dec 18 16:34:56 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted /priority backlog


Hi , I followed the instructions in examples/logging-demo, and there was no "external IP" field for elasticsearch-logging or kibana-logging, as shown below: I start up my cluster via cluster/kube- Did I miss something or is it a bug? Thanks.
Correct. This demo does not create an ELB so there is no external IP address. Access is via service proxy at the master. Type: and you should see the URLs for Elasticsearch and Kibana.
please let us know whether: a. Satnam's comment resolves your issue b. the demo should be updated to prevent this misunderstanding c. the issue isn't resolved Thank you!
Yes, Satnam's comment resolve the issue. I updated the demo in .
Awesome, thanks for sending the patch! We'll discuss there. Always fantastic to have docs updated regularly by folks trying them.


https://k8s- Failed: TestNamespaceController {namespace}


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-slow [sig-storage] Dynamic Provisioning DynamicProvisioner [Slow] should provision storage with different parameters 5/13 https://k8s- v1beta1.StorageClass got removed in We should remove the v1beta1 test.
/assign can you help take a look?
/triage accepted
/sig storage
I run the test with the latest from master without removing the beta test and it failed with a different error before getting to the beta test part: Do you have any idea what would've happened in my setup? I redeployed k/k using master HEAD
For this testcase: I saw that the provisioned PD has a size of 1.5Gi according to kubernetes: And in gcloud It's really interesting that the test was expecting 2Gi to be created, is this something that was changed recently?
The logs you posted are showing pd csi as the provisioner. Were you testing with the intree driver or csi driver? The CSI driver should to the nearest Gb. I believe intree does the .
I think I did intree because I didn't send as an argument in my command: It's interesting that the PVC has this annotation too:
mmm something interesting about my setup is that I created the cluster with CSIMigration turned on Do you think that might be forcing the usage of ?
Yes likely. Try with CSIMigration off and see if that helps. If so, then this potentially seems like a rounding bug with the csi driver
Thanks I tried with and it did indeed round up to 2Gi.


In kubectl v1.8.0+, schema info is cached in the location specified by (defaulting to ), written with world-writeable permissions (rw-rw-rw-). If is specified and pointed at a different location accessible to other users/groups, the written files may be modified by other users/groups and disrupt the kubectl invocation. CVSS score: CVSS:3.0/AV:L/AC:H/PR:L/UI:R/S:U/C:L/I:L/A:N (3.3, low) What versions are affected? kubectl v1.8.0+ What configurations are affected? Invocations that point at world-writeable locations Impact Malformed responses written to the cache directory can disrupt the kubectl invocation Workaround Use the default location in the $HOME directory or point it at a directory that is only accessible to desired users/groups. (original description follows) ==== What happened: The files inside of ".kube/http-cache" are world writeable (rw-rw-rw-). While the default for these files appears to be the home directory, using the "--cache-dir" flag could put these files into a place where world writeable files would allow any user / process to modify the cache files. Modification of the cache files could influence the kubectl utility in a negative way for other users. What you expected to happen: Apply stricter file permissions to the http-cache files. How to reproduce it (as minimally and precisely as possible): Run any generic kubectl command which is successful and then list the cache directory $ kubectl get pods --all-namespaces $ ls -la Anything else we need to know?


: I estimate this is a low severity security issue with a CVSS score of "3.3 / CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:L/A:N" - Environment: Linux Kubernetes version (use kubectl version):Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS. Running kube api server in hyperkube. OS (e.g: cat /etc/os-release): NAME="CentOS Linux" VERSION="7.1808 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="7" PRETTYNAME="CentOS Linux 7.1808 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:7" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-7" CENTOSMANTISBTPROJECTVERSION="7" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="7" OSTREEVERSION=7.1808 Kernel (e.g. uname -a): Linux hackit.internal 3.10.0-862.11.6.el7.x8664 SMP Tue Aug 14 21:49:04 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Manual installation. Others: n/a
/sig product-security
Closing this issue. <detailsIn response to : if you think this is serious, send an email thanks!


</details
Adjusted the CVSS score to to indicate user interaction is required (since they must specify a non-default cache option pointed at a shared writeable location)
There are two places this should be done: PathPerm and FilePerm should be set here: the temp file and final file should have stricter permissions (probably 0660) here:
/cc
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9.
GitHub didn't allow me to assign the following users: yuchengwu. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
help fixing this is welcome, though it would only be fixed in releases 1.12+, per
/label official-cve-feed (Related to kubernetes/sig-security)


release-master-blocking ci-kubernetes- Sub-tests: - ! 2023-01-20 09:17:28 +0000 UTC TestRoundTripSocks5AndNewConnection/proxiedwithvalidauth-https(invalidhostname+InsecureSkipVerify)TestRoundTripSocks5AndNewConnection/proxiedwithvalidauth-http /sig api-machinery cc
/assign /triage accepted Will be fixed by Thanks for noticing, great report by the way


Found some error logs when sending series events to apiserver: seems the later eventTime differed from the first one in series events After investigation, I found the problem was in the marshal. MicroTime is supposed to work on microsecond level precision, but init with nanosecond level precision, in this case, generates microsecond level precision data and generates nanosecond level precision, so when unmarshalled, the new time is not considered the exactly same as the original time (they are different at nanosecond level) MicroTime marshal should work at same precision for json and proto, so unmarshal can generate the exactly same MicroTime result: This causes the series events won't be updated on api-server after created master <details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign
Good catch... we'd probably need to truncate to microseconds when marshaling and unmarshaling proto to ensure any already persisted data is fixed up when read. Something like this:
Thanks for advice, going to send a PR soon


Cluster logging assumes kubelet stores container log at a specific directory with specific name. Currently the path is: After the CRI work, the path will be: However, we don't have test coverage for this. We don't even know whether the log files are there or not. We should add a node e2e test for this, basically: 1) Create a container which prints logs 2) Check whether the expected log file is created. 3) Check whether the logs are collected in the file as expected. And it would be better to launch a pod to do the check instead of accessing local file system directly, which makes containerizing easier. /cc because you are collecting missing test. /cc
I will try to add a node-e2e test there
I think it's ok to add this for CRI. But there maybe problems on today's interface for rkt. If I'm not wrong, rkt does't store logs at same location with docker. cc/ Is this true?
Thanks! :)
After gets merged, we need to update the test to verify the cri log path.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Currently APIServiceRegistrationController watch service change and find the related APIServices related if any. Then it AddAPIService/RemoveAPIService for APIAggregator. I dig into the kube-aggregator and find it no need totally, because update APIServices status when related service or endpoint change. So if does not watch services, it will not miss any event for APIServices. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/cc
/kind enhancement /sig api-machinery
/assign
that does not follow. AvailableConditionController modifies the APIService object. APIServiceRegistrationController must still watch the APIService objects to observe those changes.
oh, I misread the description. You're saying APIServiceRegistrationController doesn't need to watch v1.Service objects


What happened: After fixing golint failures under (now still in progress), we need to put for each and . Then common error checking code has become like: We can replace this code with which is more readable and easy to be understood what the code does. Now we have following e2e test directories which can be replaced: [x] test/e2e/apimachinery/ () [x] test/e2e/apps/ () [x] test/e2e/auth/ () [x] test/e2e/common/ () [x] test/e2e/node/ () [x] test/e2e/storage/ () [x] () [x] () [x] () [x] () [x] () [x] () [x] [x] test/e2e/storage/persistentvolumes- [x] () [x] () [x] [x] [x] [x] [x] ()
/sig testing /kind cleanup
/assign PR will be submitted soon.
Fix the following in
Fix in
Fix in
Hi, maybe I could take files beneath test/e2e/storage/vsphere/
oops, I'm finished and prepare for PR.
Thank you That's OK
Fix the following in
/assign
All related PRs were merged. I think this ticket can be closed. BTW, should we also replace to ?
Thanks for your work. thing is managed with We are almost finishing the work also :-)
/remove-kind bug
I found in these files. I'm working on :
I can work to the following item.


[x] Original approval issue: [x] Test pr: [x] Two weeks soak end date: 19 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. a Deployment with a static label the Deployment with a new Label and updated data the Deployment to ensure it's patched all Deployments in all Namespaces find the Deployment(1) ensure that the Deployment is found and is patched Namespaced Deployment(1) via a Collection with a LabelSelector Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 5 points* This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
BobyMcbobs commented in original Issue: Unsure if the endpoint replaceAppsV1NamespacedDeploymentStatus is able to be hit. When this endpoint is attempted to be updated the data after retrieving the update resource doesn't reflect the values in the update
/assign
When using the appv1.Deployment structure instead of maps of interfaces to build the structure in a patch of the Deployment, this error comes up. Still in progress yment-static":"true"}: does not match template , spec.selector: Invalid value: "null": field is immutable]
replaceAppsV1NamespacedDeploymentStatus to test. Increase cover by 1 Endpoint
/reopen PR reverted by due to flaky test
Reopened this issue. <detailsIn response to : /reopen PR reverted by due to flaky test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen Soaking, ready for promotion 19 November 2020
Reopened this issue. <detailsIn response to : /reopen Soaking, ready for promotion 19 November 2020 Instructions for interacting with me using PR comments are available .


If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
/triage accepted
/merged


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I try to backport bookmark feature to our internal branch. When I enable bookmark feature, apiserver leaks memory. I spend some time to dig the root cause and find the bookmarkWatchers of service cacher is keep increasing. I am not sure if it exists at upstream code. Please let me know if it is ok at latest code. What you expected to happen: Avoid memory leaking How to reproduce it (as minimally and precisely as possible): a cluster. not update service. will find kube-apiserver's memory keep increasing. Anything else we need to know?: Kubelet will watch service with bookmark, and add a watcher to bookmarkWatchers. and service init resource version is 0 Without updating service, we cannot update "lastProcessedResourceVersion", it will keep 0, bookmarkTimer branch will always be continued. As a result, it cannot trigger popExpiredWatchers, so the service watcher will keep increasing every watching. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scalability PLAT :)
How about making this change ?
- can you also please take a look?
Agree with u. That's same way I fix in our branch. But I wanna hear the suggestion of and .
! With the same bugfix wrote above, we can see the memory not increasing any more.
Given that watch-bookmarks doesn't create any additional go-routines (well, creates 1, but cannot leak them), it has to leak the data. The only data that can be leaked is exactly bookmarkWatchers. So I looked into the code and got into exactly the same conclusion - that we must be leaking watchers for resources that aren't changing. The fix as above makes sense (though we also need a unit test for it).
Thanks a lot.
has the fix by
Thanks for reporting the bug, I will look at this day later. In addition, also need to think carefully about whether need to send a bookmark when there are no events at all.
cc


I noticed while running e2e tests locally that some tests were apparently being incorrectly skipped: I checked, and this is also true for our Jenkins runs. Currently it looks like and have this bug, though only the former actually skips the test, I think. The issue appears to be that the code inside the top-level in these files is being run before the testContext struct is filled out (and before is even called), and so testContext.Provider is empty. I'm not immediately sure what the proper fix is; I'm also worried that there might be other things broken by the fact that these blocks are called before initialization is complete. We also probably investigate if there is a better way to mark these tests as skipped; right now, there's no indication in the test report (, Jenkins test report, etc) that these tests were skipped.
I merged and which tried to disable these tests for non-{gke,gce} but apparently didn't follow-up and verify that they were still running on our platforms. /cc
Seems easy enough to fix. I'll have a PR out to fix this before Monday.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Testgrid: https://k8s-
/assign
The test's assumption about test parameter is wrong. Previously it assumed the group name is the same across all zones, but they are different in regional clusters. Looking into using node taints as the permanent fix, as suggested by the TODO comment in the test.
Hm I think taint based evictions is still an alpha feature.
OK. I propose the following solution: Get the zone of Schedule the StatefulSet pod to that zone using node affinity; wait for the pod to run Remove the node affinity from StatefulSet Delete
Can it be simplified? Schedule the pod Figure out which zone that pod got scheduled in Delete the instance group of that zone
Unfortunately I don't know of a good way to look up the instance group given the zone, since there could be instance groups from multiple clusters in the same project
How about: Schedule the pod Figure out the zone Taint all the nodes in the zone Kill the pod
Taint with NoSchedule? yeah that sounds good


Given a cronjob created with a scedule of "0 6 *" that failed at last run (status.lastSuccessfulTime is now "before" status.lastScheduleTime). Then I trigger the cronjob by hand and this jobs ends successfully. However status.lastSuccessfulTime is not updated, thus making it difficult to monitor failed cronjobs. status.lastSuccessfulTime should be updated with a timestamp of the succeeded job endTime Install the attached files: Wait until cronjob has actually trigged, test with to see that lastScheduleTime and lastSuccessfulTime are populated ok (as expected). Then apply the fail config-map, now the job will not complete OK: Again wait until cronjob has trigged automatically, observe to see that lastScheduleTime is updated but lastSuccessfulTime is not. Then apply the "OK" configmap and then create the job by hand: when the job has finished and completed ok, observe that lastSuccessfulTime is still not updated. Wait until next schedule and observe that the field is now updated. No response <details</details<detailsTested on kind cluster on Mac </details<details</details<details</details<details</details<details</details
Had to upload the files as .txt files as github wouldn't accept .yaml or .yml files
(Double-check: I may read wrong messagess ) /sig apps
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : When I tried without the prefix in the job name, the status is updated. /sig apps /area job not sure if this is a bug or an expected behavior. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area workload-api/cronjob
I have just tested in my kind cluster and neither with the same naming pattern as the normal jobs (which apparently isn't epoch) or with epoch - this will not trigger a update of the field. I have also tested it on another cluster (rke2 based) and here I'm not able to get the field updated no matter which naming pattern I use.
Sorry for the mistake. I can repro the problem you have and not related to the prefix. /cc
/assign
/remove-sig apps /sig cli
/triage accepted


AFAIK, we don't have a documentation on how to add a new API group. Currently we have and that try to add a new API group. I will draft a guide to ease the future endeavor of adding groups. And probably by writing this guide, I will see how can we make the API group machinery easier to use. cc


What happened: No endpoints created in the EndpointSlice for my service due to the port on the service being too long. Logs show: EndpointSlice created has no endpoint: What you expected to happen: Service that works with Endpoints should work with EndpointSlice. If this is intentional, it should be clearly documented How to reproduce it (as minimally and precisely as possible): Create a service with a long port name Environment: Kubernetes version (use ): 1.16 Cloud provider or hardware configuration: kind
/sig networking
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /sig networking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
For a rough measure of impact, Istio had 5 ports that were over this limit (and many more that were right at 15 characters), so I suspect it may not be terribly rare.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This needs to be resolved before endpointslices graduate from alpha
/assign
/remove-triage unresolved


Which jobs are failing: pull-kubernetes-bazel-test Which test(s) are failing: bazel test //pkg/scheduler/internal/queue:godefaulttest Since when has it been failing: Testgrid link: Reason for failure: WARNING: DATA RACE Read at 0x00c000230db0 by goroutine 47: runtime.mapiterinit() +0x0 (PriorityQueue).flushUnschedulableQLeftover() +0x17a (PriorityQueue).flushUnschedulableQLeftover-fm() +0x41 () +0x61 () +0xce () +0x5a Previous write at 0x00c000230db0 by goroutine 45: runtime.mapassignfaststr() +0x0 (UnschedulablePodsMap).addOrUpdate() +0x30e () +0x601 testing.tRunner() +0x162 Goroutine 47 (running) created at: (PriorityQueue).run() +0x172 () +0xe0b () +0x50 () +0x5a5 testing.tRunner() +0x162 Goroutine 45 (running) created at: testing.(T).Run() +0x659 testing.runTests.func1() +0xa8 testing.tRunner() +0x162 testing.runTests() +0x4ee testing.(M).Run() +0x2ee () bazel-out/k8- +0x2fb Anything else we need to know: I'm very sorry about that. UnschedulablePodsMap is not thread-safe. In , lock will be acquired before operating UnschedulablePodsMap. But in , lock isn‘t be acquired. In the previous case, only a single thread runs so it's ok. But after add goroutine , there will be data race. So I need to acquire lock before operating UnschedulablePodsMap.
/sig testing /sig scheduling


<!-- Please only use this template for submitting enhancement requests --What would you like to be : Current constructors: Could we export a new constructor like to inject custom queue. Why is this needed: In my project, i want to create a with a custom queue based on workqueue.Interface but have no way to do that.
/sig api-machinery
thanks for the issue! We've reviewed it in the bug triage, and the consensus was that to ask you to send a PR with this and discuss it there. Would you be able to do it? thanks!
Yes, i will do that.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: when I run , found bound variable err. What you expected to happen: Build success How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ):Linux paasn1 4.14.78-300.el7.Centos.x8664 SMP Sat Jan 5 15:49:43 CST 2019 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others: GNU bash version 4.2.46(2)-release (x8664-redhat-linux-gnu) Copyright (C) 2011 Free Software Foundation, Inc.
/sig release


What happened: kubelet doesn't seem to be able to manage containers started with CRI-O any more I'm testing kubelet + CRI-O on an Ubuntu 18.04 server installation, inside a qemu/kvm VM managed by Vagrant (see ), in order to be able to test k8s + KubeVirt + KataContainers, which are all compatible with CRI-O kubelet will obsess on starting new containers, which looks like and endless process, and the cluster never gets ready. For instance here are some syslogs : What you expected to happen: It used to work (see asciinema recording at ) a few days ago, but I fear some recent changes broke it. How to reproduce it (as minimally and precisely as possible): Testing with vagrant up --provider=libvirt on Linux with a clone of that commit: vagrant ssh into the VM and ps -edf f or tail -f /var/log/syslog would do... Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: qemu/kvm VM +OS (e.g: ): Ubuntu 18.04.2 LTS Kernel (e.g. ): Linux kubernetes-vagrant-01 4.15.0-48-generic -Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm as in Network plugin and version (if this is a network-related bug): Others:
/sig node
I can confirm this. If I start cri-o from a systemd service it fails with the above mentioned error message. Starting crio with without systemd works just fine. systemd file:
Thanks for your detailed report :) Just want to do some quick version confirming - you've been using k8s 1.14.1 both when it did and did not work? Any other version changes?
Not sure for before, when it used to work, but according to the asciinema recording mentioned above (), it seems the same. However, not sure for CRI-O, underlying OS stuff (systemd, kernel, etc.). Sorry.
I was looking for some information for days. I have exactly the same issue when I use podman + systemd on 18.04.2.


It is the same for both 14.15 and 14.18 kernels. Podman without systemd runs fine. Podman with systemd fails at runc with the same error podman[]: open /dev/stderr: no such device or address podman[]: NAME: runc - Open Container Initiative runtime podman[]: runc is a command line client for running applications packaged according to ... podman[]: time="2019-05-11T13:04:59-07:00" level=error msg="open /dev/stderr: no such device or addressn" podman[]: open /dev/stderr: no such device or address
I've tried to reproduce it on 18.04.2 using Podman from but I am not able to reproduce it. Do you have any reproducer? How did you install Podman?
I used ceph unit files to run ceph in podman rather than docker. Systemd unit successfully starts docker but fails with podman. I can successfully run the same podman container run command from the command line. How were you trying to replicate this? Can you post your unit file?
Hi, I'am having the same problem with Kubernetes with CRI-O. After joining a second worker node it fails to start flannel. gives the following error: What I've tried: Installing previous version of Kubernetes (from to ) Installing previous version of crio (from to ) Building runc myself from their github page and selecting this in the Running it on Ubunut 18.04.2 Issue: (the one giuseppe linked)
I wonder if the issue is with systemd and runc. In both cases, crio and podman, I see runc help output which tells me that something is not being passed to runc that it needs.
I've only used so it is different than your test and you confirmed it works from the CLI for you as well. Could you share your service file so I test it?
I've tried systemd and cgroupfs but no luck so far.
so it seems that runc fails to re-open . That seems like a runc issue, as it doesn't really need to reopen it. To fix the failure, you could use something like:
Just an example.


I use ceph-ansible to install cluster so it may need to install additional daemons like mon before running mgr but the error, in this case, should be different, [Unit] Description=Ceph Manager [Service] EnvironmentFile=-/etc/environment ExecStartPre=-/usr/bin/podman stop ceph-mgr-storage2n1-la ExecStartPre=-/usr/bin/podman rm ceph-mgr-storage2n1-la ExecStart=/usr/bin/podman run --rm --net=host  --memory=16G  --cpus=4  -v /var/lib/ceph:/var/lib/ceph:z  -v /etc/ceph:/etc/ceph:z  -v /var/run/ceph:/var/run/ceph:z  -v /etc/localtime:/etc/localtime:ro  -e CLUSTER=ceph  -e CEPHDAEMON=MGR  -e CONTAINERIMAGE=  {{ cephmgrdockerextraenv }}  --name=ceph-mgr-storage2n1-la  ExecStopPost=-/usr/bin/podman stop ceph-mgr-storage2n1-la Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi- delighted <:
BTW, I projectatomic ppa and used apt-get -y podman to install. My podman version is 1.3.0 Sasha Litvak — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
Corrected file.


[Unit] Description=Ceph Manager [Service] EnvironmentFile=-/etc/environment ExecStartPre=-/usr/bin/podman stop ceph-mgr-storage2n1-la ExecStartPre=-/usr/bin/podman rm ceph-mgr-storage2n1-la ExecStart=/usr/bin/podman run --rm --net=host  --memory=16G  --cpus=4  -v /var/lib/ceph:/var/lib/ceph:z  -v /etc/ceph:/etc/ceph:z  -v /var/run/ceph:/var/run/ceph:z  -v /etc/localtime:/etc/localtime:ro  -e CLUSTER=ceph  -e CEPHDAEMON=MGR  -e CONTAINERIMAGE=  --name=ceph-mgr-storage2n1-la  ExecStopPost=-/usr/bin/podman stop ceph-mgr-storage2n1-la Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi- Giuseppe Scrivano <:
Would it have an effect system wide? Can I add it to a unit file or unit override instead? Giuseppe Scrivano <:
yes, I think it is better to add it only to the unit file
Why would runc work outside of systemd? And if it needs stderr, giving it Null in settings will redirect stderr to Null I presume? On Tue, May 14, 2019, 9:46 AM Giuseppe Scrivano <:
Running into same issues on Ubuntu 18.04.2 using cri-o-1.13 via PPA.
PR here: opencontainers/runc
Until it is avaialble with project atomic and such, can I use a null workaround? Giuseppe Scrivano <:
yes, but you will lose anything written to stderr. You could probably also use a file, like
What about actually mounting dev and using /dev/stderr ? On Tue, May 14, 2019, 3:11 PM Giuseppe Scrivano <:
is a symlink to . It is present, the issue here is that is a socket. The fix for runc got merged, could you use an updated runc?
I tried apt update/upgrade and restarting crio, but I'm still getting the issue.


Could it be that crio's runc is not updated yet? Edit Build new runc from and it is working now. Think crio hasn't updated it yet with their repository.
I'm running this version from PPA repo for ubuntu and it fails: could you point to version/commit or whatever helps identify the fixed version/snapshot ? Thanks in advance.
at the moment it is only fixed upstream. You'd need to build runc manually
Is there a version of the package on ppa we can downgrade to? On Wed, May 15, 2019, 4:35 AM Giuseppe Scrivano <:
could we update the runc version in the PPA repo?
Thank you. I seem them built but nut pushed to ppa yet.
should be available now.. let me know..
cri-o-runc (1.0.0-rc6-1~dev~ubuntu18.04~ppa42) bionic; urgency=medium autobuilt -- Lokesh Mandvekar (Bot) <, 15 May 2019 16:54:40 +0000
Thank you, everyone, for a quick turn around. Quick testing showed that fix is working, but more testing is needed. Lokesh Mandvekar <:
I confirm my initial tests with seem to confirm the regression is fixed. Many thanks
I also confirm the fix working for podman. Olivier Berger <:


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): initContainers imagePullPolicy initContainers required value Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: cluster/kube- Others: What happened: When trying to create either a ReplicaSet or Replication Controller whose Pod template contains an Init Container, the validation fails saying is a required value. What you expected to happen: The create should succeed and the Init Container should receive a default . How to reproduce it (as minimally and precisely as possible): Try to create either of the following: ReplicaSet Replication Controller Anything else do we need to know: If I add to the Init Container in the ReplicaSet or Replication Controller Pod template, the create succeeds. When I create an Init Container without in a StatefulSet Pod template, it succeeds. When I create an Init Container without in a bare Pod, it succeeds:
Looks like defaulting not being applied on server? That is probably due to removing defaulting from convert. Surprised we didn't catch this earlier
Some values still not being defaulted
, I'm going to create a PR if none works on it.
We decode from bytes, set defaults and then convert; but for init containers, we delay the decode from "decode phase" to "convert phase" at .