<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet terminated unexpectedly during hack/local-up-cluster kubelet log as a result apiserver cannot find the node to schedule pods What you expected to happen: kubelet runs successfully and hack/local-up-cluster creates a single node cluster How to reproduce it (as minimally and precisely as possible): run hack/local-up-cluster from HEAD Anything else we need to know?: additional cgroups check seemed to be introduced in ; hack/local-up-cluster works when I check out the parent commit have CI for hack/local-up-cluster and , so I'm not sure if it's an error on my side. It would be greatly appreciated if someone could explain it to me Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Debian GNU/Linux Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node cc
Thanks for raising such a detailed issue! cc any thoughts here? Confusingly, I think the kubelet has been assigned a different cgroup for memory and for cpu... however the fact that its using makes me think the user is relying on system defaults. out of curiosity, have you made any mods to how your system assigns cgroups?
It sounds like we might want to handle the error more gracefully. If we can't get the kubelet's cgroup, we should fall back to collecting metrics from all cgroups.
collecting metrics from all the cgroups on the system? Any concerns about how many cgroups there could be? Also, or any interest in implementing this fix (assuming we go with the fallback behavior)? If not, I can try.
thanks for looking into this! Sorry I'm not familiar with cgroups. I'm don't know any modification to my system assigns cgroups in particular
I can put out a PR this week
xref
Thanks!


Some kubelet metrics use non-standard prometheus collectors unnecessarily. Specifically, we have which can be simple gauges. Since we cannot migrate non-standard prometheus collectors using the internal metrics stability framework, we will need to convert these metrics into standard prometheus variants. /sig instrumentation /assign
GitHub didn't allow me to assign the following users: solodov. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : Some kubelet metrics use non-standard prometheus collectors unnecessarily. Specifically, we have which can be simple gauges. Since we cannot migrate non-standard prometheus collectors using the internal metrics stability framework, we will need to convert these metrics into standard prometheus variants. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
recommends that we generate those metrics in the GenericPLEG during the relist. We can generate the data directly from the pods call there ().
I would be interested to work on this issue if no one else is working on it currently
Go for it!
/assign


What happened: tried promoting a test with a tag in its full test description. The hierarchy of ginko-relevant AST's was . The tag was a in the top level . The full test description only came from , so the tag was missed. What you expected to happen: The conformance test list generator should have logged and ERROR and failed a test on the PR, calling out Disruptive as an invalid tag How to reproduce it (as minimally and precisely as possible): Check out locally and follow the directions in to generate the conformance list. The invalid test is in the list. Anything else we need to know?: Is there a function we can use to get the full test description from Ginkgo instead of rolling our own? If not, we should probably be maintaining a stack rather than just the last FYI who ran into this FYI who wrote the code to keep track of last
/sig architecture /area conformance
/milestone v1.16
/priority important-soon
/assign
GitHub didn't allow me to assign the following users: devaips. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: attach/WaitForAttach timeout value is too small in CSI driver, it's all 15s which is really too small: Compared to built-in driver timeout, it's really too small: is , total is What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.13 - 1.15 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig storage /assign


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: cd $ $ make ../image- bin webhook ../image- line 145: GOARM: unbound variable Makefile:23: recipe for target 'bin' failed make: [bin] Error 1 What you expected to happen: The binary or image should be built successfully. How to reproduce it (as minimally and precisely as possible): see my desc above, type "make" will see the error message. Anything else we need to know?: Environment*: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): Ubuntu Kernel (e.g. ): all Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
I'm pretty sure this is supposed to be built via from which does default this environment variable.
, thanks for the information! I think the harness of the shell scripts could bring us a little more, GOARM should only been set when the ARCH is ARM Make a binary is possible where a Makefile is presented
That seems reasonable, but we should probably avoid computing this in many places, currently the GOARM=${GOARM:-7} is going to always be 7 when building in the currently supported way. We should change that if we're going to compute it elsewhere. cc-ing for test/images fun ...


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: login system root, and run below command from "test/images" dir, Will fail with below error message, ./image- line 91: : command not found This is cause by the empty string like "" cannot be understood as a shell command. What you expected to happen: image should be built successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ):all Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-bazel-test Which test(s) are failing: Test_buildClientCertificateManager Testgrid link: Reason for failure: Anything else we need to know:
/remove-kind failing-test /kind flake /sig scheduling
/sig node


What happened: The new TestScorePlugin for the scheduler framework is flaky How to reproduce it (as minimally and precisely as possible): An example: Anything else we need to know?: The test evaluates that a pod should be scheduled on specific node with a higher score. I think the test sometimes fails because the higher score node that the test creates didn't propagate yet to the scheduler (i.e., the node exists in etcd, but scheduler hasn't been notified that it exists yet, and so it is not evaluated at all). /kind bug /sig scheduling /assign
/kind flake


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Error msg override when saveVolumeData fails. What you expected to happen: return the right error. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage
/assign


What happened: When using a yaml file for input, silently fails at the first missing resource, not returning any further results. What you expected to happen: Searching with cli arguments properly ignores missing resources and continues the query, eg: Would expect similar results with a file input, eg: How to reproduce it (as minimally and precisely as possible): With an input file of : the command fails at apparently the first missing resource: Note: that rearranging the inputs in (so that the missing resource is last) produces the "expected" output: Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: local OS (e.g: ): fedora 30 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: x-ref:
/sig cli
/assign
Got around to reproducing it and can confirm that I'm getting the same behavior. I'm looking into fixing it now
thanks, any updates on this?
I've learned that in the case of a fake entry, kubectl stops sending API get requests altogether. Learned by running kubectl get -f --ignore-not-found --v=10, the result is shown below. Till now I've been looking into the cobra command framework but now I'm trying to find out where kubectl is making these API calls from(working on finding out where this connection is happening from Slack+debugging, I've asked a more detailed question here ). <img width="1440" alt="Screen Shot 2019-08-12 at 4 13 41 PM" src="https://user-"
that's an interesting find, definitely helpful. When I looked into this I got to the section of get's main run function, and it seems like it should be continuing on any errors (see ) but maybe the ResourceBuilder is actually stopping when it hits a 404?
This behavior comes from where the not found errors are being ignored and where the input list is flattened. Errors are ignored here: While flattening occurs here: As the visitor iterates through the objects in the list if an error occurs, such as not found, it will stop iterating over the objects in the list and return that not found error. The logic will then ignore it and continue. Any objects that came before the not found object in the list will have been visited and will then be included in the output. To make the behavior consistent between the two ways of getting the set of objects, the would need to be updated to support filtering errors.
Just submitted a PR adding that functionality to the FlattenListVisitor. The behavior is now consistent between the two methods.


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: CRI-O e2e Which test(s) are failing: e2e suite [sig-network] Services should be able to change the type from ExternalName to NodePort e2e suite [sig-network] Services should be able to create a functioning NodePort service 19s See link: https://openshift-gce- Since when has it been failing: Anything else we need to know:
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/milestone v1.16 /assign FYI Looks like two other release-blocking tests too: - The changes link testgrid gives () suggests two other PR's, but I agree that seems the likely culprit
Based on the , Service is reachable on and but failing to reach on node's IP over nodePort i.e. . From the log, it is not clear why its not reachable on node's internal IP. Could you provide from the CRI-O based node when trying to reach nodePort service. Secondly, need to make sure, node is listening on allocated nodePort. As this issue seems much on system-level, I need to setup k8s cluster/node with CRI-O runtime and troubleshoot to find the root cause.
[sig-network] ESIPP [Slow] [DisabledForLargeClusters] should work from pods [sig-network] Services should only allow access from service loadbalancer source ranges [Slow] seems flaky to me. Most of time execPod starts running and might be terminating immediately after some time which causes this issue. [1.] e2e is failing as might have terminated and so no PodIP leading to invalid . [2.] empty . The same issue had been observed in for E2E and to fix the issue, we introduced execPod deployment to make sure execPod would remain in desired running state. It's better to address this change in another PR and emphasize using pausePod deployment.
Thanks, for helping me setting up CRI-O based cluster. Kubernetes e2e suite [sig-network] Services should be able to change the type from ExternalName to NodePort e2e suite [sig-network] Services should be able to create a functioning NodePort service 19s From log, it is clear that, these services are failing to reach over allocated nodePort. But on my CRIO based testing cluster, they are reachable. or Could you please check if they failing at your end too or it's testgrid Job-specific issue?
Hey, I did some local tests and was able to reproduce the failure. I started the cluster via (check that the variable is correct and does not point to ): Executed the tests by (again, watch out for the variable to be correct): Error output: Since I'm now able to reproduce the issue I can go further in debugging and find out how to fix this. Is anyone furhter, or If not, then I will target for fixing the issue this week. :)
Thanks for reproducing this. FYI we've seem similar failure on , but I haven't reproduced the problem locally yet. We are looking to at least turn on verbose on the test command for debugging:
The issue is that we try to automatically discover the node IP by obtaining its status here: In our setup the node has IP which for sure does not work if we try to access it within the pod. I'm thinking about a proper solution and will propose it as a PR.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/assign


[x] write e2e tests for all admission operations [x] (for resource APIs) [x] (for discovery docs) [x] switch to v1 admission APIs in e2e - [x] remove skips/time.Sleeps - and promote to conformance tests - AdmissionWebhook should have tests for all stable operations. Those tests should go through the conformance process before being released. This ticket is to help track what operations are yet to be covered by conformance tests. These stable endpoints should be hit by e2e tests tagged with [conformance]. There are tests currently hitting 6 of the corresponding beta endpoints that could be updated to use stable. Tests should be written to exercise the remaining 9 untested stable endpoints as well. These twelve tests could be updated to use the stable API, and subsequently promoted to conformance. / ! ! These will need tests written for them.  untested beta/admissionregistration !
/sig api-machinery
/assign
/area conformance
/assign ^ This is a requirement for graduation to GA.
can we get another apisnoop run? We've coverage for admissionregistration and apiextensions and I'd like to get confirmation from apisnoop that we've got full coverage now.
Keeping this open until tests are actually promoted to conformance
is ready for review. It adds our admission e2e tests to conformance as part of graduation to GA.
is updated currently not hit by any tests, one will need to be updated or written to cover it. Might want to ensure you've promoted one test that hit's each endpoint. !
The PR to do that is at


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With kubectl v1.16.0-alpha.2, completion with override flags fails. then hit TAB will produce error "resource type must be specified" What you expected to happen: It must show the namespaces without error. How to reproduce it (as minimally and precisely as possible): With kubectl v1.16.0-alpha.2, then hit TAB key. Anything else we need to know?: It looks like introduced this regression. Environment: Kubernetes version (use ): kubectl v1.16.0-alpha.2 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
I also reproduced this bug with the following steps:
I'm fixing it.


<!-- Please only use this template for submitting enhancement requests --What would you like to be : We need to enhance Azure cloud provider code to support both AAD and ADFS authentication. Why is this needed: On Azure Stack customer can have two type of deployment .
/sig azure
/assign


What happened: In reviewing a test for promotion to conformance I noticed the kubelet API was being accessed via the API server's node proxy endpoint (ref: ). This seemed off, so I reviewed . It doesn't explicitly call out kubelet API access as allowed or disallowed. . and agreed kubelet API access seems like a bad idea. I'm now surveying what we have that does_ assume kubelet API access. It's definitely used by parts of e2e framework and some Conformance tests. I'm trying to verify how critical its use is. It appears that much of it is related to debugging or verifying whether kubelets and the control plane agree on who is running what. What you expected to happen: Find a blanket policy that bans kubelet API access for Conformance tests, whether directly, or indirectly through the API server's node proxy endpoint, at any time. Alternatively, find a policy that says kubelet API access may be allowed, but only for debugging or failure purposes (must not be on the critical path to make the test pass) Alternatively, find a policy that says kubelet API access is allowed, but only indirectly through the API server, and evaluated against the same constraints we use for control plane access (GA, non-optional) If we want to say kubelet API access is banned, we can use this as a tracking issue to demote whichever tests rely on it, and come up with a plan to verify the same behavior without kubelet API access. /area conformance /sig architecture /priority important-soon /milestone v1.16 /assign
Look for anything that ends up calling calls this in BeforeEach, which will happen before some Conformance tests (if this fails, the tests will fail) Conformance test requires kubelet API access to pass may end up calling it if the test fails, via: Some Conformance tests may end up calling this if the test fails specifically Other areas this is used: only if the test fails required for a pre-stop hook test to pass required to make sure the readonly kubelet apis aren't exposed via api server proxy resource usage tracking tests require kubelet api access to pass tests call this in
Looking for doesn't turn up any additional conformance tests, and I don't feel compelled to link the two other tests that use it as they are unlikely to ever be eligible.
The examples pointed out by on slack, the kubelet's endpoints is required to be available for these to pass
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 5 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?
are there further changes for this targeted at 1.16?
Any update on this?
/milestone v1.17
Bug Triage Release Team checking in. Code freeze for the 1.17 release is on November 14. Will this make it in? Thanks!
/assign
hi is this still targetting for 1.17 or can we move to 1.18? Code freeze is Nov 14th. thanks!
/milestone v1.18
So we've updated the documented policy. My suggestion to close this out is to drop the [Conformance] tag from the tests that currently require the kubelet API to pass, and open issues that they need to be rewritten. Ignore whether the API is needed for diagnostic info in failure modes. SchedulerPredicates [Serial] calls this in BeforeEach, which will happen before some Conformance tests (if this fails, the tests will fail) Conformance test Pods should be submitted and removed [NodeConformance] requires kubelet API access to pass
i'd support this ^^
Hello , Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March. Is this issue still intended for milestone 1.18?
Hello There's been no movement on this issue this year. Should this get pushed to the next milestone?
yes, please John T Skarbek <:
/milestone v1.19
/cc


This issue was reported in the Description The container manager used in kubelet checks for docker daemon process either via pidfile or process name. While the pidfile points to the docker daemon process PID, the dockerProcessName constant stores a docker cli name (docker) instead of docker daemon name (dockerd). Figure 30.1: Constants in file. Recommendation Correct the docker process name to dockerd. Anything else we need to know?: See for current status of all issues created from these findings. The vendor gave this issue an ID of TOB-K8S-025 and it was finding 31 of the report. The vendor considers this issue Informational Severity. To view the original finding, begin on page 77 of the Environment: Kubernetes version: 1.13.4
/sig node /kind bug /remove-kind feature
/lgtm
lol, this is the issue. my bad!


<!-- Please only use this template for submitting enhancement requests --What would you like to be : The scale client () currently works fine with namespaced resources. However, it doesn't work for non-namespaced resources e.g. cluster-scoped CRDs. The current implementation should be extended in order to support non-namespaced resources. The current interface: /area custom-resources /sig autoscaling
I think scale subresource support for CRDs falls more into sig-apimachinery domain. After DirectXMan12 moved on from sig-autoscaling I don't think there is anyone in our sig-autoscaling with knowledge of scale subresource internals. /sig api-machinery
/assign
The bug report sounds legit. Does the generic scale client need to be changed (I think so), or does solve your issue?
No, the dashboard PR doesn't solve this. I discovered this issue while working on that PR so when gets merged, it will only work for namespaced CRDs. I think the generic scale client needs to be updated, maybe by adding another getter to the ScalesGetter interface: /kind bug
PR:


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-enabled-default Which test(s) are failing: CSI hostpath-v0 tests Since when has it been failing: 8/8 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: We need to update client-go to get the fix for selfLink, build new nfs provisioner images, and update them in the e2e test. The fix for client-go is being cherry-picked to 1.15 if you want to use a tagged release:
/assign
/assign
This implies rebase to 1.15.x in external-storage, which seem to be challenging (at least to me :-) )
14 cherry pick also in progress:
OK, I guess this line in the lib is the culprit: So I need to update lib then external-storage to pull it in
I don't think the lib itself needs to be updated. Only updating go-client to get the fix to is needed.
Right, lib doesn't dictate what version of client-go to use. Will wait for client-go release/commit
- thanks a lot for bumping the dependency. Did you have a chance to build the new container with it so that we could update it here:
yeah, opened . (Not sure how to trigger test, or we can just merge and see.)
You can manually create a PVC on an alpha cluster to test.


<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add a sort parameter (or similar concept) to the command. Why is this needed: With the numerous amount of API resources (not counting custom ones), it is sometimes difficult locating them in the output. By default, API resources are not sorted by the name column (the first on the left) and often this is the column holding the information one wishes to query. A sort parameter would help more easily identify the object to be understood.
/sig usability


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-node-kubelet-orphans Which test(s) are failing: [sig-storage] EmptyDir volumes [It] pod should support shared volumes between containers [Conformance] Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know:
please don't CC sig-testing on every test failure :( we own the infrastructure etc. the sig-storage owns these tests.


What happened: There's a typo typo in and What you expected to happen: “depreciated” should be “deprecated”. How to reproduce it (as minimally and precisely as possible): View the release notes Anything else we need to know?: /lifecycle active I have a PR to fix this /sig docs Environment: Kubernetes version (use ): v1.16 Cloud provider or hardware configuration: not applicable OS (e.g: ): not applicable Kernel (e.g. ): not applicable Install tools: not applicable


What would you like to be : Currently the apiserver has request latencies metrics, which are a histogram of requests to the apiserver. However, these request metrics do not include healthz. While we have recently the ability to get back health check metrics from the healthz endpoint directly, we should also emit this data as a metric, so that we can monitor health check latencies. /assign /sig api-machinery instrumentation
GitHub didn't allow me to assign the following users: jktomer. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : What would you like to be : /sig api-machinery instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I noticed this is informally assigned. Has there been any progress? Is this a place I can fit in?
Yes, I'm working on a PR.
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


What happened: Provisioning of a volume with failed and this event were created on a pod: The event does not say anything useful and only confuses users. What you expected to happen: Either no event (event on PVC about failed provisioning could be enough) or something generic about "waiting for provisioning to finish". How to reproduce it (as minimally and precisely as possible): Tested on AWS. It should be the same with GCE and other block-based in-tree volumes. StorageClass with PVC requesting RWX (so provisioning fails): pod that uses the PVC. Pod events: Anything else we need to know?: I've been able to track down what happens: Scheduler adds to the PVC Provisioner tries to create a volume for the PVC and fails. The annotation is removed in  The scheduler expects that the annotation exists here: I am not sure how to fix it though. Ignoring the error might be the easiest thing to do. Environment: Kubernetes version (use ): today's master /sig storage /sig scheduling /assign
yes, the error message is confusing. "FailedScheduling" event will be emitted if the scheduler failed to schedule the pod. How about wrapping the internal error message in function, e.g. ? Users will know the pod failed because pod volumes not bound successfully. Sometimes, users can't just wait for the scheduler to recover from the error state, manual intervention might be required. The internal reason will be useful. In this scenario, .
's suggestion sounds good to me


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The kubelet metrics related to , except for one, are missing from the kubelet metric endpoint. This applies to these metrics: What you expected to happen: I expect that the metrics mentioned above should be found in the endpoint on the kubelet. How to reproduce it (as minimally and precisely as possible): Run kubelet with a valid , and (defaults to true). or run the node e2e tests: Anything else we need to know?: All e2e tests related to DynamicKubeletConfig fails in master (and in v1.16): This bug/error was introduced in Environment: Kubernetes version (use ): && Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node /cc logicalhan tallclair mtaufen
/cc
/assign WIP proof of concept solution to fix (make it a bit easier to understand the problem) here:
/assign
edit: I misunderstood the issue. discussed with offline, and he has an idea to restore previous behavior
That sounds perfect! He answered in the PR, so hopefully we can get this working again. I have wrapped my head around this issue for quite a few hours now, so I understand that it is kinda hard to understand by just reading this issue. Here is some more background info that hopefully makes sense.
/priority important-soon /milestone v1.17
Looks like the CI jobs are cleaning up now! ! Finally down to single digit number test failures, and away from the 30+ :smile:


<!-- Please only use this template for submitting enhancement requests --What would you like to be : Change scheduler metric from Gauge to Histogram Why is this needed: is recorded for each preemption attempt. A histogram is better. /sig scheduling /help /good-first-issue
/help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can work on this issue.
/assign
I could work on this as well if doesn't have time to work on this.
I am working on it, I will create the initial PR in less than a week. Sorry for the delay.


Currently custom predicate and priorities can be created through the . The problem is that a custom predicate/priority, like ServiceAffinity, can be multiple times in the Policy with different names, possibly with different arguments. This semantic (i.e., specifying multiple instances of the same custom predicate with different arguments) is only useful if different s were specified because in all cases of custom predicates/priorities the arguments are merge-able (for example, from multiple CheckServiceAffinity custom predicates can be merged into one set). In the world of framework plugins, we don't currently have a way to maintain those semantics. to run. The question here: do we really want to maintain the old semantics of custom predicates and priorities? in other words, do we need to change the ComponentConfig API to somehow allow specifying multiple instances of the same Plugin with different arguments/weights, or should we consider that this is not supported? I always lean towards simplifying things, so my suggestion is to not support that, but I would like to know if there are actual deployments that rely on this. /sig scheduling /cc
/cc
We discussed this in the sig meeting, and put the issue for discussion on the scheduler and kubernetes-dev mailing list, I believe this behavior is not intentional, and so we should add logic to fail the scheduler configuration when a policy includes more than one custom predicate/priority of the same type. /assign
/assign
any progress on this one?
I haven't made any progress on this yet, but it's on my priority now
In migrating custom predicates to filter plugins , I am gonna ignore the custom predicate name and use the default name.


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: cloud-provider-openstack jobs Which test(s) are failing: hack/local-up- itself fails. Since when has it been failing: Recent change , checks for node ready by doing a grep of KUBELETHOST. But when HOSTNAMEOVERRIDE is set , its unable to find the ip and exits. Testgrid link: Reason for failure: Anything else we need to know:
Repalce KUBELETHOST with HOSTNAMEOVERRIDE shoud fix this issue, I have created a pull request
Thanks. But what if HOSTNAME_OVERRIDE is not set , then again it would fail? I guess its optional parameter
cc
If not set, the default value is 127.0.0.1, won't fail
Hi ramineni, nodpad is correct. HOSTNAME_OVERRIDE do have default value which is 127.0.0.1.
Thanks. Then grep HOSTNAME_OVERRIDE is ideal solution. That would work for me. Hopefully PR gets merged soon


What would you like to be : A unit test that verifies the evaluation order of the nodes in the scheduler. Why is this needed: The scheduler evaluates the nodes in a specific order until it finds a pre-defined minimum number of feasible nodes. To give all nodes a fair chance of being evaluated, in each scheduling cycle, the scheduler starts evaluating from where it left off in the previous cycle. This is captured by the parameter in  We need to add a unit test that captures regressions related to this behavior to avoid bugs like /sig scheduling /priority important-soon /help
/assign
/remove-help


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: EndpointSlice for masters was not created after enabling EndpointSlice on a pre-existing cluster. Only those EndpointSlices managed by kube-controller-manager were created. What you expected to happen: EndpointSlice for masters should be created after enabling EndpointSlice on a pre-existing cluster, the same as the EndpointSlices managed by kube-controller-manager. How to reproduce it (as minimally and precisely as possible): kubeadm to create a cluster: the EndpointSlice feature by editing the kube-apiserver, kube-controller-manager, kube-proxy manifest kube-apiserver and kube-controller-manager is restarted, the EndpointSlice in namespace will be missing. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools:Network plugin and version (if this is a network-related bug): Others:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved /assign


What happened: Housekeeping sync loop finds an orphaned pod and tries to clean it up. The kubepod slice, as seen below, is not removed by with no indication as to why. This leads to the docker scope timing out and entering a failed state. The returned error(s) from runc are swallowed making debugging difficult. What you expected to happen: As per other areas of the sync loop, useful errors are logged at debug level due to the async calls. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: N/A Environment: Kubernetes version (use ): master
/sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Have you seen the comment above pcm.Destroy() call ? Did you have a chance to look at kubelet log to see if the above was true ?
/remove-lifecycle stale
This would give us some clue about the error:
yes, definitely looked! In this specific situation, the pods are never retried. What we see happen with these pods: ADD -RECONCILE -Started Volumes -Created slice -No sandbox for pod -Started libcontainer -ContainerStarted -ContainerStarted -RECONSILE -ContainerDied -RECONCILE -Unmounts -Killing unwanted pod -Orphaned removing pod cgroups -container kill failed because of 'container not found' or 'no such process' -cleanup: failed to unmount secrets -ContainerDied -Container garbage collection succeeded -SyncLoop (SYNC): 141 pods (including this one) -stopping timed out. Killing. -Stopped libcontainer -Removed slice -DELETE -REMOVE -err: pod not found Destroy all cgroups of the pod that should not be running, by first killing all the attached processes to these cgroups. We ignore errors thrown by the method, as the housekeeping loop would again try to delete these unwanted pod cgroups Despite running, the slice is not removed and we do not log or indicate why. Subsequently, the pod is NOT determined to be either an active pod or for some reason not in the cgroup list that is passed into kl.cleanupOrphanedPodCgroups(cgroupPods, activePods) during housekeeping loops. A more detailed look, which I believe is accurate ;): housekeepingCh -HandlePodCleanups -cgroupPods=UID+CGroupName=GetAllPodsFromCgroups -allPods=GetPodsAndMirrorPods -activePods=filterOutTerminatedPods -cleanupOrphanedPodCgroups I do not think desiredPods and runningPods are relevant here.. So either: a) the pod is not found and to allPods (seems unlikely) but we can't know for sure.. b) the pod is incorrectly determined to be active via activePods which again we can not determine.. Any pod that is not Failed, Succeeded or has a DeletionTimestamp set and a status of Terminated or Waiting - anything else returns false and is deemed active c) the pod cgroup is not found in m.subsystems.MountPoints and to cgroupPods - unlikely, but could be a bug but the root of this issue is the fact that all of the scopes and slices exist and keep building up until things start failing... GetAllPodsFromCgroups could use some logging... Maybe a V 5 log that prints out foundPods at the end... From there, they're passed into the cleanup: kl.cleanupOrphanedPodCgroups(cgroupPods, activePods) 1) All activePod UIDs are put into a podSet 2) A range loop of uid and cgroup name from cgroupPods looks to see if the uid from cgroupPods also exists in the podSet (active pods) 3) If the uid from cgroupPods exists in the podSet we skip out of the loop 4) If it doesn't, the pod is orphaned (not active w/ cgroups on the filesystem) and we call pcm.Destroy(val) on the cgroup name. Doesn't make sense why pcm.Destroy(val) ignores errors... it would be helpful! tryKillingCgroupProcesses logs any error and also returns the an error which is always swallowed.. we don't see any issues here More problematic, the call to cgroupManager.Destroy only returns an error which would be swallowed... Could probably use a similar level 3 log here like what you have for tryKillingCgroupProcesses because as you can see, were totally blind here...
your change is of a similar nature to what I proposed in my PR for this issue, just a level higher. I think there are a few areas noted above the could help further debug this.. Still haven't been able to get my PR merged with additional logging.
The following code was relatively new (compared to when you logged this issue):
Indeed... I looked into it a while ago and those changes are new via: - Could definitely make a different if the issue is related to pod getting filtered out from activePods when they have not been fully cleaned up yet on the subsequent housekeeping syncloop runs.
I think is promising, but I still think it makes sense to add some logging here. Do you want want to proceed with my open PR, your suggestion in or both?
You can proceed with your PR.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: It returns an error when I create a pod of yaml file over 1M. I checked the codes and found that the coded wrong. It should be “json.Unmarshal(p.patchBytes, &v)”, not “json.Unmarshal(p.patchBytes, v)” in the 344 and 368 line. What you expected to happen: It can be parsed correctly. How to reproduce it (as minimally and precisely as possible): Look at the code and you'll see or you can test it. Anything else we need to know?: no. Environment: Kubernetes version (use ): Community latest code Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind bug
/sig api-machinery
/assign


What happened: While testing out the flag, I created a manifest with the following contents: I ran , and the resources appeared on my minikube cluster: Then, I removed the deployment from the manifest: And re-ran the same command, . To my surprise, however, the deployment wasn't removed; the only output from kubectl was . What you expected to happen: I expected the pruner to find the deployment with the matching label, , and prune it. How to reproduce it (as minimally and precisely as possible): I believe my example above is minimal and easily reproducible. Anything else we need to know?: This small change to the code for resolves the issue: Environment: Kubernetes version (use ): v1.16.3 Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig cli
/assign cc


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When trying to use the webhookConversionServer, im getting . I was able to figure out that the unicode character displayed maps to port 443. see code What you expected to happen: If the service reference doesnt define 443, it should show an error that the service doesnt have the port number (whatever the number may be). How to reproduce it (as minimally and precisely as possible): your service (missing the default port 443) your CRD (one version no conversion webhook) your resource at this old version crd with new version and conversion webhook, and use new version for storage. to get the resource yaml, and see error Anything else we need to know?: current implementation will convert port number into a unicode character. ie. im seeing where . Must use instead of to maintain decimal. Environment: Kubernetes version (use ): v1.15.4+IKS Cloud provider or hardware configuration: IBM IKS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
Are you planning to send out a PR ?
i was going to.. but didnt know which repo was relevant and didnt feel like reading through all contibution gorp.
Created a PR to fix the message output.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider is sending unexpected LoadBalancer update requests even if the LoadBalancer rules are already identical. The reason is GetIdleTimeout() returns nil when the annotation is not set While find rule doens't check whether it is nil or not. What you expected to happen: Azure cloud provider shouldn't send unexpected LoadBalancer updates. It is easily to get throttled if there're too many services running on the cluster. How to reproduce it (as minimally and precisely as possible): kubectl annotate service nginx testlabel=value Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /area provider/azure /assign
/sig cloud-provider


If you give a prerelease version to , it'll match both and . Based on this logic: We'll end up trying to pull binaries for this prerelease from the CI bucket () rather than the release bucket (). These pre-release versions are usually* pushed to the release bucket as part of the release process (e.g. ), and so should probably prefer these over the CI versions (which may or may not exist, depending on whether a commit was pushed to the release branch between when the tag was created and the build occurred). More explicitly, the current logic is: the release bucket by default. a version matches the CI regex, use the CI bucket. , if a version matches the release regex, we're good - continue to use the default (release bucket). , bail out - invalid version. The correct logic would probably be something like the release bucket by default. a version matches the release version regex, continue to use the release bucket. , if a version doesn't match the release version regex, but it matches the CI regex, use the CI bucket. it matches neither, bail out. Important note: due to our hotfix for , there is a somewhat fake prerelease tag () which doesn't have any bits in the release bucket, and we expect others soon. We should not fix this bug in until the latest pointers in the CI bucket do not point to valid release/prerelease versions.
/sig testing /sig release I guess?
/area release-eng /priority critical-urgent /assign /milestone v1.18
Hello With code freeze approaching for 1.18 (5 March), can you please check if the current issue is still intended for milestone 1.18?
-- I'm pretty sure I can get to this before the cycle is out.
if our current approach for tagging release branch pre-releases doesn't trigger this, this seems like it is not critical for the milestone, is that correct?
-- Good point! /milestone v1.19 /remove-priority critical-urgent /priority important-soon
Given the priority dip on this one, I think it's fine for someone in Releng other than Tim or I to pick this up. /unassign -- This is a good one to work on. /help
I can look that, after the release 1.18, if is that fine then can we have a short discussion about this one? /remove-help /assign
-- SGTM


What would you like to be : There are many useful scripts in "". For example, I often use and for creating simple k8s cluster quickly. However, there are only few explanation or nothing how to use it. I think it becomes more useful and helpful if some instruction or guidance is explained in each script. Why is this needed: It aims and expects to increase new k8s developers by describing how to use each script. My proposal Add the following description under the copyright for each script file. Overview of the script Simple usage Supplementary explanation(This is option) Other concern I would like to know the purpose of use about hack directory. (e.g. Where to put tools for developers(therefore there is no guidance?).)
I don't know who is in charge of each scripts of hack, but if he agrees the idea of this feature request , I am going to do this task myself!!
/sig testing
I believe there is a script generating swagger/openapi specfilters under hack, but I am still struggling with it. It could be of great help if there is some how-to doc for each scripts.
this basis, why not create a file under the hack folder, because a lot of kk CIs are derived from multiple script commands under the hack. There are already 268+ files under the hack file. For contributors unfamiliar with these, these comments and instructions are helpful to them, and it is also helpful for new contributors to handle pr ci error reports.
/cc What do you think
I think it's good idea to create (It may be easy to understand if the script's explanation is described by category or use in ) Apart from that, it is helpful that the script file itself has a brief description.
When we create , it may tough work if creating it including all scripts explanation at one time. (There are many scripts and I understand only some scripts now.) Regarding , current my idea is to add a description to each script little by little and update the at the same time. This means that will be completed little by little in conjunction with adding the description of each script. In the future, if someone create new script in hack, he also may have to add description of his script into
Self-documenting commands sound useful. A high level overview also sounds useful.
/cc /cc Regarding adding help information, I suggest that we can submit pr in one or several files as a unit, so that we can track and statistics. For the documentation, I think it should be concise and not too long. How about this advice? We can claim tasks through , the pr title yes , i agree with you. [x] kubernetes/hack/benchmark- [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] kubernetes/hack/boilerplate/BUILD [ ] kubernetes/hack/boilerplate/test/BUILD [ ] [ ] [ ] [ ] [ ] kubernetes/hack/BUILD [x] kubernetes/hack/build- [x] kubernetes/hack/build- [x] [ ] kubernetes/hack/conformance/BUILD [ ] [x] kubernetes/hack/dev-build-and- [x] kubernetes/hack/dev-build-and- [x] kubernetes/hack/dev-push- [ ] kubernetes/hack/e2e-internal/BUILD [ ] kubernetes/hack/e2e-internal/e2e-cluster- [ ] kubernetes/hack/e2e-internal/e2e- [ ] kubernetes/hack/e2e-internal/e2e-grow- [ ] kubernetes/hack/e2e-internal/e2e-shrink- [ ] kubernetes/hack/e2e-internal/e2e- [ ] kubernetes/hack/e2e-internal/e2e- [x] kubernetes/hack/e2e-node- [ ] kubernetes/hack/generate- [x] kubernetes/hack/generate- [ ] kubernetes/hack/gen-swagger-doc/gen-swagger- [x] kubernetes/hack/get- [x] kubernetes/hack/ginkgo- [x] kubernetes/hack/grab- [x] kubernetes/hack/install- [ ] kubernetes/hack/jenkins/benchmark- [ ] kubernetes/hack/jenkins/OWNERS [ ] [ ] kubernetes/hack/jenkins/test- [ ] kubernetes/hack/jenkins/verify- [ ] kubernetes/hack/lib/BUILD [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [x] kubernetes/hack/lint- [x] kubernetes/hack/list-feature- [x] kubernetes/hack/local-up- [ ] kubernetes/hack/make-rules/BUILD [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make-rules/helpers/go2make/BUILD [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with- [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with- [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/skipme/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/skipme/subsubdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/skipme/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/subdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/testdata/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/testdata/subsubdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/testdata/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/underscore/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/underscore/subsubdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/underscore/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-without-gofiles/README [ ] kubernetes/hack/make-rules/Makefile.manifest [ ] kubernetes/hack/make-rules/make- [ ] kubernetes/hack/make-rules/test- [ ] kubernetes/hack/make-rules/test-e2e- [ ] kubernetes/hack/make-rules/test-e2e- [x] kubernetes/hack/make-rules/test- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/OWNERS [x] kubernetes/hack/pin- [x] kubernetes/hack/print-workspace- [x] kubernetes/hack/run-in- [ ] kubernetes/hack/testdata/auth- [x] [x] [x] kubernetes/hack/testdata/CRD/foo-- [x] kubernetes/hack/testdata/CRD/foo-deleted- [x] kubernetes/hack/testdata/CRD/foo-updated- [x] [x] kubernetes/hack/testdata/CRD/multi-crd-list-- [x] kubernetes/hack/testdata/CRD/multi-crd-list-deleted- [x] kubernetes/hack/testdata/CRD/multi-crd-list-updated- [x] kubernetes/hack/testdata/CRD/multi-crd- [x] [x] [x] kubernetes/hack/testdata/deployment-label- [x] kubernetes/hack/testdata/deployment-label- [x] kubernetes/hack/testdata/deployment-label- [x] kubernetes/hack/testdata/deployment-multicontainer- [x] kubernetes/hack/testdata/deployment- [x] kubernetes/hack/testdata/deployment- [x] kubernetes/hack/testdata/deployment- [x] kubernetes/hack/testdata/deployment-with- [x] [x] [x] [x] [x] [x] kubernetes/hack/testdata/filter/pod-apply- [x] kubernetes/hack/testdata/filter/pod-dont- [x] kubernetes/hack/testdata/frontend- [x] kubernetes/hack/testdata/frontend- [x] kubernetes/hack/testdata/invalid-rc-with-empty- [x] kubernetes/hack/testdata/kubernetes- [x] [x] [x] [x] [x] [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-json- [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-list- [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-rclist- [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-svclist- [x] kubernetes/hack/testdata/multi-resource-yaml- [x] kubernetes/hack/testdata/multi-resource- [x] kubernetes/hack/testdata/null-propagation/deployment- [x] kubernetes/hack/testdata/null-propagation/deployment- [x] kubernetes/hack/testdata/OWNERS [x] kubernetes/hack/testdata/pod- [x] kubernetes/hack/testdata/pod- [x] kubernetes/hack/testdata/pod-with-api- [x] kubernetes/hack/testdata/pod-with-large- [x] kubernetes/hack/testdata/pod-with- [x] [x] [x] [x] [x] kubernetes/hack/testdata/prune- [x] kubernetes/hack/testdata/prune- [x] kubernetes/hack/testdata/recursive/deployment/deployment/nginx- [x] [x] [x] [x] [x] kubernetes/hack/testdata/recursive/pod/pod/busybox- [x] [x] [x] kubernetes/hack/testdata/recursive/pod- [x] kubernetes/hack/testdata/recursive/pod-modify/pod/busybox- [x] kubernetes/hack/testdata/recursive/pod- [x] kubernetes/hack/testdata/recursive/pod- [x] [x] kubernetes/hack/testdata/recursive/rc/rc/busybox- [x] [x] [x] kubernetes/hack/testdata/redis-slave- [x] kubernetes/hack/testdata/retainKeys/deployment/deployment- [x] kubernetes/hack/testdata/retainKeys/deployment/deployment- [x] kubernetes/hack/testdata/rollingupdate-daemonset- [x] kubernetes/hack/testdata/rollingupdate- [x] kubernetes/hack/testdata/rollingupdate-statefulset- [x] kubernetes/hack/testdata/rollingupdate- [x] kubernetes/hack/testdata/scale-deploy-1.yaml [x] kubernetes/hack/testdata/scale-deploy-2.yaml [x] kubernetes/hack/testdata/scale-deploy-3.yaml [x] [x] kubernetes/hack/testdata/service- [x] kubernetes/hack/testdata/service- [x] kubernetes/hack/testdata/sorted-pods/sorted- [x] kubernetes/hack/testdata/sorted-pods/sorted- [x] kubernetes/hack/testdata/sorted-pods/sorted- [ ] [ ] [x] kubernetes/hack/test- [ ] kubernetes/hack/test- [x] kubernetes/hack/update- [ ] kubernetes/hack/update- [x] kubernetes/hack/update- [ ] kubernetes/hack/update-generated-api-compatibility- [x] kubernetes/hack/update-generated-device-plugin- [ ] kubernetes/hack/update-generated-device- [ ] kubernetes/hack/update-generated- [ ] kubernetes/hack/update-generated-kms- [ ] kubernetes/hack/update-generated- [x] kubernetes/hack/update-generated-kubelet-plugin-registration- [ ] kubernetes/hack/update-generated-kubelet-plugin- [x] kubernetes/hack/update-generated-pod-resources- [ ] kubernetes/hack/update-generated-pod- [ ] kubernetes/hack/update-generated-protobuf- [ ] kubernetes/hack/update-generated- [ ] kubernetes/hack/update-generated-runtime- [ ] kubernetes/hack/update-generated- [ ] kubernetes/hack/update-generated-swagger- [x] kubernetes/hack/update- [x] kubernetes/hack/update-import- [ ] kubernetes/hack/update-openapi- [x] kubernetes/hack/update- [ ] kubernetes/hack/update-vendor- [ ] kubernetes/hack/update- [ ] kubernetes/hack/update-workspace- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-api- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-cli- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-conformance- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-external-dependencies- [ ] kubernetes/hack/verify-flags/excluded- [x] kubernetes/hack/verify-flags/OWNERS [x] kubernetes/hack/OWNERS [ ] kubernetes/hack/verify-flags- [x] kubernetes/hack/verify-generated-device- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated-files- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated-kubelet-plugin- [x] kubernetes/hack/verify-generated-pod- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated-swagger- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-import- [x] kubernetes/hack/verify-import- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-no-vendor- [x] kubernetes/hack/verify-openapi- [x] kubernetes/hack/verify-pkg- [x] kubernetes/hack/verify-publishing- [x] kubernetes/hack/verify-readonly- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-staging-meta- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-test- [x] kubernetes/hack/verify-test- [x] kubernetes/hack/verify-test- [x] kubernetes/hack/verify-typecheck- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-vendor- [x] kubernetes/hack/verify- [x]
i will pick up , i will do it step by step.
I agree with your idea, and thanks the checkbox to check progress. there are many scripts, so let's work together!
/assign /assign
I will pick up hack/verify-
I will pick up hack/verify-
I will pick up hack/verify- (all of verify scripts). But there are large number of scripts(over 40), I want to split it into two or three PRs.
I will pick up hack/update-
I will pick up these scripts. kubernetes/hack/benchmark- kubernetes/hack/build- kubernetes/hack/build- kubernetes/hack/dev-build-and- kubernetes/hack/dev-build-and- kubernetes/hack/dev-push- kubernetes/hack/e2e-node- kubernetes/hack/generate-
I will pick up these scripts get- ginkgo- grab- install- lint- list-feature- local-up- pin- print-workspace- run-in- test- test-
How about this now? /help Marking this tag so that others can easily access it.
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : How about this now? /help Marking this tag so that others can easily access it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think we've finished adding simple explanations to the almost scripts. It seems that several scripts of update- does not complete yet. I think we may able to close after that work.
I will pick up these scripts:
/assign


In a few utility function in current test/e2e/storage/utils, we found that the err variable gets shadowed. Because of this, it might get nil pointer error. See code
fixed by


What would you like to be : An integration test for NodeResourceLimits plugin. Why is this needed: No end-to-end test coverage current exist for this alpha feature. /sig scheduling /help
/assign /remove-help


When trying to see why pods don't start using either kubectl get events or kubectl describe pod , the base message is not detailed enough. Warning FailedScheduling 5s (x4 over 65s) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate. What would you like to be : Details of the taint and details of what it didn't tolerate Why is this needed: To get much much better debug information by default when trying to understand why pods can't launch.
/sig cli
/sig scheduling /priority important-longterm
/cc
/remove-sig cli
Those labels are not set on the issue: <detailsIn response to : /remove-sig cli </details
/good-first-issue
/assign starizard
Can I get an idea of what this would involve? I might be able to put some work on it.
I assume is taking this already. The error message is produced in We probably also require changes in to get the name of the missing toleration.
/remove-help


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I am testing API Priority and Fairness. I launched it like this: After the logging said the cluster is running and told me how to set KUBECONFIG, I did so in another shell and went looking for FlowSchema objects. But kubectl erroneously told me there is no such thing. However, looking in the kube-apiserver log shows flowschemas and prioritylevelconfigurations being read and written; for example: Also, I am able to read them with against the insecure port. Interestingly, the kube-apiserver log also includes this complaint: I will attach the kube-apiserver log. What you expected to happen: I expected to list them. How to reproduce it (as minimally and precisely as possible): I think the report above is pretty minimal. Anything else we need to know?: Environment: Kubernetes version (use ): this is the branch, git commit 916edd922e528f2f27542d3ca5967ca11fe153f8Cloud provider or hardware configuration: bare metal, intel, 16 "CPUs", 64 GB of memory OS (e.g: ): 18.04.2 LTS (Bionic Beaver) Kernel (e.g. ): Linux mjs-dev-1 4.15.0-74-generic -Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: hack/local-up- Network plugin and version (if this is a network-related bug): Others: Docker version 18.09.7 go version go1.13.4 linux/amd64
/sig api-machinery
/cc /cc /cc /cc
/assign
the "" API group is not included in the list of known groups to register at startup: That means resources inside that group will not appear in discovery, and clients like kubectl will not know how to find them.
You can see all resources visible via discovery by running
just as said, for now we have to manually create an apiservice into the cluster to get kubectl work. i dont think we're adding this to the list until the beta stage, can we make an exception for the case?
It is fine to add alpha things to the list I linked so they get created automatically if needed. They only get registered if the relevant runtime-config is enabled.
Interrupts control plane upgrade in lazy scenario where it's assumed dns name isn't changed properly to active node or or single node control plane
oh cool actually it succeeds on the second main node but the kube apiserver pod doesnt start
the hacky way being to adjust that api creation based on this?
And I removed the flags too
Wait what is this? This was to my api manifest during kubeadm upgrade?


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With the exec credentials plugin configured in my kubeconf, and a stale/expired token present in the same file issuing e.g. triggers the authentication flow as handled by the plugin - once the flow is returned to kubectl (by printing the ExecCredential object) the new token is stored to the kubeconfig but not immediately used to resolve the request, leading to: Immediately issuing again works as kubectl now seem to use the credentials (token) stored following the first request. What you expected to happen: That kubectl would read the new token as fetched by the exec credential plugin and use it already for the first command issued. How to reproduce it (as minimally and precisely as possible): the kubernetes API server with the OIDC authenticator enabled. kubectl to use an exec credential plugin printing the token received out of band. that the token was indeed written to the kubeconf file. that the first kubectl command failed. that the next identical command succeeds. Anything else we need to know?: This works as expected when there is no token previously stored in the kubeconf - the behavior is only triggered when there is a token stored but it expired, or for whatever other reason not accepted by the API server. Environment: Kubernetes version (use ): 1.14, also tried with 1.17 Cloud provider or hardware configuration: OS (e.g: ): Mac OS for kubectl client Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig auth /sig cli
could you provide more details around how is configured. For example, it is unclear to me if you are using the built in oidc plugin or a custom exec plugin. I think you are using a custom exec plugin but its also trying to write tokens to the file (which it should not do)? /triage needs-information
Thanks for looking into it . You are indeed correct in your assumption that I'm using a which stores the token in the after retrieval. I'd love to learn why this should not be done, but the problem remains also without persisting the token after retrieval. Basically the steps to reproduce could be reduced to these: containing an expired token. same configuring an exec plugin which retrieves a valid token out of band and echoes it. Expected behavior: The valid token retrieved is used for the subsequent authentication. Possibly expected behavior: The exec plugin flow not started (or exited with a warning) due to an expired token being present in the . Current behavior seems to acknowledge the invalid token and hence tries to retrieve a new, valid one - which it then discards as soon as it arrives.
Seems we can reduce it even further by removing the token altogether from the file and instead provide it directly. So the only step needed to reproduce is essentially this: configuring an exec plugin which retrieves a valid token out of band and echoes it as part of an . Now any will trigger the exec plugin to retrieve a valid token, but will still not use it for subsequent authentication.
This one is caused by config merging with clientcmd overrides: and exec.Auth not being in this switch:
Interesting, thanks Will add it to the switch along with some tests and if everything works out well I'll submit a PR next week.
Having spent some time on this in the debugger, it seems somewhat more intricate than I first thought. Turns out the exec plugin is executed whether the presented is invalid or valid - the exec action is actually called before the first call is even dispatched. This due to the exec credential plugin possibly being needed to setup the client certificate requested for the transport. Once the exec plugin's is called it assign the authenticators function to : ..which is then called when setting up the transport, thus triggering the exec plugin regardless of a token being present or not: Since the exec plugin has no way of knowing whether a token or a client certificate will be returned by the exec action, this kind of makes sense. It however defeats letting any present to override the exec plugin, which clearly seems to be the intent: Possible options here that I can think of: (and fix) behavior of letting override the exec plugin. This looks doable by adding a check for token auth early in the 's method body: But might be there are consequences to this I'm not aware of. the behavior of letting override the exec plugin being called. third, more complex option, would be to keep the token override but dispatch to the exec credential plugin should the response from the first call with the token come back with a 401. This was what I originally believed was the intent as my first tests always included an invalid token. While this behavior seems nice I'm more and more thinking it may violate the principle of least surprise. Either you present a token and use that for auth or you let the exec plugin fetch the credentials for you. What do you guys think?
As a colleague just made me aware of, certain tools like kubetail seem to have kubectl call the exec plugin so many times that eventually the threshold of 1000 is reached and warn logs (exec:201) start spewing - even though no certificate (and hence no rotation) should be involved. This has me more convinced that the first option is the right one, skipping the client TLS ceremonies should a token be present for authentication.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


What happened: Used kube- to bring up a GCE cluster and then logged on to the master. Run kubectl on the master, is hangs for a while (30 seconds) and then returns an error. $ kubectl get nodes Error from server (InternalError): an error on the server ("") has prevented the request from succeeding What you expected to happen: I expected to get the node list. If I use one of the system account's kubeconfig files it works. $ sudo kubectl --kubeconfig=/mnt/disks/master-pd/srv/kubernetes/kube-bootstrap/kubeconfig get nodes NAME STATUS ROLES AGE VERSION kubernetes-master Ready,SchedulingDisabled <none26m v1.18.0-alpha.2.55+-dirty kubernetes-minion-group-jfk6 Ready <none26m v1.18.0-alpha.2.55+-dirty kubernetes-minion-group-mqnp Ready <none26m v1.18.0-alpha.2.55+-dirty kubernetes-minion-group-x540 Ready <none26m v1.18.0-alpha.2.55+-dirty How to reproduce it (as minimally and precisely as possible): Bring up a GCE cluster from head, ssh to master and run "kubectl get nodes" Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.0-alpha.2.55+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2020-01-23T00:38:39Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.0-alpha.2.55+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2020-01-23T00:36:29Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE OS (e.g: ): COS m_77 Kernel (e.g. ): 4.19.76 Install tools: cluster/kube- Network plugin and version (if this is a network-related bug): Others: I believe this used to work based on talking to KAS on the insecure port using no authentication. The insecure port is now turned off and we require authentication materials. We should probably setup an admin account specifically for the master.
/sig cli
/assign
I also would not have expected a hang, but an immediate error, and I would have expected an authentication error, not an internal error
I believe the internal error is due to it defaulting to port 8080. Nothing is listening on that port. This used to work but I would guess it stopped working when we stopped listening on the insecure port.
changed behavior of client-go on connection refused errors, which changed a short wait and an "invalid host/port" error into a long retry wait and a confusing "internal server" error /assign
if kubectl is intended to work automatically against the local API server, then a kubeconfig file should be set up, but the 30 second delay and incorrect error are a bug that should be fixed in client-go /sig api-machinery
/remove-sig api-machinery /unassign actually, moved the client-go issue to will keep this one for the GCE $KUBECONFIG resolution
I agree that seems like the right fix.
I'm not sure this belong to sig-cli... looks like made this change /sig cluster-lifecycle /cc
Re: set up a kubeconfig That's what we do for GKE: create an authenticated kubeconfig and set the env var. E.g. if you have an authenticated kubeconfig at :
Yes, GKE sets up an admin system account for the master(s) as part of the internal-configure-helper. I believe we should do the same thing for GCE.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


What happened: When running external CSI Driver tests with the Kubernetes E2E: We seem to be getting the following error. From initial investigation it looks like this happens in the during check but only when the test is being skipped. Hunch is that maybe since the test is skipped some resources are never created or they are cleaned up already and they are nil when the is called and expecting them. Kind of low priority since it doesn't make any tests actually fail. /cc /sig storage
see
I think should address this


Not sure if this is a bug, feature requests, or a documentation issue. What happened: I am trying to apply some configs with --prune, and was surprised by the behavior. I have a config file with objects in multiple namespaces, and apply with (note: no flag). By kube context was set to a namespace that was not in the manifests. Here is what I found: If my context is set to a random namespace, nothing is pruned If its set to namespace X, only objects in namespace X are applied If I pass a flag I can control which namespace gets pruned, but that will break the apply because the file has multiple namespaces in it What you expected to happen: I expect some way to be able to apply a config file with objects in multiple namespace and having pruning across all namespaces. Seems there are a few options: all namespaces referenced in the file, prune these across all namespaces warn about why we are not pruning due to namespace mismatch How to reproduce it (as minimally and precisely as possible): Run the script from namespace default, a, and b. You will see different behavior on each run. Related? Environment: Kubernetes version (use ): 1.17.0 Others: /sig cli
I encountered this issue. I think has the bug that o.namespace gets "default" if no namespace is given. Then only a single namespace is visited with or without namespace. If my observation is correct, it changes the bahaviour drastically. In my work, pruning just stops working.


What happened: Using leaderelection with ConfigMapLock, I get a panic when I fail to renew the lease. What you expected to happen: No crash How to reproduce it (as minimally and precisely as possible): Create a leaderelection setup with ReleaseOnCancel=true Run the program, allow a lock to be acquired. Change RBAC permissions to block configmap access Eventually we fail to acquire the lock and run into the panic above
/sig api-machinery
/assign
let me know if you need help getting more info, I am not sure if its easy to reproduce or not. I can add some extra debug logging, etc
I will see if I can reproduce via unit test.
/cc
Can you check the log around the time of crash to see if there was the following ? Checking the events around the time of crash would also help: thanks
I have the log snippet at the top, there is a
I can also hit race conditions from a unit test. I can make this a new issue if preferred. and similar but slightly different:
There is some race around accessing observedRecord. We can use a Lock to protect the access. Please log a new issue. Thanks


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubelet dies with a panic exception if CSI plugin returns NodeId == "" and nil error. Exception log see at the end of the message What you expected to happen: Kubelet should just raise an error to the plugin and ignore the registration. How to reproduce it (as minimally and precisely as possible): Create a CSI plugin with only Node that would implement GetNodeInfo as follows: Register it as a DaemonSet and see kublet crush Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.10-gke.17 Cloud provider or hardware configuration: GKE OS (e.g: ): Container-Optimized OS (cos) Kernel (e.g. ): - Install tools: Network plugin and version (if this is a network-related bug): Others: Related exception Related line of code: - no check whether error is not nil before calling Error() on it.
/sig storage
/sig node is this the version of the kubelet too? 14 is out of support. is the issue reproducible with a 1.15+ kubelet?
/assign
I checked the code - same defect still exists in current codebase.
thanks /priority important-soon
Wow, that was fast. Thanks!


Which jobs are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /milestone v1.18 /priority critical-urgent /cc /sig cluster-lifecycle /area provider/gcp
/reopen Reopening, to observe the job.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Several of the recent failures appear to be this: cc
next run should pick up the change in we have had boskos outage clearing up now.
This is still not going to work because k/k has removed the .deb files we've been downloading from the CI artifacts:
FYI we are discussing in Slack how to proceed to resolve this. I doubt it will be a simple, quick fix, though.
me and did mention this on the k-dev: !msg/kubernetes-sig-cluster-lifecycle/q934Ah3SD64/JukDA9wsAgAJ given there were no replies we assumed there are no users. but this is definitely for the better as we removed the Bazel build of packages from k/k. 's PR seems to use binaries directly: i will have a look.
Closed via There is a possibility we will publish debs again in the future (via ), but while that tool is in flux, I don't want anyone depending on it. Thanks for the fix, /close
Closing this issue. <detailsIn response to : Closed via /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details