<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When creating a new cluster using , panics with SIGSEGV on first run: Note that after the crash the pod restarts, and the second run is successful: and everything seems to be working as expected. What you expected to happen: starts successfully the first time. How to reproduce it (as minimally and precisely as possible): Create a new cluster using , with in mode. Anything else we need to know?: Environment: Kubernetes version (use ): 1.18.0 Cloud provider or hardware configuration: Hetzner VPS OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 5.3.0-45-generic (Ubuntu's default HWE kernel) Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others: using CRI and mode Please let me know if you need any additional information.
/sig network
/assign
According to the panic output, the crashed code is For some reason, might be nil, we should always check it before using.
Do we understand WHY it is nil?
One more thing I've noticed while fiddling with the cluster is that if I remove the node and re-join it, starts just fine, so I'm guessing it has to do with the initial state of the server / ipvs.
That would be my guess too. And that sort of race is worth fixing :)
do you see anything in the logs in the form: ??
I think somewhere here we're failing to initialize the IPVS interface. We probably shouldn't ignore the error from also :grimacing:
Nope, I posted the (full) logs in the issue description.
Thanks, will look into it deeper
do you happen to have a start up script that loads the necessary IPVS modules or is it preloaded on your nodes? Wondering if there could be a race between when the modules get loaded and when proxy starts up. Either way we should log the error ().
Neither, but if you give me the module list I can try to load them before creating the cluster to see if I still get the panic.
For kernel <4.19: For = 4.19:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
I can confirm that, if I explicitly preload the modules, starts successfully the first time.
According to this log , we can confirm second check success, missing modules loaded after first failed check. Because missing modules loaded but not put to


Hi, I have a issue with a dualstack cluster for my Kubernetes Dashboard deployment. I opened a ticket here (), but I am informed that it is not related to Kubernetes Dashboard. What is the problem? Errors in kube-api: The service is created with the correct ipFamily (=IPv6). My endpoint is fully in IPv6. Indeed, my pod has one IP in IPv4 and another in IPv6. Do you have also the issue? Kubernetes version: 1.18 OS: RHEL7
/sig network /sig api-machinery
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think the following should fix the error you saw:
/remove-sig api-machinery


What happened: In a dual-stack cluster when specifying both ipv6 and ipv4 addresses in the entries are inserted in ipvs for the ipv6 nodeports but with an invalid ipv4 address; The example shows the ipv4 loopback address but any ipv4 address in will cause a faulty entry. What you expected to happen: No ipv4 entry shall be inserted for the ipv6 nodeport. How to reproduce it (as minimally and precisely as possible): Configure with both ipv4 and ipv6 addresses Bring up a dual-stack cluster with proxy-mode Create a service with ipFamily:IPv6 Do Anything else we need to know?: Proxy-mode=iptables seems to work. With no ipv4 entries are inserted for ipv6 nodeports. In it is recommended to use but doing so makes kube-proxy enter rules for address (in addition to "127.0.0.1"). While these entries cause no harm they can't be used and should not be there. These entries are created for both proxy-mode "iptables" and "ipvs". This should not be handled in this issue but in another (new) issue if needed. Example; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Kvm's on Ubuntu OS (e.g: ): xcluster Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): calico/cni:v3.12.0 Others: /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/area ipvs /area ipv6
/remove-triage unresolved /assign I take this myself if no one objects.
The ipv4 address is obviously a ipv4-converted ipv6 from .
According to the code here (), we may need to filter the before passing it to .
Looks that that's it. Please go ahead and make a PR, or I can do it if you prefer. I will help test the PR of course. Here are some review comments right away; Use v1.IPFamily as param to the filter function rather than The filter function seem usable and generic and may be placed in some "util" package, but leave it as-is and let the maintainers decide that.
Hi feel free to send a PR please, as you have took this already :) I'd like to help review it!
The "addresses" are really CIDRs which make the fix even simpler.


What happened: In handler#handleSummary, return value from request.Request.ParseForm() is ignored. What you expected to happen: The error return should be checked.
/sig node
Hi this seems to be easy to fix. Could I solve this?
Please go ahead.


What would you like to be : Please see Implement Instances::NodeAddresses so it doesn't ignore the node input. Why is this needed: The current implementation assumes it will be only run on the Kubelet. That is no longer a valid assumption and in fact it the nearish future it will no longer be run on the Kubelet.
/sig cloud-provider /area provider/gcp
This issue is about the GCE implementation found at This problem cannot be fixed in the cloud node controller.
According to is indeed only used for the address of the calling instance. Maybe we should use instead? Note that implements
Or, we should definitely remove the if its TODO item is no longer true.
I believe the TODO is no longer true as there is also a call in


What happened: In , there is mismatch between method name and the method in logs. e.g. RemoteRuntimeService#Attach has the following: What you expected to happen: The method names should be consistent with what appear in the logs. Otherwise people would be confused when troubleshooting. /sig node
could i try to fix this?Thanks
Yes. fixes method name for ContainerStats. You can cover the rest. thanks
/assign


For external cloud provider - Kubelet setting the TaintExternalCloudProvider taint: The results in calling function in cloud node-controller: This (in a healthy case) sends 4 GCE API calls to: node addresses: (as part of getNodeModifier call) instance type: (as part of getNodeModifiers call) instance: (as part of updateNodeAddresses call) node addresses (again): (as part of updateNodeAddresses call) All those requests are read requests and could be replaced by a single call. We could do that by introducing a simple struct like: and adding a method (or rather replacing the existing InstanceTypeByProviderID which is used only for this purpose) to Instances cloud-provider interface; With that change, we would be able to replace all that 4 calls with just a single API call. This would be useful especially for creating large clusters (like 5k nodes), which may saturate the quota for API calls for quite some time. This is quite significant size of refactoring, though it should be relatively mechanical task to do. /sig cloud-provider /sig scalability /priority important-longterm /help-wanted - FYI
The InstanceMetadataByProviderID would be to every cloud provider, such as aws, azure, etc. right ?
/assign Hi , if you're interested in it, I can work with you.
Yes - we need to do that (which is why I wrote it's quite significant refectoring). It also makes reviews harder, because we will need to ask maintainers of all cloud-providers. So it would be good to separate changes to different cloud providers into separate commits.
I have one question: and first query by providerID, if failed then query by nodeName. While the logic in is converse(if providerID is nil, get providerID by nodeName, then ensure its existence by providerID). If we merge these three method into , how to deal with the query sequence of nodeName and providerID?
I bet it's a consequence of different people adding different code paths. Those conceptually seem to be the same to me. Unifying this will be additional benefit from this change imho. So I would say that choose whatever is easier to implement.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I think and are the only PRs left for this. One other optimization that can be made is to cache the instance metadata but I think that's a provider specific implementation
Plus enabling it for GCE if we decide to remove it from the first PR. But yeah, in general I agree.
- can you please address ?
Yes, I have seen that.
I think this issue has been resolved, feel free to reopen it. /close
Closing this issue. <detailsIn response to : I think this issue has been resolved, feel free to reopen it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


We have some scripts that install docker/containerd etc and are having trouble installing on focal based images as does not yet support focal fossa. So in we have a patch that uses debs for focal. Additional context, please see ci test failures in and This issue is to track fixing the script back up when focal support is to
/sig release /sig testing
/priority important-soon /area release-eng
/assign


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: sig-windows jobs: Which test(s) are failing: [sig-network] Services should be able to change the type from ClusterIP to ExternalName [Conformance] [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] Since when has it been failing: It was a false positive until recently. Testgrid link: Reason for failure: Partially Qualified Domain Names are not currently supported on Windows containers. The tests are checking PQDNs for validation. The test was passing until recently because was still returning the exit code 0, even if the given DNS name was not found: [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] [BeforeEach] [sig-network] Services STEP: Creating a kubernetes client Apr 28 21:24:38.587: INFO: kubeConfig: /home/ubuntu/kube/config200424 STEP: Building a namespace api object, basename services STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in services-2633 STEP: Waiting for a default service account to be provisioned in namespace [BeforeEach] [sig-network] Services [It] should be able to change the type from NodePort to ExternalName [Conformance] STEP: creating a service nodeport-service with the type=NodePort in namespace services-2633 STEP: Creating active service to test reachability when its FQDN is referred as externalName for another service STEP: creating service externalsvc in namespace services-2633 STEP: creating replication controller externalsvc in namespace services-2633 I0428 21:24:42. ] Created replication controller with name: externalsvc, namespace: services-2633, replica count: 2 I0428 21:24:45. ] externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:48. ] externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:51. ] externalsvc Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:54. ] externalsvc Pods: 2 out of 2 created, 2 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady STEP: changing the NodePort service to type=ExternalName Apr 28 21:24:55.544: INFO: Creating new exec pod Apr 28 21:25:06.371: INFO: Running '/home/ubuntu/rune2e/kubernetes/output/dockerized/bin/linux/amd64/kubectl --server=https://kubetest--83ef-11ea-ae60- --kubeconfig=/home/ubuntu/kube/config200424 exec --namespace=services-2633 agnhost-pod -- /bin/sh -x -c nslookup nodeport-service' Apr 28 21:25:09.610: INFO: stderr: "+ nslookup nodeport-servicen kube- can't find nodeport-service: Non-existent domainrrn" Apr 28 21:25:09.610: INFO: stdout: "Server: kube-rnAddress: 10.0.0.10rnrn" STEP: deleting ReplicationController externalsvc in namespace services-2633, will wait for the garbage collector to delete the pods Apr 28 21:25:10.429: INFO: Deleting ReplicationController externalsvc took: 423.628373ms Apr 28 21:25:10.830: INFO: Terminating ReplicationController externalsvc pods took: 400.686463ms Apr 28 21:25:15.792: INFO: Cleaning up the NodePort to ExternalName test service [AfterEach] [sig-network] Services Apr 28 21:25:16.230: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready STEP: Destroying namespace "services-2633" for this suite. [AfterEach] [sig-network] Services â¢{"msg":"PASSED [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance]","total":2,"completed":2,"skipped":2913,"failed":0} The issue was observed with an updated, which doesn't return 0 if the given DNS name was not found. Anything else we need to know*: /sig testing /sig windows


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: On running kubetest with external storage driver option with driver name "", encountered error of flag redefinition. The issue happens because: As part of this patch , a flag is defined for number of inline volumes. The driver name is used to construct the flag name. Thus, when running tests on an externally provided driver (in this instance the Persistent Disk CSI driver) with driver name "", a redefinition error happens since the ephemeral test suite code path is executed once for and once for External PD CSI driver. [example run command: kubetest --test --check-version-skew=false --testargs="-- --storage.testdriver=/path/to/gcp-compute-persistent-disk-csi-driver/test/k8s-integration/config/test-" Test config looks something like this: { StorageClass: FromFile: ... SnapshotClass: FromFile: ... DriverInfo: Name: ... } ] What you expected to happen: Test should run and succeed. How to reproduce it (as minimally and precisely as possible): Use an test-, with driver name as "", and run kubetest with focus as "External.Storage" kubetest --test --check-version-skew=false --testargs="-- --storage.testdriver=/path/to/gcp-compute-persistent-disk-csi-driver/test/k8s-integration/config/test-" Anything else we need to know?: Possible fixes options: , (), when using a FromFile or FromExistingClassName option to pass in SnapshotClass information, pickup the provisioner name from the class definition's "driver" field. the flag which has a dependency on driver names. (need to understand the impact of removing the command line flag) Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage
/assign Do you know why this parameter is set as a flag and not a part of the ?
To me, is something that defines what the driver can do. This flag is about what the test does and not related to driver capabilities. Making it depend on the driver name was probably unnecessary. A single flag for all incarnations of that test probably would have been sufficient and thus better ("keep it simple, stupid!" - the stupid being me here...). It also shouldn't have used the global flag set. In other words:
/reopen I don't really want to get into the habit of defining per test case flags in There are thousands of test cases and it would be unwieldy to discover all the flags available and understand how to use them. What factors would influence how many volumes you want the test to create? Can we just hardcode a value in the test case?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The defaults must be usable so that users do not need to understand them. The flags can be discovered with - not ideal, but at least they show up. That's effectively what the default value for the flag does. This is a way to to run a custom stress test: the default is so low, it just tests that the feature works. By increasing it as high as the underlying CSI driver and storage system supports, one can run this (and potentially other tests) as part of stress testing. Not sure whether it's worth supporting this. It seemed worthwhile at the time, and better than hard-coding just one value.
The bigger issue is: how do users of the test suite actually learn what tests they run and what those tests do? I suspect the answer to that is simply "read the source", in which case they'll also find the knobs that can be used to tweak testing.
Since increasing the parameter depends on the csi driver/storage system, then I would recommending splitting this up into 2 test cases: 1) test basic functionality 2) test stress scenario, and add the parameter to the driver test config similarly to other stress tests:
At least for the storage tests, because we are intending that they be run externally, having all the tunables in one place I think makes it easier to discover.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle stale /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
I saw that the flag was removed in , I think we can close this one if doesn't apply anymore
/close The flag was removed in #diff-20517805986874f69e3254bed93d59996d5a6fd571a70ab8120736ded5aafa24
You can't close an active issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /close The flag was removed in #diff-20517805986874f69e3254bed93d59996d5a6fd571a70ab8120736ded5aafa24 </details
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


[x] Original approval issue: Start 9 June 2020 3 Flakes No Flakes for 2 weeks According to this APIsnoop query, there are still some remaining ReplicationController endpoints which are untested. a ReplicationController with a static label the ReplicationController with a new Label and updated data the ReplicationController's status /Scale the ReplicationController's Replica count to 2 Replicas the ReplicationController the ReplicationController's Status all ReplicationControllers in all Namespaces find the ReplicationControllers(1) ensure that the ReplicationController is found and is patched Namespaced ReplicationControllers(1) via a Collection with a LabelSelector Discover useragents: See all endpoints hit from this test Display test coverage change If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
Can you please add in PR
/reopen Merged. Promotion to be created
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


[x] Original approval issue: [x] two weeks soak end date: 2 July 2020 This issue is created to allow edit by According to this APIsnoop query, there are still an endpoint which is untested. /wip /hold a set of 3 Pod Templates with a static label within a Namespace. that all 3 Pod Templates with the label in the Namespace are created. the set of Namespaced Pod Templates with a label via DeleteCollection. that all Pod Templates with the label have been deleted. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
/area conformance
During Conformance Meeting: Is there a way to write this test so it is resource agnostic? If we are just changing the resource... can be parameterize the resource-type
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen Merge 17 July 2020
Reopened this issue. <detailsIn response to : /reopen Merge 17 July 2020 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


What happened: I checked if kubelet () has permissions for with SelfsubjectAccessReviews and I saw that it has. But when I checked its permissions with SelfsubjectRulesReviews it didn't show it. What you expected to happen: I expected that the API SelfsubjectRulesReviews will show that has permssions for . How to reproduce it (as minimally and precisely as possible): You can check it on the master or the worker. Test SelfsubjectAccessReviews: You will get answer that it has permissions: Test SelfsubjectRulesReviews: It won't show the permissions. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm /area/api /kind/bug /sig api-machinery
/sig api-machinery
This is a more clean display that show the bug: I think the problem is related to the fact that the file contain the kubelet certificate which is part of the group :
/sig auth
What is the SelfSubjectRulesReview response from the server ()? Note that not all authorizers support enumerating permissions, and the API object status includes an attribute which explicitly indicates the result is incomplete. See
It has Details:
I noticed that it also doesn't show that it has permissions to view and . It probably related to the Node authorizer: Because I assume that these rules doesn't appear in RBAC.
The node authorizer does not enumerate permissions, so it does not contribute to SelfSubjectRulesReview results
fixes setting correctly when the API is requested with a kubelet's credentials


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /priority critical-urgent /milestone v1.19
/sig api-machinery
Suspect it has to do with cc /assign (for CI signal ownership)
Seems unlikely since multiple kind e2e tests passed on the PR itself
I was thinking that as well, but I don't see any other relevant changes
Actually, I take that back. Looks like the same failure was encountered as a flake on some of the PR runs. Looking at the test, it appears to assume the resource version of the kubernetes service will always be in the watch window, which is a bad assumption. I'd expect us to fix that assumption in the test, but also make sure we didn't unintentionally shrink the watch window for resources.
/assign
Note: this also appears to be breaking
Failure seems to be timeout related as well shows the runtime ~tripled after merged
revert open in can investigate test failure and timeouts and bring the change back in
I take it back, the test tolerates an expired error explicitly.
showed 17 of the 22 runs since merged took ~1 hour. The first run after the revert at returned to less than 30 minutes for the run. Will keep an eye on that to see if subsequent runs match.
- thanks a lot for looking into it. I don't think my PR changed anything that anyone should rely on. I will take a deeper look into it and try to fix that (hopefully later this week).
contains the explanation and describes the bug (in fact this bug still exists in the codebase, that PR was only increasing the probability of this particular test hitting it)


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: UT TestValidateNodeIPParam does not really tests . The first IP and interface are always 127.0.0.1 and lo and will break the loop. So will never be appended into . What you expected to happen: gets tested correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/sig node


apiserverrequest (total, latency histogram) metrics are likely the two most important metrics emitted by the kube-apiserver, aggregator, and aggregated-apiserver binaries. has been open for a year regarding some of the issues that have been ongoing problems. I want to open up a discussion on: (1) cleaning this metric up so that it can be used reliably and consistently for many versions addressing as many of the cardinality concerns as possible (2) explicitly defining a set of prescriptions for adding new metrics which may require joins to this very centrally important metric These metrics are critically important because people use them for SLIs/SLOs. In fact we define SLOs for them here (). Yet, due to the fact that people can inject arbitrary webhooks into the request flow process, we can never actually guarantee that the apiserver response time actually meets our stated objectives. Which brings us to the other dimension in which these metrics are flawed. They do not account for apiserver request processing time, exclusive of webhook processing time. As long as we do not break out internal request processing time, it is possible that arbitrary clusters will never be able to meet our internally stated service level objectives given the one metrics we provide.
/sig api-machinery instrumentation /assign
The cardinality issues hit us here pretty badly, just one example for this, I suppose: Soo :100: % agree.
/milestone v1.20
Heyho I wanted to check how it is going? The code freeze is starting 12.11.2020 (about 3-4 weeks from now) and while there is still plenty of time, we want to ensure that each relevant issue has a chance to be considered. As the issue is tagged for 1.20, is it still planned for this release?
Ah, this is going to be slated for 1.21. /milestone v1.21
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/open
/re-open
Hello! I am from the Bug Triage team! Friendly reminder that code freeze is starting March 9th, 2021 (today!). As the Issue is tagged for 1.21, and the PRs linked are merged and marked for 1.21, can we close this one?
Yes! :)
Thanks !


What happened: does not handle "holes" in interface device-number correctly, dropping later interfaces. Create instance with eth0, eth1, eth2 ENIs as usual. Remove (eg: by selectively deleting pods) eth1. Only the addresses on eth0 are reported by NodeAddresses. What you expected to happen: Addresses on eth0 and eth2 should be reported.
/sig cloud-provider These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.


See - currently there's just one test that triggers Unreserve through a pre-bind failure.
/sig scheduling
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I shall work on it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/assign
i saw commented the only test code, do you have any idea why this going on?
well, I found some functions missing, maybe because of the code refactor.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


<!-- Please only use this template for submitting enhancement requests --What would you like to be : The ebtables utility in is missing 3 useful features: method in its Interface table chain Why is this needed: The missing features are useful to allow for: rule deletion as there is a way to add (ensure) a rule but not removing it currently. manipulating the broute table and brouting chain The first feature would also align ebtables with iptables, as a DeleteRule method is available there.
/sig network
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Is there some feature/bugfix you're working on in kubernetes that needs this?
Yes, in kubernetes/dns (node-cache) actually. That PR is working, but it's only in my fork of kubernetes/dns at the moment as it depends on this. It solves this: Adding this code to ebtables would help me avoid re-inventing the wheel there.
... not sure if you've seen that ... but I have created a PR to solve this issue already:
Yes, I saw the PR first. So, kubernetes/kubernetes isn't really intended to be vendored into other repos and kubernetes/dns is being bad here. It's dubious to add methods to that kubernetes itself isn't using because someone else might just end up removing them later when they notice that they're not being used... OTOH the PR needs access to the internals of the ebtables code so you couldn't just add it to kubernetes/dns directly without forking all of ...
Thanks for your comments. I saw iptables used that way in node-cache and I thought it would be good for consistency to go down the same route with ebtables. To be honest I am not very keen on forking the entire package, for obvious reasons, but I can understand your reservations about this change.
/assign
Hi Did you manage to discuss this?
yes, sorry, didn't get a chance to get back to this on Friday. (I /lgtm'ed the PR) /remove-triage unresolved
Thanks!


/sig testing /sig release
I'll work on this one /assign
thanks! /remove-sig release (on second thought this is just SIG Testing)
/lifecycle active


What happened: Running manually includes this file in LICENSES. Running complains if it is present. The file was incorrectly dropped in This needs to be resolved before 1.19 ships /priority important-soon /milestone v1.19 /cc
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
fixed in


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet fails to create kubepods cgroup on larger machine with 512 cpus, fails with the following error: What you expected to happen: Run kubelet properly without any error How to reproduce it (as minimally and precisely as possible): Start the kubelet on a machine which has a more processors Anything else we need to know?: Environment: Kubernetes version (use ): 18.5 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
When I changed CgroupDriver to cgroupfs, kubelet is running is fine:
And here is the sample code which is creating this issue:
so this is essentially a runc/libcontainer problem or a problem with the host configuration - e.g. the host having less than the requested ? this line: hints that your snippet is for an older version of runc. rc8 renamed it to "Manager": is that the version of the kubelet too? k/k 1.18 is using rc10: so ideally the match test for kubelet 1.18.5 should be against rc10 of runc. and current master of k/k is at rc90: are you able to reproduce the same with rc90?
I tried running with master kublet which does use rc90 build and kubelet fails with the same error: complete log: I think this is happening because we are trying to create a cgroup more than a supported range for the Linux kernel: 2^18 = is the max value we can use for the CPUShare, so wondering whether can add logic into the kubelet to address this issue when we run on bigger machines(more than 256 processors)
/cc
interesting snippet in moby: ! where
i think we have a misconception here that CPU shares == # of CPU. "cpu shares" is a weighting proportion to give a container more resources in comparison to other running containers: the reason for 2^18 in the kernel is to have a larger integer scale to work with. unlike docker that has (nanocpu / fractional cpus) (): does not have such an option. but it does expose (CpusetCpus) which accepts a such as . i have a wild guess that this same problem is not present if you use dockershim. to see more on what is happening you could: run the kubelet with --v=4 or more. patch the kubelet to add debug code.
sorry for the late reply. I think the issue is not in libcontainer, but in the Kubelet asking for a out of range value for cpu shares. I am not sure how it makes sense to set the number of shares in relation to the number of available CPUs, since it is a weight and not a limit. The error you are seeing is coming straight from systemd: Since there is already a check for the minimum value, I think it makes sense to add also a check for the maximum allowed (while we figure out why cpu shares are used this way). Could you give a try to the current patch? I've nothing close to 512 cores to validate it really solves the issue you've seen :smile: I'll open a PR once we are sure it solves the problem
You are right, this is coming from the systemd library which is getting used in the libcontainer With the above patch, I see kubelet is running fine but I don't think this is just good enough because while requesting the cpus in the pod is in terms of milicores which is 1000th of 1 cpu which is getting converted into and assigned to the pod's cgroup. e.g: if the user requests a cpu of which 1024/ = ~2 cpu share out 512 which brakes the current logic in the code. I think we need to add an additional logic for allocation based on the number of cpus present in the machine. Simple math should work fine if cpus are more than 256 in the system: e.g: if the request is 1024m 1024/(()/) = 512 shares
the value doesn't really limit the cpu time a process gets. It is a weight used for the cgroups at the same level. It looks to me that the Kubelet is using this mechanism simply to maintain some sort of priorities among the different pods (i.e. if pod A has double millicpus than pod B, then A gets double weight than B). Before adding more complexity, I'd make sure why it is done this way, I think it is just wrong to put in relation to the number of cores. The cpu shares assigned to each pod though are not dependent on the number of cores, but they are converted directly from the spec file. You've reported the issue on systemd, and it happens since systemd adds another check that is not present with cgroupfs or the kernel. The kernel does the same clamping I've proposed above: so the patch is just to make sure the systemd cgroup driver behaves in the same way as cgroupfs; and I think it doesn't have any impact on the pods
I've opened a PR here:
Thanks for the PR, my only worry about - how k8s will prioritize when we have two pods requesting more than the MaxShares how this logic will prioritize it!
for now it works in the same way as in the cgroupfs case. Pod requests are not in relationship with the # of cores, that AFAICS happens only when setting up the node cgroups. I think a static allocation would work better for such cgroups, as anyway is a weight.


These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions


Which jobs are failing: ci-kubernetes-node-kubelet-conformance Which test(s) are failing: Variable Expansion should succeed in writing subpaths in container Variable Expansion should verify that a failing subpath expansion can be modified during the lifecycle of a container Since when has it been failing: Since Fri Jul 10 15:41:46 2020 -0700, when was merged Testgrid link: https://k8s- Reason for failure: Used DeletePropagationForeground option to delete test pod introduced by Anything else we need to know: The test pod stuck in Terminating state until 5m timeout of DeletePodWithWait expired. It happens with empty pod as well. Here is a simplified ginkgo test case that triggers this failure: The pod phase is "Failed" and container reason is "Error" for some reason: Reverting above commit or changing delete option to DeletePropagationBackground should fix this.
/cc
/sig node
/sig storage cc
/assign
I think change just triggered the underlying issue. DeletePropagationForeground option doesn't work as expected it seems.
does the ci-kubernetes-node-kubelet-conformance job run the controller-manager? the kube-controller-manager is responsible for deleting objects with a foreground finalizer on them, but I don't see any kube-controller-manager logs gathered in the artifacts for one of those test failures -
I could reproduce the issue for commit (before 'generic ephemeral inline volumes' PR was merged) + 'use foreground deletion' commit.
yeah, I tend to think switching that e2e helper method for all callers was not a good idea
it looks like node e2e runs only etcd, api server and namespace controller: I don't think it makes sense to run controller-manager just to fix this failure. Would modifying test framework to set DeletePropagationBackground option for e2e node tests be correct solution?
For a quick fix, I would suggest just reverting the test change for now. Long term, yes, it should be configurable per test.
Also do we have plans to make this job a blocking presubmit test so we can find these problems earlier?
I agree, we should revert , both to avoid changing behavior for existing callers and to match how pods are normally deleted
if we need a PR for that I can prepare it.
that would be great, thanks
putting my hat on, probably not? more hour+ tests that run on every PR are a non-goal
The SIG / OWNERs should enable testgrid alerts on failures, if periodic testing is not sufficient you can use a postsubmit to run whenever we merge code. We're not looking to rely further on presubmit, as is we're running too much there leading to excessive resource usage and inability to merge fixes like this in a timely manner between one thing or the other flaking heavily (and our odds get much worse with each new presubmit)


Which jobs are flaking: ci-kubernetes-kind-e2e-parallel pr:pull-kubernetes-e2e-gce pr:pull-kubernetes-e2e-gce-ubuntu-containerd pr:pull-kubernetes-e2e-gce-ubuntu-containerd ci-kubernetes-kind-ipv6-e2e-parallel Which test(s) are flaking: Second flakiest test in the e2e job according to Testgrid link: Reason for failure: Anything else we need to know: /sig network /assign
needs investigation since we're switching kube-proxy to endpointslice in this release
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
made me think of , but that's in already.
It's possible it's a test timeout that's just too close, but we need verification of that checking the controller and apiserver logs to make sure there isn't a latent bug
Thanks for catching this! It looks like the timeouts are actually associated with waiting for 2 pods to be ready, not actually anything related to EndpointSlice functionality - timeout is here: Given that, it seems like the obvious solution would be to update the Services to set to and update this block to wait for the Pods to have IPs, not for them to be ready. If that sounds reasonable, I'll work on getting a fix in tomorrow.
/remove-triage unresolved
If that is still exercises the same conditions (I vaguely remember publishNotReadyAddresses changing code paths in the endpoints controller... does it do the same in the endpointslice controller?), that seems reasonable to me.
I think the code path is very similar here, the only real difference should be how the EndpointSlice controller calculates readiness. I've gone ahead and filed a PR that uses this approach, hopefully it helps:
This flaked again this morning /reopen
Reopened this issue. <detailsIn response to : This flaked again this morning Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This should be /priority critical-urgent given
The top level error message does not make this clear but the issue here is that too many EndpointSlices are being created - 3 instead of 2.
Interesting... that condition should not persist for O(minutes), right? that seems like it might be a bug, not just a test issue
There's potential that it's not consistently merging 2 ports with 1 endpoint into a single slice and instead creating a slice per port. It also could represent a larger bug. Trying to find a way to consistently reproduce this.
Despite my best efforts I have been unable to reproduce this on my own e2e cluster. I've had 10 test runs going at the same time repeatedly without any luck, any ideas for how to reproduce flaky test failures local would be welcome. In the meantime, I've created a follow up PR that will provide more helpful information if/when this test fails again: My leading theory right now is that this could be a caching issue. The EndpointSlice controller runs a sync, sees that we don't have an EndpointSlice for the given port combination in cache and then goes ahead and creates one, even though a matching EndpointSlice already exists and has not been cached yet. That would result in an annoying situation where an essentially duplicate EndpointSlice was created due to a stale cache. I would think the EndpointSliceTracker would resolve this issue for us though, so not sure. Getting the above PR in will help validate this theory. It's worth noting that if this is actually a duplicate EndpointSlice, clients like kube-proxy will handle this seamlessly because they already have to handle potential duplication in EndpointSlices. That would potentially explain why a test failure like this does not seem to be causing other things to break.
It appears that my theory was correct as witnessed here: I create a . The key points I've noticed so far: 1 EndpointSlice references pod2 (example-named-port-f7vq5) 2 EndpointSlices reference pod1 (example-named-port-fkgcc and example-named-port-vfp2q) 2 EndpointSlices were created at "2020-07-28T04:57:51Z": example-named-port-f7vq5 (references pod2) example-named-port-vfp2q (references pod1) 1 EndpointSlice was created 1 second later ("2020-07-28T04:57:52Z") example-named-port-fkgcc (references pod1) All 3 EndpointSlices are: On generation 1 - no updates have occurred With a state of ready: true The 2 EndpointSlices referencing pod1 are effectively identical  Of course this is something the EndpointSliceTracker should be handling for us, but even that approach seems less than ideal here. It wouldn't actually prevent the duplicate EndpointSlices from being created, it should just be cleaning up duplicates after they happen. This combined with the race conditions referenced in make me think we may want to find a way to slow down the sync cycle for each Service here. This also feels at least tangentially related to the need for better rate limiting in both this and the Endpoints controller discussed in . /cc
Hmm - actually I don't understand why this is happening. Maybe I'm missing something, but I don't think slowing down controller is needed to solve this problem. Even though the controller usually is not lagging my more than 1s, it is possible that it can be lagging by a minute or more in pathological cases. So we can't just rely on the fact that it's guaranteed to be synced. Can you explain it a bit deeper why this is happening? i.e. some more detailed timeline like: T1: state of cache is XYZ T2: controller computes the ES x1, x2 T3: state of cache in ABC T4: controller computes ES y1 [why?]
Agree, I'm not in favor of slowing down the controller globally to address this. Delaying resync of a particular service until we observe the existence of new endpointslices we know we created (with some max delay) could be reasonable.
That's exactly where I was hoping to end up here, although I'm not quite sure how to do that cleanly yet. To be clear, this is not just to fix this flaky test. This flaky test is a symptom of a much larger and more annoying issue with the current implementation. The EndpointSlice controller is consistently running into errors where it's trying to update EndpointSlices that are stale. This is caused by follow up calls to running before the informer cache is up to date, which happens all the time during rolling updates where rapid churn of EndpointSlices is expected. Although this eventually resolves itself, it's causing a lot more load on API Server than necessary and can potentially result in some bad states where an update fails but a delete succeeds. In this specific instance, it looks like a syncService run is occurring after the first 2 EndpointSlices have been created but before the one referencing pod1 is in the informer cache. That results in the controller creating a new EndpointSlice to reference pod1 and thus we end up with a duplicate. If anything in the future triggers another syncService call that would be resolved. I'd initially thought that EndpointSliceTracker would be thing to trigger that follow up syncService, but I was wrong. That tracker just tracks the latest resource version for each EndpointSlice the controller has created/updated and compares that with EndpointSlices received through create/update events. If the controller doesn't think that EndpointSlice should exist or the controller's resource version for that slice does not match it triggers another syncService. In this case neither of those conditions would be met so no additional syncService call would be triggered. So that's a long way of saying that I think our best possible solution here is to introduce some delay between when Services can be synced. I think the easiest solution that would solve 99% of our problems would be to add something like a 1s delay in between syncs simply by delaying when was called. This would not be ideal, but it would result in an immediate improvement and be easy to backport. It's worth noting that although duplicate EndpointSlices like this or even attempts to update a stale EndpointSlice are not ideal, they are not actually invalid and will eventually resolve themselves. Consumers of this API already have to do some form of deduping. With that said, I'm hopeful that there's a clean way to implement a solution that actually waited for each EndpointSlice change the controller has made to be reflected in the cache because that would be a better solution. I'll spend some time looking into possible implementations here.
I've created a draft PR that contains my initial progress towards the approach I described above. I don't particularly love this approach so I'm very open to other ways we could solve this issue. It's also still very much a WIP and has significant work remaining. I opened the PR early just so I could get some initial feedback on the direction and potentially inspire other ideas around approaches we could take here.
is still the best approach I can come up with here. It is ready for review if anyone has time to take a look.
Quick update here, I've limited the scope here to the following changes as discussed in Update the flaky test to consider duplicate EndpointSlices valid. Update the docs to make it very clear that consumers of the EndpointSlice API will have to do deduplication (as kube-proxy already does). Update the controller error handling to return immediately if a create, update, or delete errors (in that order). Reviews would be appreciated on all of the above.
Reopening since is not yet merged.
Thanks for catching that Closing now that is merged. /close
Closing this issue. <detailsIn response to : Thanks for catching that Closing now that is merged. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: gce-master-scale-correctness Which test(s) are flaking: [sig-network] Ingress API should support creating Ingress API operations [Conformance] Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /kind flake /sig scalability /sig network /cc
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
this is caused by the issue described in a different test is creating a default ingressclass, which breaks creation of ingress objects if there was already a default ingressclass set up in the cluster /assign
this should resolve the issue:
I take it back, looks like the conformance test is also creating default ingressclass objects... it should not fix in


The statement "returns [...], or nil" cannot be true since the return type is a struct: Suggest it could say ", or an empty struct if ..." /sig api-machinery
I can file a PR to fix the incorrect comment if you don't mind :)
Sure, I didn't get round to it yet.
/assign


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: not fixed the SMB mount issue totally, when there is a SMB mount with same remote path, original SMB mapping would be removed and SMB mapping re-established, that would cause temporally data inaccessible with the original container. should change to should use to check whether original smb mapping is valid, here is an example showing that after SMB server password changed, could validate whether connection is ok: the main diff on smb mount between linux and windows is that on LInux, every smb mount is independent, while on Windows, one remote smb server address(e.g. ) can only have one SMB-GlobalMapping. So on Windows, there is a difficulty: in some cases, we should skip create SMB-GlobalMapping since it already exists in other cases, original password is incorrect or expired, user wants to delete SMB-GlobalMapping, and then create a new SMB-GlobalMapping (fixed by ) On Windows, there is no way to check whether original SMB-GlobalMapping is valid or not, that was the answer about 1 years ago from windows team, will update this thread(also about the max SMB-GlobalMapping limit) if there is any new update recently. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig storage /assign


What happened: I two CSI plugins to my Kubernetes cluster via a single call to . When I did this, the annotation was improperly set on 2 out of 3 nodes. Only one of the plugins appeared: What you expected to happen: Both and plugins should have appeared in the annotation. How to reproduce it (as minimally and precisely as possible): Apply two or more CSI plugins via a single call to . Anything else we need to know?: Kubelet logs indicate that it registered both plugins at the same time: The CSINode object looks complete: Only the node's annotation did not include Restarting kubelet fixed the issue. It looks like a race condition when InstallCSIDriver is called concurrently for multiple drivers. The code here: could lead to each thread observing the annotation as being empty, and then setting its own annotation that includes only itself. Environment: Kubernetes version (use ): v1.17.11 Cloud provider or hardware configuration: AWS OS (e.g: ): Ubuntu 20.04.1 LTS Kernel (e.g. ): Linux ip-172-31-17-163 5.4.0-1021-aws -Ubuntu SMP Fri Jul 24 09:42:29 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: juju
/sig storage
/assign
I can reproduce this issue with commands below ! The correct csi nodeid would be like this


What happened: I download fluentd-elasticsearch in addons. The following error occurs then I use "kubectl apply -f es- " Then, i find the elasticsearch-logging pod restart lots of time, and keep status with CrashLoopBackOff. [root ~]# kubectl logs -f elasticsearch-logging-0 -n kube-system export NODENAME=elasticsearch-logging-0 NODENAME=elasticsearch-logging-0 export NODEMASTER=true NODEMASTER=true export NODEDATA=true NODEDATA=true export HTTPPORT=9200 HTTPPORT=9200 export TRANSPORTPORT=9300 TRANSPORTPORT=9300 export MINIMUMMASTERNODES=2 MINIMUMMASTERNODES=2 chown -R elasticsearch:elasticsearch /data ./bin/elasticsearchloggingdiscovery I0902 14:25:06. 9 ] Kubernetes Elasticsearch logging discovery I0902 14:25:06. 9 ] Found [] I0902 14:25:16. 9 ] Found [] I0902 14:25:26. 9 ] Found [] I0902 14:25:36. 9 ] Found [] I0902 14:25:46. 9 ] Found [] I0902 14:25:56. 9 ] Found []
unclear whether the addon is up to date and supported at this point. /sig instrumentation
cc
/assign I will take a look this week.
I have the same problem k8s version:v1.18.8
same
I've pushed a change to the stateful set that should unblock people as there was a 1.18 compatibility problem. however, please make sure you are also creating the service via , the discovery program finds the other elasticsearch nodes via service registration.
please feel free to re-open this issue if the latest version doesn't work for you.
Why was this PR to v1.20?
probably because that is the next release. most people who use this addon do so from master. Do you need this packaged in some other way?


Is this a valid test? My understanding is we cannot rely on events for programming against. Which jobs are flaking: ci-cri-containerd-node-e2e-features ci-kubernetes-node-kubelet-features among others, mainly 'node-e2e-features' based tests. Which test(s) are flaking: Testgrid link: Reason for failure: expected total number of events was 4, actual events counted was 7 Anything else we need to know: It doesn't fail in a specific environment, should this test be running at all outside of this job? The comments inside the test aren't very helpful for understanding why we're checking what we're checking.
I heard this is blocking you. /sig node /cc
I see flakes going back at least a month In addition to an earlier issue clarifying the error message This might be a spurious match,
NodeProblemDetector is broken on nearly every run:
Maybe they should be marked [serial] or [disruptive]?
We can mark it and move it to the flake job, but I fear it will work fine over there.
Edit in upstream issue says "needs to be marked serial" I'll make a PR


Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: TestDevicePluginReRegistrationProbeMode Testgrid link: Reason for failure: Anything else we need to know: Reproduce with /sig node
/assign
See
not sure this is actually fixed, I see a failure from today


/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig autoscaling Looks like it just started happening today (@ 2020-09-15 03:54:19 +0000 UTC) !
cc the namespace from the provided object "kube-system" does not match the namespace "horizontal-pod-autoscaling-830". You must pass '--namespace=kube-system' to perform this operation. the namespace from the provided object "custom-metrics" does not match the namespace "horizontal-pod-autoscaling-830". You must pass '--namespace=custom-metrics' to perform this operation. the namespace from the provided object "custom-metrics" does not match the namespace "horizontal-pod-autoscaling-830". You must pass '--namespace=custom-metrics' to perform this operation.
related fix:
No I don't think that's the fix for this (that is just a little cleanup). Actually I think we have to delete namespace parameter in line : Because before , the will have no actually effect. But now with the change of , it will have namespace : at the end of the command. But the namespace in yaml file is which is different from . We should not have parameter when applying from file because this test didn't have before, right? Related PR:
/assign


Which jobs are flaking: https://k8s- Which test(s) are flaking: Services should have session affinity timeout work for service with type clusterIP Services should have session affinity timeout work for NodePort service We have other flaky cases, like , that should also bu resolved. But cases also run in , so I suggest we solve this problem first. Testgrid link: https://k8s- Reason for failure: In ipvs mode, we set session affinity timeout as 125s, the test case is: run host command to send a request, mark returned hostName 125s another request, if return different hostName, test succed. I think the reason is, in the second request, ipvs connection may still in sessionAffinity, send another request will result in seffinity restart count, eventually result in this test case flaky.
/sig network /area ipvs
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign /remove-triage unresolved
This keeps failing https://k8s-


Inspired from Our default Filter plugins have the following order: IIUC, this order was chosen to have the fastest Filters or the most probability of reporting Unschedulable first. Arguably, some are pretty much equivalent and could be reordered. The major problem with this order is that plugin is almost at the beginning. This plugin only reports . A Node that fails this plugin will be evaluated during preemption. But perhaps this Node doesn't satisfy other UnschedulabeAndUnresolvable requirements (taints, node selectors, nodeaffinity). Theoretically, having those Filters run first could improve preemption latency. The only open question is whether or not the ordering we have today is an API guarantee. I believe it's not (at least in component config), because the only way to change the ordering of Plugins is disabling all of them first. Decided order: /sig scheduling
for comment
and :)
Even we changed the order of plugins, the framework.RunFilterPlugins still exits early when returned by plugin. So should we change the current behavior to run the filter plugins until there is one plugin returned ?
That is correct, and that's what it should be.
No, that's what the original conversation suggested and it's likely not ideal in the average case.
+1 to reordering. Taints, affinity, node name can all be executed before checking for resource fit. No, it is not.
+1. Lightweight plugins that may return status can be moved ahead of other plugins that only return status.
FYI: from who also had this concern: So how about , is it also a good candidate to be moved ahead of ? NodeAffinity and NodeSelector both incorporated in , I'm pretty sure NodeSelector logic is simply, but not 100% sure NodeAffinity logic falls into the category of "lightweight".
Let me propose an order, for extra clarity:
internally composes and invokes , which IMO is a bit more expensive than . should be cheaper than . So my 2 cents is:
Both proposals are fine by me.
/triage accepted /help /good-first-issue
/assign
I want to help here :)
/remove-help


What would you like to be : Scheduler VolumeBinding plugin reports error when a PVC doesn't exist for example: We need to take a second look at the VolumeBinding plugin and make sure that errors are returned only when there is a system error, not a user error. Why is this needed: To fix the scheduler's error reporting metrics. /sig scheduling /sig storage /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I'll take a look.
If the PVC object does not exist, the plugin will return status. I guess you mean the plugin will return an error when a referenced PV does not exist? I've created a PR for this:
Thanks! The errors I am seeing are on 1.18 clusters, can we look at 1.18 and try to patch? This is still throwing the scheduler's error metrics off.
sure
I've submitted a patch for the 1.18 release branch:
I guess on the master branch should be reviewed and merged too? cc
hello! the fix you made only apply for 1.18? how about 1.19 and 1.17? thanks!
after the fix on the master branch () is merged, I'll cherry-pick it into the 1.19 branch (the code are almost the same). I'm not sure if it's necessary to fix the 1.17 branch. It's 3 versions behind 1.20.
hi, This is because of this patch: Do you think this fix needs to be cherry-picked into 1.19 and 1.18 branch too?
can you send a patch for 1.19?
ah, here it is
to sum up: master: 1.19: 1.18: and I think this change should be backported to 1.19/1.18 too: I'll send PRs soon.
summary of the issue that an error will be reported instead of UnschedulableAndUnresolvable when PVC not found: master: already implemented, see 1.19: 1.18: can't backport as it does not implement the PreFilter extension point


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I restart containerd.service, kubelet can not connect to it again and cause 100% cpu usage. there are many logs like this: I can show container info via crictl at the meantime and restart kubelet.service can resolve the problem. What you expected to happen: kubelet can reconnect to containerd after containerd restart. How to reproduce it (as minimally and precisely as possible): cpu usage and syslog. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: armbian(debian on arm64) OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign
Reproduce: can reproduce the issue by restarting when is the container runtime another setup where was used used as container runtime, didn't experience the issue by restarting , but restarted the underlying , same issue shown up again and had to restart to resolve the issue. Will continue with the investigation, might need to reconnect to after the socket is broken.
To me it seems there are a couple of problems here. 1. kubelet does not reconnect to the containerd socket. 2. The reconnect is agressive and consumes a lot if cpu. I don't know the cause for 1, but for 2 this might be an indication that the retry interval should be relaxed.
Spent a few hours on the investigation, it's likely to be some internal states being messed up in .
TL;DR: It should be related to gRPC. Hacked the vendored gRPC mod and changed its to something like this: Then CPU only spike for a few seconds after restart then comes back to normal instead of failing the all the time. kubelet passes a customized dialer to the gRPC client. I logged the address during kubelet startup: As you can see, the address of the is at . Before restarts, I can see from log that every time, the gRPC function is using the customized dialer that kubelet passed: However, once got restarted, the address of the dialer is changed: And that function at keeps failing. As I forced to use the original customized dialer, the issue is gone. Without too much knowledge on gRPC, my blind guess is as is passed by reference to gRPC client, some component could modify it during the process lifetime and the change got persisted. Anyway, I'll check with gRPC team on this issue and see if there's already a fix on their side. If so then I can port the fix here.
/cc
Hi, are there any updates on this. I am experiencing what appears to be the same issue. If I restart containerd while the kubelet is running, my kubelet logs get flooded with: Restarting the kubelet appears to be the only way to re-establish the connection with containerd. Version information is:
Hi thanks for the info, I'm still working on the root cause. As I mentioned above, it's something related to gRPC. I collected the object info, this is the object before containerd restart: ! And this is the object after containerd restart: ! There's one thing made me concern is, the connection state is still 2 (i.e. , see ) after the failure. I'm not sure if this is related to the cause but anyway it still should be something related to gRPC. Btw I've followed their to use the latest gRPC client without customized dialer and still see this issue, will still work on the root cause.
We're seeing this now when we update our nodes to 1.19 where our Kata Containers install does a . Our current workaround is to also do , but would be good to get this fixed.
Is this MR relative to the problem?
No I don't think so, I've cherry-picked that change and tried in my environment weeks back, and the issue is still there. I only get around the issue by modifying the gRPC code as I mentioned in the above, I'll continue to check which component modified the function in after thanksgiving.
For the sake of priority. This hit us hard these days too, as security updates for container.d triggered a restart and a ton of nodes flooded the central log management. ;) Thanks for working hard on a fix.
Same here. Two systems were affected by automatic container.d upgrades & restarts. But now I wonder if - generally - "unattended-upgrades" (Ubuntu 18.04) is a wise thing to do? Or should the OS for kubernetes deployments always be updated manually & controlled? I can not find best practices for this in the documentation.
After the recent CVE-2020-15257many users will upgrade and restart containerd. It would be a big problem.
I see, will prioritize on the root cause and fix. At the mean time, you need to restart after you restart .
Had another round of debugging, and made one step towards the root cause. I following snipple to the as I mentioned in previous : When I reproduced the error, I got following information for the dialer function: That leads to cadvisor, which is the only component that uses the in the repo: So I'm convinced that this should be related to and will update the thread once I've made further progress.
EOD updates: issue is indeed caused by cadvisor, reproduced the issue in this . confusing timeout in log should be . root cause might be a bug for using context that already exceeded deadline in cadvisor. To validate that, I need to explore all methods of this : Will work on the fix on cadvisor for the rest of the week.
Fix was made in Copying from to ensure we can capture it on the issue here: I've cut new cAdvisor releases (v0.37.3 and v0.38.6) with this fix cherrypicked. We should get the cAdvisor fix back into kubernetes. We'll need 3 PRs to k/k with following updates: [x] k/k master to update cAdvisor to v0.38.6 -- [x] k/k release-1.20 to update cAdvisor to v0.38.6 -- [x] k/k release-1.19 to update cAdvisor to v0.37.3 --
Thanks for fixing this one, definitely a tricky one to track down.
Thanks for all the patience and team efforts on getting this issue fixed, also credits to on the guides and discussions, won't be able to get it done without your helps!
The fix was applied on master as well as cherrypicked into and releases.
I remember you told me, but I forgot. Does issue exist in 1.18 and 1.17?
I don't believe 1.18 or 1.17 should be affected. I think this issue was introduced with which commit was included in cAdvisor v0.37 release. cAdvisor v0.37 was vendored into k8s 1.19.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Windows nodes running with high (90%) CPU usage are having calls to /stats/summary time out with "Context Deadline Exceeded". When this happens metrics-server cannot gather accurate CPU/memory usage for the node or pods running on the node which results in HPA not functioning returning <unknowninstead of values What you expected to happen: Windows nodes should be able to run with high CPU utilization without negatively impacting Kubernetes functionality How to reproduce it (as minimally and precisely as possible): a cluster with Windows Server 2019 nodes CPU intensive workloads into cluster ex: workload to put node into high CPU state node metrics with Anything else we need to know?: This issue appears to be easier to reproduce on 1.18.x nodes vs 1.16.x or 1.17.x nodes Ex: ! ! metrics-server logs show E1020 10:58:24. 1 ] unable to fully collect metrics: unable to fully scrape metrics from source kubeletsummary:akswin000000: unable to fetch metrics from Kubelet akswin000000 (10.240.0.115): Get https://10.240.0.115:/stats/summary?onlycpuandmemory=true: context deadline exceeded Kubelet logs show and Environment: Kubernetes version (use ): 1.18.8 Cloud provider or hardware configuration: AKS / Azure OS (e.g: ): WIndows Server 2019 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: Docker EE version 19.3.11
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows /area kubelet
I was able to reproduce on 1.18.8 during a scale operation: where it fully subscribes all CPU's in Allocatable: Metric server logs: I do not see "context deadline" in kubelet but I did find some logs assocated: Once the pods are scheduled the node remains at 100% cpu and I no longer see errors but is not accurate: on the node (run after the pod deployment stabilized)
Adding to kubelete's flag and I no longer see the errors. This reserves some cpu for kubelet and other processes. We call this out in the but do not do the same for cpu.
Also, without setting system reserved CPU, it looks like does not accurately reflect CPU usage on the nodes when CPU utilization is very high. When I got some nodes into this state I did not see any error messages in logs from metrics-server and also I noticed that that querying the metrics API directly from showed recent timestamps.
I run into this again during scaling, collectiong logs for analysis here: Metric Server logs: Kubelet logs:
have you experienced this at all?
I believe has seen high CPU issues before. Not particularly on this. I've heard there were kernel issues in the past that triggered high CPU cases.
Some investigations on our end are showing that the DiagTrack service running in the infra/pause container can consume non-trivial amounts of CPU. I'll create an issue to address that problem as it might help out some here. That certainly will not eliminate the problem we are seeing tho.
is an issue for DiagTrack
The differences in behavior between 1.17. and 1.18. clusters looks like it is caused by (which was NOT backported to any prior releases) K8s version <= 1.17 behavior Prior to these changes CPU weights for containers were inferred based on limit values set in the pod specs. The observed behavior on Windows system when CPU weights were used was that under high-CPU load pods/containers were not guaranteed their scheduling time. The good news here is that system critical services like kubelet/docker/etc were still able to get enough CPU time for the nodes to function properly. K8s version = 1.18 behavior After these changes CPU limits were enforced for pods. The observed behavior on 1.18 clusters is the when a Windows node enters a high-CPU state is that containers are guaranteed their scheduling time but host resources start to get starved. Usually this manifests as the node entering the state, metrics not working correctly (because calls to /stats/summary are timing out) or both. Suggestion / possible fixes Set the Windows process priority of the kubelet to  to help ensure it gets CPU time Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) Set reserve CPU with with --system-reserves=[cpu=500m] This needs to scaled based on number of CPUs on host based on my experimentations Currently guidance on for Windows only mentions reserving system-memory It is still possible to over-commit CPU and starve kubelet/docker/HCS if users schedule pods with mis-matching requests and limits Update documentation.
Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) I can work on this, I am wondering if we should do the same for CRI(like docker) and CNI processes.
Sure, if you want to work on this go ahead. I think we should probably try to make this be toggleable via a kubelet flag. I believe there are other windows specific flags already implemented in the kubelet (like one to run as a windows service IIRC)
docker/containerd should probably have a flag that allows them to run as high priory as well but would likely be different issues in the respective projects
/reopen I don't think this was ever fully addressed
Reopened this issue. <detailsIn response to : /reopen I don't think this was ever fully addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
also related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In k8s 1.19.2 If the seccomp annotations of psp doesn't contain , pod with seccomp annotations would not start with error: psp with works psp without doesn't work Is this due to ? What you expected to happen: In k8s 1.18, psp annotations without works. I think in k8s 1.19, it should also work.
/sig auth
/assign I think this was a regression with the seccomp graduation to GA. Since we auto-convert 'docker/default' to the runtime default in fields, but PSPs that only allow 'docker/default' aren't accepting 'runtime/default'. I think the easiest fix is to just add both and to the allowed profiles map whenever either is set (here: ) This will need to be cherry-picked /priority important-soon
I'm happy to work on this one tomorrow if is not tackling the issue earlier :upsidedownface:
Fix is incoming in
Thanks for the quick fix ! Can you open a cherry-pick for v1.19 too? (see )
Forgot to link it, but this is already done: :)
Is there an easy way to check in which version of k8s this fix is applied? Label, milestone, etc? Another question, what might be a workaround without updating cluster?
Find the fix commit and run And there's also a cherry-pick commit: The workaround could be add annotations to your PSPs, take a look at this PR:


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: - Also flaking on: Which test(s) are failing: Since when has it been failing: ~Since Oct 21? Testgrid link: CAPG conformance: CAPA conformance: gce cos master default: Reason for failure: Example spyglass link: Anything else we need to know: Triage shows that this is happening across several jobs (not just the ones I listed above) and the spike started happening earlier this week. /sig cli /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Fixed by
This is affecting ppc64le jobs as well.
/remove-kind failing-test /kind flake This is not a continuously failing test, it is instead flaking
/priority critical-urgent the top flake for per /area conformance this is a conformance test, it definitely shouldn't be flaking, and should be demoted if it can't be fixed
/reopen For CI signal to observe before marking as resolved
Reopened this issue. <detailsIn response to : /reopen For CI signal to observe before marking as resolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Last failure 7 days ago 2020-11-03 05:40:32 +0000 UTC
Please review to see if we can close this now.
Let's close this - also looks like it is no longer flaking as well


Instead of relying on the clock: we should do something similar to the golang rate-limitter is doing:
/sig api-machinery /priority important-longterm
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, I would like to help on this.
/assign Thanks! /triage accepted
can you please provide more context or some guidance on how to proceed?
Sorry for delayed response. - what we should do is to basically: instead of having an additional go-routine that every 1s refreshes the budget change it roughly to: (a) keep the value of budget and the last time when we were refreshing it (b) on takeAvailable method: first recompute takeAvailable, which will be "min(current budget + time elapsed since last refresh * refreshValue, maxAllowedBudget) update last refresh time to now return that value [We also would need to update the existing tests to match the new logic.] - would you be willing to work on it?
-- I can take this up. One question I had was should the method rely on or the current time should be passed on by the caller like in the following: Having the caller pass the time completely removes the dependency on the clock at least for the implementation.
/assign
Let's may be start with: passing on budget creation to it relying on
-- thank you for the thoughts. I drafted a rough implementation in which still depends on . I will make the change as you suggested and rely solely on a instance passed on initialization. I have to rethink the testing strategy a bit, possibly using a .
Yes - FakeClock is the way to test this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopening - this didn't happen.


/sig storage What happened: When you interrupt the e2e storage stress test, namespaced objects are cleaned up, but non-namespaced objects like StorageClasses are not cleaned up. What you expected to happen: Non-namespaced objects like StorageClasses are cleaned up so long as the user doesn't interrupt a second time. How to reproduce it (as minimally and precisely as possible):
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


What happened: ExecUtil in has mandatory logging side effect hardcoded to INFO What you expected to happen: Since INFO level logging may or may not be an implementation detail of a test that is worth showing all the time, it would be better if we can turn this logging off optionally with a Quiet (false by default) boolean field How to reproduce it (as minimally and precisely as possible): Run any test that execs across pods, i.e. exec tests
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With a /22 pod network and a /24 assigned to each node, a third node cannot get a pod CIDR if kube-controller-manager has ever been restarted. What you expected to happen: Once a third or fourth node joins, it gets a /24. How to reproduce it (as minimally and precisely as possible): 150% usage on the IPAM controller is not a good sign. Anything else we need to know?: During the creation of the rangeAllocator, Occupy() is called twice on the pod networks. The first time is before subscribing to the node informer. The second time is during the node informer callbacks because on subscribing it calls Add for all existing nodes. I will submit a proposed fix shortly. Environment: Kubernetes version (use ): 1.19.2
/sig network
/triage accepted thanks for finding the bug I 've introduced :)


What happened: Test create method for non-namespaced subresources generate invalid code. I have the following code generation annotations: which generated test method with generated fake client: What you expected to happen: It should be: How to reproduce it (as minimally and precisely as possible): Generate clients for non-namespaced resources with subresources (like scale) like this: /sig api-machinery /area code-generator /assign
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: What you expected to happen: It should be: How to reproduce it (as minimally and precisely as possible): Generate clients for non-namespaced resources with subresources (like scale) like this: /sig api-machinery /area code-generator /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted


When configuring new behaviors section in HPA I two scenarios that I think are incorrect. First one: stabilizationWindowSeconds set up to 5 minutes for both scale up and scale down min replica count 2 Application consumes big amount of resources during warm-up 0 minute: current replica count: 2 1 minute: desired replica count: 3 - no scale up stabilization window worked 2 minute: desired replica count: 1 (!) - scale up to 3 (max number from previous recommendations during stabilization window period) ! Second one Stabilization window for scale up 0 Stabilization window for scale down 5 minutes min replica count 2 behavior set to scale up new pod in 2 minutes scaleUp: policies: type: Pods value: 1 periodSeconds: 120 What happens: 0 minute: 2 replicas 1 minute : performance spike scale up to 3 (desired value 4) 2 minute : no spike no scale up 3 minute: no spike scale up to 4(!) - supposedly desired value 1 and supposedly worked this way because of scale down window set to 5 minutes picking max value from that 5 minutes window that was 4 replicas Expected behaviors 1 Scenario no scale up for 5 minutes 2 Scenario correct measurement and no scale up when there is no need for it Environment: Kubernetes version (use ): client 1.19.3 Server 1.19.0 Cloud provider or hardware configuration: Azure aks
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
/assign josephburnett
The HPA controller keeps a flat history of recommendations for stabilization. ! When downscale stabilizing the maximum recommendation is selected. Purple is pre-stabilized recommendation. Green is final recommendation. Blue is max recommendation within window. ! When upscale stabilization the minimum recommendation is selected. ! However when both up and down scale stabilization is configured, the interpretation of the history changes depending on the direction of movement. Red is the undesired, actuated recommendation. ! What we want is to keep the stabilized recommendation within the envelope of the minimum and maximum over configured stabilization windows. We should only move when the envelope forces a move. ! We first stabilize up and down, and check if the current value fits within it. If it does (or the range is "inverted"), we do nothing. If it does not, we emit the "nearest possible" value, which will be lower or upper bound, depending on where the current size is in relation to the range. (thanks and for the paragraph above)