<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I change default hugepage size 2Mi to 1Gi and reboot, kubelet can't update node status What you expected to happen: kubelet can update node info and node status was ready How to reproduce it (as minimally and precisely as possible): change the node condition validate Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
Hi, Ref. the error message, it looks like you have allocated hugepages for multiple sizes, . Can you run: to verify? If more than one of the numbers is , that is the reason. This is currently being discussed in , and the reason for implementing the check is here: Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size. Because of how it is implemented, adding support for allowing a pod to request huge pages with multiple sizes isn't trivial, and will require some potentially breaking changes, but that is not relevant to this problem. /cc
Also, can you also provide the result of , that will list the supported huge page sizes (both when is set, and when it is not.).
Thanks ! Looks like your kernel only enables huge pages with the default size. My kernel 5.0+ enable all (atleast 2MiB and 1GiB on x8664) by default. When the node booted with defaulthugepagesz=1GB it "removed support" the 2MiB one, and becuse kubelet use patch on the node status when connecting to the api server, you ended up with both and , even tho 2Mi wast supported by your machine (the sw, not the hw). Even if the error is removed, this will result in a crash when kubelet tries to write to the 2MiB hugetlb cgroup files. I think using should work, otherwise the only current solution is to remove the node from the cluster, and rejoin afterwords. Ref. it looks like you have sorted out the problem, but I still think this is a valid issue, so don't close it. The correct behaviour for kubelet is to remove unsupported sizes, and only allow use of the supported ones.
"Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size." Could you fix this one as well? We use 2M pages only for pods but we need 1G pages for system processes. So it was little surprising when k8s doesn't start because two different pages were configured no matter they are really used or not. Thanks!
I am hitting the same, I used tuned to change the default size from 2Mi to 1Gi, i hit the same problem, after the reboot node is NotReady and it cannot rejoin , giving the error: Node "test-worker-0" is invalid: [status.capacity.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"20Mi", Format:"BinarySI"}: may not have pre-allocated hugepages for multiple page sizes, status.allocatable.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, My system just reports information for 1Gi, not 2Mi anymore. Removing the node and rebooting fixed it, the node rejoined again reporting the right capacity. But this is not a solution for us, changing sizes should be supported.
Hitting same K8s installed with my ubuntu 18 enabled 2Mi hugepage by default. As I change my master node's hugepage to 1Gi, this happened. I have only one master, so I have to rebuild the k8s cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm a little confused. when i used tuned to change the hugepagesz from "defaulthugepagesz=2M hugepagesz=2M hugepages=8192" to " defaulthugepagesz=1G hugepagesz=1G hugepages=16", i hit the same problem, after the reboot node is NotReadybut when i used tuned to change the hugepagesz from " defaulthugepagesz=1G hugepagesz=1G hugepages=16" to "default_hugepagesz=2M hugepagesz=2M hugepages=8192", after the rebootkubelet status can still keep Ready. why?
Hi This will be fixed in , but it is still not merged. Hoping to get it into next release. In order to solve the issue (and use 1GiB huge pages) you have to delete the node in kubernetes via , and join the node again.
Thanks
Thanks
I think this is already fixed by can you confirm?
Hi WIth this would not cause a crash, but is still required in order to fix this properly.
I am facing the same issue. After changing the hugepage setting to . I stopped the kubelet process on the worker node and removed the node from the cluster using command and rebooted it. But after it comes back online and kubelet starts, it struggles to join the cluster . Do I have to run the join command again ?


/assign /cc /sig instrumentation
GitHub didn't allow me to assign the following users: serathius. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc /sig instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey Looks like this test is passing on testgrid, https://k8s- Which would point to problem being related to specific environment. From your log looks like problem is specific to Macs. I will try to reproduce it to confirm it. My current ideas for solution are: Disabling test on Mac Instead of passing files as arguments, pass them in STDIN WDYT?
FWIW ARG_MAX can technically vary even on linux ...? STDIN SGTM


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-enabled-default Which test(s) are failing: CSI hostpath-v0 tests Since when has it been failing: 8/8 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: We are removing CSI 0.3 support in 1.18, which is before selflink is going to be removed in 1.20. So we should just disable hostpath-v0 from alpha tests. /assign
/milestone v1.16


/help /kind cleanup /area e2e-testing-framework mini guide: make sure you have signed the CNCF clone kubernetes;/kubernetes search for ginkgo.Fail calls under test/e2e/framework e.g. replace them with this that was in this PR: send a PR for kubernetes/kubernetes
/remove-kind feature /area e2e-test-framework
/sig testing
I will take this.
thank you! Please feel free to ping #testing-commons in the Kubernetes slack if you have any questions :smiley: /cc
/remove-help
/assign


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Several tests Eg. this run: Since when has it been failing: Unknown Testgrid link: Reason for failure: (happens to all the images). Pods in the e2e test suites use pull policy , , and it looks like the test during DiskPressure, resulting in kubelet removing some of the pre pulled images. Anything else we need to know: The runs after each test. This should also be the case in the test too.
/assign /sig node
These tests are now constantly failing due to missing images. This makes it hard to see if some of the stuff we do now, or have done in the last few weeks, breaks some of the tests. PR should hopefully fix this, but it is not in the 1.16 milestone, but has and . Should we get this merged into master so that we can see if if fixes the problem, or should we wait until code thaw? Any thoughts? /cc /cc /cc
/priority important-longterm
Thanks for all the work you've put into this ! I think we should add to the milestone
+1 to - thanks for your hard work :)
still has a lot of failing tests. What are the next steps? /reopen
Reopened this issue. <detailsIn response to : still has a lot of failing tests. What are the next steps? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, it has over the few last weeks become quite a mess... Looks like fixed some of the issues about image eviction problems, but there are still a few left. I think I have a solution, so ill try to submit a ~PR in a few hours~ edit: will do it tomorrow. A lot of new tests started failing 8-28, and that may be related to , but I am not totally sure..
Looks like fixed the flaking in ! The last 7 runs was executed after that was merged! ! We still have a lot of failing tests, but those are not flaky; they are failing (ref. comment above). Ill create a new issue about all the failing tests where we can track them each together. Closing this issue, as this was only filed for the flaking becuase of the evicted images. :smile: /close cc
Closing this issue. <detailsIn response to : Looks like fixed the flaking in ! The last 7 runs was executed after that was merged! We still have a lot of failing tests, but those are not flaky; they are failing (ref. comment above). cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


There are multiple cert controllers that use the same base struct, so when they log, you cannot differentiate which is which.
/sig auth


MaxInt const is exported in scheduler api although it is not needed to be exported. It is only used by . discussed that the common practice is to have API constants in /sig scheduling
cc
The problem is that it is part of a v1 API :(
Ok, it turns out they are not actually considered part of the API, and so we can remove them:
Should I open a PR to address removing those const values from ?
Yes, we can remove those in a followup PR to the one handling sized values.
Yes, this was the plan. /assign
remember to remove as well :)


Which jobs are failing: ci-kubernetes-node-kubelet-orphans Which test(s) are failing: [sig-storage] EmptyDir volumes pod should support shared volumes between containers Since when has it been failing: Testgrid link: Reason for failure:
/sig testing
/assign
Thank for comment
Tests are now passing, /close
Closing this issue. <detailsIn response to : Tests are now passing, Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


I tried to use a local volume block device in minikube. Its trying to call losetup with the -j option and its failing as the losetup in minikube does not provide this option. This may effect other distro's as well. The end result is lock block devices don't work on the platform.
/sig storage
Probably caused by this code:
Did a strace on losetup -j. looks like the same info can be had by: cat /sys/block/loop0/loop/backing_file
Just had the same problem with Ceph cluster creation in minikube, the osd-prepare doesn't start because of this:
FYI: has the same problem in minikube. Replacing the /sbin/losetup busybox link with a "real" losetup helped for now, but isn't a solution of course.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help </details


<!-- Please only use this template for submitting enhancement requests --What would you like to be : In block volume tests, wait for the Pod to be deleted instead of letting it be handled by the framework's namespace deletion Why is this needed: I think the regression caused by could have been caught if the tests check for Pod deletion because the Pod object shouldn't be deleted if TearDown fails.
/assign
This issue is not specific to block tests. We should in ideally wait for pod deletion at the end of all types of tests to confirm cleanup logic is functioning
I checked the tests under test/e2e/storage/testsuites/ where DeletePodWithWait is used to wait for the deletion.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten is this still an issue? One way to check is to make a breaking change in cleanup and see if the pull job fails.
I went through all block volume usage in e2e tests: All of them but one wait for their pods to be deleted. I'm fixing the remaining test in


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: jobs primarely, like: Which test(s) are failing: [] Container Runtime blackbox test when running a container with a new image should be able to pull image from [NodeConformance] Since when has it been failing: Since this commit merged: Testgrid link: Reason for failure: The test should always pull the image from (), it is not configurable. Anything else we need to know: /sig testing /sig windows /area conformance This issue was fixed once before:


<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add a metric to track number of scheduler binding goroutines. This will be a generic metric with as the only label for now. metric name Description -- -- scheduler_goroutines Number of running binding goroutines Why is this needed: Helps us track how many binding are running and monitor issues such as goroutine leak. /sig scheduling
/assign


In order to initialize a client-go , you need an . and friends initialize a new : launches a goroutine: This goroutine is stoppable with . However, the interface returned above does not expose anything that calls that shutdown method, AFAICT. This means that that goroutine cannot ever be cleaned up. That's not great. It's fine if you're using in like k/k does and never stopping anything, but if you ever want to stop your controllers and start up new ones, it becomes an issue. Tested/tracked down against master (as of ). /sig apimachinery /kind bug
The label(s) cannot be applied. These labels are supported: <detailsIn response to : In order to initialize a client-go , you need an . and friends initialize a new : /kind bug </details
/sig api-machinery I always forget the dash


A significant number of the of are commented out, possibly was done by mistake in the past. I uncommented the tests to see if they pass, but one of them failed. What we need is to uncomment those tests, fix the one that is not passing and remove the redundant ones if there are any. /sig scheduling /help
/assign I will be moving some of the tests to unit tests and leave a cleaner set of integration tests.
/remove-help
would you like to take this? I can provide guidance.
/help /unassign
/assign
/remove-help


Which jobs are failing: Which test(s) are failing: Since when has it been failing: 18 Oct 13:17 PDT Testgrid link: https://k8s- Reason for failure: check
/sig network /cc /cc
/priority critical-urgent /milestone v1.17
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I believe this is caused by this merge cc.
Most of this is caused by and should be fixed by . Is there any way to run these tests against that PR? /remove-triage unresolved
The job is still failing. Can you check?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage unresolved
OK, fixed some of the tests, got the rest to the point where they failed differently...
Thanks for the update. In testgrid we still see the same error. We will keep observing the next runs.
- still failing. This time the error is We probably need to clean up sanityChecks a bit more - this seems to be a similar issue.
yeah, I'll temporarily un- the tests in my next PR so I can make sure they're actually fixed this time
After the last fix, it seemed to finally recover.


What happened: During the work for reducing invalid dependencies from the e2e core framework to subpackages of the core framework, we found a lot of particular test specific functions. That means these each function is used at a single e2e test module only. These functions are unnecessary to be shared as e2e framework function, so we've just moved these functions to the particular test module like This cleanup work will help us to maintain the e2e core framework in long term, so it is helpful to share this work in volunteers as a community. This issue is for tracking this working items. How to do that?: volunteer selects a single file which is empty (no PR, no assignee) from the following e2e core framework all functions in the file all callers of each function finding the function is called at a single e2e test package only, let's move the function to the module with a PR Implementation rules: If a function is called from 2+ test packages, don't move the function from the core package. If moving of the function requires invalid code dependencies, don't move the function. (Dependency from e2e core framework to other repository) List of files of e2e core framework: [x] ( ) [x] () [x] ( [x] () [x] [x] () [x] get-kubemark-resource- [x] ( ) [x] (Done) [x] ( ) [x] () [x] ( ) [x] ( ) [x] ( ) [x] ( ) [x] ( ) [x] ( ) [x] () [x] ( ) [x] () [x] () References: This working item was under discussion as
/kind cleanup /remove-kind bug /area e2e-test-framework /sig testing /priority important-longterm
Thanks for taking care of this quickly :-)
The proposed policy to follow for what methods are to be kept within the framework: What do you think ? does that sound good?
Thanks for bring it up here. As I said on the slack, I agree with his idea.
I looked through files , and , and found some of the functions are used in a single module. Then fixed them.
i will pick up
I will pick
I will pick up
/assign I will pick
i will pick and
I will pick up ,,,
The PR for is already created by . Please check what other people picked already before posting a PR.
is already picked/posted by
That is a nice point. I wrote point as one of the above Implementation Rules, and another rule comes from which adds another invalid dependency.
I didn't noticed that already picked by , sorry about that. After looking around, there is noting need to do with: ,,,,(not picked). I will pick the left: . Found that is also no need to move, is the same.
Thanks for your investigation, I've updated the above list based on your input :-)
I think is does not need to be modified, Cloud you please help me confirm it?
and
There is noting need to do with , and I have filed a PR for
Thanks for your help :-) I've updated the above list based on your input.
Thank you also, I approved the PR and updated the above list from your work.
and do not need to be modified. Could you help me confirm it? THX.
I think there is noting need to do with
Thanks for your work, I've updated the above list based on your input.
I checked but there is no func which is called from less than 2 test packages.
Thanks for your investigation, I've updated the above list based on your input.
Nice!
Yes, thank you so much everyone!! :-)


Minimal Repro: Create 2 different disks Create 2 Pods each using one of the disks Start both pods on the same node You will notice one of the disks gets ed to a path: , then both disks are ed from to a target path: which is similar but at least uses the pod . Both of these volumes are being treated as the same volume in the Kubelet even though they are definitely not the same volume. Note: it may not be "treated" as the same volume - it may just be that since the paths are the same the following volumes see that the path is already a mountpoint and skip the step. This is the more likely scenario based on observations. The value is the "name" of the inline-volume. These can obviously be the same across different pods - we should actually be placing it in a more unique directory (based off the translated maybe?). /cc
/sig storage
The inline volume example I'm using (ignore the fact that is a complete misnomer): I've confirmed that changing the "name" makes inline volumes work well since there is no longer a collision on the paths
/assign
Alright, problem is sort of twofold: The referred to above for inline-volumes is set here: As we can see from the comment: it is set to the because the in-tree CSI Plugin makes a lot of naming assumptions based on the which for PVC/PVs is the PV name (unique in a cluster). Primarily, during unmount the CSI Plugin tries to from a "well known ": which happens to be some path based on the which for the inline volume is just the ( above). So we run into the problem that if we use the directory is not sufficiently unique for different disks. And if we use something else like then the unmount pathway is unable to find the . I think using is a no-go. So we need to use something sufficiently unique (like ) for the ( above) and fix up some of the other codepaths to either use different "well-known paths" to write to, or to fix up path generation code so they give us the correct locations. I'm also going to check to see whether 's refactor fixes this issue (because it might since it translates things earlier)
OK. Really good news. your refactor in its current state actually fixes this issue if we set the to something unique here: Looks like the early translation is also making the pathway from the "correct" dir for inline migration. So I will create a PR that fixes this issue that should merge after


What happened: We were running a service with LoadBalancer type with AWS cloud provider and after deleting this service, the cleanup process on AWS removed a wrong ingress input rule which provoked that all traffic between nodes was unable to reach out the destination. The service we were running had a configuration similar to: After applying this manfiest on the cluster, the cloud provider created the associated load balancer with two security groups: sg- (due to extra security group annotation) sg- (a new SG created by the AWS cloud provider with the rules defined on the ) Cloud provider also created a new input rule on the nodes security group which allows all traffic from all ports from (which is the sg created for the newly created load balancer in our example). One particularity of our setup is that the security group we as was the nodes security group. The input rules of this security group after the creation of this service was similar to: Everything worked fine with this setup until we deleted this service and the cloud provider started the deletion process of the LoadBalancer on AWS. We've been debugging the deletion code and we found that the first thing it does is cleaning up the security group. To do it, it first removes the input rule from the nodes security group which allows traffic from the load balancer security group (sg-, on this example) and then, as it's not referenced, it removes the security group. The expected output of this process, would be that nodes security groups would be: But, in our case, it ended up as: which dropped all the traffic between nodes (as it didn't allow any traffic between them). The cleanup process didn't properly remove the input rule from the node security group (the one which allows all traffic from source sg-) and it also did not remove the sg- security group. In our case, as we were using the nodes security group on the extra security groups, this bug provoked a network partition between nodes, but we think that if is used, the cleanup process may be broken and have unexpected side-effects like those we observed. We've been debugging and we found that the process which removes the input rule from the nodes security groups starts here: The first thing this function does is this: : When extra security groups are used, will contain the security group created for the LoadBalancer and all the extra security groups. We also found that AWS returns this SecurityGroups list in distinct orders after changing the LoadBalancer configuration. That code is always assuming that latest security group on the list is the , which may not be true when extra security groups are used. On the cases where the load balancer security group is not returned on the latest position, the cleanup process does not remove the security group and the input rule from the nodes security group is not removed as expected. We've seen that master branch still contains this code and we think this may be hapenning on newer versions: ), so we think that newer versions may be affected by this bug (we didn't try it). In our case, we saw several times this log during the removal process: Additionally, the code tried to remove the security group (but being unable to do so), after wrongly revoking the rule we have talked about before. What you expected to happen: The cleanup process should properly detect the load balancer security group, remove the input rule from nodes security group and remove the load balancer security group, regardless of the order in which AWS returns the security group list on . How to reproduce it (as minimally and precisely as possible): We did not find any consistent process to create the scenario, as it happens randomly (we are not sure if due to AWS or any kind of race condition on AWS cloud provider). Setup: Create a Service with running controller manager with AWS cloud provider with tha annotation containing one or more security groups (as the one the section) Wait until load balancer is created Run Check if does not have the newly created security group for the balancer in the latest position. Repeat until the balancer does not have the balancer security group on the latest position Trigger the bug: Result:The log will appear multiple times on controller-manager logs Load balancer associated security group is not removed Nodes security group input rule allowing all traffic from load balancer security group is not removed If the latest security group on the list was the nodes security group, the input rule allowing all traffic on all ports from all the other nodes is removed and no traffic is allowed between nodes on the cluster. Environment: Kubernetes version (use ): version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: aws OS (e.g: ):Kernel (e.g. ): Linux ip-10-2-13-67 4.9.0-7-amd64 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x8664 GNU/Linux Install tools: kops
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area provider/aws /lifecycle frozen
Working on a repro for this issue (on 1.11 first, then I'll try on master) and I've not quite observed the same behavior yet. Depending on the order of the security groups on the load balancer, I only see the ingress rule for the new LB security group to the node security group if the new security group comes last. Otherwise, if the NODE security group comes last, the ingress rule is not , and I see in the logs. Even without a full repro yet, I find it troubling that behavior is dependent on the order of security groups attached to the load balancer. One question I have related to your setup, can you describe the reason why you're adding the node security group as an ? I'd like to understand your use case a little better.
Hi Nicholas. I have a working patch for this specific issue: This bug was the subject of a talk for a local meetup I did recently. I also published a personal blog post talking about it: There you can see more details about how it can be reproduced easily. If you think the patch is valid, I'll try to add a few test cases for it and then propose the change upstream. Thanks!
It has been a while since we got this error and I'm not sure if we saw this error. What I experienced is that the order is changed by Kubernetes or by some call to AWS. It took me not more than 3 attempts to get this behavior. There's something I can do to help you reproduce the error? Do you need more details of our setup? This was a misconfiguration on our side. When we noticed that this setup was not correct we removed this security group from , which removed the mentioned security group rule and took down the Kubernetes cluster (no communication between nodes was allowed).


cc who found it. the reason is the calls from the import here: i suspect the problem surfaced with this bug fix. cc more refs: is an anti-pattern in this case. ideally metrics registration should not trigger warnings for a simple version output. mitigation on the user side is to pipe stderr to a null device (as long as klog is writing to stderr) /kind bug /priority important-longterm /area kubelet /wg component-standard
/sig node
should be fixed before 1.17.0, doesn't block beta.2
cc
Could this be caused by the following in init() of ? It was marked I wonder when deprecatedRequestLatency can be dropped.
looks like was deprecated in 1.14 - Is that the only one that needs to be dropped? are there more.
Dropping restclientrequestlatencyseconds should remove the warning for --version. In the 1.14 changelog, there were quite a few other metrics deprecated. I guess we don't have to drop them now.
/assign It's an interesting issue. I will investigate it and get back to you. By the way: I tried this morning, I got 4 warnings, things getting worse as we deprecated more metrics this weekend. (the log can not point the exact name of a metric, we intended to improve it in v1.18)
it's being invoked from here: the message is not helpful, since it doesn't tell you which metric is deprecated
Yes. It's a known issue and we intended to improve it in v1.18. Now, I suggest removing the log temporarily, Fired a PR , for reasons as follows: Too many binaries have been affected.(kubelet, kube-proxy, kube-apiserver, kube-controller-manager, kube-proxy, kube-scheduler). We should consider more about this issue. v1.17.0 is fast approaching, we should make a quick fix. All components try to register a deprecated metrics will be affected. Maybe we should record deprecated/hidden metrics by a new metric instead of the log.
Summary the root cause here: According to metrics deprecated policy, if a metric's - = 1, it will become a hidden metric. Hidden means we will not register it to the Prometheus registry by default. If we try to register a hidden metric to the stability framework, the warning log will be printed out. Because some metrics registered by init() calls from the import, so...
I have tried to print the metric's name at local: kubelet: kube-proxy: kube-apiserver: kube-controller-manager: kube-scheduler:
ideally the registration should happen outside of init(), which means calling it on demand. but i will defer to the owning sigs.
In principle, yes, I agree. Otherwise, we need to be very very careful with every log in the stability framework. I think we can track it and work out by iteration. cc for comments.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When developing unit tests related to PDBs/Deletion, mutates pod resource via adding deletion timestamp, and this directly conflicts with checking resource version. What you expected to happen: No resource version mutation. How to reproduce it (as minimally and precisely as possible): I'm attempting to add preconditions to delete call in eviction API: Anything else we need to know?: Happens locally and in CI. Environment: Kubernetes version (use ): Head of master: Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you provide more details about what you're observing?
You can see an example here: Here's the relevant section of the tests: There really shouldn't be anything modifying the pod after creation until the delete request, thus I don't believe we should be hitting this precondition failure. Resource Version in precondition I believe to be correct, but when we contact etcd to verify, it's increased by one. It's not clear why this is happening, it appears to be a bug.
Okay, I've sorted this out. We have a conflict with 'graceful' deletion with this approach: This updates the object to add a deletion timestamp, thus the first call to eviction will always fail if we need to set ResourceVersion.


Was just perusing the code and looks like we translate regional disks into the "zonal volume ID fmt", and there doesn't seem to be any tests for regional disks running so we should confirm that this works and fix it if it doesn't /assign /cc
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen /assign
/assign


<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should allow collecting both default scheduler metrics (Prometheus) and custom metrics easily. At a high level, these collectors should produce a json formatted result which will then be consumed by perf-dash to present in the dashboard. Prometheus metrics We should provide a metrics collector that collects and summarizes default scheduler metrics defined in . To begin with, we can manually pick a list of interesting metrics. Longer term, we could allow configuring the metric list in test definition. already had a PoC . As a reference, is how clusterloader collects scheduler latency metrics. Custom metrics The most important metric is scheduling throughput. is how clusterloader calculates the throughput. We can start with just calculating the average throughput without the percentiles. We may want other custom metrics that are either common to all tests or specific to certain tests. We should allow these custom collectors to be easily and ideally configurable. Why is this needed: Give more insights into scheduler performance. Part of /sig scheduling /assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [] [sig-node] Pods Extended [] Delete Grace Period should be submitted and removed [Conformance] Since when has it been failing: The ipv6 jobs have flakiness, but since I had 3 flakies in a row with this test Testgrid link: (IPv6),%20master%20(dev)%20%5Bnon-serial%5D&width=5&sort-by-flakiness= Reason for failure: It fails because is not able to delete de pod in 30 seconds, seems that this is the cause of that Nov 30 07:41:07 kind-worker kubelet[677]: W1130 07:41:07. 677 ] Failed to process watch event {EventType:0 Name:/docker/f35390ea97c6bae92f61d7d297fe7eacb4cfb3f2effafb23f3aca00e1a9d2341/kubepods/besteffort/podd8d966f8-2a03-4529-9d70-/243828b3fed0edf1137a9fa5ade6ffbbe468b79bbf212206480bbd37ab20fc7d WatchSource:0}: task 243828b3fed0edf1137a9fa5ade6ffbbe468b79bbf212206480bbd37ab20fc7d not found: not found Anything else we need to know: The job view seems to report the same test failures twice but the log output shows different timestamps
/sig node
/cc
interestingly, the serial ipv6 job doesn't look flaky: (IPv6),%20master%20(dev)&width=5&include-filter-by-regex=Delete%20Grace%20Period
this one is failing quite a lot in the 1.17 windows job - could someone confirm if it's a bug or a test issue? Thanks
or is it actually a completely different failure on windows? Since it's being tracked here as well
will close this issue because the patch addressing the issue was merged and to avoid having duplicates We can reopen if necessary /close
Closing this issue. <detailsIn response to : will close this issue because the patch addressing the issue was merged and to avoid having duplicates We can reopen if necessary /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
by the way, this one is unrelated to the windows one :-)
ACK, thank you :)
Two failures seen on after the related PR merged, that was only a cleanup, not a root cause. /reopen
Reopened this issue. <detailsIn response to : two failures seen on after the related PR merged Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind failing-testing /kind flake
Those labels are not set on the issue: <detailsIn response to : /remove-kind failing-testing /kind flake Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind failing-test
I think that I'm making progress trying to deflake the ipv6 tests I have submitted PR to fix the portmap plugin issue that should be able to improve the situation
Retitling because I don't think this specific test failure is IPv6 specific anymore
triage: flaking in pull-kubernetes-e2e-gce: /priority important-soon This is the fourth flakiest test for pull-kubernetes-e2e-gce
debugging in
bumped priority, a conformance test must not be flaky. this is now the top flake in the e2e suite.
cc
debugging log messages merged 2019-12-19T08:38:10Z, will check for output in next failure
debug output from failure:
debug output from success:
failure in
based on my reading of this, the timeline was: 00:02:23. - we got a pod running 00:02:23. - deleted it with a 30 second grace period 00:02:54. - the kubelet finished handling the delete request (30.18 seconds) 00:02:54. - test failed with a timeout (30.25 seconds) That seems like extremely tight tolerances, especially since the grace period refers to termination of the pod process, not removal from the API. I would be inclined to allow up to 2x the grace period to observe complete removal from the API.
opened to allow time for the kubelet to remove the terminated pod from the API
top flake according to
Opened to tag as [Flaky] and kick out of [Conformance]
Test is no longer run in blocking presubmit. - who owns resolving this and reintroducing the test before 1.18 is released? Investigation with combined timeline at shows that the kubelet stopped the container and destroyed it within ~15 seconds (observed deletion at 06:20:11., destroyed container at 06:20:46.), but then took another 30 before actually deleting the pod from the API (06:21:12.).
To evaluate the effectiveness of a potential fix, I would recommend opening a PR that removes the tag along with the proposed fix, and repeatedly on that PR to accumulate 10-20 test runs.
/assign assigning sig-node leads for visibility/routing
/milestone v1.19
/milestone v1.18 we need an ack from sig-leads whether this is release blocking or not
This issue is opened for 2 releases in a row. I guess the flake is not a release-blocking. In addition, there are many PRs linked to this issue, and many attempts happened to make the situation better. However, we should keep this issue open and make sure that the flake will disappear. WDYT? cc
The test has been flaky for multiple releases, but it was marked flaky and completely removed mid-1.18, which is new. I'm wanting to make sure that absence of any test signal around this is known/acceptable to sig-node.
Links to the investigations I was able to do: combined e2e/api/kubelet log from failure with extended timeout:
Added to agenda for 3/24 sig-node call


This issue proposes the writing of a new test for the API endpoint with the intention of promoting it to Conformance. This issue contains a very masic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this query, there is only one API endpoint which is for Namespace that isn't being covered. We selected patchCoreV1Namespace to write a test for. Dependencies: The mock test: If a test with these calls is build and merged, conformance will go up by 1 endpoint.
/sig testing /sig architecture /area conformance


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With client-go v0.17.0, I observed many "Throttling request" when DiscoveryClient is used to get API group resources from API server. This didn't happen with client-go kubernetes-1.16.0. It looks like originally this kind of discovery requests are not throttled because no rate limiter provided and QPS == 0. But with rate limiter is always set if nil rate limiter is provided. is particularly suspicious because it is always true because DefaultQPS is nonzero. What you expected to happen: Discovery requests should not be throttled as before. How to reproduce it (as minimally and precisely as possible): Create DiscoveryClient and then call with it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The NewTokenBucketRateLimiter was via Always negotiate a decoder using ClientNegotiator /cc
To be clear, that was fixing a bug where the documented defaults for rate limiting were not taking effect. I don't think we should completely disable throttling until more of the server-side priority work is completed, but discovery is expected to be bursty, so I could see defaulting to a different amount in the discovery client constructor, similar to
Yeah, everyone must be rate limited. I think we could give discovery a big burst value.
opened


What happened: make quick-release-images fails on v1.17.0, works fine on v1.16.4 What you expected to happen: Should work on v1.17.0 How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: The immediate cause of the problem seems to be that the hyperkube image creation fails because it doesn't find the kubectl and kubelet binaries in , apparently because they weren't built.
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Which jobs are flaking: pull-kubernetes-e2e-kind Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig apps
Failure at
Hi I looked into the failures and think it was because the test required the job succeeds 3 times with pseudorandom 50% failure chance within 15 minutes, while there is an exponential back-off delay (10s, 20s, 40s …) capped at 6 minutes before recreating failed pods. As 7 consecutive failures (1/128 chance) could exceed the timeout, the test failed intermittently because of "timed out waiting for the condition". For example, in the failure of , the job pods succeeded and failed in the following orders: After the last failure at , the delay has increased to 640s so no pod was created any more before the test failed at . I think it can be fixed by using a deterministic approach to make the job pods succeed and fail alternately, and its executing time can be reduced to tens of seconds stably. Assigning myself. /assign


This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to these query results from APIsnoop, we are lacking in tests for the follow endpoints related to Secrets. We chose the follow endpoints for this mock test. Test outline: a Secret in the default namespace with custom values Secrets in all namespaces search for Secret created in  the data in Secret  Secret  Secrets in all namespaces ensure that Secret  doesn't exist Discover useragents Display endpoints hit by the test Review the change in coverage If a test with these calls gets merged in, conformance will go up by 3 endpoints. /sig architecture /area conformance
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yeah! We got a test written and a promotion (and we linked them to the mock-ticket that started it all). /close
Closing this issue. <detailsIn response to : Yeah! We got a test written and a promotion (and we linked them to the mock-ticket that started it all). /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


/sig scheduling /priority important-soon See as an example. Part of /assign
GitHub didn't allow me to assign the following users: angao. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig scheduling /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thx, I will do it soon.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The apiserver, controller-manager and scheduler are configured to use IPv4 addresses however, are serving on all addresses IPv4 and IPv6 What you expected to happen: To listen only on the IP family specified How to reproduce it (as minimally and precisely as possible): or checking in one of the gce presubmits jobs Anything else we need to know?: it seems that the apiserver, controller-manager and kube-scheduler listen on all IP families by default despite the IP family address configured Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /cc
binding 0.0.0.0 to all interfaces is the documented behavior of net.Listen:
The unspecified ip (0.0.0.0) is a special case that means "bind to all interfaces". This is currently working as intended. Is there a specific problem this causes?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
If (0.0.0.0) means "bind to all interfaces on all IP families", then is a misconfiguration problem and not a bug . I was worried because this causes that some components use IPv6 for their communications on dual-stack hosts despite the cluster is IPv4 single family (and viceversa), as we can see in this snippet from one of the gce presubmits job and I don't know if this can impact on some services or from a security point of view
From the listen godoc: That means binding to or or will bind to all interfaces
I would expect anyone using network interfaces as security boundaries to be binding to a specific interface, not binding to all and hoping "all" is family-partitioned
Clarifying the handling of unspecified addresses like that in the flag/config help seems reasonable, but as far as I can tell, this is working as intended.
yeah, thanks for the clarification, I totally misinterpreted the net.Listen function
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
So, is there any way to listen on IPv4 addresses only (all of them)?
hmm, that will requires to expose this setting just curious, what is the use case for having IPv6 on the host and not in the apiserver, is not possible to disable ipv6 in the host?
I'm considering options to have a load balancer (HAProxy static pod in hostNetwork) for api-server on control plane nodes listening on the same port as api-server. Then I would point api-server VIP to via hosts file on the control plane nodes. This is a workaround for the absence of hairpin in AWS NLB (with instances by id) or Azure LB. I'm looking for a generic solution that will work for AWS/Azure/on-prem with nodes in the same/different subnets. If you have other suggestions, feel free to share, I would appreciate that. For now, my solution is to call , then patch the generated static manifest with set to the node's default ip, and then bootstrap with . Binding the load balancer to IPv6 and and api-server to IPv4, then using in kubeadm's config will be a dirty, but cleaner from the code perspective, approach. Btw, is there any reason behind having set to by default, but not the node's default ip, just like ?
I can't understand how do you solve the hairpin problem that way, is not the traffic been directed to the LB vip too? or do you mean that you don't use the AWS NLB and use the haproxy instead? what VIP do you use then? I don't remember the details, but the apiserver should be available to the pods too, that don't reach it through the node, they use the the kubernetes.default Service
External clients use AWS load balancer, while requests from control plane nodes go to haproxy on localhost thanks to trick, thus do not reach AWS LB.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If you create a service that reference a Basic IP in a standard load balancer cluster, the service will stuck at "Pending" state for days without detailed error from Azure response. When you describe the service, it only has following: What you expected to happen: In this case Azure responded 400 with detailed error message that the referenced IP is a basic IP and cannot be used on standard LB. I would expect the detailed error message show up as events for easier diagnostic. How to reproduce it (as minimally and precisely as possible): Create a service that reference a Basic IP in a standard load balancer cluster Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /area provider/azure /sig cloud-provider
/area cloudprovider


Which jobs are failing: Which test(s) are failing: Testgrid link: Reason for failure: check Anything else we need to know: /cc /milestone v1.18 /sig cloud-provider-gcp
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Which jobs are failing: Which test(s) are failing: Testgrid link: check /cc /milestone v1.18 /sig cloud-provider-gcp </details
/sig cloud-provider
/area provider/gcp
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /area provider/gke </details
Probably related to
shows the merge shows the errant merge that inadvertantly disabled the test This test was probably failing since forever.
Any news?
this one is on cloud provider GCP.
Can you please take a look?
Follow up from today's release meeting: If the test has been broken for as long as we know it, theres value in just removing it. If the test owners don't have capacity to look into it, CI Signal can investigate.
Any advice on which team is responsible? It seems there isn't any maintainer for this job.
try talking to please
/assign
Turning to the question of whether these tests are actually useful... One of the tests that has been continuously failing is this one Which essentially does because From the logs (example ) it waits for 2 minutes (120 seconds) for each node to not be ready and then for the next couple minutes it just checks this condition until it fails So it looks like the e2e tests are not failing because of inherent infra issues (i.e., kube-up decided to do some weird thing) but the issue seems to be with the time to reach a condition. Again if we look at the logs we can see that switching the network interface does have an effect and leads to the nodes reaching a non-ready status but ultimately fail to come back up within the desired 5m time mark - this bit of the tests is where it all goes bad . The question then turns to turning off the network interface for a node in such a way () and then expecting the node to come back to live within 5 minutes of turning the interfac back on something reasonable? If it is, then either we have to look at why it takes longer than 5 minutes or we can increase the timeouts If it isn't, then how can we test the ability of nodes to be rebooted?
/assign
Wanted to triple check on the test looking for nodes to be back up after being rebooted: I dont think 5 minutes is a hard deadline - only SLOs I could find were for scalability


What happened: We enabled the feature gate on to start switching from non-expiring to expiring ServiceAccount tokens as . It works well. We then wanted to completely get rid of the "legacy" non-expiring ServiceAccount token Secrets. To do that we configured with (so it wouldn't auto-create the Secrets), removed the existing Secrets and removed the reference from the respective ServiceAccount to the Secret. This works as expected: No non-expiring token Secrets are generated. However, when creating a Pod we now see: This happens because the rejects the Pod because it cannot find a ServiceAccount token Secret (because it doesn't exist anymore as intended). What you expected to happen: When using the feature the legacy ServiceAccount token Secrets aren't used anymore, so a Pod should not be rejected if the Secret doesn't exist. The Pod doesn't even use the Secret at all. How to reproduce it (as minimally and precisely as possible): Enable feature gate Start with Create a new ServiceAccount Deploy a dummy app using that ServiceAccount Observe no Pod being created with Describe the corresponding ReplicaSet to see the above Error event. Anything else we need to know?: Looking at the it seems that at least one legacy Secret must exist in order for the Pod to get admitted. This is enforced even when and we're relying on projected token volumes. Following the code path when is enabled reveals that the token is never going to be used later on as there's no SecretVolume to be mounted in . In other words:  is just fine when . During the transition period it's fine to have the legacy Secrets lying around but ultimately it's desirable to get rid of them because as far as I can tell they are still happily accepted by the API server. Environment: Kubernetes version: tested on but code seems the same in and Cloud provider or hardware configuration: OS (e.g: ): Ubuntu 18.04.3 LTS Kernel (e.g. ): 4.15.0-1054-aws Install tools: /kind bug /sig auth
Fixed in
Thank you!


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux


Which jobs are flaking: pull-kubernetes-e2e-gce pull-kubernetes-e2e-gce-rbe Which test(s) are flaking: Stage Testgrid link: Reason for failure: This isn't technically a test that's failing, it's part of the build-stage-up-test cycle used by our e2e jobs. There are two main clusters of failures that I can tell. One is timeouts. The other looks like: Example -gce jobs: () () () () Example -gce-rbe jobs: (source gist ) Anything else we need to know: Triage link for the job that actually reports to PRs: ends up running Some googling for "bazel error code 14" leads me to believe that bazel (or its jvm) is getting killed due to OOM. which is a value that's been cargo culted around as far as I can tell. I was unable to find the decision that led to this value. I'm wondering if a bump to 8Gi of memory would help reduce occurrences of this flake. I'm referencing the -rbe jobs to point out that this happens even if we use RBE, but they're not blocking
/sig testing Top flake according to
With merged we'll wait and see if the Stage flakes on pull-kubernetes-e2e-gce go down. If they do, there are other presubmit jobs that could benefit from that. If they don't, we should revert.
/priority important-soon
/cc
Still seeing flakes, so I don't think just bumping memory request helped. ref: my theory is that making this non-privileged broke our ability to free memory This happened via around 2019-04-11 - flakes were still occurring prior to this merge - but not prior to 2019-04-01 - and very infrequently prior 2019-03-15 It could be that bazel's memory usage grew when we bumped it to 0.23.2 on 2019-03-29 (or maybe on 2019-04-02 is what actually deployed the change) Either way, I still think it looks like bazel is eating more memory than the container is allowed, and is getting oomkilled. Two ideas: Instead of allowing the bazel server to exist as a long lived process, have kubetest call in between the Build and Stage steps, in addition to trying to flush memory Constrain bazel's host JVM to a hardcoded memory limit via in bazelrc
Idea 3: switch build modes and see how expensive / flaky that is. Including: use RBE use the dockerized build instead
I opened to try and constrain the JVM's memory. Jobs that use RBE behave the same way, so try doing the same to them for comparison's sake.
this is still being seen extremely frequently (seen multiple times in a row on the same PR at )
I opened for the last idea I have for working around bazel's behavior, beyond this I am out of my depth on how to replace bazel for the release/push- script
... but I'm willing to try with a canary job and see what happens
bumping priority, this looks like 99/122 flakes according to /priority critical-urgent
Another occurrence:
I'd like to see if we can root cause what changed around 1/27-1/29 that greatly increased this
kubernetes/kubernetes changes: 1/26, ~6am UTC: 1/28: ~6pm UTC: test-infra changes: 1/25, ~3pm UTC: 1/28: ~8pm UTC: in that delta kubekins changed from v20200123- to v20200127- nothing jumps out at me in those diffs, but wanted to link to them in case it helped anyone else see something
It doesn't have to be one, the CI is run overcommitted with no limits, something else could be upping memory pressure..? On Mon, Feb 10, 2020, 08:38 Jordan Liggitt <:
ok... after sweeping k/k and test-infra for that time period, nothing clear emerged, and the bazel bandaids seem to be holding for now. unsatisfying :-/ /close
Closing this issue. <detailsIn response to : ok... after sweeping k/k and test-infra for that time period, nothing clear emerged, and the bazel bandaids seem to be holding for now. unsatisfying :-/ Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


What happened: Currently, the mutating admission happens before updating managedFields in create requests, but in the update and patch handlers, mutating admission happens after updating managedFields. This causes the ownership of fields modified by mutating webhooks to be inconsistent based on the request verb. /sig api-machinery /cc /priority backlog
/wg apply
/assign


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened. But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: , from whatever combination of inputs we stuck into it, segfaulted. The segfault, which originates from a slightly modified fork of v1.17, looks as such: Translated to master, where the problem appears to be present, the segfault originates from this call: And that is called from here: This seems odd, because The only pointer involved in the segfaulting call is itself The caller nil-checks before calling a method on . I did some digging and found , which is a fix for this problem. However, this fix appears to be undone in . There doesn't seem to be a reason for this to have been changed in the latter PR. What you expected to happen: should not segfault from any bizarre combination of user input. If the situation is invalid, it should fail gracefully. How to reproduce it (as minimally and precisely as possible): Honestly, I'm not 100% sure what combination of inputs we crammed into , but I can locate them and post them if necessary. Anything else we need to know?: I don't feel confident opening a PR to fix this myself, but in addition to recreating the fix from , a comment should be explaining why there is this weird, seemingly pointless layer of indirection when assigning to the interface, so that this doesn't get un-fixed again.
Maybe decompose the above statement into multiple lines which should give us more idea which field was nil.
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
can't be nil, as it is of type , which is not an interface or pointer is an interface. However, when built from the exported constructor, , it is set to a literal , which is likewise not a pointer Therefore, the only remaining field is itself. Plus, I don't quite recall my go segfault stack trace syntax, but I'm relatively sure that indicates that the first argument to the method is , nil, and that the first argument to methods is the object being called on.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
When config.HealthzBindAddress is empty, we would encounter the scenario depicted in the stack trace. I have proposed a fix.
/assign
That commit, , is part of the PR that I believe introduced the problem. I strongly believe the problem to be one of a non-nil interface with a nil receiver. This is the only way that could be nil in the snippet of code from . Before PR (the changes from which are present in the version I ran), there was this odd looking set of code: The variable healthzServer, which is a pointer of type , was assigned to the variable , which is an interface of type . Then, later on, the variable was passed to , as an argument of type . This looks unexpected, and it appears that the variable is extraneous. After the changes in PR , the variable was removed, and was passed directly to as the argument for interface type . The former set of changes, with the intermediate variable, are meant to avoid the situation of a non-nil interface with a nil value. If the variable is never set in the conditional statement, then it is a nil interface. If is passed directly, then the result is the argument to is an an interface who's value is and who's concrete type is . This means that is called with as the receiver. I strongly believe this to be the root cause of the issue, but I do not know enough about kubernetes to open a PR with the correct fix, which would be to reinstate the variable (after the renaming in , this would be a variable of type ) to use as a go-between, to once again avoid this situation of a non-nil interface with a nil receiver. I believe that the removal of the variable was out of a misunderstanding of its purpose. If my assessment of the problem is incorrect, please let me know so I can understand where my logic is wrong. It is very possible that I am misunderstanding the phenomenon of a non-nil interface with a nil receiver. I would open a PR myself, but I do not know enough about Kubernetes to be confident in my ability to correctly fix and test this problem.
/remove-triage unresolved


Seen in the KIND e2e test: a) maybe we shouldn't log this more than 1x per second or something? b) maybe the error could make it clearer that the problem is the user's data? c) maybe we could fix the test triggering this? /sig api-machinery /wg apply Logs that I saw this in:
The 'duplicate entries for key' error came from validatingObjectWalker#visitListItems.
/assign
Yeah, they have multiple environment variables with the same name. I don't think that should be an error though ...
/cc
At the same time, it's not really useful to have multiple times the same variable and is probably a recipe for trouble.
It's definitely user error. Spamming apiserver logs about it doesn't help things, though.
I don't want to hide it, it's really not supposed to happen :-) it's not really user-data though. The fact that the error triggers here rather than in the actual validation means that there is a discrepancy between our validation and the regular validation. That's probably a problem that we have to fix. In that specific example, we can see that Kubernetes allows multiple environment variable with the same name, while we have assumed that it's not OK. It's a very legitimate error and we have to think about it to fix it. I think the test is relevant, and people out there are doing that. We could consider doing ratcheting if we want to stop allowing multiple variables, but I don't think that's how we want to fix that problem.
For history: should fix that.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
We've definitely improved that, maybe let's close it for now.
/reopen We still see this happening. Kube users get nervous, I would too. Please fix the root cause.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: when i installed k8s using kubeadm and my custom config like then output is It is a bug that the default value(Node,RBAC) is entered but a warning message is displayed. A function must be to compare two string slices. What you expected to happen: If the default value and custom value are the same, no warning message should be displayed. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 17.3 Install tools: kubeadm /area kubeadm /feature bug /sig cluster-lifecycle
/lifecycle active


Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestStressingCascadingDeletion Testgrid link: Reason for failure: Maybe related: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig api-machinery
I'm seeing lots of these.
post-submit failures (all of these are flakes):
first failure seen in postsubmit is , part of ... preceeding many runs looked clean
I verified the delete options as seen by the server are identical before and after , so I don't think it is a functional change related to that PR
notably, many other tests in this suite started flaking on 3/7 and after ()
figured it out, dropped the detection of empty deletion options which would trigger the test to wait for all pods to be observed by the GC controller


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If gid is configured in Pod's securityContext.fsGroup, csi inline will panic. log: E0320 16:50:59. ] Operation for """ ("-ae1b-4b5c-be91-")" failed. No retries permitted until 2020-03-20 16:51:31. +0800 CST m=+. (durationBeforeRetry 32s). Error: "recovered from panic "runtime error: invalid memory address or nil pointer dereference". (err=<nil) Call stack:ngoroutine [running]:(0xc001497e70)n +0xb5npanic(0x3b713e0, 0x7bd2050)n +0x1b5(csiMountMgr).applyFSGroup(0xc0005372c0, 0xc001a2f100, 0x4, 0xc00205ed70, 0x44, 0x0)n +0x64(csiMountMgr).SetUpAt(0xc0005372c0, 0xc000b6cb80, 0x78, 0xc00205ed70, 0x0, 0x0, 0x0)n +0x1180(csiMountMgr).SetUp(0xc0005372c0, 0xc00205ed70, 0x0, 0x0, 0x0)n +0x4d(operationGenerator).GenerateMountVolumeFunc.func1(0x8, 0x453f9f8, 0xc001497e70, 0xc0025ba750)n +0x4a4(GeneratedOperations).Run(0xc0009742a0, 0x452f628, 0xc000d56480, 0xc00135a050, 0x45)n +0xaf(nestedPendingOperations).Run.func1(0xc000d56480, 0xc00135a050, 0x45, 0xc001106d50, 0x24, 0xc0009742a0, 0x0, 0x0, 0x0, 0x0)n +0xcbncreated by (*nestedPendingOperations).Runn +0x378n" code: If using inline volume, c.spec.PersistentVolume will be nil What you expected to happen: fsGroup is successfully applied Environment: Kubernetes version (use ): 1.16.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig sig/storage
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig/storage Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
ran into this bug recently on v1.18, and I don't see anything that would fix this issue on later releases either. /reopen /lifecycle freeze /assign
Reopened this issue. <detailsIn response to : ran into this bug recently on v1.18, and I don't see anything that would fix this issue on later releases either. /reopen /lifecycle freeze /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/lifecycle frozen
I tested this scenario with the hostpath driver on k8s 1.24 (built from master) on a local VM using a slightly modified version of the examples/csi-app- spec ( fsGroup and fsType): And this does not panic kubelet It does not panic because of this check: Which was in However, I do believe we can and should support fsgroup for CSI inline volumes. Instead of checking the PV spec (which is not set for inline volumes) we could pass the directly into , which will be RWO for inline volumes.
Sorry, it was not obvious that the panic was already fixed when I first looked at this. Re-purposing this bug to address the issue that fsgroup still does not get applied for inline volumes. /retitle CSI inline volumes do not apply fsGroup
THX, also link to this
/retitle CSI inline volumes should support fsGroup


After reading up on some corner cases with etcd client behavior (connections, streams, etc.) I came across the following option that has been introduced in etcd a while ago: From the official etcd docs on client behavior: Source: I do not see this option being used in the API server when is . Is there code in the API server that deals with these cases (etcd/API server node paritioned) and disconnects clients (REST, SDK) appropriately? Or are we at risk of having dangling (hanging) consumer-side client watches due to not using this option in the etcd client? Related discussions: /sig api-machinery cc/
was this something that was considered already?
We should be using for watches. (see also the ). It should just need to be . It would also be valuable to confirm that the behavior works as expected both when establishing watches and when a etcd member is partitioned after the watch has been established.
If anyone wants to own fixing this please self assign, else feel free to assign to me.
For this method: It seems we should do:
Thanks! This is a good question. I am not exactly sure what is the semantics on kubernetes watch. It is correct that without , etcd server will not cancel watch stream when itself is isolated from the cluster. However, when this happens, other requests to etcd (including health checks on etcd) will fail. So the control plane is already in a bad state and is likely to be replaced / repaired either automatically or manually. IIUC, this means apiserver will restart, the its clients will restart their watches.
Oh I just saw Joe's earlier comment. If using aligns with k8s watch semantics, we should use it:)
i'd support moving to 3.4.5 to pick this up!
I am happy to contribute but want to do some more testing and validation on the current behavior of and with the proposed fix. Please give me some days.
Quick update: after some quick tests it seems that does not solve the issue with an active after the connected etcd endpoint becomes partitioned. I think we have the following scenarios to consider: 1) Single node API server/etcd 2) Quorum-based cluster (e.g. 3 nodes) where K8s control plane is colocated with the etcd nodes (e.g. one API server and one etcd instance on one node) 3) Quorum-based cluster (e.g. 3 nodes) where K8s control plane is not colocated with the etcd endpoints (separate nodes for K8s control plane and etcd nodes) We can rule out (1) because technically a partition with just one node in the cluster (K8s control plane and etcd) cannot happen. (3) is mostly addressed with existing gRPC timeouts and keep-alives so when an API server connected to an etcd node that gets partitioned this API server will reconnect to other reachable etcd nodes (endpoints). However, this is only true if the API server itself is not within that partition (e.g. multi-zone deployment where a whole zone becomes partitioned; read on for my observations). (2) is the interesting one: an API server is connected to an etcd node and both become partitioned from the majority. But the API server has active watches against this etcd node and serves these (Kubernetes) watches to Kubernetes clients (controllers, etc.). I was expecting that the gets evaluated even after establishing the . But it seems my assumption is not correct, i.e. it is only evaluated (checked) when is called. At least I did not see an existing being dropped after partitioning the nodes from the quorum (I did check the underlying gRPC transport status and it remained in stayed for as long as the partition was not resolved. So what does that mean for the issue raised here? One could argue that if more than just one etcd node is partitioned, i.e. also the API server and perhaps other Kubernetes controllers running in that partitioned environment (e.g. a zone), it makes no difference using because: the clients (custom controllers, etc.) are also partitioned and thus cannot connect to any other API server anyways; would be nice to return an error here instead of hanging but the system won't self-heal during the partition even if the clients (custom controllers) would not be in that partition, the corresponding API server could not be reached anymore and client-go session handling should address this (e.g. reconnect to another API server) you write: I did not see an existing watch being dropped, e.g. using this simple code: Once I partition the etcd node including this client it will not drop the . Looking at the definition of it seems that this is performed only during establishing the but not reevaluated as long as the client has a successful connection to the etcd node (e.g. node is partitioned but not dead). Since I am by no means an expert on the etcd semantics here, I might be doing something wrong. It would be great to get some clarification whether this issue (I raised, I know :) ...) needs to be addressed, documented, or can actually be addressed at all with the current implementation of .
Please see analysis above. should this work when the etcd + apiserver are on the same node or not?
Thanks for the careful analysis My take on this is: For case (2) where the apiserver and etcd are co-located (and the apiserver only talks to that etcd), I don't see any reason why dropping the watch would help. In this type of setup, when etcd becomes partitioned, the co-located apiserver becomes unavailable and apiserver clients should route away from that apiserver entirely until the partition is healed. Even if the apiserver were to drop the watch, it is still unavailable. For case (3) the behavior is already better than I expected. optimizes the case slightly by ensuring the initial watch connection is against a non-partitioned etcd, so it's still worth enabling, but not urgently needed since failover is already in place. We should document in etcd that only applies to establishing the initial watch connection, the documentation is currently misleading.
It is correct that without WithRequireLeader, etcd server will not cancel watch stream when itself is isolated from the cluster This is correct. Edit: Sorry I missread the test. IIUC, this is not correct. After receiving the watch request with , etcd server will continue to monitor its leader for this watch stream. When there is leader lost observed for "maxNoLeaderCnt election timeouts", server will cancel the watch stream.
Thanks that means the documentation is correct then. So while we haven't found any pressing issues that would fix, it does do what it should do, enabling it in the kube-apiserver won't make anything worse, and might make failover slightly faster?
Make sense to me.
BTW, I just did a quick test. an etcd cluster of 3 members: A, B and C. member B and C some time, the etcdctl command prints
Sorry for my delayed reply, wanted to do some more tests before replying. I can confirm this scenario. However, the tests that I did and which caught my attention because it was not what I expected from reading the doc string was to not shut down the peers but isolate one peer, i.e. simulate a partition which technically is also a "lost leader" scenario. In this case I saw the watches not being closed (neither for nor programmatically). Not sure if this is a bug or by design but I think it's more likely to have partitions than loosing the whole cluster where a majority of nodes is terminated/crashed (stop-failure). If you think it's worth and not expected behavior I am happy to open an issue in the etcd repo and provide more details. Even though in etcd makes it sound should address the partition scenario. Note: I'm currently reviewing the network partition tests in etcd which state that the results I'm seeing are not expected. Update: looks like I'm experiencing what was reported a while ago in but it's unclear to me whether it's fixed, unfixable or what the workaround (from K8s perspective) would be.
I'm in favor of you opening a new against etcd for the watch not being closed in the lost leader scenario (and linking to this discussion). If it turns out it is a duplicate we can always mark as such.
has some detailed watch behaviors.
FYI. There was a bug in . The fix was backported to = v3.3.19 and v3.4.5
Thank you and for your feedback. After reviewing the docs and I think the issue with and how to deal with partitions is still open (see TODO below). The scenario where clients (Watchers or any client indirectly consuming from an etcd ) would hang (forever?) is when the client gets partitioned with the corresponding partitioned etcd node OR when a whole zone/dc is partitioned. Since etcd picks safety (consistency) over liveness (availability) in this case at least not data corruption occurs but the overall system would be stuck when using watches instead of PUT/GET RPCs with timeouts. It would be ideal to throw a useful error, e.g. "lost leader", instead of hanging in "gRPC TRANSIENT_ERROR" state (which is what I'm seeing). Until this is resolved in etcd, perhaps the docs could be updated as is a bit misleading here. I will comment on the etcd issue above and file a PR for the K8s watcher implementation to make use of as it is still a useful option when (re)establishing the stream. Depending on the environment, API server instances could have a health/liveness probe (or systemd equivalent) which could restart the process in case of a partition. Upon reconnect to the etcd node with the option it would immediately fail and error out accordingly.


Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error. I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO. However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).
FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestSubresourcePatch Testgrid link: Reason for failure: Unknown. A diff with the original object would be helpful. Anything else we need to know: /sig api-machinery
OK Trying to reproduce, see what's going on.
Clearly this is because SSA is updating the tracked entry's time. It happens when the two request don't happen in the same second. Having a between the two requests makes this into a definitive error.
/wg apply
This seems to be because of . A repeated patch like will always trigger a new managedFields entry to be created, since it's always going to change the object (status or spec will be wiped depending on the endpoint), so the patch will always cause the object to change, which will always cause the entry to be created.
/assign
/assign
/assign
Still seeing this, bumped into it over in
deflake in while the underlying issue is being worked on in