I'm using K8S 1.10 and have a specific use case where I'm trying to use an emptyDir to share mounts between two containers in a pod. I'm using the MountPropagation feature successfully, but while debugging another issue I noticed that the node was running a very long running du command. Looking into the source I noted: Which runs du on any emptyDir, do measure disk usage. I recommend that in this case we should add the -x flag that will skip any mounted directories as they do not count against the file usage and would make these scans a lot faster. I could of course add a simple pull request for this but first wanted to see if this was known and on purpose. To reproduce bind mount/loop mount/device mount any large/slow directory into an emptyDir volume and watch as the du process slogs away Environment: Kubernetes 1.10 GKE <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think it makes sense to ignore submounts.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
do you still want to submit a PR to fix this? Thanks!
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


/lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


[X] Two weeks soak end date: Nov 18 According to this APIsnoop query, there are still a remaining Service endpoint which is untested. A service resource didn’t support endpoint until was merged. Create a watch to track service events Create a service with a static label. Confirm via the watch that the service was created. Delete the service via the service with the dynamic client with . Confirm via the watch that the service was deleted. The above steps have been implemented in the follwing . This query shows all the endpoints hit within a short period of running the above e2e test If a test with these calls gets merged, test coverage will go up by 1 point This test is also created with the goal of conformance promotion. /sig testing /sig network /area conformance
/triage accepted
Waiting on 2 week soak before promoting the test to conformance /reopen
Reopened this issue. <detailsIn response to : Waiting on 2 week soak before promoting the test to conformance /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


During
any chance this is related to pass: https://k8s- fail: https://k8s- https://k8s-
This is because for some reason the submit queue rebuild didn't catch the flake before merging. Protobuf landed first, merging a new api object like petset should have failed.
Quick fix is to rebuild the proto marshalers and push.
Also, hack/verify-generated- is broken, which probably is the root cause.
Looking at it now
Three problems: -generated-protobuf wasn't updated when we made update-generated-protobuf use docker, so it didn't fail out script was verifying the old API groups, not any of the new API groups that have been merged without protobuf serializations (although pkg/api/serialization_test should have failed, possibly the job didn't build with the latest master?), which caused the tests to fail
Something about the merge bot is broken - it merged without all the tests being green: . That's why this broke -
Spawned since there is something wrong with update-generated-protobuf - will merge that to fix the failing test and then debug the generator in the morning.
The bot can race with pushes between when it starts testing and when it merges...


What happened: states that Container.SecurityContext applies to a Pod, which can be misleading since we already have PodSecurityContext. What you expected to happen: Documentation states that it applies to container How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment:
/sig api-machinery /kind documentation
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Not stale.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle rotten
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Currently watches only log when they complete, which can make analyzing kube-apiserver log files a bit tricky. (did component x never start watching after that list or did apiserver shut down?)


on the end of script there is a line: It produces high CPU load when it is run without tty. However it can be replaced with something like this:
Looks like this has been fixed.


https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .


https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
The test pod gets deleted by the test upon failure and so there is not enough information about the state of the test pod. I'm changing the test to not delete the pod and instead let the namespace deletion logic in the test framework handle deletions. I'm also increasing the pod lifetime from to a longer duration to avoid any possible race conditions in the test. The test will also retry if the logs output from the test pod does not contain non-space characters.


I'm lowering the priority of this test once my changes go in and see if the flake re-appears.
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
I suspect a race between container starting up and the test querying the logs. I'm adding a retry in the test to avoid the race. Lowering the priority since I still don't suspect any kubelet bugs.
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .


Flaked a couple of times on serial lane with the same error. Test is relatively new. This run only has the single test flake: /sig node /kind flake /cc
/assign
New recent failures: /triage accepted


The pod security policy capability test is using wrong CAP format with prefix. In our document, we explicitly say: It fails for cri-containerd. I'll find out why it doesn't fail for Docker, maybe docker has handled it. But we should fix the test anyway.
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


[ ] OWNERSALIASES:581: # conformance aliases [x] // More info: ~~ [x] // See [x] fs.MarkDeprecated("redirect-container-streaming", "Container streaming redirection has been removed from the kubelet as of v1.20, and this flag will be removed in v1.22. For more details, see ") [x] // More info: [x] // More info: ~~ [x] [x] [x] CHANGELOG/CHANGELOG-[scalability validation report]: ~~ [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [ ] link: [ ] link: Some needs to be regenerated
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help
/kind documentation
/remove-kind bug
/good-first-issue
/assign I shall fix .
/assign i'm working in
i have create a pr for
i see , you need
/assign fixing
/assign
/assign CHANGELOG/CHANGELOG-
/assign CHANGELOG/CHANGELOG- Edit : Link is already resolved, unassigning
/assign CHANGELOG/CHANGELOG-
/assign ,111
Sorry, already assigned
/assign i will take
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
  is merged. Most KEP links are fixed. You may scan it again. fixed most of them.
/sig architecture
/assign I will work on and
/remove-help


pull-kubernetes-unit TestDifferentWidths As far back as triage board history shows No response /sig api-machinery
/cc /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available .


If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Running into this one a fair bit in my PRs, opened as a possible fix
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen I got this in --
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


/close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen Got this in
Reopened this issue. <detailsIn response to : /reopen Got this in </details
/remove-lifecycle rotten


The based container images for are shipped with the wrong architecture within the manifest if not amd64: Refers to We updated the images a few minutes ago to buster-v1.3.0, which are now ready to be used as base for the multi-architecture images. If the images contain the right architectures, then it may magically happen that the kube-proxy images are fixed, too. This is because we already use the build syntax for it: /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release


Similar to issue , building kubernetes1.23 resulted in 1 variation: builds should be fully determined by inputs on openSUSE or Debian, This bug was found while working on . <details23.0 </details<details</details<details</details<details</details<details</details<details</details
/sig release /area build-release
/assign
/sig cli
/triage accepted /assign


What happened: e2e tests detect whether PSP is enabled based on presence of PSP API objects. An addon that proactively includes a PSP in a manifest can trigger that detection and enable PSP-specific e2e tests that should not be run. What you expected to happen: In addition to detecting presence of PSP objects, consider dry-run creating a pod and verifying the annotation indicating the selected PSP policy is , indicating the admission plugin is active. /sig auth /assign


I ran conformance tests on a cluster that had a controller enabled that syncs a configmap across all clusters. It failed: Appears to be due to a 2 configmap limit: These clusters should pass conformance. This is a very common pattern (even Kubernetes itself does it!). I didn't run it myself, but apparently just install Istio and run the conformance suite. See for details Kubernetes version (use ): v1.20.2
/assign
my guess is the CA configmap created by default in 1.20 exacerbated this issue. I agree the conformance test should make fewer assumptions about the cluster it is running in and should accomodate default configmaps. left a comment on the PR in flight for this. I'd like to see the change picked to 1.20 as well.
/triage accepted /priority important-soon


https://k8s- It seems that when it takes a while, it tries to collect debug info. And that collection is flawed because kubectl needs a container for log collection when the pod has more than one container.


removed this test. We need to continue testing this. This wasn't about testing upgrades, the prefix has nothing to do with upgrades. It's just there so people can share etcd. We maybe don't need to double everything but we do need to make sure that keeps working.
To be honest - I think that what we was testing before didn't give us completely any information. We didn't testing what information we are storing in etcd and where. That said, if we had a hardcoded prefix somewhere in our code, we wouldn't discover it at all, and those tests would still be passing. We should have a test that is directly checking etcd contents. I think it should be possible to write a simple unit test for it. Will try to put something together next week.


I0921 20:36:33. ] Could not connect to D-Bus system bus: %sdial unix /var/run/dbus/systembussocket: connect: no such file or directory


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In https://k8s- failed due to . Afterwards crashed as follows: What you expected to happen: Graceful exit.
/assign


What would you like to be : Once a metric be deprecated, we should update the unit test accordingly if there is any. We need to work out a way for a unit test freely select codebase version, and also should make sure the unit test won't fail in the subsequent release. Why is this needed: Deprecated also needs to be protected. /sig instrumentation /assign


appears to have been trying to improve the logging in e2e test runs, but it seems to have affected much more than that, and now when a build fails, it logs so many useless messages that you have to scroll back to see why your compile actually failed. before: after: /assign cc (re )
GitHub didn't allow me to assign the following users: oxxenix. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : appears to have been trying to improve the logging in e2e test runs, but it seems to have affected much more than that, and now when a build fails, it logs so many useless messages that you have to scroll back to see why your compile actually failed. after: /assign cc (re ) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks for tracking this down. I agree that the additional error message is distracting and would like to see implemented differently so that it's not shown for manual invocations.
/sig release
do you have any tips or recommendations how to implement it differently? Thanks
It looks like that extra ouput is a from . I'd suggest to either revert or strip BASHCOMMAND from the error message if it's not that useful.
It looks like what you wanted in is the location and time stamp ("The update now displays both the specific location of the error in the code and a timestamp."), which is what we also get from the calltree after "make build". I would revert and then investigate why that shorter calltree is not shown after an E2E test failure. Keeping the calltree around as in Dan's "before" seems reasonable to me.


It shows a little bit more location info with the change. Here is how failure looks like with the change (but without BASH_COMMAND: and without the change:
I mean, the call tree is pretty useless in the "compile error" case... if your problem is that the code doesn't compile, you don't need to be told what part of the build scripts it is that invokes . It's just that before it was a small amount of useless output that was easy to ignore, whereas now it's a lot of useless output, which completely pushes the useful output offscreen. But IMO, "no backtrace" would be nicer than "short backtrace"... (I realize there are probably other cases where the backtrace is useful, and I'm not saying we should get rid of it in those cases, whatever they are.)
I agree, stack trace for compilation errors is quite useless. Unfortunately a lot of targets 'make test-e2e-node' included, build binaries, so we can't easily show traceback in one case and don't show in another. Removing BASH_COMMAND from the output looks like a good trade-off to me, though. It has a little bit more info(sometimes usefull) about callstack and less verbose than currently. I'm also ok with reverting and investigating later.
/assign


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /sig azure What happened: The internal IP address is got from instance metadata. I suspect the instance metadata may not be stable, e.g. it has undocumented rate limits (refer ). Cross refer What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:


Looks like some kubelet unit tests perform a lookupIp which ends up depending on the health of the local resolver. Specifically has this failing: Which leads to the test in: And we recently swapped out the node names from testnode to 127.0.0.1, so probably the culprit?
/ref
Interesting. I did swap out the node names in the unit test as part of to be 127.0.0.1, party because I thought that it wouldn't have any problems resolving during unit tests.
/cc


cc
What is the environment where you are running the tests? They pass for me on my mac, a linux box, and shippable seems to be passing them as well.
I couldn 't reproduce the failure either.
I'm out right now; i'll try them when i get back. For the record, I run hack/test- Paul Morie wrote:
Ahem FTR This acronym is going to go quad-plat Paul Morie wrote:
I ran on both platforms and saw the tests pass.
I'm still seeing it. Won't have bandwidth to dig in personally until tomorrow.
seeing a similar failure after a git pull
I knew I wasn't crazy
environment: fedora 21, golang: 1.4, docker: 1.6
I think I see the problem. When I set the node name to in the test I was assuming it would parse as a valid ip address and not end up at the LookupIP call. But only the node name (and not the kubelet hostname) is being set (except in my new test where I ) so it's trying to do a lookup. Do you think you could test just changing the default hostname from "testnode" to "127.0.0.1" and see if that fixes it for you?
Trying now
I still get failures but no panic / threaddump.
gist coming
this looks like again?
Good that it isn't panicing any longer. I can dig into the test failures tomorrow (I think it should be relatively simple to resolve).
cool, thanks for looking
Passes in master now for me. Closing.
FYI, they will break again once goes in, and then we can merge my PR to see if that fixes it for you.
:+1:
is in -- are these tests breaking again?
I just rebased which I think will fix them.
They are breaking again.
is on it.


apiserver reports creation failure if a node is not ready (unhealthy, or kubelet is not ready), but the api object is already stored in master. extracted from and A short term solution is to have health_registry check node health upon creation, not just on retrieving. Or ignore unhealthy node. A long term solution might fall into the interface revamp
Why can't apiserver accept an unhealthy minion.
What's the point of having an unhealthy minion that we'll never be able to use? One reason I can think of is that the minion can later becomes healthy and thus a valid node in k8s. But for now, as we don't have the ability to handle unhealthy node, delegate the responsibility to user makes more sense. However, I admit the title is misleading. What I really what to convey is that kubernetes currently allow registrying unhealthy node, by storing it in RESTStorage. But the create operation reports fail because the node can't pass health checking. I don't think this is the right behavior. We can either return success and do health check later, or return failure and do not store it. (i prefer the latter one for the above reason).
I tend to think that we should accept unhealthy minions - minions may go in and out of health, or go up and down at any time - so trying ensure health at registration time seems like a fruitless effort since we have to handle the error cases anyway.
I've no problem with accepting unhealthy minions, and I agree with your point. The real problem here is that the apiserver returns an inconsistent status. For example, if I have an unhealthy minion that I want to registry. From user's perspective, the registration fails, because apiserver returns an error (resource not found). So I retry, but apiserver returns failure again; this time, the error is resource already exists, no matter the minion is healthy or not.
- yes, we should remove the health check as part of the registration and just report success. agree there is inconsistency.
+1 On Nov 7, 2014 8:48 AM, "Derek Carr" wrote:
Seems like it's a clients responsibility to warn the user that a node might not be available prior to it being , but the the master's responsibility to assume that all properly specified nodes are reachable before making decisions regarding using a node.


There's very little difference between an admin stopping a VM without clearing Kube and adding a node to the cluster prior to it being fully reachable (I think).
"clients responsibility" ? if we enable kubectl to create minions, it would be kubectl who should check if the minion is reachable and complain? I'm not sure what 'client' means. (but I agree 100%, don't report failure adding a minion just because it isn't reachable)
node controller plugin should also give warning


There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: VirtualMachineScaleSetVM and VirtualMachine are different data structures (because of different API versions), so the disk attach operation seems duplicate between vm and vmss. That means although disk operations are in same logic, some duplication are still required now. They should use same api version in the future and should be merged into same code base, e.g. move sharing logic to Refer What you expected to happen: vmss and vm disk operations should share same code logic. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I have checked with currently we can’t merge, e.g. following two structs are different since they using different sdk-for-azure versions. While in the end when sdk-for-azure version is upgraded, above two versions should be same and at that time, we should merge functions like ‘AttachDisk’, ‘DetachDiskByName’, 'GetDiskLun', GetNextDiskLun, DisksAreAttached


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --We have even encountered such case that pod continuously reports “Unable to attach or mount volumes for pod;...” when kubelet restarted. The reasons are listed in details as below: , as result of apiserver instantaneous unstablility, dswp get pvc‘s pv spec failed, then the pvc volume does not be to dsw. , syncStates() lists all the volume directories locally, and recontructs the volume from local volume, if the volume isn't in dsw, it will put the recontructed volume to asw, but now the volume.OuterVolumeSpecName isn't correct. chance, at this time, In , the fail to pvc volume is to dsw again normally in reconcile() -mountAttachVolumes(), it compares dsw and asw by volume's innerVolumename, it's all the same. but it could not sense the OuterVolumeSpecName dismatch. so it will not re-mount again to update asw cache , it checks pod's volume attach or mount success by comparing volume.OuterVolumeSpecName and , but this pv'c OuterVolumeSpecName isn't correct in asw. so pod continuously reports “Unable to attach or mount volumes for pod;...” reconcile() could sense the reconstruct volume's dismatch to the dsw, then re-mount again to make volume cache consistency Kubernetes version (use ): 1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
CC
I think this is a point that can be optimized. /cc
I will give a fix for this
Thanks for reporting the issue, this innerSpecName and outerSpecName are confusing and cause issues. Please ping me for review when you have a fix. Thanks!
Please don't triage your own issues immediately, give someone else a chance to review them :( /sig storage
okay, I know


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: [sig-scheduling] SchedulerPreemption [Serial] PodTopologySpread Preemption validates proper pods are preempted Which test(s) are flaking: Testgrid link: Reason for failure: It's likely when the preemptor preempts low-priority pods, and gets running, the low-priority pods are still terminating. tries to wait for all low-priority pods to be totally deleted and do the final check. Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig scheduling /assign


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: kubernetes/test/integration/scheduler.TestPreemptionRaces Since when has it been failing: Not exactly known. The flake rate is not high. That's probably why it has not received attention. Testgrid link: Reason for failure: Logs are currently unaccessible. Test grid returns an error when clicking on the failed instances. Anything else we need to know: /sig scheduling /label help-wanted
Let me try to fix it. /assign /remove-help
These are the last 4 failure log in . Although The flake rate is not high, but the fail point is same. I will try to solve this error based on this information.
This is currently the flakiest test for the flakiest PR job for kubernetes/kubernetes
/remove-kind failing-test /kind flake /priority important-soon /milestone v1.15
I will try to work out a PR if is not working on it.
Sure, you can take over it if you want. And It is strange about the flake in . I ran about 3,000 times on a local machine, but never failed. :joy: I also can help you to do someting ~
Indeed, flaky tests are usually difficult to repro and debug.. A heads-up: I seem to have found the root cause. I'm working on a solution and will raise a PR after thorough testing.
this is still flaking (e.g. )
I will take another look.
There are two additional flakes since the one reported on May 20: It's no longer that flaky , but the PR I gave earlier proved not to be a thorough solution but mitigation. I may not be able to continue looking into it in the next couple of weeks. Anyone is in interested, feel free to take it.
Last failure was 4 days ago. will continue observing and see if it clears out /milestone v1.16
Been observing this one for a bit and it seems like its getting better Other than some failures a couple days ago it has been stable. Will keep an eye for a couple more days to make sure.
Seems to have stabilized. /close
Closing this issue. <detailsIn response to : Seems to have stabilized.


/close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Heapster pod: And logs: cc p0 for triage
Did the required node scope change?
Its either the API on the console or monitoring write scopes on the node. , Mike Danese wrote:
I think this affects only GCE
I will fix this issue since is on vacation.
Need to cherry pick so i'll leave open
is out for review.
has been closed and the cherrypick here () has been merged so I'm closing this issue. Thanks everyone for the debugging.


<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Four tests: Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is force deleted while kubelet is down [Disruptive][Slow][LinuxOnly] is failing since June 29th 2021 (but the first failure was different than more recent ones). Commit , infra commit , . Other three tests are failing since July 9th 2021. Commit , infra commit , . Testgrid: https://k8s- Prow history: For CSI Volumes: For In-tree Volumes: There is another open issues for failing tests in this job: Failures reported there are about missing metrics, and were occuring before unmounting problems. So I opened a new issue, even though it's the same SIG. These tests don't look related to me, but I don't have enough context, maybe they somehow are. /sig storage /kind failing-test /milestone 1.22
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.


<detailsIn response to : <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Four tests: Testgrid: https://k8s- Prow history: For CSI Volumes: For In-tree Volumes: #### Anything else we need to know: /kind failing-test /milestone 1.22 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
/cc
/cc (my close/reopen yesterday was accidental, sorry)
I can take a look /assign
The non-subpath counterpart of this test is passing: These tests seem to use the default storageclass, whereas failing tests use in-tree hostpath and csi-hostpath
The CSI ones are failing because the hostpath driver never received a NodeUnstage (whereas it did in passing tests), causing the driver to error on volume deletion.
In the failed run kubelet has this error: which seems to have prevented kubelet from triggering a UnmountDevice. This error didn't exist in a successful run.
Hostpath driver can run with and without support for attach. Do you know which mode is enabled for this test?
Between the last passing time and when the test started failing, are there any PRs merged that look suspicious?
According to csi-attacher logs, attach is disabled for both the good run and the failing run. Haven't spotted any obvious PRs so far, but I did find this: in the successful run, the log message for the successful is this: and the failed run: so the reconstructed volume names are somehow different. Checking reconstruction logic
This merged 12 days ago and was a pretty big change to pod lifecycle and could have trickling impact to volume manager. Does the timing of test failure match?
There is also this reconstruction fix that merged yesterday and seems related to outer vs inner spec name. I don't know if that will help fix this issue:
cc
Last successful run started at 7/8 17:49 PT, first failing run started at 7/9 2:23 PT.


was merged at 7/8 16:48 PT, so it's possible (if there's a delay in picking up master) Also this is in the failing kubelet logs but not in the successful one: I0720 18:20:42. ] "Reconciler sync states: could not find volume information in desired state, clean up the mount points" podName=-e508-45c2-8509- volumeSpecName="pvc--a615-4e89-afe8-"
From , the newly introduced log message here exists in the failing kubelet logs but not the successful one:
Failures starting 07/08: /sig node for visibility
I think caused the failures. Before after kubelet restarts, pod will still be in desired state (even through it is in terminating state), so that clean up mounts can be done completely. I think now, terminating pods will not be back to desired state, but reconstruction to actual state cannot completely clean up mounts. I am testing it about this.
Is there a reason why reconstruction to actual state cannot clean up mounts? Note that a pod can be force deleted at any time, so a terminating pod that is force deleted would still have to be capable of being cleaned up (it won't be in desired state)? (note too that the expectation of the changes in is that terminating and fully terminated and "never seen before" should all completely equivalent from the perspective of other loops in the kubelet) If I understand you correctly, is this an existing bug that we're now triggering consistently?
We have both "grace pod deletion" and "force pod deletion" test cases that are failing.
There were a couple of key changes to volume loops in : Roughly before was "it looks like it's got no running containers", and after my change is "we are going to be stopping all the containers from now on". So this change meant that once a pod starts terminating, we are in "tearing down mode" and so my understanding of this method was that "any pod where containers are being torn down is not in the desired state of the world".


After a restart, we would definitely not add a pod back that was in terminating state into desired state of the world, but "restart + this change" should be roughly equivalent to "restart and force delete while restart is processing" The other change was in which changed a live CRI read (which was trying to enforce that the dsowp is synchronized with the pod worker) into a threadsafe check against the pod worker directly (the pod worker can provide authoritative answers to "this pod has no running containers" now). The latter change seems less likely to impact you.
Thank you for the details! I think your change is making the right change, but unfortunately some of volume management still have issues, and so the tests rely on some behaviors that is not very right (pod volume is back to desired state even after it is in terminating) The cleanup mount currently does not have the logic to clean up global mount point if the volume plugin sets it up. I am still testing more and will share more details.
is this a release blocker?
Yes, this impacts volume cleanup if kubelet restarts for any reason.
On a kube-up cluster running the latest master against the latest e2e binary, I was able to reproduce the in-tree failures but not the CSI ones for some reason. Will focus on in-tree failures for the time being
/triage accepted
marking release-blocker per note on slack:
/priority critical-urgent (once upon a time we used to not ship a release if any priority/critical-urgent issues were open, not sure if we decided to stop doing that)
We were able to find the trigger for the CSI failures - on 6/29 there was a change in the hostpath driver test manifest, and the driver's is enabled. This makes the driver return an error instead of only logging a warning message on if the volume isn't unstaged properly. IIUC there's a long standing issue that reconstruction doesn't handle node unstage. could you confirm? If that's the case, then these CSI failures are not release blocking. Furthermore, the CSI test with graceful delete started to fail on 7/9 because of since graceful and force deletes are now considered the same from the point of view of the volume manager. cc
When that check was enabled, I don't think we realized it would impact the reconstruction path; it was enabled to validate a different issue.


Are we running into the issue that reconstruction of staging directories isn't supported () OR is it that our reconstruction path requires drivers to have real mount points ()? Either way since both are known gaps with reconstruction, I think it makes sense to to disable the flag until we can address those issues.
I was looking at this today too, trying to understand the underlying cause. That is why the failures are being reported now, but the part that I didn't understand is why NodeUnstageVolume is not being called for these volumes. In this instance: The DeleteVolume request fails because a staged path is still present: And in the kubelet log, NodeUnpublishVolume is called Then we hit this error: And NodeUnstageVolume is never called for this volume (same for other failures I looked at) so DeleteVolume keeps failing. So I think you're right that it's triggered by reconstruction, and it looks similar to , except in this case the restart happened before NodeUnpublishVolume.
And it's probably not since the test uses a PVC
Will look deeper into why UnmountDevice wasn't issued as well, but some update on the in-tree tests: the force delete test is skipped for the in-tree hostpath CSI driver because of Graceful tests are now failing as well because they behave like force tests due to
I don't think the issue is related to PVC or not. I think it's related to drivers that do not actually mount devices as a separate filesystem than the root.
There are three possible flows during kubelet restart pod gracefully delete case, the pod still exists (it is not considered as terminated in function isPodTerminated(); note that this logic has some changes after , see , so pod volume will be back to desired state, and reconciler will try to set up volume and fill in actual state. When pod volume is finally deleted from desired state, reconciler will trigger teardown with information from actual state to unpublish and unstage volume. pod is forcefully deleted, pod volume will not be in desired state. If volume reconstruction succeeds (be able to reconstruct volume spec), volume will be back into actual state. Then reconciler will trigger teardown with information from actual state to unpublish and unstage volume (this part is similar to case 1).


last case is reconstruction failed (in this csi-host-path test, reconstruction will fail due to issue related to , it fail to check mount point for csi hostpath driver because it is using a bind mount of the same device ). In this case, reconciler will try to clean up mounts directly, but only published volume path, not the staged volume path (this logic is not implemented due to missing information). As a workaround, we can now disable check-volume-lifecycle if it does not cause other issues. Then we will work on to make sure csi-host-path driver can pass volume mountpoint check during reconstruction.
This issue is now mitigated with merged. There are two longer-term issues that are isolated to hostpath plugins (and maybe other plugins that are similar in behavior, perhaps emptydir and similar?): related to IsLikelyNotMountPoint as mentioned. Because these don't impact the core volume machinery and are not regressions, this issue doesn't need to be release blocking. We'll close this issue once (2) is tracked.
is the IsLikelyNotMountPoint issue
after is merged, two csi-hostpath driver subpath test can pass, and two in-tree driver gracefully delete tests were skipped. https://k8s-
can you please cherry-pick to release-1.22 branch?
Opened cherry-pick, please review:
- what is the plan for reenabling them?
cherry-pick merged!
Cheng and I will work on solving IsLikelyNotMountPoint issue for hostpath drivers. That should allow test to pass and we can enable them.
Can this be removed from the milestone now that the workarounds have been merged, while leaving this issue open to track further work?
We can use existing issue to track the real fix. /close
Closing this issue. <detailsIn response to : We can use existing issue to track the real fix. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-kubernetes-e2e-gce-cos-k8sbeta-default Networking Granular Checks: Services should function for service endpoints using hostNetwork Triage looks like it's mostly flaking on this job only: Examples /sig network <!-- Please see the deflaking doc () for more guidance! --
/milestone v1.21 (marking in the current milestone until we get confirmation whether it is relevant to the release or not)
Slack Thread:
/assign
This is fixed by , let's wait some runs to confirm before closing
/milestone clear (since this has been deemed to be non release-blocking)
/triage accepted
/close
Closing this issue. <detailsIn response to : This is fixed by , let's wait some runs to confirm before closing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "LRUExpireCache" Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT What happened: requires write lock since needs to manipulate its list to track recently used item. Currently it uses read lock so it may introduce race condition. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else do we need to know:
This was fixed in


/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
<img width="1312" alt="image" src=""
This looks like a test you a few days ago: /assign Can you look into these failures?
GitHub didn't allow me to assign the following users: chengjoey. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This looks like a test you a few days ago: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
As mentioned in the previous comments, there should be only one expected state in the end. From the logs of these failed tests, after stopping the container, the phase changes to Failed. I think this does not meet expectations.
Thanks to for revert my previous PR. I think the pod final state should always be in one state. From the current e2e test results, the final state may be Succeed or Failed, seems pod.Status.Phase is empty. I think this seems to be wrong. I will study whether the node status (sufficient memory) and other factors interfere with the final state. Thanks again to for his help.


Add the log with phase "Succeed": The normal update status steps are 1-2-3-4, but the result of failure tests is that step 2 is executed last, which seems a bit strange. Because phase is empty
feel free to reopen the PR. But let's make sure to run the tests that this reported as failing before we merge. Thank you for your help on improving test coverage!
The links to the failed E2E tests in the overview of this issue all show the Pod phase as , but is there a link available somewhere for cases where it's empty string?
In fact, these failed tests are all empty strings, because not terminated: should be followed by phase Because my original test was written like this as I commented earlier, different order of updating status may lead to different final state
The Pod's final state is shown as in the dumped result immediately after the test fails. The referenced inside - is that coming from the Pod information fetched within ? Could it be that it's showing up as an empty string because it's not able to access the value externally?
In the tests submitted previously, and in my local(minnikube) tests, the pod status will change from to again see this log step 3 So I think the order of state changes affects the final result, and this is not very stable
I think this is a separate issue, so as I mentioned in , I thought it might be fine to wait for either the or phase. However, after looking at , it seems that we may need to resolve this unstable final result first. WDYT?
If we can explain different results, it is fine. I don't want unexplained check for either condition in our tests


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: [sig-auth] [Feature:NodeAuthorizer] A node shouldn't be able to create another node Since when has it been failing: N/A Testgrid link: N/A Reason for failure: The test creates a new ` node, but it does not delete it after the test ends: kubectl get nodes NAME STATUS ROLES AGE VERSION 1954k8s000 Ready agent 24d v1.14.1 1954k8s001 Ready agent 24d v1.14.1 foo NotReady <none4m3s k8s-master--0 Ready master 24d v1.14.1 Anything else we need to know: /sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .


Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


and show that we run this job every few minutes, but it never passes. I'm going to send out a PR to disable this job until someone has the time to investigate and resolve.
It's failing before tests even start:
At this point I don't think it's a flake, we needed flannel for a release and it's less important now so I'm not investigating why the cluster isn't coming up soon. I do believe if the cluster comes up a majority if not all of the tests will pass, and the ones that don't will flake for known reasons.


The "service" file in this document contains a trailing comma on the line ' "name": "my-service", ` and thus kubectl fails to parse it properly.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --This is probably just a documentation / example issue. I'm a real noob so I can just tell this is an issue, but I can't fix the docs through a PR, otherwise I'd do it happily. Hope it's not a dupe. Following the guide at with IPVS. Using the following script: This generates the following YAML (only the interesting part): You see that the line has an extra comma. It's instead of (extra comma after second argument) This generates the following log: Removing the extra comma fixes everything. Setup a cluster with Kube-DNS IPVS mode Generate the nodelocaldns Deploy the manifest Pods are crashing Kubernetes version v1.21.0 Cloud provider or hardware configuration: Bare Metal with HA setup (behind an external LB, Kube-proxy as IPVS) OS (e.g: ): CentOS Linux 8 Kernel (e.g. ): Linux 4.18.0-240.10.1.el83.x8664 SMP Mon Jan 18 17:05:51 UTC 2021 x8664 x8664 x86_64 GNU/LinuxInstall tools: none, a classic kubeadm initNetwork plugin and version (if this is a network-related bug): Not relevant, but Canal (Calico + Flannel) Others: nothing. Love you all for making K8S. Thank you a billion times.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig docs
, it seems like that don't get a value return. i saw command in Can you try this again and check if can kubedns get a value?
/area dns /sig network /assign Documentation here is not changed since Sep 2019 and needs to be updated as well.


Dual-stack and ipv6 are used more widely now.
I do get a value, but is replaced with as the extra comma doesn't seem to be replaced.
/assign


More detail can be seen in e2e test
cc
, Marek Grabowski wrote:
you are running an older version of the test. can you get the latest and try again ? This test had not failed at all yesterday in our automated test runs yesterday. Closing issue for now.
This failure depends on if the pod is really deleted and kube-proxy (the test container on) catch that the endpoint is deleted in 5s . After sending "delete pod" request for 5s, if the pod was still there and kube-proxy still contains this endpoints(this may happen under pressure), hit test can still detect 5 endpoints. At least we should ensure the pod was deleted. Anyway, I think will be helpful to this test. What do you think?
It doesn't matter if the pod is actually deleted or not. We just care that kube-proxy thinks that it should not be part of the service endpoint. Having said that, it still doesn't hurt to do the check that you are doing. I will assign your PR to myself and review it. In the meantime, can you rebase your PR ?
Thanks for your review. The PR is up to date.
Failed twice today in kubernetes-e2e-gce-parallel
One more today. The error message is not very helpful.
18:28:37 STEP: dialing(http) endpointPodIP:endpointHttpPort from node1 18:28:37 STEP: Dialing from node. command:for i in ' wc -l This is the point where the test fails. Am looking into it. We need to have more tries, we just do seq (1 1), instead of seq (6 6). this is causing flakiness.
Fixed by PR


Blocking merges, P0
Looks like this e2e code hasn't changed since .
Ugh. What changed is that we moved it off teh flaky list. Perhaps wrongly so. , Stephen Milner wrote:
Ah, I see. removed a number of e2e service items from variables. Relevant here:
the test is actually still running " Services should serve identically named services in different namespaces on different load-balancers" is running green.. Saad's fix where we correctly wait for GCE operations to complete, is checked in. Closing this issue.
Looks like this test is failing forever in and
Somehow I lost this one. Will look at it ASAP , Janet Kuo wrote:
The problem is that this test depends on the api proxy which doesn't do UDP, as far as I can tell, so it could never have worked when UDP support was . I will see what I can do to rectify.
Fixed test is - it's a doozy.
I've asked oncall to move this to the head of the queue. It's hard to figure out existing flakes when the test driving them has been totally changed. I want to re-examine all flakes after tghis
is in. I am going to close a few issues and wait for reoccurence, which is to be handled P(-1)


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Seems to be impacting a few KIND jobs on the master branch the most, but also assorted other jobs see the triage link above Reported by Sample failure: /sig node <!-- Please see the deflaking doc () for more guidance! --
xref probing tests
/assign ref /triage accepted
Thanks Elana, still trying today... Does anyone know if we can use the kubelet logs in e2e tests?
Not very portably in cluster e2e tests at least. Maybe in node_e2e instead.
Have you considered that the issue is not with the test but with the underlying code in kubelet like detailed in this issue Like detailed in the issue I can reliably reproduce (on k8s 1.18) the startup probe passing within a few seconds, but the container is only marked as ready after about a minute.
there is already a fix in the 1.21 release for that (part of a bigger PR): And actually this test was to verify that part of the code, but unfortunately sometimes it doesn't respond as quick as it should (but never on my PC, only in ci/cd tests)...
Some recent flakes:
one of the most common flakes in kind jobs currently failures like: Mar 25 17:03:42.508: Pod became ready in 42.007286524s, more than 5s after startupProbe succeeded. It means that the delay readiness probes were not initiated immediately after startup finished.
Maybe we could speed up the review on
Thank you I will take a look. I'm making a pass through flakes tonight trying to identify if anything is on kind's end
I don't think it's kind... since running on my PC with kind it works 100%, I think the issue is either the resources... or contention in the queue as mentioned to in the comment
/milestone v1.22
commented on PR with the suggested fix
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.


</details
Yeah...
Thanks for your great work in keeping things in check
FYI
I have (finally) understood what's happening, and I can even explain why the deflake made things worse... Please have a look at the sleep before the worker starts to process probes: So, you have to remember that we have one worker per pod and per probe, and since we have these probe settings for the test: We have (PeriodSeconds=10) starting sometimes immediately (if the Kubelet has started since more than 10s) or after a quick sleep of rand10s. On the contrary, will likely sleep most of the time for rand60s. This causes a delay in the processing of notifications: If we are lucky, this delay is less than 5s and the test passes: Now you can understand why the deflaked settings were actually worse:
The reason I didn't spot these was that I usually start my cluster once for all and iterate on the tests... so the kubelet had plenty of time to start and I never ended in that sleep.
For the solution, I suggest we either remove the sleep completely, or we find something that would make all 3 probe workers start at the same time to avoid sync issues... For the record, I was able to spot this issue thanks to the following log error (because the channel was overflowing): What do you think ?
I have found another issue in that code: If has already ran (and failed), we go to L245 and don't probe via and therefore wait for the next tick, making the test fail.
This seems consistent with what I found when I was adding e2es for the new probe-level termination grace period and why I did my timing math the way I did.
/reopen I don't think this is fixed, I just hit it again and gubernator is still showing plenty of flakes:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
how to check the exact commit used for each test?


For instance, I am not sure (from the name) that should include my latest fix...
In your PR the fix wasn't there:
you may send cherry-pick to 1.21 for which is using kubelet 1.21, not 1.22.
is using kubelet. Cherry-pick to 1.20 may be needed as well.
I have found this one that failed even with the fix: corresponding to:
I don't know what could explain this atm... but it seems we still have delay propagating container statuses.
In the job artifacts there are multiple places with this info including the clone records/logs, and In testgrid it's a row.
yeah afaict since the tests ran on my PR after your fix was merged into master, it showed the flake was still present :( Prow will always use the latest master when running CI.
Sorry for doubting what you said... I was probably in deny and hoped this annoyance was fixed once for all :(
Plan: we'll see if Clayton's pod lifecycle refactor and/or the jitter refactor for probes affects this.
Clayton's PR jitter refactor
since isn't (likely) coming to 1.22 we could cut off 90% of flakes by increasing to 15s delay:
it seems it fails even after merging so let's continue our discussion on slack
this is not a regression, our suggestion is not block release on it. feel free to remove the milestone
/milestone v1.23
In Kubelet maybe, but in the flakiness of the e2e tests it was. Unfortunately we can't easily explore the current state :/
In the release blocking jobs the frequency is not horrible now, so unblocking is probably fine, it is not as bad as it was. Outside of the release cycle this is still the flakiest test in some jobs, see e.g. where it's the only flake source we have on record currently.
if we end up needing it, we have a timing/period adjustment PR we can take that should help with the flakiness with


Looks like similar issue with
That's the PV controller this is the attach controller, thus separate issues.


Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No. Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST in we have that perform unnecessary shallow copy on pods () : we get the pods from the cache i think we do not mutate the pods (i.e only reading values) so we can avoid this deepCopy i think.
/assign
/remove-kind enhancement /kind technical-debt
That's actually a shallow copy
- I'm sorry i was thinking about another issue while filing this one.. I will update the issue, Thanks.
/sig network


1 tests failed, 11 jobs failed, 241 builds failed. Failure stats cover 1 day time range '17 Aug 2017 22:57 UTC' to '18 Aug 2017 22:57 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Seeing this failure on the e2e release-master-blocking https://k8s- Example failure https://k8s- Looks like an RBAC issue cc
this is currently blocking the alpha.3 release
looks like this new test was by in which was merged about 6 days ago and is failing consistently
This has started failing again on our GKE test suite https://k8s- https://k8s- cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/api-machinery: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
Seems to be flaking now with the error: https://k8s-
- we need to determine (today if possible) if this is truly release blocking. If so, please add the release-blocker label. And, if not, how do we best continue work on this for 1.8.x/1.9.0?
this test has two passes and four failures on the same commit I'm seeing gke-specific authz grants in that test that are incorrect: That commit removed the wait for the authz grant to become effective (which can lead to flakes), and granted superuser permissions to all users, which is incorrect and invalidates any other authz-related tests run in parallel with this test
cc
I can add the wait back.


edit: and fix the test given admin to all authenticated users.
Actually after staring at this test for about half an hour I can't figure out what different users exist or what permissions they're being granted. ClientSet, InternalClientset, and AggregatorClient are all initialized from the same config so I don't see how one would be able to create an RBAC binding but another would fail later. any thoughts here?
I honestly think the gke specific BindClusterRole is a red-herring. It is needed so the client has permission to perform one of the setup steps. (I think it was either to create the wardler cluster role or to bind that role to the anonymous user) Once that setup step is complete we no longer need the that cluster role bound and so I don't think its related.
The gke authorizer allows the “bind” verb, so the client can create a binding to the cluster-admin. It cannot create a role directly unless it has permissions via RBAC. Since we don’t have a way to determine the username associated with , binding to all authenticated users is what was done as a workaround.
I agree that the point at which the tests are failing indicate that the previous authorization issues are not the cause.
/open
/reopen
So a lot more information to work with now but the error is still occurring. I am still looking into this.
any update on the investigation?
Friendly v1.8 release team ping. This failure still seems to be happening, is this actively being worked? Does this need to be in the v1.8 milestone?


We've seen this a few times on GCE now, and I saw it flake on now as well (cc
cc Victor integrated cadvisor into the kubelet. I wonder if a fake version of cadvisor gets integrated into kubelet in the case of builds.
I noticed this first on GCE. (FWIW, GKE uses the same binaries as GCE.)
Jenkins shows the following four changes for the GCE build where this test started consistently failing: Commit by mattmoor Enable usage of a "json key" for authenticating with Commit by dbsmith Add a system modeler to scheduler Commit by dawnchen Convert RestartPolicy to string for v1beta3. Commit by abshah Build statically linked binaryies. With the change to go 1.4, we probably were generating dynamically linked binaries accidentally.
If I had to guess, I would take a stab and say it was , . cc
Feel free to revert by 5pm if no fix.
I believe and are correct, makes the build not use cgo which places a fake cAdvisor in the binary and that fails. Looking into the fix.
is out, it is a revert of the old one along with a plan for not needing static binaries.


First, the formatting. Second, "init-ctr-demo_demos" is an internal artifact, not fit for human-facing errors. Third, the container DOES exist, it's just in a terminating state.
The formatting issue should be fixed by . Regarding to the pod name, I think is better. As for the pod name, the error message is from the HTTP response of kubelet server. I've looked at the code, functions defined in interface are using ($podName_podNamespace) to identify a pod. Without spiting the parameter into two separated parameters, it could be tricky to change the output format of an error message. Personally I would prefer to change the signatures, but I don't know if we are planing to do it. In order to improve the readability, I a client-side check for pod status. See .


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: Discovery should validate PreferredVersion for each APIGroup Testgrid link: Reason for failure: The test does not handle API groups which are or removed dynamically during the run of the test. Just because an API group is present in at the beginning of the test does not mean or will be present by the end of the test. This is particularly true for the dynamic/custom-resource API groups registered/unregistered during the e2e test run /sig api-machinery /assign
mitigation opened at
Saw this today in It seems to happen relatively often Likely due to the CRD created in: Should the ValidatingAdmissionPolicy test CRD have its group name template changed to match the existing test exclusion for groups ending in /cc


[x] Original approval issue: Start 9 June 2020 3 Flakes No Flakes for 2 weeks According to this APIsnoop query, there are still some remaining ReplicationController endpoints which are untested. a ReplicationController with a static label the ReplicationController with a new Label and updated data the ReplicationController's status /Scale the ReplicationController's Replica count to 2 Replicas the ReplicationController the ReplicationController's Status all ReplicationControllers in all Namespaces find the ReplicationControllers(1) ensure that the ReplicationController is found and is patched Namespaced ReplicationControllers(1) via a Collection with a LabelSelector Discover useragents: See all endpoints hit from this test Display test coverage change If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
Can you please add in PR
/reopen Merged. Promotion to be created
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? http proxy, httpproxy, httpsproxy, AWS, kubeadm, kubectl, cloud-provider Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): v1.4.4 Environment: Cloud provider or hardware configuration: kubeadm on EC2 instances on AWS OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Linux ip-10-0-133-99 4.4.0-45-generic -Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: httpproxy, httpsproxy and noproxy set in /etc/environment, Docker config as well on startup scripts that run What happened: won't complete and gets stuck on it's first deployment. When one looks through the logs it shows that the pods are downloading fine but we are stuck due to the kube-controller-manager continually crashing as it can't contact AWS. Elsewhere in kubernetes I see references to and if the AWS cloud provider could use that it would be able to connect as well. From the POD if I manually set httpproxy and httpsproxy and no_proxy I am able to use the AWS CLI without any issues. What you expected to happen: I would expect to successfully complete and use the environment proxies set up on the system so as not to depend on a default Internet Gateway being connected to the VPC just for AWS access. How to reproduce it (as minimally and precisely as possible): Set up a public VPC with Squid set up as a proxy Set up a private VPC peered to the public VPC Set up a master node and run Set up a regular node and run will never fully initialize. If you remove the --cloud-provider aws` then it will complete and nodes will register. Anything else do we need to know:
what do you have in proxy variables ? (env | grep -i proxy) it tries to connect to (tcp 176.32.118.53:443) directly. You're expecting to connect there via proxy ? then do you have something like in your no_proxy ?


Unfortunately on the pod itself for the the environment variables aren't passed through. As this is created using I don't have control over the pod definition. So basically none of the *_proxy variables are set inside the pod in the docker image, however if I and export the variables and try to wget that URL it works. Basically I'm hoping we can update to respect and pass the environment variables down to the system pods if they exist, eg, add: Without it I can't stand up a cluster with AWS integration on a completely private VPC and for such a small change it would make a rather big difference.
ok, so request is actually to pass proxy variables into containers. interesting. I'll check how it can be done. BTW, for noproxy value in your example: things like 10.0.0.0/8 would not work on 99% of software that respects noproxy variable. it must be always exact IP match.
you're right that what I'm actually asking for is to pass proxy variables through either from a commandline setting on kubeadm or pick it up from /etc/environment or shell environment that launches kubeadm. I didn't realize it till I dug in to try and think about how one might go about fixing this. At first I thought we would just pick up the variables from the system environment but then I realized kube-controller-manager is deployed in a container and so we'd need to update the yaml file that launches it to contain the environment variables. As I'd like to go to production with this soon hopefully this is a small change that can be .
This issue was moved to kubernetes/kubeadm


What would you like to be : Deprecate the flag which enables exposing logs of apiserver. Why is this needed: As per , in order to disable accessing apiserver logs in the future, we need to deprecate the current default and wait for the requisite time period.
/sig api-machinery
/assign
I'm in favor of deprecating this. It's potentially disruptive, I think an email blast to kubernetes-dev would be a good idea.
Thanks , I have posted in kubernetes-dev mailing list.
I'm also in favor of removing this functionality from the apiserver. Since this is GA behavior, the deprecation period would be 12 months.
cc


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Today there are many obstacles preventing users from setting up FlexVolume on a GCE COS image easily. The default FlexVolume driver directory is not writable. Custom mount utilities can't be installed. Solutions exist for these problems but they are time-consuming to execute manually. On top of these, there are also the common FlexVolume pain points such as manual driver and mount utilities installation, and manually restarting kubelet. Ideally all these tasks should be automated, making it a quick and simple experience to have custom out-of-tree volume plugins. Is this a BUG REPORT or FEATURE REQUEST?: FEATURE REQUEST /kind feature
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig storage
/assign


The path to the server tarball resolves to: is hardcoded, and does not exist in release builds (v1.3.0-beta.2) The file in question is located at: Correcting the path is a workaround for release builds
cc
Hey, sorry about that. I think I got a fix, but I need your help to test it. I'm traveling and my cellular upload speeds are not so great. Please take a look at this and let me know if it solves the issue:


What happened: The stackdriver logging agent is installed and start running no matter logging is enabled/disabled in the cluster or not. What you expected to happen: Do not install and start the logging agent if logging is disabled.
/sig windows
cc


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When using IPVS and not natting (no cluster-cidr specified) requests from the host to service IPs are send with the service ip as the source ip. Let be the kubernetes API IP. Kubeproxy will set up with . This means that traffic from the host (which sees the ) will originate from (best matching ip/network). What you expected to happen: The source ip should be natted to the host ip. How to reproduce it (as minimally and precisely as possible): a ipvs based cluster any cluster-cidr from kube-proxy traffic to a service ip (e.g. http to the kubernetes api server) from the node / host network pod the outgoing interface looking for the service ip () Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: baremetal OS (e.g. from /etc/os-release): Kernel (e.g. ): (based on the Ubuntu 18.04 kernel) Install tools: custom repackaging of kubernetes release tarballs Others: /sig network
We have a similar issue in kube-router FWIW resolution is included in the issue
Ah nice, I reproduced the source IP selection on my local Ubuntu 18.04 without any kubernetes installed (use ip link add / ip addr add and then ping the new dummy IP / tcpdump lo). Might be related to the kernel version (4.15) as you couldn't reproduce it.... Changing the routing table would avoid an extra NAT rule. That sounds interesting but not doable without changing the kubernetes sources Murali Reddy < am Fr., 27. Apr. 2018, 14:06:
let me try with to reproduce. but its no brainer, this issue must be fixed.
getting closer... -CLUSTER-IP is only populated if masquerade-all or cluster-cidr is used: is no OUTPUT masquerading for service IPs if cluster-cidr is not specified (no branch matches): With cluster-cidr enabled I've saved the ipset.... I've disabled cluster-cidr and rebooted (to reset all iptables/ipset/...).


I think the code should: Always populate Set a masquerade rule for service IPs (src == service IP should not happen on OUTPUT)
How to test this with a local cluster.... the node (install , & modprobe the modules , , , ). a new shell: define a service with manual external endpoints tcpdump on the outgoing interface of the node the service ip the tcpdump, it will show outgoing traffic from_ the service IP:
Can you please take a look at this?
/assign
I assume this issue is related to some specific kernel versions? I just neither can't reproduce it nor can't see the failures in our test grid.
I was aware of there is a PR() in flight, my team will evaluate the effects of that PR. But, please rebase the PR first, thanks!


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e tests which use RBD server image fail with What you expected to happen: These tests succeed. How to reproduce it (as minimally and precisely as possible): Run e2e tests which use RBD server image. Anything else we need to know?: /sig storage This issue is introduced by below commit: It change the base image from fedora 26 to fedora 28. In fedora 26, hostname command is installed as a dependency for  However, in fedora 28, it is not installed, as a result, this issue happens due to lack of hostname command. Environment: Kubernetes version (use ): k8s 1.12 alpha
Please see the difference of the behaviour between fedora 26 and 28. [Fedora26] [Fedora28]
Could you help solving this issue? I assume the fix for the code would be as below, but I'm not familiar with the build and push process for the test image. ( in doesn't work.) So, I need your help to make further progress.
(or) Both need to be run from directory
Thank you for letting me know it. It should be s/all-containers/all-container/, but it seems to work. I will test with above patch and above build commands, then make a PR, once I confirm that it works well.
sorry typing from memory :)
I need to ask you two more things: Do we need to increment the version for this image, like 1.0.1, even for this small change? (I guess that and test/images/volume/rbd/VERSION will need to be updated in this case.) How the image push process will be? Does it pushed by CI, if there are a change in the code or docker file, or do someone who have permission will push it manually?
creates below three docker images: Is my understanding right that to make each test access to each arch image, needs to be modified like below? (Without doing so, each test seems to keep trying to access to , without arch.)
For fedora28 image, just installing hostname doesn't seem to work well. Please see the log below and attached . I guess that this is due to the difference of ceph version, which will be beyond my beyond my expertise.


So, could you help debugging this issue? Meanwhile, I will try testing with the image by using fedora26 for workaround.
I confirmed that below commit, which change the base image back to fedora 26, fix this issue. Please see the log . After I checked the image-, I come to think that by pushing the image, each image will be able to be pulled without -arch, so I guess the last parameter for ImageConfig should be false. (I couldn't test it in my environment.) I will fix the commit as so and make a PR once I confirm it. Also, if it will easy to fix with Fedora 28, I will wait for the actual fix and quit making a PR for the workaround.
thanks yes, a quick fix is welcome, looking forward
Thank you for your comment. I will continue with the workaround. I saw a commit for deleting multi arch consideration. I rebased the patch to the commit and made a PR. Please review it.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: VMAS nodes are reporting error in vmss cluster (vmType set to vmss). What you expected to happen: Node should register correctly in the kubernetes cluster. How to reproduce it (as minimally and precisely as possible): Create a VMAS nodes inside vmss cluster. Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure


0/8 is with a company's intranet, so it can be insecure. This is similiar to "For GCE, allow insecure registries anywhere in 10.0.0.0/8" Thanks!
Linking to and .
This was


What would you like to be : There is a tab in the k8s perf dashaboard for the scheduler perf benchmarks, but it is showing nothing, we should fix that: http://perf- What this dashboard should be showing is multiple runs per day of We can run it every hour, and summarize the 24 runs of each day to show 50%, 90% and 99% percentile. Why is this needed: Having daily runs of the benchmarks is important to track performance regressions. /sig scheduling /assign /priority important-soon
Ref
/cc
The ci job's config is at And one of failed logs could be found at I tried to reproduce it on my local machine(because I do not have Gloud credentials, I run the task directly in the container): However it works well on my local machine. It is so weird. It will be very appreciated if anyone could help reproduce it with Gloud credentials.
can you try and reach out to the test-infra people? ping them on the slack channel perhaps.
yes, I have tried to get help from , he is busy now. I'll try more.
While working on reproducing this in order to file a bug against test-infra folks I think I found a solution. Will send PRs shortly. /assign
/retitle sig-scalability-benchmarks#scheduler fails consistently due to build issues
/sig scalability Testgrid link: https://k8s- Error logs:
A word about the solution, the job uses the deprecated script which is not supported anymore by test-infra oncall / folks. I've rewritten the job definition to use pod utilities and it started working (will require to get merged first).
Thanks
Thanks !


/kind cleanup /sig testing /priority important-longterm This is a followup issue to address the "standard build globals" nit on At the very least I think we shouldn't capture our custom env var. Also think it'd be worth documenting the assumed variables in a list, similar to
cc
/assign


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I have tried FlexVolume on a Windows node, it does not work. I am now working on this feature now. Below is the FlexVolume initialization error What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.7.7 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig windows /assign
Detailed steps about how to use flexvolume on Windows with this PR:
cc:


With the move to running the kube-apiserver in a pod, I am getting errors of the form: The root null pointer is from this line: The line numbers in the stack are from my continuing debug of the original null pointer. We need to protect against a null pointer in this path, which appears to only be invoked when doing a list call that matches a single item based on field selector. Will submit a PR to address.
With proper line numbers:


I am tracking a changed behavior in networking with kubernetes 1.26. If I create a service of type externalTrafficPolicy: Local, and there are no local endpoints in a node, kube-proxy does not create hns loadbalancers for the clusterIP of that service. With kubernetes 1.25, it creates a loadbalancer for clusterIP (but not externalIP). This is the relevant log message: Looks like this comes with a result from a new function isAllEndpointsTerminating introduced in kubernetes 1.26. In Linux, we can still access clusterIP even if there are no local endpoints for the service. Thoughts? The end result is that in Windows, clients cannot access the clusterIP of such a service with there are no local endpoints.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
cc
Does this issue occur when there are no local endpoints for a given service? If there is a local endpoint, then you can reach the service again, right?
, Also let us know, do you have any InternalTrafficPolicy configured ?
Does this issue occur when there are no local endpoints for a given service? If there is a local endpoint, then you can reach the service again, right? That is correct , Also let us know, do you have any InternalTrafficPolicy configured ? We do not have that configured


download- refers to release 1.1.4 but the latest is 1.1.7
Resolved by PR


Looks like CSI volume limit test is flaking. /sig storage /assign
This flake isn't happening in normal clusters, appears to be alpha only issue.


What happened: In handler#handleSummary, return value from request.Request.ParseForm() is ignored. What you expected to happen: The error return should be checked.
/sig node
Hi this seems to be easy to fix. Could I solve this?
Please go ahead.


BUG REPORT When using with multiple files (like when referencing a directory in ), the json or yaml output is not a valid list format, and when fed to just the first file gets created. Kubernetes version (use ): latest master What happened: only the first file in directory gets created. What you expected to happen: all files in directory must be created. How to reproduce it (as minimally and precisely as possible):


needs a test to ensure that clauses are implemented correctly. Specifically, that they are not implemented as "deny" rules. eg, given a client pod with IP , if you add a NetworkPolicy with: spec: ingress: from: ipBlock: cidr: 0.0.0.0/0 except: A.B.C.0/24 then it should block ingress from that pod. But if you then add a second policy with: spec: ingress: from: ipBlock: cidr: A.B.C.D/32 then it should reallow ingress from that pod. Then if you delete the first policy, ingress should still be allowed, and if you create the first policy again, ingress should still be allowed. (ie, the result does not depend on which order the two policies were created in). /sig network /priority important-longterm
I can work on this as my first issue related to sig-network.
just noticed this issue while working on a similar effort. since there was no activity for the past 2 months I pushed a PR for this scenario