Question: stdoutBuf := bytes.NewBuffer(nil) stderrBuf := bytes.NewBuffer(nil) w := newLogWriter(stdoutBuf, stderrBuf, &LogOptions{since: test.since, timestamp: test.timestamp, bytes: -1}) <del> err := w.write(msg) </del> <ins> err := w.write(msg, true) </ins> assert.NoError(t, err) assert.Equal(t, test.expectStdout, stdoutBuf.String()) assert.Equal(t, test.expectStderr, stderrBuf.String())

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen will fix it.
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: stdoutBuf := bytes.NewBuffer(nil) stderrBuf := bytes.NewBuffer(nil) w := newLogWriter(stdoutBuf, stderrBuf, &LogOptions{since: test.since, timestamp: test.timestamp, bytes: -1}) <del> err := w.write(msg) </del> <ins> err := w.write(msg, true) </ins> assert.NoError(t, err) assert.Equal(t, test.expectStdout, stdoutBuf.String()) assert.Equal(t, test.expectStderr, stderrBuf.String())

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .


Question: stdoutBuf := bytes.NewBuffer(nil) stderrBuf := bytes.NewBuffer(nil) w := newLogWriter(stdoutBuf, stderrBuf, &LogOptions{since: test.since, timestamp: test.timestamp, bytes: -1}) <del> err := w.write(msg) </del> <ins> err := w.write(msg, true) </ins> assert.NoError(t, err) assert.Equal(t, test.expectStdout, stdoutBuf.String()) assert.Equal(t, test.expectStderr, stderrBuf.String())

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I received a bug report about this issue. The root cause is that the buffer reads up to a new line. If the line is greater than 4096 bytes and '--timestamps' is turrned on the kubelet will write the timestamp and the partial log line. We will need to refactor the ReadLogs function to allow for a partial line read.


Question: } func (npm *nominatedPodMap) update(oldPod, newPod *v1.Pod) { <ins> // In some cases, an Update event with no "NominatedNode" present is received right // after a node("NominatedNode") is reserved for this pod in memory. // In this case, we need to keep reserving the NominatedNode when updating the pod pointer. nodeName := "" // We won't fall into below `if` block if the Update event represents: // (1) NominatedNode info is added // (2) NominatedNode info is updated // (3) NominatedNode info is removed if NominatedNodeName(oldPod) == "" && NominatedNodeName(newPod) == "" { if nnn, ok := npm.nominatedPodToNode[oldPod.UID]; ok { // This is the only case we should continue reserving the NominatedNode nodeName = nnn } } </ins> // We update irrespective of the nominatedNodeName changed or not, to ensure // that pod pointer is updated. npm.delete(oldPod) <del> npm.add(newPod, "") </del> <ins> npm.add(newPod, nodeName) </ins> } func (npm *nominatedPodMap) podsForNode(nodeName string) []*v1.Pod {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: kubernetes/test/integration/scheduler.TestPreemptionRaces Since when has it been failing: Not exactly known. The flake rate is not high. That's probably why it has not received attention. Testgrid link: Reason for failure: Logs are currently unaccessible. Test grid returns an error when clicking on the failed instances. Anything else we need to know: /sig scheduling /label help-wanted
Let me try to fix it. /assign /remove-help
These are the last 4 failure log in . Although The flake rate is not high, but the fail point is same. I will try to solve this error based on this information.
This is currently the flakiest test for the flakiest PR job for kubernetes/kubernetes
/remove-kind failing-test /kind flake /priority important-soon /milestone v1.15
I will try to work out a PR if is not working on it.
Sure, you can take over it if you want. And It is strange about the flake in . I ran about 3,000 times on a local machine, but never failed. :joy: I also can help you to do someting ~
Indeed, flaky tests are usually difficult to repro and debug.. A heads-up: I seem to have found the root cause. I'm working on a solution and will raise a PR after thorough testing.
this is still flaking (e.g. )
I will take another look.
There are two additional flakes since the one reported on May 20: It's no longer that flaky , but the PR I gave earlier proved not to be a thorough solution but mitigation. I may not be able to continue looking into it in the next couple of weeks. Anyone is in interested, feel free to take it.
Last failure was 4 days ago. will continue observing and see if it clears out /milestone v1.16
Been observing this one for a bit and it seems like its getting better Other than some failures a couple days ago it has been stable. Will keep an eye for a couple more days to make sure.
Seems to have stabilized. /close
Closing this issue. <detailsIn response to : Seems to have stabilized.


Question: } func (npm *nominatedPodMap) update(oldPod, newPod *v1.Pod) { <ins> // In some cases, an Update event with no "NominatedNode" present is received right // after a node("NominatedNode") is reserved for this pod in memory. // In this case, we need to keep reserving the NominatedNode when updating the pod pointer. nodeName := "" // We won't fall into below `if` block if the Update event represents: // (1) NominatedNode info is added // (2) NominatedNode info is updated // (3) NominatedNode info is removed if NominatedNodeName(oldPod) == "" && NominatedNodeName(newPod) == "" { if nnn, ok := npm.nominatedPodToNode[oldPod.UID]; ok { // This is the only case we should continue reserving the NominatedNode nodeName = nnn } } </ins> // We update irrespective of the nominatedNodeName changed or not, to ensure // that pod pointer is updated. npm.delete(oldPod) <del> npm.add(newPod, "") </del> <ins> npm.add(newPod, nodeName) </ins> } func (npm *nominatedPodMap) podsForNode(nodeName string) []*v1.Pod {

/close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // ListRegisteredPriorityFunctions returns the registered priority functions. func ListRegisteredPriorityFunctions() []string { <del> schedulerFactoryMutex.Lock() defer schedulerFactoryMutex.Unlock() </del> <ins> schedulerFactoryMutex.RLock() defer schedulerFactoryMutex.RUnlock() </ins> var names []string for name := range priorityFunctionMap {

What would you like to be : Currently Mutex lock is acquired in for accessing different maps. Some functions only read map entry: We can utilize RWMutex where the above function acquires read lock. Why is this needed: For concurrent calls to IsFitPredicateRegistered / IsPriorityFunctionRegistered, read lock would lower contention.
/sig scheduling
functions getFitPredicateFunctions, getPriorityFunctionConfigs, ListRegisteredFitPredicates, ListRegisteredPriorityFunctions and GetAlgorithmProvider can also use read lock.
I happened to raise earlier today. Anyone can finish resting ones.
Good to know,
Thanks for raising this ! My PR covered and . Feel free to raise another PR to cover others.
Once is merged, I will send my PR thanks
it's merged now.


Question: #   google         - Heapster, Google Cloud Monitoring, and Google Cloud Logging #   googleinfluxdb - Enable influxdb and google (except GCM) #   standalone     - Heapster only. Metrics available via Heapster REST API. <del> ENABLE_CLUSTER_MONITORING="${KUBE_ENABLE_CLUSTER_MONITORING:-googleinfluxdb}" </del> <ins> ENABLE_CLUSTER_MONITORING="${KUBE_ENABLE_CLUSTER_MONITORING:-influxdb}" </ins> # Optional: Enable node logging. ENABLE_NODE_LOGGING="${KUBE_ENABLE_NODE_LOGGING:-true}"

Heapster pod: And logs: cc p0 for triage
Did the required node scope change?
Its either the API on the console or monitoring write scopes on the node. , Mike Danese wrote:
I think this affects only GCE
I will fix this issue since is on vacation.
Need to cherry pick so i'll leave open
is out for review.
has been closed and the cherrypick here () has been merged so I'm closing this issue. Thanks everyone for the debugging.


Question: want      []string } testFn := func(test *testcase, t *testing.T) { <del> SortControllerRevisions(test.revisions) for i := range test.revisions { if test.revisions[i].Name != test.want[i] { t.Errorf("%s: want %s at %d got %s", test.name, test.want[i], i, test.revisions[i].Name) </del> <ins> t.Run(test.name, func(t *testing.T) { SortControllerRevisions(test.revisions) for i := range test.revisions { if test.revisions[i].Name != test.want[i] { t.Errorf("%s: want %s at %d got %s", test.name, test.want[i], i, test.revisions[i].Name) } </ins> } <del> } </del> <ins> }) </ins> } ss1 := newStatefulSet(3, "ss1", types.UID("ss1"), map[string]string{"foo": "bar"}) ss1.Status.CollisionCount = new(int32)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Statefulsets are sometimes continually rolling pods and I don't know why statefulset created: storageclass referenced: kubectl describe statefulset statefulset-test: As you can see from the above describe, the pod in the statefulset has been restarted a lot of times. Observing the created pod, 1) The pod comes up ready 2) immediately transitions to terminating 3) Is terminated 4) A new pod is created 5) Go back to (1) (repeats forever) Despite no updates to the statefulset spec ocurring. Making a change to the statefulset template (for example a trivial change adding a nonsense annotation) stops the rolling and fixes the issue When this statefulset was created, there was a mutating webhook enabled. After the observed pod rolling. I disabled this by making its configuration invalid and making it non required, and the rolling persisted. The status of the statefulset: : You can see that there is a discrepency between currentRevision and updateRevision. currentRevision: statefulset-test- updateRevision: statefulset-test- A short while later the status is changed to contain currentRevision: statefulset-test- updateRevision: statefulset-test- This change happened at Sun Apr 1 19:48:48 2018 Presumably after the rollout of statefulset-test- is complete.


Question: want      []string } testFn := func(test *testcase, t *testing.T) { <del> SortControllerRevisions(test.revisions) for i := range test.revisions { if test.revisions[i].Name != test.want[i] { t.Errorf("%s: want %s at %d got %s", test.name, test.want[i], i, test.revisions[i].Name) </del> <ins> t.Run(test.name, func(t *testing.T) { SortControllerRevisions(test.revisions) for i := range test.revisions { if test.revisions[i].Name != test.want[i] { t.Errorf("%s: want %s at %d got %s", test.name, test.want[i], i, test.revisions[i].Name) } </ins> } <del> } </del> <ins> }) </ins> } ss1 := newStatefulSet(3, "ss1", types.UID("ss1"), map[string]string{"foo": "bar"}) ss1.Status.CollisionCount = new(int32)

Here are all the controllerrevisions from above: kubectl get controllerrevision statefulset-test- -o yaml kubectl get controllerrevision statefulset-test- -o yaml: kubectl get controllerrevision statefulset-test- -o yaml: The differences between the above controllerrevisions are insignificant (the hash, the creation timestamp, the name, the resource version, the uid, the self link) Interestingly all these controllerrevisions were created very close together in time shortly after I created the statefulset The current controllerrevisions are After watching the state change for a while, it actually seems like the statefulset is just cycling through the updateRevision being one of the above 10 controllerrevisions Leaders during this time From kube-controller-manager logs during this time, here is one excerpt: The error sync in there is suspicous, and as you can see the controllerrevisions are changing in the logs I have the full logs from a 5 minute window on either side of the above if necessary. The reason I'm inclined to think this is a kubernetes bug is that any update to the statefulset template fixes the problem. How to reproduce it (as minimally and precisely as possible): Not really sure. Worth just trying to create the above statefulset (requires creating the I tried deleting and recreating the above statefulset and it didn't happen, so something flaky is going on Anything else we need to know?: kubernetes 1.9.3 (client and server) cloud provider: aws
/sig apps
I realize this will probably need some further debugging, happy to get some direction
Consistent repro: Create a statefulset that creates a controllerrevision whose hash is between max int32 and max uint32 (this shouldn't be that hard to do, possibly a 50/50 chance of this). Manually create a controllerrevision identical to the first (same revision), incrementing both the name and hash by one. See that the statefulset controller will continually switch which controller revision should be the target of the statefulset.
Due to the issue fixed here: , under some circumstances controllerrevisions whose revision is the same were being considered equal.


Question: want      []string } testFn := func(test *testcase, t *testing.T) { <del> SortControllerRevisions(test.revisions) for i := range test.revisions { if test.revisions[i].Name != test.want[i] { t.Errorf("%s: want %s at %d got %s", test.name, test.want[i], i, test.revisions[i].Name) </del> <ins> t.Run(test.name, func(t *testing.T) { SortControllerRevisions(test.revisions) for i := range test.revisions { if test.revisions[i].Name != test.want[i] { t.Errorf("%s: want %s at %d got %s", test.name, test.want[i], i, test.revisions[i].Name) } </ins> } <del> } </del> <ins> }) </ins> } ss1 := newStatefulSet(3, "ss1", types.UID("ss1"), map[string]string{"foo": "bar"}) ss1.Status.CollisionCount = new(int32)

This ends up causing this bug to manifest as follows: ListRevisions finds many controllerrevisions with the same revision (unclear how this state happens, but it can happen) The sort here randomly sorts them Due to the linked bug , FindEqualRevisions incorrectly finds that all these controller revisions are equal to the proposed udpate. The updateRevision selects the last revision found here Therefore, updateRevision, is randomly selected and causes the bug. This fixes the immediate issue, but the sort should probably be made stable (i.e not just look at the revision number, perhaps just consider the hash for tie breakers)
As an update, fixing the equality check will not be sufficient because it is possible for multiple controller revisions to be created that are actually equal, including the same hash, that have the same revision number. I believe the fix will need to be equal controllerrevisions with the same revision number from being created sort the controllerrevisions so that the controllerrevision that is the target of the statefulset is stable
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
ping I use the K8S which version is 1.11.4 and encounter the same problem. I want to reproduce the problem and find your way: Manually create a controllerrevision identical to the first (same revision), incrementing both the name and hash by one. Could you tell me how to mannually create a same controllerrevision identical to the first? Thanks in anvance.


Question: l.migrationCheck.validateMigrationVolumeOpCounts() } <del> ginkgo.It("should write files of various sizes, verify size, validate content [Slow][LinuxOnly]", func() { </del> <ins> ginkgo.It("should write files of various sizes, verify size, validate content [Slow]", func() { </ins> init() defer cleanup()

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Four tests: Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Kubernetes e2e suite.[sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is force deleted while kubelet is down [Disruptive][Slow][LinuxOnly] is failing since June 29th 2021 (but the first failure was different than more recent ones). Commit , infra commit , . Other three tests are failing since July 9th 2021. Commit , infra commit , . Testgrid: https://k8s- Prow history: For CSI Volumes: For In-tree Volumes: There is another open issues for failing tests in this job: Failures reported there are about missing metrics, and were occuring before unmounting problems. So I opened a new issue, even though it's the same SIG. These tests don't look related to me, but I don't have enough context, maybe they somehow are. /sig storage /kind failing-test /milestone 1.22
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.


Question: l.migrationCheck.validateMigrationVolumeOpCounts() } <del> ginkgo.It("should write files of various sizes, verify size, validate content [Slow][LinuxOnly]", func() { </del> <ins> ginkgo.It("should write files of various sizes, verify size, validate content [Slow]", func() { </ins> init() defer cleanup()

<detailsIn response to : <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Four tests: Testgrid: https://k8s- Prow history: For CSI Volumes: For In-tree Volumes: #### Anything else we need to know: /kind failing-test /milestone 1.22 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
/cc
/cc (my close/reopen yesterday was accidental, sorry)
I can take a look /assign
The non-subpath counterpart of this test is passing: These tests seem to use the default storageclass, whereas failing tests use in-tree hostpath and csi-hostpath
The CSI ones are failing because the hostpath driver never received a NodeUnstage (whereas it did in passing tests), causing the driver to error on volume deletion.
In the failed run kubelet has this error: which seems to have prevented kubelet from triggering a UnmountDevice. This error didn't exist in a successful run.
Hostpath driver can run with and without support for attach. Do you know which mode is enabled for this test?
Between the last passing time and when the test started failing, are there any PRs merged that look suspicious?
According to csi-attacher logs, attach is disabled for both the good run and the failing run. Haven't spotted any obvious PRs so far, but I did find this: in the successful run, the log message for the successful is this: and the failed run: so the reconstructed volume names are somehow different. Checking reconstruction logic
This merged 12 days ago and was a pretty big change to pod lifecycle and could have trickling impact to volume manager. Does the timing of test failure match?
There is also this reconstruction fix that merged yesterday and seems related to outer vs inner spec name. I don't know if that will help fix this issue:
cc
Last successful run started at 7/8 17:49 PT, first failing run started at 7/9 2:23 PT.


Question: l.migrationCheck.validateMigrationVolumeOpCounts() } <del> ginkgo.It("should write files of various sizes, verify size, validate content [Slow][LinuxOnly]", func() { </del> <ins> ginkgo.It("should write files of various sizes, verify size, validate content [Slow]", func() { </ins> init() defer cleanup()

was merged at 7/8 16:48 PT, so it's possible (if there's a delay in picking up master) Also this is in the failing kubelet logs but not in the successful one: I0720 18:20:42. ] "Reconciler sync states: could not find volume information in desired state, clean up the mount points" podName=-e508-45c2-8509- volumeSpecName="pvc--a615-4e89-afe8-"
From , the newly introduced log message here exists in the failing kubelet logs but not the successful one:
Failures starting 07/08: /sig node for visibility
I think caused the failures. Before after kubelet restarts, pod will still be in desired state (even through it is in terminating state), so that clean up mounts can be done completely. I think now, terminating pods will not be back to desired state, but reconstruction to actual state cannot completely clean up mounts. I am testing it about this.
Is there a reason why reconstruction to actual state cannot clean up mounts? Note that a pod can be force deleted at any time, so a terminating pod that is force deleted would still have to be capable of being cleaned up (it won't be in desired state)? (note too that the expectation of the changes in is that terminating and fully terminated and "never seen before" should all completely equivalent from the perspective of other loops in the kubelet) If I understand you correctly, is this an existing bug that we're now triggering consistently?
We have both "grace pod deletion" and "force pod deletion" test cases that are failing.
There were a couple of key changes to volume loops in : Roughly before was "it looks like it's got no running containers", and after my change is "we are going to be stopping all the containers from now on". So this change meant that once a pod starts terminating, we are in "tearing down mode" and so my understanding of this method was that "any pod where containers are being torn down is not in the desired state of the world".


Question: l.migrationCheck.validateMigrationVolumeOpCounts() } <del> ginkgo.It("should write files of various sizes, verify size, validate content [Slow][LinuxOnly]", func() { </del> <ins> ginkgo.It("should write files of various sizes, verify size, validate content [Slow]", func() { </ins> init() defer cleanup()

After a restart, we would definitely not add a pod back that was in terminating state into desired state of the world, but "restart + this change" should be roughly equivalent to "restart and force delete while restart is processing" The other change was in which changed a live CRI read (which was trying to enforce that the dsowp is synchronized with the pod worker) into a threadsafe check against the pod worker directly (the pod worker can provide authoritative answers to "this pod has no running containers" now). The latter change seems less likely to impact you.
Thank you for the details! I think your change is making the right change, but unfortunately some of volume management still have issues, and so the tests rely on some behaviors that is not very right (pod volume is back to desired state even after it is in terminating) The cleanup mount currently does not have the logic to clean up global mount point if the volume plugin sets it up. I am still testing more and will share more details.
is this a release blocker?
Yes, this impacts volume cleanup if kubelet restarts for any reason.
On a kube-up cluster running the latest master against the latest e2e binary, I was able to reproduce the in-tree failures but not the CSI ones for some reason. Will focus on in-tree failures for the time being
/triage accepted
marking release-blocker per note on slack:
/priority critical-urgent (once upon a time we used to not ship a release if any priority/critical-urgent issues were open, not sure if we decided to stop doing that)
We were able to find the trigger for the CSI failures - on 6/29 there was a change in the hostpath driver test manifest, and the driver's is enabled. This makes the driver return an error instead of only logging a warning message on if the volume isn't unstaged properly. IIUC there's a long standing issue that reconstruction doesn't handle node unstage. could you confirm? If that's the case, then these CSI failures are not release blocking. Furthermore, the CSI test with graceful delete started to fail on 7/9 because of since graceful and force deletes are now considered the same from the point of view of the volume manager. cc
When that check was enabled, I don't think we realized it would impact the reconstruction path; it was enabled to validate a different issue.


Question: l.migrationCheck.validateMigrationVolumeOpCounts() } <del> ginkgo.It("should write files of various sizes, verify size, validate content [Slow][LinuxOnly]", func() { </del> <ins> ginkgo.It("should write files of various sizes, verify size, validate content [Slow]", func() { </ins> init() defer cleanup()

Are we running into the issue that reconstruction of staging directories isn't supported () OR is it that our reconstruction path requires drivers to have real mount points ()? Either way since both are known gaps with reconstruction, I think it makes sense to to disable the flag until we can address those issues.
I was looking at this today too, trying to understand the underlying cause. That is why the failures are being reported now, but the part that I didn't understand is why NodeUnstageVolume is not being called for these volumes. In this instance: The DeleteVolume request fails because a staged path is still present: And in the kubelet log, NodeUnpublishVolume is called Then we hit this error: And NodeUnstageVolume is never called for this volume (same for other failures I looked at) so DeleteVolume keeps failing. So I think you're right that it's triggered by reconstruction, and it looks similar to , except in this case the restart happened before NodeUnpublishVolume.
And it's probably not since the test uses a PVC
Will look deeper into why UnmountDevice wasn't issued as well, but some update on the in-tree tests: the force delete test is skipped for the in-tree hostpath CSI driver because of Graceful tests are now failing as well because they behave like force tests due to
I don't think the issue is related to PVC or not. I think it's related to drivers that do not actually mount devices as a separate filesystem than the root.
There are three possible flows during kubelet restart pod gracefully delete case, the pod still exists (it is not considered as terminated in function isPodTerminated(); note that this logic has some changes after , see , so pod volume will be back to desired state, and reconciler will try to set up volume and fill in actual state. When pod volume is finally deleted from desired state, reconciler will trigger teardown with information from actual state to unpublish and unstage volume. pod is forcefully deleted, pod volume will not be in desired state. If volume reconstruction succeeds (be able to reconstruct volume spec), volume will be back into actual state. Then reconciler will trigger teardown with information from actual state to unpublish and unstage volume (this part is similar to case 1).


Question: l.migrationCheck.validateMigrationVolumeOpCounts() } <del> ginkgo.It("should write files of various sizes, verify size, validate content [Slow][LinuxOnly]", func() { </del> <ins> ginkgo.It("should write files of various sizes, verify size, validate content [Slow]", func() { </ins> init() defer cleanup()

last case is reconstruction failed (in this csi-host-path test, reconstruction will fail due to issue related to , it fail to check mount point for csi hostpath driver because it is using a bind mount of the same device ). In this case, reconciler will try to clean up mounts directly, but only published volume path, not the staged volume path (this logic is not implemented due to missing information). As a workaround, we can now disable check-volume-lifecycle if it does not cause other issues. Then we will work on to make sure csi-host-path driver can pass volume mountpoint check during reconstruction.
This issue is now mitigated with merged. There are two longer-term issues that are isolated to hostpath plugins (and maybe other plugins that are similar in behavior, perhaps emptydir and similar?): related to IsLikelyNotMountPoint as mentioned. Because these don't impact the core volume machinery and are not regressions, this issue doesn't need to be release blocking. We'll close this issue once (2) is tracked.
is the IsLikelyNotMountPoint issue
after is merged, two csi-hostpath driver subpath test can pass, and two in-tree driver gracefully delete tests were skipped. https://k8s-
can you please cherry-pick to release-1.22 branch?
Opened cherry-pick, please review:
- what is the plan for reenabling them?
cherry-pick merged!
Cheng and I will work on solving IsLikelyNotMountPoint issue for hostpath drivers. That should allow test to pass and we can enable them.
Can this be removed from the milestone now that the workarounds have been merged, while leaving this issue open to track further work?
We can use existing issue to track the real fix. /close
Closing this issue. <detailsIn response to : We can use existing issue to track the real fix. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } }) <del> ginkgo.It("should function for pod-Service(hostNetwork): udp", func() { config := e2enetwork.NewNetworkingTestConfig(f, e2enetwork.EndpointsUseHostNetwork) ginkgo.By(fmt.Sprintf("dialing(udp) %v --> %v:%v (config.clusterIP)", config.TestContainerPod.Name, config.ClusterIP, e2enetwork.ClusterUDPPort)) err := config.DialFromTestContainer("udp", config.ClusterIP, e2enetwork.ClusterUDPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } ginkgo.By(fmt.Sprintf("dialing(udp) %v --> %v:%v (nodeIP)", config.TestContainerPod.Name, config.NodeIP, config.NodeUDPPort)) err = config.DialFromTestContainer("udp", config.NodeIP, config.NodeUDPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } }) </del> // if the endpoints pods use hostNetwork, several tests can't run in parallel // because the pods will try to acquire the same port in the host. // We run the test in serial, to avoid port conflicts.

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-kubernetes-e2e-gce-cos-k8sbeta-default Networking Granular Checks: Services should function for service endpoints using hostNetwork Triage looks like it's mostly flaking on this job only: Examples /sig network <!-- Please see the deflaking doc () for more guidance! --
/milestone v1.21 (marking in the current milestone until we get confirmation whether it is relevant to the release or not)
Slack Thread:
/assign
This is fixed by , let's wait some runs to confirm before closing
/milestone clear (since this has been deemed to be non release-blocking)
/triage accepted
/close
Closing this issue. <detailsIn response to : This is fixed by , let's wait some runs to confirm before closing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } else { handler = restfulGetResource(getter, exporter, reqScope) } <del> handler = metrics.InstrumentRouteFunc(action.Verb, resource, subresource, namespaceScope, handler) </del> <ins> if needOverride { // need change the reported verb handler = metrics.InstrumentRouteFunc(verbOverrider.OverrideMetricsVerb(action.Verb), resource, subresource, namespaceScope, handler) } else { handler = metrics.InstrumentRouteFunc(action.Verb, resource, subresource, namespaceScope, handler) } </ins> if a.enableAPIResponseCompression { handler = genericfilters.RestfulWithCompression(handler, a.group.Context) }

Is this a BUG REPORT or FEATURE REQUEST?: What happened: use kubectl search log with follow option get api latency (apiserverrequestlatenciessummary{quantile="0.99",resource="pods",verb="GET"} )high to give a alert in my cluster. the apiserver log is below and we can see it costs 1m25s. What you expected to happen: do not make this api latency account in getting pod request api latency How to reproduce it (as minimally and precisely as possible): use kubectl to search a pod log with follow option long enough and you will get this metric apiserverrequestlatenciessummary{quantile="0.99",resource="pods",verb="GET"} higher than the ordinary case. Anything else we need to know?: I am happy to fix it. Environment: Kubernetes version (use ): 1.6 Cloud provider or hardware configuration**: bare-metal OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): 4.4 Install tools: kubeadm Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig API Machinery
/area apiserver
/sig api-machinery
/assign Thanks... looking forward to your PR!
fixed with PTAL


Question: case "get": // Return an existing (matching) node on get. return &api.Node{ <del> ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}, Spec:       api.NodeSpec{ExternalID: "127.0.0.1"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}, Spec:       api.NodeSpec{ExternalID: testKubeletHostname}, </ins> }, nil default: return nil, fmt.Errorf("no reaction implemented for %s", action.Action)

Looks like some kubelet unit tests perform a lookupIp which ends up depending on the health of the local resolver. Specifically has this failing: Which leads to the test in: And we recently swapped out the node names from testnode to 127.0.0.1, so probably the culprit?
/ref
Interesting. I did swap out the node names in the unit test as part of to be 127.0.0.1, party because I thought that it wouldn't have any problems resolving during unit tests.
/cc


Question: case "get": // Return an existing (matching) node on get. return &api.Node{ <del> ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}, Spec:       api.NodeSpec{ExternalID: "127.0.0.1"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}, Spec:       api.NodeSpec{ExternalID: testKubeletHostname}, </ins> }, nil default: return nil, fmt.Errorf("no reaction implemented for %s", action.Action)

cc
What is the environment where you are running the tests? They pass for me on my mac, a linux box, and shippable seems to be passing them as well.
I couldn 't reproduce the failure either.
I'm out right now; i'll try them when i get back. For the record, I run hack/test- Paul Morie wrote:
Ahem FTR This acronym is going to go quad-plat Paul Morie wrote:
I ran on both platforms and saw the tests pass.
I'm still seeing it. Won't have bandwidth to dig in personally until tomorrow.
seeing a similar failure after a git pull
I knew I wasn't crazy
environment: fedora 21, golang: 1.4, docker: 1.6
I think I see the problem. When I set the node name to in the test I was assuming it would parse as a valid ip address and not end up at the LookupIP call. But only the node name (and not the kubelet hostname) is being set (except in my new test where I ) so it's trying to do a lookup. Do you think you could test just changing the default hostname from "testnode" to "127.0.0.1" and see if that fixes it for you?
Trying now
I still get failures but no panic / threaddump.
gist coming
this looks like again?
Good that it isn't panicing any longer. I can dig into the test failures tomorrow (I think it should be relatively simple to resolve).
cool, thanks for looking
Passes in master now for me. Closing.
FYI, they will break again once goes in, and then we can merge my PR to see if that fixes it for you.
:+1:
is in -- are these tests breaking again?
I just rebased which I think will fix them.
They are breaking again.
is on it.


Question: function save-logs() { local -r node_name="${1}" local -r dir="${2}" <del> local files="${3} ${common_logfiles}" </del> <ins> local files="${3}" </ins> if [[ "${KUBERNETES_PROVIDER}" == "gce" || "${KUBERNETES_PROVIDER}" == "gke" ]]; then files="${files} ${gce_logfiles}" fi if [[ "${KUBERNETES_PROVIDER}" == "aws" ]]; then files="${files} ${aws_logfiles}" fi if ssh-to-node "${node_name}" "sudo systemctl status kubelet.service" &> /dev/null; then ssh-to-node "${node_name}" "sudo journalctl --output=cat -u kubelet.service" > "${dir}/kubelet.log" || true ssh-to-node "${node_name}" "sudo journalctl --output=cat -u docker.service" > "${dir}/docker.log" || true <ins> ssh-to-node "${node_name}" "sudo journalctl --output=cat -k" > "${dir}/kern.log" || true </ins> else <del> files="${files} ${initd_logfiles} ${supervisord_logfiles}" </del> <ins> files="${kern_logfile} ${files} ${initd_logfiles} ${supervisord_logfiles}" </ins> fi copy-logs-from-node "${node_name}" "${dir}" "${files}" }

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "LRUExpireCache" Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT What happened: requires write lock since needs to manipulate its list to track recently used item. Currently it uses read lock so it may introduce race condition. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else do we need to know:
This was fixed in


Question: return checkMirrorPodDisappear(ctx, f.ClientSet, pod.Name, pod.Namespace) }, f.Timeouts.PodDelete, f.Timeouts.Poll).Should(gomega.BeNil()) }) <del> // Regression test for an extended scenario for https://issues.k8s.io/123980 ginkgo.It("should evict running pods that do not meet the affinity after the kubelet restart", func(ctx context.Context) { nodeLabelKey := string(uuid.NewUUID()) nodeLabelValueRequired := string(uuid.NewUUID()) podName := "affinity-pod" + string(uuid.NewUUID()) nodeName := getNodeName(ctx, f) ginkgo.By(fmt.Sprintf("Adding node label for node (%s) to satisify pod (%s/%s) affinity", nodeName, f.Namespace.Name, podName)) e2enode.AddOrUpdateLabelOnNode(f.ClientSet, nodeName, nodeLabelKey, nodeLabelValueRequired) ginkgo.DeferCleanup(func() { e2enode.RemoveLabelOffNode(f.ClientSet, nodeName, nodeLabelKey) }) pod := e2epod.MustMixinRestrictedPodSecurity(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      podName, Namespace: f.Namespace.Name, }, Spec: v1.PodSpec{ Affinity: &v1.Affinity{ NodeAffinity: &v1.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &v1.NodeSelector{ NodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{ { Key:      nodeLabelKey, Operator: v1.NodeSelectorOpIn, Values:   []string{nodeLabelValueRequired}, }, }, }, }, }, }, }, Containers: []v1.Container{ { Name:  podName, Image: imageutils.GetPauseImageName(), }, }, }, }) // Create the pod bound to the node. It will start, but will be rejected after kubelet restart. ginkgo.By(fmt.Sprintf("Creating a pod (%s/%s)", f.Namespace.Name, podName)) e2epod.NewPodClient(f).Create(ctx, pod) ginkgo.By(fmt.Sprintf("Waiting for the pod (%s/%s) to be running", f.Namespace.Name, pod.Name)) err := e2epod.WaitForPodNameRunningInNamespace(ctx, f.ClientSet, pod.Name, f.Namespace.Name) framework.ExpectNoError(err, "Failed to await for the pod to be running: (%v/%v)", f.Namespace.Name, pod.Name) // Remove node label e2enode.RemoveLabelOffNode(f.ClientSet, nodeName, nodeLabelKey) ginkgo.By("Restart the kubelet") restartKubelet(true) gomega.Eventually(ctx, func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, recoverTimeout, f.Timeouts.Poll). Should(gomega.BeTrueBecause("kubelet should be healthy after restart")) // Pod should be terminated, maybe not immediately, should allow a few seconds for the kubelet to kill the pod // after kubelet restart, pod admission denied, kubelet will reject the pod and kill container. gomega.Eventually(ctx, func() bool { pod, err = e2epod.NewPodClient(f).Get(ctx, podName, metav1.GetOptions{}) framework.ExpectNoError(err) // pod is in a final state, the following are the behaviors of pods after kubelet restarted: // 1. kubelet `canAdmitPod` reject pod by reason Pod admission denied by nodeAffinity // 2. kubelet stop/kill container // the final state of the pod is related to the kill container. // if an error occurs in the preStop of the container or the exitCode is not 0, // the phase is PodFailed. if the exitCode of the StopContainer is 0, the phase is PodSucceeded. // in this case, stop and kill container is successful(exitCode 0), the pod phase should be PodSucceeded. return pod.Status.Phase == v1.PodSucceeded }, recoverTimeout, f.Timeouts.Poll).Should(gomega.BeTrueBecause("Pod %s not terminated: %s", pod.Name, pod.Status.Phase)) }) </del> }) })

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
<img width="1312" alt="image" src=""
This looks like a test you a few days ago: /assign Can you look into these failures?
GitHub didn't allow me to assign the following users: chengjoey. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This looks like a test you a few days ago: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
As mentioned in the previous comments, there should be only one expected state in the end. From the logs of these failed tests, after stopping the container, the phase changes to Failed. I think this does not meet expectations.
Thanks to for revert my previous PR. I think the pod final state should always be in one state. From the current e2e test results, the final state may be Succeed or Failed, seems pod.Status.Phase is empty. I think this seems to be wrong. I will study whether the node status (sufficient memory) and other factors interfere with the final state. Thanks again to for his help.


Question: return checkMirrorPodDisappear(ctx, f.ClientSet, pod.Name, pod.Namespace) }, f.Timeouts.PodDelete, f.Timeouts.Poll).Should(gomega.BeNil()) }) <del> // Regression test for an extended scenario for https://issues.k8s.io/123980 ginkgo.It("should evict running pods that do not meet the affinity after the kubelet restart", func(ctx context.Context) { nodeLabelKey := string(uuid.NewUUID()) nodeLabelValueRequired := string(uuid.NewUUID()) podName := "affinity-pod" + string(uuid.NewUUID()) nodeName := getNodeName(ctx, f) ginkgo.By(fmt.Sprintf("Adding node label for node (%s) to satisify pod (%s/%s) affinity", nodeName, f.Namespace.Name, podName)) e2enode.AddOrUpdateLabelOnNode(f.ClientSet, nodeName, nodeLabelKey, nodeLabelValueRequired) ginkgo.DeferCleanup(func() { e2enode.RemoveLabelOffNode(f.ClientSet, nodeName, nodeLabelKey) }) pod := e2epod.MustMixinRestrictedPodSecurity(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      podName, Namespace: f.Namespace.Name, }, Spec: v1.PodSpec{ Affinity: &v1.Affinity{ NodeAffinity: &v1.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &v1.NodeSelector{ NodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{ { Key:      nodeLabelKey, Operator: v1.NodeSelectorOpIn, Values:   []string{nodeLabelValueRequired}, }, }, }, }, }, }, }, Containers: []v1.Container{ { Name:  podName, Image: imageutils.GetPauseImageName(), }, }, }, }) // Create the pod bound to the node. It will start, but will be rejected after kubelet restart. ginkgo.By(fmt.Sprintf("Creating a pod (%s/%s)", f.Namespace.Name, podName)) e2epod.NewPodClient(f).Create(ctx, pod) ginkgo.By(fmt.Sprintf("Waiting for the pod (%s/%s) to be running", f.Namespace.Name, pod.Name)) err := e2epod.WaitForPodNameRunningInNamespace(ctx, f.ClientSet, pod.Name, f.Namespace.Name) framework.ExpectNoError(err, "Failed to await for the pod to be running: (%v/%v)", f.Namespace.Name, pod.Name) // Remove node label e2enode.RemoveLabelOffNode(f.ClientSet, nodeName, nodeLabelKey) ginkgo.By("Restart the kubelet") restartKubelet(true) gomega.Eventually(ctx, func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, recoverTimeout, f.Timeouts.Poll). Should(gomega.BeTrueBecause("kubelet should be healthy after restart")) // Pod should be terminated, maybe not immediately, should allow a few seconds for the kubelet to kill the pod // after kubelet restart, pod admission denied, kubelet will reject the pod and kill container. gomega.Eventually(ctx, func() bool { pod, err = e2epod.NewPodClient(f).Get(ctx, podName, metav1.GetOptions{}) framework.ExpectNoError(err) // pod is in a final state, the following are the behaviors of pods after kubelet restarted: // 1. kubelet `canAdmitPod` reject pod by reason Pod admission denied by nodeAffinity // 2. kubelet stop/kill container // the final state of the pod is related to the kill container. // if an error occurs in the preStop of the container or the exitCode is not 0, // the phase is PodFailed. if the exitCode of the StopContainer is 0, the phase is PodSucceeded. // in this case, stop and kill container is successful(exitCode 0), the pod phase should be PodSucceeded. return pod.Status.Phase == v1.PodSucceeded }, recoverTimeout, f.Timeouts.Poll).Should(gomega.BeTrueBecause("Pod %s not terminated: %s", pod.Name, pod.Status.Phase)) }) </del> }) })

Add the log with phase "Succeed": The normal update status steps are 1-2-3-4, but the result of failure tests is that step 2 is executed last, which seems a bit strange. Because phase is empty
feel free to reopen the PR. But let's make sure to run the tests that this reported as failing before we merge. Thank you for your help on improving test coverage!
The links to the failed E2E tests in the overview of this issue all show the Pod phase as , but is there a link available somewhere for cases where it's empty string?
In fact, these failed tests are all empty strings, because not terminated: should be followed by phase Because my original test was written like this as I commented earlier, different order of updating status may lead to different final state
The Pod's final state is shown as in the dumped result immediately after the test fails. The referenced inside - is that coming from the Pod information fetched within ? Could it be that it's showing up as an empty string because it's not able to access the value externally?
In the tests submitted previously, and in my local(minnikube) tests, the pod status will change from to again see this log step 3 So I think the order of state changes affects the final result, and this is not very stable
I think this is a separate issue, so as I mentioned in , I thought it might be fine to wait for either the or phase. However, after looking at , it seems that we may need to resolve this unstable final result first. WDYT?
If we can explain different results, it is fine. I don't want unexplained check for either condition in our tests


Question: }, &api.Pod{ ObjectMeta: api.ObjectMeta{ <del> UID:       "222", Name:      "bar" + "-" + hostname, Namespace: "default", SelfLink: getSelfLink("bar-"+hostname, kubetypes.NamespaceDefault), </del> <ins> UID:         "222", Name:        "bar" + "-" + hostname, Namespace:   "default", Annotations: map[string]string{kubetypes.ConfigHashAnnotationKey: "222"}, SelfLink:    getSelfLink("bar-"+hostname, kubetypes.NamespaceDefault), </ins> }, Spec: api.PodSpec{ NodeName:                      hostname,

See for more context. Currently, kubelet compares the spec of a mirror pod (created via apiserver) with that of a static pod (config stored as a file) to decide whether the two are considered identical. If they differ, kubelet would delete the mirror pod and re-create it with the correct spec. Due to the fact that apiserver may set default values for some fields in the spec of the mirror pod, the specs could deviate undesirably. To avoid problems like this, we can store the config of a static pod as an annotation in the mirror pod, and compare the static pod with the annotation.
Would a hash of the config be sufficient if we're only going to compare? The whole spec would generally be more useful, but I wonder if that is too much data to put in an annotation.
This seems totally reasonable to me, but I'm still desperately curious why you can't just bypass defaulting when you generate mirror pod from static pod? (i.e. just use the static pod's configuration verbatim)
It's not kubelet who applies the default values here. The admission plugin was only run at the apiserver, and to bypass that, you have to ask apiserver to treat mirror pods differently and potentially at multiple places.
On May 22, 2015 10:38 AM, "Yu-Ju Hong" wrote: Understood, but is it really multiple places? If so then I agree it's a bad idea. Otherwise it might be worth considering as an alternative.
It's untenable for Kubelet to assume to know/circumvent all fields that will be set by other entities: admission control, defaulting, late initialization, auto-something, whatever. It needs to obey the same rules as other clients.
I don't understand why you would ever want to apply any of the things you mentioned to static pods. Static pods should be copied verbatim into the mirror pods, right? That's what I'm proposing.
What if the master is updated before the kubelet, and the master adds a new API field with a default value?
That case should be discussed in
I agree that mirror pod should reflect the true status of the static pod running on the node, and no default values should be set in this case.


Question: }, &api.Pod{ ObjectMeta: api.ObjectMeta{ <del> UID:       "222", Name:      "bar" + "-" + hostname, Namespace: "default", SelfLink: getSelfLink("bar-"+hostname, kubetypes.NamespaceDefault), </del> <ins> UID:         "222", Name:        "bar" + "-" + hostname, Namespace:   "default", Annotations: map[string]string{kubetypes.ConfigHashAnnotationKey: "222"}, SelfLink:    getSelfLink("bar-"+hostname, kubetypes.NamespaceDefault), </ins> }, Spec: api.PodSpec{ NodeName:                      hostname,

However, I think it'd be harder to implement this way since, as stated above, that there are many things to circumvent in the apiserver itself. It could also be harder to keep this special case from being broken by a yet another new feature/plugin. Keeping the hash of the config in the annotation seems like a good compromise even though it may not reflect the true status 100% accurately. I think this is acceptable. What do you think?
Sure, if you think the annotation approach is easier, I would go with that. I guess in the case Brian mentoined (master adds a new API field with a default value), you would not update the mirror pod, because the static pod definition has not changed?
Yes, a hash would be sufficient.
It shouldn't be necessary to update the mirror pod in the case that a new default field was updated, because that field should be populated automatically by the apiserver on decode from etcd.
We should fix this so we don't have this problem again in 1.2.


Question: apiVersion: v1 metadata: name: etcd-discovery <del> creationTimestamp: </del> labels: name: etcd-discovery spec:

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Vagrant OS (e.g. from /etc/os-release): CoreOS Kernel (e.g. ):Install tools: Vagrant & Others: What happened: What you expected to happen: Success How to reproduce it (as minimally and precisely as possible): Followed instructions at Tried to run Received: Anything else do we need to know: From googling, it seems several of the fields in the config are for when using OpenShift origin. But creating the replication controller is a prerequisite to launching OpenShift Origin. Chicken & egg problem.


Question: } ginkgo.By(fmt.Sprintf("Create node foo by user: %v", asUser)) _, err := c.CoreV1().Nodes().Create(context.TODO(), node, metav1.CreateOptions{}) <ins> // NOTE: If the test fails and a new node IS created, we need to delete it. If we don't, we'd have // a zombie node in a NotReady state which will delay further tests since we're waiting for all // tests to be in the Ready state. defer func() { f.ClientSet.CoreV1().Nodes().Delete(context.TODO(), node.Name, metav1.DeleteOptions{}) }() </ins> framework.ExpectEqual(apierrors.IsForbidden(err), true) })

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: [sig-auth] [Feature:NodeAuthorizer] A node shouldn't be able to create another node Since when has it been failing: N/A Testgrid link: N/A Reason for failure: The test creates a new ` node, but it does not delete it after the test ends: kubectl get nodes NAME STATUS ROLES AGE VERSION 1954k8s000 Ready agent 24d v1.14.1 1954k8s001 Ready agent 24d v1.14.1 foo NotReady <none4m3s k8s-master--0 Ready master 24d v1.14.1 Anything else we need to know: /sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .


Question: } ginkgo.By(fmt.Sprintf("Create node foo by user: %v", asUser)) _, err := c.CoreV1().Nodes().Create(context.TODO(), node, metav1.CreateOptions{}) <ins> // NOTE: If the test fails and a new node IS created, we need to delete it. If we don't, we'd have // a zombie node in a NotReady state which will delay further tests since we're waiting for all // tests to be in the Ready state. defer func() { f.ClientSet.CoreV1().Nodes().Delete(context.TODO(), node.Name, metav1.DeleteOptions{}) }() </ins> framework.ExpectEqual(apierrors.IsForbidden(err), true) })

Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err != nil { return nil, err } <ins> errors := ds.performPlatformSpecificContainerForContainer(r.ContainerId) if len(errors) != 0 { return nil, fmt.Errorf("failed to run platform-specific clean ups for container %q: %v", r.ContainerId, errors) } </ins> err = ds.client.RemoveContainer(r.ContainerId, dockertypes.ContainerRemoveOptions{RemoveVolumes: true, Force: true}) if err != nil { return nil, fmt.Errorf("failed to remove container %q: %v", r.ContainerId, err) } return &runtimeapi.RemoveContainerResponse{}, nil }

What would you like to be : Periodically remove cleanupInfos for old containers from dockershim's map. One way of doing that would be to just forget about s older than say an hour, as they should always be short-lived. Why is this needed: introduced a new in-memory map for the dockershim, that could potentially (in pathological cases) cause memory leaks - for containers that use GMSA cred specs, get created successfully, but then never get started nor removed. This might be okay for the alpha release of the GMSA feature, but should be addressed in the beta. See
/assign wk8
/sig windows
/milestone v1.15
got moved out. let's move this issue with it as well /milestone v1.16
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
: yup we'd love to have this in 1.16, still waiting for kind reviewers on :) ...


Question: bash -c "${common_commands}" diff -ruN "${output_dir}/upstream" "${output_dir}/patch" >"${diff_file}" || true <ins> if [[ -t 1 ]]; then  # Attached to a terminal? less "${diff_file}" fi </ins> echo echo "  *** Diff saved in ${diff_file} ***"

and show that we run this job every few minutes, but it never passes. I'm going to send out a PR to disable this job until someone has the time to investigate and resolve.
It's failing before tests even start:
At this point I don't think it's a flake, we needed flannel for a release and it's less important now so I'm not investigating why the cluster isn't coming up soon. I do believe if the cluster comes up a majority if not all of the tests will pass, and the ones that don't will flake for known reasons.


Question: "type": "string" }, "ports": { <del> "description": "List of ports to expose from the container. Exposing a port here gives the system additional information about the network connections a container uses, but is primarily informational. Not specifying a port here DOES NOT prevent that port from being exposed. Any port which is listening on the default "0.0.0.0" address inside a container will be accessible from the network. Cannot be updated.", </del> <ins> "description": "List of ports to expose from the container. Not specifying a port here DOES NOT prevent that port from being exposed. Any port which is listening on the default "0.0.0.0" address inside a container will be accessible from the network. Modifying this array with strategic merge patch may corrupt the data. For more information See https://github.com/kubernetes/kubernetes/issues/108255. Cannot be updated.", </ins> "items": { "allOf": [ {

The comments in says that : However there are several cases where this is false : When using a Service with a targetPort that references a name in the ports arrays of the pod. Some software (ab)uses this API to get which ports are used (See Although unfixable, my goal is to fix the documentation so other people don't have the same problem). The Azure Load Balancer implementation does this for instance. Some other network/security software might use this information. Also, is it possible to document/reference the above issue in the comment ? Since it is described as informational I expected for the network to work without this information. But it's definitely not the case. Not needed since it's mostly a doc issue. I'm interested in contributing this patch but would like some feedbacks before doing it. :) <details</details<details</details<details</details<details</details<details</details<details</details
/sig network
/triage accepted -- thoughts? I believe the doc are due for a refresh.
I'd take this one. /assign
I didn't see that you wanted to do this patch, I am very sorry and now I unassign it. /unassign
No worries But I'm still not sure where / what to add. Should I only fix the part saying it is only informational ? Is there anything else ? (cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: if err != nil { t.Fatalf("unexpected error: %v", err) } <ins> } func TestDeleteNamespaceWithIncompleteFinalizers(t *testing.T) { now := util.Now() fakeEtcdClient, helper := newHelper(t) fakeEtcdClient.ChangeIndex = 1 fakeEtcdClient.Data["/registry/namespaces/foo"] = tools.EtcdResponseWithError{ R: &etcd.Response{ Node: &etcd.Node{ Value: runtime.EncodeOrDie(latest.Codec, &api.Namespace{ ObjectMeta: api.ObjectMeta{ Name:              "foo", DeletionTimestamp: &now, }, Spec: api.NamespaceSpec{ Finalizers: []api.FinalizerName{api.FinalizerKubernetes}, }, Status: api.NamespaceStatus{Phase: api.NamespaceActive}, }), ModifiedIndex: 1, CreatedIndex:  1, }, }, } storage, _, _ := NewStorage(helper) _, err := storage.Delete(api.NewDefaultContext(), "foo", nil) if err == nil { t.Fatalf("expected error: %v", err) } } </ins> <del> // TODO: when we add life-cycle, this will go to Terminating, and then we need to test Terminating to gone </del> <ins> func TestDeleteNamespaceWithCompleteFinalizers(t *testing.T) { now := util.Now() fakeEtcdClient, helper := newHelper(t) fakeEtcdClient.ChangeIndex = 1 fakeEtcdClient.Data["/registry/namespaces/foo"] = tools.EtcdResponseWithError{ R: &etcd.Response{ Node: &etcd.Node{ Value: runtime.EncodeOrDie(latest.Codec, &api.Namespace{ ObjectMeta: api.ObjectMeta{ Name:              "foo", DeletionTimestamp: &now, }, Spec: api.NamespaceSpec{ Finalizers: []api.FinalizerName{}, }, Status: api.NamespaceStatus{Phase: api.NamespaceActive}, }), ModifiedIndex: 1, CreatedIndex:  1, }, }, } storage, _, _ := NewStorage(helper) _, err := storage.Delete(api.NewDefaultContext(), "foo", nil) if err != nil { t.Fatalf("unexpected error: %v", err) } </ins> }

There is a funny subtle behvaiour in get namespaces, in which if i call more than one time, it seems that the removal from Terminating state is accelarated. It seems this might indicate a bug, or maybe its not a bug and the eventual semantics of is a little funny sometimes. in any case, here's what we are seeing : 1) when i delete a namespace the first time usning kubectl, we get a state change to . that is correct, of course. 2) but ... if make the call a second time, then the same status line is immediately removed from . Expected behaviour : A second call to "kubectl delete ..." shouldn't have any effect on the timing of removal of a namespace from 'Terminating' state*... So i guess there is some "eventual consistency" subtle bug that is resolved by repeated calls to
suggests this may not be e bug, maybe its just the etcd eventual deleter thingy.
Looks to definitely be a bug. The 2nd delete seems to have interrupted the delete process and artifacts (like events) from the namespace still exist in etcd.
investigating...
oops, see the error, will submit a fix.
I have a fix, but need a test case, will submit PR tomorrow AM


Question: "kind": "Service", "apiVersion": "v1beta3", "metadata": { <del> "name": "my-service", </del> <ins> "name": "my-service" </ins> }, "spec": { "selector": {

The "service" file in this document contains a trailing comma on the line ' "name": "my-service", ` and thus kubectl fails to parse it properly.


Question: // during "reserve" extension point or later. func (mc CommunicatingPlugin) Unreserve(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeName string) { if pod.Name == "my-test-pod" { <del> state.Lock() </del> // The pod is at the end of its lifecycle -- let's clean up the allocated // resources. In this case, our clean up is simply deleting the key written // in the Reserve operation. state.Delete(framework.StateKey(pod.Name)) <del> state.Unlock() </del> } }

What happened: kube-scheduler () crashes due to a panic while running : The race condition is triggered when an out-of-tree plugin, that writes to during its Filter phase, is registered through . Access to in general happens from a single goroutine and there are no practical concurrency issues, with the exception of the phase, that parallelizes calls for each node across plugins: A quick glance at the tip of the branch seems to indicate the bug is still present there. What you expected to happen: Ideally access to should be synchronized everywhere for correctness, including by all in-tree plugins, since there are no guarantees about how will call multiple plugins concurrently. It seems to me it is something that was considered at some point, given that has methods. Sadly, almost no in-tree plugins are synchronizing access. For now, I will try to work around the problem by doing a single write to during and storing a data bag structure where I will write state into during (with proper synchronization), instead of writing to directly. How to reproduce it (as minimally and precisely as possible): Install a scheduler plugin that writes to during its phase. In-tree (builtin) plugins are not synchronizing their access to . Environment: Kubernetes version (use ): v1.18.8 Cloud provider or hardware configuration: linux-amd64 (AWS) OS (e.g: ): Ubuntu 18.04.5 LTS (Bionic Beaver) Others: custom kube-scheduler binary built with an out-of-tree plugin using
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
AFAICT all in-tree plugins write to the cycle state in PreFilter and then only read from cycle state in Filter. So this shouldn't be a problem with in-tree plugins as concurrent read access should be fine during RunFilterPlugins. But since your out-of-tree plugin writes in Filter, I can see why this can be a problem.


Question: // during "reserve" extension point or later. func (mc CommunicatingPlugin) Unreserve(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeName string) { if pod.Name == "my-test-pod" { <del> state.Lock() </del> // The pod is at the end of its lifecycle -- let's clean up the allocated // resources. In this case, our clean up is simply deleting the key written // in the Reserve operation. state.Delete(framework.StateKey(pod.Name)) <del> state.Unlock() </del> } }

Aside from adding Lock/Unlock in in-tree plugin, offering mutual exclusion on each key as a built-in feature within CycleState could be an alternative solution. This way, reads from in-tree plugins during Filter wouldn't interfere with writes during Filter from out-of-tree plugins.
cc
should we change the Read/Write CycleState methods to be thread safe?
Adding locks without a proper justification would just increase the latency. could you clarify your use case? Why do you need to write anything during Filter?
yeah, we should measure it, but I don't expect it to have significant effect...
Adding locks without a proper justification would just increase the latency. Without contention (which is the case with in-tree plugins), I don't think this is a concern.
Adapting source from : source: essentially, 11.3 ns/Lock-Unlock pair
I remember locks showing up in profiling for PreScore/Score for certain plugins. Although I have to admit that the accesses were not properly implemented. I'd like to hear an answer for before doing any experiments.
+1 to that I'd also like to understand the motivations / requirements of mutating CycleState in Filter().
My particular use-case is that we have resources being dynamically to/removed from Nodes, and those resources need to be allocated (and tracked) in the scheduling loop. Technically I could pre-compute all the available dynamic resources for all Nodes at the phase, and snapshot it, but that would require me to pre-compute everything for all Nodes on all scheduling cycles. Instead, what I'm currently doing is computing what dynamic resources can be utilized on-demand during the phase (kind of a lazy loading of dynamic resources for each pair), and saving state of what extra resources were considered during for each Node that was evaluated. Then, later in the scheduler plugin can lock-in the exact dynamic resources for the particular Node that was selected. I understand the concern with adding extra synchronization overhead without a real need. Even if my use-case is not good enough to justify it, there is another argument for it: technically, the lack of access synchronization on violates the . Even if in the end we decide synchronizing all access is not worth the overhead (for correctness), I would strongly encourage better documentation saying which plugin phases are allowed to modify , and perhaps even forcing a snapshot-read-only copy of to be used by plugins.
Using the scheduler to track resources is not a good idea.


Question: // during "reserve" extension point or later. func (mc CommunicatingPlugin) Unreserve(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeName string) { if pod.Name == "my-test-pod" { <del> state.Lock() </del> // The pod is at the end of its lifecycle -- let's clean up the allocated // resources. In this case, our clean up is simply deleting the key written // in the Reserve operation. state.Delete(framework.StateKey(pod.Name)) <del> state.Unlock() </del> } }

The scheduler should be stateless. Can't you track the resources with an external controller or with a device plugin and making it part of the Node spec?
there may be some misunderstanding in my poor choice of words. What I mean by "tracking" is that the scheduler needs to track usage and availability of these extra per-Node resources, to be able to allocate them during placement/scheduling. Exactly the same it does with volumes, cpus, memory, ephemeral disk, etc Doing computations in a controller outside of the scheduling loop would be similar to "precomputing" these resources. In my particular use case I chose a different way because: introduces another point of failure, if the external controller stops pre-computing extra resources and publishing them somewhere (CRD, Node spec, etc) scheduling halts (at least scheduling on these extra resources) Node resources in my use case are volatile (they are coming and going frequently), and the extra latency by an extra controller (vs doing all computations right in the scheduling loop) may be problematic for my use case. Even if you consider my use case not enough to add proper synchronization to , I'd still encourage making it explicit what scheduling phases are allowed to modify it (and even passing in a snapshot read-only version of to phases that are not expected to update it), to avoid confusion and race conditions.
Ok, if you are not tracking the existence of resources in the scheduler, I wouldn't suggest the external controller. Still, have you considered the device plugin framework? That way, the default kube-scheduler could keep track of resources usage. Otherwise, I still don't see why you would need to update the CycleState during Filter. As its name indicates, CycleState only lives during a scheduling cycle.
Regardless of the correctness of usage of CycleState to do what wants, I still think we need to do at least one of the following: that writing to CycleState during Filter will lead to data races, a read-only CycleState to Filter plugins (whether we do this at the interface level or if we simply drop Write requests during Filter is an implementation detail), or make CycleState concurrency safe CycleState as it exists right now allows plugin authors to shoot themselves in the foot IMO.
I'm leaning towards 1, but it's fair to continue the discussion before deciding.
Can you run the "controller" as a goroutine along with your scheduler plugin?


Question: // during "reserve" extension point or later. func (mc CommunicatingPlugin) Unreserve(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeName string) { if pod.Name == "my-test-pod" { <del> state.Lock() </del> // The pod is at the end of its lifecycle -- let's clean up the allocated // resources. In this case, our clean up is simply deleting the key written // in the Reserve operation. state.Delete(framework.StateKey(pod.Name)) <del> state.Unlock() </del> } }

So it's not an external controller binary, and hence don't introduce another point of failure. It's pretty like how the default scheduler manages its internal cache. But note that cache is only needed when you need a more efficient data structure. If a {key:value} map is good enough, spinning up a XYZInformer and obtain the latest XYZ resource on-demand is the most recommended way. I think it quite depends on how you design the internal XYZ cache. For example, it should only tweak the cache in an incremental manner - in other words, unchanged resource will be kept unmodified. You can check out the "doubly-pointer snapshot" of scheduler's cache. With above said, if I understand your case correctly, you may need to build a sophisticated XYZ cache, which runs in a separate goroutine and responds to your XYZ resource changes. And then upon each scheduling cycle, take a snapshot (may needs a specific snapshotting algorithm) in PreFilter() and WRITE to CycleState, and then in Filter() READ from CycleState. My personal option is to suggest all plugin developers to read from CycleState in Filter().
nailed it. What you describe is almost exactly what I’m doing today. There are some details particular to my use case (e.g. just tracking current state with an informer and an incremental cache is not enough, because there’s some logic involved in deciding if currently available resources are usable by each pair, and I currently do that computation in as pairs are evaluated). In any case, I just wanted to share my use case to provide some color, but I don’t want what I’m doing distracting us from the more important questions, mentioned by above. My personal preference would be as a very short term next step, then when someone has cycles to do it. I would be OK with punting on until a more strong need shows up.
Note that limiting to only reading from works for me, as long as I can still read my own data structure from it and do writes on it (i.e. don’t deepclone for the phase).
also adds overhead (from copying the state, which can be very expensive).
Could implement just by changing the interface passed to (e.g. ). That wouldn’t be backwards compatible, but very cheap.


Question: // during "reserve" extension point or later. func (mc CommunicatingPlugin) Unreserve(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeName string) { if pod.Name == "my-test-pod" { <del> state.Lock() </del> // The pod is at the end of its lifecycle -- let's clean up the allocated // resources. In this case, our clean up is simply deleting the key written // in the Reserve operation. state.Delete(framework.StateKey(pod.Name)) <del> state.Unlock() </del> } }

I guess that would work in the sense that you cannot change the high level values of the CycleState map, but you could still override what's inside them, if they have pointers. But maybe that's ok.
but you could still override what's inside them, if they have pointers. But maybe that's ok. yeah, if it was pointers and if the underlying data needed to be modified, I'd imagine it would have its own Lock/Unlock semantics
for such implementation, the only remaining concern is backwards compatibility. So I would leave that decision to who leads the scheduling-plugins repo.
I rethink 's original problem a little bit: the original racing issue occurs when reading CycleState in in-tree plugins, this indicates one of the (out-of-tree) plugins is (over)writing the most high-level value of the CycleState map. But if we control the locking inside the out-of-tree plugin, and mutate that particular in-place, that would be good enough? Case 3 in the above program demonstrates the correct way to manipulate the specific out-of-tree state. Let me know your thoughts. If it resolves 's concern, we don't need to make any change on the Filter() interface.
Case 3 is how I am working around the problem today to avoid the race. I write a "data bag" once into top level in , then only modify my own inner structure (with proper locking) in , without doing any further modifications to directly (top level) in . That is acceptable, but I'd encourage you to consider making the interface (and documentation) more explicit for plugins, so they avoid the mistake of writing directly into . Without changing anything (no documentation, no readonly interface), it is pretty easy for a plugin to use in a way that you don't want to support.
Thanks for confirming that. Yes, documenting it, or(and) publishing examples would be quite helpful to clarify the semantics. As a wrap-up: a keyed value can be read from CycleState, literally , in any phase. It's always race-free.


Question: // during "reserve" extension point or later. func (mc CommunicatingPlugin) Unreserve(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeName string) { if pod.Name == "my-test-pod" { <del> state.Lock() </del> // The pod is at the end of its lifecycle -- let's clean up the allocated // resources. In this case, our clean up is simply deleting the key written // in the Reserve operation. state.Delete(framework.StateKey(pod.Name)) <del> state.Unlock() </del> } }

if you want to mutate the top-level keyed value in CycleState, if it's a phase that CycleState is accessed consecutively, such as PreFilter, PreScore, it's safe to mutate CycleState in any manner if it's a phase that CycleState is accessed parallelly, to be race-free, (for now) you only have 1 option: keep the keyed value's pointer unchanged, and use an internal lock to mutate that pointer obj [TBD] we may come up with changes to add read locks to all in-tree plugins when reading values from CycleState, so that the out-of-tree plugin can surround its mutating logic between and - th mutating logic thereby can be any style Any thoughts on the TBD item? I would assume read locks wouldn't cost too much.
For now, I think the documentation is correct, it clearly says Read() and Write() are not thread-safe: The key thing here is whether in-tree plugin should use RLock/RUnlock or not. If yes, out-of-tree plugins can be freed to use Lock/Unlock. (see the TBD item in the previous comment)
I like the TBD, and optimizing for correctness in how in-tree plugins are accessing . I understand the concern with synchronization overhead, but I would be really surprised if it is the bottleneck in scheduling code.
It can be, when you have 5k-nodes clusters, or bigger.
I don't think it will be expensive, a rlock should be equivalent to an atomic add operation (basically it increments the number of readers). As a comparison, looking up the state from the map is likely to be more expensive.
Of course we are assuming no contention in the default plugins.
This is my impression as well. A vanilla version of scheduler doesn't impose write lock in Filter() and Score(), so should be good.
We can always see with the benchmarks and revert if its not negligible.
I will run the benchmarks to see if there is perf downgrade. /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale This is still relevant, yes?


Question: // during "reserve" extension point or later. func (mc CommunicatingPlugin) Unreserve(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeName string) { if pod.Name == "my-test-pod" { <del> state.Lock() </del> // The pod is at the end of its lifecycle -- let's clean up the allocated // resources. In this case, our clean up is simply deleting the key written // in the Reserve operation. state.Delete(framework.StateKey(pod.Name)) <del> state.Unlock() </del> } }

IIUC the volumebinding plugin has been fixed, but external plugins can still shoot themselves in the foot.
/priority backlog
Yes. I'm working on this, late this week I will send out a PR along with perf test result.


Question: ### Negative caching <del> The `denial` cache TTL has been reduced to the minimum of 5 seconds [here](https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml#L37). In the unlikely event that this impacts performance, setting this TTL to a higher value make help alleviate issues, but be aware that operations that rely on DNS polling for orchestration may fail (for example operators with StatefulSets). </del> <ins> The `denial` cache TTL has been reduced to the minimum of 5 seconds [here](https://github.com/kubernetes/kubernetes/blob/a38ed2c5ceedf682cbc19442aac5224ae6e10eaa/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml#L61). In the unlikely event that this impacts performance, setting this TTL to a higher value make help alleviate issues, but be aware that operations that rely on DNS polling for orchestration may fail (for example operators with StatefulSets). </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --This is probably just a documentation / example issue. I'm a real noob so I can just tell this is an issue, but I can't fix the docs through a PR, otherwise I'd do it happily. Hope it's not a dupe. Following the guide at with IPVS. Using the following script: This generates the following YAML (only the interesting part): You see that the line has an extra comma. It's instead of (extra comma after second argument) This generates the following log: Removing the extra comma fixes everything. Setup a cluster with Kube-DNS IPVS mode Generate the nodelocaldns Deploy the manifest Pods are crashing Kubernetes version v1.21.0 Cloud provider or hardware configuration: Bare Metal with HA setup (behind an external LB, Kube-proxy as IPVS) OS (e.g: ): CentOS Linux 8 Kernel (e.g. ): Linux 4.18.0-240.10.1.el83.x8664 SMP Mon Jan 18 17:05:51 UTC 2021 x8664 x8664 x86_64 GNU/LinuxInstall tools: none, a classic kubeadm initNetwork plugin and version (if this is a network-related bug): Not relevant, but Canal (Calico + Flannel) Others: nothing. Love you all for making K8S. Thank you a billion times.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig docs
, it seems like that don't get a value return. i saw command in Can you try this again and check if can kubedns get a value?
/area dns /sig network /assign Documentation here is not changed since Sep 2019 and needs to be updated as well.


Question: ### Negative caching <del> The `denial` cache TTL has been reduced to the minimum of 5 seconds [here](https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml#L37). In the unlikely event that this impacts performance, setting this TTL to a higher value make help alleviate issues, but be aware that operations that rely on DNS polling for orchestration may fail (for example operators with StatefulSets). </del> <ins> The `denial` cache TTL has been reduced to the minimum of 5 seconds [here](https://github.com/kubernetes/kubernetes/blob/a38ed2c5ceedf682cbc19442aac5224ae6e10eaa/cluster/addons/dns/nodelocaldns/nodelocaldns.yaml#L61). In the unlikely event that this impacts performance, setting this TTL to a higher value make help alleviate issues, but be aware that operations that rely on DNS polling for orchestration may fail (for example operators with StatefulSets). </ins>

Dual-stack and ipv6 are used more widely now.
I do get a value, but is replaced with as the extra comma doesn't seem to be replaced.
/assign


Question: }) } <ins> // waitForRCPodToDisappear returns nil if the pod from the given replication controller (described by rcName) no longer exists. // In case of failure or too long waiting time, an error is returned. func waitForRCPodToDisappear(c *client.Client, ns, rcName, podName string) error { label := labels.SelectorFromSet(labels.Set(map[string]string{"name": rcName})) return waitForPodToDisappear(c, ns, podName, label, 20*time.Second, 5*time.Minute) } </ins> // waitForService waits until the service appears (exist == true), or disappears (exist == false) func waitForService(c *client.Client, namespace, name string, exist bool, interval, timeout time.Duration) error { err := wait.Poll(interval, timeout, func() (bool, error) {

More detail can be seen in e2e test
cc
, Marek Grabowski wrote:
you are running an older version of the test. can you get the latest and try again ? This test had not failed at all yesterday in our automated test runs yesterday. Closing issue for now.
This failure depends on if the pod is really deleted and kube-proxy (the test container on) catch that the endpoint is deleted in 5s . After sending "delete pod" request for 5s, if the pod was still there and kube-proxy still contains this endpoints(this may happen under pressure), hit test can still detect 5 endpoints. At least we should ensure the pod was deleted. Anyway, I think will be helpful to this test. What do you think?
It doesn't matter if the pod is actually deleted or not. We just care that kube-proxy thinks that it should not be part of the service endpoint. Having said that, it still doesn't hurt to do the check that you are doing. I will assign your PR to myself and review it. In the meantime, can you rebase your PR ?
Thanks for your review. The PR is up to date.
Failed twice today in kubernetes-e2e-gce-parallel
One more today. The error message is not very helpful.
18:28:37 STEP: dialing(http) endpointPodIP:endpointHttpPort from node1 18:28:37 STEP: Dialing from node. command:for i in ' wc -l This is the point where the test fails. Am looking into it. We need to have more tries, we just do seq (1 1), instead of seq (6 6). this is causing flakiness.
Fixed by PR


Question: "name":        "The name of this port.  This must match the 'name' field in the corresponding ServicePort. Must be a DNS_LABEL. Optional only if one port is defined.", "port":        "The port number of the endpoint.", "protocol":    "The IP protocol for this port. Must be UDP, TCP, or SCTP. Default is TCP.", <del> "appProtocol": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and http://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", </del> <ins> "appProtocol": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and https://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", </ins> } func (EndpointPort) SwaggerDoc() map[string]string {

We should make the change in PR API reference links to should use HTTPS, not HTTP Visit h and look at the IANA hyperlinks Once this fix is made, a PR to is needed to update the docs (by autogenerating them). This typically would be deferred until the next Kubernetes release. v1.23 not applicable not applicable not applicable not applicable not applicable
/sig docs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I opened this issue so that someone else can open a new PR that is equivalent to .
/retitle Update links to IANA for Service Name and Transport Protocol Port Number Registry
I would like to work on this issue. May I assign it to myself?
Anyone - including you - is welcome to work on this issue! So long as you've you can open a pull request to propose a fix.
Thanks
/assign
I have made the PR. I hope this works. If not then will be happy to do the necessary changes.
Had to close the previous PR because of issues, here is the new one I made.


Question: }, []string{"operation", "type"}, ) <ins> etcdObjectCounts = compbasemetrics.NewGaugeVec( &compbasemetrics.GaugeOpts{ Name:              "etcd_object_counts", DeprecatedVersion: "1.21.0", Help:              "Number of stored objects at the time of last check split by kind. This metric is replaced by apiserver_storage_object_counts.", StabilityLevel:    compbasemetrics.ALPHA, }, []string{"resource"}, ) </ins> objectCounts = compbasemetrics.NewGaugeVec( &compbasemetrics.GaugeOpts{ <del> Name:           "etcd_object_counts", </del> <ins> Name:           "apiserver_storage_object_counts", </ins> Help:           "Number of stored objects at the time of last check split by kind.", <del> StabilityLevel: compbasemetrics.ALPHA, </del> <ins> StabilityLevel: compbasemetrics.STABLE, </ins> }, []string{"resource"}, )

What would you like to be : We want to start migrating metrics which people depend on for operations to stable status. etcdobjectcounts was mentioned in a recent SIG meeting as a viable candidate. However, the 'etcd' prefix is not in line with stated metric conventions. SIG consensus determined that creating a new version of the metric with the metric name 'apiserverstorageobjectcounts' would be sufficient allow this metric to be promoted. We'd want to simultaneously deprecate 'etcdobjectcounts' (since it would be a duplicate) and remove it after the deprecation grace period window. /assign /cc
GitHub didn't allow me to assign the following users: erain. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : What would you like to be : /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig instrumentation api-machinery /cc
Ack. I am going to work on this.
/assign /triage accepted
In the SIG api-machinery meeting today, we agreed that the stable version of this metric should be named 'apiserverstorageobject_count'
Thanks and will use in the PR. (And sorry for missing today's meeting - due to a conflict)


Question: Status:     status, } } <ins> func TestPrioritiesRegistered(t *testing.T) { var functionNames []string // Files and directories which priorities may be referenced targetFiles := []string{ "./../../algorithmprovider/defaults/defaults.go", // Default algorithm "./../../factory/plugins.go",                     // Registered in init() } // List all golang source files under ./priorities/, excluding test files and sub-directories. files, err := codeinspector.GetSourceCodeFiles(".") if err != nil { t.Errorf("unexpected error: %v when listing files in current directory", err) } // Get all public priorities in files. for _, filePath := range files { functions, err := codeinspector.GetPublicFunctions(filePath) if err == nil { functionNames = append(functionNames, functions...) } else { t.Errorf("unexpected error when parsing %s", filePath) } } // Check if all public priorities are referenced in target files. for _, functionName := range functionNames { args := []string{"-rl", functionName} args = append(args, targetFiles...) err := exec.Command("grep", args...).Run() if err != nil { switch err.Error() { case "exit status 2": t.Errorf("unexpected error when checking %s", functionName) case "exit status 1": t.Errorf("priority %s is implemented as public but seems not registered or used in any other place", functionName) } } } } </ins>

I just noticed you forgot to register the image locality priority function in in I also missed it when I reviewed your PR. I have fixed this in . But it would be great if you could figure out a mechanism that would ensure that all predicate/priority functions that are tested (you can assume they will always live under ) are registered, and fail the test if not. That will allow us to prevent this mistake in the future.
Thanks! Will fix this soon!
Seems need to wait for in before I send PR? Otherwise this test will always fail ...