Question:
we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?

<ins> Shell2Junit License Information Feb, 2010 shell2junit library and sample code is licensed under Apache License, v.2.0. (c) 2009 Manolo Carrasco (Manuel Carrasco Moñino) ===== Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: a. You must give any other recipients of the Work or Derivative Works a copy of this License; and b. You must cause any modified files to carry prominent notices stating that You changed the files; and c. You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and d. If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. </ins>


Question:
we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?

<ins> The MIT License (MIT) Copyright (c) 2015-2016 Manfred Touron Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. </ins>


Question:
we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?

<ins> Copyright (c) 2013 Joshua Tacoma. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. </ins>


Question:
we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?

<ins> Copyright (c) 2012 The Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. </ins>


Question:
we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?

<ins> This project is covered by two different licenses: MIT and Apache. #### MIT License #### The following files were ported to Go from C files of libyaml, and thus are still covered by their original MIT license, with the additional copyright staring in 2011 when the project was ported over: apic.go emitterc.go parserc.go readerc.go scannerc.go writerc.go yamlh.go yamlprivateh.go Copyright (c) 2006-2010 Kirill Simonov Copyright (c) 2006-2011 Kirill Simonov Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ### Apache License ### All the remaining project files are covered by the Apache license: Copyright (c) 2011-2019 Canonical Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. </ins>


Question:
we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?

<ins> Copyright 2011-2016 Canonical Ltd. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. </ins>


Question:
we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?

<ins> The MIT License (MIT) Copyright (c) 2018 QRI, Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. </ins>


Question:
we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?

mv "${TMP_LICENSE_FILE}" "${dest_dir}/LICENSE" done <ins> # copy licenses for forked code from vendor and third_party directories (cd "${KUBE_ROOT}" &&  find vendor third_party -iname 'licen[sc]e*' -o -iname 'notice*' -o -iname 'copying*' |  grep -E 'third_party|forked' |  xargs tar -czf - | tar -C "${TMP_LICENSES_DIR}" -xzf -) </ins> # Leave things like OWNERS alone. rm -f "${LICENSES_DIR}/LICENSE" rm -rf "${LICENSES_DIR}/vendor" <ins> rm -rf "${LICENSES_DIR}/third_party" </ins> mv "${TMP_LICENSES_DIR}"/* "${LICENSES_DIR}"


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI -- [sig-node] Pods should delete a collection of pods [Conformance] TBD links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
cc /sig windows
I've looked through the logs for a bit and the test itself. What the test is doing: create 3 pods. expect to find 3 pods (it doesn't seem to wait for the to start, they only need to exist). delete all 3 of them as a collection. expect all 3 pods to dissapear. All 3 pods ended up on the same node, plus some other tests were running in parallel, which could also contribute to the failure. Looking at the kubelet logs, it seems that was the remaining pod to be deleted. Filtering the logs, we can see: The test failed at, while the other 2 pods did: Those times correspond with the number of pods observed by the test: Now, my question is, why are there such big gaps between the ` events? It's 12s for test-pod-2 and 21s for test-pod-3? Seems like a long time for pods / containers to linger after they've been killed. PS: It seems that Containers are still being created / started / stopped, even if the delete request came less than 0.2s after their creation. It doesn't seem to me like a reasonable thing to do, especially if that Pod would have to pull a huge image; I would have expected to be able to cancel that.
/triage accepted
Let's discuss at the 8/17 SIG-Windows community meeting
Hit again on 8/15
I think was planning on opening up a PR to use longer timeouts in this test. We see that some Windows containers do not get scheduled until 1+ minutes into the test - especially when there are many tests running in parallel.

}}, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create pod") framework.Logf("created %v", podTestName) <ins> framework.ExpectNoError(e2epod.WaitForPodNameRunningInNamespace(f.ClientSet, podTestName, f.Namespace.Name)) framework.Logf("running and ready %v", podTestName) </ins> } // wait as required for all 3 pods to be found


Question:
I am using client-go/tools/cache package NewInformer, find the StreamWatcher and seeing extremely high memory allocations. Attached is a memory profile: From this we have 2.3 GB of memory allocated by the StreamWatcher. To get to this state I had a script to create pod and delete pod over and over again, and every 10 seconds I run it again. I left this running 30 minutes and saw this memory usage. watch code as follows, cache.NewInformer return controller and storewriter is chan queue, receiving event and distribute to queue
/sig api-machinery
Thanks for taking a look
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
i find that infomer handler be stuck will lead to streamWatcher memory high allocationsdon't hang informer handler and you will resolve that question.
/assign /triage accepted
can you check with a cpu profile if some handler is blocking?
cpu profile is normalas followscode callback handler func (w *Watcher) AddEvent(obj interface{}) { data := w.genSyncData(obj, action.SyncDataActionAdd) if data == nil { return } w.writer.Sync(data) } writer is a buffer chan with length and write chan timeout for one secondprobably is will be stuck when produce lots of eventsfinallyI store events to fifo queue when it produce lots of eventsand consume fifo queue event to writer chanthe issue is solved
So, the problem is that your handlers are slower processing the objects from the delta fifo queue, than the rate at which the events are obtained from the apiserver, this is because of this objects are processed by the handlers locking the deltaFifo queue, hence the streamwatcher accumulates the object in memory until the handlers process the object and Pop() release the lock. The code comments inform about this problem // Add() and Get(), won't be blocked for too long. the only solution I see is logging some errors warning about this (processing withouth locking doesn't seem an option, right?) I've tried to use queue depth, but that is not an option, since the contention happens before events get into the queue and it nevers goes high. I've tried adding trace for slow handlers operations, however, those maybe harmless if the incoming rate is still lower than the draining rate, unless we decide that we want to trace process handlers that take more than X seconds Is adding another intermediate queue between the streamwatecher and the deltaFifo an option? that will free the streamwatchers ...
Yeah. I don't see it as a high priority, but that could be helpful. Following the standard hander to workqueue pattern, it isn't a practical problem. /priority backlog
i don't hang informer hander by fifo queue, i find that memory will come down when lots of create and delete podsbut i running code onlinethe streamwatcher still exist high memory usage as follows pprof Please take a look at it
i am also facing same issue any guidance for the watcher memory increase in high pod churn

"errors" "fmt" "sync" <ins> "time" </ins> "k8s.io/apimachinery/pkg/util/sets" "k8s.io/klog/v2" <ins> utiltrace "k8s.io/utils/trace" </ins> ) // DeltaFIFOOptions is the configuration parameters for DeltaFIFO. All are


Question:
I am using client-go/tools/cache package NewInformer, find the StreamWatcher and seeing extremely high memory allocations. Attached is a memory profile: From this we have 2.3 GB of memory allocated by the StreamWatcher. To get to this state I had a script to create pod and delete pod over and over again, and every 10 seconds I run it again. I left this running 30 minutes and saw this memory usage. watch code as follows, cache.NewInformer return controller and storewriter is chan queue, receiving event and distribute to queue
/sig api-machinery
Thanks for taking a look
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
i find that infomer handler be stuck will lead to streamWatcher memory high allocationsdon't hang informer handler and you will resolve that question.
/assign /triage accepted
can you check with a cpu profile if some handler is blocking?
cpu profile is normalas followscode callback handler func (w *Watcher) AddEvent(obj interface{}) { data := w.genSyncData(obj, action.SyncDataActionAdd) if data == nil { return } w.writer.Sync(data) } writer is a buffer chan with length and write chan timeout for one secondprobably is will be stuck when produce lots of eventsfinallyI store events to fifo queue when it produce lots of eventsand consume fifo queue event to writer chanthe issue is solved
So, the problem is that your handlers are slower processing the objects from the delta fifo queue, than the rate at which the events are obtained from the apiserver, this is because of this objects are processed by the handlers locking the deltaFifo queue, hence the streamwatcher accumulates the object in memory until the handlers process the object and Pop() release the lock. The code comments inform about this problem // Add() and Get(), won't be blocked for too long. the only solution I see is logging some errors warning about this (processing withouth locking doesn't seem an option, right?) I've tried to use queue depth, but that is not an option, since the contention happens before events get into the queue and it nevers goes high. I've tried adding trace for slow handlers operations, however, those maybe harmless if the incoming rate is still lower than the draining rate, unless we decide that we want to trace process handlers that take more than X seconds Is adding another intermediate queue between the streamwatecher and the deltaFifo an option? that will free the streamwatchers ...
Yeah. I don't see it as a high priority, but that could be helpful. Following the standard hander to workqueue pattern, it isn't a practical problem. /priority backlog
i don't hang informer hander by fifo queue, i find that memory will come down when lots of create and delete podsbut i running code onlinethe streamwatcher still exist high memory usage as follows pprof Please take a look at it
i am also facing same issue any guidance for the watcher memory increase in high pod churn

} id := f.queue[0] f.queue = f.queue[1:] <ins> depth := len(f.queue) </ins> if f.initialPopulationCount > 0 { f.initialPopulationCount-- }


Question:
I am using client-go/tools/cache package NewInformer, find the StreamWatcher and seeing extremely high memory allocations. Attached is a memory profile: From this we have 2.3 GB of memory allocated by the StreamWatcher. To get to this state I had a script to create pod and delete pod over and over again, and every 10 seconds I run it again. I left this running 30 minutes and saw this memory usage. watch code as follows, cache.NewInformer return controller and storewriter is chan queue, receiving event and distribute to queue
/sig api-machinery
Thanks for taking a look
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
i find that infomer handler be stuck will lead to streamWatcher memory high allocationsdon't hang informer handler and you will resolve that question.
/assign /triage accepted
can you check with a cpu profile if some handler is blocking?
cpu profile is normalas followscode callback handler func (w *Watcher) AddEvent(obj interface{}) { data := w.genSyncData(obj, action.SyncDataActionAdd) if data == nil { return } w.writer.Sync(data) } writer is a buffer chan with length and write chan timeout for one secondprobably is will be stuck when produce lots of eventsfinallyI store events to fifo queue when it produce lots of eventsand consume fifo queue event to writer chanthe issue is solved
So, the problem is that your handlers are slower processing the objects from the delta fifo queue, than the rate at which the events are obtained from the apiserver, this is because of this objects are processed by the handlers locking the deltaFifo queue, hence the streamwatcher accumulates the object in memory until the handlers process the object and Pop() release the lock. The code comments inform about this problem // Add() and Get(), won't be blocked for too long. the only solution I see is logging some errors warning about this (processing withouth locking doesn't seem an option, right?) I've tried to use queue depth, but that is not an option, since the contention happens before events get into the queue and it nevers goes high. I've tried adding trace for slow handlers operations, however, those maybe harmless if the incoming rate is still lower than the draining rate, unless we decide that we want to trace process handlers that take more than X seconds Is adding another intermediate queue between the streamwatecher and the deltaFifo an option? that will free the streamwatchers ...
Yeah. I don't see it as a high priority, but that could be helpful. Following the standard hander to workqueue pattern, it isn't a practical problem. /priority backlog
i don't hang informer hander by fifo queue, i find that memory will come down when lots of create and delete podsbut i running code onlinethe streamwatcher still exist high memory usage as follows pprof Please take a look at it
i am also facing same issue any guidance for the watcher memory increase in high pod churn

continue } delete(f.items, id) <ins> // Only log traces if the queue depth is greater than 10 and it takes more than // 100 milliseconds to process one item from the queue. // Queue depth never goes high because processing an item is locking the queue, // and new items can't be added until processing finish. // https://github.com/kubernetes/kubernetes/issues/103789 if depth > 10 { trace := utiltrace.New("DeltaFIFO Pop Process", utiltrace.Field{Key: "ID", Value: id}, utiltrace.Field{Key: "Depth", Value: depth}, utiltrace.Field{Key: "Reason", Value: "slow event handlers blocking the queue"}) defer trace.LogIfLong(100 * time.Millisecond) } </ins> err := process(item) if e, ok := err.(ErrRequeue); ok { f.addIfNotPresent(id, item)


Question:
I can't find a single instance of this test passing in testgrid. Run via and potentially other suites. /sig node /kind failing-test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The test fails because there is no GPUs reported by kubelet [1]. Looking at the node info right before the failure, the node does not have any gpu capacity [2]. The artifacts only have kubelet log, but not device plugin or driver installer log. So I am not sure if there's any error on them. Is there any way to see the device plugin or driver installer log? [1] [2]
/assign I have a partial fix here - The actual feature seems broken too tho - hidden by the fact that it's been failing to provision properly for a really long time :(
Since merged this test has been relatively stable. Going to close this issue now (kept it around while I was on PTO to take a look when I got back). Seems like there are some infrequent flakes that we should follow up on after we remove DKK. /close
Closing this issue. <detailsIn response to : Since merged this test has been relatively stable. Going to close this issue now (kept it around while I was on PTO to take a look when I got back). Seems like there are some infrequent flakes that we should follow up on after we remove DKK. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen This is failing again, I'm not sure what regressed but it was 2wks ago.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I keep seeing errors like But our serial suite doesn't have GPUs attached. I'd expect this to be skipped: do you have any idea why sometimes these tests pass, sometimes they get skipped, and sometimes they time out? It makes no sense to me:
Wrote to try to dump what's matching and not causing these tests to get skipped
The tests get skipped when they run on nodes without GPUs attached, which is 2 of 3 runs every time, but cos-stable2 has 2 GPUs (). The failures happen because something in the GPU setup process broke again (I assume a bad image bump bc we don't pin), and so the test fails waiting for the GPUs to show up on the node after installing the plugin, but they are attached to the node.
I think I have a fix here -

runcmd: - modprobe configs <del> # Setup the installation target at make it executable - mkdir -p /home/kubernetes/bin/nvidia - mount --bind /home/kubernetes/bin/nvidia /home/kubernetes/bin/nvidia - mount -o remount,exec /home/kubernetes/bin/nvidia # Compile and install the nvidia driver (precompiled driver installation currently fails) - docker run --net=host --pid=host -v /dev:/dev -v /:/root -v /home/kubernetes/bin/nvidia:/usr/local/nvidia -e NVIDIA_INSTALL_DIR_HOST=/home/kubernetes/bin/nvidia -e NVIDIA_INSTALL_DIR_CONTAINER=/usr/local/nvidia -e NVIDIA_DRIVER_VERSION=460.91.03 --privileged gcr.io/cos-cloud/cos-gpu-installer:latest # Run the installer again, as on the first try it doesn't detect the libnvidia-ml.so # on the second attempt we detect it and update the ld cache. - docker run --net=host --pid=host -v /dev:/dev -v /:/root -v /home/kubernetes/bin/nvidia:/usr/local/nvidia -e NVIDIA_INSTALL_DIR_HOST=/home/kubernetes/bin/nvidia -e NVIDIA_INSTALL_DIR_CONTAINER=/usr/local/nvidia -e NVIDIA_DRIVER_VERSION=460.91.03 --privileged gcr.io/cos-cloud/cos-gpu-installer:latest </del> <ins> # Install GPU drivers - https://cloud.google.com/container-optimized-os/docs/how-to/run-gpus - cos-extensions install gpu - mount --bind /var/lib/nvidia /var/lib/nvidia - mount -o remount,exec /var/lib/nvidia /var/lib/nvidia # Run nvidia-smi to verify installation - /var/lib/nvidia/bin/nvidia-smi </ins> # Remove build containers. They're very large. - docker rm -f $(docker ps -aq) # Standard installation proceeds


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --On cgroup v1 when using systemd cgroup driver, kubelet regularly freeze and thaw its control groups, and that is causing issues further down. No freeze Running the command below should never print anything, since should always be . However, when using master/v1.22.0, this returns regularly. When using runc v1.0.0 or earlier, this can cause containers to end in a permanently frozen state. This state is tricky to debug, since no log output is given. To see if an environment is having problems, one can use: More info in and Kubernetes version (use ): : introduced in Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node /priority critical-urgent
Thanks for opening this. IIUC, this issue is limited to: cgroup v1 systemd cgroup driver runc 1.0.1+
systemd cgroup driver runc 1.0.1+ Well, yeah, kinda. It is a bit of a matrix As long as cgroup version = v1 cgroup driver = systemd the freezing will occur. The freezing is unnecessary, and creates a big overhead on machines with huge number of processes. However, if runc (the binary used by the container runtime) is v1.0.0 or before, the permanent freeze of containers can occur. If using runc (the binary) v1.0.1, the permanent freeze is mitigated and does not occur. So in one way, as long as the user is using runc binary v1.0.1 or later, this is not completely critical.
I am seeing a lot of flakiness lately in systemd node e2e jobs using crio, Almost all of them fail with container not being able to come up with following error, That seems to be coming from I am wondering if that flakiness is due to frozen cgroups issue reported in this issue. Unfortunately I do no have access to the system to query sysfs to see if cgroups is in FROZEN state.
I think is right and the flakes we're seeing in the CRI-O jobs seems to be related. runc considers a frozen cgroup state as paused container: This means that in our test, the container creation works as intended, but the fails sometimes which causes test to flake and container to get stuck in the container creation error. The code in runc seems intentional and exists since a long time, are we aware of any recent changes in k/k to this kind of cgroup handling?
cc
Yes that looks like a result of this issue. Thanks for the detailed information! This was issue in k8s (the freezing) was introduced in As disucssed here: , freezing would cause issues, and the runc bump was merged before it was properly tested. If you are talking about runc as in the binary, there was previously an issue with the definition of a "paused/frozen" container; should give some more context. That issue is fixed in runc v1.0.1 (the binary).
/triage accepted
This issue will be fixed by or any similar way to avoid the freeze.
cc:

err := kl.syncPod(pod, dockerContainers) if err != nil { glog.Errorf("Error syncing pod, skipping: %v", err) <ins> record.Eventf(pod, "", "failedSync", "Error syncing pod, skipping: %v", err) </ins> } }) }


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- - At least 08/08 sequentially performs operations that take ~20s on average. Example: For a 5000 nodes test this causes a major timeout. excludes tests with tag which SCTP had until was merged. This is why there is a shift in red cells in the testgrid after 08/15. We will add a pattern excluding until this issue is resolved. /assign /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/assign

jig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("getting the state of the sctp module on nodes") <del> nodes, err := e2enode.GetReadySchedulableNodes(cs) </del> <ins> nodes, err := e2enode.GetBoundedReadySchedulableNodes(cs, 2) </ins> framework.ExpectNoError(err) sctpLoadedAtStart := CheckSCTPModuleLoadedOnNodes(f, nodes)


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- - At least 08/08 sequentially performs operations that take ~20s on average. Example: For a 5000 nodes test this causes a major timeout. excludes tests with tag which SCTP had until was merged. This is why there is a shift in red cells in the testgrid after 08/15. We will add a pattern excluding until this issue is resolved. /assign /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/assign

framework.ExpectNoError(err, fmt.Sprintf("error while waiting for service:%s err: %v", serviceName, err)) hostExec := utils.NewHostExec(f) defer hostExec.Cleanup() <del> node, err := e2enode.GetRandomReadySchedulableNode(cs) framework.ExpectNoError(err) </del> <ins> node := &nodes.Items[0] </ins> cmd := "iptables-save" if framework.TestContext.ClusterIsIPv6() { cmd = "ip6tables-save"


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --E2E test fails. The test is implemented at: I would expect the test pass with these objects in place: AFAIK this fulfills the general requirement of the test that there's a corresponding endpointslice object with same addressing information as is on the endpoint object. IMO the test should rather look for epslice which is owned by the endpoint rather than relying on a hardcoded name. Add flag to kube-apiserver Manually create the endpoint with proper addressing for the api-server epslice "mirroring" create new endpointslice with name like --The test fails as it's hardcoded to look for endpointslice named I'm working on a PR to provide a fix proposal Kubernetes version (use ): 1.22.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: k0s Network plugin and version (if this is a network-related bug): kube-router 1.3.1 Others:
/sig testing
/remove-sig testing /sig network EndpointSlices are owned by sig network.
/kind failing-test
/remove kind-bug /remove kind-failing-test The test is correct, apiserver must generates both endpoint and enpointslices with the special name "kubernetes" , that are the backends for the "kubernetes.default" service with the IP addresses of the apiservers. This special service "kubernetes.default" and its endpoints are used by clients with In-Cluster auto discovery. I think that whatever cluster that is not generating the endpoint and endpointslices for the "kubernetes.default" service should not meet Conformance, at least I can't see how automatic in-cluster configuration will work without that ... I only see option used for webhook tests, the other 2 reconcilers always create the endpoints and slices for "kubernetes.default" should we clarify the goal of this "none" option?
Yes, I do understand this. And in our case this is true, there is a svc called as there is also endpoints object called . This makes the in-cluster discovery work. Otherwise we'd get much more errors in the conformance instead of this only one... :) In our case we're relying on the endpointslice mirroring to take care of creating the endpointslice object, which makes this test fail as it uses generated "suffix" on the object name. I have not dived too deep (yet) into the internals of endpointslices but I fail to see how the endpointslice object name really matters when both svc and endpoint object does have proper name.
oh, this seems interesting then ... who is creating the corresponding endpoint? can you paste a kubectl get endpoints -o yaml?
this is how it looks like in my cluster
In our case (k0s distro) it's our custom controller that is triggered when the controlplane is configured in HA mode. So this basically sets the api to mode . When our controller runs it has a configured LB address which it resolves to IP address(es) and creates the corresponding EP object. We do NOT create the endpointslice as we rely on the mirroring functionality to let it create and sync it. Here's the object details: Reading the code for API built-in endpoint reconcilers I see they set and create the endpointslice object by themselves. Yes, we could do that too in k0s but I fail to see why the name of the endpointslice object really matters at all.
I agree the conformance test should not be looking up endpointslice by name. It should be using a label selector of to locate one or more endpointslices and verifying the ports and endpoints across them.
Fair enough, on the good side this also will simplify the KEP from to dual-stack the apiserver you are the expert in this area, can you PTAL at this discussion? /kind failing-test
hmm the test description explicitly says the name of the object has to be "kubernetes", if the name is mandatory the test is correct, otherwise we need to rewrite the test description and talk with conformance people /area conformance
the description should be rewritten... the test is enforcing stricter requirements on endpointslices than required/expected
/triage accepted

description: The discovery.k8s.io API group MUST exist in the /apis discovery document. The discovery.k8s.io/v1 API group/version MUST exist in the /apis/discovery.k8s.io discovery document. The endpointslices resource MUST exist in the /apis/discovery.k8s.io/v1 <del> discovery document. API Server should create self referential Endpoints and EndpointSlices named "kubernetes" in the default namespace. </del> <ins> discovery document. The cluster MUST have a service named "kubernetes" on the default namespace referencing the API servers. The "kubernetes.default" service MUST have Endpoints and EndpointSlices pointing to each API server instance. </ins> release: v1.21 file: test/e2e/network/endpointslice.go - testname: EndpointSlice API


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --E2E test fails. The test is implemented at: I would expect the test pass with these objects in place: AFAIK this fulfills the general requirement of the test that there's a corresponding endpointslice object with same addressing information as is on the endpoint object. IMO the test should rather look for epslice which is owned by the endpoint rather than relying on a hardcoded name. Add flag to kube-apiserver Manually create the endpoint with proper addressing for the api-server epslice "mirroring" create new endpointslice with name like --The test fails as it's hardcoded to look for endpointslice named I'm working on a PR to provide a fix proposal Kubernetes version (use ): 1.22.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: k0s Network plugin and version (if this is a network-related bug): kube-router 1.3.1 Others:
/sig testing
/remove-sig testing /sig network EndpointSlices are owned by sig network.
/kind failing-test
/remove kind-bug /remove kind-failing-test The test is correct, apiserver must generates both endpoint and enpointslices with the special name "kubernetes" , that are the backends for the "kubernetes.default" service with the IP addresses of the apiservers. This special service "kubernetes.default" and its endpoints are used by clients with In-Cluster auto discovery. I think that whatever cluster that is not generating the endpoint and endpointslices for the "kubernetes.default" service should not meet Conformance, at least I can't see how automatic in-cluster configuration will work without that ... I only see option used for webhook tests, the other 2 reconcilers always create the endpoints and slices for "kubernetes.default" should we clarify the goal of this "none" option?
Yes, I do understand this. And in our case this is true, there is a svc called as there is also endpoints object called . This makes the in-cluster discovery work. Otherwise we'd get much more errors in the conformance instead of this only one... :) In our case we're relying on the endpointslice mirroring to take care of creating the endpointslice object, which makes this test fail as it uses generated "suffix" on the object name. I have not dived too deep (yet) into the internals of endpointslices but I fail to see how the endpointslice object name really matters when both svc and endpoint object does have proper name.
oh, this seems interesting then ... who is creating the corresponding endpoint? can you paste a kubectl get endpoints -o yaml?
this is how it looks like in my cluster
In our case (k0s distro) it's our custom controller that is triggered when the controlplane is configured in HA mode. So this basically sets the api to mode . When our controller runs it has a configured LB address which it resolves to IP address(es) and creates the corresponding EP object. We do NOT create the endpointslice as we rely on the mirroring functionality to let it create and sync it. Here's the object details: Reading the code for API built-in endpoint reconcilers I see they set and create the endpointslice object by themselves. Yes, we could do that too in k0s but I fail to see why the name of the endpointslice object really matters at all.
I agree the conformance test should not be looking up endpointslice by name. It should be using a label selector of to locate one or more endpointslices and verifying the ports and endpoints across them.
Fair enough, on the good side this also will simplify the KEP from to dual-stack the apiserver you are the expert in this area, can you PTAL at this discussion? /kind failing-test
hmm the test description explicitly says the name of the object has to be "kubernetes", if the name is mandatory the test is correct, otherwise we need to rewrite the test description and talk with conformance people /area conformance
the description should be rewritten... the test is enforcing stricter requirements on endpointslices than required/expected
/triage accepted

Description: The discovery.k8s.io API group MUST exist in the /apis discovery document. The discovery.k8s.io/v1 API group/version MUST exist in the /apis/discovery.k8s.io discovery document. The endpointslices resource MUST exist in the /apis/discovery.k8s.io/v1 discovery document. <del> API Server should create self referential Endpoints and EndpointSlices named "kubernetes" in the default namespace. </del> <ins> The cluster MUST have a service named "kubernetes" on the default namespace referencing the API servers. The "kubernetes.default" service MUST have Endpoints and EndpointSlices pointing to each API server instance. </ins> */ framework.ConformanceIt("should have Endpoints and EndpointSlices pointing to API Server", func() { namespace := "default" name := "kubernetes" <ins> // verify "kubernetes.default" service exist _, err := cs.CoreV1().Services(namespace).Get(context.TODO(), name, metav1.GetOptions{}) framework.ExpectNoError(err, "error obtaining API server "kubernetes" Service resource on "default" namespace") // verify Endpoints for the API servers exist </ins> endpoints, err := cs.CoreV1().Endpoints(namespace).Get(context.TODO(), name, metav1.GetOptions{}) <del> framework.ExpectNoError(err, "error creating Endpoints resource") if len(endpoints.Subsets) != 1 { framework.Failf("Expected 1 subset in endpoints, got %d: %#v", len(endpoints.Subsets), endpoints.Subsets) </del> <ins> framework.ExpectNoError(err, "error obtaining API server "kubernetes" Endpoint resource on "default" namespace") if len(endpoints.Subsets) == 0 { framework.Failf("Expected at least 1 subset in endpoints, got %d: %#v", len(endpoints.Subsets), endpoints.Subsets) </ins> } <del> endpointSubset := endpoints.Subsets[0] endpointSlice, err := cs.DiscoveryV1().EndpointSlices(namespace).Get(context.TODO(), name, metav1.GetOptions{}) framework.ExpectNoError(err, "error creating EndpointSlice resource") if len(endpointSlice.Ports) != len(endpointSubset.Ports) { framework.Failf("Expected EndpointSlice to have %d ports, got %d: %#v", len(endpointSubset.Ports), len(endpointSlice.Ports), endpointSlice.Ports) </del> <ins> // verify EndpointSlices for the API servers exist endpointSliceList, err := cs.DiscoveryV1().EndpointSlices(namespace).List(context.TODO(), metav1.ListOptions{ LabelSelector: "kubernetes.io/service-name=" + name, }) framework.ExpectNoError(err, "error obtaining API server "kubernetes" EndpointSlice resource on "default" namespace") if len(endpointSliceList.Items) == 0 { framework.Failf("Expected at least 1 EndpointSlice, got %d: %#v", len(endpoints.Subsets), endpoints.Subsets) </ins> } <del> numExpectedEndpoints := len(endpointSubset.Addresses) + len(endpointSubset.NotReadyAddresses) if len(endpointSlice.Endpoints) != numExpectedEndpoints { framework.Failf("Expected EndpointSlice to have %d endpoints, got %d: %#v", numExpectedEndpoints, len(endpointSlice.Endpoints), endpointSlice.Endpoints) </del> <ins> if !endpointSlicesEqual(endpoints, endpointSliceList) { framework.Failf("Expected EndpointSlice to have same addresses and port as Endpoints, got %#v: %#v", endpoints, endpointSliceList) </ins> } })


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --E2E test fails. The test is implemented at: I would expect the test pass with these objects in place: AFAIK this fulfills the general requirement of the test that there's a corresponding endpointslice object with same addressing information as is on the endpoint object. IMO the test should rather look for epslice which is owned by the endpoint rather than relying on a hardcoded name. Add flag to kube-apiserver Manually create the endpoint with proper addressing for the api-server epslice "mirroring" create new endpointslice with name like --The test fails as it's hardcoded to look for endpointslice named I'm working on a PR to provide a fix proposal Kubernetes version (use ): 1.22.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: k0s Network plugin and version (if this is a network-related bug): kube-router 1.3.1 Others:
/sig testing
/remove-sig testing /sig network EndpointSlices are owned by sig network.
/kind failing-test
/remove kind-bug /remove kind-failing-test The test is correct, apiserver must generates both endpoint and enpointslices with the special name "kubernetes" , that are the backends for the "kubernetes.default" service with the IP addresses of the apiservers. This special service "kubernetes.default" and its endpoints are used by clients with In-Cluster auto discovery. I think that whatever cluster that is not generating the endpoint and endpointslices for the "kubernetes.default" service should not meet Conformance, at least I can't see how automatic in-cluster configuration will work without that ... I only see option used for webhook tests, the other 2 reconcilers always create the endpoints and slices for "kubernetes.default" should we clarify the goal of this "none" option?
Yes, I do understand this. And in our case this is true, there is a svc called as there is also endpoints object called . This makes the in-cluster discovery work. Otherwise we'd get much more errors in the conformance instead of this only one... :) In our case we're relying on the endpointslice mirroring to take care of creating the endpointslice object, which makes this test fail as it uses generated "suffix" on the object name. I have not dived too deep (yet) into the internals of endpointslices but I fail to see how the endpointslice object name really matters when both svc and endpoint object does have proper name.
oh, this seems interesting then ... who is creating the corresponding endpoint? can you paste a kubectl get endpoints -o yaml?
this is how it looks like in my cluster
In our case (k0s distro) it's our custom controller that is triggered when the controlplane is configured in HA mode. So this basically sets the api to mode . When our controller runs it has a configured LB address which it resolves to IP address(es) and creates the corresponding EP object. We do NOT create the endpointslice as we rely on the mirroring functionality to let it create and sync it. Here's the object details: Reading the code for API built-in endpoint reconcilers I see they set and create the endpointslice object by themselves. Yes, we could do that too in k0s but I fail to see why the name of the endpointslice object really matters at all.
I agree the conformance test should not be looking up endpointslice by name. It should be using a label selector of to locate one or more endpointslices and verifying the ports and endpoints across them.
Fair enough, on the good side this also will simplify the KEP from to dual-stack the apiserver you are the expert in this area, can you PTAL at this discussion? /kind failing-test
hmm the test description explicitly says the name of the object has to be "kubernetes", if the name is mandatory the test is correct, otherwise we need to rewrite the test description and talk with conformance people /area conformance
the description should be rewritten... the test is enforcing stricter requirements on endpointslices than required/expected
/triage accepted

framework.ExpectNoError(err, "error deleting Service") return svc } <ins> // endpointSlicesEqual compare if the Endpoint and the EndpointSliceList contains the same endpoints values // as in addresses and ports, considering Ready and Unready addresses func endpointSlicesEqual(endpoints *v1.Endpoints, endpointSliceList *discoveryv1.EndpointSliceList) bool { // get the apiserver endpoint addresses epAddresses := sets.NewString() epPorts := sets.NewInt32() for _, subset := range endpoints.Subsets { for _, addr := range subset.Addresses { epAddresses.Insert(addr.IP) } for _, addr := range subset.NotReadyAddresses { epAddresses.Insert(addr.IP) } for _, port := range subset.Ports { epPorts.Insert(port.Port) } } framework.Logf("Endpoints addresses: %v , ports: %v", epAddresses.List(), epPorts.List()) // Endpoints are single stack, and must match the primary IP family of the Service kubernetes.default // However, EndpointSlices can be IPv4 or IPv6, we can only compare the Slices that match the same IP family // framework.TestContext.ClusterIsIPv6() reports the IP family of the kubernetes.default service var addrType discoveryv1.AddressType if framework.TestContext.ClusterIsIPv6() { addrType = discoveryv1.AddressTypeIPv6 } else { addrType = discoveryv1.AddressTypeIPv4 } // get the apiserver addresses from the endpoint slice list sliceAddresses := sets.NewString() slicePorts := sets.NewInt32() for _, slice := range endpointSliceList.Items { if slice.AddressType != addrType { framework.Logf("Skipping slice %s: wanted %s family, got %s", slice.Name, addrType, slice.AddressType) continue } for _, s := range slice.Endpoints { sliceAddresses.Insert(s.Addresses...) } for _, ports := range slice.Ports { if ports.Port != nil { slicePorts.Insert(*ports.Port) } } } framework.Logf("EndpointSlices addresses: %v , ports: %v", sliceAddresses.List(), slicePorts.List()) if sliceAddresses.Equal(epAddresses) && slicePorts.Equal(epPorts) { return true } return false } </ins>


Question:
TestRecordOperation TestRecordOperation TestGetHistogramVecFromGatherer /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --/sig testing This may be related with
/assign I will do some initial researches on this. is similar
/sig instrument
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig instrument Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig instrumentation
/triage accepted /assign cc
To fix it gracefully, we may refactor the code in like The refactor would change a lot and need some confirmations or demo.(I prefer) A temporary workaround before refactoring it: is another try to ignore the dup error in but it is odd as should panic as designed if one register failed.
Three testings failed due to . Haven't found package promauto () being used in these testings. It might be other issues. These testings are: /filterwithonelabel
What are you guys doing here? What test is this? You cannot change the component-base metrics code base it is doing what it is supposed to be doing. Does this test suite involve multiple sequential runs of the same test?
I think this is odd to change the component-base metrics code base. But the refactor to use is another approach. You can try to reproduce it with is with
Oh, yes. This involves multiple sequential runs of the same test to . It uses
You can use to register the registry only once. It is important to reset the metrics, or next iteration in the test will fail
Thanks I other fixes in

"github.com/stretchr/testify/assert" <del> "k8s.io/component-base/metrics/legacyregistry" </del> <ins> compbasemetrics "k8s.io/component-base/metrics" </ins> runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1alpha2" "k8s.io/kubernetes/pkg/kubelet/metrics" ) func TestRecordOperation(t *testing.T) { <del> legacyregistry.MustRegister(metrics.RuntimeOperations) legacyregistry.MustRegister(metrics.RuntimeOperationsDuration) legacyregistry.MustRegister(metrics.RuntimeOperationsErrors) </del> <ins> // Use local registry var registry = compbasemetrics.NewKubeRegistry() var gather compbasemetrics.Gatherer = registry registry.MustRegister(metrics.RuntimeOperations) registry.MustRegister(metrics.RuntimeOperationsDuration) registry.MustRegister(metrics.RuntimeOperationsErrors) registry.Reset() </ins> l, err := net.Listen("tcp", "127.0.0.1:0") assert.NoError(t, err)


Question:
TestRecordOperation TestRecordOperation TestGetHistogramVecFromGatherer /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --/sig testing This may be related with
/assign I will do some initial researches on this. is similar
/sig instrument
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig instrument Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig instrumentation
/triage accepted /assign cc
To fix it gracefully, we may refactor the code in like The refactor would change a lot and need some confirmations or demo.(I prefer) A temporary workaround before refactoring it: is another try to ignore the dup error in but it is odd as should panic as designed if one register failed.
Three testings failed due to . Haven't found package promauto () being used in these testings. It might be other issues. These testings are: /filterwithonelabel
What are you guys doing here? What test is this? You cannot change the component-base metrics code base it is doing what it is supposed to be doing. Does this test suite involve multiple sequential runs of the same test?
I think this is odd to change the component-base metrics code base. But the refactor to use is another approach. You can try to reproduce it with is with
Oh, yes. This involves multiple sequential runs of the same test to . It uses
You can use to register the registry only once. It is important to reset the metrics, or next iteration in the test will fail
Thanks I other fixes in

prometheusURL := "http://" + l.Addr().String() + "/metrics" mux := http.NewServeMux() <del> //lint:ignore SA1019 ignore deprecated warning until we move off of global registries mux.Handle("/metrics", legacyregistry.Handler()) </del> <ins> handler := compbasemetrics.HandlerFor(gather, compbasemetrics.HandlerOpts{}) mux.Handle("/metrics", handler) </ins> server := &http.Server{ Addr:    l.Addr().String(), Handler: mux,


Question:
TestRecordOperation TestRecordOperation TestGetHistogramVecFromGatherer /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --/sig testing This may be related with
/assign I will do some initial researches on this. is similar
/sig instrument
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig instrument Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig instrumentation
/triage accepted /assign cc
To fix it gracefully, we may refactor the code in like The refactor would change a lot and need some confirmations or demo.(I prefer) A temporary workaround before refactoring it: is another try to ignore the dup error in but it is odd as should panic as designed if one register failed.
Three testings failed due to . Haven't found package promauto () being used in these testings. It might be other issues. These testings are: /filterwithonelabel
What are you guys doing here? What test is this? You cannot change the component-base metrics code base it is doing what it is supposed to be doing. Does this test suite involve multiple sequential runs of the same test?
I think this is odd to change the component-base metrics code base. But the refactor to use is another approach. You can try to reproduce it with is with
Oh, yes. This involves multiple sequential runs of the same test to . It uses
You can use to register the registry only once. It is important to reset the metrics, or next iteration in the test will fail
Thanks I other fixes in

"fmt" "math" "reflect" <del> "sync" </del> "testing" "github.com/google/go-cmp/cmp" dto "github.com/prometheus/client_model/go" "k8s.io/component-base/metrics" <del> "k8s.io/component-base/metrics/legacyregistry" </del> "k8s.io/utils/pointer" )


Question:
TestRecordOperation TestRecordOperation TestGetHistogramVecFromGatherer /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --/sig testing This may be related with
/assign I will do some initial researches on this. is similar
/sig instrument
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig instrument Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig instrumentation
/triage accepted /assign cc
To fix it gracefully, we may refactor the code in like The refactor would change a lot and need some confirmations or demo.(I prefer) A temporary workaround before refactoring it: is another try to ignore the dup error in but it is odd as should panic as designed if one register failed.
Three testings failed due to . Haven't found package promauto () being used in these testings. It might be other issues. These testings are: /filterwithonelabel
What are you guys doing here? What test is this? You cannot change the component-base metrics code base it is doing what it is supposed to be doing. Does this test suite involve multiple sequential runs of the same test?
I think this is odd to change the component-base metrics code base. But the refactor to use is another approach. You can try to reproduce it with is with
Oh, yes. This involves multiple sequential runs of the same test to . It uses
You can use to register the registry only once. It is important to reset the metrics, or next iteration in the test will fail
Thanks I other fixes in

} func TestGetHistogramVecFromGatherer(t *testing.T) { <del> var registerMetrics sync.Once </del> tests := []struct { name    string lvMap   map[string]string


Question:
TestRecordOperation TestRecordOperation TestGetHistogramVecFromGatherer /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --/sig testing This may be related with
/assign I will do some initial researches on this. is similar
/sig instrument
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig instrument Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig instrumentation
/triage accepted /assign cc
To fix it gracefully, we may refactor the code in like The refactor would change a lot and need some confirmations or demo.(I prefer) A temporary workaround before refactoring it: is another try to ignore the dup error in but it is odd as should panic as designed if one register failed.
Three testings failed due to . Haven't found package promauto () being used in these testings. It might be other issues. These testings are: /filterwithonelabel
What are you guys doing here? What test is this? You cannot change the component-base metrics code base it is doing what it is supposed to be doing. Does this test suite involve multiple sequential runs of the same test?
I think this is odd to change the component-base metrics code base. But the refactor to use is another approach. You can try to reproduce it with is with
Oh, yes. This involves multiple sequential runs of the same test to . It uses
You can use to register the registry only once. It is important to reset the metrics, or next iteration in the test will fail
Thanks I other fixes in

Buckets:   buckets, } vec := metrics.NewHistogramVec(HistogramOpts, labels) <del> registerMetrics.Do(func() { legacyregistry.MustRegister(vec) }) </del> <ins> // Use local registry var registry = metrics.NewKubeRegistry() var gather metrics.Gatherer = registry registry.MustRegister(vec) </ins> // Observe two metrics with same value for label1 but different value of label2. vec.WithLabelValues("value1-0", "value2-0").Observe(1.5) vec.WithLabelValues("value1-0", "value2-1").Observe(2.5) vec.WithLabelValues("value1-1", "value2-0").Observe(3.5) vec.WithLabelValues("value1-1", "value2-1").Observe(4.5) metricName := fmt.Sprintf("%s_%s_%s", HistogramOpts.Namespace, HistogramOpts.Subsystem, HistogramOpts.Name) <del> histogramVec, _ := GetHistogramVecFromGatherer(legacyregistry.DefaultGatherer, metricName, tt.lvMap) </del> <ins> histogramVec, _ := GetHistogramVecFromGatherer(gather, metricName, tt.lvMap) </ins> if diff := cmp.Diff(tt.wantVec, histogramVec); diff != "" { t.Errorf("Got unexpected HistogramVec (-want +got):n%s", diff) }


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --- post-kubernetes-push-image-etcd n/a 2020-08-11 Started failing when merged No new versions of etcd are getting pushed until this gets fixed, when means the following PRs haven't fully taken effect: /sig api-machinery /priority critical-urgent /milestone v1.23 /assign FYI
/sig release /area release-eng
I've opened to rollback the offending commit, would like to keep this open until we can figure out the correct fix-forward
/triage accepted
was triggered by the merge of , which pushed So that's now current / includes whatever PRs linked in the description may have tried to change (excluding )
does this need promotion at all or are we good with the existing image?
/remove-priority critical-urgent /priority important-soon The job is back to passing
Recent updates are related to Debian base image update and golang upgrade. Nice to get a promotion. Good enough to use .
Yeah, looks like that the env in the build job does not search for docker CLI plugins in the path, which makes it unavailable during the run.
I think we have to set the environment variable like we already do in other build jobs, proposing the fix in
Latest test run is green:

# REVISION provides a version number for this image and all it's bundled # artifacts. It should start at zero for each LATEST_ETCD_VERSION and increment # for each revision of this image at that etcd version. <del> REVISION?=3 </del> <ins> REVISION?=4 </ins> # IMAGE_TAG Uniquely identifies k8s.gcr.io/etcd docker image with a tag of the form "<etcd-version>-<revision>". IMAGE_TAG=$(LATEST_ETCD_VERSION)-$(REVISION)


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --- post-kubernetes-push-image-etcd n/a 2020-08-11 Started failing when merged No new versions of etcd are getting pushed until this gets fixed, when means the following PRs haven't fully taken effect: /sig api-machinery /priority critical-urgent /milestone v1.23 /assign FYI
/sig release /area release-eng
I've opened to rollback the offending commit, would like to keep this open until we can figure out the correct fix-forward
/triage accepted
was triggered by the merge of , which pushed So that's now current / includes whatever PRs linked in the description may have tried to change (excluding )
does this need promotion at all or are we good with the existing image?
/remove-priority critical-urgent /priority important-soon The job is back to passing
Recent updates are related to Debian base image update and golang upgrade. Nice to get a promotion. Good enough to use .
Yeah, looks like that the env in the build job does not search for docker CLI plugins in the path, which makes it unavailable during the run.
I think we have to set the environment variable like we already do in other build jobs, proposing the fix in
Latest test run is green:

RUNNERIMAGE?=gcr.io/distroless/static:latest <ins> QEMUVERSION?=5.2.0-2 </ins> build: # Explicitly copy files to the temp directory $(BIN_INSTALL) migrate-if-needed.sh $(TEMP_DIR)


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --- post-kubernetes-push-image-etcd n/a 2020-08-11 Started failing when merged No new versions of etcd are getting pushed until this gets fixed, when means the following PRs haven't fully taken effect: /sig api-machinery /priority critical-urgent /milestone v1.23 /assign FYI
/sig release /area release-eng
I've opened to rollback the offending commit, would like to keep this open until we can figure out the correct fix-forward
/triage accepted
was triggered by the merge of , which pushed So that's now current / includes whatever PRs linked in the description may have tried to change (excluding )
does this need promotion at all or are we good with the existing image?
/remove-priority critical-urgent /priority important-soon The job is back to passing
Recent updates are related to Debian base image update and golang upgrade. Nice to get a promotion. Good enough to use .
Yeah, looks like that the env in the build job does not search for docker CLI plugins in the path, which makes it unavailable during the run.
I think we have to set the environment variable like we already do in other build jobs, proposing the fix in
Latest test run is green:

cd $(TEMP_DIR) && echo "ENV ETCD_UNSUPPORTED_ARCH=$(ARCH)" >> Dockerfile endif <ins> docker run --rm --privileged multiarch/qemu-user-static:$(QEMUVERSION) --reset -p yes docker buildx version BUILDER=$(shell docker buildx create --use) </ins> # And build the image <del> docker build  </del> <ins> docker buildx build  </ins> --pull  <ins> --load  --platform linux/$(ARCH)  </ins> -t $(REGISTRY)/etcd-$(ARCH):$(IMAGE_TAG)  --build-arg BASEIMAGE=$(BASEIMAGE)  --build-arg RUNNERIMAGE=$(RUNNERIMAGE)  $(TEMP_DIR) <ins> docker buildx rm $$BUILDER </ins> push: build docker tag $(REGISTRY)/etcd-$(ARCH):$(IMAGE_TAG) $(MANIFEST_IMAGE)-$(ARCH):$(IMAGE_TAG)


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --- post-kubernetes-push-image-etcd n/a 2020-08-11 Started failing when merged No new versions of etcd are getting pushed until this gets fixed, when means the following PRs haven't fully taken effect: /sig api-machinery /priority critical-urgent /milestone v1.23 /assign FYI
/sig release /area release-eng
I've opened to rollback the offending commit, would like to keep this open until we can figure out the correct fix-forward
/triage accepted
was triggered by the merge of , which pushed So that's now current / includes whatever PRs linked in the description may have tried to change (excluding )
does this need promotion at all or are we good with the existing image?
/remove-priority critical-urgent /priority important-soon The job is back to passing
Recent updates are related to Debian base image update and golang upgrade. Nice to get a promotion. Good enough to use .
Yeah, looks like that the env in the build job does not search for docker CLI plugins in the path, which makes it unavailable during the run.
I think we have to set the environment variable like we already do in other build jobs, proposing the fix in
Latest test run is green:

- IMAGE=gcr.io/$PROJECT_ID/etcd - BUILD_IMAGE=debian-build - TMPDIR=/workspace <ins> - HOME=/root  # for docker buildx </ins> args: - '-c' - |


Question:
I am trying out For that I enabled kubelet memory manager, cpu manager and topology manager. This is tried on a dual socket NUMA server KUBELETKUBEADMARGS="--container-runtime=remote --container-runtime- --cpu-manager-policy=static --feature-gates=RotateKubeletServerCertificate=true,AllAlpha=false,RemoveSelfLink=false,EphemeralContainers=true,BoundServiceAccountTokenVolume=false,KubeletPodResourcesGetAllocatable=true,TopologyManager=true,MemoryManager=true --kube-api-burst=40 --kube-api-qps=20 --kube-reserved=cpu=4,memory=4Gi --memory-manager-policy=static --reserved-cpus=0,1,2,3 --reserved-memory 0:memory=4Gi,1:memory=4196Mi --system-reserved=cpu=4,memory=4Gi --topology-manager-policy=single-numa-node" topologyManagerPolicy: single-numa-node is set on This throws an error 2021-10-20T11:28:49.+00:00 localhost kubelet[]: E1020 11:28:49. ] "Failed to parse kubelet flag" err="invalid argument "0:memory=4Gi,1:memory=4196Mi" for "--reserved-memory" flag: the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got 0:memory=4Gi,1:memory=4196Mi" This is on kubernetes 1.21.1 Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-09-29T11:36:35Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} Any hints on specifying "reserved-memory" for multi-socket NUMA system ???? One way is specifying multiple "reserved-memory" like But problem with this is is map, so it will get over-written. we should be able to specify reserved memory in a multi socket system. Enable above flags and run kubelet No response Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-09-29T11:36:35Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} none SLES 15 sp2 <details</detailscontainerd No response
/sig node
this is the right way to config
Problem with that is KubeletExtraArgs map[string]string is a map, so multiple "reserved-memory" will get over-written.
there is a tracking issue for kubeadm's api to support multiple arguments with the same name: but it is a backwards incompatible change. you should beusing the KubeletConfiguration for this instead:
/remove-kind support /kind bug /assign Looks like a bug in a parsing logic + better error would be beneficial
Is this a parsing bug?, if yes can you just point me where I need to fix this. I will try to fix this in the downstream release(our internal release).
Here the parsing logic - and here it is failing - Give me know if you need some additional help.
Are we planning to add support for multiple numa nodes in single "--reserved-memory" flag, or just displaying a better error msg?
We should support multiple NUMA nodes under the same --reserved-memory flag.
/triage accepted
Can you provide a sample configuration for 2 numa nodes? "." comma seperator is already there for single numa configuration.
Yes, it is the problem, initially we needed to split it by comma, but probably it will not work as expected, so maybe we can split it by and update documentation. Something like:
can you please review the PR and provide your comments

return nil } <del> numaNodeReservation := strings.Split(s, ":") if len(numaNodeReservation) != 2 { return fmt.Errorf("the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got %s", s) } memoryTypeReservations := strings.Split(numaNodeReservation[1], ",") if len(memoryTypeReservations) < 1 { return fmt.Errorf("the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got %s", s) } numaNodeID, err := strconv.Atoi(numaNodeReservation[0]) if err != nil { return fmt.Errorf("failed to convert the NUMA node ID, exptected integer, got %s", numaNodeReservation[0]) } memoryReservation := kubeletconfig.MemoryReservation{ NumaNode: int32(numaNodeID), Limits:   map[v1.ResourceName]resource.Quantity{}, } for _, reservation := range memoryTypeReservations { limit := strings.Split(reservation, "=") if len(limit) != 2 { return fmt.Errorf("the reserved limit has incorrect value, expected type=quantatity, got %s", reservation) </del> <ins> numaNodeReservations := strings.Split(s, ";") for _, reservation := range numaNodeReservations { numaNodeReservation := strings.Split(reservation, ":") if len(numaNodeReservation) != 2 { return fmt.Errorf("the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got %s", reservation) </ins> } <del> resourceName := v1.ResourceName(limit[0]) if resourceName != v1.ResourceMemory && !corev1helper.IsHugePageResourceName(resourceName) { return fmt.Errorf("memory type conversion error, unknown type: %q", resourceName) </del> <ins> memoryTypeReservations := strings.Split(numaNodeReservation[1], ",") if len(memoryTypeReservations) < 1 { return fmt.Errorf("the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got %s", reservation) </ins> } <del> q, err := resource.ParseQuantity(limit[1]) </del> <ins> numaNodeID, err := strconv.Atoi(numaNodeReservation[0]) </ins> if err != nil { <del> return fmt.Errorf("failed to parse the quantatity, expected quantatity, got %s", limit[1]) </del> <ins> return fmt.Errorf("failed to convert the NUMA node ID, exptected integer, got %s", numaNodeReservation[0]) </ins> } <del> memoryReservation.Limits[v1.ResourceName(limit[0])] = q } </del> <ins> memoryReservation := kubeletconfig.MemoryReservation{ NumaNode: int32(numaNodeID), Limits:   map[v1.ResourceName]resource.Quantity{}, } </ins> <del> *v.Value = append(*v.Value, memoryReservation) </del> <ins> for _, memoryTypeReservation := range memoryTypeReservations { limit := strings.Split(memoryTypeReservation, "=") if len(limit) != 2 { return fmt.Errorf("the reserved limit has incorrect value, expected type=quantatity, got %s", memoryTypeReservation) } resourceName := v1.ResourceName(limit[0]) if resourceName != v1.ResourceMemory && !corev1helper.IsHugePageResourceName(resourceName) { return fmt.Errorf("memory type conversion error, unknown type: %q", resourceName) } q, err := resource.ParseQuantity(limit[1]) if err != nil { return fmt.Errorf("failed to parse the quantatity, expected quantatity, got %s", limit[1]) } memoryReservation.Limits[v1.ResourceName(limit[0])] = q } *v.Value = append(*v.Value, memoryReservation) } </ins> return nil }


Question:
I am trying out For that I enabled kubelet memory manager, cpu manager and topology manager. This is tried on a dual socket NUMA server KUBELETKUBEADMARGS="--container-runtime=remote --container-runtime- --cpu-manager-policy=static --feature-gates=RotateKubeletServerCertificate=true,AllAlpha=false,RemoveSelfLink=false,EphemeralContainers=true,BoundServiceAccountTokenVolume=false,KubeletPodResourcesGetAllocatable=true,TopologyManager=true,MemoryManager=true --kube-api-burst=40 --kube-api-qps=20 --kube-reserved=cpu=4,memory=4Gi --memory-manager-policy=static --reserved-cpus=0,1,2,3 --reserved-memory 0:memory=4Gi,1:memory=4196Mi --system-reserved=cpu=4,memory=4Gi --topology-manager-policy=single-numa-node" topologyManagerPolicy: single-numa-node is set on This throws an error 2021-10-20T11:28:49.+00:00 localhost kubelet[]: E1020 11:28:49. ] "Failed to parse kubelet flag" err="invalid argument "0:memory=4Gi,1:memory=4196Mi" for "--reserved-memory" flag: the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got 0:memory=4Gi,1:memory=4196Mi" This is on kubernetes 1.21.1 Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-09-29T11:36:35Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} Any hints on specifying "reserved-memory" for multi-socket NUMA system ???? One way is specifying multiple "reserved-memory" like But problem with this is is map, so it will get over-written. we should be able to specify reserved memory in a multi socket system. Enable above flags and run kubelet No response Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-09-29T11:36:35Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} none SLES 15 sp2 <details</detailscontainerd No response
/sig node
this is the right way to config
Problem with that is KubeletExtraArgs map[string]string is a map, so multiple "reserved-memory" will get over-written.
there is a tracking issue for kubeadm's api to support multiple arguments with the same name: but it is a backwards incompatible change. you should beusing the KubeletConfiguration for this instead:
/remove-kind support /kind bug /assign Looks like a bug in a parsing logic + better error would be beneficial
Is this a parsing bug?, if yes can you just point me where I need to fix this. I will try to fix this in the downstream release(our internal release).
Here the parsing logic - and here it is failing - Give me know if you need some additional help.
Are we planning to add support for multiple numa nodes in single "--reserved-memory" flag, or just displaying a better error msg?
We should support multiple NUMA nodes under the same --reserved-memory flag.
/triage accepted
Can you provide a sample configuration for 2 numa nodes? "." comma seperator is already there for single numa configuration.
Yes, it is the problem, initially we needed to split it by comma, but probably it will not work as expected, so maybe we can split it by and update documentation. Something like:
can you please review the PR and provide your comments

}, }, { <ins> desc: "valid input with ';' as separator for multiple reserved-memory arguments", argc: "blah --reserved-memory=0:memory=1Gi,hugepages-1Gi=1Gi;1:memory=1Gi", expectVal: []kubeletconfig.MemoryReservation{ { NumaNode: 0, Limits: v1.ResourceList{ v1.ResourceMemory:        memory1Gi, resourceNameHugepages1Gi: memory1Gi, }, }, { NumaNode: 1, Limits: v1.ResourceList{ v1.ResourceMemory: memory1Gi, }, }, }, }, { </ins> desc:      "invalid input", argc:      "blah --reserved-memory=bad-input", expectVal: nil,


Question:
configmap, like edit configmap test, and make some error and write-save, will see below when the content after edited is invalid, should not remove data , thus mean expect content below configmap 2 edit configmap make invalid No response version kubeadm <details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
PTAL
Hi, I am new to this project and would love to contribute. Can I work on this issue?
/sig cli

stripped := []byte{} lines := bytes.Split(file, []byte("n")) for i, line := range lines { <del> if bytes.HasPrefix(bytes.TrimSpace(line), []byte("#")) { </del> <ins> trimline := bytes.TrimSpace(line) if bytes.HasPrefix(trimline, []byte("#")) && !bytes.HasPrefix(trimline, []byte("#!")) { </ins> continue } stripped = append(stripped, line...)


Question:
doesn't do the same embedded resource and x-kubernetes-list-type "map" and "set" invariant validation that happens in custom resource create/update validation. Does that mean embedded objects and list-type fields under custom resource status can contain invalid data? if so, that needs to be fixed in a way that tolerates existing invalid data (ratcheting validation) Originally posted by in /sig api-machinery /kind bug /priority important-longterm
/triage accepted
I would like to collaborate for fixing the validation issue for custom resources and tolerate the existing invalid data. Could you please guide me with some code pointers or files and elaborate on how can I proceed for fixing this bug.
I haven't tested this, but something along the lines of this: we'd need tests to exercise the following scenarios on a CRD that specifies a status subresource with a list property with x-kubernetes-list-type=map and x-kubernetes-list-type=set a custom resource instance that already has duplicate items in the status property list can be updated successfully while still having duplicate items in the list (validation doesn't block an invalid → invalid status update) a custom resource instance that already has duplicate items in the status property list can be updated successfully to set unique items in the list (validation doesn't block an invalid → valid status update) a custom resource instance that does not have duplicate items in the status property list cannot be updated to set duplicate items in the list (validation blocks a valid → invalid status update)
since it's been a couple of months, i'm going to try and take a stab at this /assign
/assign
Hi k8s 1.25 Bug Triage Shadow here, just checking in to see if this is still on track for k8s 1.25? Thank you.
has stalled needing tests and a rebase
Thanks for the update.
Hello with the code freeze approaching, is this still on track for v1.25? Thanks!
not really, the status on is the same (needs tests)
/milestone clear

"sigs.k8s.io/structured-merge-diff/v4/fieldpath" "k8s.io/apiextensions-apiserver/pkg/apiserver/schema/cel" <ins> structurallisttype "k8s.io/apiextensions-apiserver/pkg/apiserver/schema/listtype" </ins> "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/validation/field"