Question:
When installing the metrics-server to implement the API and the prometheus-adapter to implement the API, it is possible to make the OpenAPI handler fail to initialize with this error log in apiserver: This will result in kubectl discovery calls to /openapi/v2 to fail with a 503: The OpenAPI handler should not fail to initialize if both and are serving the same path for different API groups/versions. I can provide exact installation steps if needed, however, a high-level repro is: install metrics-server to serve install prometheus-adapter to serve ensure resourceRules is populated in the prometheus-adapter config so that resource metrics are also served by restarting apiserver will result in the OpenAPI failure and 503 responses to /openapi/v2 path No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
Please let me know if I should reopen the issue in
/sig api-machinery
/sig instrumentation [metrics apis]
Would you mind taking a look when have time? Thank you^^ /triage accepted
I have a question. should be serving under . Why does an apiservice serving another gvk register paths under ? cc we've made the assumption that an APIService (such as metrics-server or prometheus-adapter) serves only gvks under its own registered group version. I'll do some more investigation but just wondering if prometheus doesn't follow that assumption?
the prometheus-adapter can also be configured to serve and replace the metrics-server by enabling these metrics, However, if the prometheus-adapter fronts and metrics-server fronts , it does seem like the correct behavior would be for prometheus-adapter to serve until an operator updates the APIService to reference prometheus-adapter rather than metrics-server. We're seeing this issue occur for upgrades between 1.27 and 1.28 so it does look this path duplication did not cause the OpenAPI handler to fail on 1.27.
I just upgraded from 1.27 to 1.28 and running into this problem. Some kubectl apply operations fail, and ArgoCD cannot load its custom resources. Any suggested workaround? Would removing the from the prometheus configuration resolve it? Or perhaps disable metrics-server?
Simply looking at the code diff between 1.27 and 1.28 I can't pinpoint the cause. Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question. In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.
I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in

apiService := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "apiservicegroup", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy"}, }, }


Question:
When installing the metrics-server to implement the API and the prometheus-adapter to implement the API, it is possible to make the OpenAPI handler fail to initialize with this error log in apiserver: This will result in kubectl discovery calls to /openapi/v2 to fail with a 503: The OpenAPI handler should not fail to initialize if both and are serving the same path for different API groups/versions. I can provide exact installation steps if needed, however, a high-level repro is: install metrics-server to serve install prometheus-adapter to serve ensure resourceRules is populated in the prometheus-adapter config so that resource metrics are also served by restarting apiserver will result in the OpenAPI failure and 503 responses to /openapi/v2 path No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
Please let me know if I should reopen the issue in
/sig api-machinery
/sig instrumentation [metrics apis]
Would you mind taking a look when have time? Thank you^^ /triage accepted
I have a question. should be serving under . Why does an apiservice serving another gvk register paths under ? cc we've made the assumption that an APIService (such as metrics-server or prometheus-adapter) serves only gvks under its own registered group version. I'll do some more investigation but just wondering if prometheus doesn't follow that assumption?
the prometheus-adapter can also be configured to serve and replace the metrics-server by enabling these metrics, However, if the prometheus-adapter fronts and metrics-server fronts , it does seem like the correct behavior would be for prometheus-adapter to serve until an operator updates the APIService to reference prometheus-adapter rather than metrics-server. We're seeing this issue occur for upgrades between 1.27 and 1.28 so it does look this path duplication did not cause the OpenAPI handler to fail on 1.27.
I just upgraded from 1.27 to 1.28 and running into this problem. Some kubectl apply operations fail, and ArgoCD cannot load its custom resources. Any suggested workaround? Would removing the from the prometheus configuration resolve it? Or perhaps disable metrics-server?
Simply looking at the code diff between 1.27 and 1.28 I can't pinpoint the cause. Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question. In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.
I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in

apiServiceFailed := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "failed", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy"}, }, }


Question:
We should check the images configured in test/e2e to ensure we're not adding new images: In almost all cases, should be extended to support any missing functionality and used as the sole image for new tests. This has been the policy in SIG Testing for at least a few years now. To prevent regressing on this, we should check the list of images and permit tag updates but NOT adding any more names to the list. We can do this with an allowlist against /sig testing /kind test Others are welcome to work on this, but I'll submit something soon if nobody beats me to it. /assign Follow up to , we should aim to ship something this release (1.30).
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : We should check the images configured in test/e2e to ensure we're not adding new images: /kind test /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello! Release Signal shadow here. This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting (less than a month from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As this issue is tagged for 1.30, is it still planned for this release?
We still intend to do this and it's a test-only change so we're aiming for test freeze.
Hello! Release Signal shadow here. I'd like to check what's the status. The code freeze is starting in a week, and we want to ensure that each PR has a chance to be merged on time. Is it still planned for this release?
As has no update recently, /milestone clear we may make it in v1.31.
Yeah I think we have to revisit this later.

<ins> #!/usr/bin/env bash # Copyright 2024 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. cd "${KUBE_ROOT}" source hack/lib/init.sh # NOTE: Please do NOT add any to this list!! # # We are aiming to consolidate on: registry.k8s.io/e2e-test-images/agnhost # The sources for which are in test/images/agnhost. # If agnhost is missing functionality for your tests, please reach out to SIG Testing. kube::util::read-array PERMITTED_IMAGES < <(sed '/^#/d' ./test/images/.permitted-images) # get current list of images, ignoring tags echo "Getting e2e image list ..." make WHAT=test/e2e/e2e.test e2e_test="$(kube::util::find-binary e2e.test)" kube::util::read-array IMAGES < <("${e2e_test}" --list-images | sed -E 's/^(.+):[^:]+$/1/' | LC_ALL=C sort -u) # diff versus known permitted images ret=0 >&2 echo "Diffing e2e image list ..." diff -Naupr <(printf '%sn' "${IMAGES[@]}") <(printf '%sn' "${PERMITTED_IMAGES[@]}") || ret=$? if [[ $ret -eq 0 ]]; then >&2 echo "PASS: e2e images used are OK." else >&2 echo "FAIL: e2e images do not match the approved list!" >&2 echo "" >&2 echo "Please use registry.k8s.io/e2e-test-images/agnhost wherever possible, we are consolidating test images." >&2 echo "See: test/images/agnhost/README.md" >&2 echo "" >&2 echo "You can reach out to https://git.k8s.io/community/sig-testing for help." exit 1 fi </ins>


Question:
We should check the images configured in test/e2e to ensure we're not adding new images: In almost all cases, should be extended to support any missing functionality and used as the sole image for new tests. This has been the policy in SIG Testing for at least a few years now. To prevent regressing on this, we should check the list of images and permit tag updates but NOT adding any more names to the list. We can do this with an allowlist against /sig testing /kind test Others are welcome to work on this, but I'll submit something soon if nobody beats me to it. /assign Follow up to , we should aim to ship something this release (1.30).
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : We should check the images configured in test/e2e to ensure we're not adding new images: /kind test /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello! Release Signal shadow here. This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting (less than a month from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As this issue is tagged for 1.30, is it still planned for this release?
We still intend to do this and it's a test-only change so we're aiming for test freeze.
Hello! Release Signal shadow here. I'd like to check what's the status. The code freeze is starting in a week, and we want to ensure that each PR has a chance to be merged on time. Is it still planned for this release?
As has no update recently, /milestone clear we may make it in v1.31.
Yeah I think we have to revisit this later.

<ins> # NOTE: Please do NOT add any to this list!! # # We are aiming to consolidate on: registry.k8s.io/e2e-test-images/agnhost # The sources for which are in test/images/agnhost. # If agnhost is missing functionality for your tests, please reach out to SIG Testing. gcr.io/authenticated-image-pulling/alpine gcr.io/authenticated-image-pulling/windows-nanoserver gcr.io/k8s-authenticated-test/agnhost invalid.registry.k8s.io/invalid/alpine registry.k8s.io/build-image/distroless-iptables registry.k8s.io/cloud-provider-gcp/gcp-compute-persistent-disk-csi-driver registry.k8s.io/e2e-test-images/agnhost registry.k8s.io/e2e-test-images/apparmor-loader registry.k8s.io/e2e-test-images/busybox registry.k8s.io/e2e-test-images/cuda-vector-add registry.k8s.io/e2e-test-images/httpd registry.k8s.io/e2e-test-images/ipc-utils registry.k8s.io/e2e-test-images/jessie-dnsutils registry.k8s.io/e2e-test-images/kitten registry.k8s.io/e2e-test-images/nautilus registry.k8s.io/e2e-test-images/nginx registry.k8s.io/e2e-test-images/node-perf/npb-ep registry.k8s.io/e2e-test-images/node-perf/npb-is registry.k8s.io/e2e-test-images/node-perf/tf-wide-deep registry.k8s.io/e2e-test-images/nonewprivs registry.k8s.io/e2e-test-images/nonroot registry.k8s.io/e2e-test-images/perl registry.k8s.io/e2e-test-images/redis registry.k8s.io/e2e-test-images/regression-issue-74839 registry.k8s.io/e2e-test-images/resource-consumer registry.k8s.io/e2e-test-images/sample-apiserver registry.k8s.io/e2e-test-images/volume/iscsi registry.k8s.io/e2e-test-images/volume/nfs registry.k8s.io/etcd registry.k8s.io/pause registry.k8s.io/prometheus-dummy-exporter registry.k8s.io/prometheus-to-sd registry.k8s.io/sd-dummy-exporter registry.k8s.io/sig-storage/csi-attacher registry.k8s.io/sig-storage/csi-external-health-monitor-controller registry.k8s.io/sig-storage/csi-node-driver-registrar registry.k8s.io/sig-storage/csi-provisioner registry.k8s.io/sig-storage/csi-resizer registry.k8s.io/sig-storage/csi-snapshotter registry.k8s.io/sig-storage/hello-populator registry.k8s.io/sig-storage/hostpathplugin registry.k8s.io/sig-storage/livenessprobe registry.k8s.io/sig-storage/nfs-provisioner registry.k8s.io/sig-storage/volume-data-source-validator </ins>


Question:
Part of From For now posted as a "help wanted" issue, however if we cannot hit the timeline for 1.30 code freeze () I would take it over so we don't miss. Still issue is simple enough that it can be quickly implemented. Implementation tips: Add a metric around waitUnitlFreshAndBlock method Metric should be a histogram with latency buckets similar to api latency Metric should have label for resource version. A way to monitor watch cache impact on request latency.
/sig api-machinery /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig api-machinery /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

}, []string{"resource"}, ) <ins> WatchCacheReadWait = compbasemetrics.NewHistogramVec( &compbasemetrics.HistogramOpts{ Namespace:      namespace, Subsystem:      subsystem, Name:           "read_wait_seconds", Help:           "Histogram of time spent waiting for a watch cache to become fresh.", StabilityLevel: compbasemetrics.ALPHA, Buckets:        []float64{0.005, 0.025, 0.05, 0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 1.25, 1.5, 2, 3}, }, []string{"resource"}) </ins> ) var registerMetrics sync.Once


Question:
Part of From For now posted as a "help wanted" issue, however if we cannot hit the timeline for 1.30 code freeze () I would take it over so we don't miss. Still issue is simple enough that it can be quickly implemented. Implementation tips: Add a metric around waitUnitlFreshAndBlock method Metric should be a histogram with latency buckets similar to api latency Metric should have label for resource version. A way to monitor watch cache impact on request latency.
/sig api-machinery /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig api-machinery /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

legacyregistry.MustRegister(watchCacheCapacityDecreaseTotal) legacyregistry.MustRegister(WatchCacheCapacity) legacyregistry.MustRegister(WatchCacheInitializations) <ins> legacyregistry.MustRegister(WatchCacheReadWait) </ins> }) }


Question:
Part of From For now posted as a "help wanted" issue, however if we cannot hit the timeline for 1.30 code freeze () I would take it over so we don't miss. Still issue is simple enough that it can be quickly implemented. Implementation tips: Add a metric around waitUnitlFreshAndBlock method Metric should be a histogram with latency buckets similar to api latency Metric should have label for resource version. A way to monitor watch cache impact on request latency.
/sig api-machinery /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig api-machinery /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

// You HAVE TO explicitly call w.RUnlock() after this function. func (w *watchCache) waitUntilFreshAndBlock(ctx context.Context, resourceVersion uint64) error { startTime := w.clock.Now() <ins> defer func() { if resourceVersion > 0 { metrics.WatchCacheReadWait.WithContext(ctx).WithLabelValues(w.groupResource.String()).Observe(w.clock.Since(startTime).Seconds()) } }() </ins> // In case resourceVersion is 0, we accept arbitrarily stale result. // As a result, the condition in the below for loop will never be


Question:
Part of From For now posted as a "help wanted" issue, however if we cannot hit the timeline for 1.30 code freeze () I would take it over so we don't miss. Still issue is simple enough that it can be quickly implemented. Implementation tips: Add a metric around waitUnitlFreshAndBlock method Metric should be a histogram with latency buckets similar to api latency Metric should have label for resource version. A way to monitor watch cache impact on request latency.
/sig api-machinery /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig api-machinery /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

"k8s.io/apimachinery/pkg/watch" "k8s.io/apiserver/pkg/features" "k8s.io/apiserver/pkg/storage" <ins> "k8s.io/apiserver/pkg/storage/cacher/metrics" </ins> utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/tools/cache" featuregatetesting "k8s.io/component-base/featuregate/testing" <ins> k8smetrics "k8s.io/component-base/metrics" "k8s.io/component-base/metrics/testutil" </ins> "k8s.io/utils/clock" testingclock "k8s.io/utils/clock/testing" )


Question:
Part of From For now posted as a "help wanted" issue, however if we cannot hit the timeline for 1.30 code freeze () I would take it over so we don't miss. Still issue is simple enough that it can be quickly implemented. Implementation tips: Add a metric around waitUnitlFreshAndBlock method Metric should be a histogram with latency buckets similar to api latency Metric should have label for resource version. A way to monitor watch cache impact on request latency.
/sig api-machinery /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig api-machinery /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

store.updateCache(add) } } <ins> func TestHistogramCacheReadWait(t *testing.T) { registry := k8smetrics.NewKubeRegistry() if err := registry.Register(metrics.WatchCacheReadWait); err != nil { t.Errorf("unexpected error: %v", err) } ctx := context.Background() testedMetrics := "apiserver_watch_cache_read_wait_seconds" store := newTestWatchCache(2, &cache.Indexers{}) defer store.Stop() // In background, update the store. go func() { if err := store.Add(makeTestPod("foo", 2)); err != nil { t.Errorf("unexpected error: %v", err) } if err := store.Add(makeTestPod("bar", 5)); err != nil { t.Errorf("unexpected error: %v", err) } }() testCases := []struct { desc            string resourceVersion uint64 want            string }{ { desc:            "resourceVersion is non-zero", resourceVersion: 5, want: ` # HELP apiserver_watch_cache_read_wait_seconds [ALPHA] Histogram of time spent waiting for a watch cache to become fresh. # TYPE apiserver_watch_cache_read_wait_seconds histogram apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.005"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.025"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.05"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.1"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.2"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.4"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.6"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.8"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="1"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="1.25"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="1.5"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="2"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="3"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="+Inf"} 1 apiserver_watch_cache_read_wait_seconds_sum{resource="pods"} 0 apiserver_watch_cache_read_wait_seconds_count{resource="pods"} 1 `, }, { desc:            "resourceVersion is 0", resourceVersion: 0, want:            ``, }, } for _, test := range testCases { t.Run(test.desc, func(t *testing.T) { defer registry.Reset() if _, _, _, err := store.WaitUntilFreshAndGet(ctx, test.resourceVersion, "prefix/ns/bar"); err != nil { t.Errorf("unexpected error: %v", err) } if err := testutil.GatherAndCompare(registry, strings.NewReader(test.want), testedMetrics); err != nil { t.Errorf("unexpected error: %v", err) } }) } } </ins>


Question:
07-02 and 13-02 No response No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
I might be related to the npd.
The same test failures happening again on 25.02:
/sig node
/sig node /cc The restart of caused the flake. I also opened to track it in NPD repo.
I missed this log. It is caused by oom. Adding some memory may help. I submitted to fix it. Does mean killing metrics-server?
/remove-sig node Sorry, I misread the log. The NPD found the metric server is OOM. It is about the metrics server, not the NPD. Sorry for disturb. I may add some memory for metric-server and check why metric server memory changed. I heard that metrics server recently released a new version. I will check if it is related.
/sig node
OOM metrics shows the metric server is OOMed. This is odd, that only metrics-server-nanny is with a memory limit. It uses whose command is . metrics-server container has no memory limit, but it got killed by OOM. Does this indicate that the node is OOM? (edited)
The last run failed in node-problem-detector ContainerStateTerminated. I0227 05:03:31. ] PodStartupLatency: perc50: 14m29.018886827s, perc90: 28m44.625518807s, perc99: 32m2.19303496s; threshold 1h0m0s I0227 05:03:32. ] Loaded restart count threshold overrides: map[coredns:2 fluentd-gcp:999 konnectivity-agent:2 kube-controller-manager:1 kube-proxy:2 kube-scheduler:1 l7-lb-controller:1 metadata-proxy:2 prometheus-to-sd-exporter:2] I0227 05:03:32. ] ClusterOOMsTracker: gathering cluster OOMs tracking measurement E0227 05:03:32. ] TestMetrics: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0] I0227 05:03:32. ] Step "[step: 29] gathering measurements" ended W0227 05:03:32. ] Got errors during step execution: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] I0227 05:03:32. ] Waiting for the chaos monkey subroutine to end... I0227 05:03:32. ] Chaos monkey ended. I0227 05:03:32. ] I0227 05:03:32. ] Stopping DnsLookupLatency probe... I0227 05:03:47. ] Stopping InClusterNetworkLatency probe... I0227 05:04:22. ] Resources cleanup time: 50.539700215s E0227 05:04:22. ]
E0227 05:04:22. ] Test Finished E0227 05:04:22. ] Test: E0227 05:04:22. ] Status: Fail E0227 05:04:22. ] Errors: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] E0227 05:04:22. ]
I0227 05:04:22. ]
I0227 05:04:22. ] Running testing/huge- I0227 05:04:22. ]

MINION_TAG="${INSTANCE_PREFIX}-minion" MINION_NAMES=($(eval echo ${INSTANCE_PREFIX}-minion-{1..${NUM_MINIONS}})) MINION_IP_RANGES=($(eval echo "10.245.{1..${NUM_MINIONS}}.0/24")) <del> MINION_SCOPES="" </del> <ins> MINION_SCOPES="storage-ro,compute-rw" </ins> # Increase the sleep interval value if concerned about API rate limits. 3, in seconds, is the default. POLL_SLEEP_INTERVAL=3 PORTAL_NET="10.0.0.0/16"


Question:
07-02 and 13-02 No response No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
I might be related to the npd.
The same test failures happening again on 25.02:
/sig node
/sig node /cc The restart of caused the flake. I also opened to track it in NPD repo.
I missed this log. It is caused by oom. Adding some memory may help. I submitted to fix it. Does mean killing metrics-server?
/remove-sig node Sorry, I misread the log. The NPD found the metric server is OOM. It is about the metrics server, not the NPD. Sorry for disturb. I may add some memory for metric-server and check why metric server memory changed. I heard that metrics server recently released a new version. I will check if it is related.
/sig node
OOM metrics shows the metric server is OOMed. This is odd, that only metrics-server-nanny is with a memory limit. It uses whose command is . metrics-server container has no memory limit, but it got killed by OOM. Does this indicate that the node is OOM? (edited)
The last run failed in node-problem-detector ContainerStateTerminated. I0227 05:03:31. ] PodStartupLatency: perc50: 14m29.018886827s, perc90: 28m44.625518807s, perc99: 32m2.19303496s; threshold 1h0m0s I0227 05:03:32. ] Loaded restart count threshold overrides: map[coredns:2 fluentd-gcp:999 konnectivity-agent:2 kube-controller-manager:1 kube-proxy:2 kube-scheduler:1 l7-lb-controller:1 metadata-proxy:2 prometheus-to-sd-exporter:2] I0227 05:03:32. ] ClusterOOMsTracker: gathering cluster OOMs tracking measurement E0227 05:03:32. ] TestMetrics: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0] I0227 05:03:32. ] Step "[step: 29] gathering measurements" ended W0227 05:03:32. ] Got errors during step execution: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] I0227 05:03:32. ] Waiting for the chaos monkey subroutine to end... I0227 05:03:32. ] Chaos monkey ended. I0227 05:03:32. ] I0227 05:03:32. ] Stopping DnsLookupLatency probe... I0227 05:03:47. ] Stopping InClusterNetworkLatency probe... I0227 05:04:22. ] Resources cleanup time: 50.539700215s E0227 05:04:22. ]
E0227 05:04:22. ] Test Finished E0227 05:04:22. ] Test: E0227 05:04:22. ] Status: Fail E0227 05:04:22. ] Errors: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] E0227 05:04:22. ]
I0227 05:04:22. ]
I0227 05:04:22. ] Running testing/huge- I0227 05:04:22. ]

<ins> apiVersion: v1beta1 desiredState: manifest: containers: - image: kubernetes/pause name: testpd volumeMounts: - mountPath: "/testpd" name: "testpd" id: testpd version: v1beta1 volumes: - name: testpd source: persistentDisk: # This GCE PD must already exist and be formatted ext4 pdName: "%insert_pd_name_here%" fsType: ext4 id: testpd kind: Pod labels: test: testpd </ins>


Question:
07-02 and 13-02 No response No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
I might be related to the npd.
The same test failures happening again on 25.02:
/sig node
/sig node /cc The restart of caused the flake. I also opened to track it in NPD repo.
I missed this log. It is caused by oom. Adding some memory may help. I submitted to fix it. Does mean killing metrics-server?
/remove-sig node Sorry, I misread the log. The NPD found the metric server is OOM. It is about the metrics server, not the NPD. Sorry for disturb. I may add some memory for metric-server and check why metric server memory changed. I heard that metrics server recently released a new version. I will check if it is related.
/sig node
OOM metrics shows the metric server is OOMed. This is odd, that only metrics-server-nanny is with a memory limit. It uses whose command is . metrics-server container has no memory limit, but it got killed by OOM. Does this indicate that the node is OOM? (edited)
The last run failed in node-problem-detector ContainerStateTerminated. I0227 05:03:31. ] PodStartupLatency: perc50: 14m29.018886827s, perc90: 28m44.625518807s, perc99: 32m2.19303496s; threshold 1h0m0s I0227 05:03:32. ] Loaded restart count threshold overrides: map[coredns:2 fluentd-gcp:999 konnectivity-agent:2 kube-controller-manager:1 kube-proxy:2 kube-scheduler:1 l7-lb-controller:1 metadata-proxy:2 prometheus-to-sd-exporter:2] I0227 05:03:32. ] ClusterOOMsTracker: gathering cluster OOMs tracking measurement E0227 05:03:32. ] TestMetrics: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0] I0227 05:03:32. ] Step "[step: 29] gathering measurements" ended W0227 05:03:32. ] Got errors during step execution: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] I0227 05:03:32. ] Waiting for the chaos monkey subroutine to end... I0227 05:03:32. ] Chaos monkey ended. I0227 05:03:32. ] I0227 05:03:32. ] Stopping DnsLookupLatency probe... I0227 05:03:47. ] Stopping InClusterNetworkLatency probe... I0227 05:04:22. ] Resources cleanup time: 50.539700215s E0227 05:04:22. ]
E0227 05:04:22. ] Test Finished E0227 05:04:22. ] Test: E0227 05:04:22. ] Status: Fail E0227 05:04:22. ] Errors: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] E0227 05:04:22. ]
I0227 05:04:22. ]
I0227 05:04:22. ] Running testing/huge- I0227 05:04:22. ]

<ins> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Launches a container and verifies it can be reached. Assumes that # we're being called by hack/e2e-test.sh (we use some env vars it sets up). # TODO: Convert uses of gcutil to gcloud set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../.. source "${KUBE_ROOT}/cluster/kube-env.sh" source "${KUBE_ROOT}/cluster/$KUBERNETES_PROVIDER/util.sh" if [[ "$KUBERNETES_PROVIDER" != "gce" ]]; then echo "PD test is only run for GCE" return 0 fi disk_name="e2e-$(date +%H-%M-%s)" config="/tmp/${disk_name}.yaml" function teardown() { echo "Cleaning up test artifacts" ${KUBECFG} delete pods/testpd rm -rf ${config} echo "Waiting for disk to become unmounted" sleep 20 gcutil deletedisk -f --zone=${ZONE} ${disk_name} } trap "teardown" EXIT perl -p -e "s/%.*%/${disk_name}/g" ${KUBE_ROOT}/examples/gce-pd/testpd.yaml > ${config} # Create and mount the disk. gcutil adddisk --size_gb=10 --zone=${ZONE} ${disk_name} gcutil attachdisk --disk ${disk_name} ${MASTER_NAME} gcutil ssh ${MASTER_NAME} sudo rm -rf /mnt/tmp gcutil ssh ${MASTER_NAME} sudo mkdir /mnt/tmp gcutil ssh ${MASTER_NAME} sudo /usr/share/google/safe_format_and_mount /dev/disk/by-id/google-${disk_name} /mnt/tmp gcutil ssh ${MASTER_NAME} sudo umount /mnt/tmp gcloud compute instances detach-disk --disk ${disk_name} --zone ${ZONE} ${MASTER_NAME} ${KUBECFG} -c ${config} create pods pod_id_list=$($KUBECFG '-template={{range.items}}{{.id}} {{end}}' -l test=testpd list pods) # Pod turn up on a clean cluster can take a while for the docker image pull. all_running=0 for i in $(seq 1 24); do echo "Waiting for pods to come up." sleep 5 all_running=1 for id in $pod_id_list; do current_status=$($KUBECFG -template '{{.currentState.status}}' get pods/$id) || true if [[ "$current_status" != "Running" ]]; then all_running=0 break fi done if [[ "${all_running}" == 1 ]]; then break fi done if [[ "${all_running}" == 0 ]]; then echo "Pods did not come up in time" exit 1 fi </ins>


Question:
07-02 and 13-02 No response No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
I might be related to the npd.
The same test failures happening again on 25.02:
/sig node
/sig node /cc The restart of caused the flake. I also opened to track it in NPD repo.
I missed this log. It is caused by oom. Adding some memory may help. I submitted to fix it. Does mean killing metrics-server?
/remove-sig node Sorry, I misread the log. The NPD found the metric server is OOM. It is about the metrics server, not the NPD. Sorry for disturb. I may add some memory for metric-server and check why metric server memory changed. I heard that metrics server recently released a new version. I will check if it is related.
/sig node
OOM metrics shows the metric server is OOMed. This is odd, that only metrics-server-nanny is with a memory limit. It uses whose command is . metrics-server container has no memory limit, but it got killed by OOM. Does this indicate that the node is OOM? (edited)
The last run failed in node-problem-detector ContainerStateTerminated. I0227 05:03:31. ] PodStartupLatency: perc50: 14m29.018886827s, perc90: 28m44.625518807s, perc99: 32m2.19303496s; threshold 1h0m0s I0227 05:03:32. ] Loaded restart count threshold overrides: map[coredns:2 fluentd-gcp:999 konnectivity-agent:2 kube-controller-manager:1 kube-proxy:2 kube-scheduler:1 l7-lb-controller:1 metadata-proxy:2 prometheus-to-sd-exporter:2] I0227 05:03:32. ] ClusterOOMsTracker: gathering cluster OOMs tracking measurement E0227 05:03:32. ] TestMetrics: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0] I0227 05:03:32. ] Step "[step: 29] gathering measurements" ended W0227 05:03:32. ] Got errors during step execution: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] I0227 05:03:32. ] Waiting for the chaos monkey subroutine to end... I0227 05:03:32. ] Chaos monkey ended. I0227 05:03:32. ] I0227 05:03:32. ] Stopping DnsLookupLatency probe... I0227 05:03:47. ] Stopping InClusterNetworkLatency probe... I0227 05:04:22. ] Resources cleanup time: 50.539700215s E0227 05:04:22. ]
E0227 05:04:22. ] Test Finished E0227 05:04:22. ] Test: E0227 05:04:22. ] Status: Fail E0227 05:04:22. ] Errors: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] E0227 05:04:22. ]
I0227 05:04:22. ]
I0227 05:04:22. ] Running testing/huge- I0227 05:04:22. ]

"regexp" "strings" "syscall" <ins> "github.com/golang/glog" </ins> ) const MOUNT_MS_BIND = syscall.MS_BIND


Question:
07-02 and 13-02 No response No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
I might be related to the npd.
The same test failures happening again on 25.02:
/sig node
/sig node /cc The restart of caused the flake. I also opened to track it in NPD repo.
I missed this log. It is caused by oom. Adding some memory may help. I submitted to fix it. Does mean killing metrics-server?
/remove-sig node Sorry, I misread the log. The NPD found the metric server is OOM. It is about the metrics server, not the NPD. Sorry for disturb. I may add some memory for metric-server and check why metric server memory changed. I heard that metrics server recently released a new version. I will check if it is related.
/sig node
OOM metrics shows the metric server is OOMed. This is odd, that only metrics-server-nanny is with a memory limit. It uses whose command is . metrics-server container has no memory limit, but it got killed by OOM. Does this indicate that the node is OOM? (edited)
The last run failed in node-problem-detector ContainerStateTerminated. I0227 05:03:31. ] PodStartupLatency: perc50: 14m29.018886827s, perc90: 28m44.625518807s, perc99: 32m2.19303496s; threshold 1h0m0s I0227 05:03:32. ] Loaded restart count threshold overrides: map[coredns:2 fluentd-gcp:999 konnectivity-agent:2 kube-controller-manager:1 kube-proxy:2 kube-scheduler:1 l7-lb-controller:1 metadata-proxy:2 prometheus-to-sd-exporter:2] I0227 05:03:32. ] ClusterOOMsTracker: gathering cluster OOMs tracking measurement E0227 05:03:32. ] TestMetrics: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0] I0227 05:03:32. ] Step "[step: 29] gathering measurements" ended W0227 05:03:32. ] Got errors during step execution: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] I0227 05:03:32. ] Waiting for the chaos monkey subroutine to end... I0227 05:03:32. ] Chaos monkey ended. I0227 05:03:32. ] I0227 05:03:32. ] Stopping DnsLookupLatency probe... I0227 05:03:47. ] Stopping InClusterNetworkLatency probe... I0227 05:04:22. ] Resources cleanup time: 50.539700215s E0227 05:04:22. ]
E0227 05:04:22. ] Test Finished E0227 05:04:22. ] Test: E0227 05:04:22. ] Status: Fail E0227 05:04:22. ] Errors: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] E0227 05:04:22. ]
I0227 05:04:22. ]
I0227 05:04:22. ] Running testing/huge- I0227 05:04:22. ]

// Wraps syscall.Mount() func (mounter *DiskMounter) Mount(source string, target string, fstype string, flags uintptr, data string) error { <ins> glog.V(5).Infof("Mounting %s %s %s %d %s", source, target, fstype, flags, data) </ins> return syscall.Mount(source, target, fstype, flags, data) }


Question:
Kubernetes 1.27 with feature gate enabled and disabled, a rebooted node is not able to re-start pods that were running there before the reboot. The node reports in Pod events: The Pod should start just fine. On Kubernetes 1.27: NewVolumeManagerReconstruction feature gate. a StatefulSet that uses a volume. -reboot the node which runs the StatefulSet Pod. The reason is that kubelet's VolumeManager reconstructs the volume as uncertain during startup and it uses a provisional outer volume name for it, because it has not seen the API server and Pods in it yet. This provisional volume ID is then used in ActualStateOfWorld forever. While kubelet does mount the volume after reboot, it's not able to match it with the Pod that needs it. Kubelet reports , while the volume is mounted with ID in my case (= the provisional name, we use for it) and not (=). It is not reproducible with enabled. <details</details<detailsAWS + AWS EBS CSI driver, but it should not be relevant. </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-soon This is reproducible when SELinuxMountReadWriteOncePod is disabled and NewVolumeManagerReconstruction enabled. They both got enabled by default as beta in 1.28. So someone has to explicitly disable SELinuxMountReadWriteOncePod to hit this bug.

// Update uncertain volumes - the new markVolumeOpts may have updated information. // Especially reconstructed volumes (marked as uncertain during reconstruction) need // an update. <del> updateUncertainVolume = utilfeature.DefaultFeatureGate.Enabled(features.SELinuxMountReadWriteOncePod) && podObj.volumeMountStateForPod == operationexecutor.VolumeMountUncertain </del> <ins> updateUncertainVolume = utilfeature.DefaultFeatureGate.Enabled(features.NewVolumeManagerReconstruction) && podObj.volumeMountStateForPod == operationexecutor.VolumeMountUncertain </ins> } if !podExists || updateUncertainVolume { // Add new mountedPod or update existing one.


Question:
Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted

utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/kubernetes" "k8s.io/client-go/transport" <ins> "k8s.io/component-base/tracing" </ins> "k8s.io/component-base/version" v1 "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1" v1helper "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1/helper"


Question:
Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted

// rejectForwardingRedirects is whether to allow to forward redirect response rejectForwardingRedirects bool <ins> // tracerProvider is used to wrap the proxy transport and handler with tracing tracerProvider tracing.TracerProvider </ins> } // Complete fills in any fields not set that are required to have valid data. It's mutating the receiver.


Question:
Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted

openAPIV3Config:            c.GenericConfig.OpenAPIV3Config, proxyCurrentCertKeyContent: func() (bytes []byte, bytes2 []byte) { return nil, nil }, rejectForwardingRedirects:  c.ExtraConfig.RejectForwardingRedirects, <ins> tracerProvider:             c.GenericConfig.TracerProvider, </ins> } // used later  to filter the served resource by those that have expired.


Question:
Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted

proxyTransportDial:         s.proxyTransportDial, serviceResolver:            s.serviceResolver, rejectForwardingRedirects:  s.rejectForwardingRedirects, <ins> tracerProvider:             s.tracerProvider, </ins> } proxyHandler.updateAPIService(apiService) if s.openAPIAggregationController != nil {


Question:
Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted

"k8s.io/apiserver/pkg/endpoints/handlers/responsewriters" endpointmetrics "k8s.io/apiserver/pkg/endpoints/metrics" genericapirequest "k8s.io/apiserver/pkg/endpoints/request" <ins> genericfeatures "k8s.io/apiserver/pkg/features" utilfeature "k8s.io/apiserver/pkg/util/feature" </ins> utilflowcontrol "k8s.io/apiserver/pkg/util/flowcontrol" apiserverproxyutil "k8s.io/apiserver/pkg/util/proxy" "k8s.io/apiserver/pkg/util/x509metrics" "k8s.io/client-go/transport" <ins> "k8s.io/component-base/tracing" </ins> "k8s.io/klog/v2" apiregistrationv1api "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1" apiregistrationv1apihelper "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1/helper"


Question:
Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted

// reject to forward redirect response rejectForwardingRedirects bool <ins> // tracerProvider is used to wrap the proxy transport and handler with tracing tracerProvider tracing.TracerProvider </ins> } type proxyHandlingInfo struct {


Question:
Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted

proxyRoundTripper = transport.NewAuthProxyRoundTripper(user.GetName(), user.GetGroups(), user.GetExtra(), proxyRoundTripper) <ins> if utilfeature.DefaultFeatureGate.Enabled(genericfeatures.APIServerTracing) && !upgrade { tracingWrapper := tracing.WrapperFor(r.tracerProvider) proxyRoundTripper = tracingWrapper(proxyRoundTripper) } </ins> // If we are upgrading, then the upgrade path tries to use this request with the TLS config we provide, but it does // NOT use the proxyRoundTripper.  It's a direct dial that bypasses the proxyRoundTripper.  This means that we have to // attach the "correct" user headers to the request ahead of time.


Question:
Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted

"golang.org/x/net/websocket" <ins> "go.opentelemetry.io/otel/propagation" sdktrace "go.opentelemetry.io/otel/sdk/trace" "go.opentelemetry.io/otel/sdk/trace/tracetest" "go.opentelemetry.io/otel/trace" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" utilnet "k8s.io/apimachinery/pkg/util/net" "k8s.io/apimachinery/pkg/util/proxy" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apiserver/pkg/authentication/user" <ins> "k8s.io/apiserver/pkg/endpoints/filters" </ins> genericapirequest "k8s.io/apiserver/pkg/endpoints/request" "k8s.io/apiserver/pkg/server/egressselector" utilflowcontrol "k8s.io/apiserver/pkg/util/flowcontrol"


Question:
Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted

"k8s.io/component-base/metrics/legacyregistry" apiregistration "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1" "k8s.io/utils/pointer" <ins> "k8s.io/utils/ptr" </ins> ) type targetHTTPHandler struct {


Question:
Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted

} <ins> func TestTracerProvider(t *testing.T) { fakeRecorder := tracetest.NewSpanRecorder() otelTracer := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(fakeRecorder)) target := &targetHTTPHandler{} user := &user.DefaultInfo{ Name:   "username", Groups: []string{"one", "two"}, } path := "/request/path" apiService := &apiregistration.APIService{ ObjectMeta: metav1.ObjectMeta{Name: "v1.foo"}, Spec: apiregistration.APIServiceSpec{ Service:  &apiregistration.ServiceReference{Name: "test-service", Namespace: "test-ns", Port: ptr.To(int32(443))}, Group:    "foo", Version:  "v1", CABundle: testCACrt, }, Status: apiregistration.APIServiceStatus{ Conditions: []apiregistration.APIServiceCondition{ {Type: apiregistration.Available, Status: apiregistration.ConditionTrue}, }, }, } targetServer := httptest.NewUnstartedServer(target) serviceCert := svcCrt if cert, err := tls.X509KeyPair(serviceCert, svcKey); err != nil { t.Fatalf("TestTracerProvider: failed to parse key pair: %v", err) } else { targetServer.TLS = &tls.Config{Certificates: []tls.Certificate{cert}} } targetServer.StartTLS() defer targetServer.Close() serviceResolver := &mockedRouter{destinationHost: targetServer.Listener.Addr().String()} handler := &proxyHandler{ localDelegate:              http.NewServeMux(), serviceResolver:            serviceResolver, proxyCurrentCertKeyContent: func() ([]byte, []byte) { return emptyCert(), emptyCert() }, tracerProvider:             otelTracer, } server := httptest.NewServer(contextHandler(filters.WithTracing(handler, otelTracer), user)) defer server.Close() handler.updateAPIService(apiService) curr := handler.handlingInfo.Load().(proxyHandlingInfo) handler.handlingInfo.Store(curr) var propagator propagation.TraceContext req, err := http.NewRequest(http.MethodGet, server.URL+path, nil) if err != nil { t.Errorf("expected new request: %v", err) return } t.Logf("Sending request: %v", req) _, err = http.DefaultClient.Do(req) if err != nil { t.Errorf("http request failed: %v", err) return } t.Log("Ensure the target server received the traceparent header") id, ok := target.headers["Traceparent"] if !ok { t.Error("expected traceparent header") return } t.Log("Get the span context from the traceparent header") h := http.Header{ "Traceparent": id, } ctx := propagator.Extract(context.Background(), propagation.HeaderCarrier(h)) span := trace.SpanFromContext(ctx) t.Log("Ensure that the span context is valid and remote") if !span.SpanContext().IsValid() { t.Error("expected valid span context") return } if !span.SpanContext().IsRemote() { t.Error("expected remote span context") return } t.Log("Ensure that the span ID and trace ID match the expected values") expectedSpanCtx := fakeRecorder.Ended()[0].SpanContext() if expectedSpanCtx.TraceID() != span.SpanContext().TraceID() { t.Errorf("expected trace id to match. expected: %v, but got %v", expectedSpanCtx.TraceID(), span.SpanContext().TraceID()) return } if expectedSpanCtx.SpanID() != span.SpanContext().SpanID() { t.Errorf("expected span id to match. expected: %v, but got: %v", expectedSpanCtx.SpanID(), span.SpanContext().SpanID()) return } t.Log("Ensure that the expected spans were recorded when sending a request through the proxy") expectedSpanNames := []string{"HTTP GET", "GET"} spanNames := []string{} for _, span := range fakeRecorder.Ended() { spanNames = append(spanNames, span.Name()) } if e, a := expectedSpanNames, spanNames; !reflect.DeepEqual(e, a) { t.Errorf("expected span names %v, got %v", e, a) return } } </ins> func TestNewRequestForProxyWithAuditID(t *testing.T) { tests := []struct { name    string


Question:
We use a to nil out the : However on a re-sync, this results in a data race that was observed when running unit tests: The read had occurred previously during a callback. On re-sync, the retrieves every object from the cache, invokes the transform function, and re-queues them. However, the object instances are the same as those stored in the cache so mutating them outside the cache without lock protection is unsafe. It is documented that the "TransformFunc sees the object before any other actor, and it is now safe to mutate the object in place instead of making a copy" however this is not the case on a re-sync. The should make a copy of the objects retrieved from the cache store on a re-sync or perhaps elide the on a re-sync since the objects would already have been transformed. As a workaround, I changed our to only nil the if not already nil but this is a latent bug that others will likely run into.
/assign That means that and introduced possible panic regressions into kube-controller-manager and kube-scheduler /priority critical-urgent /kind regression
/transfer kubernetes
/sig api-machinery
kube-controller-manager and kube-scheduler do the same in 1.29+ and likely have the same issue
I think delete has the same problem, and replace may have the same problem: Passing the same object to the transformer after it is already in the cache is unsafe and we should not do that at all.
/assign /cc
Looking into this. I may take time to come up with a proper fix. But would like to evaluate the risk for the current situation. Not entirely sure how delete and replace would be triggered in normal settings, but resync is fortunately low frequency as the default setting for resync period is between 12h and 24h for (--min-resync-period duration Default: 12h0m0s, see ) . As for the resync function is Just think out aloud, I read a thread discussion somewhere that resync was a mechanism in the past to prevent data inconsistency due to bugs but client-go has been mature these days, resync is no-longer needed in theory but for a safe guard. So maybe disabling resync is a reasonable fix if delete/replace can be fixed.
You're mixing resync with relist - I believe relist is no longer needed. Resync is actually used. FWIW - I opened an initial version of the fix:
I think it's resync. The thread I mentioned is
/triage accepted

"fmt" "math/rand" "sync" <ins> "sync/atomic" </ins> "testing" "time"


Question:
We use a to nil out the : However on a re-sync, this results in a data race that was observed when running unit tests: The read had occurred previously during a callback. On re-sync, the retrieves every object from the cache, invokes the transform function, and re-queues them. However, the object instances are the same as those stored in the cache so mutating them outside the cache without lock protection is unsafe. It is documented that the "TransformFunc sees the object before any other actor, and it is now safe to mutate the object in place instead of making a copy" however this is not the case on a re-sync. The should make a copy of the objects retrieved from the cache store on a re-sync or perhaps elide the on a re-sync since the objects would already have been transformed. As a workaround, I changed our to only nil the if not already nil but this is a latent bug that others will likely run into.
/assign That means that and introduced possible panic regressions into kube-controller-manager and kube-scheduler /priority critical-urgent /kind regression
/transfer kubernetes
/sig api-machinery
kube-controller-manager and kube-scheduler do the same in 1.29+ and likely have the same issue
I think delete has the same problem, and replace may have the same problem: Passing the same object to the transformer after it is already in the cache is unsafe and we should not do that at all.
/assign /cc
Looking into this. I may take time to come up with a proper fix. But would like to evaluate the risk for the current situation. Not entirely sure how delete and replace would be triggered in normal settings, but resync is fortunately low frequency as the default setting for resync period is between 12h and 24h for (--min-resync-period duration Default: 12h0m0s, see ) . As for the resync function is Just think out aloud, I read a thread discussion somewhere that resync was a mechanism in the past to prevent data inconsistency due to bugs but client-go has been mature these days, resync is no-longer needed in theory but for a safe guard. So maybe disabling resync is a reasonable fix if delete/replace can be fixed.
You're mixing resync with relist - I believe relist is no longer needed. Resync is actually used. FWIW - I opened an initial version of the fix:
I think it's resync. The thread I mentioned is
/triage accepted

close(stopCh) } <ins> func TestTransformingInformerRace(t *testing.T) { // source simulates an apiserver object endpoint. source := fcache.NewFakeControllerSource() label := "to-be-transformed" makePod := func(name string) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      name, Namespace: "namespace", Labels:    map[string]string{label: "true"}, }, Spec: v1.PodSpec{ Hostname: "hostname", }, } } badTransform := atomic.Bool{} podTransformer := func(obj interface{}) (interface{}, error) { pod, ok := obj.(*v1.Pod) if !ok { return nil, fmt.Errorf("unexpected object type: %T", obj) } if pod.ObjectMeta.Labels[label] != "true" { badTransform.Store(true) return nil, fmt.Errorf("object already transformed: %#v", obj) } pod.ObjectMeta.Labels[label] = "false" return pod, nil } numObjs := 5 for i := 0; i < numObjs; i++ { source.Add(makePod(fmt.Sprintf("pod-%d", i))) } type event struct{} events := make(chan event, numObjs) recordEvent := func(eventType watch.EventType, previous, current interface{}) { events <- event{} } checkEvents := func(count int) { for i := 0; i < count; i++ { <-events } } store, controller := NewTransformingInformer( source, &v1.Pod{}, 5*time.Millisecond, ResourceEventHandlerDetailedFuncs{ AddFunc:    func(obj interface{}, isInInitialList bool) { recordEvent(watch.Added, nil, obj) }, UpdateFunc: func(oldObj, newObj interface{}) { recordEvent(watch.Modified, oldObj, newObj) }, DeleteFunc: func(obj interface{}) { recordEvent(watch.Deleted, obj, nil) }, }, podTransformer, ) stopCh := make(chan struct{}) go controller.Run(stopCh) checkEvents(numObjs) // Periodically fetch objects to ensure no access races. wg := sync.WaitGroup{} errors := make(chan error, numObjs) for i := 0; i < numObjs; i++ { wg.Add(1) go func(index int) { defer wg.Done() key := fmt.Sprintf("namespace/pod-%d", index) for { select { case <-stopCh: return default: } obj, ok, err := store.GetByKey(key) if !ok || err != nil { errors <- fmt.Errorf("couldn't get the object for %v", key) return } pod := obj.(*v1.Pod) if pod.ObjectMeta.Labels[label] != "false" { errors <- fmt.Errorf("unexpected object: %#v", pod) return } } }(i) } // Let resyncs to happen for some time. time.Sleep(time.Second) close(stopCh) wg.Wait() close(errors) for err := range errors { t.Error(err) } if badTransform.Load() { t.Errorf("unexpected transformation happened") } } </ins> func TestDeletionHandlingObjectToName(t *testing.T) { cm := &v1.ConfigMap{ ObjectMeta: metav1.ObjectMeta{


Question:
We use a to nil out the : However on a re-sync, this results in a data race that was observed when running unit tests: The read had occurred previously during a callback. On re-sync, the retrieves every object from the cache, invokes the transform function, and re-queues them. However, the object instances are the same as those stored in the cache so mutating them outside the cache without lock protection is unsafe. It is documented that the "TransformFunc sees the object before any other actor, and it is now safe to mutate the object in place instead of making a copy" however this is not the case on a re-sync. The should make a copy of the objects retrieved from the cache store on a re-sync or perhaps elide the on a re-sync since the objects would already have been transformed. As a workaround, I changed our to only nil the if not already nil but this is a latent bug that others will likely run into.
/assign That means that and introduced possible panic regressions into kube-controller-manager and kube-scheduler /priority critical-urgent /kind regression
/transfer kubernetes
/sig api-machinery
kube-controller-manager and kube-scheduler do the same in 1.29+ and likely have the same issue
I think delete has the same problem, and replace may have the same problem: Passing the same object to the transformer after it is already in the cache is unsafe and we should not do that at all.
/assign /cc
Looking into this. I may take time to come up with a proper fix. But would like to evaluate the risk for the current situation. Not entirely sure how delete and replace would be triggered in normal settings, but resync is fortunately low frequency as the default setting for resync period is between 12h and 24h for (--min-resync-period duration Default: 12h0m0s, see ) . As for the resync function is Just think out aloud, I read a thread discussion somewhere that resync was a mechanism in the past to prevent data inconsistency due to bugs but client-go has been mature these days, resync is no-longer needed in theory but for a safe guard. So maybe disabling resync is a reasonable fix if delete/replace can be fixed.
You're mixing resync with relist - I believe relist is no longer needed. Resync is actually used. FWIW - I opened an initial version of the fix:
I think it's resync. The thread I mentioned is
/triage accepted

} // TransformFunc allows for transforming an object before it will be processed. <del> // TransformFunc (similarly to ResourceEventHandler functions) should be able // to correctly handle the tombstone of type cache.DeletedFinalStateUnknown. // // New in v1.27: In such cases, the contained object will already have gone // through the transform object separately (when it was added / updated prior // to the delete), so the TransformFunc can likely safely ignore such objects // (i.e., just return the input object). </del> // // The most common usage pattern is to clean-up some parts of the object to // reduce component memory usage if a given component doesn't care about them. // <del> // New in v1.27: unless the object is a DeletedFinalStateUnknown, TransformFunc // sees the object before any other actor, and it is now safe to mutate the // object in place instead of making a copy. </del> <ins> // New in v1.27: TransformFunc sees the object before any other actor, and it // is now safe to mutate the object in place instead of making a copy. // // It's recommended for the TransformFunc to be idempotent. // It MUST be idempotent if objects already present in the cache are passed to // the Replace() to avoid re-mutating them. Default informers do not pass // existing objects to Replace though. </ins> // // Note that TransformFunc is called while inserting objects into the // notification queue and is therefore extremely performance sensitive; please


Question:
We use a to nil out the : However on a re-sync, this results in a data race that was observed when running unit tests: The read had occurred previously during a callback. On re-sync, the retrieves every object from the cache, invokes the transform function, and re-queues them. However, the object instances are the same as those stored in the cache so mutating them outside the cache without lock protection is unsafe. It is documented that the "TransformFunc sees the object before any other actor, and it is now safe to mutate the object in place instead of making a copy" however this is not the case on a re-sync. The should make a copy of the objects retrieved from the cache store on a re-sync or perhaps elide the on a re-sync since the objects would already have been transformed. As a workaround, I changed our to only nil the if not already nil but this is a latent bug that others will likely run into.
/assign That means that and introduced possible panic regressions into kube-controller-manager and kube-scheduler /priority critical-urgent /kind regression
/transfer kubernetes
/sig api-machinery
kube-controller-manager and kube-scheduler do the same in 1.29+ and likely have the same issue
I think delete has the same problem, and replace may have the same problem: Passing the same object to the transformer after it is already in the cache is unsafe and we should not do that at all.
/assign /cc
Looking into this. I may take time to come up with a proper fix. But would like to evaluate the risk for the current situation. Not entirely sure how delete and replace would be triggered in normal settings, but resync is fortunately low frequency as the default setting for resync period is between 12h and 24h for (--min-resync-period duration Default: 12h0m0s, see ) . As for the resync function is Just think out aloud, I read a thread discussion somewhere that resync was a mechanism in the past to prevent data inconsistency due to bugs but client-go has been mature these days, resync is no-longer needed in theory but for a safe guard. So maybe disabling resync is a reasonable fix if delete/replace can be fixed.
You're mixing resync with relist - I believe relist is no longer needed. Resync is actually used. FWIW - I opened an initial version of the fix:
I think it's resync. The thread I mentioned is
/triage accepted

// queueActionLocked appends to the delta list for the object. // Caller must lock first. func (f *DeltaFIFO) queueActionLocked(actionType DeltaType, obj interface{}) error { <ins> return f.queueActionInternalLocked(actionType, actionType, obj) } // queueActionInternalLocked appends to the delta list for the object. // The actionType is emitted and must honor emitDeltaTypeReplaced. // The internalActionType is only used within this function and must // ignore emitDeltaTypeReplaced. // Caller must lock first. func (f *DeltaFIFO) queueActionInternalLocked(actionType, internalActionType DeltaType, obj interface{}) error { </ins> id, err := f.KeyOf(obj) if err != nil { return KeyError{obj, err} } // Every object comes through this code path once, so this is a good <del> // place to call the transform func.  If obj is a // DeletedFinalStateUnknown tombstone, then the containted inner object // will already have gone through the transformer, but we document that // this can happen. In cases involving Replace(), such an object can // come through multiple times. </del> <ins> // place to call the transform func. // // If obj is a DeletedFinalStateUnknown tombstone or the action is a Sync, // then the object have already gone through the transformer. // // If the objects already present in the cache are passed to Replace(), // the transformer must be idempotent to avoid re-mutating them, // or coordinate with all readers from the cache to avoid data races. // Default informers do not pass existing objects to Replace. </ins> if f.transformer != nil { <del> var err error obj, err = f.transformer(obj) if err != nil { return err </del> <ins> _, isTombstone := obj.(DeletedFinalStateUnknown) if !isTombstone && internalActionType != Sync { var err error obj, err = f.transformer(obj) if err != nil { return err } </ins> } }


Question:
We use a to nil out the : However on a re-sync, this results in a data race that was observed when running unit tests: The read had occurred previously during a callback. On re-sync, the retrieves every object from the cache, invokes the transform function, and re-queues them. However, the object instances are the same as those stored in the cache so mutating them outside the cache without lock protection is unsafe. It is documented that the "TransformFunc sees the object before any other actor, and it is now safe to mutate the object in place instead of making a copy" however this is not the case on a re-sync. The should make a copy of the objects retrieved from the cache store on a re-sync or perhaps elide the on a re-sync since the objects would already have been transformed. As a workaround, I changed our to only nil the if not already nil but this is a latent bug that others will likely run into.
/assign That means that and introduced possible panic regressions into kube-controller-manager and kube-scheduler /priority critical-urgent /kind regression
/transfer kubernetes
/sig api-machinery
kube-controller-manager and kube-scheduler do the same in 1.29+ and likely have the same issue
I think delete has the same problem, and replace may have the same problem: Passing the same object to the transformer after it is already in the cache is unsafe and we should not do that at all.
/assign /cc
Looking into this. I may take time to come up with a proper fix. But would like to evaluate the risk for the current situation. Not entirely sure how delete and replace would be triggered in normal settings, but resync is fortunately low frequency as the default setting for resync period is between 12h and 24h for (--min-resync-period duration Default: 12h0m0s, see ) . As for the resync function is Just think out aloud, I read a thread discussion somewhere that resync was a mechanism in the past to prevent data inconsistency due to bugs but client-go has been mature these days, resync is no-longer needed in theory but for a safe guard. So maybe disabling resync is a reasonable fix if delete/replace can be fixed.
You're mixing resync with relist - I believe relist is no longer needed. Resync is actually used. FWIW - I opened an initial version of the fix:
I think it's resync. The thread I mentioned is
/triage accepted

return KeyError{item, err} } keys.Insert(key) <del> if err := f.queueActionLocked(action, item); err != nil { </del> <ins> if err := f.queueActionInternalLocked(action, Replaced, item); err != nil { </ins> return fmt.Errorf("couldn't enqueue object: %v", err) } }


Question:
The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows

return fmt.Errorf("failed to try resolving symlinks in path %q: %v", path, err) } path = evaluated <del> f, err := os.Open(path) </del> <ins> f, err := openFileShareDelete(path) </ins> if err != nil { return fmt.Errorf("failed to open log file %q: %v", path, err) }


Question:
The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows

return err } if recreated { <del> newF, err := os.Open(path) </del> <ins> newF, err := openFileShareDelete(path) </ins> if err != nil { if os.IsNotExist(err) { continue


Question:
The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows

<ins> //go:build !windows // +build !windows /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package logs import ( "os" ) func openFileShareDelete(path string) (*os.File, error) { // Noop. Only relevant for Windows. return os.Open(path) } </ins>


Question:
The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows

"io" "os" "path/filepath" <del> goruntime "runtime" </del> "testing" "time"


Question:
The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows

} func TestReadRotatedLog(t *testing.T) { <del> if goruntime.GOOS == "windows" { // TODO: remove skip once the failing test has been fixed. t.Skip("Skip failing test on Windows.") } </del> tmpDir := t.TempDir() file, err := os.CreateTemp(tmpDir, "logfile") if err != nil {