Question:
an E2E test that assumes that a service named "kube-dns" exists in the "kube-system" namespace. This is a kubeadm-ism, and the test fails if the cluster DNS is not configured the way kubeadm configures it. Although this E2E test isn't a conformance test, making it functional on non-kubeadm clusters would increase its utility for distributors that may wish to run this E2E test as a conformance test. [sig-network] DNS HostNetwork should resolve DNS of partial qualified names for services on hostNetwork pods with dnsPolicy: ClusterFirstWithHostNet Since we ran it on an OpenShift cluster. No response The test makes assumptions about how the cluster DNS is configured. The test uses (not ), so it does not need to specify , as far as I can tell. Omitting obviates the need to look up the service. I believe that the service lookup was left in as an oversight during the development of the test when it was changed from specifying "None" to specifying "ClusterFirst": Cc: /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
responded on the pr.
It seems kubeadm inherited the status-quo, I can't find any place that mandates the Service name for DNS Asking for double checking, but the docs seems to mention that this is more a recommendation than a preescription If it seems that the test can be done in a more portable way we should do it

pod := createDNSPod(f.Namespace.Name, wheezyProbeCmd, jessieProbeCmd, dnsTestPodHostName, dnsTestServiceName) pod.Spec.DNSPolicy = v1.DNSClusterFirst pod.Spec.DNSConfig = &v1.PodDNSConfig{ <del> Nameservers: []string{kubednsIP}, Searches:    testSearchPaths, </del> <ins> Searches: testSearchPaths, </ins> Options: []v1.PodDNSConfigOption{ { Name:  "ndots",


Question:
In , we discovered that our plan for eventually removing the scheduling plugin when CSI migration is GA is not possible because the plugin also applies to non-migratable volume types. However, if we leave the plugin as is, then even migratable PVs will be broken if we remove beta topology labels from Nodes in the future. This is because the plugin does not have any handling for migrated volumes. I suggest we add CSI migration handling to the VolumeZone plugin. I think we just need to detect if the volume is migrated, and if so, skip the VolumeZone processing. For migrated volumes, we translate into so they end up getting processed by the plugin. In a cluster with csi migration for a specific intree volume type enabled, an intree PV with beta topology labels the beta topology label from a node to schedule a pod using that pv No response <details</details<detailsAny that supports csi migration </details<details</details<details</details<details</details<details</details
/sig storage /triage accepted /priority important-soon For e2e testing, this needs an environment that is configured with csi migration
Hi,I'd like to work on this issue. I previously wrote for plugin, so I'm familiar with it's logic. However I'm not familiar with the whole thing, and I may ask you some questions during the work :( Can I try this one? Here are some questions ahead: I think what you want to do is to schedule a pv with old labels to a node with only new labels. And you think we can translate into in volumezone during (maybe), and update this pv so that in stage, can , so we don't have compare Topology labels in . My question is: Can't we just translate this for all pvs? Why do we need to detect if the volume is migrated? What's the difference between a migrated pv and a non-migratable pv in this scenario?
I think this is also another alternative solution that could also work, but from , it may not scalable to do an automatic migration for all the different APIs and controllers that use the node labels. The main difference with CSI migration, is we have a good idea of all the different components that would be impacted and also have reasonable control over those components.
A few more questions Will the volumes be migrated before we take a pod off the scheduler queue? If so, how can I detect this. Currently I find a function in , but nothing like . Is this translation done in plugin or before scheduling? I think currently in , we consider topology label and beta label as different labels. That is ,if we have a pv with and a node with , even if they have same values, will let the pod pass filter. If we're going to remove beta labels on node,maybe we should try to consider two labels as the same in ? Here's my understanding: For migrated volumes, we translate Topology labels to pv.NodeAffinity and update the pv, then we skip volume_zone(that is,don't compare labels with nodes). What about non-migratable volumes, I think we should continue to compare with node labels for non-migratable volumes?
For detecting if a PV is migrated, you can use the same method that's used in the VolumeBinding plugin: The only difference is you don't actually need to translate the PV in the VolumeZone plugin since it's already handled by the VolumeBinding plugin. For non-migratable volumes, I think we can continue with the same VolumeZone logic. For now, we'll rely on users to update their PVs with the GA labels.
Thanks ,for your detailed explanation. This is the implementation plan I prepared: In , we check if a pv is migrtable by using ,if so we skip volumezone since will translate this pv and handle it later, otherwise we save the label of this pv in state,and compare it with node's label in .Is that feasible to you?
Yup that sounds good to me!
Thanks! I'll try to work on this one. I'll file a pr a few days later and we can talk about the details in there. In the meantime, I think I previously wrote a bug in plugin, I wonder if you have time to confirm it /assign
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted

v1.LabelTopologyRegion, } <ins> func translateToGALabel(label string) string { if label == v1.LabelFailureDomainBetaRegion { return v1.LabelTopologyRegion } if label == v1.LabelFailureDomainBetaZone { return v1.LabelTopologyZone } return label } </ins> // Name returns name of the plugin. It is used in logs, etc. func (pl *VolumeZone) Name() string { return Name


Question:
In , we discovered that our plan for eventually removing the scheduling plugin when CSI migration is GA is not possible because the plugin also applies to non-migratable volume types. However, if we leave the plugin as is, then even migratable PVs will be broken if we remove beta topology labels from Nodes in the future. This is because the plugin does not have any handling for migrated volumes. I suggest we add CSI migration handling to the VolumeZone plugin. I think we just need to detect if the volume is migrated, and if so, skip the VolumeZone processing. For migrated volumes, we translate into so they end up getting processed by the plugin. In a cluster with csi migration for a specific intree volume type enabled, an intree PV with beta topology labels the beta topology label from a node to schedule a pod using that pv No response <details</details<detailsAny that supports csi migration </details<details</details<details</details<details</details<details</details
/sig storage /triage accepted /priority important-soon For e2e testing, this needs an environment that is configured with csi migration
Hi,I'd like to work on this issue. I previously wrote for plugin, so I'm familiar with it's logic. However I'm not familiar with the whole thing, and I may ask you some questions during the work :( Can I try this one? Here are some questions ahead: I think what you want to do is to schedule a pv with old labels to a node with only new labels. And you think we can translate into in volumezone during (maybe), and update this pv so that in stage, can , so we don't have compare Topology labels in . My question is: Can't we just translate this for all pvs? Why do we need to detect if the volume is migrated? What's the difference between a migrated pv and a non-migratable pv in this scenario?
I think this is also another alternative solution that could also work, but from , it may not scalable to do an automatic migration for all the different APIs and controllers that use the node labels. The main difference with CSI migration, is we have a good idea of all the different components that would be impacted and also have reasonable control over those components.
A few more questions Will the volumes be migrated before we take a pod off the scheduler queue? If so, how can I detect this. Currently I find a function in , but nothing like . Is this translation done in plugin or before scheduling? I think currently in , we consider topology label and beta label as different labels. That is ,if we have a pv with and a node with , even if they have same values, will let the pod pass filter. If we're going to remove beta labels on node,maybe we should try to consider two labels as the same in ? Here's my understanding: For migrated volumes, we translate Topology labels to pv.NodeAffinity and update the pv, then we skip volume_zone(that is,don't compare labels with nodes). What about non-migratable volumes, I think we should continue to compare with node labels for non-migratable volumes?
For detecting if a PV is migrated, you can use the same method that's used in the VolumeBinding plugin: The only difference is you don't actually need to translate the PV in the VolumeZone plugin since it's already handled by the VolumeBinding plugin. For non-migratable volumes, I think we can continue with the same VolumeZone logic. For now, we'll rely on users to update their PVs with the GA labels.
Thanks ,for your detailed explanation. This is the implementation plan I prepared: In , we check if a pv is migrtable by using ,if so we skip volumezone since will translate this pv and handle it later, otherwise we save the label of this pv in state,and compare it with node's label in .Is that feasible to you?
Yup that sounds good to me!
Thanks! I'll try to work on this one. I'll file a pr a few days later and we can talk about the details in there. In the meantime, I think I previously wrote a bug in plugin, I wonder if you have time to confirm it /assign
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted

for _, pvTopology := range podPVTopologies { v, ok := node.Labels[pvTopology.key] <ins> if !ok { // if we can't match the beta label, try to match pv's beta label with node's ga label v, ok = node.Labels[translateToGALabel(pvTopology.key)] } </ins> if !ok || !pvTopology.values.Has(v) { logger.V(10).Info("Won't schedule pod onto node due to volume (mismatch on label key)", "pod", klog.KObj(pod), "node", klog.KObj(node), "PV", klog.KRef("", pvTopology.pvName), "PVLabelKey", pvTopology.key) return framework.NewStatus(framework.UnschedulableAndUnresolvable, ErrReasonConflict)


Question:
In , we discovered that our plan for eventually removing the scheduling plugin when CSI migration is GA is not possible because the plugin also applies to non-migratable volume types. However, if we leave the plugin as is, then even migratable PVs will be broken if we remove beta topology labels from Nodes in the future. This is because the plugin does not have any handling for migrated volumes. I suggest we add CSI migration handling to the VolumeZone plugin. I think we just need to detect if the volume is migrated, and if so, skip the VolumeZone processing. For migrated volumes, we translate into so they end up getting processed by the plugin. In a cluster with csi migration for a specific intree volume type enabled, an intree PV with beta topology labels the beta topology label from a node to schedule a pod using that pv No response <details</details<detailsAny that supports csi migration </details<details</details<details</details<details</details<details</details
/sig storage /triage accepted /priority important-soon For e2e testing, this needs an environment that is configured with csi migration
Hi,I'd like to work on this issue. I previously wrote for plugin, so I'm familiar with it's logic. However I'm not familiar with the whole thing, and I may ask you some questions during the work :( Can I try this one? Here are some questions ahead: I think what you want to do is to schedule a pv with old labels to a node with only new labels. And you think we can translate into in volumezone during (maybe), and update this pv so that in stage, can , so we don't have compare Topology labels in . My question is: Can't we just translate this for all pvs? Why do we need to detect if the volume is migrated? What's the difference between a migrated pv and a non-migratable pv in this scenario?
I think this is also another alternative solution that could also work, but from , it may not scalable to do an automatic migration for all the different APIs and controllers that use the node labels. The main difference with CSI migration, is we have a good idea of all the different components that would be impacted and also have reasonable control over those components.
A few more questions Will the volumes be migrated before we take a pod off the scheduler queue? If so, how can I detect this. Currently I find a function in , but nothing like . Is this translation done in plugin or before scheduling? I think currently in , we consider topology label and beta label as different labels. That is ,if we have a pv with and a node with , even if they have same values, will let the pod pass filter. If we're going to remove beta labels on node,maybe we should try to consider two labels as the same in ? Here's my understanding: For migrated volumes, we translate Topology labels to pv.NodeAffinity and update the pv, then we skip volume_zone(that is,don't compare labels with nodes). What about non-migratable volumes, I think we should continue to compare with node labels for non-migratable volumes?
For detecting if a PV is migrated, you can use the same method that's used in the VolumeBinding plugin: The only difference is you don't actually need to translate the PV in the VolumeZone plugin since it's already handled by the VolumeBinding plugin. For non-migratable volumes, I think we can continue with the same VolumeZone logic. For now, we'll rely on users to update their PVs with the GA labels.
Thanks ,for your detailed explanation. This is the implementation plan I prepared: In , we check if a pv is migrtable by using ,if so we skip volumezone since will translate this pv and handle it later, otherwise we save the label of this pv in state,and compare it with node's label in .Is that feasible to you?
Yup that sounds good to me!
Thanks! I'll try to work on this one. I'll file a pr a few days later and we can talk about the details in there. In the meantime, I think I previously wrote a bug in plugin, I wonder if you have time to confirm it /assign
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted

}, wantFilterStatus: framework.NewStatus(framework.UnschedulableAndUnresolvable, ErrReasonConflict), }, <ins> { name: "pv with beta label,node with ga label, matched", Pod:  createPodWithVolume("pod_1", "Vol_1", "PVC_1"), Node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name: "host1", Labels: map[string]string{ v1.LabelTopologyZone: "us-west1-a", }, }, }, }, { name: "pv with beta label,node with ga label, don't match", Pod:  createPodWithVolume("pod_1", "vol_1", "PVC_1"), Node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name: "host1", Labels: map[string]string{ v1.LabelTopologyZone: "us-west1-b", }, }, }, wantFilterStatus: framework.NewStatus(framework.UnschedulableAndUnresolvable, ErrReasonConflict), }, </ins> } for _, test := range tests {


Question:
In a cluster with large enough numbers of objects, it seems to be impossible to clean them up with . Because the deletecollection is failing while listing the objects from etcd. The failure happens after exactly 34 seconds, due to the 34-second max timeout introduced in PR (cc for insight on that PR). The deletecollection should be able to run for longer, and succeed. I recognise that there are still risks of the API server using too much RAM, if deleting a very large collection due to issue . But, until that issue is fixed, it seems like allowing to run for longer might be a good idea. As far as I can tell, the 34 second timeout does not apply to normal LIST calls, so on that basis I feel it should not apply to the giant unchunked LIST call that happens inside . a few hundred thousand objects of some particular type. (Argo workflowtaskresults happen to be the type I'm looking at while I write this) deletecollection. (Optionally passing some kind of fieldSelector, but I don't think that's essential) that a 504 failure code is logged in kube-audit logs, and that a log entry similar to this is logged by the API Server (In practice, the total time may be a little over ms). far as I can tell, the delete collection really has failed at this point. I notice that PR says that, in general, "The go-routine can keep running, while the timeout logic will return a timeout to the client" but I don't think that's what's happening here. I think that, because the 34 sec timeout is used in the context of the etcd call, the etcd list call really does fail, and so the code never proceeds to actually deleting anything. No response 22 AKS <details</details<details</details<details</details<details</details
/sig scalability
/cc
the deadline has been in effect before PR was introduced If we remove this deadline from here, then the deletecollection request will have at most to complete (default). The list call that is failing in your case, is a direct call to the storage, right? on the contrary, we don't enforce the timeout for LIST thinking out loud here: I don't know if it makes sense to remove the from should we allow timeout for DELETECOLLECTION to exceed ? thoughts?
The list call that is failing in your case, is a direct call to the storage, right? It reads from etcd, not from API Server Cache. Is that what you were asking Looking at the bigger picture, I think your question about 60s timeout is a good one. Especially in the context of this: In particular, changing the timeouts won't solve all of the problems with DELETECOLLECTION. My assumption, when logging this issue, is that it might be a quick way to solve just some of them.
I think we should remove this - it's effectively a regression that we introduced with the PR. That's a good question, and I'm not entirely sure. Maybe we should, but I would do that separately. In the meantime, I would much more prefer to switch the DeleteCollection implementation to internally use pagination, as it would allow us to 1) not block on full list (and as a result make some progress) 2) would be some preparation towards potentially exposing a Limit option in the API... /kind regression
/sig api-machinery
/triage accepted
I suggest the following: - can you please take care of reverting the 34s timeout for DeleteCollection calls? this is done, let's close this issue and continue operating on
sounds good, thanks! /assign

return } <del> // enforce a timeout of at most requestTimeoutUpperBound (34s) or less if the user-provided // timeout inside the parent context is lower than requestTimeoutUpperBound. ctx, cancel := context.WithTimeout(ctx, requestTimeoutUpperBound) defer cancel() </del> <ins> // DELETECOLLECTION can be a lengthy operation, // we should not impose any 34s timeout here. // NOTE: This is similar to LIST which does not enforce a 34s timeout. </ins> ctx = request.WithNamespace(ctx, namespace) outputMediaType, _, err := negotiation.NegotiateOutputMediaType(req, scope.Serializer, scope)


Question:
In a cluster with large enough numbers of objects, it seems to be impossible to clean them up with . Because the deletecollection is failing while listing the objects from etcd. The failure happens after exactly 34 seconds, due to the 34-second max timeout introduced in PR (cc for insight on that PR). The deletecollection should be able to run for longer, and succeed. I recognise that there are still risks of the API server using too much RAM, if deleting a very large collection due to issue . But, until that issue is fixed, it seems like allowing to run for longer might be a good idea. As far as I can tell, the 34 second timeout does not apply to normal LIST calls, so on that basis I feel it should not apply to the giant unchunked LIST call that happens inside . a few hundred thousand objects of some particular type. (Argo workflowtaskresults happen to be the type I'm looking at while I write this) deletecollection. (Optionally passing some kind of fieldSelector, but I don't think that's essential) that a 504 failure code is logged in kube-audit logs, and that a log entry similar to this is logged by the API Server (In practice, the total time may be a little over ms). far as I can tell, the delete collection really has failed at this point. I notice that PR says that, in general, "The go-routine can keep running, while the timeout logic will return a timeout to the client" but I don't think that's what's happening here. I think that, because the 34 sec timeout is used in the context of the etcd call, the etcd list call really does fail, and so the code never proceeds to actually deleting anything. No response 22 AKS <details</details<details</details<details</details<details</details
/sig scalability
/cc
the deadline has been in effect before PR was introduced If we remove this deadline from here, then the deletecollection request will have at most to complete (default). The list call that is failing in your case, is a direct call to the storage, right? on the contrary, we don't enforce the timeout for LIST thinking out loud here: I don't know if it makes sense to remove the from should we allow timeout for DELETECOLLECTION to exceed ? thoughts?
The list call that is failing in your case, is a direct call to the storage, right? It reads from etcd, not from API Server Cache. Is that what you were asking Looking at the bigger picture, I think your question about 60s timeout is a good one. Especially in the context of this: In particular, changing the timeouts won't solve all of the problems with DELETECOLLECTION. My assumption, when logging this issue, is that it might be a quick way to solve just some of them.
I think we should remove this - it's effectively a regression that we introduced with the PR. That's a good question, and I'm not entirely sure. Maybe we should, but I would do that separately. In the meantime, I would much more prefer to switch the DeleteCollection implementation to internally use pagination, as it would allow us to 1) not block on full list (and as a result make some progress) 2) would be some preparation towards potentially exposing a Limit option in the API... /kind regression
/sig api-machinery
/triage accepted
I suggest the following: - can you please take care of reverting the 34s timeout for DeleteCollection calls? this is done, let's close this issue and continue operating on
sounds good, thanks! /assign

"context" "io" "net/http" <ins> "net/http/httptest" </ins> "strings" <ins> "sync/atomic" </ins> "testing" <ins> metainternalversion "k8s.io/apimachinery/pkg/apis/meta/internalversion" </ins> metainternalversionscheme "k8s.io/apimachinery/pkg/apis/meta/internalversion/scheme" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime"


Question:
In a cluster with large enough numbers of objects, it seems to be impossible to clean them up with . Because the deletecollection is failing while listing the objects from etcd. The failure happens after exactly 34 seconds, due to the 34-second max timeout introduced in PR (cc for insight on that PR). The deletecollection should be able to run for longer, and succeed. I recognise that there are still risks of the API server using too much RAM, if deleting a very large collection due to issue . But, until that issue is fixed, it seems like allowing to run for longer might be a good idea. As far as I can tell, the 34 second timeout does not apply to normal LIST calls, so on that basis I feel it should not apply to the giant unchunked LIST call that happens inside . a few hundred thousand objects of some particular type. (Argo workflowtaskresults happen to be the type I'm looking at while I write this) deletecollection. (Optionally passing some kind of fieldSelector, but I don't think that's essential) that a 504 failure code is logged in kube-audit logs, and that a log entry similar to this is logged by the API Server (In practice, the total time may be a little over ms). far as I can tell, the delete collection really has failed at this point. I notice that PR says that, in general, "The go-routine can keep running, while the timeout logic will return a timeout to the client" but I don't think that's what's happening here. I think that, because the 34 sec timeout is used in the context of the etcd call, the etcd list call really does fail, and so the code never proceeds to actually deleting anything. No response 22 AKS <details</details<details</details<details</details<details</details
/sig scalability
/cc
the deadline has been in effect before PR was introduced If we remove this deadline from here, then the deletecollection request will have at most to complete (default). The list call that is failing in your case, is a direct call to the storage, right? on the contrary, we don't enforce the timeout for LIST thinking out loud here: I don't know if it makes sense to remove the from should we allow timeout for DELETECOLLECTION to exceed ? thoughts?
The list call that is failing in your case, is a direct call to the storage, right? It reads from etcd, not from API Server Cache. Is that what you were asking Looking at the bigger picture, I think your question about 60s timeout is a good one. Especially in the context of this: In particular, changing the timeouts won't solve all of the problems with DELETECOLLECTION. My assumption, when logging this issue, is that it might be a quick way to solve just some of them.
I think we should remove this - it's effectively a regression that we introduced with the PR. That's a good question, and I'm not entirely sure. Maybe we should, but I would do that separately. In the meantime, I would much more prefer to switch the DeleteCollection implementation to internally use pagination, as it would allow us to 1) not block on full list (and as a result make some progress) 2) would be some preparation towards potentially exposing a Limit option in the API... /kind regression
/sig api-machinery
/triage accepted
I suggest the following: - can you please take care of reverting the 34s timeout for DeleteCollection calls? this is done, let's close this issue and continue operating on
sounds good, thanks! /assign

auditapis "k8s.io/apiserver/pkg/apis/audit" "k8s.io/apiserver/pkg/audit" "k8s.io/apiserver/pkg/endpoints/handlers/negotiation" <ins> "k8s.io/apiserver/pkg/registry/rest" </ins> "k8s.io/utils/pointer" )


Question:
In a cluster with large enough numbers of objects, it seems to be impossible to clean them up with . Because the deletecollection is failing while listing the objects from etcd. The failure happens after exactly 34 seconds, due to the 34-second max timeout introduced in PR (cc for insight on that PR). The deletecollection should be able to run for longer, and succeed. I recognise that there are still risks of the API server using too much RAM, if deleting a very large collection due to issue . But, until that issue is fixed, it seems like allowing to run for longer might be a good idea. As far as I can tell, the 34 second timeout does not apply to normal LIST calls, so on that basis I feel it should not apply to the giant unchunked LIST call that happens inside . a few hundred thousand objects of some particular type. (Argo workflowtaskresults happen to be the type I'm looking at while I write this) deletecollection. (Optionally passing some kind of fieldSelector, but I don't think that's essential) that a 504 failure code is logged in kube-audit logs, and that a log entry similar to this is logged by the API Server (In practice, the total time may be a little over ms). far as I can tell, the delete collection really has failed at this point. I notice that PR says that, in general, "The go-routine can keep running, while the timeout logic will return a timeout to the client" but I don't think that's what's happening here. I think that, because the 34 sec timeout is used in the context of the etcd call, the etcd list call really does fail, and so the code never proceeds to actually deleting anything. No response 22 AKS <details</details<details</details<details</details<details</details
/sig scalability
/cc
the deadline has been in effect before PR was introduced If we remove this deadline from here, then the deletecollection request will have at most to complete (default). The list call that is failing in your case, is a direct call to the storage, right? on the contrary, we don't enforce the timeout for LIST thinking out loud here: I don't know if it makes sense to remove the from should we allow timeout for DELETECOLLECTION to exceed ? thoughts?
The list call that is failing in your case, is a direct call to the storage, right? It reads from etcd, not from API Server Cache. Is that what you were asking Looking at the bigger picture, I think your question about 60s timeout is a good one. Especially in the context of this: In particular, changing the timeouts won't solve all of the problems with DELETECOLLECTION. My assumption, when logging this issue, is that it might be a quick way to solve just some of them.
I think we should remove this - it's effectively a regression that we introduced with the PR. That's a good question, and I'm not entirely sure. Maybe we should, but I would do that separately. In the meantime, I would much more prefer to switch the DeleteCollection implementation to internally use pagination, as it would allow us to 1) not block on full list (and as a result make some progress) 2) would be some preparation towards potentially exposing a Limit option in the API... /kind regression
/sig api-machinery
/triage accepted
I suggest the following: - can you please take care of reverting the 34s timeout for DeleteCollection calls? this is done, let's close this issue and continue operating on
sounds good, thanks! /assign

}) } } <ins> func TestDeleteCollectionWithNoContextDeadlineEnforced(t *testing.T) { var invokedGot, hasDeadlineGot int32 fakeDeleterFn := func(ctx context.Context, _ rest.ValidateObjectFunc, _ *metav1.DeleteOptions, _ *metainternalversion.ListOptions) (runtime.Object, error) { // we expect CollectionDeleter to be executed once atomic.AddInt32(&invokedGot, 1) // we don't expect any context deadline to be set if _, hasDeadline := ctx.Deadline(); hasDeadline { atomic.AddInt32(&hasDeadlineGot, 1) } return nil, nil } // do the minimum setup to ensure that it gets as far as CollectionDeleter scope := &RequestScope{ Namer: &mockNamer{}, Serializer: &fakeSerializer{ serializer: runtime.NewCodec(runtime.NoopEncoder{}, runtime.NoopDecoder{}), }, } handler := DeleteCollection(fakeCollectionDeleterFunc(fakeDeleterFn), false, scope, nil) request, err := http.NewRequest("GET", "/test", nil) if err != nil { t.Fatalf("unexpected error: %v", err) } // the request context should not have any deadline by default if _, hasDeadline := request.Context().Deadline(); hasDeadline { t.Fatalf("expected request context to not have any deadline") } recorder := httptest.NewRecorder() handler.ServeHTTP(recorder, request) if atomic.LoadInt32(&invokedGot) != 1 { t.Errorf("expected collection deleter to be invoked") } if atomic.LoadInt32(&hasDeadlineGot) > 0 { t.Errorf("expected context to not have any deadline") } } type fakeCollectionDeleterFunc func(ctx context.Context, deleteValidation rest.ValidateObjectFunc, options *metav1.DeleteOptions, listOptions *metainternalversion.ListOptions) (runtime.Object, error) func (f fakeCollectionDeleterFunc) DeleteCollection(ctx context.Context, deleteValidation rest.ValidateObjectFunc, options *metav1.DeleteOptions, listOptions *metainternalversion.ListOptions) (runtime.Object, error) { return f(ctx, deleteValidation, options, listOptions) } type fakeSerializer struct { serializer runtime.Serializer } func (n *fakeSerializer) SupportedMediaTypes() []runtime.SerializerInfo { return []runtime.SerializerInfo{ { MediaType:        "application/json", MediaTypeType:    "application", MediaTypeSubType: "json", }, } } func (n *fakeSerializer) EncoderForVersion(serializer runtime.Encoder, gv runtime.GroupVersioner) runtime.Encoder { return n.serializer } func (n *fakeSerializer) DecoderToVersion(serializer runtime.Decoder, gv runtime.GroupVersioner) runtime.Decoder { return n.serializer } </ins>


Question:
handles cases when csiNode is nil. This is condition is used by cluster-autoscaler internal working. However, the log line refers to causing seg fault
/sig scheduling /sig storage
/sig storage /priority important-soon /triage accepted
/sig storage /sig scheduling Note that the lister returns an error for the case when the doesn't exist. But it's ignored here: It looks like the log line that causes the panic was introduced in in 1.25+
cc
/assign
As far as I can tell this issue is only fixed in version 1.27 (by ). Is there a plan to port it back to releases 1.25 and 1.26 as well?
it has already been picked to 1.25 and 1.26 () and will be in the next patch releases for those versions
Great. Thank you for the pointer.

var _ storagelisters.CSINodeLister = CSINodeLister{} // CSINodeLister declares a storagev1.CSINode type for testing. <del> type CSINodeLister storagev1.CSINode </del> <ins> type CSINodeLister []storagev1.CSINode </ins> // Get returns a fake CSINode object. func (n CSINodeLister) Get(name string) (*storagev1.CSINode, error) { <del> csiNode := storagev1.CSINode(n) return &csiNode, nil </del> <ins> for _, cn := range n { if cn.Name == name { return &cn, nil } } return nil, fmt.Errorf("csiNode %q not found", name) </ins> } // List lists all CSINodes in the indexer.


Question:
handles cases when csiNode is nil. This is condition is used by cluster-autoscaler internal working. However, the log line refers to causing seg fault
/sig scheduling /sig storage
/sig storage /priority important-soon /triage accepted
/sig storage /sig scheduling Note that the lister returns an error for the case when the doesn't exist. But it's ignored here: It looks like the log line that causes the panic was introduced in in 1.25+
cc
/assign
As far as I can tell this issue is only fixed in version 1.27 (by ). Is there a plan to port it back to releases 1.25 and 1.26 as well?
it has already been picked to 1.25 and 1.26 () and will be in the next patch releases for those versions
Great. Thank you for the pointer.

return fmt.Errorf("looking up provisioner name for volume %v: %w", vol, err) } if !isCSIMigrationOn(csiNode, inTreeProvisionerName) { <ins> csiNodeName := "" if csiNode != nil { csiNodeName = csiNode.Name } </ins> klog.V(5).InfoS("CSI Migration is not enabled for provisioner", "provisioner", inTreeProvisionerName, <del> "pod", klog.KObj(pod), "csiNode", csiNode.Name) </del> <ins> "pod", klog.KObj(pod), "csiNode", csiNodeName) </ins> return nil } // Do translation for the in-tree volume.


Question:
handles cases when csiNode is nil. This is condition is used by cluster-autoscaler internal working. However, the log line refers to causing seg fault
/sig scheduling /sig storage
/sig storage /priority important-soon /triage accepted
/sig storage /sig scheduling Note that the lister returns an error for the case when the doesn't exist. But it's ignored here: It looks like the log line that causes the panic was introduced in in 1.25+
cc
/assign
As far as I can tell this issue is only fixed in version 1.27 (by ). Is there a plan to port it back to releases 1.25 and 1.26 as well?
it has already been picked to 1.25 and 1.26 () and will be in the next patch releases for those versions
Great. Thank you for the pointer.

wantStatus:       framework.NewStatus(framework.Unschedulable, ErrReasonMaxVolumeCountExceeded), }, { <ins> newPod:           inTreeInlineVolPod, existingPods:     []*v1.Pod{inTreeTwoVolPod}, filterName:       "csi", maxVols:          2, driverNames:      []string{csilibplugins.AWSEBSInTreePluginName, ebsCSIDriverName}, migrationEnabled: true, limitSource:      "node", test:             "nil csi node", }, { </ins> newPod:           pendingVolumePod, existingPods:     []*v1.Pod{inTreeTwoVolPod}, filterName:       "csi",


Question:
handles cases when csiNode is nil. This is condition is used by cluster-autoscaler internal working. However, the log line refers to causing seg fault
/sig scheduling /sig storage
/sig storage /priority important-soon /triage accepted
/sig storage /sig scheduling Note that the lister returns an error for the case when the doesn't exist. But it's ignored here: It looks like the log line that causes the panic was introduced in in 1.25+
cc
/assign
As far as I can tell this issue is only fixed in version 1.27 (by ). Is there a plan to port it back to releases 1.25 and 1.26 as well?
it has already been picked to 1.25 and 1.26 () and will be in the next patch releases for those versions
Great. Thank you for the pointer.

} pvLister = append(pvLister, pv) } } return pvLister }


Question:
handles cases when csiNode is nil. This is condition is used by cluster-autoscaler internal working. However, the log line refers to causing seg fault
/sig scheduling /sig storage
/sig storage /priority important-soon /triage accepted
/sig storage /sig scheduling Note that the lister returns an error for the case when the doesn't exist. But it's ignored here: It looks like the log line that causes the panic was introduced in in 1.25+
cc
/assign
As far as I can tell this issue is only fixed in version 1.27 (by ). Is there a plan to port it back to releases 1.25 and 1.26 as well?
it has already been picked to 1.25 and 1.26 () and will be in the next patch releases for those versions
Great. Thank you for the pointer.

} func getFakeCSINodeLister(csiNode *storagev1.CSINode) fakeframework.CSINodeLister { <ins> csiNodeLister := fakeframework.CSINodeLister{} </ins> if csiNode != nil { <del> return fakeframework.CSINodeLister(*csiNode) </del> <ins> csiNodeLister = append(csiNodeLister, *csiNode.DeepCopy()) </ins> } <del> return fakeframework.CSINodeLister{} </del> <ins> return csiNodeLister </ins> } func getNodeWithPodAndVolumeLimits(limitSource string, pods []*v1.Pod, limit int64, driverNames ...string) (*framework.NodeInfo, *storagev1.CSINode) {


Question:
handles cases when csiNode is nil. This is condition is used by cluster-autoscaler internal working. However, the log line refers to causing seg fault
/sig scheduling /sig storage
/sig storage /priority important-soon /triage accepted
/sig storage /sig scheduling Note that the lister returns an error for the case when the doesn't exist. But it's ignored here: It looks like the log line that causes the panic was introduced in in 1.25+
cc
/assign
As far as I can tell this issue is only fixed in version 1.27 (by ). Is there a plan to port it back to releases 1.25 and 1.26 as well?
it has already been picked to 1.25 and 1.26 () and will be in the next patch releases for those versions
Great. Thank you for the pointer.

initCSINode := func() { csiNode = &storagev1.CSINode{ <del> ObjectMeta: metav1.ObjectMeta{Name: "csi-node-for-max-pd-test-1"}, </del> <ins> ObjectMeta: metav1.ObjectMeta{Name: "node-for-max-pd-test-1"}, </ins> Spec: storagev1.CSINodeSpec{ Drivers: []storagev1.CSINodeDriver{}, },


Question:
[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below. This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted

API Server using the current Aggregator [Conformance]' description: Ensure that the sample-apiserver code from 1.17 and compiled against 1.17 will work on the current Aggregator/API-Server. <del> release: v1.17, v1.21 </del> <ins> release: v1.17, v1.21, v1.27 </ins> file: test/e2e/apimachinery/aggregator.go - testname: Custom Resource Definition Conversion Webhook, convert mixed version list codename: '[sig-api-machinery] CustomResourceConversionWebhook [Privileged:ClusterAdmin]


Question:
[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below. This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted

apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" <ins> "k8s.io/apimachinery/pkg/labels" </ins> "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/discovery" clientset "k8s.io/client-go/kubernetes" <ins> "k8s.io/client-go/util/retry" </ins> apiregistrationv1 "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1" aggregatorclient "k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset" rbacv1helpers "k8s.io/kubernetes/pkg/apis/rbac/v1" "k8s.io/kubernetes/test/e2e/framework" <ins> e2eauth "k8s.io/kubernetes/test/e2e/framework/auth" </ins> e2edeployment "k8s.io/kubernetes/test/e2e/framework/deployment" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" imageutils "k8s.io/kubernetes/test/utils/image"


Question:
[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below. This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted

const ( aggregatorServicePort = 7443 <ins> apiServiceRetryPeriod  = 1 * time.Second apiServiceRetryTimeout = 2 * time.Minute </ins> ) var _ = SIGDescribe("Aggregator", func() {


Question:
[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below. This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted

}) /* <del> Release: v1.17, v1.21 </del> <ins> Release: v1.17, v1.21, v1.27 </ins> Testname: aggregator-supports-the-sample-apiserver Description: Ensure that the sample-apiserver code from 1.17 and compiled against 1.17 will work on the current Aggregator/API-Server.


Question:
[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below. This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted

_, err := client.CoreV1().Secrets(namespace).Create(ctx, secret, metav1.CreateOptions{}) framework.ExpectNoError(err, "creating secret %s in namespace %s", secretName, namespace) <del> // kubectl create -f clusterrole.yaml _, err = client.RbacV1().ClusterRoles().Create(ctx, &rbacv1.ClusterRole{ </del> <ins> if e2eauth.IsRBACEnabled(ctx, client.RbacV1()) { // kubectl create -f clusterrole.yaml _, err = client.RbacV1().ClusterRoles().Create(ctx, &rbacv1.ClusterRole{ </ins> <del> ObjectMeta: metav1.ObjectMeta{Name: "sample-apiserver-reader"}, Rules: []rbacv1.PolicyRule{ rbacv1helpers.NewRule("get", "list", "watch").Groups("").Resources("namespaces").RuleOrDie(), rbacv1helpers.NewRule("get", "list", "watch").Groups("admissionregistration.k8s.io").Resources("*").RuleOrDie(), }, }, metav1.CreateOptions{}) framework.ExpectNoError(err, "creating cluster role %s", "sample-apiserver-reader") </del> <ins> ObjectMeta: metav1.ObjectMeta{Name: "sample-apiserver-reader"}, Rules: []rbacv1.PolicyRule{ rbacv1helpers.NewRule("get", "list", "watch").Groups("").Resources("namespaces").RuleOrDie(), rbacv1helpers.NewRule("get", "list", "watch").Groups("admissionregistration.k8s.io").Resources("*").RuleOrDie(), }, }, metav1.CreateOptions{}) framework.ExpectNoError(err, "creating cluster role %s", "sample-apiserver-reader") </ins> <del> _, err = client.RbacV1().ClusterRoleBindings().Create(ctx, &rbacv1.ClusterRoleBinding{ ObjectMeta: metav1.ObjectMeta{ Name: "wardler:" + namespace + ":sample-apiserver-reader", }, RoleRef: rbacv1.RoleRef{ APIGroup: "rbac.authorization.k8s.io", Kind:     "ClusterRole", Name:     "sample-apiserver-reader", }, Subjects: []rbacv1.Subject{ { APIGroup:  "", Kind:      "ServiceAccount", Name:      "default", Namespace: namespace, </del> <ins> _, err = client.RbacV1().ClusterRoleBindings().Create(ctx, &rbacv1.ClusterRoleBinding{ ObjectMeta: metav1.ObjectMeta{ Name: "wardler:" + namespace + ":sample-apiserver-reader", </ins> }, <del> }, }, metav1.CreateOptions{}) framework.ExpectNoError(err, "creating cluster role binding %s", "wardler:"+namespace+":sample-apiserver-reader") </del> <ins> RoleRef: rbacv1.RoleRef{ APIGroup: "rbac.authorization.k8s.io", Kind:     "ClusterRole", Name:     "sample-apiserver-reader", }, Subjects: []rbacv1.Subject{ { APIGroup:  "", Kind:      "ServiceAccount", Name:      "default", Namespace: namespace, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err, "creating cluster role binding %s", "wardler:"+namespace+":sample-apiserver-reader") </ins> <del> // kubectl create -f authDelegator.yaml _, err = client.RbacV1().ClusterRoleBindings().Create(ctx, &rbacv1.ClusterRoleBinding{ ObjectMeta: metav1.ObjectMeta{ Name: "wardler:" + namespace + ":auth-delegator", }, RoleRef: rbacv1.RoleRef{ APIGroup: "rbac.authorization.k8s.io", Kind:     "ClusterRole", Name:     "system:auth-delegator", }, Subjects: []rbacv1.Subject{ { APIGroup:  "", Kind:      "ServiceAccount", Name:      "default", Namespace: namespace, </del> <ins> // kubectl create -f authDelegator.yaml _, err = client.RbacV1().ClusterRoleBindings().Create(ctx, &rbacv1.ClusterRoleBinding{ ObjectMeta: metav1.ObjectMeta{ Name: "wardler:" + namespace + ":auth-delegator", </ins> }, <del> }, }, metav1.CreateOptions{}) framework.ExpectNoError(err, "creating cluster role binding %s", "wardler:"+namespace+":auth-delegator") </del> <ins> RoleRef: rbacv1.RoleRef{ APIGroup: "rbac.authorization.k8s.io", Kind:     "ClusterRole", Name:     "system:auth-delegator", }, Subjects: []rbacv1.Subject{ { APIGroup:  "", Kind:      "ServiceAccount", Name:      "default", Namespace: namespace, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err, "creating cluster role binding %s", "wardler:"+namespace+":auth-delegator") } </ins> // kubectl create -f deploy.yaml deploymentName := "sample-apiserver-deployment"


Question:
[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below. This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted

_, err = client.CoreV1().ServiceAccounts(namespace).Create(ctx, sa, metav1.CreateOptions{}) framework.ExpectNoError(err, "creating service account %s in namespace %s", "sample-apiserver", namespace) <del> // kubectl create -f auth-reader.yaml _, err = client.RbacV1().RoleBindings("kube-system").Create(ctx, &rbacv1.RoleBinding{ ObjectMeta: metav1.ObjectMeta{ Name: "wardler-auth-reader", Annotations: map[string]string{ rbacv1.AutoUpdateAnnotationKey: "true", </del> <ins> if e2eauth.IsRBACEnabled(ctx, client.RbacV1()) { // kubectl create -f auth-reader.yaml _, err = client.RbacV1().RoleBindings("kube-system").Create(ctx, &rbacv1.RoleBinding{ ObjectMeta: metav1.ObjectMeta{ Name: "wardler-auth-reader", Annotations: map[string]string{ rbacv1.AutoUpdateAnnotationKey: "true", }, </ins> }, <del> }, RoleRef: rbacv1.RoleRef{ APIGroup: "", Kind:     "Role", Name:     "extension-apiserver-authentication-reader", }, Subjects: []rbacv1.Subject{ { Kind:      "ServiceAccount", Name:      "default", Namespace: namespace, </del> <ins> RoleRef: rbacv1.RoleRef{ APIGroup: "", Kind:     "Role", Name:     "extension-apiserver-authentication-reader", </ins> }, <del> }, }, metav1.CreateOptions{}) framework.ExpectNoError(err, "creating role binding %s in namespace %s", "wardler-auth-reader", "kube-system") </del> <ins> Subjects: []rbacv1.Subject{ { Kind:      "ServiceAccount", Name:      "default", Namespace: namespace, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err, "creating role binding %s in namespace %s", "wardler-auth-reader", "kube-system") } </ins> // Wait for the extension apiserver to be up and healthy // kubectl get deployments -n <aggregated-api-namespace> && status == Running


Question:
[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below. This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted

Service: &apiregistrationv1.ServiceReference{ Namespace: namespace, Name:      "sample-api", <del> Port:      pointer.Int32Ptr(aggregatorServicePort), </del> <ins> Port:      pointer.Int32(aggregatorServicePort), </ins> }, Group:                "wardle.example.com", Version:              "v1alpha1",


Question:
[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below. This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted

framework.Failf("Unable to find v1alpha1.wardle.example.com in APIServiceList") } <ins> // As the APIService doesn't have any labels currently set we need to // set one so that we can select it later when we call deleteCollection ginkgo.By("Adding a label to the APIService") apiServiceName := "v1alpha1.wardle.example.com" apiServiceClient := aggrclient.ApiregistrationV1().APIServices() apiServiceLabel := map[string]string{"e2e-apiservice": "patched"} apiServiceLabelSelector := labels.SelectorFromSet(apiServiceLabel).String() apiServicePatch, err := json.Marshal(map[string]interface{}{ "metadata": map[string]interface{}{ "labels": apiServiceLabel, }, }) framework.ExpectNoError(err, "failed to Marshal APIService JSON patch") _, err = apiServiceClient.Patch(ctx, apiServiceName, types.StrategicMergePatchType, []byte(apiServicePatch), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch APIService") patchedApiService, err := apiServiceClient.Get(ctx, apiServiceName, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to retrieve api service %s", apiServiceName) framework.Logf("APIService labels: %v", patchedApiService.Labels) ginkgo.By("Updating APIService Status") var updatedStatus, wardle *apiregistrationv1.APIService err = retry.RetryOnConflict(retry.DefaultRetry, func() error { var statusToUpdate *apiregistrationv1.APIService statusContent, err = restClient.Get(). AbsPath("/apis/apiregistration.k8s.io/v1/apiservices/v1alpha1.wardle.example.com/status"). SetHeader("Accept", "application/json").DoRaw(ctx) framework.ExpectNoError(err, "No response for .../apiservices/v1alpha1.wardle.example.com/status. Error: %v", err) err = json.Unmarshal([]byte(statusContent), &statusToUpdate) framework.ExpectNoError(err, "Failed to process statusContent: %v | err: %v ", string(statusContent), err) statusToUpdate.Status.Conditions = append(statusToUpdate.Status.Conditions, apiregistrationv1.APIServiceCondition{ Type:    "StatusUpdated", Status:  "True", Reason:  "E2E", Message: "Set from e2e test", }) updatedStatus, err = apiServiceClient.UpdateStatus(ctx, statusToUpdate, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "Failed to update status. %v", err) framework.Logf("updatedStatus.Conditions: %#v", updatedStatus.Status.Conditions) ginkgo.By("Confirm that v1alpha1.wardle.example.com /status was updated") statusContent, err = restClient.Get(). AbsPath("/apis/apiregistration.k8s.io/v1/apiservices/v1alpha1.wardle.example.com/status"). SetHeader("Accept", "application/json").DoRaw(ctx) framework.ExpectNoError(err, "No response for .../apiservices/v1alpha1.wardle.example.com/status. Error: %v", err) err = json.Unmarshal([]byte(statusContent), &wardle) framework.ExpectNoError(err, "Failed to process statusContent: %v | err: %v ", string(statusContent), err) foundUpdatedStatusCondition := false for _, cond := range wardle.Status.Conditions { if cond.Type == "StatusUpdated" && cond.Reason == "E2E" && cond.Message == "Set from e2e test" { framework.Logf("Found APIService %v with Labels: %v & Condition: %v", wardle.ObjectMeta.Name, wardle.Labels, cond) foundUpdatedStatusCondition = true break } else { framework.Logf("Observed APIService %v with Labels: %v & Condition: %v", wardle.ObjectMeta.Name, wardle.Labels, cond) } } framework.ExpectEqual(foundUpdatedStatusCondition, true, "The updated status condition was not found. %#v", wardle.Status.Conditions) framework.Logf("Found updated status condition for %s", wardle.ObjectMeta.Name) ginkgo.By(fmt.Sprintf("Replace APIService %s", apiServiceName)) var updatedApiService *apiregistrationv1.APIService err = retry.RetryOnConflict(retry.DefaultRetry, func() error { currentApiService, err := apiServiceClient.Get(ctx, apiServiceName, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to get APIService %s", apiServiceName) currentApiService.Labels = map[string]string{ apiServiceName: "updated", } updatedApiService, err = apiServiceClient.Update(ctx, currentApiService, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err) framework.ExpectEqual(updatedApiService.Labels[apiServiceName], "updated", "should have the updated label but have %q", updatedApiService.Labels[apiServiceName]) framework.Logf("Found updated apiService label for %q", apiServiceName) </ins> // kubectl delete flunder test-flunder <ins> ginkgo.By(fmt.Sprintf("Delete APIService %q", flunderName)) </ins> err = dynamicClient.Delete(ctx, flunderName, metav1.DeleteOptions{}) validateErrorWithDebugInfo(ctx, f, err, pods, "deleting flunders(%v) using dynamic client", unstructuredList.Items)


Question:
[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below. This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted

framework.Failf("failed to get back the correct deleted flunders list %v from the dynamic client", unstructuredList) } <ins> ginkgo.By("Recreating test-flunder before removing endpoint via deleteCollection") jsonFlunder, err = json.Marshal(testFlunder) framework.ExpectNoError(err, "marshalling test-flunder for create using dynamic client") unstruct = &unstructured.Unstructured{} err = unstruct.UnmarshalJSON(jsonFlunder) framework.ExpectNoError(err, "unmarshalling test-flunder as unstructured for create using dynamic client") _, err = dynamicClient.Create(ctx, unstruct, metav1.CreateOptions{}) framework.ExpectNoError(err, "listing flunders using dynamic client") // kubectl get flunders unstructuredList, err = dynamicClient.List(ctx, metav1.ListOptions{}) framework.ExpectNoError(err, "listing flunders using dynamic client") if len(unstructuredList.Items) != 1 { framework.Failf("failed to get back the correct flunders list %v from the dynamic client", unstructuredList) } ginkgo.By("Read v1alpha1.wardle.example.com /status before patching it") statusContent, err = restClient.Get(). AbsPath("/apis/apiregistration.k8s.io/v1/apiservices/v1alpha1.wardle.example.com/status"). SetHeader("Accept", "application/json").DoRaw(ctx) framework.ExpectNoError(err, "No response for .../apiservices/v1alpha1.wardle.example.com/status. Error: %v", err) wardle.Reset() err = json.Unmarshal([]byte(statusContent), &wardle) framework.ExpectNoError(err, "Failed to process statusContent: %v | err: %v ", string(statusContent), err) ginkgo.By("Patch APIService Status") patch := map[string]interface{}{ "status": map[string]interface{}{ "conditions": append(wardle.Status.Conditions, apiregistrationv1.APIServiceCondition{ Type:    "StatusPatched", Status:  "True", Reason:  "E2E", Message: "Set by e2e test", }), }, } payload, err := json.Marshal(patch) framework.ExpectNoError(err, "Failed to marshal JSON. %v", err) _, err = restClient.Patch(types.MergePatchType). AbsPath("/apis/apiregistration.k8s.io/v1/apiservices/v1alpha1.wardle.example.com/status"). SetHeader("Accept", "application/json"). Body([]byte(payload)). DoRaw(ctx) framework.ExpectNoError(err, "Patch failed for .../apiservices/v1alpha1.wardle.example.com/status. Error: %v", err) ginkgo.By("Confirm that v1alpha1.wardle.example.com /status was patched") statusContent, err = restClient.Get(). AbsPath("/apis/apiregistration.k8s.io/v1/apiservices/v1alpha1.wardle.example.com/status"). SetHeader("Accept", "application/json").DoRaw(ctx) framework.ExpectNoError(err, "No response for .../apiservices/v1alpha1.wardle.example.com/status. Error: %v", err) wardle.Reset() err = json.Unmarshal([]byte(statusContent), &wardle) framework.ExpectNoError(err, "Failed to process statusContent: %v | err: %v ", string(statusContent), err) foundPatchedStatusCondition := false for _, cond := range wardle.Status.Conditions { if cond.Type == "StatusPatched" && cond.Reason == "E2E" && cond.Message == "Set by e2e test" { framework.Logf("Found APIService %v with Labels: %v & Conditions: %v", wardle.ObjectMeta.Name, wardle.Labels, cond) foundPatchedStatusCondition = true break } else { framework.Logf("Observed APIService %v with Labels: %v & Conditions: %v", wardle.ObjectMeta.Name, wardle.Labels, cond) } } framework.ExpectEqual(foundPatchedStatusCondition, true, "The patched status condition was not found. %#v", wardle.Status.Conditions) framework.Logf("Found patched status condition for %s", wardle.ObjectMeta.Name) ginkgo.By(fmt.Sprintf("APIService deleteCollection with labelSelector: %q", apiServiceLabelSelector)) err = aggrclient.ApiregistrationV1().APIServices().DeleteCollection(ctx, metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: apiServiceLabelSelector}) framework.ExpectNoError(err, "Unable to delete apiservice %s", apiServiceName) ginkgo.By("Confirm that the generated APIService has been deleted") err = wait.PollImmediate(apiServiceRetryPeriod, apiServiceRetryTimeout, checkApiServiceListQuantity(ctx, aggrclient, apiServiceLabelSelector, 0)) framework.ExpectNoError(err, "failed to count the required APIServices") framework.Logf("APIService %s has been deleted.", apiServiceName) </ins> cleanTest(ctx, client, aggrclient, namespace) }


Question:
[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below. This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted

} return fmt.Sprintf("%s-%d", base, id) } <ins> func checkApiServiceListQuantity(ctx context.Context, aggrclient *aggregatorclient.Clientset, label string, quantity int) func() (bool, error) { return func() (bool, error) { var err error framework.Logf("Requesting list of APIServices to confirm quantity") list, err := aggrclient.ApiregistrationV1().APIServices().List(ctx, metav1.ListOptions{LabelSelector: label}) if err != nil { return false, err } if len(list.Items) != quantity { return false, err } framework.Logf("Found %d APIService with label %q", quantity, label) return true, nil } } </ins>


Question:
After the cluster was updated to enable PD CSI Driver, there was a race condition where AD Controller (kube-controller-manager) started to BEFORE the PD CSI Driver was up, thus AD Controller gave us the error Then AD Controller triggered the disk detach because the disk/volume was not populated in the desired state of world. Unfortunately the disk was in use during this detach and got corrupted. Since cluster upgrades can cause the controller manager and CSI drivers to start at different times, the AD controller needs to be more conservative when detaching We have not been able to reproduce this, it came up from a customer. /sig storage /assign <detailsk8s 1.23, CSI migration enabled </details<detailsGKE </details<details</details<details</details<details</details<details</details
/triage accepted
I think there's potentially two changes we should do: 1) for the initial DSW population, we should skip the check that if the csi driver is not installed. 2) in general, we should not skip volumes during dsw initial population if an error is returned. If we had some temporary api server error then that would cause volumes to not get to dsw and similar behavior happen. cc
for a repro, maybe you can try something like: a cluster with csi migration enabled (control plane + nodes) a pod with intree-volume and get it running the csi driver node daemonset. verify the driver is removed from CSINode kube-controller-manager
cc
/cc was trying to repro this, but on GKE with the managed driver which made it complicated.
Yeah maybe gce kube-up would be easier for a repro
If CSINode says that it has migrated GCP PD volume plugin to CSI, then IMO A/D controller should always attach GCP PD volumes to it using CSI, regardless if the CSI driver is present or not. In-tree volume plugin in the kubelet is disabled and can't mount any in-tree volumes anyway. To flip the migration feature gate / upgrade / downgrade kubelet, the cluster admin must drain the node first to make sure everything is detached. A/D controller should not see change of CSINode migration annotations with any volume attached to it.
/priority important-soon
Another option could be to process these volumes anyways but add them to DSOW with migration status as "uncertain", so as they can be re-processed later. But surely it looks like might be easiest unless it has any downsides.
There wasn't a node update in this case---there was a control plane update which restarted the PD CSI driver, and it looks like the CSIDriver resource was updated or changed in some way which raced with the kube-controller-manager restart.
Is there anyway we can know if the csi driver is up and running(like a health check ping) before populating DSW?
We use the CSINode registration to determine that. The discussion here I think is concluding that we should not use that as a criteria for adding the volume to DSW. We only need to check that CSI migration is enabled on the node.
+1 we need to be very resilient against races, I think we have to assume a driver is in process of being installed at any point in the AD controller reconcilliation.
/assign sunnylovestiramisu
Reproduced with the steps: In /var/log/kube-controller-
Tested with the and doing the reproduce steps above. The error of does not show up anymore. The error message still get logged in the kube-controller- for debugging purpose: Ran Ran On the GCP console, the disk is used by node: Ran I redeploy the pd-csi-driver and then make sure it is up and running. Ran

isMigratedOnNode := mpaSet.Has(pluginName) <del> if isMigratedOnNode { installed := false driverName, err := csiMigratedPluginManager.GetCSINameFromInTreeName(pluginName) if err != nil { return isMigratedOnNode, err } for _, driver := range csiNode.Spec.Drivers { if driver.Name == driverName { installed = true break } } if !installed { return true, fmt.Errorf("in-tree plugin %s is migrated on node %s but driver %s is not installed", pluginName, string(nodeName), driverName) } } </del> return isMigratedOnNode, nil }


Question:
After the cluster was updated to enable PD CSI Driver, there was a race condition where AD Controller (kube-controller-manager) started to BEFORE the PD CSI Driver was up, thus AD Controller gave us the error Then AD Controller triggered the disk detach because the disk/volume was not populated in the desired state of world. Unfortunately the disk was in use during this detach and got corrupted. Since cluster upgrades can cause the controller manager and CSI drivers to start at different times, the AD controller needs to be more conservative when detaching We have not been able to reproduce this, it came up from a customer. /sig storage /assign <detailsk8s 1.23, CSI migration enabled </details<detailsGKE </details<details</details<details</details<details</details<details</details
/triage accepted
I think there's potentially two changes we should do: 1) for the initial DSW population, we should skip the check that if the csi driver is not installed. 2) in general, we should not skip volumes during dsw initial population if an error is returned. If we had some temporary api server error then that would cause volumes to not get to dsw and similar behavior happen. cc
for a repro, maybe you can try something like: a cluster with csi migration enabled (control plane + nodes) a pod with intree-volume and get it running the csi driver node daemonset. verify the driver is removed from CSINode kube-controller-manager
cc
/cc was trying to repro this, but on GKE with the managed driver which made it complicated.
Yeah maybe gce kube-up would be easier for a repro
If CSINode says that it has migrated GCP PD volume plugin to CSI, then IMO A/D controller should always attach GCP PD volumes to it using CSI, regardless if the CSI driver is present or not. In-tree volume plugin in the kubelet is disabled and can't mount any in-tree volumes anyway. To flip the migration feature gate / upgrade / downgrade kubelet, the cluster admin must drain the node first to make sure everything is detached. A/D controller should not see change of CSINode migration annotations with any volume attached to it.
/priority important-soon
Another option could be to process these volumes anyways but add them to DSOW with migration status as "uncertain", so as they can be re-processed later. But surely it looks like might be easiest unless it has any downsides.
There wasn't a node update in this case---there was a control plane update which restarted the PD CSI driver, and it looks like the CSIDriver resource was updated or changed in some way which raced with the kube-controller-manager restart.
Is there anyway we can know if the csi driver is up and running(like a health check ping) before populating DSW?
We use the CSINode registration to determine that. The discussion here I think is concluding that we should not use that as a criteria for adding the volume to DSW. We only need to check that CSI migration is enabled on the node.
+1 we need to be very resilient against races, I think we have to assume a driver is in process of being installed at any point in the AD controller reconcilliation.
/assign sunnylovestiramisu
Reproduced with the steps: In /var/log/kube-controller-
Tested with the and doing the reproduce steps above. The error of does not show up anymore. The error message still get logged in the kube-controller- for debugging purpose: Ran Ran On the GCP console, the disk is used by node: Ran I redeploy the pd-csi-driver and then make sure it is up and running. Ran

<ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "os" "reflect" "strings" "testing" "time" "github.com/google/go-cmp/cmp" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" kubetypes "k8s.io/apimachinery/pkg/types" "k8s.io/client-go/informers" csitrans "k8s.io/csi-translation-lib" fakeframework "k8s.io/kubernetes/pkg/scheduler/framework/fake" "k8s.io/kubernetes/pkg/volume/csimigration" "k8s.io/kubernetes/pkg/volume/fc" utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/kubernetes/fake" utiltesting "k8s.io/client-go/util/testing" "k8s.io/kubernetes/pkg/volume" volumetest "k8s.io/kubernetes/pkg/volume/testing" ) const ( testHostName      = "test-hostname" socketPath        = "/var/run/kmsplugin" migratedVolume    = "migrated-volume-name" nonMigratedVolume = "non-migrated-volume-name" testNodeName      = "test-node-name" ) var ( dirOrCreate = v1.HostPathType(v1.HostPathDirectoryOrCreate) nodeName    = kubetypes.NodeName(testNodeName) hostPath    = &v1.HostPathVolumeSource{ Path: socketPath, Type: &dirOrCreate, } migratedObjectReference    = v1.ObjectReference{Namespace: "default", Name: "migrated-pvc"} nonMigratedObjectReference = v1.ObjectReference{Namespace: "default", Name: "non-migrated-pvc"} fsVolumeMode               = new(v1.PersistentVolumeMode) ) type vaTest struct { desc                 string createNodeName       kubetypes.NodeName pod                  *v1.Pod wantVolume           *v1.Volume wantPersistentVolume *v1.PersistentVolume wantErrorMessage     string } func Test_CreateVolumeSpec(t *testing.T) { for _, test := range []vaTest{ { desc:           "inline volume type that does not support csi migration", createNodeName: nodeName, pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "kube-apiserver", Namespace: "default", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: migratedVolume, VolumeSource: v1.VolumeSource{ HostPath: hostPath, }, }, }, }, }, wantVolume: &v1.Volume{ Name: migratedVolume, VolumeSource: v1.VolumeSource{ HostPath: hostPath, }, }, }, { desc:           "inline volume type that supports csi migration", createNodeName: nodeName, pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "kube-apiserver", Namespace: "default", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: migratedVolume, VolumeSource: v1.VolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName:    "test-disk", FSType:    "ext4", Partition: 0, ReadOnly:  false, }, }, }, }, }, }, wantPersistentVolume: &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: "pd.csi.storage.gke.io-test-disk", }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ CSI: &v1.CSIPersistentVolumeSource{ Driver:           "pd.csi.storage.gke.io", VolumeHandle:     "projects/UNSPECIFIED/zones/UNSPECIFIED/disks/test-disk", FSType:           "ext4", ReadOnly:         false, VolumeAttributes: map[string]string{"partition": ""}, }, }, AccessModes: []v1.PersistentVolumeAccessMode{"ReadWriteOnce"}, VolumeMode:  fsVolumeMode, }, }, }, { desc:           "pv type that does not support csi migration", createNodeName: nodeName, pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "kube-apiserver", Namespace: "default", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: nonMigratedVolume, VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "non-migrated-pvc", ReadOnly:  false, }, }, }, }, }, }, wantPersistentVolume: &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: nonMigratedVolume, }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ ScaleIO: &v1.ScaleIOPersistentVolumeSource{}, }, ClaimRef: &nonMigratedObjectReference, }, }, }, { desc:           "pv type that supports csi migration", createNodeName: nodeName, pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "kube-apiserver", Namespace: "default", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: migratedVolume, VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "migrated-pvc", ReadOnly:  false, }, }, }, }, }, }, wantPersistentVolume: &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: migratedVolume, }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ CSI: &v1.CSIPersistentVolumeSource{ Driver:           "pd.csi.storage.gke.io", VolumeHandle:     "projects/UNSPECIFIED/zones/UNSPECIFIED/disks/test-disk", FSType:           "ext4", ReadOnly:         false, VolumeAttributes: map[string]string{"partition": ""}, }, }, ClaimRef: &migratedObjectReference, }, }, }, { desc:           "CSINode not found for a volume type that supports csi migration", createNodeName: kubetypes.NodeName("another-node"), pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "kube-apiserver", Namespace: "default", }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: migratedVolume, VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "migrated-pvc", ReadOnly:  false, }, }, }, }, }, }, wantErrorMessage: "csiNode "another-node" not found", }, } { t.Run(test.desc, func(t *testing.T) { plugMgr, intreeToCSITranslator, csiTranslator, pvLister, pvcLister := setup(testNodeName, t) actualSpec, err := CreateVolumeSpec(test.pod.Spec.Volumes[0], test.pod, test.createNodeName, plugMgr, pvcLister, pvLister, intreeToCSITranslator, csiTranslator) if actualSpec == nil && (test.wantPersistentVolume != nil || test.wantVolume != nil) { t.Errorf("got volume spec is nil") } if (len(test.wantErrorMessage) > 0 && err == nil) || (err != nil && !strings.Contains(err.Error(), test.wantErrorMessage)) { t.Errorf("got err %v, want err with message %v", err, test.wantErrorMessage) } if test.wantPersistentVolume != nil { if actualSpec.PersistentVolume == nil { t.Errorf("gotVolumeWithCSIMigration is nil") } gotVolumeWithCSIMigration := *actualSpec.PersistentVolume if gotVolumeWithCSIMigration.Name != test.wantPersistentVolume.Name { t.Errorf("got volume name is %v, want volume name is %v", gotVolumeWithCSIMigration.Name, test.wantPersistentVolume.Name) } if !reflect.DeepEqual(gotVolumeWithCSIMigration.Spec, test.wantPersistentVolume.Spec) { t.Errorf("got volume.Spec and want.Spec diff is %s", cmp.Diff(gotVolumeWithCSIMigration.Spec, test.wantPersistentVolume.Spec)) } } if test.wantVolume != nil { if actualSpec.Volume == nil { t.Errorf("gotVolume is nil") } gotVolume := *actualSpec.Volume if !reflect.DeepEqual(gotVolume, *test.wantVolume) { t.Errorf("got volume and want diff is %s", cmp.Diff(gotVolume, test.wantVolume)) } } }) } } func setup(nodeName string, t *testing.T) (*volume.VolumePluginMgr, csimigration.PluginManager, csitrans.CSITranslator, fakeframework.PersistentVolumeLister, fakeframework.PersistentVolumeClaimLister) { tmpDir, err := utiltesting.MkTmpdir("csi-test") if err != nil { t.Fatalf("can't make a temp dir: %v", err) } defer os.RemoveAll(tmpDir) *fsVolumeMode = v1.PersistentVolumeFilesystem csiTranslator := csitrans.New() intreeToCSITranslator := csimigration.NewPluginManager(csiTranslator, utilfeature.DefaultFeatureGate) kubeClient := fake.NewSimpleClientset() factory := informers.NewSharedInformerFactory(kubeClient, time.Minute) csiDriverInformer := factory.Storage().V1().CSIDrivers() csiDriverLister := csiDriverInformer.Lister() volumeAttachmentInformer := factory.Storage().V1().VolumeAttachments() volumeAttachmentLister := volumeAttachmentInformer.Lister() plugMgr := &volume.VolumePluginMgr{} fakeAttachDetachVolumeHost := volumetest.NewFakeAttachDetachVolumeHostWithCSINodeName(t, tmpDir, kubeClient, fc.ProbeVolumePlugins(), nodeName, csiDriverLister, volumeAttachmentLister, ) plugMgr.Host = fakeAttachDetachVolumeHost pvLister := fakeframework.PersistentVolumeLister{ { ObjectMeta: metav1.ObjectMeta{Name: migratedVolume}, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName:    "test-disk", FSType:    "ext4", Partition: 0, ReadOnly:  false, }, }, ClaimRef: &migratedObjectReference, }, }, { ObjectMeta: metav1.ObjectMeta{Name: nonMigratedVolume}, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ ScaleIO: &v1.ScaleIOPersistentVolumeSource{}, }, ClaimRef: &nonMigratedObjectReference, }, }, } pvcLister := fakeframework.PersistentVolumeClaimLister{ { ObjectMeta: metav1.ObjectMeta{Name: "migrated-pvc", Namespace: "default"}, Spec:       v1.PersistentVolumeClaimSpec{VolumeName: migratedVolume}, Status: v1.PersistentVolumeClaimStatus{ Phase: v1.ClaimBound, }, }, { ObjectMeta: metav1.ObjectMeta{Name: "non-migrated-pvc", Namespace: "default"}, Spec:       v1.PersistentVolumeClaimSpec{VolumeName: nonMigratedVolume}, Status: v1.PersistentVolumeClaimStatus{ Phase: v1.ClaimBound, }, }, } return plugMgr, intreeToCSITranslator, csiTranslator, pvLister, pvcLister } </ins>