Question:
Minimal Repro: Create 2 different disks Create 2 Pods each using one of the disks Start both pods on the same node You will notice one of the disks gets ed to a path: , then both disks are ed from to a target path: which is similar but at least uses the pod . Both of these volumes are being treated as the same volume in the Kubelet even though they are definitely not the same volume. Note: it may not be "treated" as the same volume - it may just be that since the paths are the same the following volumes see that the path is already a mountpoint and skip the step. This is the more likely scenario based on observations. The value is the "name" of the inline-volume. These can obviously be the same across different pods - we should actually be placing it in a more unique directory (based off the translated maybe?). /cc
/sig storage
The inline volume example I'm using (ignore the fact that is a complete misnomer): I've confirmed that changing the "name" makes inline volumes work well since there is no longer a collision on the paths
/assign
Alright, problem is sort of twofold: The referred to above for inline-volumes is set here: As we can see from the comment: it is set to the because the in-tree CSI Plugin makes a lot of naming assumptions based on the which for PVC/PVs is the PV name (unique in a cluster). Primarily, during unmount the CSI Plugin tries to from a "well known ": which happens to be some path based on the which for the inline volume is just the ( above). So we run into the problem that if we use the directory is not sufficiently unique for different disks. And if we use something else like then the unmount pathway is unable to find the . I think using is a no-go. So we need to use something sufficiently unique (like ) for the ( above) and fix up some of the other codepaths to either use different "well-known paths" to write to, or to fix up path generation code so they give us the correct locations. I'm also going to check to see whether 's refactor fixes this issue (because it might since it translates things earlier)
OK. Really good news. your refactor in its current state actually fixes this issue if we set the to something unique here: Looks like the early translation is also making the pathway from the "correct" dir for inline migration. So I will create a PR that fixes this issue that should merge after

return &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ <del> // A.K.A InnerVolumeSpecName required to match for Unmount Name: volume.Name, </del> <ins> // Must be unique per disk as it is used as the unique part of the // staging path Name: fmt.Sprintf("%s-%s", GCEPDDriverName, pdSource.PDName), </ins> }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{


Question:
Minimal Repro: Create 2 different disks Create 2 Pods each using one of the disks Start both pods on the same node You will notice one of the disks gets ed to a path: , then both disks are ed from to a target path: which is similar but at least uses the pod . Both of these volumes are being treated as the same volume in the Kubelet even though they are definitely not the same volume. Note: it may not be "treated" as the same volume - it may just be that since the paths are the same the following volumes see that the path is already a mountpoint and skip the step. This is the more likely scenario based on observations. The value is the "name" of the inline-volume. These can obviously be the same across different pods - we should actually be placing it in a more unique directory (based off the translated maybe?). /cc
/sig storage
The inline volume example I'm using (ignore the fact that is a complete misnomer): I've confirmed that changing the "name" makes inline volumes work well since there is no longer a collision on the paths
/assign
Alright, problem is sort of twofold: The referred to above for inline-volumes is set here: As we can see from the comment: it is set to the because the in-tree CSI Plugin makes a lot of naming assumptions based on the which for PVC/PVs is the PV name (unique in a cluster). Primarily, during unmount the CSI Plugin tries to from a "well known ": which happens to be some path based on the which for the inline volume is just the ( above). So we run into the problem that if we use the directory is not sufficiently unique for different disks. And if we use something else like then the unmount pathway is unable to find the . I think using is a no-go. So we need to use something sufficiently unique (like ) for the ( above) and fix up some of the other codepaths to either use different "well-known paths" to write to, or to fix up path generation code so they give us the correct locations. I'm also going to check to see whether 's refactor fixes this issue (because it might since it translates things earlier)
OK. Really good news. your refactor in its current state actually fixes this issue if we set the to something unique here: Looks like the early translation is also making the pathway from the "correct" dir for inline migration. So I will create a PR that fixes this issue that should merge after

cinderSource := volume.Cinder pv := &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ <del> // A.K.A InnerVolumeSpecName required to match for Unmount Name: volume.Name, </del> <ins> // Must be unique per disk as it is used as the unique part of the // staging path Name: fmt.Sprintf("%s-%s", CinderDriverName, cinderSource.VolumeID), </ins> }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{


Question:
Minimal Repro: Create 2 different disks Create 2 Pods each using one of the disks Start both pods on the same node You will notice one of the disks gets ed to a path: , then both disks are ed from to a target path: which is similar but at least uses the pod . Both of these volumes are being treated as the same volume in the Kubelet even though they are definitely not the same volume. Note: it may not be "treated" as the same volume - it may just be that since the paths are the same the following volumes see that the path is already a mountpoint and skip the step. This is the more likely scenario based on observations. The value is the "name" of the inline-volume. These can obviously be the same across different pods - we should actually be placing it in a more unique directory (based off the translated maybe?). /cc
/sig storage
The inline volume example I'm using (ignore the fact that is a complete misnomer): I've confirmed that changing the "name" makes inline volumes work well since there is no longer a collision on the paths
/assign
Alright, problem is sort of twofold: The referred to above for inline-volumes is set here: As we can see from the comment: it is set to the because the in-tree CSI Plugin makes a lot of naming assumptions based on the which for PVC/PVs is the PV name (unique in a cluster). Primarily, during unmount the CSI Plugin tries to from a "well known ": which happens to be some path based on the which for the inline volume is just the ( above). So we run into the problem that if we use the directory is not sufficiently unique for different disks. And if we use something else like then the unmount pathway is unable to find the . I think using is a no-go. So we need to use something sufficiently unique (like ) for the ( above) and fix up some of the other codepaths to either use different "well-known paths" to write to, or to fix up path generation code so they give us the correct locations. I'm also going to check to see whether 's refactor fixes this issue (because it might since it translates things earlier)
OK. Really good news. your refactor in its current state actually fixes this issue if we set the to something unique here: Looks like the early translation is also making the pathway from the "correct" dir for inline migration. So I will create a PR that fixes this issue that should merge after

package csitranslation import ( <ins> "fmt" </ins> "reflect" "testing" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/uuid" </ins> "k8s.io/csi-translation-lib/plugins" )


Question:
Minimal Repro: Create 2 different disks Create 2 Pods each using one of the disks Start both pods on the same node You will notice one of the disks gets ed to a path: , then both disks are ed from to a target path: which is similar but at least uses the pod . Both of these volumes are being treated as the same volume in the Kubelet even though they are definitely not the same volume. Note: it may not be "treated" as the same volume - it may just be that since the paths are the same the following volumes see that the path is already a mountpoint and skip the step. This is the more likely scenario based on observations. The value is the "name" of the inline-volume. These can obviously be the same across different pods - we should actually be placing it in a more unique directory (based off the translated maybe?). /cc
/sig storage
The inline volume example I'm using (ignore the fact that is a complete misnomer): I've confirmed that changing the "name" makes inline volumes work well since there is no longer a collision on the paths
/assign
Alright, problem is sort of twofold: The referred to above for inline-volumes is set here: As we can see from the comment: it is set to the because the in-tree CSI Plugin makes a lot of naming assumptions based on the which for PVC/PVs is the PV name (unique in a cluster). Primarily, during unmount the CSI Plugin tries to from a "well known ": which happens to be some path based on the which for the inline volume is just the ( above). So we run into the problem that if we use the directory is not sufficiently unique for different disks. And if we use something else like then the unmount pathway is unable to find the . I think using is a no-go. So we need to use something sufficiently unique (like ) for the ( above) and fix up some of the other codepaths to either use different "well-known paths" to write to, or to fix up path generation code so they give us the correct locations. I'm also going to check to see whether 's refactor fixes this issue (because it might since it translates things earlier)
OK. Really good news. your refactor in its current state actually fixes this issue if we set the to something unique here: Looks like the early translation is also making the pathway from the "correct" dir for inline migration. So I will create a PR that fixes this issue that should merge after

} } <ins> func TestTranslateInTreeInlineVolumeToCSINameUniqueness(t *testing.T) { for driverName := range inTreePlugins { t.Run(driverName, func(t *testing.T) { ctl := New() vs1, err := generateUniqueVolumeSource(driverName) if err != nil { t.Fatalf("Couldn't generate random source: %v", err) } pv1, err := ctl.TranslateInTreeInlineVolumeToCSI(&v1.Volume{ VolumeSource: vs1, }) if err != nil { t.Fatalf("Error when translating to CSI: %v", err) } vs2, err := generateUniqueVolumeSource(driverName) if err != nil { t.Fatalf("Couldn't generate random source: %v", err) } pv2, err := ctl.TranslateInTreeInlineVolumeToCSI(&v1.Volume{ VolumeSource: vs2, }) if err != nil { t.Fatalf("Error when translating to CSI: %v", err) } if pv1 == nil || pv2 == nil { t.Fatalf("Did not expect either pv1: %v or pv2: %v to be nil", pv1, pv2) } if pv1.Name == pv2.Name { t.Errorf("PV name %s not sufficiently unique for different volumes", pv1.Name) } }) } } func generateUniqueVolumeSource(driverName string) (v1.VolumeSource, error) { switch driverName { case plugins.GCEPDDriverName: return v1.VolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName: string(uuid.NewUUID()), }, }, nil case plugins.AWSEBSDriverName: return v1.VolumeSource{ AWSElasticBlockStore: &v1.AWSElasticBlockStoreVolumeSource{ VolumeID: string(uuid.NewUUID()), }, }, nil case plugins.CinderDriverName: return v1.VolumeSource{ Cinder: &v1.CinderVolumeSource{ VolumeID: string(uuid.NewUUID()), }, }, nil case plugins.AzureDiskDriverName: return v1.VolumeSource{ AzureDisk: &v1.AzureDiskVolumeSource{ DiskName:    string(uuid.NewUUID()), DataDiskURI: string(uuid.NewUUID()), }, }, nil case plugins.AzureFileDriverName: return v1.VolumeSource{ AzureFile: &v1.AzureFileVolumeSource{ SecretName: string(uuid.NewUUID()), ShareName:  string(uuid.NewUUID()), }, }, nil default: return v1.VolumeSource{}, fmt.Errorf("couldn't find logic for driver: %v", driverName) } } </ins> func TestPluginNameMappings(t *testing.T) { testCases := []struct { name             string


Question:
What happened: We were running a service with LoadBalancer type with AWS cloud provider and after deleting this service, the cleanup process on AWS removed a wrong ingress input rule which provoked that all traffic between nodes was unable to reach out the destination. The service we were running had a configuration similar to: After applying this manfiest on the cluster, the cloud provider created the associated load balancer with two security groups: sg- (due to extra security group annotation) sg- (a new SG created by the AWS cloud provider with the rules defined on the ) Cloud provider also created a new input rule on the nodes security group which allows all traffic from all ports from (which is the sg created for the newly created load balancer in our example). One particularity of our setup is that the security group we as was the nodes security group. The input rules of this security group after the creation of this service was similar to: Everything worked fine with this setup until we deleted this service and the cloud provider started the deletion process of the LoadBalancer on AWS. We've been debugging the deletion code and we found that the first thing it does is cleaning up the security group. To do it, it first removes the input rule from the nodes security group which allows traffic from the load balancer security group (sg-, on this example) and then, as it's not referenced, it removes the security group. The expected output of this process, would be that nodes security groups would be: But, in our case, it ended up as: which dropped all the traffic between nodes (as it didn't allow any traffic between them). The cleanup process didn't properly remove the input rule from the node security group (the one which allows all traffic from source sg-) and it also did not remove the sg- security group. In our case, as we were using the nodes security group on the extra security groups, this bug provoked a network partition between nodes, but we think that if is used, the cleanup process may be broken and have unexpected side-effects like those we observed. We've been debugging and we found that the process which removes the input rule from the nodes security groups starts here: The first thing this function does is this: : When extra security groups are used, will contain the security group created for the LoadBalancer and all the extra security groups. We also found that AWS returns this SecurityGroups list in distinct orders after changing the LoadBalancer configuration. That code is always assuming that latest security group on the list is the , which may not be true when extra security groups are used. On the cases where the load balancer security group is not returned on the latest position, the cleanup process does not remove the security group and the input rule from the nodes security group is not removed as expected. We've seen that master branch still contains this code and we think this may be hapenning on newer versions: ), so we think that newer versions may be affected by this bug (we didn't try it). In our case, we saw several times this log during the removal process: Additionally, the code tried to remove the security group (but being unable to do so), after wrongly revoking the rule we have talked about before. What you expected to happen: The cleanup process should properly detect the load balancer security group, remove the input rule from nodes security group and remove the load balancer security group, regardless of the order in which AWS returns the security group list on . How to reproduce it (as minimally and precisely as possible): We did not find any consistent process to create the scenario, as it happens randomly (we are not sure if due to AWS or any kind of race condition on AWS cloud provider). Setup: Create a Service with running controller manager with AWS cloud provider with tha annotation containing one or more security groups (as the one the section) Wait until load balancer is created Run Check if does not have the newly created security group for the balancer in the latest position. Repeat until the balancer does not have the balancer security group on the latest position Trigger the bug: Result:The log will appear multiple times on controller-manager logs Load balancer associated security group is not removed Nodes security group input rule allowing all traffic from load balancer security group is not removed If the latest security group on the list was the nodes security group, the input rule allowing all traffic on all ports from all the other nodes is removed and no traffic is allowed between nodes on the cluster. Environment: Kubernetes version (use ): version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: aws OS (e.g: ):Kernel (e.g. ): Linux ip-10-2-13-67 4.9.0-7-amd64 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x8664 GNU/Linux Install tools: kops
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area provider/aws /lifecycle frozen
Working on a repro for this issue (on 1.11 first, then I'll try on master) and I've not quite observed the same behavior yet. Depending on the order of the security groups on the load balancer, I only see the ingress rule for the new LB security group to the node security group if the new security group comes last. Otherwise, if the NODE security group comes last, the ingress rule is not , and I see in the logs. Even without a full repro yet, I find it troubling that behavior is dependent on the order of security groups attached to the load balancer. One question I have related to your setup, can you describe the reason why you're adding the node security group as an ? I'd like to understand your use case a little better.
Hi Nicholas. I have a working patch for this specific issue: This bug was the subject of a talk for a local meetup I did recently. I also published a personal blog post talking about it: There you can see more details about how it can be reproduced easily. If you think the patch is valid, I'll try to add a few test cases for it and then propose the change upstream. Thanks!
It has been a while since we got this error and I'm not sure if we saw this error. What I experienced is that the order is changed by Kubernetes or by some call to AWS. It took me not more than 3 attempts to get this behavior. There's something I can do to help you reproduce the error? Do you need more details of our setup? This was a misconfiguration on our side. When we noticed that this setup was not correct we removed this security group from , which removed the mentioned security group rule and took down the Kubernetes cluster (no communication between nodes was allowed).

return sgList, setupSg, nil } <ins> // sortELBSecurityGroupList returns a list of sorted securityGroupIDs based on the original order // from buildELBSecurityGroupList. The logic is: //  * securityGroups specified by ServiceAnnotationLoadBalancerSecurityGroups appears first in order //  * securityGroups specified by ServiceAnnotationLoadBalancerExtraSecurityGroups appears last in order func (c *Cloud) sortELBSecurityGroupList(securityGroupIDs []string, annotations map[string]string) { annotatedSGList := getSGListFromAnnotation(annotations[ServiceAnnotationLoadBalancerSecurityGroups]) annotatedExtraSGList := getSGListFromAnnotation(annotations[ServiceAnnotationLoadBalancerExtraSecurityGroups]) annotatedSGIndex := make(map[string]int, len(annotatedSGList)) annotatedExtraSGIndex := make(map[string]int, len(annotatedExtraSGList)) for i, sgID := range annotatedSGList { annotatedSGIndex[sgID] = i } for i, sgID := range annotatedExtraSGList { annotatedExtraSGIndex[sgID] = i } sgOrderMapping := make(map[string]int, len(securityGroupIDs)) for _, sgID := range securityGroupIDs { if i, ok := annotatedSGIndex[sgID]; ok { sgOrderMapping[sgID] = i } else if j, ok := annotatedExtraSGIndex[sgID]; ok { sgOrderMapping[sgID] = len(annotatedSGIndex) + 1 + j } else { sgOrderMapping[sgID] = len(annotatedSGIndex) } } sort.Slice(securityGroupIDs, func(i, j int) bool { return sgOrderMapping[securityGroupIDs[i]] < sgOrderMapping[securityGroupIDs[j]] }) } </ins> // buildListener creates a new listener from the given port, adding an SSL certificate // if indicated by the appropriate annotations. func buildListener(port v1.ServicePort, annotations map[string]string, sslPorts *portSets) (*elb.Listener, error) {


Question:
What happened: We were running a service with LoadBalancer type with AWS cloud provider and after deleting this service, the cleanup process on AWS removed a wrong ingress input rule which provoked that all traffic between nodes was unable to reach out the destination. The service we were running had a configuration similar to: After applying this manfiest on the cluster, the cloud provider created the associated load balancer with two security groups: sg- (due to extra security group annotation) sg- (a new SG created by the AWS cloud provider with the rules defined on the ) Cloud provider also created a new input rule on the nodes security group which allows all traffic from all ports from (which is the sg created for the newly created load balancer in our example). One particularity of our setup is that the security group we as was the nodes security group. The input rules of this security group after the creation of this service was similar to: Everything worked fine with this setup until we deleted this service and the cloud provider started the deletion process of the LoadBalancer on AWS. We've been debugging the deletion code and we found that the first thing it does is cleaning up the security group. To do it, it first removes the input rule from the nodes security group which allows traffic from the load balancer security group (sg-, on this example) and then, as it's not referenced, it removes the security group. The expected output of this process, would be that nodes security groups would be: But, in our case, it ended up as: which dropped all the traffic between nodes (as it didn't allow any traffic between them). The cleanup process didn't properly remove the input rule from the node security group (the one which allows all traffic from source sg-) and it also did not remove the sg- security group. In our case, as we were using the nodes security group on the extra security groups, this bug provoked a network partition between nodes, but we think that if is used, the cleanup process may be broken and have unexpected side-effects like those we observed. We've been debugging and we found that the process which removes the input rule from the nodes security groups starts here: The first thing this function does is this: : When extra security groups are used, will contain the security group created for the LoadBalancer and all the extra security groups. We also found that AWS returns this SecurityGroups list in distinct orders after changing the LoadBalancer configuration. That code is always assuming that latest security group on the list is the , which may not be true when extra security groups are used. On the cases where the load balancer security group is not returned on the latest position, the cleanup process does not remove the security group and the input rule from the nodes security group is not removed as expected. We've seen that master branch still contains this code and we think this may be hapenning on newer versions: ), so we think that newer versions may be affected by this bug (we didn't try it). In our case, we saw several times this log during the removal process: Additionally, the code tried to remove the security group (but being unable to do so), after wrongly revoking the rule we have talked about before. What you expected to happen: The cleanup process should properly detect the load balancer security group, remove the input rule from nodes security group and remove the load balancer security group, regardless of the order in which AWS returns the security group list on . How to reproduce it (as minimally and precisely as possible): We did not find any consistent process to create the scenario, as it happens randomly (we are not sure if due to AWS or any kind of race condition on AWS cloud provider). Setup: Create a Service with running controller manager with AWS cloud provider with tha annotation containing one or more security groups (as the one the section) Wait until load balancer is created Run Check if does not have the newly created security group for the balancer in the latest position. Repeat until the balancer does not have the balancer security group on the latest position Trigger the bug: Result:The log will appear multiple times on controller-manager logs Load balancer associated security group is not removed Nodes security group input rule allowing all traffic from load balancer security group is not removed If the latest security group on the list was the nodes security group, the input rule allowing all traffic on all ports from all the other nodes is removed and no traffic is allowed between nodes on the cluster. Environment: Kubernetes version (use ): version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: aws OS (e.g: ):Kernel (e.g. ): Linux ip-10-2-13-67 4.9.0-7-amd64 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x8664 GNU/Linux Install tools: kops
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area provider/aws /lifecycle frozen
Working on a repro for this issue (on 1.11 first, then I'll try on master) and I've not quite observed the same behavior yet. Depending on the order of the security groups on the load balancer, I only see the ingress rule for the new LB security group to the node security group if the new security group comes last. Otherwise, if the NODE security group comes last, the ingress rule is not , and I see in the logs. Even without a full repro yet, I find it troubling that behavior is dependent on the order of security groups attached to the load balancer. One question I have related to your setup, can you describe the reason why you're adding the node security group as an ? I'd like to understand your use case a little better.
Hi Nicholas. I have a working patch for this specific issue: This bug was the subject of a talk for a local meetup I did recently. I also published a personal blog post talking about it: There you can see more details about how it can be reproduced easily. If you think the patch is valid, I'll try to add a few test cases for it and then propose the change upstream. Thanks!
It has been a while since we got this error and I'm not sure if we saw this error. What I experienced is that the order is changed by Kubernetes or by some call to AWS. It took me not more than 3 attempts to get this behavior. There's something I can do to help you reproduce the error? Do you need more details of our setup? This was a misconfiguration on our side. When we noticed that this setup was not correct we removed this security group from , which removed the mentioned security group rule and took down the Kubernetes cluster (no communication between nodes was allowed).

} } <del> err = c.updateInstanceSecurityGroupsForLoadBalancer(loadBalancer, instances) </del> <ins> err = c.updateInstanceSecurityGroupsForLoadBalancer(loadBalancer, instances, annotations) </ins> if err != nil { klog.Warningf("Error opening ingress rules for the load balancer to the instances: %q", err) return nil, err


Question:
What happened: We were running a service with LoadBalancer type with AWS cloud provider and after deleting this service, the cleanup process on AWS removed a wrong ingress input rule which provoked that all traffic between nodes was unable to reach out the destination. The service we were running had a configuration similar to: After applying this manfiest on the cluster, the cloud provider created the associated load balancer with two security groups: sg- (due to extra security group annotation) sg- (a new SG created by the AWS cloud provider with the rules defined on the ) Cloud provider also created a new input rule on the nodes security group which allows all traffic from all ports from (which is the sg created for the newly created load balancer in our example). One particularity of our setup is that the security group we as was the nodes security group. The input rules of this security group after the creation of this service was similar to: Everything worked fine with this setup until we deleted this service and the cloud provider started the deletion process of the LoadBalancer on AWS. We've been debugging the deletion code and we found that the first thing it does is cleaning up the security group. To do it, it first removes the input rule from the nodes security group which allows traffic from the load balancer security group (sg-, on this example) and then, as it's not referenced, it removes the security group. The expected output of this process, would be that nodes security groups would be: But, in our case, it ended up as: which dropped all the traffic between nodes (as it didn't allow any traffic between them). The cleanup process didn't properly remove the input rule from the node security group (the one which allows all traffic from source sg-) and it also did not remove the sg- security group. In our case, as we were using the nodes security group on the extra security groups, this bug provoked a network partition between nodes, but we think that if is used, the cleanup process may be broken and have unexpected side-effects like those we observed. We've been debugging and we found that the process which removes the input rule from the nodes security groups starts here: The first thing this function does is this: : When extra security groups are used, will contain the security group created for the LoadBalancer and all the extra security groups. We also found that AWS returns this SecurityGroups list in distinct orders after changing the LoadBalancer configuration. That code is always assuming that latest security group on the list is the , which may not be true when extra security groups are used. On the cases where the load balancer security group is not returned on the latest position, the cleanup process does not remove the security group and the input rule from the nodes security group is not removed as expected. We've seen that master branch still contains this code and we think this may be hapenning on newer versions: ), so we think that newer versions may be affected by this bug (we didn't try it). In our case, we saw several times this log during the removal process: Additionally, the code tried to remove the security group (but being unable to do so), after wrongly revoking the rule we have talked about before. What you expected to happen: The cleanup process should properly detect the load balancer security group, remove the input rule from nodes security group and remove the load balancer security group, regardless of the order in which AWS returns the security group list on . How to reproduce it (as minimally and precisely as possible): We did not find any consistent process to create the scenario, as it happens randomly (we are not sure if due to AWS or any kind of race condition on AWS cloud provider). Setup: Create a Service with running controller manager with AWS cloud provider with tha annotation containing one or more security groups (as the one the section) Wait until load balancer is created Run Check if does not have the newly created security group for the balancer in the latest position. Repeat until the balancer does not have the balancer security group on the latest position Trigger the bug: Result:The log will appear multiple times on controller-manager logs Load balancer associated security group is not removed Nodes security group input rule allowing all traffic from load balancer security group is not removed If the latest security group on the list was the nodes security group, the input rule allowing all traffic on all ports from all the other nodes is removed and no traffic is allowed between nodes on the cluster. Environment: Kubernetes version (use ): version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: aws OS (e.g: ):Kernel (e.g. ): Linux ip-10-2-13-67 4.9.0-7-amd64 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x8664 GNU/Linux Install tools: kops
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area provider/aws /lifecycle frozen
Working on a repro for this issue (on 1.11 first, then I'll try on master) and I've not quite observed the same behavior yet. Depending on the order of the security groups on the load balancer, I only see the ingress rule for the new LB security group to the node security group if the new security group comes last. Otherwise, if the NODE security group comes last, the ingress rule is not , and I see in the logs. Even without a full repro yet, I find it troubling that behavior is dependent on the order of security groups attached to the load balancer. One question I have related to your setup, can you describe the reason why you're adding the node security group as an ? I'd like to understand your use case a little better.
Hi Nicholas. I have a working patch for this specific issue: This bug was the subject of a talk for a local meetup I did recently. I also published a personal blog post talking about it: There you can see more details about how it can be reproduced easily. If you think the patch is valid, I'll try to add a few test cases for it and then propose the change upstream. Thanks!
It has been a while since we got this error and I'm not sure if we saw this error. What I experienced is that the order is changed by Kubernetes or by some call to AWS. It took me not more than 3 attempts to get this behavior. There's something I can do to help you reproduce the error? Do you need more details of our setup? This was a misconfiguration on our side. When we noticed that this setup was not correct we removed this security group from , which removed the mentioned security group rule and took down the Kubernetes cluster (no communication between nodes was allowed).

// Open security group ingress rules on the instances so that the load balancer can talk to them // Will also remove any security groups ingress rules for the load balancer that are _not_ needed for allInstances <del> func (c *Cloud) updateInstanceSecurityGroupsForLoadBalancer(lb *elb.LoadBalancerDescription, instances map[InstanceID]*ec2.Instance) error { </del> <ins> func (c *Cloud) updateInstanceSecurityGroupsForLoadBalancer(lb *elb.LoadBalancerDescription, instances map[InstanceID]*ec2.Instance, annotations map[string]string) error { </ins> if c.cfg.Global.DisableSecurityGroupIngress { return nil } // Determine the load balancer security group id <del> loadBalancerSecurityGroupID := "" for _, securityGroup := range lb.SecurityGroups { if aws.StringValue(securityGroup) == "" { continue } if loadBalancerSecurityGroupID != "" { // We create LBs with one SG klog.Warningf("Multiple security groups for load balancer: %q", aws.StringValue(lb.LoadBalancerName)) } loadBalancerSecurityGroupID = *securityGroup } if loadBalancerSecurityGroupID == "" { </del> <ins> lbSecurityGroupIDs := aws.StringValueSlice(lb.SecurityGroups) if len(lbSecurityGroupIDs) == 0 { </ins> return fmt.Errorf("could not determine security group for load balancer: %s", aws.StringValue(lb.LoadBalancerName)) } <ins> c.sortELBSecurityGroupList(lbSecurityGroupIDs, annotations) loadBalancerSecurityGroupID := lbSecurityGroupIDs[0] </ins> // Get the actual list of groups that allow ingress from the load-balancer var actualGroups []*ec2.SecurityGroup


Question:
What happened: We were running a service with LoadBalancer type with AWS cloud provider and after deleting this service, the cleanup process on AWS removed a wrong ingress input rule which provoked that all traffic between nodes was unable to reach out the destination. The service we were running had a configuration similar to: After applying this manfiest on the cluster, the cloud provider created the associated load balancer with two security groups: sg- (due to extra security group annotation) sg- (a new SG created by the AWS cloud provider with the rules defined on the ) Cloud provider also created a new input rule on the nodes security group which allows all traffic from all ports from (which is the sg created for the newly created load balancer in our example). One particularity of our setup is that the security group we as was the nodes security group. The input rules of this security group after the creation of this service was similar to: Everything worked fine with this setup until we deleted this service and the cloud provider started the deletion process of the LoadBalancer on AWS. We've been debugging the deletion code and we found that the first thing it does is cleaning up the security group. To do it, it first removes the input rule from the nodes security group which allows traffic from the load balancer security group (sg-, on this example) and then, as it's not referenced, it removes the security group. The expected output of this process, would be that nodes security groups would be: But, in our case, it ended up as: which dropped all the traffic between nodes (as it didn't allow any traffic between them). The cleanup process didn't properly remove the input rule from the node security group (the one which allows all traffic from source sg-) and it also did not remove the sg- security group. In our case, as we were using the nodes security group on the extra security groups, this bug provoked a network partition between nodes, but we think that if is used, the cleanup process may be broken and have unexpected side-effects like those we observed. We've been debugging and we found that the process which removes the input rule from the nodes security groups starts here: The first thing this function does is this: : When extra security groups are used, will contain the security group created for the LoadBalancer and all the extra security groups. We also found that AWS returns this SecurityGroups list in distinct orders after changing the LoadBalancer configuration. That code is always assuming that latest security group on the list is the , which may not be true when extra security groups are used. On the cases where the load balancer security group is not returned on the latest position, the cleanup process does not remove the security group and the input rule from the nodes security group is not removed as expected. We've seen that master branch still contains this code and we think this may be hapenning on newer versions: ), so we think that newer versions may be affected by this bug (we didn't try it). In our case, we saw several times this log during the removal process: Additionally, the code tried to remove the security group (but being unable to do so), after wrongly revoking the rule we have talked about before. What you expected to happen: The cleanup process should properly detect the load balancer security group, remove the input rule from nodes security group and remove the load balancer security group, regardless of the order in which AWS returns the security group list on . How to reproduce it (as minimally and precisely as possible): We did not find any consistent process to create the scenario, as it happens randomly (we are not sure if due to AWS or any kind of race condition on AWS cloud provider). Setup: Create a Service with running controller manager with AWS cloud provider with tha annotation containing one or more security groups (as the one the section) Wait until load balancer is created Run Check if does not have the newly created security group for the balancer in the latest position. Repeat until the balancer does not have the balancer security group on the latest position Trigger the bug: Result:The log will appear multiple times on controller-manager logs Load balancer associated security group is not removed Nodes security group input rule allowing all traffic from load balancer security group is not removed If the latest security group on the list was the nodes security group, the input rule allowing all traffic on all ports from all the other nodes is removed and no traffic is allowed between nodes on the cluster. Environment: Kubernetes version (use ): version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: aws OS (e.g: ):Kernel (e.g. ): Linux ip-10-2-13-67 4.9.0-7-amd64 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x8664 GNU/Linux Install tools: kops
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area provider/aws /lifecycle frozen
Working on a repro for this issue (on 1.11 first, then I'll try on master) and I've not quite observed the same behavior yet. Depending on the order of the security groups on the load balancer, I only see the ingress rule for the new LB security group to the node security group if the new security group comes last. Otherwise, if the NODE security group comes last, the ingress rule is not , and I see in the logs. Even without a full repro yet, I find it troubling that behavior is dependent on the order of security groups attached to the load balancer. One question I have related to your setup, can you describe the reason why you're adding the node security group as an ? I'd like to understand your use case a little better.
Hi Nicholas. I have a working patch for this specific issue: This bug was the subject of a talk for a local meetup I did recently. I also published a personal blog post talking about it: There you can see more details about how it can be reproduced easily. If you think the patch is valid, I'll try to add a few test cases for it and then propose the change upstream. Thanks!
It has been a while since we got this error and I'm not sure if we saw this error. What I experienced is that the order is changed by Kubernetes or by some call to AWS. It took me not more than 3 attempts to get this behavior. There's something I can do to help you reproduce the error? Do you need more details of our setup? This was a misconfiguration on our side. When we noticed that this setup was not correct we removed this security group from , which removed the mentioned security group rule and took down the Kubernetes cluster (no communication between nodes was allowed).

{ // De-authorize the load balancer security group from the instances security group <del> err = c.updateInstanceSecurityGroupsForLoadBalancer(lb, nil) </del> <ins> err = c.updateInstanceSecurityGroupsForLoadBalancer(lb, nil, service.Annotations) </ins> if err != nil { klog.Errorf("Error deregistering load balancer from instance security groups: %q", err) return err


Question:
What happened: We were running a service with LoadBalancer type with AWS cloud provider and after deleting this service, the cleanup process on AWS removed a wrong ingress input rule which provoked that all traffic between nodes was unable to reach out the destination. The service we were running had a configuration similar to: After applying this manfiest on the cluster, the cloud provider created the associated load balancer with two security groups: sg- (due to extra security group annotation) sg- (a new SG created by the AWS cloud provider with the rules defined on the ) Cloud provider also created a new input rule on the nodes security group which allows all traffic from all ports from (which is the sg created for the newly created load balancer in our example). One particularity of our setup is that the security group we as was the nodes security group. The input rules of this security group after the creation of this service was similar to: Everything worked fine with this setup until we deleted this service and the cloud provider started the deletion process of the LoadBalancer on AWS. We've been debugging the deletion code and we found that the first thing it does is cleaning up the security group. To do it, it first removes the input rule from the nodes security group which allows traffic from the load balancer security group (sg-, on this example) and then, as it's not referenced, it removes the security group. The expected output of this process, would be that nodes security groups would be: But, in our case, it ended up as: which dropped all the traffic between nodes (as it didn't allow any traffic between them). The cleanup process didn't properly remove the input rule from the node security group (the one which allows all traffic from source sg-) and it also did not remove the sg- security group. In our case, as we were using the nodes security group on the extra security groups, this bug provoked a network partition between nodes, but we think that if is used, the cleanup process may be broken and have unexpected side-effects like those we observed. We've been debugging and we found that the process which removes the input rule from the nodes security groups starts here: The first thing this function does is this: : When extra security groups are used, will contain the security group created for the LoadBalancer and all the extra security groups. We also found that AWS returns this SecurityGroups list in distinct orders after changing the LoadBalancer configuration. That code is always assuming that latest security group on the list is the , which may not be true when extra security groups are used. On the cases where the load balancer security group is not returned on the latest position, the cleanup process does not remove the security group and the input rule from the nodes security group is not removed as expected. We've seen that master branch still contains this code and we think this may be hapenning on newer versions: ), so we think that newer versions may be affected by this bug (we didn't try it). In our case, we saw several times this log during the removal process: Additionally, the code tried to remove the security group (but being unable to do so), after wrongly revoking the rule we have talked about before. What you expected to happen: The cleanup process should properly detect the load balancer security group, remove the input rule from nodes security group and remove the load balancer security group, regardless of the order in which AWS returns the security group list on . How to reproduce it (as minimally and precisely as possible): We did not find any consistent process to create the scenario, as it happens randomly (we are not sure if due to AWS or any kind of race condition on AWS cloud provider). Setup: Create a Service with running controller manager with AWS cloud provider with tha annotation containing one or more security groups (as the one the section) Wait until load balancer is created Run Check if does not have the newly created security group for the balancer in the latest position. Repeat until the balancer does not have the balancer security group on the latest position Trigger the bug: Result:The log will appear multiple times on controller-manager logs Load balancer associated security group is not removed Nodes security group input rule allowing all traffic from load balancer security group is not removed If the latest security group on the list was the nodes security group, the input rule allowing all traffic on all ports from all the other nodes is removed and no traffic is allowed between nodes on the cluster. Environment: Kubernetes version (use ): version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: aws OS (e.g: ):Kernel (e.g. ): Linux ip-10-2-13-67 4.9.0-7-amd64 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x8664 GNU/Linux Install tools: kops
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area provider/aws /lifecycle frozen
Working on a repro for this issue (on 1.11 first, then I'll try on master) and I've not quite observed the same behavior yet. Depending on the order of the security groups on the load balancer, I only see the ingress rule for the new LB security group to the node security group if the new security group comes last. Otherwise, if the NODE security group comes last, the ingress rule is not , and I see in the logs. Even without a full repro yet, I find it troubling that behavior is dependent on the order of security groups attached to the load balancer. One question I have related to your setup, can you describe the reason why you're adding the node security group as an ? I'd like to understand your use case a little better.
Hi Nicholas. I have a working patch for this specific issue: This bug was the subject of a talk for a local meetup I did recently. I also published a personal blog post talking about it: There you can see more details about how it can be reproduced easily. If you think the patch is valid, I'll try to add a few test cases for it and then propose the change upstream. Thanks!
It has been a while since we got this error and I'm not sure if we saw this error. What I experienced is that the order is changed by Kubernetes or by some call to AWS. It took me not more than 3 attempts to get this behavior. There's something I can do to help you reproduce the error? Do you need more details of our setup? This was a misconfiguration on our side. When we noticed that this setup was not correct we removed this security group from , which removed the mentioned security group rule and took down the Kubernetes cluster (no communication between nodes was allowed).

return nil } <del> err = c.updateInstanceSecurityGroupsForLoadBalancer(lb, instances) </del> <ins> err = c.updateInstanceSecurityGroupsForLoadBalancer(lb, instances, service.Annotations) </ins> if err != nil { return err }


Question:
What happened: We were running a service with LoadBalancer type with AWS cloud provider and after deleting this service, the cleanup process on AWS removed a wrong ingress input rule which provoked that all traffic between nodes was unable to reach out the destination. The service we were running had a configuration similar to: After applying this manfiest on the cluster, the cloud provider created the associated load balancer with two security groups: sg- (due to extra security group annotation) sg- (a new SG created by the AWS cloud provider with the rules defined on the ) Cloud provider also created a new input rule on the nodes security group which allows all traffic from all ports from (which is the sg created for the newly created load balancer in our example). One particularity of our setup is that the security group we as was the nodes security group. The input rules of this security group after the creation of this service was similar to: Everything worked fine with this setup until we deleted this service and the cloud provider started the deletion process of the LoadBalancer on AWS. We've been debugging the deletion code and we found that the first thing it does is cleaning up the security group. To do it, it first removes the input rule from the nodes security group which allows traffic from the load balancer security group (sg-, on this example) and then, as it's not referenced, it removes the security group. The expected output of this process, would be that nodes security groups would be: But, in our case, it ended up as: which dropped all the traffic between nodes (as it didn't allow any traffic between them). The cleanup process didn't properly remove the input rule from the node security group (the one which allows all traffic from source sg-) and it also did not remove the sg- security group. In our case, as we were using the nodes security group on the extra security groups, this bug provoked a network partition between nodes, but we think that if is used, the cleanup process may be broken and have unexpected side-effects like those we observed. We've been debugging and we found that the process which removes the input rule from the nodes security groups starts here: The first thing this function does is this: : When extra security groups are used, will contain the security group created for the LoadBalancer and all the extra security groups. We also found that AWS returns this SecurityGroups list in distinct orders after changing the LoadBalancer configuration. That code is always assuming that latest security group on the list is the , which may not be true when extra security groups are used. On the cases where the load balancer security group is not returned on the latest position, the cleanup process does not remove the security group and the input rule from the nodes security group is not removed as expected. We've seen that master branch still contains this code and we think this may be hapenning on newer versions: ), so we think that newer versions may be affected by this bug (we didn't try it). In our case, we saw several times this log during the removal process: Additionally, the code tried to remove the security group (but being unable to do so), after wrongly revoking the rule we have talked about before. What you expected to happen: The cleanup process should properly detect the load balancer security group, remove the input rule from nodes security group and remove the load balancer security group, regardless of the order in which AWS returns the security group list on . How to reproduce it (as minimally and precisely as possible): We did not find any consistent process to create the scenario, as it happens randomly (we are not sure if due to AWS or any kind of race condition on AWS cloud provider). Setup: Create a Service with running controller manager with AWS cloud provider with tha annotation containing one or more security groups (as the one the section) Wait until load balancer is created Run Check if does not have the newly created security group for the balancer in the latest position. Repeat until the balancer does not have the balancer security group on the latest position Trigger the bug: Result:The log will appear multiple times on controller-manager logs Load balancer associated security group is not removed Nodes security group input rule allowing all traffic from load balancer security group is not removed If the latest security group on the list was the nodes security group, the input rule allowing all traffic on all ports from all the other nodes is removed and no traffic is allowed between nodes on the cluster. Environment: Kubernetes version (use ): version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: aws OS (e.g: ):Kernel (e.g. ): Linux ip-10-2-13-67 4.9.0-7-amd64 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x8664 GNU/Linux Install tools: kops
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area provider/aws /lifecycle frozen
Working on a repro for this issue (on 1.11 first, then I'll try on master) and I've not quite observed the same behavior yet. Depending on the order of the security groups on the load balancer, I only see the ingress rule for the new LB security group to the node security group if the new security group comes last. Otherwise, if the NODE security group comes last, the ingress rule is not , and I see in the logs. Even without a full repro yet, I find it troubling that behavior is dependent on the order of security groups attached to the load balancer. One question I have related to your setup, can you describe the reason why you're adding the node security group as an ? I'd like to understand your use case a little better.
Hi Nicholas. I have a working patch for this specific issue: This bug was the subject of a talk for a local meetup I did recently. I also published a personal blog post talking about it: There you can see more details about how it can be reproduced easily. If you think the patch is valid, I'll try to add a few test cases for it and then propose the change upstream. Thanks!
It has been a while since we got this error and I'm not sure if we saw this error. What I experienced is that the order is changed by Kubernetes or by some call to AWS. It took me not more than 3 attempts to get this behavior. There's something I can do to help you reproduce the error? Do you need more details of our setup? This was a misconfiguration on our side. When we noticed that this setup was not correct we removed this security group from , which removed the mentioned security group rule and took down the Kubernetes cluster (no communication between nodes was allowed).

assert.Equal(t, testCase.region, result) } } <ins> func TestCloud_sortELBSecurityGroupList(t *testing.T) { type args struct { securityGroupIDs []string annotations      map[string]string } tests := []struct { name                 string args                 args wantSecurityGroupIDs []string }{ { name: "with no annotation", args: args{ securityGroupIDs: []string{"sg-1"}, annotations:      map[string]string{}, }, wantSecurityGroupIDs: []string{"sg-1"}, }, { name: "with service.beta.kubernetes.io/aws-load-balancer-security-groups", args: args{ securityGroupIDs: []string{"sg-2", "sg-1", "sg-3"}, annotations: map[string]string{ "service.beta.kubernetes.io/aws-load-balancer-security-groups": "sg-3,sg-2,sg-1", }, }, wantSecurityGroupIDs: []string{"sg-3", "sg-2", "sg-1"}, }, { name: "with service.beta.kubernetes.io/aws-load-balancer-extra-security-groups", args: args{ securityGroupIDs: []string{"sg-2", "sg-1", "sg-3", "sg-4"}, annotations: map[string]string{ "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups": "sg-3,sg-2,sg-1", }, }, wantSecurityGroupIDs: []string{"sg-4", "sg-3", "sg-2", "sg-1"}, }, { name: "with both annotation", args: args{ securityGroupIDs: []string{"sg-2", "sg-1", "sg-3", "sg-4", "sg-5", "sg-6"}, annotations: map[string]string{ "service.beta.kubernetes.io/aws-load-balancer-security-groups":       "sg-3,sg-2,sg-1", "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups": "sg-6,sg-5", }, }, wantSecurityGroupIDs: []string{"sg-3", "sg-2", "sg-1", "sg-4", "sg-6", "sg-5"}, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { c := &Cloud{} c.sortELBSecurityGroupList(tt.args.securityGroupIDs, tt.args.annotations) assert.Equal(t, tt.wantSecurityGroupIDs, tt.args.securityGroupIDs) }) } } </ins>


Question:
cc who found it. the reason is the calls from the import here: i suspect the problem surfaced with this bug fix. cc more refs: is an anti-pattern in this case. ideally metrics registration should not trigger warnings for a simple version output. mitigation on the user side is to pipe stderr to a null device (as long as klog is writing to stderr) /kind bug /priority important-longterm /area kubelet /wg component-standard
/sig node
should be fixed before 1.17.0, doesn't block beta.2
cc
Could this be caused by the following in init() of ? It was marked I wonder when deprecatedRequestLatency can be dropped.
looks like was deprecated in 1.14 - Is that the only one that needs to be dropped? are there more.
Dropping restclientrequestlatencyseconds should remove the warning for --version. In the 1.14 changelog, there were quite a few other metrics deprecated. I guess we don't have to drop them now.
/assign It's an interesting issue. I will investigate it and get back to you. By the way: I tried this morning, I got 4 warnings, things getting worse as we deprecated more metrics this weekend. (the log can not point the exact name of a metric, we intended to improve it in v1.18)
it's being invoked from here: the message is not helpful, since it doesn't tell you which metric is deprecated
Yes. It's a known issue and we intended to improve it in v1.18. Now, I suggest removing the log temporarily, Fired a PR , for reasons as follows: Too many binaries have been affected.(kubelet, kube-proxy, kube-apiserver, kube-controller-manager, kube-proxy, kube-scheduler). We should consider more about this issue. v1.17.0 is fast approaching, we should make a quick fix. All components try to register a deprecated metrics will be affected. Maybe we should record deprecated/hidden metrics by a new metric instead of the log.
Summary the root cause here: According to metrics deprecated policy, if a metric's - = 1, it will become a hidden metric. Hidden means we will not register it to the Prometheus registry by default. If we try to register a hidden metric to the stability framework, the warning log will be printed out. Because some metrics registered by init() calls from the import, so...
I have tried to print the metric's name at local: kubelet: kube-proxy: kube-apiserver: kube-controller-manager: kube-scheduler:
ideally the registration should happen outside of init(), which means calling it on demand. but i will defer to the owning sigs.
In principle, yes, I agree. Otherwise, we need to be very very careful with every log in the stability framework. I think we can track it and work out by iteration. cc for comments.

return } if shouldHide(&version, selfVersion) { <del> klog.Warningf("This metric(%s) has been deprecated for more than one release, hiding.", d.fqName) </del> <ins> // TODO(RainbowMango): Remove this log temporarily. https://github.com/kubernetes/kubernetes/issues/85369 // klog.Warningf("This metric(%s) has been deprecated for more than one release, hiding.", d.fqName) </ins> d.isHidden = true } })


Question:
cc who found it. the reason is the calls from the import here: i suspect the problem surfaced with this bug fix. cc more refs: is an anti-pattern in this case. ideally metrics registration should not trigger warnings for a simple version output. mitigation on the user side is to pipe stderr to a null device (as long as klog is writing to stderr) /kind bug /priority important-longterm /area kubelet /wg component-standard
/sig node
should be fixed before 1.17.0, doesn't block beta.2
cc
Could this be caused by the following in init() of ? It was marked I wonder when deprecatedRequestLatency can be dropped.
looks like was deprecated in 1.14 - Is that the only one that needs to be dropped? are there more.
Dropping restclientrequestlatencyseconds should remove the warning for --version. In the 1.14 changelog, there were quite a few other metrics deprecated. I guess we don't have to drop them now.
/assign It's an interesting issue. I will investigate it and get back to you. By the way: I tried this morning, I got 4 warnings, things getting worse as we deprecated more metrics this weekend. (the log can not point the exact name of a metric, we intended to improve it in v1.18)
it's being invoked from here: the message is not helpful, since it doesn't tell you which metric is deprecated
Yes. It's a known issue and we intended to improve it in v1.18. Now, I suggest removing the log temporarily, Fired a PR , for reasons as follows: Too many binaries have been affected.(kubelet, kube-proxy, kube-apiserver, kube-controller-manager, kube-proxy, kube-scheduler). We should consider more about this issue. v1.17.0 is fast approaching, we should make a quick fix. All components try to register a deprecated metrics will be affected. Maybe we should record deprecated/hidden metrics by a new metric instead of the log.
Summary the root cause here: According to metrics deprecated policy, if a metric's - = 1, it will become a hidden metric. Hidden means we will not register it to the Prometheus registry by default. If we try to register a hidden metric to the stability framework, the warning log will be printed out. Because some metrics registered by init() calls from the import, so...
I have tried to print the metric's name at local: kubelet: kube-proxy: kube-apiserver: kube-controller-manager: kube-scheduler:
ideally the registration should happen outside of init(), which means calling it on demand. but i will defer to the owning sigs.
In principle, yes, I agree. Otherwise, we need to be very very careful with every log in the stability framework. I think we can track it and work out by iteration. cc for comments.

return } if shouldHide(&version, selfVersion) { <del> klog.Warningf("This metric has been deprecated for more than one release, hiding.") </del> <ins> // TODO(RainbowMango): Remove this log temporarily. https://github.com/kubernetes/kubernetes/issues/85369 // klog.Warningf("This metric has been deprecated for more than one release, hiding.") </ins> r.isHidden = true } })


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When developing unit tests related to PDBs/Deletion, mutates pod resource via adding deletion timestamp, and this directly conflicts with checking resource version. What you expected to happen: No resource version mutation. How to reproduce it (as minimally and precisely as possible): I'm attempting to add preconditions to delete call in eviction API: Anything else we need to know?: Happens locally and in CI. Environment: Kubernetes version (use ): Head of master: Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you provide more details about what you're observing?
You can see an example here: Here's the relevant section of the tests: There really shouldn't be anything modifying the pod after creation until the delete request, thus I don't believe we should be hitting this precondition failure. Resource Version in precondition I believe to be correct, but when we contact etcd to verify, it's increased by one. It's not clear why this is happening, it appears to be a bug.
Okay, I've sorted this out. We have a conflict with 'graceful' deletion with this approach: This updates the object to add a deletion timestamp, thus the first call to eviction will always fail if we need to set ResourceVersion.

// TODO: remove the check, because we support no-op updates now. if graceful || pendingFinalizers || shouldUpdateFinalizers { err, ignoreNotFound, deleteImmediately, out, lastExisting = e.updateForGracefulDeletionAndFinalizers(ctx, name, key, options, preconditions, deleteValidation, obj) <ins> // Update the preconditions.ResourceVersion if set since we updated the object. if err == nil && deleteImmediately && preconditions.ResourceVersion != nil { accessor, err = meta.Accessor(out) if err != nil { return out, false, kubeerr.NewInternalError(err) } resourceVersion := accessor.GetResourceVersion() preconditions.ResourceVersion = &resourceVersion } </ins> } // !deleteImmediately covers all cases where err != nil. We keep both to be future-proof.


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When developing unit tests related to PDBs/Deletion, mutates pod resource via adding deletion timestamp, and this directly conflicts with checking resource version. What you expected to happen: No resource version mutation. How to reproduce it (as minimally and precisely as possible): I'm attempting to add preconditions to delete call in eviction API: Anything else we need to know?: Happens locally and in CI. Environment: Kubernetes version (use ): Head of master: Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you provide more details about what you're observing?
You can see an example here: Here's the relevant section of the tests: There really shouldn't be anything modifying the pod after creation until the delete request, thus I don't believe we should be hitting this precondition failure. Resource Version in precondition I believe to be correct, but when we contact etcd to verify, it's increased by one. It's not clear why this is happening, it appears to be a bug.
Okay, I've sorted this out. We have a conflict with 'graceful' deletion with this approach: This updates the object to add a deletion timestamp, thus the first call to eviction will always fail if we need to set ResourceVersion.

} } <ins> func TestStoreGracefulDeleteWithResourceVersion(t *testing.T) { podA := &example.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Spec:       example.PodSpec{NodeName: "machine"}, } testContext := genericapirequest.WithNamespace(genericapirequest.NewContext(), "test") destroyFunc, registry := NewTestGenericStoreRegistry(t) defer destroyFunc() defaultDeleteStrategy := testRESTStrategy{scheme, names.SimpleNameGenerator, true, false, true} registry.DeleteStrategy = testGracefulStrategy{defaultDeleteStrategy} // test failure condition _, _, err := registry.Delete(testContext, podA.Name, rest.ValidateAllObjectFunc, nil) if !errors.IsNotFound(err) { t.Errorf("Unexpected error: %v", err) } // create pod _, err = registry.Create(testContext, podA, rest.ValidateAllObjectFunc, &metav1.CreateOptions{}) if err != nil { t.Errorf("Unexpected error: %v", err) } // try to get a item which should be deleted obj, err := registry.Get(testContext, podA.Name, &metav1.GetOptions{}) if errors.IsNotFound(err) { t.Errorf("Unexpected error: %v", err) } accessor, err := meta.Accessor(obj) if err != nil { t.Errorf("Unexpected error: %v", err) } resourceVersion := accessor.GetResourceVersion() options := metav1.NewDeleteOptions(0) options.Preconditions = &metav1.Preconditions{ResourceVersion: &resourceVersion} // delete object _, wasDeleted, err := registry.Delete(testContext, podA.Name, rest.ValidateAllObjectFunc, options) if err != nil { t.Errorf("Unexpected error: %v", err) } if !wasDeleted { t.Errorf("unexpected, pod %s should have been deleted immediately", podA.Name) } // try to get a item which should be deleted _, err = registry.Get(testContext, podA.Name, &metav1.GetOptions{}) if !errors.IsNotFound(err) { t.Errorf("Unexpected error: %v", err) } } </ins> // TestGracefulStoreCanDeleteIfExistingGracePeriodZero tests recovery from // race condition where the graceful delete is unable to complete // in prior operation, but the pod remains with deletion timestamp


Question:
Was just perusing the code and looks like we translate regional disks into the "zonal volume ID fmt", and there doesn't seem to be any tests for regional disks running so we should confirm that this works and fix it if it doesn't /assign /cc
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen /assign
/assign

if err != nil { return nil, fmt.Errorf("failed to get region from zones: %v", err) } <del> volID = fmt.Sprintf(volIDZonalFmt, UnspecifiedValue, region, pv.Spec.GCEPersistentDisk.PDName) </del> <ins> volID = fmt.Sprintf(volIDRegionalFmt, UnspecifiedValue, region, pv.Spec.GCEPersistentDisk.PDName) </ins> } else { // Unspecified volID = fmt.Sprintf(volIDZonalFmt, UnspecifiedValue, UnspecifiedValue, pv.Spec.GCEPersistentDisk.PDName)


Question:
Was just perusing the code and looks like we translate regional disks into the "zonal volume ID fmt", and there doesn't seem to be any tests for regional disks running so we should confirm that this works and fix it if it doesn't /assign /cc
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen /assign
/assign

v1 "k8s.io/api/core/v1" storage "k8s.io/api/storage/v1" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> ) func NewStorageClass(params map[string]string, allowedTopologies []v1.TopologySelectorTerm) *storage.StorageClass {


Question:
Was just perusing the code and looks like we translate regional disks into the "zonal volume ID fmt", and there doesn't seem to be any tests for regional disks running so we should confirm that this works and fix it if it doesn't /assign /cc
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen /assign
/assign

t.Errorf("got am %v, expected access mode of ReadOnlyMany", ams[0]) } } <ins> func TestTranslateInTreePVToCSIVolIDFmt(t *testing.T) { g := NewGCEPersistentDiskCSITranslator() pdName := "pd-name" tests := []struct { desc               string topologyLabelKey   string topologyLabelValue string wantVolId          string }{ { desc:               "beta topology key zonal", topologyLabelKey:   v1.LabelFailureDomainBetaZone, topologyLabelValue: "us-east1-a", wantVolId:          "projects/UNSPECIFIED/zones/us-east1-a/disks/pd-name", }, { desc:               "v1 topology key zonal", topologyLabelKey:   v1.LabelTopologyZone, topologyLabelValue: "us-east1-a", wantVolId:          "projects/UNSPECIFIED/zones/us-east1-a/disks/pd-name", }, { desc:               "beta topology key regional", topologyLabelKey:   v1.LabelFailureDomainBetaZone, topologyLabelValue: "us-central1-a__us-central1-c", wantVolId:          "projects/UNSPECIFIED/regions/us-central1/disks/pd-name", }, { desc:               "v1 topology key regional", topologyLabelKey:   v1.LabelTopologyZone, topologyLabelValue: "us-central1-a__us-central1-c", wantVolId:          "projects/UNSPECIFIED/regions/us-central1/disks/pd-name", }, } for _, tc := range tests { t.Run(tc.desc, func(t *testing.T) { translatedPV, err := g.TranslateInTreePVToCSI(&v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{tc.topologyLabelKey: tc.topologyLabelValue}, }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName: pdName, }, }, }, }) if err != nil { t.Errorf("got error translating in-tree PV to CSI: %v", err) } if got := translatedPV.Spec.PersistentVolumeSource.CSI.VolumeHandle; got != tc.wantVolId { t.Errorf("got translated volume handle: %q, want %q", got, tc.wantVolId) } }) } } </ins>


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should allow collecting both default scheduler metrics (Prometheus) and custom metrics easily. At a high level, these collectors should produce a json formatted result which will then be consumed by perf-dash to present in the dashboard. Prometheus metrics We should provide a metrics collector that collects and summarizes default scheduler metrics defined in . To begin with, we can manually pick a list of interesting metrics. Longer term, we could allow configuring the metric list in test definition. already had a PoC . As a reference, is how clusterloader collects scheduler latency metrics. Custom metrics The most important metric is scheduling throughput. is how clusterloader calculates the throughput. We can start with just calculating the average throughput without the percentiles. We may want other custom metrics that are either common to all tests or specific to certain tests. We should allow these custom collectors to be easily and ideally configurable. Why is this needed: Give more insights into scheduler performance. Part of /sig scheduling /assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

"//staging/src/k8s.io/client-go/informers/core/v1:go_default_library", "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/rest:go_default_library", <ins> "//staging/src/k8s.io/component-base/metrics/legacyregistry:go_default_library", </ins> "//test/integration/util:go_default_library", <ins> "//vendor/github.com/prometheus/client_model/go:go_default_library", "//vendor/k8s.io/klog:go_default_library", </ins> ], )


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should allow collecting both default scheduler metrics (Prometheus) and custom metrics easily. At a high level, these collectors should produce a json formatted result which will then be consumed by perf-dash to present in the dashboard. Prometheus metrics We should provide a metrics collector that collects and summarizes default scheduler metrics defined in . To begin with, we can manually pick a list of interesting metrics. Longer term, we could allow configuring the metric list in test definition. already had a PoC . As a reference, is how clusterloader collects scheduler latency metrics. Custom metrics The most important metric is scheduling throughput. is how clusterloader calculates the throughput. We can start with just calculating the average throughput without the percentiles. We may want other custom metrics that are either common to all tests or specific to certain tests. We should allow these custom collectors to be easily and ideally configurable. Why is this needed: Give more insights into scheduler performance. Part of /sig scheduling /assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

configFile = "config/performance-config.yaml" ) <ins> var ( defaultMetrics = []string{ "scheduler_scheduling_algorithm_predicate_evaluation_seconds", "scheduler_scheduling_algorithm_priority_evaluation_seconds", "scheduler_binding_duration_seconds", "scheduler_e2e_scheduling_duration_seconds", } ) </ins> // testCase configures a test case to run the scheduler performance test. Users should be able to // provide this via a YAML file. //


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should allow collecting both default scheduler metrics (Prometheus) and custom metrics easily. At a high level, these collectors should produce a json formatted result which will then be consumed by perf-dash to present in the dashboard. Prometheus metrics We should provide a metrics collector that collects and summarizes default scheduler metrics defined in . To begin with, we can manually pick a list of interesting metrics. Longer term, we could allow configuring the metric list in test definition. already had a PoC . As a reference, is how clusterloader collects scheduler latency metrics. Custom metrics The most important metric is scheduling throughput. is how clusterloader calculates the throughput. We can start with just calculating the average throughput without the percentiles. We may want other custom metrics that are either common to all tests or specific to certain tests. We should allow these custom collectors to be easily and ideally configurable. Why is this needed: Give more insights into scheduler performance. Part of /sig scheduling /assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

} func BenchmarkPerfScheduling(b *testing.B) { <ins> dataItems := DataItems{Version: "v1"} </ins> tests := getSimpleTestCases(configFile) for _, test := range tests {


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should allow collecting both default scheduler metrics (Prometheus) and custom metrics easily. At a high level, these collectors should produce a json formatted result which will then be consumed by perf-dash to present in the dashboard. Prometheus metrics We should provide a metrics collector that collects and summarizes default scheduler metrics defined in . To begin with, we can manually pick a list of interesting metrics. Longer term, we could allow configuring the metric list in test definition. already had a PoC . As a reference, is how clusterloader collects scheduler latency metrics. Custom metrics The most important metric is scheduling throughput. is how clusterloader calculates the throughput. We can start with just calculating the average throughput without the percentiles. We may want other custom metrics that are either common to all tests or specific to certain tests. We should allow these custom collectors to be easily and ideally configurable. Why is this needed: Give more insights into scheduler performance. Part of /sig scheduling /assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

for feature, flag := range test.FeatureGates { defer featuregatetesting.SetFeatureGateDuringTest(b, utilfeature.DefaultFeatureGate, feature, flag)() } <del> perfScheduling(test, b) </del> <ins> dataItems.DataItems = append(dataItems.DataItems, perfScheduling(test, b)...) </ins> }) } <ins> if err := dataItems2JSONFile(dataItems, b.Name()); err != nil { klog.Fatalf("%v: unable to write measured data: %v", b.Name(), err) } </ins> } <del> func perfScheduling(test testCase, b *testing.B) { </del> <ins> func perfScheduling(test testCase, b *testing.B) []DataItem { </ins> var nodeStrategy testutils.PrepareNodeStrategy = &testutils.TrivialNodePrepareStrategy{} if test.Nodes.NodeAllocatableStrategy != nil { nodeStrategy = test.Nodes.NodeAllocatableStrategy


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should allow collecting both default scheduler metrics (Prometheus) and custom metrics easily. At a high level, these collectors should produce a json formatted result which will then be consumed by perf-dash to present in the dashboard. Prometheus metrics We should provide a metrics collector that collects and summarizes default scheduler metrics defined in . To begin with, we can manually pick a list of interesting metrics. Longer term, we could allow configuring the metric list in test definition. already had a PoC . As a reference, is how clusterloader collects scheduler latency metrics. Custom metrics The most important metric is scheduling throughput. is how clusterloader calculates the throughput. We can start with just calculating the average throughput without the percentiles. We may want other custom metrics that are either common to all tests or specific to certain tests. We should allow these custom collectors to be easily and ideally configurable. Why is this needed: Give more insights into scheduler performance. Part of /sig scheduling /assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

// start benchmark b.ResetTimer() <ins> // Start measuring throughput stopCh := make(chan struct{}) throughputCollector := newThroughputCollector(podInformer) go throughputCollector.run(stopCh) // Scheduling the main workload </ins> config = testutils.NewTestPodCreatorConfig() config.AddStrategy(testNamespace, test.PodsToSchedule.Num, testPodStrategy) podCreator = testutils.NewTestPodCreator(clientset, config) podCreator.CreatePods() <-completedCh <ins> close(stopCh) </ins> // Note: without this line we're taking the overhead of defer() into account. b.StopTimer() <ins> setNameLabel := func(dataItem *DataItem) DataItem { if dataItem.Labels == nil { dataItem.Labels = map[string]string{} } dataItem.Labels["Name"] = b.Name() return *dataItem } dataItems := []DataItem{ setNameLabel(throughputCollector.collect()), } for _, metric := range defaultMetrics { dataItem := newPrometheusCollector(metric).collect() if dataItem == nil { continue } dataItems = append(dataItems, setNameLabel(dataItem)) } return dataItems </ins> } func getPodStrategy(pc podCase) testutils.TestPodCreateStrategy {


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should allow collecting both default scheduler metrics (Prometheus) and custom metrics easily. At a high level, these collectors should produce a json formatted result which will then be consumed by perf-dash to present in the dashboard. Prometheus metrics We should provide a metrics collector that collects and summarizes default scheduler metrics defined in . To begin with, we can manually pick a list of interesting metrics. Longer term, we could allow configuring the metric list in test definition. already had a PoC . As a reference, is how clusterloader collects scheduler latency metrics. Custom metrics The most important metric is scheduling throughput. is how clusterloader calculates the throughput. We can start with just calculating the average throughput without the percentiles. We may want other custom metrics that are either common to all tests or specific to certain tests. We should allow these custom collectors to be easily and ideally configurable. Why is this needed: Give more insights into scheduler performance. Part of /sig scheduling /assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

package benchmark import ( <ins> "encoding/json" "flag" "fmt" "io/ioutil" "math" "path" "sort" "time" dto "github.com/prometheus/client_model/go" </ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime/schema" coreinformers "k8s.io/client-go/informers/core/v1" clientset "k8s.io/client-go/kubernetes" restclient "k8s.io/client-go/rest" <ins> "k8s.io/component-base/metrics/legacyregistry" "k8s.io/klog" </ins> "k8s.io/kubernetes/test/integration/util" ) <ins> const ( dateFormat                = "2006-01-02T15:04:05Z" throughputSampleFrequency = time.Second ) var dataItemsDir = flag.String("data-items-dir", "", "destination directory for storing generated data items for perf dashboard") </ins> // mustSetupScheduler starts the following components: // - k8s api server (a.k.a. master) // - scheduler