Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The apiserver, controller-manager and scheduler are configured to use IPv4 addresses however, are serving on all addresses IPv4 and IPv6 What you expected to happen: To listen only on the IP family specified How to reproduce it (as minimally and precisely as possible): or checking in one of the gce presubmits jobs Anything else we need to know?: it seems that the apiserver, controller-manager and kube-scheduler listen on all IP families by default despite the IP family address configured Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /cc
binding 0.0.0.0 to all interfaces is the documented behavior of net.Listen:
The unspecified ip (0.0.0.0) is a special case that means "bind to all interfaces". This is currently working as intended. Is there a specific problem this causes?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
If (0.0.0.0) means "bind to all interfaces on all IP families", then is a misconfiguration problem and not a bug . I was worried because this causes that some components use IPv6 for their communications on dual-stack hosts despite the cluster is IPv4 single family (and viceversa), as we can see in this snippet from one of the gce presubmits job and I don't know if this can impact on some services or from a security point of view
From the listen godoc: That means binding to or or will bind to all interfaces
I would expect anyone using network interfaces as security boundaries to be binding to a specific interface, not binding to all and hoping "all" is family-partitioned
Clarifying the handling of unspecified addresses like that in the flag/config help seems reasonable, but as far as I can tell, this is working as intended.
yeah, thanks for the clarification, I totally misinterpreted the net.Listen function
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
So, is there any way to listen on IPv4 addresses only (all of them)?
hmm, that will requires to expose this setting just curious, what is the use case for having IPv6 on the host and not in the apiserver, is not possible to disable ipv6 in the host?
I'm considering options to have a load balancer (HAProxy static pod in hostNetwork) for api-server on control plane nodes listening on the same port as api-server. Then I would point api-server VIP to via hosts file on the control plane nodes. This is a workaround for the absence of hairpin in AWS NLB (with instances by id) or Azure LB. I'm looking for a generic solution that will work for AWS/Azure/on-prem with nodes in the same/different subnets. If you have other suggestions, feel free to share, I would appreciate that. For now, my solution is to call , then patch the generated static manifest with set to the node's default ip, and then bootstrap with . Binding the load balancer to IPv6 and and api-server to IPv4, then using in kubeadm's config will be a dirty, but cleaner from the code perspective, approach. Btw, is there any reason behind having set to by default, but not the node's default ip, just like ?
I can't understand how do you solve the hairpin problem that way, is not the traffic been directed to the LB vip too? or do you mean that you don't use the AWS NLB and use the haproxy instead? what VIP do you use then? I don't remember the details, but the apiserver should be available to the pods too, that don't reach it through the node, they use the the kubernetes.default Service
External clients use AWS load balancer, while requests from control plane nodes go to haproxy on localhost thanks to trick, thus do not reach AWS LB.

fs.IPVar(&s.BindAddress, "bind-address", s.BindAddress, ""+ "The IP address on which to listen for the --secure-port port. The "+ "associated interface(s) must be reachable by the rest of the cluster, and by CLI/web "+ <del> "clients. If blank, all interfaces will be used (0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces).") </del> <ins> "clients. If blank or an unspecified address (0.0.0.0 or ::), all interfaces will be used.") </ins> desc := "The port on which to serve HTTPS with authentication and authorization." if s.Required {


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If you create a service that reference a Basic IP in a standard load balancer cluster, the service will stuck at "Pending" state for days without detailed error from Azure response. When you describe the service, it only has following: What you expected to happen: In this case Azure responded 400 with detailed error message that the referenced IP is a basic IP and cannot be used on standard LB. I would expect the detailed error message show up as events for easier diagnostic. How to reproduce it (as minimally and precisely as possible): Create a service that reference a Basic IP in a standard load balancer cluster Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /area provider/azure /sig cloud-provider
/area cloudprovider

// shouldRetryHTTPRequest determines if the request is retriable. func shouldRetryHTTPRequest(resp *http.Response, err error) bool { if resp != nil { <del> // HTTP 412 (StatusPreconditionFailed) means etag mismatch, hence we shouldn't retry. if resp.StatusCode == http.StatusPreconditionFailed { </del> <ins> // HTTP 412 (StatusPreconditionFailed) means etag mismatch // HTTP 400 (BadRequest) means the request cannot be accepted, hence we shouldn't retry. if resp.StatusCode == http.StatusPreconditionFailed || resp.StatusCode == http.StatusBadRequest { </ins> return false }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If you create a service that reference a Basic IP in a standard load balancer cluster, the service will stuck at "Pending" state for days without detailed error from Azure response. When you describe the service, it only has following: What you expected to happen: In this case Azure responded 400 with detailed error message that the referenced IP is a basic IP and cannot be used on standard LB. I would expect the detailed error message show up as events for easier diagnostic. How to reproduce it (as minimally and precisely as possible): Create a service that reference a Basic IP in a standard load balancer cluster Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /area provider/azure /sig cloud-provider
/area cloudprovider

{ code: http.StatusBadRequest, expected: &Error{ <del> Retriable:      true, </del> <ins> Retriable:      false, </ins> HTTPStatusCode: http.StatusBadRequest, RawError:       fmt.Errorf("HTTP response: 400"), },


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If you create a service that reference a Basic IP in a standard load balancer cluster, the service will stuck at "Pending" state for days without detailed error from Azure response. When you describe the service, it only has following: What you expected to happen: In this case Azure responded 400 with detailed error message that the referenced IP is a basic IP and cannot be used on standard LB. I would expect the detailed error message show up as events for easier diagnostic. How to reproduce it (as minimally and precisely as possible): Create a service that reference a Basic IP in a standard load balancer cluster Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /area provider/azure /sig cloud-provider
/area cloudprovider

}{ { code:     http.StatusBadRequest, <del> expected: true, </del> <ins> expected: false, </ins> }, { code:     http.StatusInternalServerError,


Question:
Which jobs are failing: Which test(s) are failing: Testgrid link: Reason for failure: check Anything else we need to know: /cc /milestone v1.18 /sig cloud-provider-gcp
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Which jobs are failing: Which test(s) are failing: Testgrid link: check /cc /milestone v1.18 /sig cloud-provider-gcp </details
/sig cloud-provider
/area provider/gcp
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /area provider/gke </details
Probably related to
shows the merge shows the errant merge that inadvertantly disabled the test This test was probably failing since forever.
Any news?
this one is on cloud provider GCP.
Can you please take a look?
Follow up from today's release meeting: If the test has been broken for as long as we know it, theres value in just removing it. If the test owners don't have capacity to look into it, CI Signal can investigate.
Any advice on which team is responsible? It seems there isn't any maintainer for this job.
try talking to please
/assign
Turning to the question of whether these tests are actually useful... One of the tests that has been continuously failing is this one Which essentially does because From the logs (example ) it waits for 2 minutes (120 seconds) for each node to not be ready and then for the next couple minutes it just checks this condition until it fails So it looks like the e2e tests are not failing because of inherent infra issues (i.e., kube-up decided to do some weird thing) but the issue seems to be with the time to reach a condition. Again if we look at the logs we can see that switching the network interface does have an effect and leads to the nodes reaching a non-ready status but ultimately fail to come back up within the desired 5m time mark - this bit of the tests is where it all goes bad . The question then turns to turning off the network interface for a node in such a way () and then expecting the node to come back to live within 5 minutes of turning the interfac back on something reasonable? If it is, then either we have to look at why it takes longer than 5 minutes or we can increase the timeouts If it isn't, then how can we test the ability of nodes to be rebooted?
/assign
Wanted to triple check on the test looking for nodes to be back up after being rebooted: I dont think 5 minutes is a hard deadline - only SLOs I could find were for scalability

ginkgo.It("each node by switching off the network interface and ensure they function upon switch on", func() { // switch the network interface off for a while to simulate a network outage // We sleep 10 seconds to give some time for ssh command to cleanly finish before network is down. <del> testReboot(f.ClientSet, "nohup sh -c 'sleep 10 && sudo ip link set eth0 down && sleep 120 && sudo ip link set eth0 up && (sudo dhclient || true)' >/dev/null 2>&1 &", nil) </del> <ins> cmd := "nohup sh -c '" + "sleep 10; " + "echo Shutting down eth0 | sudo tee /dev/kmsg; " + "sudo ip link set eth0 down | sudo tee /dev/kmsg; " + "sleep 120; " + "echo Starting up eth0 | sudo tee /dev/kmsg; " + "sudo ip link set eth0 up | sudo tee /dev/kmsg; " + "sleep 10; " + "echo Retrying starting up eth0 | sudo tee /dev/kmsg; " + "sudo ip link set eth0 up | sudo tee /dev/kmsg; " + "echo Running dhclient | sudo tee /dev/kmsg; " + "sudo dhclient | sudo tee /dev/kmsg; " + "echo Starting systemd-networkd | sudo tee /dev/kmsg; " + "sudo systemctl restart systemd-networkd | sudo tee /dev/kmsg" + "' >/dev/null 2>&1 &" testReboot(f.ClientSet, cmd, nil) </ins> }) ginkgo.It("each node by dropping all inbound packets for a while and ensure they function afterwards", func() {


Question:
What happened: We enabled the feature gate on to start switching from non-expiring to expiring ServiceAccount tokens as . It works well. We then wanted to completely get rid of the "legacy" non-expiring ServiceAccount token Secrets. To do that we configured with (so it wouldn't auto-create the Secrets), removed the existing Secrets and removed the reference from the respective ServiceAccount to the Secret. This works as expected: No non-expiring token Secrets are generated. However, when creating a Pod we now see: This happens because the rejects the Pod because it cannot find a ServiceAccount token Secret (because it doesn't exist anymore as intended). What you expected to happen: When using the feature the legacy ServiceAccount token Secrets aren't used anymore, so a Pod should not be rejected if the Secret doesn't exist. The Pod doesn't even use the Secret at all. How to reproduce it (as minimally and precisely as possible): Enable feature gate Start with Create a new ServiceAccount Deploy a dummy app using that ServiceAccount Observe no Pod being created with Describe the corresponding ReplicaSet to see the above Error event. Anything else we need to know?: Looking at the it seems that at least one legacy Secret must exist in order for the Pod to get admitted. This is enforced even when and we're relying on projected token volumes. Following the code path when is enabled reveals that the token is never going to be used later on as there's no SecretVolume to be mounted in . In other words:  is just fine when . During the transition period it's fine to have the legacy Secrets lying around but ultimately it's desirable to get rid of them because as far as I can tell they are still happily accepted by the API server. Environment: Kubernetes version: tested on but code seems the same in and Cloud provider or hardware configuration: OS (e.g: ): Ubuntu 18.04.3 LTS Kernel (e.g. ): 4.15.0-1054-aws Install tools: /kind bug /sig auth
Fixed in
Thank you!


Question:
What happened: We enabled the feature gate on to start switching from non-expiring to expiring ServiceAccount tokens as . It works well. We then wanted to completely get rid of the "legacy" non-expiring ServiceAccount token Secrets. To do that we configured with (so it wouldn't auto-create the Secrets), removed the existing Secrets and removed the reference from the respective ServiceAccount to the Secret. This works as expected: No non-expiring token Secrets are generated. However, when creating a Pod we now see: This happens because the rejects the Pod because it cannot find a ServiceAccount token Secret (because it doesn't exist anymore as intended). What you expected to happen: When using the feature the legacy ServiceAccount token Secrets aren't used anymore, so a Pod should not be rejected if the Secret doesn't exist. The Pod doesn't even use the Secret at all. How to reproduce it (as minimally and precisely as possible): Enable feature gate Start with Create a new ServiceAccount Deploy a dummy app using that ServiceAccount Observe no Pod being created with Describe the corresponding ReplicaSet to see the above Error event. Anything else we need to know?: Looking at the it seems that at least one legacy Secret must exist in order for the Pod to get admitted. This is enforced even when and we're relying on projected token volumes. Following the code path when is enabled reveals that the token is never going to be used later on as there's no SecretVolume to be mounted in . In other words:  is just fine when . During the transition period it's fine to have the legacy Secrets lying around but ultimately it's desirable to get rid of them because as far as I can tell they are still happily accepted by the API server. Environment: Kubernetes version: tested on but code seems the same in and Cloud provider or hardware configuration: OS (e.g: ): Ubuntu 18.04.3 LTS Kernel (e.g. ): 4.15.0-1054-aws Install tools: /kind bug /sig auth
Fixed in
Thank you!

} func (s *Plugin) mountServiceAccountToken(serviceAccount *corev1.ServiceAccount, pod *api.Pod) error { <del> // Find the name of a referenced ServiceAccountToken secret we can mount serviceAccountToken, err := s.getReferencedServiceAccountToken(serviceAccount) if err != nil { return fmt.Errorf("Error looking up service account token for %s/%s: %v", serviceAccount.Namespace, serviceAccount.Name, err) </del> <ins> var ( // serviceAccountToken holds the name of a secret containing a legacy service account token serviceAccountToken string err                 error ) if !s.boundServiceAccountTokenVolume { // Find the name of a referenced ServiceAccountToken secret we can mount serviceAccountToken, err = s.getReferencedServiceAccountToken(serviceAccount) if err != nil { return fmt.Errorf("Error looking up service account token for %s/%s: %v", serviceAccount.Namespace, serviceAccount.Name, err) } </ins> } <del> if len(serviceAccountToken) == 0 { </del> <ins> if len(serviceAccountToken) == 0 && !s.boundServiceAccountTokenVolume { </ins> // We don't have an API token to mount, so return if s.RequireAPIToken { // If a token is required, this is considered an error


Question:
What happened: We enabled the feature gate on to start switching from non-expiring to expiring ServiceAccount tokens as . It works well. We then wanted to completely get rid of the "legacy" non-expiring ServiceAccount token Secrets. To do that we configured with (so it wouldn't auto-create the Secrets), removed the existing Secrets and removed the reference from the respective ServiceAccount to the Secret. This works as expected: No non-expiring token Secrets are generated. However, when creating a Pod we now see: This happens because the rejects the Pod because it cannot find a ServiceAccount token Secret (because it doesn't exist anymore as intended). What you expected to happen: When using the feature the legacy ServiceAccount token Secrets aren't used anymore, so a Pod should not be rejected if the Secret doesn't exist. The Pod doesn't even use the Secret at all. How to reproduce it (as minimally and precisely as possible): Enable feature gate Start with Create a new ServiceAccount Deploy a dummy app using that ServiceAccount Observe no Pod being created with Describe the corresponding ReplicaSet to see the above Error event. Anything else we need to know?: Looking at the it seems that at least one legacy Secret must exist in order for the Pod to get admitted. This is enforced even when and we're relying on projected token volumes. Following the code path when is enabled reveals that the token is never going to be used later on as there's no SecretVolume to be mounted in . In other words:  is just fine when . During the transition period it's fine to have the legacy Secrets lying around but ultimately it's desirable to get rid of them because as far as I can tell they are still happily accepted by the API server. Environment: Kubernetes version: tested on but code seems the same in and Cloud provider or hardware configuration: OS (e.g: ): Ubuntu 18.04.3 LTS Kernel (e.g. ): 4.15.0-1054-aws Install tools: /kind bug /sig auth
Fixed in
Thank you!

"k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/diff" </ins> "k8s.io/apiserver/pkg/admission" admissiontesting "k8s.io/apiserver/pkg/admission/testing" "k8s.io/client-go/informers"


Question:
What happened: We enabled the feature gate on to start switching from non-expiring to expiring ServiceAccount tokens as . It works well. We then wanted to completely get rid of the "legacy" non-expiring ServiceAccount token Secrets. To do that we configured with (so it wouldn't auto-create the Secrets), removed the existing Secrets and removed the reference from the respective ServiceAccount to the Secret. This works as expected: No non-expiring token Secrets are generated. However, when creating a Pod we now see: This happens because the rejects the Pod because it cannot find a ServiceAccount token Secret (because it doesn't exist anymore as intended). What you expected to happen: When using the feature the legacy ServiceAccount token Secrets aren't used anymore, so a Pod should not be rejected if the Secret doesn't exist. The Pod doesn't even use the Secret at all. How to reproduce it (as minimally and precisely as possible): Enable feature gate Start with Create a new ServiceAccount Deploy a dummy app using that ServiceAccount Observe no Pod being created with Describe the corresponding ReplicaSet to see the above Error event. Anything else we need to know?: Looking at the it seems that at least one legacy Secret must exist in order for the Pod to get admitted. This is enforced even when and we're relying on projected token volumes. Following the code path when is enabled reveals that the token is never going to be used later on as there's no SecretVolume to be mounted in . In other words:  is just fine when . During the transition period it's fine to have the legacy Secrets lying around but ultimately it's desirable to get rid of them because as far as I can tell they are still happily accepted by the API server. Environment: Kubernetes version: tested on but code seems the same in and Cloud provider or hardware configuration: OS (e.g: ): Ubuntu 18.04.3 LTS Kernel (e.g. ): 4.15.0-1054-aws Install tools: /kind bug /sig auth
Fixed in
Thank you!

} } <ins> func TestAssignsDefaultServiceAccountAndBoundTokenWithNoSecretTokens(t *testing.T) { ns := "myns" admit := NewServiceAccount() informerFactory := informers.NewSharedInformerFactory(nil, controller.NoResyncPeriodFunc()) admit.SetExternalKubeInformerFactory(informerFactory) admit.MountServiceAccountToken = true admit.RequireAPIToken = true admit.boundServiceAccountTokenVolume = true // Add the default service account for the ns into the cache informerFactory.Core().V1().ServiceAccounts().Informer().GetStore().Add(&corev1.ServiceAccount{ ObjectMeta: metav1.ObjectMeta{ Name:      DefaultServiceAccountName, Namespace: ns, }, }) pod := &api.Pod{ Spec: api.PodSpec{ Containers: []api.Container{{}}, }, } attrs := admission.NewAttributesRecord(pod, nil, api.Kind("Pod").WithVersion("version"), ns, "myname", api.Resource("pods").WithVersion("version"), "", admission.Create, &metav1.CreateOptions{}, false, nil) err := admissiontesting.WithReinvocationTesting(t, admit).Admit(context.TODO(), attrs, nil) if err != nil { t.Fatalf("Expected success, got: %v", err) } expectedVolumes := []api.Volume{{ Name: "cleared", VolumeSource: api.VolumeSource{ Projected: &api.ProjectedVolumeSource{ Sources: []api.VolumeProjection{ {ServiceAccountToken: &api.ServiceAccountTokenProjection{ExpirationSeconds: 3600, Path: "token"}}, {ConfigMap: &api.ConfigMapProjection{LocalObjectReference: api.LocalObjectReference{Name: "kube-root-ca.crt"}, Items: []api.KeyToPath{{Key: "ca.crt", Path: "ca.crt"}}}}, {DownwardAPI: &api.DownwardAPIProjection{Items: []api.DownwardAPIVolumeFile{{Path: "namespace", FieldRef: &api.ObjectFieldSelector{APIVersion: "v1", FieldPath: "metadata.namespace"}}}}}, }, }, }, }} expectedVolumeMounts := []api.VolumeMount{{ Name:      "cleared", ReadOnly:  true, MountPath: "/var/run/secrets/kubernetes.io/serviceaccount", }} // clear generated volume names for i := range pod.Spec.Volumes { if len(pod.Spec.Volumes[i].Name) > 0 { pod.Spec.Volumes[i].Name = "cleared" } } for i := range pod.Spec.Containers[0].VolumeMounts { if len(pod.Spec.Containers[0].VolumeMounts[i].Name) > 0 { pod.Spec.Containers[0].VolumeMounts[i].Name = "cleared" } } if !reflect.DeepEqual(expectedVolumes, pod.Spec.Volumes) { t.Errorf("unexpected volumes: %s", diff.ObjectReflectDiff(expectedVolumes, pod.Spec.Volumes)) } if !reflect.DeepEqual(expectedVolumeMounts, pod.Spec.Containers[0].VolumeMounts) { t.Errorf("unexpected volumes: %s", diff.ObjectReflectDiff(expectedVolumeMounts, pod.Spec.Containers[0].VolumeMounts)) } } </ins> func TestFetchesUncachedServiceAccount(t *testing.T) { ns := "myns"


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux

return status, true, nil } <del> func getPublicIPDomainNameLabel(service *v1.Service) string { </del> <ins> func getPublicIPDomainNameLabel(service *v1.Service) (string, bool) { </ins> if labelName, found := service.Annotations[ServiceAnnotationDNSLabelName]; found { <del> return labelName </del> <ins> return labelName, found </ins> } <del> return "" </del> <ins> return "", false </ins> } // EnsureLoadBalancer creates a new load balancer 'name', or updates the existing one. Returns the status of the balancer


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux

return lbStatus.Ingress[0].IP, nil } <del> func (az *Cloud) ensurePublicIPExists(service *v1.Service, pipName string, domainNameLabel, clusterName string, shouldPIPExisted bool) (*network.PublicIPAddress, error) { </del> <ins> func (az *Cloud) ensurePublicIPExists(service *v1.Service, pipName string, domainNameLabel, clusterName string, shouldPIPExisted, foundDNSLabelAnnotation bool) (*network.PublicIPAddress, error) { </ins> pipResourceGroup := az.getPublicIPAddressResourceGroup(service) pip, existsPip, err := az.getPublicIPAddress(pipResourceGroup, pipName) if err != nil {


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux

} klog.V(2).Infof("ensurePublicIPExists for service(%s): pip(%s) - creating", serviceName, *pip.Name) } <del> if len(domainNameLabel) == 0 { pip.PublicIPAddressPropertiesFormat.DNSSettings = nil } else { pip.PublicIPAddressPropertiesFormat.DNSSettings = &network.PublicIPAddressDNSSettings{ DomainNameLabel: &domainNameLabel, </del> <ins> if foundDNSLabelAnnotation { if len(domainNameLabel) == 0 { pip.PublicIPAddressPropertiesFormat.DNSSettings = nil } else { pip.PublicIPAddressPropertiesFormat.DNSSettings = &network.PublicIPAddressDNSSettings{ DomainNameLabel: &domainNameLabel, } </ins> } }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux

if err != nil { return nil, err } <del> domainNameLabel := getPublicIPDomainNameLabel(service) pip, err := az.ensurePublicIPExists(service, pipName, domainNameLabel, clusterName, shouldPIPExisted) </del> <ins> domainNameLabel, found := getPublicIPDomainNameLabel(service) pip, err := az.ensurePublicIPExists(service, pipName, domainNameLabel, clusterName, shouldPIPExisted, found) </ins> if err != nil { return nil, err }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux

if !isInternal && wantLb { // Confirm desired public ip resource exists var pip *network.PublicIPAddress <del> domainNameLabel := getPublicIPDomainNameLabel(service) if pip, err = az.ensurePublicIPExists(service, desiredPipName, domainNameLabel, clusterName, shouldPIPExisted); err != nil { </del> <ins> domainNameLabel, found := getPublicIPDomainNameLabel(service) if pip, err = az.ensurePublicIPExists(service, desiredPipName, domainNameLabel, clusterName, shouldPIPExisted, found); err != nil { </ins> return nil, err } return pip, nil


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux

pip, err := az.reconcilePublicIP("testCluster", &service, "", test.wantLb) if test.expectedID != "" { assert.Equal(t, test.expectedID, to.String(pip.ID), "TestCase[%d]: %s", i, test.desc) <del> } else { assert.Equal(t, test.expectedPIP, pip, "TestCase[%d]: %s", i, test.desc) </del> <ins> } else if test.expectedPIP != nil && test.expectedPIP.Name != nil { assert.Equal(t, *test.expectedPIP.Name, *pip.Name, "TestCase[%d]: %s", i, test.desc) </ins> } assert.Equal(t, test.expectedError, err != nil, "TestCase[%d]: %s", i, test.desc) }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux

func TestEnsurePublicIPExists(t *testing.T) { testCases := []struct { <del> desc          string existingPIPs  []network.PublicIPAddress expectedPIP   *network.PublicIPAddress expectedID    string expectedDNS   string expectedError bool </del> <ins> desc                    string existingPIPs            []network.PublicIPAddress inputDNSLabel           string foundDNSLabelAnnotation bool expectedPIP             *network.PublicIPAddress expectedID              string expectedError           bool </ins> }{ { desc:         "ensurePublicIPExists shall return existed PIP if there is any",


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux

"Microsoft.Network/publicIPAddresses/pip1", }, { <del> desc: "ensurePublicIPExists shall update existed PIP's dns label", </del> <ins> desc:                    "ensurePublicIPExists shall update existed PIP's dns label", inputDNSLabel:           "newdns", foundDNSLabelAnnotation: true, </ins> existingPIPs: []network.PublicIPAddress{{ Name: to.StringPtr("pip1"), PublicIPAddressPropertiesFormat: &network.PublicIPAddressPropertiesFormat{


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux

}, }, }, <del> expectedDNS: "newdns", </del> <ins> }, { desc:                    "ensurePublicIPExists shall delete DNS from PIP if DNS label is set empty", foundDNSLabelAnnotation: true, existingPIPs: []network.PublicIPAddress{{ Name: to.StringPtr("pip1"), PublicIPAddressPropertiesFormat: &network.PublicIPAddressPropertiesFormat{ DNSSettings: &network.PublicIPAddressDNSSettings{ DomainNameLabel: to.StringPtr("previousdns"), }, }, }}, expectedPIP: &network.PublicIPAddress{ Name: to.StringPtr("pip1"), ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg" + "/providers/Microsoft.Network/publicIPAddresses/pip1"), PublicIPAddressPropertiesFormat: &network.PublicIPAddressPropertiesFormat{ DNSSettings: nil, }, }, }, { desc:                    "ensurePublicIPExists shall not delete DNS from PIP if DNS label annotation is not set", foundDNSLabelAnnotation: false, existingPIPs: []network.PublicIPAddress{{ Name: to.StringPtr("pip1"), PublicIPAddressPropertiesFormat: &network.PublicIPAddressPropertiesFormat{ DNSSettings: &network.PublicIPAddressDNSSettings{ DomainNameLabel: to.StringPtr("previousdns"), }, }, }}, expectedPIP: &network.PublicIPAddress{ Name: to.StringPtr("pip1"), ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg" + "/providers/Microsoft.Network/publicIPAddresses/pip1"), PublicIPAddressPropertiesFormat: &network.PublicIPAddressPropertiesFormat{ DNSSettings: &network.PublicIPAddressDNSSettings{ DomainNameLabel: to.StringPtr("previousdns"), }, }, }, </ins> }, }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux

t.Fatalf("TestCase[%d] meets unexpected error: %v", i, err) } } <del> pip, err := az.ensurePublicIPExists(&service, "pip1", test.expectedDNS, "", false) </del> <ins> pip, err := az.ensurePublicIPExists(&service, "pip1", test.inputDNSLabel, "", false, test.foundDNSLabelAnnotation) </ins> if test.expectedID != "" { assert.Equal(t, test.expectedID, to.String(pip.ID), "TestCase[%d]: %s", i, test.desc) } else {


Question:
Which jobs are flaking: pull-kubernetes-e2e-gce pull-kubernetes-e2e-gce-rbe Which test(s) are flaking: Stage Testgrid link: Reason for failure: This isn't technically a test that's failing, it's part of the build-stage-up-test cycle used by our e2e jobs. There are two main clusters of failures that I can tell. One is timeouts. The other looks like: Example -gce jobs: () () () () Example -gce-rbe jobs: (source gist ) Anything else we need to know: Triage link for the job that actually reports to PRs: ends up running Some googling for "bazel error code 14" leads me to believe that bazel (or its jvm) is getting killed due to OOM. which is a value that's been cargo culted around as far as I can tell. I was unable to find the decision that led to this value. I'm wondering if a bump to 8Gi of memory would help reduce occurrences of this flake. I'm referencing the -rbe jobs to point out that this happens even if we use RBE, but they're not blocking
/sig testing Top flake according to
With merged we'll wait and see if the Stage flakes on pull-kubernetes-e2e-gce go down. If they do, there are other presubmit jobs that could benefit from that. If they don't, we should revert.
/priority important-soon
/cc
Still seeing flakes, so I don't think just bumping memory request helped. ref: my theory is that making this non-privileged broke our ability to free memory This happened via around 2019-04-11 - flakes were still occurring prior to this merge - but not prior to 2019-04-01 - and very infrequently prior 2019-03-15 It could be that bazel's memory usage grew when we bumped it to 0.23.2 on 2019-03-29 (or maybe on 2019-04-02 is what actually deployed the change) Either way, I still think it looks like bazel is eating more memory than the container is allowed, and is getting oomkilled. Two ideas: Instead of allowing the bazel server to exist as a long lived process, have kubetest call in between the Build and Stage steps, in addition to trying to flush memory Constrain bazel's host JVM to a hardcoded memory limit via in bazelrc
Idea 3: switch build modes and see how expensive / flaky that is. Including: use RBE use the dockerized build instead
I opened to try and constrain the JVM's memory. Jobs that use RBE behave the same way, so try doing the same to them for comparison's sake.
this is still being seen extremely frequently (seen multiple times in a row on the same PR at )
I opened for the last idea I have for working around bazel's behavior, beyond this I am out of my depth on how to replace bazel for the release/push- script
... but I'm willing to try with a canary job and see what happens
bumping priority, this looks like 99/122 flakes according to /priority critical-urgent
Another occurrence:
I'd like to see if we can root cause what changed around 1/27-1/29 that greatly increased this
kubernetes/kubernetes changes: 1/26, ~6am UTC: 1/28: ~6pm UTC: test-infra changes: 1/25, ~3pm UTC: 1/28: ~8pm UTC: in that delta kubekins changed from v20200123- to v20200127- nothing jumps out at me in those diffs, but wanted to link to them in case it helped anyone else see something
It doesn't have to be one, the CI is run overcommitted with no limits, something else could be upping memory pressure..? On Mon, Feb 10, 2020, 08:38 Jordan Liggitt <:
ok... after sweeping k/k and test-infra for that time period, nothing clear emerged, and the bazel bandaids seem to be holding for now. unsatisfying :-/ /close
Closing this issue. <detailsIn response to : ok... after sweeping k/k and test-infra for that time period, nothing clear emerged, and the bazel bandaids seem to be holding for now. unsatisfying :-/ Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

else bazel-release: bazel build //build/release-tars <ins> bazel shutdown </ins> endif


Question:
What happened: Currently, the mutating admission happens before updating managedFields in create requests, but in the update and patch handlers, mutating admission happens after updating managedFields. This causes the ownership of fields modified by mutating webhooks to be inconsistent based on the request verb. /sig api-machinery /cc /priority backlog
/wg apply
/assign

if len(name) == 0 { _, name, _ = scope.Namer.ObjectName(obj) } <del> admissionAttributes := admission.NewAttributesRecord(obj, nil, scope.Kind, namespace, name, scope.Resource, scope.Subresource, admission.Create, options, dryrun.IsDryRun(options.DryRun), userInfo) if mutatingAdmission, ok := admit.(admission.MutationInterface); ok && mutatingAdmission.Handles(admission.Create) { err = mutatingAdmission.Admit(ctx, admissionAttributes, scope) if err != nil { scope.err(err, w, req) return } } if scope.FieldManager != nil { liveObj, err := scope.Creater.New(scope.Kind) if err != nil { scope.err(fmt.Errorf("failed to create new object (Create for %v): %v", scope.Kind, err), w, req) return </del> <ins> trace.Step("About to store object in database") result, err := finishRequest(timeout, func() (runtime.Object, error) { if scope.FieldManager != nil { liveObj, err := scope.Creater.New(scope.Kind) if err != nil { return nil, fmt.Errorf("failed to create new object (Create for %v): %v", scope.Kind, err) } obj, err = scope.FieldManager.Update(liveObj, obj, managerOrUserAgent(options.FieldManager, req.UserAgent())) if err != nil { return nil, fmt.Errorf("failed to update object (Create for %v) managed fields: %v", scope.Kind, err) } </ins> } <del> obj, err = scope.FieldManager.Update(liveObj, obj, managerOrUserAgent(options.FieldManager, req.UserAgent())) if err != nil { scope.err(fmt.Errorf("failed to update object (Create for %v) managed fields: %v", scope.Kind, err), w, req) return </del> <ins> admissionAttributes := admission.NewAttributesRecord(obj, nil, scope.Kind, namespace, name, scope.Resource, scope.Subresource, admission.Create, options, dryrun.IsDryRun(options.DryRun), userInfo) if mutatingAdmission, ok := admit.(admission.MutationInterface); ok && mutatingAdmission.Handles(admission.Create) { if err := mutatingAdmission.Admit(ctx, admissionAttributes, scope); err != nil { return nil, err } </ins> } <del> } trace.Step("About to store object in database") result, err := finishRequest(timeout, func() (runtime.Object, error) { </del> return r.Create( ctx, name,


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened. But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)

leaveStdinOpen := o.LeaveStdinOpen waitForExitCode := !leaveStdinOpen && restartPolicy == corev1.RestartPolicyNever if waitForExitCode { <del> pod, err = waitForPod(clientset.CoreV1(), attachablePod.Namespace, attachablePod.Name, podCompleted) </del> <ins> pod, err = waitForPod(clientset.CoreV1(), attachablePod.Namespace, attachablePod.Name, opts.GetPodTimeout, podCompleted) </ins> if err != nil { return err }


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened. But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)

} // waitForPod watches the given pod until the exitCondition is true <del> func waitForPod(podClient corev1client.PodsGetter, ns, name string, exitCondition watchtools.ConditionFunc) (*corev1.Pod, error) { // TODO: expose the timeout ctx, cancel := watchtools.ContextWithOptionalTimeout(context.Background(), 0*time.Second) </del> <ins> func waitForPod(podClient corev1client.PodsGetter, ns, name string, timeout time.Duration, exitCondition watchtools.ConditionFunc) (*corev1.Pod, error) { ctx, cancel := watchtools.ContextWithOptionalTimeout(context.Background(), timeout) </ins> defer cancel() preconditionFunc := func(store cache.Store) (bool, error) {


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened. But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)

} func handleAttachPod(f cmdutil.Factory, podClient corev1client.PodsGetter, ns, name string, opts *attach.AttachOptions) error { <del> pod, err := waitForPod(podClient, ns, name, podRunningAndReady) </del> <ins> pod, err := waitForPod(podClient, ns, name, opts.GetPodTimeout, podRunningAndReady) </ins> if err != nil && err != ErrPodCompleted { return err }


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened. But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)

ctx, cancel := watchtools.ContextWithOptionalTimeout(ctx, 0*time.Second) defer cancel() <del> preconditionFunc := func(store cache.Store) (bool, error) { _, exists, err := store.Get(&metav1.ObjectMeta{Namespace: ns, Name: podName}) if err != nil { return true, err } if !exists { // We need to make sure we see the object in the cache before we start waiting for events // or we would be waiting for the timeout if such object didn't exist. // (e.g. it was deleted before we started informers so they wouldn't even see the delete event) return true, errors.NewNotFound(corev1.Resource("pods"), podName) } return false, nil } </del> fieldSelector := fields.OneTermEqualSelector("metadata.name", podName).String() lw := &cache.ListWatch{ ListFunc: func(options metav1.ListOptions) (runtime.Object, error) {


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened. But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)

intr := interrupt.New(nil, cancel) var result *corev1.Pod err := intr.Run(func() error { <del> ev, err := watchtools.UntilWithSync(ctx, lw, &corev1.Pod{}, preconditionFunc, func(ev watch.Event) (bool, error) { </del> <ins> ev, err := watchtools.UntilWithSync(ctx, lw, &corev1.Pod{}, nil, func(ev watch.Event) (bool, error) { </ins> switch ev.Type { case watch.Deleted: return false, errors.NewNotFound(schema.GroupResource{Resource: "pods"}, "")


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened. But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)

"//build/visible_to:pkg_kubectl_cmd_rollout_CONSUMERS", ], deps = [ <del> "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", </del> "//staging/src/k8s.io/apimachinery/pkg/api/meta:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured:go_default_library",


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened. But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)

"github.com/spf13/cobra" <del> apierrors "k8s.io/apimachinery/pkg/api/errors" </del> "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened. But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)

}, } <del> preconditionFunc := func(store cache.Store) (bool, error) { _, exists, err := store.Get(&metav1.ObjectMeta{Namespace: info.Namespace, Name: info.Name}) if err != nil { return true, err } if !exists { // We need to make sure we see the object in the cache before we start waiting for events // or we would be waiting for the timeout if such object didn't exist. return true, apierrors.NewNotFound(mapping.Resource.GroupResource(), info.Name) } return false, nil } </del> // if the rollout isn't done yet, keep watching deployment status ctx, cancel := watchtools.ContextWithOptionalTimeout(context.Background(), o.Timeout) intr := interrupt.New(nil, cancel) return intr.Run(func() error { <del> _, err = watchtools.UntilWithSync(ctx, lw, &unstructured.Unstructured{}, preconditionFunc, func(e watch.Event) (bool, error) { </del> <ins> _, err = watchtools.UntilWithSync(ctx, lw, &unstructured.Unstructured{}, nil, func(e watch.Event) (bool, error) { </ins> switch t := e.Type; t { case watch.Added, watch.Modified: status, done, err := statusViewer.Status(e.Object.(runtime.Unstructured), o.Revision)


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened. But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)

} var pod *corev1.Pod <del> leaveStdinOpen := o.LeaveStdinOpen waitForExitCode := !leaveStdinOpen && restartPolicy == corev1.RestartPolicyNever </del> <ins> waitForExitCode := !o.LeaveStdinOpen && (restartPolicy == corev1.RestartPolicyNever || restartPolicy == corev1.RestartPolicyOnFailure) </ins> if waitForExitCode { <del> pod, err = waitForPod(clientset.CoreV1(), attachablePod.Namespace, attachablePod.Name, opts.GetPodTimeout, podCompleted) </del> <ins> // we need different exit condition depending on restart policy // for Never it can either fail or succeed, for OnFailure only // success matters exitCondition := podCompleted if restartPolicy == corev1.RestartPolicyOnFailure { exitCondition = podSucceeded } pod, err = waitForPod(clientset.CoreV1(), attachablePod.Namespace, attachablePod.Name, opts.GetPodTimeout, exitCondition) </ins> if err != nil { return err } <del> } // after removal is done, return successfully if we are not interested in the exit code if !waitForExitCode { </del> <ins> } else { // after removal is done, return successfully if we are not interested in the exit code </ins> return nil }