Question:
Both https://k8s- and https://k8s- started failing recently. Their 1.7 counterparts are passing. The difference between the last passing and the first failing tests is: Probably something in broke the tests. /sig scheduling /cc /kind bug
It looks like there isn't enough cpu on the master to run the cos-nvidia-installer daemonset. Did we add any addons recently?
My previous comment is a red herring. It is an issue with . Ill post a fix soon. From the scheduler logs on the master:
that's a benign error. The daemonset is not expected to run on the master. But there is no easy way to express that request in our APIs.
Yes, the pod not scheduling on the master is benign. The actual error is that we do not report node condition , so it is left as . The fix is to report that is false.
The tests are still failing. /reopen
/assign
I will be updating to track progress.
tests are green again! /close

} } <ins> // Set OODCondition for the node. func (kl *Kubelet) setNodeOODCondition(node *v1.Node) { currentTime := metav1.NewTime(kl.clock.Now()) var nodeOODCondition *v1.NodeCondition // Check if NodeOutOfDisk condition already exists and if it does, just pick it up for update. for i := range node.Status.Conditions { if node.Status.Conditions[i].Type == v1.NodeOutOfDisk { nodeOODCondition = &node.Status.Conditions[i] } } newOODCondition := nodeOODCondition == nil if newOODCondition || nodeOODCondition.Status == v1.ConditionUnknown { nodeOODCondition = &v1.NodeCondition{ Type:               v1.NodeOutOfDisk, Status:             v1.ConditionFalse, Reason:             "KubeletHasSufficientDisk", Message:            "kubelet has sufficient disk space available", LastTransitionTime: currentTime, } } // Update the heartbeat time irrespective of all the conditions. nodeOODCondition.LastHeartbeatTime = currentTime if newOODCondition { node.Status.Conditions = append(node.Status.Conditions, *nodeOODCondition) } } </ins> // Maintains Node.Spec.Unschedulable value from previous run of tryUpdateNodeStatus() // TODO: why is this a package var? var oldNodeUnschedulable bool


Question:
Both https://k8s- and https://k8s- started failing recently. Their 1.7 counterparts are passing. The difference between the last passing and the first failing tests is: Probably something in broke the tests. /sig scheduling /cc /kind bug
It looks like there isn't enough cpu on the master to run the cos-nvidia-installer daemonset. Did we add any addons recently?
My previous comment is a red herring. It is an issue with . Ill post a fix soon. From the scheduler logs on the master:
that's a benign error. The daemonset is not expected to run on the master. But there is no easy way to express that request in our APIs.
Yes, the pod not scheduling on the master is benign. The actual error is that we do not report node condition , so it is left as . The fix is to report that is false.
The tests are still failing. /reopen
/assign
I will be updating to track progress.
tests are green again! /close

return []func(*v1.Node) error{ kl.setNodeAddress, withoutError(kl.setNodeStatusInfo), <ins> withoutError(kl.setNodeOODCondition), </ins> withoutError(kl.setNodeMemoryPressureCondition), withoutError(kl.setNodeDiskPressureCondition), withoutError(kl.setNodeReadyCondition),


Question:
Both https://k8s- and https://k8s- started failing recently. Their 1.7 counterparts are passing. The difference between the last passing and the first failing tests is: Probably something in broke the tests. /sig scheduling /cc /kind bug
It looks like there isn't enough cpu on the master to run the cos-nvidia-installer daemonset. Did we add any addons recently?
My previous comment is a red herring. It is an issue with . Ill post a fix soon. From the scheduler logs on the master:
that's a benign error. The daemonset is not expected to run on the master. But there is no easy way to express that request in our APIs.
Yes, the pod not scheduling on the master is benign. The actual error is that we do not report node condition , so it is left as . The fix is to report that is false.
The tests are still failing. /reopen
/assign
I will be updating to track progress.
tests are green again! /close

Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { <ins> Type:               v1.NodeOutOfDisk, Status:             v1.ConditionFalse, Reason:             "KubeletHasSufficientDisk", Message:            fmt.Sprintf("kubelet has sufficient disk space available"), LastHeartbeatTime:  metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), }, { </ins> Type:               v1.NodeMemoryPressure, Status:             v1.ConditionFalse, Reason:             "KubeletHasSufficientMemory",


Question:
Both https://k8s- and https://k8s- started failing recently. Their 1.7 counterparts are passing. The difference between the last passing and the first failing tests is: Probably something in broke the tests. /sig scheduling /cc /kind bug
It looks like there isn't enough cpu on the master to run the cos-nvidia-installer daemonset. Did we add any addons recently?
My previous comment is a red herring. It is an issue with . Ill post a fix soon. From the scheduler logs on the master:
that's a benign error. The daemonset is not expected to run on the master. But there is no easy way to express that request in our APIs.
Yes, the pod not scheduling on the master is benign. The actual error is that we do not report node condition , so it is left as . The fix is to report that is false.
The tests are still failing. /reopen
/assign
I will be updating to track progress.
tests are green again! /close

Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { <ins> Type:               v1.NodeOutOfDisk, Status:             v1.ConditionFalse, Reason:             "KubeletHasSufficientDisk", Message:            fmt.Sprintf("kubelet has sufficient disk space available"), LastHeartbeatTime:  metav1.Time{}, LastTransitionTime: metav1.Time{}, }, { </ins> Type:               v1.NodeMemoryPressure, Status:             v1.ConditionFalse, Reason:             "KubeletHasSufficientMemory",


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /priority failing-test /area test-infra What happened: job is failing. See https://k8s- https://k8s- https://k8s- https://k8s- What you expected to happen: test should pass
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/assign
Reiterating the mentions to trigger a notification: <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm guessing that something got messed up in the bazel cache when reverting from 0.5.2 to 0.4.5. I'm not sure how best to fix.
How about updating everything to bazel 0.5.3 at once? I know you're trying to do a step-by-step approach to avoid breakages, but I suspect it will be cheaper to just deal with a brief outage while we update? (aka update the bazelbuild image to 0.5.3 and then merge your PR with updated and working BUILD rules?)
there's at least one other issue I've discovered with the latest go rules - the binary no longer works, due to changes introduced in that might be arguably better (at least everything still builds?), though it's frustrating that it makes things more broken in a different way.

<ins> FROM debian:jessie RUN apt-get update RUN apt-get -qy install python-seqdiag make curl WORKDIR /diagrams RUN curl -sLo DroidSansMono.ttf https://googlefontdirectory.googlecode.com/hg/apache/droidsansmono/DroidSansMono.ttf ADD . /diagrams CMD bash -c 'make >/dev/stderr && tar cf - *.png' </ins>  No newline at end of file


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /priority failing-test /area test-infra What happened: job is failing. See https://k8s- https://k8s- https://k8s- https://k8s- What you expected to happen: test should pass
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/assign
Reiterating the mentions to trigger a notification: <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm guessing that something got messed up in the bazel cache when reverting from 0.5.2 to 0.4.5. I'm not sure how best to fix.
How about updating everything to bazel 0.5.3 at once? I know you're trying to do a step-by-step approach to avoid breakages, but I suspect it will be cheaper to just deal with a brief outage while we update? (aka update the bazelbuild image to 0.5.3 and then merge your PR with updated and working BUILD rules?)
there's at least one other issue I've discovered with the latest go rules - the binary no longer works, due to changes introduced in that might be arguably better (at least everything still builds?), though it's frustrating that it makes things more broken in a different way.

%.png: %.seqdiag $(FONT) seqdiag --no-transparency -a -f '$(FONT)' $< <ins> # Build the stuff via a docker image .PHONY: docker docker: docker build -t clustering-seqdiag . docker run --rm clustering-seqdiag | tar xvf - docker-clean: docker rmi clustering-seqdiag || true docker images -q --filter "dangling=true" | xargs docker rmi fix-clock-skew: boot2docker ssh sudo date -u -D "%Y%m%d%H%M.%S" --set "$(shell date -u +%Y%m%d%H%M.%S)" </ins>


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /priority failing-test /area test-infra What happened: job is failing. See https://k8s- https://k8s- https://k8s- https://k8s- What you expected to happen: test should pass
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/assign
Reiterating the mentions to trigger a notification: <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm guessing that something got messed up in the bazel cache when reverting from 0.5.2 to 0.4.5. I'm not sure how best to fix.
How about updating everything to bazel 0.5.3 at once? I know you're trying to do a step-by-step approach to avoid breakages, but I suspect it will be cheaper to just deal with a brief outage while we update? (aka update the bazelbuild image to 0.5.3 and then merge your PR with updated and working BUILD rules?)
there's at least one other issue I've discovered with the latest go rules - the binary no longer works, due to changes introduced in that might be arguably better (at least everything still builds?), though it's frustrating that it makes things more broken in a different way.

``` Just call `make` to regenerate the diagrams. <ins> ## Building with Docker If you are on a Mac or your pip install is messed up, you can easily build with docker. ``` make docker ``` The first run will be slow but things should be fast after that. To clean up the docker containers that are created (and other cruft that is left around) you can run `make docker-clean`. If you are using boot2docker and get warnings about clock skew (or if things aren't building for some reason) then you can fix that up with `make fix-clock-skew`. ## Automatically rebuild on file changes If you have the fswatch utility installed, you can have it monitor the file system and automatically rebuild when files have changed.  Just do a `make watch`. </ins>  No newline at end of file


Question: All controllers that use the field have a default value of 10 for the field, except for , which has a default of 2. We should increase it to 10 for consistency on its default value across controllers. /sig apps

"format": "int32" }, "revisionHistoryLimit": { <del> "description": "The number of old ReplicaSets to retain to allow rollback. This is a pointer to distinguish between explicit zero and not specified. Defaults to 2.", </del> <ins> "description": "The number of old ReplicaSets to retain to allow rollback. This is a pointer to distinguish between explicit zero and not specified. Defaults to 10.", </ins> "type": "integer", "format": "int32" },


Question: All controllers that use the field have a default value of 10 for the field, except for , which has a default of 2. We should increase it to 10 for consistency on its default value across controllers. /sig apps

"revisionHistoryLimit": { "type": "integer", "format": "int32", <del> "description": "The number of old ReplicaSets to retain to allow rollback. This is a pointer to distinguish between explicit zero and not specified. Defaults to 2." </del> <ins> "description": "The number of old ReplicaSets to retain to allow rollback. This is a pointer to distinguish between explicit zero and not specified. Defaults to 10." </ins> }, "paused": { "type": "boolean",


Question: All controllers that use the field have a default value of 10 for the field, except for , which has a default of 2. We should increase it to 10 for consistency on its default value across controllers. /sig apps

revisionHistoryLimit

The number of old ReplicaSets to retain to allow rollback. This is a pointer to distinguish between explicit zero and not specified. Defaults to 2.

The number of old ReplicaSets to retain to allow rollback. This is a pointer to distinguish between explicit zero and not specified. Defaults to 10.

false

integer (int32)


Question: All controllers that use the field have a default value of 10 for the field, except for , which has a default of 2. We should increase it to 10 for consistency on its default value across controllers. /sig apps

// in extensions. These addons are: // - MaxUnavailable during rolling update set to 25% (1 in extensions) // - MaxSurge value during rolling update set to 25% (1 in extensions) <del> // - RevisionHistoryLimit set to 2 (not set in extensions) </del> <ins> // - RevisionHistoryLimit set to 10 (not set in extensions) </ins> // - ProgressDeadlineSeconds set to 600s (not set in extensions) func SetDefaults_Deployment(obj *appsv1beta2.Deployment) { // Default labels and selector to labels from pod template spec.


Question: All controllers that use the field have a default value of 10 for the field, except for , which has a default of 2. We should increase it to 10 for consistency on its default value across controllers. /sig apps

} if obj.Spec.RevisionHistoryLimit == nil { obj.Spec.RevisionHistoryLimit = new(int32) <del> *obj.Spec.RevisionHistoryLimit = 2 </del> <ins> *obj.Spec.RevisionHistoryLimit = 10 </ins> } if obj.Spec.ProgressDeadlineSeconds == nil { obj.Spec.ProgressDeadlineSeconds = new(int32)


Question: All controllers that use the field have a default value of 10 for the field, except for , which has a default of 2. We should increase it to 10 for consistency on its default value across controllers. /sig apps

MaxUnavailable: &defaultIntOrString, }, }, <del> RevisionHistoryLimit:    newInt32(2), </del> <ins> RevisionHistoryLimit:    newInt32(10), </ins> ProgressDeadlineSeconds: newInt32(600), Template:                defaultTemplate, },


Question: All controllers that use the field have a default value of 10 for the field, except for , which has a default of 2. We should increase it to 10 for consistency on its default value across controllers. /sig apps

// The number of old ReplicaSets to retain to allow rollback. // This is a pointer to distinguish between explicit zero and not specified. <del> // Defaults to 2. </del> <ins> // Defaults to 10. </ins> // +optional optional int32 revisionHistoryLimit = 6;


Question: All controllers that use the field have a default value of 10 for the field, except for , which has a default of 2. We should increase it to 10 for consistency on its default value across controllers. /sig apps

// The number of old ReplicaSets to retain to allow rollback. // This is a pointer to distinguish between explicit zero and not specified. <del> // Defaults to 2. </del> <ins> // Defaults to 10. </ins> // +optional RevisionHistoryLimit *int32 `json:"revisionHistoryLimit,omitempty" protobuf:"varint,6,opt,name=revisionHistoryLimit"`


Question: All controllers that use the field have a default value of 10 for the field, except for , which has a default of 2. We should increase it to 10 for consistency on its default value across controllers. /sig apps

"template":                "Template describes the pods that will be created.", "strategy":                "The deployment strategy to use to replace existing pods with new ones.", "minReadySeconds":         "Minimum number of seconds for which a newly created pod should be ready without any of its container crashing, for it to be considered available. Defaults to 0 (pod will be considered available as soon as it is ready)", <del> "revisionHistoryLimit":    "The number of old ReplicaSets to retain to allow rollback. This is a pointer to distinguish between explicit zero and not specified. Defaults to 2.", </del> <ins> "revisionHistoryLimit":    "The number of old ReplicaSets to retain to allow rollback. This is a pointer to distinguish between explicit zero and not specified. Defaults to 10.", </ins> "paused":                  "Indicates that the deployment is paused.", "rollbackTo":              "The config this deployment is rolling back to. Will be cleared after rollback is done.", "progressDeadlineSeconds": "The maximum time in seconds for a deployment to make progress before it is considered to be failed. The deployment controller will continue to process failed deployments and a condition with a ProgressDeadlineExceeded reason will be surfaced in the deployment status. Once autoRollback is implemented, the deployment controller will automatically rollback failed deployments. Note that progress will not be estimated during the time a deployment is paused. Defaults to 600s.",


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I was attempting to filter pods with a using client-go selecting by its . I tried multiple variations until I ended up with . When filtering by , list of pod(s) came back empty even if pod with the IP we are filtering by was running. Then I tried reproducing it by calling the Kubernetes API directly which had the same result, list of pods came back empty: What you expected to happen: When passing the IP of the previously retrieved running Pod, I expected to get that Pod returned in the result. How to reproduce it (as minimally and precisely as possible): create a simple , get its IP (For example using - query Kubernetes: Anything else we need to know?: If I leave out the pod IP, I get all the pods back. I also tried using different formats when doing the curl, not one of them return anything. Also it seems like field is . As a side note I retrieved the URL which I used from debugging which calls client-go made. Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: minikube version: v0.20.0 OS (e.g. from /etc/os-release): minikube OS Kernel (e.g. ): minikube kernel: Install tools: minikube Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Currently the for only support , , , and . I've already submitted a PR in . PTAL.
cc

podSpecificFieldsSet["spec.nodeName"] = pod.Spec.NodeName podSpecificFieldsSet["spec.restartPolicy"] = string(pod.Spec.RestartPolicy) podSpecificFieldsSet["status.phase"] = string(pod.Status.Phase) <ins> podSpecificFieldsSet["status.podIP"] = string(pod.Status.PodIP) </ins> return generic.AddObjectMetaFieldsSet(podSpecificFieldsSet, &pod.ObjectMeta, true) }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I was attempting to filter pods with a using client-go selecting by its . I tried multiple variations until I ended up with . When filtering by , list of pod(s) came back empty even if pod with the IP we are filtering by was running. Then I tried reproducing it by calling the Kubernetes API directly which had the same result, list of pods came back empty: What you expected to happen: When passing the IP of the previously retrieved running Pod, I expected to get that Pod returned in the result. How to reproduce it (as minimally and precisely as possible): create a simple , get its IP (For example using - query Kubernetes: Anything else we need to know?: If I leave out the pod IP, I get all the pods back. I also tried using different formats when doing the curl, not one of them return anything. Also it seems like field is . As a side note I retrieved the URL which I used from debugging which calls client-go made. Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: minikube version: v0.20.0 OS (e.g. from /etc/os-release): minikube OS Kernel (e.g. ): minikube kernel: Install tools: minikube Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Currently the for only support , , , and . I've already submitted a PR in . PTAL.
cc

fieldSelector: fields.ParseSelectorOrDie("status.phase=Pending"), expectMatch:   false, }, <ins> { in: &api.Pod{ Status: api.PodStatus{PodIP: "1.2.3.4"}, }, fieldSelector: fields.ParseSelectorOrDie("status.podIP=1.2.3.4"), expectMatch:   true, }, { in: &api.Pod{ Status: api.PodStatus{PodIP: "1.2.3.4"}, }, fieldSelector: fields.ParseSelectorOrDie("status.podIP=4.3.2.1"), expectMatch:   false, }, </ins> } for _, testCase := range testCases { m := MatchPod(labels.Everything(), testCase.fieldSelector)


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup /sig scheduling Descriptions: According to the discussion at , we'll move the node condition filter into a new predicates; and use feature gate to turn-off/on it. /cc
/assign

factory.RegisterFitPredicate("CheckNodeDiskPressure", predicates.CheckNodeDiskPressurePredicate), // Fit is determied by node condtions: not ready, network unavailable and out of disk. <del> factory.RegisterFitPredicate("CheckNodeCondition", predicates.CheckNodeConditionPredicate), </del> factory.RegisterMandatoryFitPredicate("CheckNodeCondition", predicates.CheckNodeConditionPredicate), // Fit is determined by volume zone requirements.


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup /sig scheduling Descriptions: According to the discussion at , we'll move the node condition filter into a new predicates; and use feature gate to turn-off/on it. /cc
/assign

schedulerFactoryMutex sync.Mutex // maps that hold registered algorithm types <del> fitPredicateMap          = make(map[string]FitPredicateFactory) mandatoryFitPredicateMap = make(map[string]FitPredicateFactory) priorityFunctionMap      = make(map[string]PriorityConfigFactory) algorithmProviderMap     = make(map[string]AlgorithmProviderConfig) </del> <ins> fitPredicateMap        = make(map[string]FitPredicateFactory) mandatoryFitPredicates = sets.NewString() priorityFunctionMap    = make(map[string]PriorityConfigFactory) algorithmProviderMap   = make(map[string]AlgorithmProviderConfig) </ins> // Registered metadata producers priorityMetadataProducer  MetadataProducerFactory


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup /sig scheduling Descriptions: According to the discussion at , we'll move the node condition filter into a new predicates; and use feature gate to turn-off/on it. /cc
/assign

schedulerFactoryMutex.Lock() defer schedulerFactoryMutex.Unlock() validateAlgorithmNameOrDie(name) <del> mandatoryFitPredicateMap[name] = func(PluginFactoryArgs) algorithm.FitPredicate { return predicate } </del> <ins> fitPredicateMap[name] = func(PluginFactoryArgs) algorithm.FitPredicate { return predicate } mandatoryFitPredicates.Insert(name) </ins> return name }


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup /sig scheduling Descriptions: According to the discussion at , we'll move the node condition filter into a new predicates; and use feature gate to turn-off/on it. /cc
/assign

predicates[name] = factory(args) } <del> // Always include required fit predicates. for name, factory := range mandatoryFitPredicateMap { if _, found := predicates[name]; !found { </del> <ins> // Always include mandatory fit predicates. for name := range mandatoryFitPredicates { if factory, found := fitPredicateMap[name]; found { </ins> predicates[name] = factory(args) } }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals

return nil, apierrors.NewBadRequest(fmt.Sprintf("namespace is not allowed on this type: %v", namespace)) } <ins> if len(tokenReview.Spec.Token) == 0 { return nil, apierrors.NewBadRequest(fmt.Sprintf("token is required for TokenReview in authentication")) } </ins> if r.tokenAuthenticator == nil { return tokenReview, nil }


Question:
There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery

"type": "array", "items": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Initializer" <del> } </del> <ins> }, "x-kubernetes-patch-merge-key": "name", "x-kubernetes-patch-strategy": "merge" </ins> }, "result": { "description": "If result is set with the Failure field, the object will be persisted to storage and then deleted, ensuring that other clients can observe the deletion.",


Question:
There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery

library = ":go_default_library", deps = [ "//vendor/k8s.io/api/admissionregistration/v1alpha1:go_default_library", <ins> "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/meta:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/runtime:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", <ins> "//vendor/k8s.io/apiserver/pkg/admission:go_default_library", "//vendor/k8s.io/apiserver/pkg/authorization/authorizer:go_default_library", </ins> ], )


Question:
There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery

glog.V(5).Infof("Modifying uninitialized resource %s", a.GetResource()) <ins> if updated != nil && len(updated.Pending) == 0 && updated.Result == nil { accessor.SetInitializers(nil) } </ins> // because we are called before validation, we need to ensure the update transition is valid. if errs := validation.ValidateInitializersUpdate(updated, existing, initializerFieldPath); len(errs) > 0 { return errors.NewInvalid(a.GetKind().GroupKind(), a.GetName(), errs)


Question:
There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery

import ( "reflect" <ins> "strings" </ins> "testing" "k8s.io/api/admissionregistration/v1alpha1" <ins> "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" </ins> "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apiserver/pkg/admission" "k8s.io/apiserver/pkg/authorization/authorizer" </ins> ) func newInitializer(name string, rules ...v1alpha1.Rule) *v1alpha1.InitializerConfiguration {


Question:
There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery

}) } } <ins> type fakeAuthorizer struct { accept bool } func (f *fakeAuthorizer) Authorize(a authorizer.Attributes) (bool, string, error) { if f.accept { return true, "", nil } return false, "denied", nil } func TestAdmitUpdate(t *testing.T) { tests := []struct { name             string oldInitializers  *metav1.Initializers newInitializers  *metav1.Initializers verifyUpdatedObj func(runtime.Object) (pass bool, reason string) err              string }{ { name:            "updates on initialized resources are allowed", oldInitializers: nil, newInitializers: nil, err:             "", }, { name:            "updates on initialized resources are allowed", oldInitializers: &metav1.Initializers{Pending: []metav1.Initializer{{Name: "init.k8s.io"}}}, newInitializers: &metav1.Initializers{}, verifyUpdatedObj: func(obj runtime.Object) (bool, string) { accessor, err := meta.Accessor(obj) if err != nil { return false, "cannot get accessor" } if accessor.GetInitializers() != nil { return false, "expect nil initializers" } return true, "" }, err: "", }, { name:            "initializers may not be set once initialized", oldInitializers: nil, newInitializers: &metav1.Initializers{Pending: []metav1.Initializer{{Name: "init.k8s.io"}}}, err:             "field is immutable once initialization has completed", }, } plugin := initializer{ config:     nil, authorizer: &fakeAuthorizer{true}, } for _, tc := range tests { oldObj := &v1.Pod{} oldObj.Initializers = tc.oldInitializers newObj := &v1.Pod{} newObj.Initializers = tc.newInitializers a := admission.NewAttributesRecord(newObj, oldObj, schema.GroupVersionKind{}, "", "foo", schema.GroupVersionResource{}, "", admission.Update, nil) err := plugin.Admit(a) switch { case tc.err == "" && err != nil: t.Errorf("%q: unexpected error: %v", tc.name, err) case tc.err != "" && err == nil: t.Errorf("%q: unexpected no error, expected %s", tc.name, tc.err) case tc.err != "" && err != nil && !strings.Contains(err.Error(), tc.err): t.Errorf("%q: expected %s, got %v", tc.name, tc.err, err) } } } </ins>


Question:
There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery

} } allErrs = append(allErrs, validateInitializersResult(initializers.Result, fldPath.Child("result"))...) <del> if len(initializers.Pending) == 0 && initializers.Result == nil { allErrs = append(allErrs, field.Invalid(fldPath.Child("pending"), nil, "must be non-empty when result is not set")) } </del> return allErrs }


Question:
There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery

// When the last pending initializer is removed, and no failing result is set, the initializers // struct will be set to nil and the object is considered as initialized and visible to all // clients. <ins> // +patchMergeKey=name // +patchStrategy=merge </ins> repeated Initializer pending = 1; // If result is set with the Failure field, the object will be persisted to storage and then deleted,


Question:
There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery

// When the last pending initializer is removed, and no failing result is set, the initializers // struct will be set to nil and the object is considered as initialized and visible to all // clients. <del> Pending []Initializer `json:"pending" protobuf:"bytes,1,rep,name=pending"` </del> <ins> // +patchMergeKey=name // +patchStrategy=merge Pending []Initializer `json:"pending" protobuf:"bytes,1,rep,name=pending" patchStrategy:"merge" patchMergeKey:"name"` </ins> // If result is set with the Failure field, the object will be persisted to storage and then deleted, // ensuring that other clients can observe the deletion. Result *Status `json:"result,omitempty" protobuf:"bytes,2,opt,name=result"`


Question:
There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery

Expect(err).NotTo(HaveOccurred()) Expect(len(pods.Items)).Should(Equal(1)) }) <ins> It("will be set to nil if a patch removes the last pending initializer", func() { ns := f.Namespace.Name c := f.ClientSet podName := "to-be-patch-initialized-pod" framework.Logf("Creating pod %s", podName) // TODO: lower the timeout so that the server responds faster. _, err := c.CoreV1().Pods(ns).Create(newUninitializedPod(podName)) if err != nil && !errors.IsTimeout(err) { framework.Failf("expect err to be timeout error, got %v", err) } uninitializedPod, err := c.CoreV1().Pods(ns).Get(podName, metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred()) Expect(uninitializedPod.Initializers).NotTo(BeNil()) Expect(len(uninitializedPod.Initializers.Pending)).Should(Equal(1)) patch := fmt.Sprintf(`{"metadata":{"initializers":{"pending":[{"$patch":"delete","name":"%s"}]}}}`, uninitializedPod.Initializers.Pending[0].Name) patchedPod, err := c.CoreV1().Pods(ns).Patch(uninitializedPod.Name, types.StrategicMergePatchType, []byte(patch)) Expect(err).NotTo(HaveOccurred()) Expect(patchedPod.Initializers).To(BeNil()) }) </ins> }) func newUninitializedPod(podName string) *v1.Pod {


Question:
Seen in https://k8s- Aug 26 09:35:45.074: Couldn't delete ns: "e2e-tests-containers-9cscj": an error on the server ("missing route (service "sample-api" not found)") has prevented the request from succeeding (get ) (&errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"an error on the server ("missing route (service "sample-api" not found)") has prevented the request from succeeding (get )", Reason:"InternalError", Details:(*v1.StatusDetails)(0xc4217a4900), Code:500}})
https://k8s-

}) func cleanTest(f *framework.Framework, block bool) { <ins> // delete the APIService first to avoid causing discovery errors aggrclient := f.AggregatorClient _ = aggrclient.ApiregistrationV1beta1().APIServices().Delete("v1alpha1.wardle.k8s.io", nil) </ins> namespace := "sample-system" client := f.ClientSet _ = client.ExtensionsV1beta1().Deployments(namespace).Delete("sample-apiserver", nil)


Question:
Seen in https://k8s- Aug 26 09:35:45.074: Couldn't delete ns: "e2e-tests-containers-9cscj": an error on the server ("missing route (service "sample-api" not found)") has prevented the request from succeeding (get ) (&errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"an error on the server ("missing route (service "sample-api" not found)") has prevented the request from succeeding (get )", Reason:"InternalError", Details:(*v1.StatusDetails)(0xc4217a4900), Code:500}})
https://k8s-

_ = client.CoreV1().Namespaces().Delete(namespace, nil) _ = client.RbacV1beta1().ClusterRoles().Delete("wardler", nil) _ = client.RbacV1beta1().ClusterRoleBindings().Delete("wardler:sample-system:anonymous", nil) <del> aggrclient := f.AggregatorClient _ = aggrclient.ApiregistrationV1beta1().APIServices().Delete("v1alpha1.wardle.k8s.io", nil) </del> if block { _ = wait.Poll(100*time.Millisecond, 5*time.Second, func() (bool, error) { _, err := client.CoreV1().Namespaces().Get("sample-system", metav1.GetOptions{})


Question:
Seen in https://k8s- Aug 26 09:35:45.074: Couldn't delete ns: "e2e-tests-containers-9cscj": an error on the server ("missing route (service "sample-api" not found)") has prevented the request from succeeding (get ) (&errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"an error on the server ("missing route (service "sample-api" not found)") has prevented the request from succeeding (get )", Reason:"InternalError", Details:(*v1.StatusDetails)(0xc4217a4900), Code:500}})
https://k8s-

return numPods, missingTimestamp, nil } <ins> // isDynamicDiscoveryError returns true if the error is a group discovery error // only for groups expected to be created/deleted dynamically during e2e tests func isDynamicDiscoveryError(err error) bool { if !discovery.IsGroupDiscoveryFailedError(err) { return false } discoveryErr := err.(*discovery.ErrGroupDiscoveryFailed) for gv := range discoveryErr.Groups { switch gv.Group { case "mygroup.example.com": // custom_resource_definition // garbage_collector case "wardle.k8s.io": // aggregator default: Logf("discovery error for unexpected group: %#v", gv) return false } } return false } </ins> // hasRemainingContent checks if there is remaining content in the namespace via API discovery func hasRemainingContent(c clientset.Interface, clientPool dynamic.ClientPool, namespace string) (bool, error) { // some tests generate their own framework.Client rather than the default


Question:
Seen in https://k8s- Aug 26 09:35:45.074: Couldn't delete ns: "e2e-tests-containers-9cscj": an error on the server ("missing route (service "sample-api" not found)") has prevented the request from succeeding (get ) (&errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:""}, Status:"Failure", Message:"an error on the server ("missing route (service "sample-api" not found)") has prevented the request from succeeding (get )", Reason:"InternalError", Details:(*v1.StatusDetails)(0xc4217a4900), Code:500}})
https://k8s-

// find out what content is supported on the server resources, err := c.Discovery().ServerPreferredNamespacedResources() <del> if err != nil { </del> <ins> if err != nil && !isDynamicDiscoveryError(err) { </ins> return false, err } groupVersionResources, err := discovery.GroupVersionResources(resources) <del> if err != nil { </del> <ins> if err != nil && !isDynamicDiscoveryError(err) { </ins> return false, err }


} debugLogThresholdsWithObservation("thresholds - reclaim not satisfied", thresholds, observations) <del> // determine the set of thresholds whose stats have been updated since the last sync thresholds = thresholdsUpdatedStats(thresholds, observations, m.lastObservations) debugLogThresholdsWithObservation("thresholds - updated stats", thresholds, observations) </del> // track when a threshold was first observed now := m.clock.Now() thresholdsFirstObservedAt := thresholdsFirstObservedAt(thresholds, m.thresholdsFirstObservedAt, now)


m.thresholdsFirstObservedAt = thresholdsFirstObservedAt m.nodeConditionsLastObservedAt = nodeConditionsLastObservedAt m.thresholdsMet = thresholds <ins> // determine the set of thresholds whose stats have been updated since the last sync thresholds = thresholdsUpdatedStats(thresholds, observations, m.lastObservations) debugLogThresholdsWithObservation("thresholds - updated stats", thresholds, observations) </ins> m.lastObservations = observations m.Unlock()


Question:
We should solve this problem in a better way
cc :))
/sig contributor-experience release
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/assign

color: bfe5bf - name: do-not-merge color: e11d21 <ins> - name: do-not-merge/work-in-progress color: e11d21 - name: do-not-merge/hold color: e11d21 - name: do-not-merge/cherry-pick-not-approved color: e11d21 - name: do-not-merge/release-note-label-needed color: e11d21 - name: do-not-merge/blocked-paths color: e11d21 </ins> - name: flake-has-meta color: fbca04 - name: for-new-contributors


Question:
https://k8s- https://k8s- cc: /kind bug /priority failing-test
Thanks a lot for reporting this! I think I have a fix PR for this issue. Or should we roll PR back?
I've kicked off testing of your PR. If it fixes the cross build, then that may be the easier path than reverting. cc: (sig-node leads)
I also fixed it in since it also touches the imports in that file. Would love to not rebase it yet again for this.
That call would be up to or they would have to determine which approach they prefer (fix on it's own in a small PR, or fix as part of a larger PR to prevent rebase).
Smaller is probably better; that way if the combined one needs to be reverted it doesn't re-break cross as a result. Will rebase if necessary (will depend on queue order.)

"k8s.io/api/core/v1" "k8s.io/client-go/tools/record" "k8s.io/kubernetes/pkg/kubelet/cadvisor" <ins> kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" </ins> "k8s.io/kubernetes/pkg/util/mount" )


Question:
A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet. For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.
Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .

// without needing to be deleted and recreated. if firewallExists { glog.Infof("EnsureLoadBalancer(%v(%v)): updating firewall", loadBalancerName, serviceName) <del> if err := gce.updateFirewall(apiService, makeFirewallName(loadBalancerName), gce.region, desc, sourceRanges, ports, hosts); err != nil { </del> <ins> if err := gce.updateFirewall(apiService, MakeFirewallName(loadBalancerName), gce.region, desc, sourceRanges, ports, hosts); err != nil { </ins> return nil, err } glog.Infof("EnsureLoadBalancer(%v(%v)): updated firewall", loadBalancerName, serviceName) } else { glog.Infof("EnsureLoadBalancer(%v(%v)): creating firewall", loadBalancerName, serviceName) <del> if err := gce.createFirewall(apiService, makeFirewallName(loadBalancerName), gce.region, desc, sourceRanges, ports, hosts); err != nil { </del> <ins> if err := gce.createFirewall(apiService, MakeFirewallName(loadBalancerName), gce.region, desc, sourceRanges, ports, hosts); err != nil { </ins> return nil, err } glog.Infof("EnsureLoadBalancer(%v(%v)): created firewall", loadBalancerName, serviceName)


Question:
A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet. For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.
Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .

// target pool to use local traffic health check. glog.V(2).Infof("Updating from nodes health checks to local traffic health checks for service %v LB %v", apiService.Name, loadBalancerName) if supportsNodesHealthCheck { <del> hcToDelete = makeHttpHealthCheck(makeNodesHealthCheckName(clusterID), GetNodesHealthCheckPath(), GetNodesHealthCheckPort()) </del> <ins> hcToDelete = makeHttpHealthCheck(MakeNodesHealthCheckName(clusterID), GetNodesHealthCheckPath(), GetNodesHealthCheckPort()) </ins> } tpNeedsUpdate = true }


Question:
A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet. For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.
Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .

tpNeedsUpdate = true } if supportsNodesHealthCheck { <del> hcToCreate = makeHttpHealthCheck(makeNodesHealthCheckName(clusterID), GetNodesHealthCheckPath(), GetNodesHealthCheckPort()) </del> <ins> hcToCreate = makeHttpHealthCheck(MakeNodesHealthCheckName(clusterID), GetNodesHealthCheckPath(), GetNodesHealthCheckPort()) </ins> } } // Now we get to some slightly more interesting logic.


Question:
A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet. For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.
Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .

// using local traffic health check or nodes health check. Attempt to delete // both to prevent leaking. hcNames = append(hcNames, loadBalancerName) <del> hcNames = append(hcNames, makeNodesHealthCheckName(clusterID)) </del> <ins> hcNames = append(hcNames, MakeNodesHealthCheckName(clusterID)) </ins> } errs := utilerrors.AggregateGoroutines( func() error { <del> fwName := makeFirewallName(loadBalancerName) </del> <ins> fwName := MakeFirewallName(loadBalancerName) </ins> err := ignoreNotFound(gce.DeleteFirewall(fwName)) if isForbidden(err) && gce.OnXPN() { glog.V(4).Infof("ensureExternalLoadBalancerDeleted(%v): do not have permission to delete firewall rule (on XPN). Raising event.", loadBalancerName)


Question:
A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet. For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.
Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .

} func (gce *GCECloud) firewallNeedsUpdate(name, serviceName, region, ipAddress string, ports []v1.ServicePort, sourceRanges netsets.IPNet) (exists bool, needsUpdate bool, err error) { <del> fw, err := gce.service.Firewalls.Get(gce.NetworkProjectID(), makeFirewallName(name)).Do() </del> <ins> fw, err := gce.service.Firewalls.Get(gce.NetworkProjectID(), MakeFirewallName(name)).Do() </ins> if err != nil { if isHTTPErrorCode(err, http.StatusNotFound) { return false, true, nil


Question:
A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet. For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.
Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .

return fmt.Sprintf(`{"kubernetes.io/service-name":"%s"}`, serviceName) } <del> // makeNodesHealthCheckName returns name of the health check resource used by </del> <ins> // MakeNodesHealthCheckName returns name of the health check resource used by </ins> // the GCE load balancers (l4) for performing health checks on nodes. <del> func makeNodesHealthCheckName(clusterID string) string { </del> <ins> func MakeNodesHealthCheckName(clusterID string) string { </ins> return fmt.Sprintf("k8s-%v-node", clusterID) }


Question:
A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet. For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.
Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .

// balancers (l4) for performing health checks. func MakeHealthCheckFirewallName(clusterID, hcName string, isNodesHealthCheck bool) string { if isNodesHealthCheck { <del> return makeNodesHealthCheckName(clusterID) + "-http-hc" </del> <ins> return MakeNodesHealthCheckName(clusterID) + "-http-hc" </ins> } return "k8s-" + hcName + "-http-hc" } <del> func makeFirewallName(name string) string { </del> <ins> // MakeFirewallName returns the firewall name used by the GCE load // balancers (l4) for serving traffic. func MakeFirewallName(name string) string { </ins> return fmt.Sprintf("k8s-fw-%s", name) }


Question:
A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet. For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.
Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .

if err != nil { return fmt.Errorf("error parsing GCE/GKE region from zone %q: %v", zone, err) } <del> if err := gceCloud.DeleteFirewall(loadBalancerName); err != nil && </del> <ins> if err := gceCloud.DeleteFirewall(gcecloud.MakeFirewallName(loadBalancerName)); err != nil && </ins> !IsGoogleAPIHTTPErrorCode(err, http.StatusNotFound) { retErr = err }


Question:
A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet. For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.
Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .

!IsGoogleAPIHTTPErrorCode(err, http.StatusNotFound) { retErr = fmt.Errorf("%vn%v", retErr, err) } <del> var hcNames []string </del> <ins> clusterID, err := GetClusterID(c) if err != nil { retErr = fmt.Errorf("%vn%v", retErr, err) return } hcNames := []string{gcecloud.MakeNodesHealthCheckName(clusterID)} </ins> hc, getErr := gceCloud.GetHttpHealthCheck(loadBalancerName) if getErr != nil && !IsGoogleAPIHTTPErrorCode(getErr, http.StatusNotFound) { retErr = fmt.Errorf("%vn%v", retErr, getErr)


Question:
A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet. For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.
Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .

if hc != nil { hcNames = append(hcNames, hc.Name) } <del> clusterID, err := GetClusterID(c) if err != nil { retErr = fmt.Errorf("%vn%v", retErr, err) return } </del> if err := gceCloud.DeleteExternalTargetPoolAndChecks(nil, loadBalancerName, region, clusterID, hcNames...); err != nil && !IsGoogleAPIHTTPErrorCode(err, http.StatusNotFound) { retErr = fmt.Errorf("%vn%v", retErr, err)