Question: expectedReconciledRole:       aggregatedRole(aggregationrule([]map[string]string{{"alpha": "bravo"}, {"foo": "bar"}})), expectedReconciliationNeeded: true, }, <ins> "unexpected aggregation": { // desired role is not aggregated expectedRole: role(rules("pods", "nodes", "secrets"), nil, nil), // existing role is aggregated actualRole:             aggregatedRole(aggregationrule([]map[string]string{{"alpha": "bravo"}})), removeExtraPermissions: false, // reconciled role should have desired permissions and not be aggregated expectedReconciledRole:       role(rules("pods", "nodes", "secrets"), nil, nil), expectedReconciliationNeeded: true, }, "unexpected aggregation with differing permissions": { // desired role is not aggregated expectedRole: role(rules("pods", "nodes", "secrets"), nil, nil), // existing role is aggregated and has other permissions actualRole: func() *rbac.ClusterRole { r := aggregatedRole(aggregationrule([]map[string]string{{"alpha": "bravo"}})) r.Rules = rules("deployments") return r }(), removeExtraPermissions: false, // reconciled role should have aggregation removed, preserve differing permissions, and include desired permissions expectedReconciledRole:       role(rules("deployments", "pods", "nodes", "secrets"), nil, nil), expectedReconciliationNeeded: true, }, </ins> } for k, tc := range tests {

/kind bug What happened: I deployed a fresh custom HA kubernetes v1.10.2 cluster with 3 nodes (all controller and worker currently, more workers planned). I then created a namespace called "stummi" and a rolebinding with "user=stummi clusterrole=admin" in this namespace. User stummi (successfully authenticated by TLS client certificate) was not able to list pods or anything else in his namespace: While trying to figure things out, I downgraded to v1.8.11 (a version without cluster role aggregation, that worked) and also tried v1.11 alpha2 (which did not work). I never purged my etcd and I sadly do not have logs from before the different versions. Shortly after posting on slack and before writing this ticket, I dumped the cluster roles in question: Note the rules. What you expected to happen: I expected user stummi to be able to do most things in his namespace. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I posted this problem on slack and helped me and wanted this issue opened. /sig auth The cluster is deployed with ansible and I can give the playbooks when required. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: 3 virtual machines (2 cores, 8GB memory) OS (e.g. from /etc/os-release): Debian Stretch Kernel (e.g. ): Install tools: custom ansible playbooks Others: Please let me know if you need more information. I'm happy to provide everything you need to fix this :)
/assign


Question: ## Set resource limits/request of a deployment # Pre-condition: no deployment exists kube::test::get_object_assert deployment "{{range.items}}{{$id_field}}:{{end}}" '' <del> # Set resources of a local file without talking to the server kubectl set resources -f hack/testdata/deployment-multicontainer.yaml -c=perl --limits=cpu=300m --requests=cpu=300m --local -o yaml "${kube_flags[@]}" ! kubectl set resources -f hack/testdata/deployment-multicontainer.yaml -c=perl --limits=cpu=300m --requests=cpu=300m --dry-run -o yaml "${kube_flags[@]}" </del> # Create a deployment kubectl create -f hack/testdata/deployment-multicontainer.yaml "${kube_flags[@]}" kube::test::get_object_assert deployment "{{range.items}}{{$id_field}}:{{end}}" 'nginx-deployment:' kube::test::get_object_assert deployment "{{range.items}}{{$deployment_image_field}}:{{end}}" "${IMAGE_DEPLOYMENT_R1}:" kube::test::get_object_assert deployment "{{range.items}}{{$deployment_second_image_field}}:{{end}}" "${IMAGE_PERL}:" # Set the deployment's cpu limits <del> kubectl set resources deployment nginx-deployment --limits=cpu=200m "${kube_flags[@]}" kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 0).resources.limits.cpu}}:{{end}}" "200m:" kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 1).resources.limits.cpu}}:{{end}}" "200m:" # Set the deployments memory limits without affecting the cpu limits kubectl set resources deployment nginx-deployment --limits=memory=256Mi "${kube_flags[@]}" kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 0).resources.limits.cpu}}:{{end}}" "200m:" kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 0).resources.limits.memory}}:{{end}}" "256Mi:" # Set the deployments resource requests without effecting the deployments limits kubectl set resources deployment nginx-deployment --requests=cpu=100m,memory=256Mi "${kube_flags[@]}" kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 0).resources.limits.cpu}}:{{end}}" "200m:" kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 0).resources.limits.memory}}:{{end}}" "256Mi:" kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 0).resources.requests.cpu}}:{{end}}" "100m:" kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 0).resources.requests.memory}}:{{end}}" "256Mi:" # Set the deployments resource requests and limits to zero kubectl set resources deployment nginx-deployment --requests=cpu=0,memory=0 --limits=cpu=0,memory=0 "${kube_flags[@]}" </del> <ins> kubectl set resources deployment nginx-deployment --limits=cpu=100m "${kube_flags[@]}" kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 0).resources.limits.cpu}}:{{end}}" "100m:" kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 1).resources.limits.cpu}}:{{end}}" "100m:" </ins> # Set a non-existing container should fail ! kubectl set resources deployment nginx-deployment -c=redis --limits=cpu=100m # Set the limit of a specific container in deployment <del> kubectl set resources deployment nginx-deployment --limits=cpu=100m "${kube_flags[@]}" </del> kubectl set resources deployment nginx-deployment -c=nginx --limits=cpu=200m "${kube_flags[@]}" kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 0).resources.limits.cpu}}:{{end}}" "200m:" kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 1).resources.limits.cpu}}:{{end}}" "100m:"

from https://k8s-
Really suspect this is
This is putting us into test-go post-commit run is broken
cc:
marked this p0 - if a fix isn't easy let's try to identify what introduced this and revert.
This flake on: 2016-10-30 07:19 PDT First instance: : It is flaking frequently, blocking PRs from getting merged, and slowing the submit queue ( at 75%). As on-call, I'm going to try and identify the cluprit and revert. Can use all help in doing so.
can you investigate whether your additional tests could cause this (by breaking the deployment of those pods, for instance) , Saad Ali wrote:
PR looks most suspect to me. It mucked with , and the errors we are seeing in each of the failures is related to : That PR was merged at Oct 30, 2016, 6:18 AM PDT and the first flake in was at Oct 30, 2016, 7:19 AM PDT. Thoughts? CC
I'm going to initiate a revert of to see if that fixes the issue
IMO we can just change the flaky test to not get pods but some other resources, such as deployments. We can't really expect to clean up pods immediately.
I'm going to proceed with revert (). Please feel free to send a follow up PR to handle this more gracefully.
Going to leave this opened until we have a few hours of runs with the revert. Want to verify, with some degree of certainty, that the revert took care of this flake.
We had 15 green runs of in a row. Considering this flake fixed.


Question: // we have removed content, so mark it finalized by us result, err := retryOnConflictError(kubeClient, namespace, finalizeNamespaceFunc) if err != nil { <ins> // in normal practice, this should not be possible, but if a deployment is running // two controllers to do namespace deletion that share a common finalizer token it's // possible that a not found could occur since the other controller would have finished the delete. if errors.IsNotFound(err) { return nil } </ins> return err }

If the namespace controller gets an error when doing a sync and the error is not a error the error will not be handled. Ran in to this when we found When deleting a namespace it received a error that wasn't reported anywhere and the namespace stuck in because a finalizer was left.
Other things noticed in the namespace controller: need a crash handler in sweep for all error return cases and determine whether should be returned determine what to do with non- errors at ... I think that currently it just hangs termination of that namespace until a resync
I will take a look. Thanks for the heads up.


Question: klog.Errorf("cannot convert to *v1.Pod: %v", obj) return } <ins> klog.V(3).Infof("add event for scheduled pod %s/%s ", pod.Namespace, pod.Name) </ins> if err := sched.SchedulerCache.AddPod(pod); err != nil { klog.Errorf("scheduler cache AddPod failed: %v", err)

This helps debugging scheduler's behavior, currently we don't have logs that indicate when the scheduler first saw a pod or a node. The log level should be v=3. /sig scheduling /cc
Do you mean add logging for callbacks in ?
yes, at least for add and delete events
Sure, I can also give this pr, before tomorrow
/assign
We actually have one v=2 log line for a Node being to the cache's tree: But it's probably good to keep this one.


Question: // we can recreate it cleanly. if cachedService.Spec.CreateExternalLoadBalancer { glog.Infof("Deleting existing load balancer for service %s that needs an updated load balancer.", namespacedName) <del> if err := s.ensureLBDeleted(cachedService); err != nil { </del> <ins> if err := s.balancer.EnsureTCPLoadBalancerDeleted(s.loadBalancerName(cachedService), s.zone.Region); err != nil { </ins> return err, retryable } }

All of our external load balancer tests failed for the better part of a day due to the GCE project that our tests run in hitting its quota for number of target pools. While the target pools have been deleted, the fact that we hit our quota indicates that there must be a leak somewhere, and we need to plug it.
For reference, the previous issue was
Could the leakage be related to ? The quota exhaustion occurred about 22 hrs after that.
I'd guess that this is due to GetTCPLoadBalancer returning that the load balancer doesn't exist if the forwarding rule is gone, ignoring the target pool, while the service controller considers its deleting done if the cloud provider says that the load balancer doesn't exist. I that logic to the cloud provider because the other implementations of DeleteTCPLoadBalancer weren't/won't be idempotent in the sense of returning ok if the LB doesn't exist. To fix it, we'll either need to make sure all Delete implementations are idempotent and remove the Get call in the controller, or add a method along the lines of IsTCPLoadBalancerDeleted to ask the cloud provider whether all pieces of it have been completely cleaned up. We can't make GetTCPLoadBalancer check that because it's relied on to tell us whether or not the balancer is fully created.
Copying over my commentary from on why the GKE e2e tests are leaking so many forwarding rules and target pools vs the GCE tests:
Is there a way for the services test to wait for the service controller to clean up the external resources before finishing (without just doing a sleep)?
Not without polling GCE directly for whether the resources still exist, although that may be worth it as a workaround. What would be perfect is that clean-up-all-cloud-provider-resources API method that we talked about a couple months ago.
Can we not perhaps use GCE namespaces or projects as a cheap way of getting this here?
I don't believe so. We can't use projects due to the fact that projects can't be programatically created (they're intended to be long-lived). I'm not aware of anything along the lines of a GCE namespace other than networks, which won't work because target pools and forwarding rules unfortunately don't seem to be scoped to networks.


Question: // we can recreate it cleanly. if cachedService.Spec.CreateExternalLoadBalancer { glog.Infof("Deleting existing load balancer for service %s that needs an updated load balancer.", namespacedName) <del> if err := s.ensureLBDeleted(cachedService); err != nil { </del> <ins> if err := s.balancer.EnsureTCPLoadBalancerDeleted(s.loadBalancerName(cachedService), s.zone.Region); err != nil { </ins> return err, retryable } }

The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.


Question: // we can recreate it cleanly. if cachedService.Spec.CreateExternalLoadBalancer { glog.Infof("Deleting existing load balancer for service %s that needs an updated load balancer.", namespacedName) <del> if err := s.ensureLBDeleted(cachedService); err != nil { </del> <ins> if err := s.balancer.EnsureTCPLoadBalancerDeleted(s.loadBalancerName(cachedService), s.zone.Region); err != nil { </ins> return err, retryable } }

I'll periodically check to see whether anything is leaked after this point. I apologize if the deletions affected any running tests - everything looked to only be from GKE and parallel e2e runs.


Question: err = wait.PollImmediate(podRetryPeriod, podRetryTimeout, checkPodListQuantity(f, "type=Testing", 0)) framework.ExpectNoError(err, "found a pod(s)") }) <ins> ginkgo.It("should run through the lifecycle of Pods and PodStatus", func() { podResource := schema.GroupVersionResource{Group: "", Version: "v1", Resource: "pods"} testNamespaceName := f.Namespace.Name testPodName := "pod-test" testPodImage := imageutils.GetE2EImage(imageutils.Agnhost) testPodImage2 := imageutils.GetE2EImage(imageutils.Httpd) testPodLabels := map[string]string{"test-pod-static": "true"} testPodLabelsFlat := "test-pod-static=true" zero := int64(0) w := &cache.ListWatch{ WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { options.LabelSelector = testPodLabelsFlat return f.ClientSet.CoreV1().Pods(testNamespaceName).Watch(context.TODO(), options) }, } podsList, err := f.ClientSet.CoreV1().Pods("").List(context.TODO(), metav1.ListOptions{LabelSelector: testPodLabelsFlat}) framework.ExpectNoError(err, "failed to list Pods") testPod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   testPodName, Labels: testPodLabels, }, Spec: v1.PodSpec{ TerminationGracePeriodSeconds: &zero, Containers: []v1.Container{ { Name:  testPodName, Image: testPodImage, }, }, }, } ginkgo.By("creating a Pod with a static label") _, err = f.ClientSet.CoreV1().Pods(testNamespaceName).Create(context.TODO(), &testPod, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create Pod %v in namespace %v", testPod.ObjectMeta.Name, testNamespaceName) ginkgo.By("watching for Pod to be ready") ctx, cancel := context.WithTimeout(context.Background(), podReadyTimeout) defer cancel() _, err = watchtools.Until(ctx, podsList.ResourceVersion, w, func(event watch.Event) (bool, error) { if pod, ok := event.Object.(*v1.Pod); ok { found := pod.ObjectMeta.Name == testPod.ObjectMeta.Name && pod.ObjectMeta.Namespace == testNamespaceName && pod.Labels["test-pod-static"] == "true" && pod.Status.Phase == v1.PodRunning if !found { framework.Logf("observed Pod %v in namespace %v in phase %v conditions %v", pod.ObjectMeta.Name, pod.ObjectMeta.Namespace, pod.Status.Phase, pod.Status.Conditions) } return found, nil } return false, nil }) framework.ExpectNoError(err, "failed to see Pod %v in namespace %v running", testPod.ObjectMeta.Name, testNamespaceName) ginkgo.By("patching the Pod with a new Label and updated data") podPatch, err := json.Marshal(v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{"test-pod": "patched"}, }, Spec: v1.PodSpec{ TerminationGracePeriodSeconds: &zero, Containers: []v1.Container{{ Name:  testPodName, Image: testPodImage2, }}, }, }) framework.ExpectNoError(err, "failed to marshal JSON patch for Pod") _, err = f.ClientSet.CoreV1().Pods(testNamespaceName).Patch(context.TODO(), testPodName, types.StrategicMergePatchType, []byte(podPatch), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch Pod %s in namespace %s", testPodName, testNamespaceName) ctx, cancel = context.WithTimeout(context.Background(), 30*time.Second) defer cancel() _, err = watchtools.Until(ctx, podsList.ResourceVersion, w, func(event watch.Event) (bool, error) { switch event.Type { case watch.Modified: if pod, ok := event.Object.(*v1.Pod); ok { found := pod.ObjectMeta.Name == pod.Name && pod.Labels["test-pod-static"] == "true" return found, nil } default: framework.Logf("observed event type %v", event.Type) } return false, nil }) framework.ExpectNoError(err, "failed to see %v event", watch.Modified) ginkgo.By("getting the Pod and ensuring that it's patched") pod, err := f.ClientSet.CoreV1().Pods(testNamespaceName).Get(context.TODO(), testPodName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to fetch Pod %s in namespace %s", testPodName, testNamespaceName) framework.ExpectEqual(pod.ObjectMeta.Labels["test-pod"], "patched", "failed to patch Pod - missing label") framework.ExpectEqual(pod.Spec.Containers[0].Image, testPodImage2, "failed to patch Pod - wrong image") ginkgo.By("getting the PodStatus") podStatusUnstructured, err := dc.Resource(podResource).Namespace(testNamespaceName).Get(context.TODO(), testPodName, metav1.GetOptions{}, "status") framework.ExpectNoError(err, "failed to fetch PodStatus of Pod %s in namespace %s", testPodName, testNamespaceName) podStatusBytes, err := json.Marshal(podStatusUnstructured) framework.ExpectNoError(err, "failed to marshal unstructured response") var podStatus v1.Pod err = json.Unmarshal(podStatusBytes, &podStatus) framework.ExpectNoError(err, "failed to unmarshal JSON bytes to a Pod object type") ginkgo.By("replacing the Pod's status Ready condition to False") podStatusUpdated := podStatus podStatusFieldPatchCount := 0 podStatusFieldPatchCountTotal := 2 for pos, cond := range podStatusUpdated.Status.Conditions { if (cond.Type == v1.PodReady && cond.Status == v1.ConditionTrue) || (cond.Type == v1.ContainersReady && cond.Status == v1.ConditionTrue) { podStatusUpdated.Status.Conditions[pos].Status = v1.ConditionFalse podStatusFieldPatchCount++ } } framework.ExpectEqual(podStatusFieldPatchCount, podStatusFieldPatchCountTotal, "failed to patch all relevant Pod conditions") podStatusUpdate, err := f.ClientSet.CoreV1().Pods(testNamespaceName).UpdateStatus(context.TODO(), &podStatusUpdated, metav1.UpdateOptions{}) framework.ExpectNoError(err, "failed to update PodStatus of Pod %s in namespace %s", testPodName, testNamespaceName) ginkgo.By("check the Pod again to ensure its Ready conditions are False") podStatusFieldPatchCount = 0 podStatusFieldPatchCountTotal = 2 for _, cond := range podStatusUpdate.Status.Conditions { if (cond.Type == v1.PodReady && cond.Status == v1.ConditionFalse) || (cond.Type == v1.ContainersReady && cond.Status == v1.ConditionFalse) { podStatusFieldPatchCount++ } } framework.ExpectEqual(podStatusFieldPatchCount, podStatusFieldPatchCountTotal, "failed to update PodStatus - field patch count doesn't match the total") ginkgo.By("deleting the Pod via a Collection with a LabelSelector") err = f.ClientSet.CoreV1().Pods(testNamespaceName).DeleteCollection(context.TODO(), metav1.DeleteOptions{GracePeriodSeconds: &zero}, metav1.ListOptions{LabelSelector: testPodLabelsFlat}) framework.ExpectNoError(err, "failed to delete Pod by collection") ginkgo.By("watching for the Pod to be deleted") ctx, cancel = context.WithTimeout(context.Background(), 1*time.Minute) defer cancel() _, err = watchtools.Until(ctx, podsList.ResourceVersion, w, func(event watch.Event) (bool, error) { switch event.Type { case watch.Deleted: if pod, ok := event.Object.(*v1.Pod); ok { found := pod.ObjectMeta.Name == pod.Name && pod.Labels["test-pod-static"] == "true" return found, nil } default: framework.Logf("observed event type %v", event.Type) } return false, nil }) framework.ExpectNoError(err, "failed to see %v event", watch.Deleted) }) </ins> }) func checkPodListQuantity(f *framework.Framework, label string, quantity int) func() (bool, error) {

[x] Original approval issue: - Reverted due to flakes [x] Two weeks soak end date: 20 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Pod endpoints which are untested. a Pod with a static label the Pod with a new Label and updated data the Pod to ensure it's patched the Pod's status the Pod's status Ready condition to False the Pod's status to check if Ready is False all Pods in all Namespaces find the Pod(1) ensure that the Pod is found and is patched Namespaced Pod(1) via a Collection with a LabelSelector the Pod to check that it's deleted Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/reopen Test PR Merge Promotion PR to be created next
Reopened this issue. <detailsIn response to : /reopen Test PR Merge Promotion PR to be created next Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
reverted by due to flaky test
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/reopen PR on test grid for 2 weeks before promotion to conformance
Reopened this issue. <detailsIn response to : /reopen PR on test grid for 2 weeks before promotion to conformance Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: watch "k8s.io/apimachinery/pkg/watch" clientset "k8s.io/client-go/kubernetes" watchtools "k8s.io/client-go/tools/watch" <ins> "k8s.io/client-go/util/retry" </ins> cloudprovider "k8s.io/cloud-provider" "k8s.io/kubernetes/test/e2e/framework" e2edeployment "k8s.io/kubernetes/test/e2e/framework/deployment"

[x] two weeks soak end date : 18 March 2021 According to this APIsnoop query, there are still some remaining RESOURCENAME endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Note: Community feedback for the e2e test has lead to a number of improvements including extending the test coverage to all outstanding endpoints listed above. As each endpoint is tested a ’watch’ confirms that the result is valid before testing the next endpoint. readCoreV1NamespacedServiceStatus (get /status) patchCoreV1NamespacedServiceStatus (patch /status) replaceCoreV1NamespacedServiceStatus (update /status) patchCoreV1NamespacedService Create a Service with a static label Patch the Service with a new label and updated data Get the Service to ensure it’s patched Upate the Service with a new label and updated data Get the Service to ensure it’s updated Delete Namespaced Service via a Collection with a LabelSelector This query by apisnoop shows that all outstanding endpoints as list at the start of this document have been hit by the e2e test. Note that the results do include other service endpoints that have been addressed in other conformance tests. If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
Can you please review the proposed test and advise if we could go ahead with a PR
/assign
It looks like it's patching metadata and spec but not status?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I've been retesting this mock test and apisnoop does log as been hit. From my testing below it looks like the patchStatus is working as expected or have I missed something?


Question: watch "k8s.io/apimachinery/pkg/watch" clientset "k8s.io/client-go/kubernetes" watchtools "k8s.io/client-go/tools/watch" <ins> "k8s.io/client-go/util/retry" </ins> cloudprovider "k8s.io/cloud-provider" "k8s.io/kubernetes/test/e2e/framework" e2edeployment "k8s.io/kubernetes/test/e2e/framework/deployment"

Using the following code before the patch request Gives this response Then after running the patch ServiceStatus Then running The response shows the patched changes.
During the Conformance Meeting on 12th of January 2021, reviewed approach with Clayton, and we will start on the test PR this week!
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
/triage accepted


Question: utilruntime.Must(events.AddToScheme(scheme)) utilruntime.Must(v1beta1.AddToScheme(scheme)) utilruntime.Must(v1.AddToScheme(scheme)) <del> utilruntime.Must(scheme.SetVersionPriority(v1beta1.SchemeGroupVersion, v1.SchemeGroupVersion)) </del> <ins> utilruntime.Must(scheme.SetVersionPriority(v1.SchemeGroupVersion, v1beta1.SchemeGroupVersion)) </ins> }

What would you like to be : The Event API is being promoted to v1 in and right now v1beta1 has higher priority than v1 in We should bump this in 1.20. Why is this needed: The new Event API is being promoted.
/sig instrumentation
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ginkgo.AfterEach(func() { // print out additional info if tests failed if ginkgo.CurrentGinkgoTestDescription().Failed { <del> // list existing priorities </del> <ins> // List existing PriorityClasses. </ins> priorityList, err := cs.SchedulingV1().PriorityClasses().List(context.TODO(), metav1.ListOptions{}) if err != nil { <del> framework.Logf("Unable to list priorities: %v", err) </del> <ins> framework.Logf("Unable to list PriorityClasses: %v", err) </ins> } else { <del> framework.Logf("List existing priorities:") </del> <ins> framework.Logf("List existing PriorityClasses:") </ins> for _, p := range priorityList.Items { framework.Logf("%v/%v created at %v", p.Name, p.Value, p.CreationTimestamp) }

The following PriorityClass endpoints was Promoted it 1.14 without Conformance tests, and remain untested: replace-, read-, patch-, list-, delete-collection- SchedulingV1PriorityClass According to this APIsnoop query, there are still some remaining PriorityClass endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints ! createSchedulingV1PriorityClass deleteSchedulingV1PriorityClass Promoted to Conformance 1.19 with /sig testing /sig architecture /area conformance
Is the ultimiate goal to use conformance tests to explicitly issue all CRUD requests against PriorityClass? And, is equivalent to "update PriorityClass"?
all APIs must have Conformance tests unless they are not eligible for Conformance due to the There is other APIGroups that have both Replace and Update as endpoints. That is the prerogative of the SIG to decide.
can you please point us to similar conformance tests used for core objects such as pods and nodes?
This is a life cycle test for Create, List, Update, Patch, DeleteCollection More can be found in the test/e2e folder. Also look for examples on the Conformance at previously Promoted Conformance test Reach out if we can help with more information on our approach to Conformance testing.
I remembered about this PR. Last week a few lines of code was to cover the last of the "Lease" endpoint group for conformance. The one file contain the Conformance tests for the complete life cycle of Lease endpoints.
/assign I will work on it.
/reopen Will monitor in test grid to move to Conformance after 2 weeks soak.
Reopened this issue. <detailsIn response to : /reopen Will monitor in test grid to move to Conformance after 2 weeks soak. </details
/triage accepted


Question: "flag" "fmt" "log" <ins> "math/rand" </ins> "net/http" "os" "time"

This was the root cause of rancher/rancher. In , when is called, it is given the oldVersion of an object and the newVersion of the object. It then uses the indexers to calculate the old values (from the old version) and the new updated values. In cases where are size of 1, are of size greater than 1, and contain the sole value from , the key will be deleted for the old value will be deleted and won't be re-. This is due to the two for loops at the end of the method which are used for optimization. When the first for loop is processed, . Because of this, the key is deleted from the index for this value. In the second for loop, the , and for at least one of the values . The continue statement causes the key to not be to the index for this value (and since the value was deleted in the prior for loop, it no longer exists in the index for this value), despite the fact that the indicated that this is the desired state. Since specified that this value was still valid for this object, the key should not be deleted from the index for this value, or it should be re- when it is re-discovered in . Either way, the end state of the index should include this key in the set for the specified value. I've created a which demonstrates the issue (branch is ). To use it (assuming that you have a running kubernetes cluster to use with): the controller: . the foo crd: the controller (assumes that you have $KUBECONFIG set to the file path of your kubeconfig file). an example foo with only one label: . Observe, in the logs of the running sample-controller, that the indexer was able to locate this foo using the indexer. the foo to include a second label: . Observe, in the logs of the running sample-controller, that the indexer was not able to locate the foo using the first key any longer, despite the fact that the label was still on the object. The sample-controller fork adds an indexer for foo objects which uses the indexerFunction of . This function looks for labels that have a key with a prefix of and uses their values as index keys for the foo objects. For example, if I have a with a label of , the indexer will allow you to retrieve using .


Question: "flag" "fmt" "log" <ins> "math/rand" </ins> "net/http" "os" "time"

The controller's method has been altered to just ensure that the foos can be retrieved using the indexer for all valid labels. If the controller can't retrieve foo by using the value from a valid label, then it returns an error (it tries for all valid labels). This was originally discovered in a fork of client-go that we maintain. After we reverted the changes introduced by , our issues stopped occurring. N/A - This was introduced in and appears to be in all versions of the client v0.23+. N/A - issue is with client-go No response No response No response No response
/sig api-machinery
I think that you have background in this issue/area of the code. Would you mind taking a look at this issue and the PR that I have up to fix it? Please let me know if I've misunderstood anything.
/assign Would you mind to take a look? Thank you /triage accepted
Thanks for reporting the issue. will address this issue
Reopening - since we need to cherrypick back to 1.23
Cherrypick is merged - closing.


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package main import ( "io" "io/ioutil" "log" "os" "os/signal" "path/filepath" "strings" "github.com/pkg/errors" ) func main() { if strings.Contains(os.Args[0], "run_e2e.sh") || strings.Contains(os.Args[0], "gorunner") { log.Print("warn: calling test with e2e.test is deprecated and will be removed in 1.25, please rely on container manifest to invoke executable") } env := envWithDefaults(map[string]string{ resultsDirEnvKey: defaultResultsDir, skipEnvKey:       defaultSkip, focusEnvKey:      defaultFocus, providerEnvKey:   defaultProvider, parallelEnvKey:   defaultParallel, ginkgoEnvKey:     defaultGinkgoBinary, testBinEnvKey:    defaultTestBinary, }) if err := configureAndRunWithEnv(env); err != nil { log.Fatal(err) } } // configureAndRunWithEnv uses the given environment to configure and then start the test run. // It will handle TERM signals gracefully and kill the test process and will // save the logs/results to the location specified via the RESULTS_DIR environment // variable. func configureAndRunWithEnv(env Getenver) error { // Ensure we save results regardless of other errors. This helps any // consumer who may be polling for the results. resultsDir := env.Getenv(resultsDirEnvKey) defer saveResults(resultsDir) // Print the output to stdout and a logfile which will be returned // as part of the results tarball. logFilePath := filepath.Join(resultsDir, logFileName) logFile, err := os.Create(logFilePath) if err != nil { return errors.Wrapf(err, "failed to create log file %v", logFilePath) } mw := io.MultiWriter(os.Stdout, logFile) cmd := getCmd(env, mw) log.Printf("Running command:n%vn", cmdInfo(cmd)) err = cmd.Start() if err != nil { return errors.Wrap(err, "starting command") } // Handle signals and shutdown process gracefully. go setupSigHandler(cmd.Process.Pid) return errors.Wrap(cmd.Wait(), "running command") } // setupSigHandler will kill the process identified by the given PID if it // gets a TERM signal. func setupSigHandler(pid int) { c := make(chan os.Signal, 1) signal.Notify(c, os.Interrupt) // Block until a signal is received. log.Println("Now listening for interrupts") s := <-c log.Printf("Got signal: %v. Shutting down test process (PID: %v)n", s, pid) p, err := os.FindProcess(pid) if err != nil { log.Printf("Could not find process %v to shut down.n", pid) return } if err := p.Signal(s); err != nil { log.Printf("Failed to signal test process to terminate: %vn", err) return } log.Printf("Signalled process %v to terminate successfully.n", pid) } // saveResults will tar the results directory and write the resulting tarball path // into the donefile. func saveResults(resultsDir string) error { log.Printf("Saving results at %vn", resultsDir) err := tarDir(resultsDir, filepath.Join(resultsDir, resultsTarballName)) if err != nil { return errors.Wrapf(err, "tar directory %v", resultsDir) } doneFile := filepath.Join(resultsDir, doneFileName) resultsTarball := filepath.Join(resultsDir, resultsTarballName) resultsTarball, err = filepath.Abs(resultsTarball) if err != nil { return errors.Wrapf(err, "failed to find absolute path for %v", resultsTarball) } return errors.Wrap( ioutil.WriteFile(doneFile, []byte(resultsTarball), os.FileMode(0777)), "writing donefile", ) } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be modified: image to be based on anything other than Docker Hub sourced Why is this needed: With Docker Hub rate-limit policy in-place, we probably should consider removing this dependency to stay in sync with other base images in Kubernetes project. Folks building their own binaries (i.e. with ) might run into this as well. Notes: Prior discussion happened on An idea from is to use the should probably be renamed to avoid confusion with k/release (h/t Currently, is set to , so migrating to may not be a drop-in move However, , so maybe we don't have to start from scratch /sig testing /area conformance /sig release (cc'ing because builds conformance image, though my quick search seem to show there isn't any build pipelines referencing )
/assign
do we still want to do this? I read somewhere that the images built in Google Cloud are being mirrored from Docker Hub, right? This would mean that the API rate limiting will not hit us any more.
Womp yeah I think DockerHub rate limit is not really the big motivator at this point. However, I think I still want to consider using distroless / go-runner to build and produce conformance image to reduce the noise from CVE scanning done in downstream places. What do you think?
Yeah I think this seems a valuable addition. Are you planning to work on this issue? :)
I will, though probably won't start until next week. If you have ideas and would like to start early feel free to go for it! Otherwise, I'd be happy to work on and deliver this.
Thank you Wilson! Feel free to ping again if you need any assistance. /assign
/triage accepted
one thing I need a little help with is to find a reference for using go-runner image that's used everywhere -- the one that I've found is inside conformance directory which seems to be different from what everyone else is using
We maintain a go-runner in k/release:
oops, I forgot to get back on this last week -- I meant to say that I need a reference for image which I can use as , so something like on Dockerfile. Is that a possible thing to do?


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package main import ( "io" "io/ioutil" "log" "os" "os/signal" "path/filepath" "strings" "github.com/pkg/errors" ) func main() { if strings.Contains(os.Args[0], "run_e2e.sh") || strings.Contains(os.Args[0], "gorunner") { log.Print("warn: calling test with e2e.test is deprecated and will be removed in 1.25, please rely on container manifest to invoke executable") } env := envWithDefaults(map[string]string{ resultsDirEnvKey: defaultResultsDir, skipEnvKey:       defaultSkip, focusEnvKey:      defaultFocus, providerEnvKey:   defaultProvider, parallelEnvKey:   defaultParallel, ginkgoEnvKey:     defaultGinkgoBinary, testBinEnvKey:    defaultTestBinary, }) if err := configureAndRunWithEnv(env); err != nil { log.Fatal(err) } } // configureAndRunWithEnv uses the given environment to configure and then start the test run. // It will handle TERM signals gracefully and kill the test process and will // save the logs/results to the location specified via the RESULTS_DIR environment // variable. func configureAndRunWithEnv(env Getenver) error { // Ensure we save results regardless of other errors. This helps any // consumer who may be polling for the results. resultsDir := env.Getenv(resultsDirEnvKey) defer saveResults(resultsDir) // Print the output to stdout and a logfile which will be returned // as part of the results tarball. logFilePath := filepath.Join(resultsDir, logFileName) logFile, err := os.Create(logFilePath) if err != nil { return errors.Wrapf(err, "failed to create log file %v", logFilePath) } mw := io.MultiWriter(os.Stdout, logFile) cmd := getCmd(env, mw) log.Printf("Running command:n%vn", cmdInfo(cmd)) err = cmd.Start() if err != nil { return errors.Wrap(err, "starting command") } // Handle signals and shutdown process gracefully. go setupSigHandler(cmd.Process.Pid) return errors.Wrap(cmd.Wait(), "running command") } // setupSigHandler will kill the process identified by the given PID if it // gets a TERM signal. func setupSigHandler(pid int) { c := make(chan os.Signal, 1) signal.Notify(c, os.Interrupt) // Block until a signal is received. log.Println("Now listening for interrupts") s := <-c log.Printf("Got signal: %v. Shutting down test process (PID: %v)n", s, pid) p, err := os.FindProcess(pid) if err != nil { log.Printf("Could not find process %v to shut down.n", pid) return } if err := p.Signal(s); err != nil { log.Printf("Failed to signal test process to terminate: %vn", err) return } log.Printf("Signalled process %v to terminate successfully.n", pid) } // saveResults will tar the results directory and write the resulting tarball path // into the donefile. func saveResults(resultsDir string) error { log.Printf("Saving results at %vn", resultsDir) err := tarDir(resultsDir, filepath.Join(resultsDir, resultsTarballName)) if err != nil { return errors.Wrapf(err, "tar directory %v", resultsDir) } doneFile := filepath.Join(resultsDir, doneFileName) resultsTarball := filepath.Join(resultsDir, resultsTarballName) resultsTarball, err = filepath.Abs(resultsTarball) if err != nil { return errors.Wrapf(err, "failed to find absolute path for %v", resultsTarball) } return errors.Wrap( ioutil.WriteFile(doneFile, []byte(resultsTarball), os.FileMode(0777)), "writing donefile", ) } </ins>

after searching around, I found this -- going to try and use and see how it works :smile:
Started a PR -- plenty of things I'd like to be discussed but we should probably have the discussions there
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale there is work in progress


Question: By(fmt.Sprintf("Restart count of pod %s in namespace %s is now %d", podDescr.Name, ns, restartCount)) if restartCount > initialRestartCount { By(fmt.Sprintf("Restart count of pod %s in namespace %s increased from %d to %d during the test", podDescr.Name, ns, initialRestartCount, restartCount)) <del> pass = true </del> <ins> restarts = true </ins> break } } <del> if !pass { Fail(fmt.Sprintf("Did not see the restart count of pod %s in namespace %s increase from %d during the test", podDescr.Name, ns, initialRestartCount)) </del> <ins> if restarts != expectRestart { Fail(fmt.Sprintf("pod %s in namespace %s - expected restarts: %v, found restarts: %v", podDescr.Name, ns, expectRestart, restarts)) </ins> } }

As far as I can tell, container liveness probes of type exec flat out don't work against clusters built at head. I had the kubelet always log the output of the exec'ed command, and it's always empty no matter what I exec, causing the probe to fail since it requires the output to be equal to the string "ok". If you want to test it, try adding a liveness probe of this form to any container: I also tried these lovely forms, to more closely match the liveness example under examples/: On the plus side, this may be an opportunistic time to switch over to
The reason that the e2e test for exec liveness probes hasn't caught this is that it only tests that containers are restarted when they fail liveness probes. It doesn't test that containers aren't restarted when their liveness checks should be succeeding. The e2e test's exec'ed commands also only ever produced empty output, rather than the expected "fail" output.
Is it possible to include this in a new pre-release, since it fixes a bug in health checking? I see it merged in master and I'm wondering what the policy is of releasing new code in a separate (0.16.*) release.


Question: return device, refCount, nil } <del> // IsNotMountPoint determines if a directory is a mountpoint. // It should return ErrNotExist when the directory does not exist. // This method uses the List() of all mountpoints // It is more extensive than IsLikelyNotMountPoint // and it detects bind mounts in linux func IsNotMountPoint(mounter Interface, file string) (bool, error) { </del> <ins> // isNotMountPoint implements Mounter.IsNotMountPoint and is shared by mounter // implementations. func isNotMountPoint(mounter Interface, file string) (bool, error) { </ins> // IsLikelyNotMountPoint provides a quick check // to determine whether file IS A mountpoint notMnt, notMntErr := mounter.IsLikelyNotMountPoint(file)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There's , and . The former is used by local and nfs volume plugins, while the latter is used by , which is also used by local and nfs volume plugins. What you expected to happen: Can we consolidate these? This is really confusing and could easily be the source of bugs. <!-- DO NOT EDIT BELOW THIS LINE --/kind cleanup
cc
/assign I'd like to clean up code.


Question: valueFrom: fieldRef: fieldPath: metadata.name <del> command: - /bin/sh - -c - /kubemark --morph=proxy --name=$(NODE_NAME) {{hollow_proxy_params}} --kubeconfig=/kubeconfig/kubeproxy.kubeconfig $(CONTENT_TYPE) --alsologtostderr 1>>/var/log/kubeproxy-$(NODE_NAME).log 2>&1 </del> <ins> command: [ "/kubemark", "--morph=proxy", "--name=$(NODE_NAME)", "--kubeconfig=/kubeconfig/kubeproxy.kubeconfig", "$(CONTENT_TYPE)", "--log-file=/var/log/kubeproxy-$(NODE_NAME).log", "--logtostderr=false", {{hollow_proxy_params}} ] </ins> volumeMounts: - name: kubeconfig-volume mountPath: /kubeconfig

There are CVEs happening in different images - we already migrated Kubmeark from busybox in , but we should go one step further and migrate the base image to distroless. There are some things that would need to be changed mentioned here: /sig scalability
Can I work on this? /assign
/remove-help
- sure that would be great.
Hi based on the discussion in , the kubemark image has dependency on and maybe more other utilities. So it seems impossible to migrate the base image to distroless? Or do we need something like debian-hyperkube-base?
The mount one we may get rid of - basically we should be able to switch to FakeMounter:


Question: // Enable usage of Provision of PVCs from snapshots in other namespaces CrossNamespaceVolumeDataSource featuregate.Feature = "CrossNamespaceVolumeDataSource" <ins> // owner: @thockin // deprecated: v1.29 // // Enables Service.status.ingress.loadBanace to be set on // services of types other than LoadBalancer. AllowServiceLBStatusOnNonLB featuregate.Feature = "AllowServiceLBStatusOnNonLB" </ins> // owner: @bswartz // alpha: v1.18 // beta: v1.24

xref I think this may be one of the few cases where we can actually change the behavior in a "breaking" way. User creates service with , is empty Controller or user updates the status subresource, sets (not wiped or validated, should be) User updates in any way, is wiped It's clear that the is not meant to be set when is not "LoadBalancer". The question is: 1) Fail validation? 2) Silently discard the value? 3) Warn and then discard the value? 4) Just warn and leave it. Because we allowed it in the past, but it is so volatile, I lean towards 3. and - the people most familair with this type. Edit: PrepareForUpdate() runs before warnings. So: (1) is easy (2) is easy (4) is slightly less easy than (2) (3) is slightly less easy than (4)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
OMG change it to clusterIP as expected Status is wiped update status as ClusterIP ... and if I edit the service back to loadbalancer it keeps the status??
/kind bug
/retitle Service can be set when !=
We talked about this one at the SIG Network meeting today. suggested we use the feature gate for deprecation option to phase the current (problematic) functionality out.
/triage accept
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: podClient.WaitForSuccess(podName, framework.PodStartTimeout) return podName } <del> It("should run the container as unprivileged when false [LinuxOnly] [NodeConformance]", func() { // This test is marked LinuxOnly since it runs a Linux-specific command, and Windows does not support Windows escalation. </del> <ins> /* Release : v1.15 Testname: Security Context, privileged=false. Description: Create a container to run in unprivileged mode by setting pod's SecurityContext Privileged option as false. Pod MUST be in Succeeded phase. [LinuxOnly]: This test is marked as LinuxOnly since it runs a Linux-specific command. */ framework.ConformanceIt("should run the container as unprivileged when false [LinuxOnly] [NodeConformance]", func() { </ins> podName := createAndWaitUserPod(false) logs, err := framework.GetPodLogs(f.ClientSet, f.Namespace.Name, podName, podName) if err != nil {

/kind cleanup /area conformance /area test /sig node /sig testing /sig architecture /assign /assign We have a number of tests that only have the tag that may make sense as tests, we should identify whether these tests cases meet (or could meet) the . We should open PR's to promote these if so. I would advise starting with the test/common cases
GitHub didn't allow me to assign the following users: mgdevstack, mkurylec. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /kind cleanup /area conformance /area test /sig node /sig testing /sig architecture /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The tests in test/e2enode don't actually get run as part of Conformance as near as I can tell. If we want to run them as part of conformance, they need to be migrated into test/e2e/common. Some of the tests in e2enode may not be able to be ported over because they assume a level access we don't have in a conformance environment.
regarding this testcase: [sig-storage] HostPath should support r/w [NodeConformance]: It uses hostPath and privileged=true, to read/write from /tmp folder. Which would you suggest is the best orientation to overcome the privilege mode setting and allow its promotion?
/assign
Regarding , the suite runs the test only if the kubelet configuration was required to have the QOS (cgroups-per-qos) enabled. This configuration for the kubelet is selected at the time the cluster is set up. Looking for where this might be configured found: where is set to true. For other settings it is also configured to true. where is set to true. BUT also found where it is mentioned that the flag is no longer available. Questions: would it be ok to promote this test to be conformance, even it can be skipped based on configuration of the cluster? And not supported in the containerized kubelet?
I would consider cgroups to be an implementation detail that Conformance shouldn't be verifying
It uses hostPath and privileged=true, to read/write from /tmp folder.


Question: podClient.WaitForSuccess(podName, framework.PodStartTimeout) return podName } <del> It("should run the container as unprivileged when false [LinuxOnly] [NodeConformance]", func() { // This test is marked LinuxOnly since it runs a Linux-specific command, and Windows does not support Windows escalation. </del> <ins> /* Release : v1.15 Testname: Security Context, privileged=false. Description: Create a container to run in unprivileged mode by setting pod's SecurityContext Privileged option as false. Pod MUST be in Succeeded phase. [LinuxOnly]: This test is marked as LinuxOnly since it runs a Linux-specific command. */ framework.ConformanceIt("should run the container as unprivileged when false [LinuxOnly] [NodeConformance]", func() { </ins> podName := createAndWaitUserPod(false) logs, err := framework.GetPodLogs(f.ClientSet, f.Namespace.Name, podName, podName) if err != nil {

Which would you suggest is the best orientation to overcome the privilege mode setting and allow its promotion? Michelle Au If my understanding is correct, on some platforms, writing to /tmp requires privileged. I don't see a way to to remove that requirement.
If my understanding is correct, on some platforms, writing to /tmp requires privileged. I don't see a way to to remove that requirement. Yeah writing to host doesn't seem like a thing we can expect of conformance tests. I wonder if it makes sense to tag this test with something to denote the requirement, ie: I see some tests in test/e2e_node use
The hostpath tests in already have a comment explaining this. Maybe the comment should be updated to be more clear that that is the reason why we can't add it to conformance.
That's /tmp on the host or in the container filesystem?
/tmp on the host
ping what else remains to be done before we close this out? I am guessing that we want to close , is there anything beyond that?
Once tests under (issue- ) and are promoted we are good to close this issue. Short update for reference- None of [NodeConformance] tests from "test/e2e_node" seems eligible for [Conformance] promotion at this moment due to node level dependencies. Remaining [NodeConformance] tests from "test/e2e" has following status - Waiting for to be closed and then below tests would be promoted to Conformance Needs to be Promoted Can not be promoted to Conformance as //Slow by design Can not be promoted due to Node level dependency
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen
/assign


Question: // EnvVarSource represents a source for the value of an EnvVar. type EnvVarSource struct { <del> // Selects a field of the pod: supports metadata.name, metadata.namespace, metadata.labels, metadata.annotations, </del> <ins> // Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['<KEY>']`, `metadata.annotations['<KEY>']`, </ins> // spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. // +optional FieldRef *ObjectFieldSelector `json:"fieldRef,omitempty" protobuf:"bytes,1,opt,name=fieldRef"`

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I tried to create a pod with env variables referencing to annotations: What you expected to happen: Should Proceed. As in It stated is a valid option How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.2 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig docs
The API docs need to be updated to indicate the correct supported values. See for correct values
You are a lifesaver. Thank you!
/assign I can try to update the api documentation


Question: out.Filter = (*config.PluginSet)(unsafe.Pointer(in.Filter)) out.PostFilter = (*config.PluginSet)(unsafe.Pointer(in.PostFilter)) out.Score = (*config.PluginSet)(unsafe.Pointer(in.Score)) <del> out.NormalizeScore = (*config.PluginSet)(unsafe.Pointer(in.NormalizeScore)) </del> out.Reserve = (*config.PluginSet)(unsafe.Pointer(in.Reserve)) out.Permit = (*config.PluginSet)(unsafe.Pointer(in.Permit)) out.PreBind = (*config.PluginSet)(unsafe.Pointer(in.PreBind))

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Consolidate ) into Score plugin set in config API. Why is this needed: NormalizeScore extension point is implemented in . We decided to treat NormalizeScore as an "optional" step of Score plugin, see . However, we deferred the removal of due to API review delays. Now that PR was merged. We can start the actual removal from the API and start the API review process. /sig scheduling
/assign


Question: <ins> # Hunting flaky tests in Kubernetes Sometimes unit tests are flaky.  This means that due to (usually) race conditions, they will occasionally fail, even though most of the time they pass. We have a goal of 99.9% flake free tests.  This means that there is only one flake in one thousand runs of a test. Running a test 1000 times on your own machine can be tedious and time consuming.  Fortunately, there is a better way to achieve this using Kubernetes. _Note: these instructions are mildly hacky for now, as we get run once semantics and logging they will get better_ There is a testing image ```brendanburns/flake``` up on the docker hub.  We will use this image to test our fix. Create a replication controller with the following config: ```yaml id: flakeController desiredState: replicas: 24 replicaSelector: name: flake podTemplate: desiredState: manifest: version: v1beta1 id: "" volumes: [] containers: - name: flake image: brendanburns/flake env: - name: TEST_PACKAGE value: pkg/tools - name: REPO_SPEC value: https://github.com/GoogleCloudPlatform/kubernetes restartpolicy: {} labels: name: flake labels: name: flake ``` ```./cluster/kubecfg.sh -c controller.yaml create replicaControllers``` This will spin up 100 instances of the test.  They will run to completion, then exit, the kubelet will restart them, eventually you will have sufficient runs for your purposes, and you can stop the replication controller: ```sh ./cluster/kubecfg.sh stop flakeController ./cluster/kubecfg.sh rm flakeController ``` Now examine the machines with ```docker ps -a``` and look for tasks that exited with non-zero exit codes (ignore those that exited -1, since that's what happens when you stop the replica controller) Happy flake hunting! </ins>

Right now, we have a mixture of user documentation and developer documentation in the top-level directory. We should make it easier to find a just user documentation. Step 1 would be to move developer documentation into a subdir like docs/development or something like that. User docs would stay in the top-level docs dir.
move to


Question: <ins> Resource Quota ======================================== This example demonstrates how resource quota and limits can be applied to a Kubernetes namespace. This example assumes you have a functional Kubernetes setup. Step 1: Create a namespace ----------------------------------------- This example will work in a custom namespace to demonstrate the concepts involved. Let's create a new namespace called quota-example: ```shell $ cluster/kubectl.sh create -f namespace.yaml $ cluster/kubectl.sh get namespaces NAME            LABELS             STATUS default         <none>             Active quota-example   <none>             Active ``` Step 2: Apply a quota to the namespace ----------------------------------------- By default, a pod will run with unbounded CPU and memory limits.  This means that any pod in the system will be able to consume as much CPU and memory on the node that executes the pod. Users may want to restrict how much of the cluster resources a given namespace may consume across all of its pods in order to manage cluster usage.  To do this, a user applies a quota to a namespace.  A quota lets the user set hard limits on the total amount of node resources (cpu, memory) and API resources (pods, services, etc.) that a namespace may consume. Let's create a simple quota in our namespace: ```shell $ cluster/kubectl.sh create -f quota.yaml --namespace=quota-example ``` Once your quota is applied to a namespace, the system will restrict any creation of content in the namespace until the quota usage has been calculated.  This should happen quickly. You can describe your current quota usage to see what resources are being consumed in your namespace. ``` $ cluster/kubectl.sh describe quota quota --namespace=quota-example Name:     quota Resource    Used  Hard --------    ----  ---- cpu     0m  20 memory      0m  1Gi persistentvolumeclaims  0m  10 pods      0m  10 replicationcontrollers  0m  20 resourcequotas    1 1 secrets     1 10 services    0m  5 ``` Step 3: Applying default resource limits ----------------------------------------- Pod authors rarely specify resource limits for their pods. Since we applied a quota to our project, let's see what happens when an end-user creates a pod that has unbounded cpu and memory by creating an nginx container. To demonstrate, lets create a replication controller that runs nginx: ```shell $ cluster/kubectl.sh run nginx --image=nginx --replicas=1 --namespace=quota-example CONTROLLER   CONTAINER(S)   IMAGE(S)   SELECTOR    REPLICAS nginx        nginx          nginx      run=nginx   1 ``` Now let's look at the pods that were created. ```shell $ cluster/kubectl.sh get pods --namespace=quota-example POD       IP        CONTAINER(S)   IMAGE(S)   HOST      LABELS    STATUS    CREATED   MESSAGE ``` What happened?  I have no pods!  Let's describe the replication controller to get a view of what is happening. ```shell cluster/kubectl.sh describe rc nginx --namespace=quota-example Name:   nginx Image(s): nginx Selector: run=nginx Labels:   run=nginx Replicas: 0 current / 1 desired Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed Events: FirstSeen       LastSeen      Count From        SubobjectPath Reason    Message Mon, 01 Jun 2015 22:49:31 -0400 Mon, 01 Jun 2015 22:52:22 -0400 7 {replication-controller }     failedCreate  Error creating: Pod "nginx-" is forbidden: Limited to 1Gi memory, but pod has no specified memory limit ``` The Kubernetes API server is rejecting the replication controllers requests to create a pod because our pods do not specify any memory usage. So let's set some default limits for the amount of cpu and memory a pod can consume: ```shell $ cluster/kubectl.sh create -f limits.yaml --namespace=quota-example limitranges/limits $ cluster/kubectl.sh describe limits limits --namespace=quota-example Name:   limits Type    Resource  Min Max Default ----    --------  --- --- --- Container cpu   - - 100m Container memory    - - 512Mi ``` Now any time a pod is created in this namespace, if it has not specified any resource limits, the default amount of cpu and memory per container will be applied as part of admission control. Now that we have applied default limits for our namespace, our replication controller should be able to create its pods. ```shell $ cluster/kubectl.sh get pods --namespace=quota-example POD           IP         CONTAINER(S)   IMAGE(S)   HOST                    LABELS      STATUS    CREATED     MESSAGE nginx-t40zm   10.0.0.2                             10.245.1.3/10.245.1.3   run=nginx   Running   2 minutes nginx          nginx                                          Running   2 minutes ``` And if we print out our quota usage in the namespace: ```shell cluster/kubectl.sh describe quota quota --namespace=quota-example Name:     quota Resource    Used    Hard --------    ----    ---- cpu     100m    20 memory      536870912 1Gi persistentvolumeclaims  0m    10 pods      1   10 replicationcontrollers  1   20 resourcequotas    1   1 secrets     1   10 services    0m    5 ``` You can now see the pod that was created is consuming explicit amounts of resources, and the usage is being tracked by the Kubernetes system properly. Summary ---------------------------- Actions that consume node resources for cpu and memory can be subject to hard quota limits defined by the namespace quota. Any action that consumes those resources can be tweaked, or can pick up namespace level defaults to meet your end goal. [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/examples/resourcequota/README.md?pixel)]() </ins>

Resource quota example is missing readme. Was it once there and was accidentally deleted?
I wrote something up in Origin repo I need to repurpose for Kubernetes. Sent from my iPhone


Question: package validation import ( <ins> "fmt" "net" </ins> "k8s.io/apimachinery/pkg/util/validation/field" "k8s.io/kubernetes/federation/apis/federation" "k8s.io/kubernetes/pkg/api/validation"

Right now it is possible to add a cluster for which it won't be possible to create a client. For example client-related code does: while validation checks only: cc:
switching this to 1.6 as it's too late for 1.5. ok? (please switch it right back if you disagree)
Not a release blocker for 1.7.
are you interested in fixing this?
of course, will do that.
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.8 milestone for more than 7 days: kind: Must specify exactly one of [, , ]. priority: Must specify exactly one of [, , ]. Removing it from the milestone. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: # Use skopeo to find these values: https://github.com/containers/skopeo # # Example <del> # Manifest: skopeo inspect docker://gcr.io/k8s-staging-build-image/debian-base:buster-v1.3.0 # Arches: skopeo inspect --raw docker://gcr.io/k8s-staging-build-image/debian-base:buster-v1.3.0 </del> <ins> # Manifest: skopeo inspect docker://gcr.io/k8s-staging-build-image/debian-base:buster-v1.4.0 # Arches: skopeo inspect --raw docker://gcr.io/k8s-staging-build-image/debian-base:buster-v1.4.0 </ins> _DEBIAN_BASE_DIGEST = { <del> "manifest": "sha256:d66137c7c362d1026dca670d1ff4c25e5b0770e8ace87ac3d008d52e4b0db338", "amd64": "sha256:a5ab028d9a730b78af9abb15b5db9b2e6f82448ab269d6f3a07d1834c571ccc6", "arm": "sha256:94e611363760607366ca1fed9375105b6c5fc922ab1249869b708690ca13733c", "arm64": "sha256:83512c52d44587271cd0f355c0a9a7e6c2412ddc66b8a8eb98f994277297a72f", "ppc64le": "sha256:9c8284b2797b114ebe8f3f1b2b5817a9c7f07f3f82513c49a30e6191a1acc1fc", "s390x": "sha256:d617637dd4df0bc1cfa524fae3b4892cfe57f7fec9402ad8dfa28e38e82ec688", </del> <ins> "manifest": "sha256:36652ef8e4dd6715de02e9b68e5c122ed8ee06c75f83f5c574b97301e794c3fb", "amd64": "sha256:afff10fcd513483e492807f8d934bdf0be4a237997f55e0f1f8e34c04a6cb213", "arm": "sha256:27e6e66ea3c4c4ca6dbfc8c949f0c4c870f038f4500fd267c242422a244f233c", "arm64": "sha256:4333a5edc9ce6d6660c76104749c2e50e6158e57c8e5956f732991bb032a8ce1", "ppc64le": "sha256:01a0ba2645883ea8d985460c2913070a90a098056cc6d188122942678923ddb7", "s390x": "sha256:610526b047d4b528d9e14b4f15347aa4e37af0c47e1307a2f7aebf8745c8a323", </ins> } # Use skopeo to find these values: https://github.com/containers/skopeo # # Example <del> # Manifest: skopeo inspect docker://gcr.io/k8s-staging-build-image/debian-iptables:buster-v1.4.0 # Arches: skopeo inspect --raw docker://gcr.io/k8s-staging-build-image/debian-iptables:buster-v1.4.0 </del> <ins> # Manifest: skopeo inspect docker://gcr.io/k8s-staging-build-image/debian-iptables:buster-v1.5.0 # Arches: skopeo inspect --raw docker://gcr.io/k8s-staging-build-image/debian-iptables:buster-v1.5.0 </ins> _DEBIAN_IPTABLES_DIGEST = { <del> "manifest": "sha256:87f97cf2b62eb107871ee810f204ccde41affb70b29883aa898e93df85dea0f0", "amd64": "sha256:da837f39cf3af78adb796c0caa9733449ae99e51cf624590c328e4c9951ace7a", "arm": "sha256:bb6677337a4dbc3e578a3e87642d99be740dea391dc5e8987f04211c5e23abcd", "arm64": "sha256:6ad4717d69db2cc47bc2efc91cebb96ba736be1de49e62e0deffdbaf0fa2318c", "ppc64le": "sha256:168ccfeb861239536826a26da24ab5f68bb5349d7439424b7008b01e8f6534fc", "s390x": "sha256:5a88d4f4c29bac5b5c93195059b928f7346be11d0f0f7f6da0e14c0bfdbd1362", </del> <ins> "manifest": "sha256:abe8cef9e116f2d5ec1175c386e33841ff3386779138b425af876384b0fd7ccb", "amd64": "sha256:b4b8b1e0d4617011dd03f20b804cc2e50bf48bafc36b1c8c7bd23fd44bfd641e", "arm": "sha256:09f79b3a00268705a8f8462f1528fed536e204905359f21e9965f08dd306c60a", "arm64": "sha256:b4fa11965f34a9f668c424b401c0af22e88f600d22c899699bdb0bd1e6953ad6", "ppc64le": "sha256:0ea0be4dec281b506f6ceef4cb3594cabea8d80e2dc0d93c7eb09d46259dd837", "s390x": "sha256:50ef25fba428b6002ef0a9dea7ceae5045430dc1035d50498a478eefccba17f5", </ins> } # Use skopeo to find these values: https://github.com/containers/skopeo

The based container images for are shipped with the wrong architecture within the manifest if not amd64: Refers to We updated the images a few minutes ago to buster-v1.3.0, which are now ready to be used as base for the multi-architecture images. If the images contain the right architectures, then it may magically happen that the kube-proxy images are fixed, too. This is because we already use the build syntax for it: /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release


Question: defaultVersion := cmdutil.OutputVersion(cmd, clientConfig.Version) results := editResults{} for { <del> obj, err := resource.AsVersionedObject(infos, false, defaultVersion) </del> <ins> objs, err := resource.AsVersionedObjects(infos, defaultVersion) </ins> if err != nil { return preservedFile(err, results.file, out) } <ins> // if input object is a list, traverse and edit each item one at a time for _, obj := range objs { // TODO: add an annotating YAML printer that can print inline comments on each field, //   including descriptions or validation errors // generate the file to edit buf := &bytes.Buffer{} if err := results.header.writeTo(buf); err != nil { return preservedFile(err, results.file, out) } if err := printer.PrintObj(obj, buf); err != nil { return preservedFile(err, results.file, out) } original := buf.Bytes() </ins> <del> // TODO: add an annotating YAML printer that can print inline comments on each field, //   including descriptions or validation errors // generate the file to edit buf := &bytes.Buffer{} if err := results.header.writeTo(buf); err != nil { return preservedFile(err, results.file, out) } if err := printer.PrintObj(obj, buf); err != nil { return preservedFile(err, results.file, out) } original := buf.Bytes() // launch the editor edit := editor.NewDefaultEditor() edited, file, err := edit.LaunchTempFile("kubectl-edit-", ext, buf) if err != nil { return preservedFile(err, results.file, out) } </del> <ins> // launch the editor edit := editor.NewDefaultEditor() edited, file, err := edit.LaunchTempFile("kubectl-edit-", ext, buf) if err != nil { return preservedFile(err, results.file, out) } </ins> <del> // cleanup any file from the previous pass if len(results.file) > 0 { os.Remove(results.file) } </del> <ins> // cleanup any file from the previous pass if len(results.file) > 0 { os.Remove(results.file) } </ins> <del> glog.V(4).Infof("User edited:n%s", string(edited)) fmt.Printf("User edited:n%s", string(edited)) lines, err := hasLines(bytes.NewBuffer(edited)) if err != nil { return preservedFile(err, file, out) } if bytes.Equal(original, edited) { if len(results.edit) > 0 { preservedFile(nil, file, out) } else { os.Remove(file) </del> <ins> glog.V(4).Infof("User edited:n%s", string(edited)) lines, err := hasLines(bytes.NewBuffer(edited)) if err != nil { return preservedFile(err, file, out) </ins> } <del> fmt.Fprintln(out, "Edit cancelled, no changes made.") return nil } if !lines { if len(results.edit) > 0 { preservedFile(nil, file, out) } else { os.Remove(file) </del> <ins> // Compare content without comments if bytes.Equal(stripComments(original), stripComments(edited)) { if len(results.edit) > 0 { preservedFile(nil, file, out) } else { os.Remove(file) } fmt.Fprintln(out, "Edit cancelled, no changes made.") continue } if !lines { if len(results.edit) > 0 { preservedFile(nil, file, out) } else { os.Remove(file) } fmt.Fprintln(out, "Edit cancelled, saved file was empty.") continue </ins> } <del> fmt.Fprintln(out, "Edit cancelled, saved file was empty.") return nil } results = editResults{ file: file, } </del> <ins> results = editResults{ file: file, } </ins> <del> // parse the edited file updates, err := rmap.InfoForData(edited, "edited-file") if err != nil { return preservedFile(err, file, out) } </del> <ins> // parse the edited file updates, err := rmap.InfoForData(edited, "edited-file") if err != nil { return fmt.Errorf("The edited file had a syntax error: %v", err) } </ins> <del> // annotate the edited object for kubectl apply if err := kubectl.UpdateApplyAnnotation(updates); err != nil { return preservedFile(err, file, out) } </del> <ins> // annotate the edited object for kubectl apply if err := kubectl.UpdateApplyAnnotation(updates); err != nil { return preservedFile(err, file, out) } </ins> <del> visitor := resource.NewFlattenListVisitor(updates, rmap) </del> <ins> visitor := resource.NewFlattenListVisitor(updates, rmap) </ins> <del> // need to make sure the original namespace wasn't changed while editing if err = visitor.Visit(resource.RequireNamespace(cmdNamespace)); err != nil { return preservedFile(err, file, out) } </del> <ins> // need to make sure the original namespace wasn't changed while editing if err = visitor.Visit(resource.RequireNamespace(cmdNamespace)); err != nil { return preservedFile(err, file, out) } </ins> <del> // use strategic merge to create a patch originalJS, err := yaml.ToJSON(original) if err != nil { return preservedFile(err, file, out) } editedJS, err := yaml.ToJSON(edited) if err != nil { return preservedFile(err, file, out) } patch, err := strategicpatch.CreateStrategicMergePatch(originalJS, editedJS, obj) // TODO: change all jsonmerge to strategicpatch // for checking preconditions preconditions := []jsonmerge.PreconditionFunc{} if err != nil { glog.V(4).Infof("Unable to calculate diff, no merge is possible: %v", err) return preservedFile(err, file, out) } else { preconditions = append(preconditions, jsonmerge.RequireKeyUnchanged("apiVersion")) preconditions = append(preconditions, jsonmerge.RequireKeyUnchanged("kind")) preconditions = append(preconditions, jsonmerge.RequireMetadataKeyUnchanged("name")) results.version = defaultVersion } </del> <ins> // use strategic merge to create a patch originalJS, err := yaml.ToJSON(original) if err != nil { return preservedFile(err, file, out) } editedJS, err := yaml.ToJSON(edited) if err != nil { return preservedFile(err, file, out) } patch, err := strategicpatch.CreateStrategicMergePatch(originalJS, editedJS, obj) // TODO: change all jsonmerge to strategicpatch // for checking preconditions preconditions := []jsonmerge.PreconditionFunc{} if err != nil { glog.V(4).Infof("Unable to calculate diff, no merge is possible: %v", err) return preservedFile(err, file, out) } else { preconditions = append(preconditions, jsonmerge.RequireKeyUnchanged("apiVersion")) preconditions = append(preconditions, jsonmerge.RequireKeyUnchanged("kind")) preconditions = append(preconditions, jsonmerge.RequireMetadataKeyUnchanged("name")) results.version = defaultVersion } </ins> <del> if hold, msg := jsonmerge.TestPreconditionsHold(patch, preconditions); !hold { fmt.Fprintf(out, "error: %s", msg) return preservedFile(nil, file, out) } </del> <ins> if hold, msg := jsonmerge.TestPreconditionsHold(patch, preconditions); !hold { fmt.Fprintf(out, "error: %s", msg) return preservedFile(nil, file, out) } </ins> <del> err = visitor.Visit(func(info *resource.Info, err error) error { patched, err := resource.NewHelper(info.Client, info.Mapping).Patch(info.Namespace, info.Name, api.StrategicMergePatchType, patch) if err != nil { fmt.Fprintln(out, results.addError(err, info)) </del> <ins> err = visitor.Visit(func(info *resource.Info, err error) error { patched, err := resource.NewHelper(info.Client, info.Mapping).Patch(info.Namespace, info.Name, api.StrategicMergePatchType, patch) if err != nil { fmt.Fprintln(out, results.addError(err, info)) return nil } info.Refresh(patched, true) cmdutil.PrintSuccess(mapper, false, out, info.Mapping.Resource, info.Name, "edited") </ins> return nil <ins> }) if err != nil { return preservedFile(err, file, out) </ins> } <del> info.Refresh(patched, true) cmdutil.PrintSuccess(mapper, false, out, info.Mapping.Resource, info.Name, "edited") return nil }) if err != nil { return preservedFile(err, file, out) } if results.retryable > 0 { fmt.Fprintf(out, "You can run `kubectl replace -f %s` to try this update again.n", file) return errExit } if results.conflict > 0 { fmt.Fprintf(out, "You must update your local resource version and run `kubectl replace -f %s` to overwrite the remote changes.n", file) return errExit </del> <ins> if results.retryable > 0 { fmt.Fprintf(out, "You can run `kubectl replace -f %s` to try this update again.n", file) return errExit } if results.conflict > 0 { fmt.Fprintf(out, "You must update your local resource version and run `kubectl replace -f %s` to overwrite the remote changes.n", file) return errExit } if len(results.edit) == 0 { if results.notfound == 0 { os.Remove(file) } else { fmt.Fprintf(out, "The edits you made on deleted resources have been saved to %qn", file) } } </ins> } if len(results.edit) == 0 { <del> if results.notfound == 0 { os.Remove(file) } else { fmt.Fprintf(out, "The edits you made on deleted resources have been saved to %qn", file) } </del> return nil }

When trying to edit multiple resources, for example: will panic. Ref as the possible cause cc
if was the cause, any fix needs to happen in 1.1 as well, was already cherry picked there
Also shows that we need a test for this case.
is the cause. When we edit multiple resources (i.e., list-type objects), the list's Accessor.annotations (type ) is nil and will panic when UpdateApplyAnnotation tries to SetAnnotations ().


Question: return err } if ipCorrect && portsCorrect { <del> return nil </del> <ins> return r.epAdapter.EnsureEndpointSliceFromEndpoints(metav1.NamespaceDefault, e) </ins> } if !ipCorrect { // We *always* add our own IP address.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: EndpointSlice for masters was not created after enabling EndpointSlice on a pre-existing cluster. Only those EndpointSlices managed by kube-controller-manager were created. What you expected to happen: EndpointSlice for masters should be created after enabling EndpointSlice on a pre-existing cluster, the same as the EndpointSlices managed by kube-controller-manager. How to reproduce it (as minimally and precisely as possible): kubeadm to create a cluster: the EndpointSlice feature by editing the kube-apiserver, kube-controller-manager, kube-proxy manifest kube-apiserver and kube-controller-manager is restarted, the EndpointSlice in namespace will be missing. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools:Network plugin and version (if this is a network-related bug): Others:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved /assign


Question: if rOpts != nil { hc.Resources = dockercontainer.Resources{ Memory:     rOpts.GetMemoryLimitInBytes(), <del> MemorySwap: -1, </del> <ins> MemorySwap: dockertools.DefaultMemorySwap(), </ins> CPUShares:  rOpts.GetCpuShares(), CPUQuota:   rOpts.GetCpuQuota(), CPUPeriod:  rOpts.GetCpuPeriod(),

What happened: Latest kubelet + Windows Docker version 1.14.0-dev refused to start any containers, error output: "invalid option: Windows does not support MemorySwap" How to reproduce it (as minimally and precisely as possible): It's always reproducible with Windows Docker 1.14.0-dev + latest kubelet code base. Anything else do we need to know: It didn't happen for Windows Docker 1.12.2-cs2-ws-beta. The reason behind the issue was because new code was for Windows Docker to check Resources parameters, it doesn't accept a non-zero value for MemorySwap: But kubelet was default to set an unlimited (-1) value. The issue was gone in my setup after applied a private fix to disable MemorySwap by set it to zero for Windows.


Question: import ( "context" <ins> "crypto/tls" </ins> "encoding/json" "flag" "fmt"

ci-kubernetes-node-kubelet-serial-hugepages when start the pod with the resources requests that contain multiple hugepages resources should set correct hugetlb mount and limit under the container cgroup Unexpected error: "pod ran to completion", Multiple sizes huge pages is going to be graduated to GA, so having this test job fixed is quite important.
/sig node
/assign
/triage accepted
/cc
I did a small analysis and I can see that the pod failed to start, I can not find any failure messages under the kubelet log, so it really hard to say why it failed.
were you able to reproduce it locally? btw, there is a "right way" to fix this - to fix dynamic update of a hugepage resources. This would eliminate need in a kubelet restart. There is even a PR for that: , we just need to make it working again.
after conversation with on slack I'd say we should stick to the current procedure - restarting kubelet when hugepages allocation changes. So, we should investigate why kubelet doesn't restart properly in this test case and fix it.
Nope, I failed to reproduce it locally, also under the CI, it does not appear often. The best thing will be to describe the pod in case if it failed, so we will see the reason.
under the code we have so from the log , I can see that the kubelet received the pod before the kubelet stopped, so the problem that pod dropped to Failed state because of the kubelet stop.
So some additional details, the one who responsible to restart the kubelet again, is the server service: restart the kubelet service are waiting for hugepages to be available are starting the pod server service restart the kubelet again because it was not healthy for too long - failed to start because of the kubelet restart
great findings! Should we just stop the kubelet and let the server service to start it again? We can also check if the kubelet is restarted successfully before starting the pod.
I tried it under the PR , also I unsure how can we check if the kubelet restarted successfully. All tests that I wrote before I checked if the node has NotReady state and waited until it will be Ready, but in case of the restart probably the node will always have Ready state because the time period is very short. I will continue to investigate and will check locally how can I fix it.


Question: import ( "context" <ins> "crypto/tls" </ins> "encoding/json" "flag" "fmt"

So the only reliable solution that I found, it just trying to start the pod number of times. I tested it 5 times and the test passed. Can you please review
Can we access kubelet service from hugepage test cases? If we can we can use its APIs like or to check if kubelet is ready after the restart. Overall I have a feeling of duplication test/e2enode/services functionality in This is a main reason for this and similar bugs I believe. It would be nice to get rid of this duplication if possible. That would nicely isolate kubelet service controls in one place and eliminate conflicts like this one. wdyt?
not really, the used mostly to start all needed components to emulate the cluster(not only the kubelet) and the test does not have access to it. I think you can separate the readiness check under the importable method, but it worth the separate PR.
I updated the PR to check the healthiness of the kubelet and looks like it works just fine.
thank you. it looks good to me so far. I'll add lgtm label when is merged and tests are all green.
the job is still flaky. can you reopen this issue?
I opened a different one - , the issue can affect any job, not only the hugepages one.
I can open a fix to W/A an issue(start the pod until it will succeed), it should fix all problems, WDYT?
i think it's ok as a workaround, until is fixed
would it be better to wait until api server reports correct resource value before running the pod?
We are waiting for it, the problem that the scheduler and the kubelet see a different picture regarding resources requested by pods.
ok. In this case restarting the pod until it succeeds seems reasonable to me to fix the flakiness until is fixed.
any progress on the workaround fix?


Question: } func runStaticPodTest(c *client.Client, configFilePath string) { <del> manifest := `version: v1beta2 id: static-pod </del> <ins> var testCases = []struct { desc         string fileContents string }{ { desc: "static-pod-from-manifest", fileContents: `version: v1beta2 id: static-pod-from-manifest </ins> containers: <del> - name: static-container image: kubernetes/pause` manifestFile, err := ioutil.TempFile(configFilePath, "") defer os.Remove(manifestFile.Name()) ioutil.WriteFile(manifestFile.Name(), []byte(manifest), 0600) // Wait for the mirror pod to be created. podName := "static-pod-localhost" namespace := kubelet.NamespaceDefault if err := wait.Poll(time.Second, time.Minute*2, podRunning(c, namespace, podName)); err != nil { if pods, err := c.Pods(namespace).List(labels.Everything()); err == nil { for _, pod := range pods.Items { glog.Infof("pod found: %s/%s", namespace, pod.Name) } } glog.Fatalf("FAILED: mirror pod has not been created or is not running: %v", err) } // Delete the mirror pod, and wait for it to be recreated. c.Pods(namespace).Delete(podName) if err = wait.Poll(time.Second, time.Second*30, podRunning(c, namespace, podName)); err != nil { glog.Fatalf("FAILED: mirror pod has not been re-created or is not running: %v", err) } // Remove the manifest file, and wait for the mirror pod to be deleted. os.Remove(manifestFile.Name()) if err = wait.Poll(time.Second, time.Second*30, podNotFound(c, namespace, podName)); err != nil { glog.Fatalf("FAILED: mirror pod has not been deleted: %v", err) </del> <ins> - name: static-container image: kubernetes/pause`, }, { desc: "static-pod-from-spec", fileContents: `{ "kind": "Pod", "apiVersion": "v1beta3", "metadata": { "name": "static-pod-from-spec" }, "spec": { "containers": [{ "name": "static-container", "image": "kubernetes/pause" }] } }`, }, </ins> } <ins> for _, testCase := range testCases { func() { desc := testCase.desc manifestFile, err := ioutil.TempFile(configFilePath, "") defer os.Remove(manifestFile.Name()) ioutil.WriteFile(manifestFile.Name(), []byte(testCase.fileContents), 0600) // Wait for the mirror pod to be created. podName := fmt.Sprintf("%s-localhost", desc) namespace := kubelet.NamespaceDefault if err := wait.Poll(time.Second, time.Minute*2, podRunning(c, namespace, podName)); err != nil { if pods, err := c.Pods(namespace).List(labels.Everything()); err == nil { for _, pod := range pods.Items { glog.Infof("pod found: %s/%s", namespace, pod.Name) } } glog.Fatalf("%s FAILED: mirror pod has not been created or is not running: %v", desc, err) } // Delete the mirror pod, and wait for it to be recreated. c.Pods(namespace).Delete(podName) if err = wait.Poll(time.Second, time.Second*30, podRunning(c, namespace, podName)); err != nil { glog.Fatalf("%s FAILED: mirror pod has not been re-created or is not running: %v", desc, err) } // Remove the manifest file, and wait for the mirror pod to be deleted. os.Remove(manifestFile.Name()) if err = wait.Poll(time.Second, time.Second*30, podNotFound(c, namespace, podName)); err != nil { glog.Fatalf("%s FAILED: mirror pod has not been deleted: %v", desc, err) } }() } </ins> } func runReplicationControllerTest(c *client.Client) {

Currently it assumes it can read ContainerManifest - it should try both
Why not hold on this until we have Kubelet-read-Pod in?
Also the HTTP source has the same problem.
It should read Pod. Changed the title.
Ideally, I'd like all Pods created directly via Kubelet to be to etcd and surfaced via the API. If updates are made via the API, I'd like those to propagate back to the node. We can discuss whether the config file should be updated or ignored. At some point, I'd like Kubelet to persistently cache pods received from the apiserver, perhaps by just writing them all out as config files.
is the issue to sync local pods back to apiserver.
this is somewhat related to other issues we talked about.
Detail: We want locally specified pods to be able to use YAML config files, but we also want to rip YAML out of the API server . The config-file behavior should be as similar to the kubectl experience. Kubelet could use parts of the kubectl library to translate YAML to JSON. Vision: Eventually, I'd like Kubelet to use the apiserver library and support exactly the same Pod API as the master, at which time I'd like kubectl be able to create pods directly on Kubelets, as well as reading from config files. I also expect to have a per-node controller . I'd like to be able to use the per-node controller to update pods started via config files (or direct kubectl-kubelet pods), overriding those original files. For availability and self-hosting, we'll eventually need local checkpointing , ideally just using serialized JSON for each object. If Kubelet used apiserver, this could be done using a custom registry implementation.
I think that is a prerequisite for it.
If noone picks it up, I'm going to start working on it tomorrow.
thanks, wojtek-t
fixed it for http channel


Question: func (o *ResourcesOptions) Validate() error { var err error if len(o.Limits) == 0 && len(o.Requests) == 0 { <del> return fmt.Errorf("you must specify an update to requests or limits (in the form of --requests/--limits)") </del> <ins> return fmt.Errorf("you must specify an update to requests or limits or  (in the form of --requests/--limits)") </ins> } o.ResourceRequirements, err = kubectl.HandleResourceRequirements(map[string]string{"limits": o.Limits, "requests": o.Requests})

from https://k8s-
Really suspect this is
This is putting us into test-go post-commit run is broken
cc:
marked this p0 - if a fix isn't easy let's try to identify what introduced this and revert.
This flake on: 2016-10-30 07:19 PDT First instance: : It is flaking frequently, blocking PRs from getting merged, and slowing the submit queue ( at 75%). As on-call, I'm going to try and identify the cluprit and revert. Can use all help in doing so.
can you investigate whether your additional tests could cause this (by breaking the deployment of those pods, for instance) , Saad Ali wrote:
PR looks most suspect to me. It mucked with , and the errors we are seeing in each of the failures is related to : That PR was merged at Oct 30, 2016, 6:18 AM PDT and the first flake in was at Oct 30, 2016, 7:19 AM PDT. Thoughts? CC
I'm going to initiate a revert of to see if that fixes the issue
IMO we can just change the flaky test to not get pods but some other resources, such as deployments. We can't really expect to clean up pods immediately.
I'm going to proceed with revert (). Please feel free to send a follow up PR to handle this more gracefully.
Going to leave this opened until we have a few hours of runs with the revert. Want to verify, with some degree of certainty, that the revert took care of this flake.
We had 15 green runs of in a row. Considering this flake fixed.


Question: }) } } <ins> func TestGetExcludedChecks(t *testing.T) { type args struct { r *http.Request } tests := []struct { name string r    *http.Request want sets.String }{ {"Should have no excluded health checks", createGetRequestWithUrl("/healthz?verbose=true"), sets.NewString(), }, {"Should extract out the ping health check", createGetRequestWithUrl("/healthz?exclude=ping"), sets.NewString("ping"), }, {"Should extract out ping and log health check", createGetRequestWithUrl("/healthz?exclude=ping&exclude=log"), sets.NewString("ping", "log"), }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { if got := getExcludedChecks(tt.r); !reflect.DeepEqual(got, tt.want) { t.Errorf("getExcludedChecks() = %v, want %v", got, tt.want) } }) } } func createGetRequestWithUrl(rawUrlString string) *http.Request { url, _ := url.Parse(rawUrlString) return &http.Request{ Method: http.MethodGet, Proto:  "HTTP/1.1", URL:    url, } } </ins>

What would you like to be : Add support to the kube-apiserver so that we can disable health-checks via query params. Why is this needed: Currently, the healthz endpoint is inflexible in that any and all health checks are currently executed when the endpoint is hit. Since the kube-apiserver only currently supports a liveness check, this means all persistent health check failures will eventually result in a kubelet driven restart. However, this may not be desirable behavior if you have additional monitoring services layered on top of your control plane. For instance, take the etcd failure from the perspective of the kube-apiserver. Currently, persistent failures during the etcd health check from the api-server healthz endpoint will cause the kube-apiserver to be restarted. This sometimes helps (in the case of a faulty etcd-client connection) but also can be detrimental (the kube-apiserver restarts until the etcd cluster is functioning, then the kube-apiserver bombards the etcd cluster with requests to update local caches, saturating the etcd cluster with requests, causing it to timeout on its own liveness probes and trigger a kubelet-driven restart). Let's say that we could now disable etcd from our liveness health check. Then if we had an external monitoring service, we could ask independently ask etcd if it was healthy and we could also ask the kube-apiserver if it thought etcd was unhealthy. Based on the combination of those answers, we could enable more intelligent responses to certain situation, like restarting the kube-apiserver when it thought etcd was unhealthy but etcd was actually reporting as healthy but not doing so when etcd was actually unhealthy. This feature is backwards-compatible and opt-in. There would be no impact to existing behavior unless these query params are actually used. /sig api-machinery /kind feature
/cc
SGTM
/assign