Question:
The method doesn't check taints and tolerations. This is problematic in 2 scenarios: If the users bypass scheduler by setting a value in If the node restarts. The node would try to restart scheduled pods once it comes back up, even if it's not ready (has a not-ready taint). Kubelet to check taints and tolerations. a taint in a node. a Pod with set to the Node above, without a matching toleration. /sig node /sig scheduling
this reminds me of your kubelet sync PR ...
I don't think this is a syncing issue. The logic doesn't check taints at all.
scenario 1 is pretty bad, i will let the #sig-node slack know about this.
It doesn't sound like a syncing issue, but we seem to have multiple issues with pod admission/management on restart currently
I'm aware of that issue. This should be unrelated.
I would like to address this. /assign
I think it would be great to proceed after merging ref:
Please add me as reviewer
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale PR in review:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale

reasons: []PredicateFailureReason{&PredicateFailureError{nodeports.Name, nodeports.ErrReason}}, name:    "hostport conflict", }, <ins> { pod: &v1.Pod{ Spec: v1.PodSpec{ Tolerations: []v1.Toleration{ {Key: "foo"}, {Key: "bar"}, }, }, }, nodeInfo: schedulerframework.NewNodeInfo(), node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: "machine1"}, Spec: v1.NodeSpec{ Taints: []v1.Taint{ {Key: "foo", Effect: v1.TaintEffectNoSchedule}, {Key: "bar", Effect: v1.TaintEffectNoExecute}, }, }, Status: v1.NodeStatus{Capacity: makeResources(10, 20, 32, 0, 0, 0).Capacity, Allocatable: makeAllocatableResources(10, 20, 32, 0, 0, 0)}, }, name: "taint/toleration match", }, { pod:      &v1.Pod{}, nodeInfo: schedulerframework.NewNodeInfo(), node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: "machine1"}, Spec: v1.NodeSpec{ Taints: []v1.Taint{ {Key: "foo", Effect: v1.TaintEffectNoSchedule}, }, }, Status: v1.NodeStatus{Capacity: makeResources(10, 20, 32, 0, 0, 0).Capacity, Allocatable: makeAllocatableResources(10, 20, 32, 0, 0, 0)}, }, name: "NoSchedule taint/toleration not match", }, { pod:      &v1.Pod{}, nodeInfo: schedulerframework.NewNodeInfo(), node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: "machine1"}, Spec: v1.NodeSpec{ Taints: []v1.Taint{ {Key: "bar", Effect: v1.TaintEffectNoExecute}, }, }, Status: v1.NodeStatus{Capacity: makeResources(10, 20, 32, 0, 0, 0).Capacity, Allocatable: makeAllocatableResources(10, 20, 32, 0, 0, 0)}, }, reasons: []PredicateFailureReason{&PredicateFailureError{tainttoleration.Name, tainttoleration.ErrReasonNotMatch}}, name:    "NoExecute taint/toleration not match", }, { pod:      &v1.Pod{}, nodeInfo: schedulerframework.NewNodeInfo(), node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: "machine1"}, Spec: v1.NodeSpec{ Taints: []v1.Taint{ {Key: "baz", Effect: v1.TaintEffectPreferNoSchedule}, }, }, Status: v1.NodeStatus{Capacity: makeResources(10, 20, 32, 0, 0, 0).Capacity, Allocatable: makeAllocatableResources(10, 20, 32, 0, 0, 0)}, }, name: "PreferNoSchedule taint/toleration not match", }, { pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ types.ConfigSourceAnnotationKey: types.FileSource, }, }, }, nodeInfo: schedulerframework.NewNodeInfo(), node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: "machine1"}, Spec: v1.NodeSpec{ Taints: []v1.Taint{ {Key: "foo", Effect: v1.TaintEffectNoSchedule}, {Key: "bar", Effect: v1.TaintEffectNoExecute}, }, }, Status: v1.NodeStatus{Capacity: makeResources(10, 20, 32, 0, 0, 0).Capacity, Allocatable: makeAllocatableResources(10, 20, 32, 0, 0, 0)}, }, name: "static pods ignore taints", }, </ins> } for _, test := range resourceTests { t.Run(test.name, func(t *testing.T) {


Question:
[x] test approval issue : [X] test pr : 26 May [x] two weeks soak end date : 9 June [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining DaemonSet endpoints which are untested. Create a watch to track daemon set events that DaemonSet with a static label Create a daemon set with a static label. Confirm that the pods are running. Get the daemon set status. Parse the response and confirm that the daemon set status conditions can be listed. Update the daemon set status. Confirm via the watch that the status has been updated. Patch the daemon set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new which validates that three new endpoints are hit. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

"text/tabwriter" "time" <ins> "k8s.io/client-go/tools/cache" </ins> "github.com/onsi/ginkgo" "github.com/onsi/gomega" appsv1 "k8s.io/api/apps/v1"


Question:
[x] test approval issue : [X] test pr : 26 May [x] two weeks soak end date : 9 June [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining DaemonSet endpoints which are untested. Create a watch to track daemon set events that DaemonSet with a static label Create a daemon set with a static label. Confirm that the pods are running. Get the daemon set status. Parse the response and confirm that the daemon set status conditions can be listed. Update the daemon set status. Confirm via the watch that the status has been updated. Patch the daemon set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new which validates that three new endpoints are hit. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime" <ins> "k8s.io/apimachinery/pkg/runtime/schema" </ins> "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/wait" <ins> watch "k8s.io/apimachinery/pkg/watch" </ins> clientset "k8s.io/client-go/kubernetes" "k8s.io/client-go/kubernetes/scheme" <ins> watchtools "k8s.io/client-go/tools/watch" "k8s.io/client-go/util/retry" </ins> podutil "k8s.io/kubernetes/pkg/api/v1/pod" extensionsinternal "k8s.io/kubernetes/pkg/apis/extensions" "k8s.io/kubernetes/pkg/controller/daemon"


Question:
[x] test approval issue : [X] test pr : 26 May [x] two weeks soak end date : 9 June [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining DaemonSet endpoints which are untested. Create a watch to track daemon set events that DaemonSet with a static label Create a daemon set with a static label. Confirm that the pods are running. Get the daemon set status. Parse the response and confirm that the daemon set status conditions can be listed. Update the daemon set status. Confirm via the watch that the status has been updated. Patch the daemon set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new which validates that three new endpoints are hit. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

framework.ExpectNoError(err, "failed to list DaemonSets") framework.ExpectEqual(len(dsList.Items), 0, "filtered list should have no daemonset") }) <ins> ginkgo.It("should verify changes to a daemon set status", func() { label := map[string]string{daemonsetNameLabel: dsName} labelSelector := labels.SelectorFromSet(label).String() dsClient := f.ClientSet.AppsV1().DaemonSets(ns) cs := f.ClientSet w := &cache.ListWatch{ WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { options.LabelSelector = labelSelector return dsClient.Watch(context.TODO(), options) }, } dsList, err := cs.AppsV1().DaemonSets("").List(context.TODO(), metav1.ListOptions{LabelSelector: labelSelector}) framework.ExpectNoError(err, "failed to list Daemon Sets") ginkgo.By(fmt.Sprintf("Creating simple DaemonSet %q", dsName)) testDaemonset, err := c.AppsV1().DaemonSets(ns).Create(context.TODO(), newDaemonSetWithLabel(dsName, image, label), metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("Check that daemon pods launch on every node of the cluster.") err = wait.PollImmediate(dsRetryPeriod, dsRetryTimeout, checkRunningOnAllNodes(f, testDaemonset)) framework.ExpectNoError(err, "error waiting for daemon pod to start") err = checkDaemonStatus(f, dsName) framework.ExpectNoError(err) ginkgo.By("Getting /status") dsResource := schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "daemonsets"} dsStatusUnstructured, err := f.DynamicClient.Resource(dsResource).Namespace(ns).Get(context.TODO(), dsName, metav1.GetOptions{}, "status") framework.ExpectNoError(err, "Failed to fetch the status of daemon set %s in namespace %s", dsName, ns) dsStatusBytes, err := json.Marshal(dsStatusUnstructured) framework.ExpectNoError(err, "Failed to marshal unstructured response. %v", err) var dsStatus appsv1.DaemonSet err = json.Unmarshal(dsStatusBytes, &dsStatus) framework.ExpectNoError(err, "Failed to unmarshal JSON bytes to a daemon set object type") framework.Logf("Daemon Set %s has Conditions: %v", dsName, dsStatus.Status.Conditions) ginkgo.By("updating the DaemonSet Status") var statusToUpdate, updatedStatus *appsv1.DaemonSet err = retry.RetryOnConflict(retry.DefaultRetry, func() error { statusToUpdate, err = dsClient.Get(context.TODO(), dsName, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to retrieve daemon set %s", dsName) statusToUpdate.Status.Conditions = append(statusToUpdate.Status.Conditions, appsv1.DaemonSetCondition{ Type:    "StatusUpdate", Status:  "True", Reason:  "E2E", Message: "Set from e2e test", }) updatedStatus, err = dsClient.UpdateStatus(context.TODO(), statusToUpdate, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "Failed to update status. %v", err) framework.Logf("updatedStatus.Conditions: %#v", updatedStatus.Status.Conditions) ginkgo.By("watching for the daemon set status to be updated") ctx, cancel := context.WithTimeout(context.Background(), dsRetryTimeout) defer cancel() _, err = watchtools.Until(ctx, dsList.ResourceVersion, w, func(event watch.Event) (bool, error) { if ds, ok := event.Object.(*appsv1.DaemonSet); ok { found := ds.ObjectMeta.Name == testDaemonset.ObjectMeta.Name && ds.ObjectMeta.Namespace == testDaemonset.ObjectMeta.Namespace && ds.Labels[daemonsetNameLabel] == dsName if !found { framework.Logf("Observed daemon set %v in namespace %v with annotations: %v & Conditions: %v", ds.ObjectMeta.Name, ds.ObjectMeta.Namespace, ds.Annotations, ds.Status.Conditions) return false, nil } for _, cond := range ds.Status.Conditions { if cond.Type == "StatusUpdate" && cond.Reason == "E2E" && cond.Message == "Set from e2e test" { framework.Logf("Found daemon set %v in namespace %v with labels: %v annotations: %v & Conditions: %v", ds.ObjectMeta.Name, ds.ObjectMeta.Namespace, ds.ObjectMeta.Labels, ds.Annotations, ds.Status.Conditions) return found, nil } framework.Logf("Observed daemon set %v in namespace %v with annotations: %v & Conditions: %v", ds.ObjectMeta.Name, ds.ObjectMeta.Namespace, ds.Annotations, ds.Status.Conditions) } } object := strings.Split(fmt.Sprintf("%v", event.Object), "{")[0] framework.Logf("Observed %v event: %+v", object, event.Type) return false, nil }) framework.ExpectNoError(err, "failed to locate daemon set %v in namespace %v", testDaemonset.ObjectMeta.Name, ns) framework.Logf("Daemon set %s has an updated status", dsName) ginkgo.By("patching the DaemonSet Status") daemonSetStatusPatch := appsv1.DaemonSet{ Status: appsv1.DaemonSetStatus{ Conditions: []appsv1.DaemonSetCondition{ { Type:   "StatusPatched", Status: "True", }, }, }, } payload, err := json.Marshal(daemonSetStatusPatch) framework.ExpectNoError(err, "Failed to marshal JSON. %v", err) _, err = dsClient.Patch(context.TODO(), dsName, types.MergePatchType, payload, metav1.PatchOptions{}, "status") framework.ExpectNoError(err, "Failed to patch daemon set status", err) ginkgo.By("watching for the daemon set status to be patched") ctx, cancel = context.WithTimeout(context.Background(), dsRetryTimeout) defer cancel() _, err = watchtools.Until(ctx, dsList.ResourceVersion, w, func(event watch.Event) (bool, error) { if ds, ok := event.Object.(*appsv1.DaemonSet); ok { found := ds.ObjectMeta.Name == testDaemonset.ObjectMeta.Name && ds.ObjectMeta.Namespace == testDaemonset.ObjectMeta.Namespace && ds.Labels[daemonsetNameLabel] == dsName if !found { framework.Logf("Observed daemon set %v in namespace %v with annotations: %v & Conditions: %v", ds.ObjectMeta.Name, ds.ObjectMeta.Namespace, ds.Annotations, ds.Status.Conditions) return false, nil } for _, cond := range ds.Status.Conditions { if cond.Type == "StatusPatched" { framework.Logf("Found daemon set %v in namespace %v with labels: %v annotations: %v & Conditions: %v", ds.ObjectMeta.Name, ds.ObjectMeta.Namespace, ds.ObjectMeta.Labels, ds.Annotations, ds.Status.Conditions) return found, nil } framework.Logf("Observed daemon set %v in namespace %v with annotations: %v & Conditions: %v", ds.ObjectMeta.Name, ds.ObjectMeta.Namespace, ds.Annotations, ds.Status.Conditions) } } object := strings.Split(fmt.Sprintf("%v", event.Object), "{")[0] framework.Logf("Observed %v event: %v", object, event.Type) return false, nil }) framework.ExpectNoError(err, "failed to locate daemon set %v in namespace %v", testDaemonset.ObjectMeta.Name, ns) framework.Logf("Daemon set %s has a patched status", dsName) }) </ins> }) // randomPod selects a random pod within pods that causes fn to return true, or nil


Question:
[x] test approval issue : [X] test pr : 26 May [x] two weeks soak end date : 9 June [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining DaemonSet endpoints which are untested. Create a watch to track daemon set events that DaemonSet with a static label Create a daemon set with a static label. Confirm that the pods are running. Get the daemon set status. Parse the response and confirm that the daemon set status conditions can be listed. Update the daemon set status. Confirm via the watch that the status has been updated. Patch the daemon set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new which validates that three new endpoints are hit. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

description: A conformant Kubernetes distribution MUST support DaemonSet RollingUpdates. release: v1.10 file: test/e2e/apps/daemon_set.go <ins> - testname: DaemonSet, status sub-resource codename: '[sig-apps] Daemon set [Serial] should verify changes to a daemon set status [Conformance]' description: When a DaemonSet is created it MUST succeed. Attempt to read, update and patch its status sub-resource; all mutating sub-resource operations MUST be visible to subsequent reads. release: v1.22 file: test/e2e/apps/daemon_set.go </ins> - testname: Deployment, completes the scaling of a Deployment subresource codename: '[sig-apps] Deployment Deployment should have a working scale subresource [Conformance]'


Question:
[x] test approval issue : [X] test pr : 26 May [x] two weeks soak end date : 9 June [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining DaemonSet endpoints which are untested. Create a watch to track daemon set events that DaemonSet with a static label Create a daemon set with a static label. Confirm that the pods are running. Get the daemon set status. Parse the response and confirm that the daemon set status conditions can be listed. Update the daemon set status. Confirm via the watch that the status has been updated. Patch the daemon set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new which validates that three new endpoints are hit. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

framework.ExpectEqual(len(dsList.Items), 0, "filtered list should have no daemonset") }) <del> ginkgo.It("should verify changes to a daemon set status", func() { </del> <ins> /*	Release: v1.22 Testname: DaemonSet, status sub-resource Description: When a DaemonSet is created it MUST succeed. Attempt to read, update and patch its status sub-resource; all mutating sub-resource operations MUST be visible to subsequent reads. */ framework.ConformanceIt("should verify changes to a daemon set status", func() { </ins> label := map[string]string{daemonsetNameLabel: dsName} labelSelector := labels.SelectorFromSet(label).String()


Question:
Originally tracked at . After fixing "not include test files in import-boss" in , we have a regression here: is using . We need remove dependencies from integration tests to e2e tests again. /assign /sig testing
/sig auth as per the required change
/kind cleanup /area test
It would be helpful to outline an argument as to why this is the case. I can imagine reasons, but none are mentioned here.
Description updated.
/triage accepted /priority backlog /milestone v1.21 This seems a simple enough fix to land (and thank you for doing this) but marking low enough priority to make it clear we may back out and wait for v1.22 if we don't have bandwidth for this
So while I think we should probably do this anyhow I want to note: This issue lists the reasons: we have no intention to stage e2e (there's been much discussion about this since then) we generally don't want anything non-informational (required jobs) running only sometimes anyhow, it's just not reliable Decoupling these seems like a good idea from a purely code-organization perspective, but the reasons in the previous issue are no longer valid.
/milestone v1.22 The PR didn't land in time. At this point I view this as nice to have. Even if it's low impact, let's wait until next release.

rules: # Discourage import of k8s.io/kubernetes/test/e2e - selectorRegexp: k8s[.]io/kubernetes/test/e2e <del> # TODO: import-boss --include-test-files is catching these; drive to zero allowedPrefixes: # test/integration/auth/bootstraptoken_test.go is using this - k8s.io/kubernetes/test/e2e/lifecycle/bootstrap </del> <ins> forbiddenPrefixes: - "" </ins>


Question:
Originally tracked at . After fixing "not include test files in import-boss" in , we have a regression here: is using . We need remove dependencies from integration tests to e2e tests again. /assign /sig testing
/sig auth as per the required change
/kind cleanup /area test
It would be helpful to outline an argument as to why this is the case. I can imagine reasons, but none are mentioned here.
Description updated.
/triage accepted /priority backlog /milestone v1.21 This seems a simple enough fix to land (and thank you for doing this) but marking low enough priority to make it clear we may back out and wait for v1.22 if we don't have bandwidth for this
So while I think we should probably do this anyhow I want to note: This issue lists the reasons: we have no intention to stage e2e (there's been much discussion about this since then) we generally don't want anything non-informational (required jobs) running only sometimes anyhow, it's just not reliable Decoupling these seems like a good idea from a purely code-organization perspective, but the reasons in the previous issue are no longer valid.
/milestone v1.22 The PR didn't land in time. At this point I view this as nice to have. Even if it's low impact, let's wait until next release.

"k8s.io/apiserver/pkg/authentication/request/bearertoken" bootstrapapi "k8s.io/cluster-bootstrap/token/api" "k8s.io/kubernetes/plugin/pkg/auth/authenticator/token/bootstrap" <del> bootstraputil "k8s.io/kubernetes/test/e2e/lifecycle/bootstrap" </del> "k8s.io/kubernetes/test/integration" "k8s.io/kubernetes/test/integration/framework" )


Question:
Originally tracked at . After fixing "not include test files in import-boss" in , we have a regression here: is using . We need remove dependencies from integration tests to e2e tests again. /assign /sig testing
/sig auth as per the required change
/kind cleanup /area test
It would be helpful to outline an argument as to why this is the case. I can imagine reasons, but none are mentioned here.
Description updated.
/triage accepted /priority backlog /milestone v1.21 This seems a simple enough fix to land (and thank you for doing this) but marking low enough priority to make it clear we may back out and wait for v1.22 if we don't have bandwidth for this
So while I think we should probably do this anyhow I want to note: This issue lists the reasons: we have no intention to stage e2e (there's been much discussion about this since then) we generally don't want anything non-informational (required jobs) running only sometimes anyhow, it's just not reliable Decoupling these seems like a good idea from a purely code-organization perspective, but the reasons in the previous issue are no longer valid.
/milestone v1.22 The PR didn't land in time. At this point I view this as nice to have. Even if it's low impact, let's wait until next release.

// TestBootstrapTokenAuth tests the bootstrap token auth provider func TestBootstrapTokenAuth(t *testing.T) { <del> tokenID, err := bootstraputil.GenerateTokenID() if err != nil { t.Fatalf("unexpected error: %v", err) } secret, err := bootstraputil.GenerateTokenSecret() if err != nil { t.Fatalf("unexpected error: %v", err) } </del> <ins> validTokenID := "token1" validSecret := "validtokensecret" </ins> var bootstrapSecretValid = &corev1.Secret{ ObjectMeta: metav1.ObjectMeta{ Namespace: metav1.NamespaceSystem,


Question:
Originally tracked at . After fixing "not include test files in import-boss" in , we have a regression here: is using . We need remove dependencies from integration tests to e2e tests again. /assign /sig testing
/sig auth as per the required change
/kind cleanup /area test
It would be helpful to outline an argument as to why this is the case. I can imagine reasons, but none are mentioned here.
Description updated.
/triage accepted /priority backlog /milestone v1.21 This seems a simple enough fix to land (and thank you for doing this) but marking low enough priority to make it clear we may back out and wait for v1.22 if we don't have bandwidth for this
So while I think we should probably do this anyhow I want to note: This issue lists the reasons: we have no intention to stage e2e (there's been much discussion about this since then) we generally don't want anything non-informational (required jobs) running only sometimes anyhow, it's just not reliable Decoupling these seems like a good idea from a purely code-organization perspective, but the reasons in the previous issue are no longer valid.
/milestone v1.22 The PR didn't land in time. At this point I view this as nice to have. Even if it's low impact, let's wait until next release.

}, Type: corev1.SecretTypeBootstrapToken, Data: map[string][]byte{ <del> bootstrapapi.BootstrapTokenIDKey:               []byte(tokenID), bootstrapapi.BootstrapTokenSecretKey:           []byte(secret), </del> <ins> bootstrapapi.BootstrapTokenIDKey:               []byte(validTokenID), bootstrapapi.BootstrapTokenSecretKey:           []byte(validSecret), </ins> bootstrapapi.BootstrapTokenUsageAuthentication: []byte("true"), }, }


Question:
Originally tracked at . After fixing "not include test files in import-boss" in , we have a regression here: is using . We need remove dependencies from integration tests to e2e tests again. /assign /sig testing
/sig auth as per the required change
/kind cleanup /area test
It would be helpful to outline an argument as to why this is the case. I can imagine reasons, but none are mentioned here.
Description updated.
/triage accepted /priority backlog /milestone v1.21 This seems a simple enough fix to land (and thank you for doing this) but marking low enough priority to make it clear we may back out and wait for v1.22 if we don't have bandwidth for this
So while I think we should probably do this anyhow I want to note: This issue lists the reasons: we have no intention to stage e2e (there's been much discussion about this since then) we generally don't want anything non-informational (required jobs) running only sometimes anyhow, it's just not reliable Decoupling these seems like a good idea from a purely code-organization perspective, but the reasons in the previous issue are no longer valid.
/milestone v1.22 The PR didn't land in time. At this point I view this as nice to have. Even if it's low impact, let's wait until next release.

}, Type: corev1.SecretTypeBootstrapToken, Data: map[string][]byte{ <del> bootstrapapi.BootstrapTokenIDKey:               []byte(tokenID), </del> <ins> bootstrapapi.BootstrapTokenIDKey:               []byte(validTokenID), </ins> bootstrapapi.BootstrapTokenSecretKey:           []byte("invalid"), bootstrapapi.BootstrapTokenUsageAuthentication: []byte("true"), }, } <ins> tokenExpiredTime := time.Now().Add(-time.Hour).Format(time.RFC3339) </ins> var expiredBootstrapToken = &corev1.Secret{ ObjectMeta: metav1.ObjectMeta{ Namespace: metav1.NamespaceSystem,


Question:
Originally tracked at . After fixing "not include test files in import-boss" in , we have a regression here: is using . We need remove dependencies from integration tests to e2e tests again. /assign /sig testing
/sig auth as per the required change
/kind cleanup /area test
It would be helpful to outline an argument as to why this is the case. I can imagine reasons, but none are mentioned here.
Description updated.
/triage accepted /priority backlog /milestone v1.21 This seems a simple enough fix to land (and thank you for doing this) but marking low enough priority to make it clear we may back out and wait for v1.22 if we don't have bandwidth for this
So while I think we should probably do this anyhow I want to note: This issue lists the reasons: we have no intention to stage e2e (there's been much discussion about this since then) we generally don't want anything non-informational (required jobs) running only sometimes anyhow, it's just not reliable Decoupling these seems like a good idea from a purely code-organization perspective, but the reasons in the previous issue are no longer valid.
/milestone v1.22 The PR didn't land in time. At this point I view this as nice to have. Even if it's low impact, let's wait until next release.

}, Type: corev1.SecretTypeBootstrapToken, Data: map[string][]byte{ <del> bootstrapapi.BootstrapTokenIDKey:               []byte(tokenID), </del> <ins> bootstrapapi.BootstrapTokenIDKey:               []byte(validTokenID), </ins> bootstrapapi.BootstrapTokenSecretKey:           []byte("invalid"), bootstrapapi.BootstrapTokenUsageAuthentication: []byte("true"), <del> bootstrapapi.BootstrapTokenExpirationKey:       []byte(bootstraputil.TimeStringFromNow(-time.Hour)), </del> <ins> bootstrapapi.BootstrapTokenExpirationKey:       []byte(tokenExpiredTime), </ins> }, } type request struct {


Question:
Originally tracked at . After fixing "not include test files in import-boss" in , we have a regression here: is using . We need remove dependencies from integration tests to e2e tests again. /assign /sig testing
/sig auth as per the required change
/kind cleanup /area test
It would be helpful to outline an argument as to why this is the case. I can imagine reasons, but none are mentioned here.
Description updated.
/triage accepted /priority backlog /milestone v1.21 This seems a simple enough fix to land (and thank you for doing this) but marking low enough priority to make it clear we may back out and wait for v1.22 if we don't have bandwidth for this
So while I think we should probably do this anyhow I want to note: This issue lists the reasons: we have no intention to stage e2e (there's been much discussion about this since then) we generally don't want anything non-informational (required jobs) running only sometimes anyhow, it's just not reliable Decoupling these seems like a good idea from a purely code-organization perspective, but the reasons in the previous issue are no longer valid.
/milestone v1.22 The PR didn't land in time. At this point I view this as nice to have. Even if it's low impact, let's wait until next release.

previousResourceVersion := make(map[string]float64) transport := http.DefaultTransport <del> token := tokenID + "." + secret </del> <ins> token := validTokenID + "." + validSecret </ins> var bodyStr string if test.request.body != "" { sub := ""


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --e2e test fails with ECDSA Service Account key e2e test to succeed with ECDSA Service Account key Use ECDSA key for the service account key. Logs from the started test container: library defaults to by default as the only supported method while should be explicitly: Relevant code: On the API server side: Kubernetes version (use ): 1.21.0-rc.0 Cloud provider or hardware configuration: OS (e.g: ): Talos Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
Changing the service account key type to RSA fixes the test:
this is a problem for us, as this test is part of test suite, and it fails with 1.21-rc.0 e2e tests
SIG Auth own their test cases (it's tagged in the test name), SIG Architecture own conformance. Testing owns the frameworks, CI, tooling, etc. EDIT: just clarifying and making sure the right people see it. I raised this to the conformance work group.
/assign /priority critical-urgent /milestone v1.21 because this is in the conformance suite, it must not restrict valid+supported cluster configurations
/reopen the image needs to be promoted and bumped in the e2e test
Reopened this issue. <detailsIn response to : /reopen the image needs to be promoted and bumped in the e2e test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
promote image PR:
I believe with merged this can be closed as all the fixes were done and backported to 1.21
Thanks!

# agnhost: bump this one first - name: "agnhost" <del> version: "2.30" </del> <ins> version: "2.31" </ins> refPaths: - path: test/images/agnhost/VERSION match: d.d


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --e2e test fails with ECDSA Service Account key e2e test to succeed with ECDSA Service Account key Use ECDSA key for the service account key. Logs from the started test container: library defaults to by default as the only supported method while should be explicitly: Relevant code: On the API server side: Kubernetes version (use ): 1.21.0-rc.0 Cloud provider or hardware configuration: OS (e.g: ): Talos Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
Changing the service account key type to RSA fixes the test:
this is a problem for us, as this test is part of test suite, and it fails with 1.21-rc.0 e2e tests
SIG Auth own their test cases (it's tagged in the test name), SIG Architecture own conformance. Testing owns the frameworks, CI, tooling, etc. EDIT: just clarifying and making sure the right people see it. I raised this to the conformance work group.
/assign /priority critical-urgent /milestone v1.21 because this is in the conformance suite, it must not restrict valid+supported cluster configurations
/reopen the image needs to be promoted and bumped in the e2e test
Reopened this issue. <detailsIn response to : /reopen the image needs to be promoted and bumped in the e2e test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
promote image PR:
I believe with merged this can be closed as all the fixes were done and backported to 1.21
Thanks!

<del> 2.30 </del> <ins> 2.31 </ins>


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --e2e test fails with ECDSA Service Account key e2e test to succeed with ECDSA Service Account key Use ECDSA key for the service account key. Logs from the started test container: library defaults to by default as the only supported method while should be explicitly: Relevant code: On the API server side: Kubernetes version (use ): 1.21.0-rc.0 Cloud provider or hardware configuration: OS (e.g: ): Talos Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
Changing the service account key type to RSA fixes the test:
this is a problem for us, as this test is part of test suite, and it fails with 1.21-rc.0 e2e tests
SIG Auth own their test cases (it's tagged in the test name), SIG Architecture own conformance. Testing owns the frameworks, CI, tooling, etc. EDIT: just clarifying and making sure the right people see it. I raised this to the conformance work group.
/assign /priority critical-urgent /milestone v1.21 because this is in the conformance suite, it must not restrict valid+supported cluster configurations
/reopen the image needs to be promoted and bumped in the e2e test
Reopened this issue. <detailsIn response to : /reopen the image needs to be promoted and bumped in the e2e test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
promote image PR:
I believe with merged this can be closed as all the fixes were done and backported to 1.21
Thanks!

func main() { rootCmd := &cobra.Command{ Use:     "app", <del> Version: "2.30", </del> <ins> Version: "2.31", </ins> } rootCmd.AddCommand(auditproxy.CmdAuditProxy)


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --e2e test fails with ECDSA Service Account key e2e test to succeed with ECDSA Service Account key Use ECDSA key for the service account key. Logs from the started test container: library defaults to by default as the only supported method while should be explicitly: Relevant code: On the API server side: Kubernetes version (use ): 1.21.0-rc.0 Cloud provider or hardware configuration: OS (e.g: ): Talos Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
Changing the service account key type to RSA fixes the test:
this is a problem for us, as this test is part of test suite, and it fails with 1.21-rc.0 e2e tests
SIG Auth own their test cases (it's tagged in the test name), SIG Architecture own conformance. Testing owns the frameworks, CI, tooling, etc. EDIT: just clarifying and making sure the right people see it. I raised this to the conformance work group.
/assign /priority critical-urgent /milestone v1.21 because this is in the conformance suite, it must not restrict valid+supported cluster configurations
/reopen the image needs to be promoted and bumped in the e2e test
Reopened this issue. <detailsIn response to : /reopen the image needs to be promoted and bumped in the e2e test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
promote image PR:
I believe with merged this can be closed as all the fixes were done and backported to 1.21
Thanks!

} log.Printf("OK: Constructed OIDC provider for issuer %v", unsafeClaims.Issuer) <del> validTok, err := iss.Verifier(&oidc.Config{ClientID: audience}).Verify(ctx, raw) </del> <ins> validTok, err := iss.Verifier(&oidc.Config{ ClientID:             audience, SupportedSigningAlgs: []string{oidc.RS256, oidc.ES256}, }).Verify(ctx, raw) </ins> if err != nil { log.Fatal(err) }


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-e2e-aks-engine-azure-file pull-kubernetes-e2e-azure-file-windows AzureFile CSI Driver End-to-End Tests: Dynamic Provisioning should create a volume on demand with mount options [] [] [Windows] AzureFile CSI Driver End-to-End Tests: Dynamic Provisioning should create multiple PV objects, bind to PVCs and attach all to different pods on the same node [] [] [Windows] AzureFile CSI Driver End-to-End Tests: Dynamic Provisioning should create a volume on demand and mount it as readOnly in a pod [] [] [Windows] If the randomly generated kubetest resource group has capital letters, the share name will contain invalid characters and fail provisioning. links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

} if shareName == "" { <del> // File share name has a length limit of 63, and it cannot contain two consecutive '-'s. </del> <ins> // File share name has a length limit of 63, it cannot contain two consecutive '-'s, and all letters must be lower case. </ins> name := util.GenerateVolumeName(a.options.ClusterName, a.options.PVName, 63) shareName = strings.Replace(name, "--", "-", -1) <ins> shareName = strings.ToLower(shareName) </ins> } if resourceGroup == "" {


Question:
With recent migration of Kubelet to structured logging introduced e2e tests that enable JSON in Kubelet. Organized migration of other components haven't yet started and is on best effort bases, yet some line were already migrated. As code-base will be slowly migrated, we need to expand the test to avoid errors like I propose to expand existing e2e test suite ( and ) to cover all control plane components (apiserver, controller-manager and scheduler). /cc
/sig instrumentation
/help
If all components have this flag do we just want to enable it on all components? We already increase the log level on all components in CI, it should be pretty straightforward to expand this to all: pick a new environment variable name for the script with a more appropriate name for all components instead of just kubelet set it on the job in addition to the old kubelet one, by sending a PR to test-infra send a PR to the kind script to set the value in all places where we set verbosity instead of just kubelet, switching to the new variable name
/triage accepted
/area logging
So currently the e2e tests do not actually read and test what the logs return the current job just makes sure that kubelet is running Would we want to write similar tests for the different control planes while we start migration or do you have something else in mind with respect to the automated testing ?
Yes, currently we just test if Kubelet is running to test that flag is working and if enabling json doesn't cause panics. This exact issue proposes to expand this to other control plane, but your idea would be a great follow up. Would be great to propose a test plan that will specify what we want to test and how. If you have ideas some ideas could you collect hem in some issue so we can discuss.
There's no portable way to get these logs. There are non portable ways but those cannot be used in a standard e2e test.
Oh so for the standard e2e tests it might be good enough to just test the startups of these control planes with the logging enabled.
I was thinking that we could get started with maybe writing similar to one for the remaining components and then maybe find a way of checking the logs generated as we have bigger discussions a) kube-apiserver --Completed b) etcd --- TODO c) kube-scheduler ---TODO d) kube-controller-manager --TODO
sound good. FYI Etcd is not a kubernetes component as so it's out of scope of this issue.
/wg structured-logging As getting core components logs is not portable, we could test a container with same logging library and using Kubenetes logs API to access logs. Created to track it
it still might not be a terrible idea to write a simple test script to probe the exported logs from a cluster with json logging enabled for any non-json lines once we know that everything should be. I think this could probably be hacked up with jq or something, but I wouldn't position it like a normal e2e test.
Could you give a hint how to implement and run such script?
/priority important-soon This issue is based on feedback from Kubelet migration retrospection, it is required before organizing migration of next component. /cc
Would this follow the same flow as in where we check if json logging is enabled and then run jq separately or within the same flow. To be more specific: Seems to be getting the data and validating it so in that case would we be running after grabbing the logs ?
For now let's focus on the original tasks of having ( and ) test scenarios, also configure other components. We just need to update Kind script to also pass to other components. Example:
Would like to take this one! /cc
Great, let me know if you need any help? /remove-help
just created a PR:
Components to be picked: [x] kube-scheduler [ ] kube-apiserver [ ] controller-manager
Attempted to cover the remaining components in this
It would be great if we had e2e setup for all components before we start structured migration next week. will you be able to work on it this week? Please let me know if there is anything I can help with.
Thanks for help with control plane components, for 1.23 we are planning to migrate kube-proxy to structured logging. Could you also try to enable JSON logging on kube-proxy?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Confirmed that in last CI run Logs from kubelet and control plane components are written in JSON: Great Job Could you take a look into changing log format of kube-proxy?
Filed for kube-proxy. Are we good to close this issue ?
sg

export PROJECT="k8s-jkns-e2e-gke-prod" export CLOUDSDK_BUCKET="gs://cloud-sdk-testing/rc" export JENKINS_USE_SERVER_VERSION="y" <ins> export CLOUDSDK_API_ENDPOINT_OVERRIDES_CONTAINER="https://container.googleapis.com/" </ins> export ZONE="asia-east1-b" - 'gke-prod-parallel': description: 'Run E2E tests on GKE prod endpoint in parallel.'


Question:
With recent migration of Kubelet to structured logging introduced e2e tests that enable JSON in Kubelet. Organized migration of other components haven't yet started and is on best effort bases, yet some line were already migrated. As code-base will be slowly migrated, we need to expand the test to avoid errors like I propose to expand existing e2e test suite ( and ) to cover all control plane components (apiserver, controller-manager and scheduler). /cc
/sig instrumentation
/help
If all components have this flag do we just want to enable it on all components? We already increase the log level on all components in CI, it should be pretty straightforward to expand this to all: pick a new environment variable name for the script with a more appropriate name for all components instead of just kubelet set it on the job in addition to the old kubelet one, by sending a PR to test-infra send a PR to the kind script to set the value in all places where we set verbosity instead of just kubelet, switching to the new variable name
/triage accepted
/area logging
So currently the e2e tests do not actually read and test what the logs return the current job just makes sure that kubelet is running Would we want to write similar tests for the different control planes while we start migration or do you have something else in mind with respect to the automated testing ?
Yes, currently we just test if Kubelet is running to test that flag is working and if enabling json doesn't cause panics. This exact issue proposes to expand this to other control plane, but your idea would be a great follow up. Would be great to propose a test plan that will specify what we want to test and how. If you have ideas some ideas could you collect hem in some issue so we can discuss.
There's no portable way to get these logs. There are non portable ways but those cannot be used in a standard e2e test.
Oh so for the standard e2e tests it might be good enough to just test the startups of these control planes with the logging enabled.
I was thinking that we could get started with maybe writing similar to one for the remaining components and then maybe find a way of checking the logs generated as we have bigger discussions a) kube-apiserver --Completed b) etcd --- TODO c) kube-scheduler ---TODO d) kube-controller-manager --TODO
sound good. FYI Etcd is not a kubernetes component as so it's out of scope of this issue.
/wg structured-logging As getting core components logs is not portable, we could test a container with same logging library and using Kubenetes logs API to access logs. Created to track it
it still might not be a terrible idea to write a simple test script to probe the exported logs from a cluster with json logging enabled for any non-json lines once we know that everything should be. I think this could probably be hacked up with jq or something, but I wouldn't position it like a normal e2e test.
Could you give a hint how to implement and run such script?
/priority important-soon This issue is based on feedback from Kubelet migration retrospection, it is required before organizing migration of next component. /cc
Would this follow the same flow as in where we check if json logging is enabled and then run jq separately or within the same flow. To be more specific: Seems to be getting the data and validating it so in that case would we be running after grabbing the logs ?
For now let's focus on the original tasks of having ( and ) test scenarios, also configure other components. We just need to update Kind script to also pass to other components. Example:
Would like to take this one! /cc
Great, let me know if you need any help? /remove-help
just created a PR:
Components to be picked: [x] kube-scheduler [ ] kube-apiserver [ ] controller-manager
Attempted to cover the remaining components in this
It would be great if we had e2e setup for all components before we start structured migration next week. will you be able to work on it this week? Please let me know if there is anything I can help with.
Thanks for help with control plane components, for 1.23 we are planning to migrate kube-proxy to structured logging. Could you also try to enable JSON logging on kube-proxy?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Confirmed that in last CI run Logs from kubelet and control plane components are written in JSON: Great Job Could you take a look into changing log format of kube-proxy?
Filed for kube-proxy. Are we good to close this issue ?
sg

export PROJECT="k8s-e2e-gke-prod-parallel" export CLOUDSDK_BUCKET="gs://cloud-sdk-testing/rc" export JENKINS_USE_SERVER_VERSION="y" <ins> export CLOUDSDK_API_ENDPOINT_OVERRIDES_CONTAINER="https://container.googleapis.com/" </ins> export GINKGO_TEST_ARGS="--ginkgo.skip=[Slow]|[Serial]|[Disruptive]|[Flaky]|[Feature:.+]" export GINKGO_PARALLEL="y" export ZONE="asia-east1-b"


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --pull-kubernetes-integration runkubectldiff_tests It only shows failure cases in recent 2 days. The last date is I tried to reproduce it locally, but can't
/sig testing
/sig cli
/cc
I’ve only briefly taken a look at this, but my understanding is that server-side diff is flaking because the timestamp is different (by like a second) between the one already applied and the diff apply. do you think this could be related to this recent change Could delaying when we update the timestamp cause it to sometimes get misaligned with what’s being diffed? I’m not super familiar with diff, but it feels like it should ignore differences between timestamps that get updated on each call, right? Is there some sort of mechanism we have in that knows to ignore timestamps that just isn’t being done for fields?
This error probably shows that we have a path that causes the object to be re-written to etcd when it shouldn't.
As we can see, the diff only shows a difference in the timestamp but not in the fields that are managed. We are not supposed to update the timestamp if the managed fields haven't changed. That timestamp only gets updated if the two steps happen in different seconds (that's why it's flaky). That is definitely not a flake but an actual failure.
Failed 81/236 times today in /priority critical-urgent I think, correct it if I am wrong.
/triage accepted
correct. who should be assigned as the owner of this?
I can take it, I'm looking into it now
/assign

// newDefaultFieldManager is a helper function which wraps a Manager with certain default logic. func newDefaultFieldManager(f Manager, typeConverter TypeConverter, objectConverter runtime.ObjectConvertor, objectCreater runtime.ObjectCreater, kind schema.GroupVersionKind, ignoreManagedFieldsFromRequestObject bool) *FieldManager { <del> f = NewManagedFieldsUpdater(f) f = NewStripMetaManager(f) f = NewBuildManagerInfoManager(f, kind.GroupVersion()) f = NewCapManagersManager(f, DefaultMaxUpdateManagers) f = NewProbabilisticSkipNonAppliedManager(f, objectCreater, kind, DefaultTrackOnCreateProbability) f = NewLastAppliedManager(f, typeConverter, objectConverter, kind.GroupVersion()) f = NewLastAppliedUpdater(f) return NewFieldManager(f, ignoreManagedFieldsFromRequestObject) </del> <ins> return NewFieldManager( NewLastAppliedUpdater( NewLastAppliedManager( NewProbabilisticSkipNonAppliedManager( NewCapManagersManager( NewBuildManagerInfoManager( NewManagedFieldsUpdater( NewStripMetaManager(f), ), kind.GroupVersion(), ), DefaultMaxUpdateManagers, ), objectCreater, kind, DefaultTrackOnCreateProbability, ), typeConverter, objectConverter, kind.GroupVersion()), ), ignoreManagedFieldsFromRequestObject, ) </ins> } // DecodeManagedFields converts ManagedFields from the wire format (api format)


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --pull-kubernetes-integration runkubectldiff_tests It only shows failure cases in recent 2 days. The last date is I tried to reproduce it locally, but can't
/sig testing
/sig cli
/cc
I’ve only briefly taken a look at this, but my understanding is that server-side diff is flaking because the timestamp is different (by like a second) between the one already applied and the diff apply. do you think this could be related to this recent change Could delaying when we update the timestamp cause it to sometimes get misaligned with what’s being diffed? I’m not super familiar with diff, but it feels like it should ignore differences between timestamps that get updated on each call, right? Is there some sort of mechanism we have in that knows to ignore timestamps that just isn’t being done for fields?
This error probably shows that we have a path that causes the object to be re-written to etcd when it shouldn't.
As we can see, the diff only shows a difference in the timestamp but not in the fields that are managed. We are not supposed to update the timestamp if the managed fields haven't changed. That timestamp only gets updated if the two steps happen in different seconds (that's why it's flaky). That is definitely not a flake but an actual failure.
Failed 81/236 times today in /priority critical-urgent I think, correct it if I am wrong.
/triage accepted
correct. who should be assigned as the owner of this?
I can take it, I'm looking into it now
/assign

live := &unstructured.Unstructured{} live.SetKind(gvk.Kind) live.SetAPIVersion(gvk.GroupVersion().String()) <del> f = NewStripMetaManager(f) f = NewManagedFieldsUpdater(f) f = NewBuildManagerInfoManager(f, gvk.GroupVersion()) f = NewProbabilisticSkipNonAppliedManager(f, &fakeObjectCreater{gvk: gvk}, gvk, DefaultTrackOnCreateProbability) f = NewLastAppliedManager(f, typeConverter, objectConverter, gvk.GroupVersion()) f = NewLastAppliedUpdater(f) </del> <ins> f = NewLastAppliedUpdater( NewLastAppliedManager( NewProbabilisticSkipNonAppliedManager( NewBuildManagerInfoManager( NewManagedFieldsUpdater( NewStripMetaManager(f), ), gvk.GroupVersion(), ), &fakeObjectCreater{gvk: gvk}, gvk, DefaultTrackOnCreateProbability, ), typeConverter, objectConverter, gvk.GroupVersion(), ), ) </ins> if chainFieldManager != nil { f = chainFieldManager(f) }


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --k8s-e2e-ltsc2019-containerd-flannel-sdnoverlay-master [] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestart http hook properly [NodeConformance] [Conformance] [] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance] Note: the tests have been excluded until they are fixed. In the case of multinode clusters, the http server pod and the test cluster can spawn on different nodes, which can be problematic for poststart / prestop hooks, as they are executed by the kubelet itself, and the cross-node lifecycle hook might fail (according to the [Kubernetes network model], it is not mandatory for kubelet to be able to access pods on a different node). However, that is not always the case, different Networking setups allow cross-node communication between kubelet and pods, which means that the above tests are passing properly (for example: ), which begs the question if this behaviour should be universal or not, especially since the tests are marked as , which would imply that it should work in all cases. [Kubernetes network model]: links to appreciated links to specific failures in spyglass appreciated Slack: /sig testing /sig networking /sig windows <!-- Please see the deflaking doc () for more guidance! --
/sig network /sig node /area conformance /assign /priority critical-urgent /cc This is important since affects conformance and defines the networking behaviour for the poststart/ prestop hooks
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /sig node /sig conformance /assign /priority critical-urgent /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i'd personally let folks have first dibs on what we need to do here. Though from the name of the test itself, it seems like the test is NOT a sig-network test :)
Indeed, it does not have any tag in its name, but the tests rely on networking and on how CNIs work. As I've stated above, it's not a problem for flannel sdnbridge, but it is for flannel sdnoverlay; other CNIs might have a similar issue too.
because the network is not a thing :) let´s tag the sig owner then /sig node I think that the key here is to decide if kubelet should be able to access pod in other nodes, and based on the link Claudiu is part of the Kubernetes networking model. I saw users that doesn't want kubelet to access even the pods on same host, so I think that is legit to say kubelet MAY reach pods in other nodes but not MUST ... but I'm not familiar with the to fully understand the context. If kubelet "may" reach pods in the other nodes, the test must only test lifecycle hooks for pods in the same node as the kubelet under test, right?
This is correct. The model also says: "For those platforms that support Pods running in the host network (e.g. Linux): pods in the host network of a node can communicate with all pods on all nodes without NAT" I believe that Windows host network is not supported, right?
Right, and there are specific tests that checks that exact scenario you've mentioned: However, the flaky tests I've mentioned in the issue require kubelet to be able to dial-in into a webserver pod for a different pod's poststart / prestop HTTP lifecycle hooks. There are couple of things regarding this: of all, in order to ensure kubelet would be able to run the HTTP lifecycle hooks, we'd have to make sure both pods are on the same node, as that is a supported scenario, according to the Kubernetes Network Model. I have sent a PR for this above. Alternatively, a Cluster IP could probably work as well, but IMO, ensuring that both pods end up on the same node is simpler and easier to clean up. , it's interesting that those HTTP lifecycle hook tests work on different networking setups, without any pods having HostNetwork enabled, even though you've mentioned they should (e.g.: Linux / Windows with flannel l2bridge, and even Linux with flannel overlay). Now, my question is: is this behaviour a feature, or a bug, especially since the tests are marked as `?
oh, I missed that part, so in Linux kubelet must be able to communicate with pods on all nodes (becuase it is running in the hostnetwork) If that is correct, it seems that one solution is to keep the test for Linux and tag it as LinuxOnly, and a new test for WindowsOnly that only spawn the pods under test in the same node EDIT Discussed in sig-net meeting , no clear answer... can be kubelet considered as a hostnetwork pod?
For the purposes of testing, I don't mind forcing the pods onto the same node., but let's document the heck out of that at the code-site?
correct but it will be supported once HostProcess containers are merged in 1.22: Only in Overlay networks (ie Flannel where these tests are flaking) does the pod have to be on the same node. It doesn't require host network mode and these tests are . I would be more inclined to disable this test for Windows Overlay test setups and have a sig-windows test for the overlay functionality. I would also question if the lifecycle hooks only running on the same node is useful and if it should just be documented that it doesn't work in the Windows Overlay setup.
is looking at test PRs from SIG Node. I will let sig-net triage this issue
Next steps on this?
/triage accepted
We still haven't merged this: I also had this, but it has been decided to be blocked:
requires conformance approval, at this point I think that has to be postponed to 1.23 right, since that behavior is not required, we shouldn't test it, because is legit to fail and having a test for something that can fail can create confusion :smile:

hook should execute poststart http hook properly [NodeConformance] [Conformance]' description: When a post start handler is specified in the container lifecycle using a HttpGet action, then the handler MUST be invoked after the start of the container. <del> A server pod is created that will serve http requests, create a second pod with a container lifecycle specifying a post start that invokes the server pod to validate that the post start is executed. </del> <ins> A server pod is created that will serve http requests, create a second pod on the same node with a container lifecycle specifying a post start that invokes the server pod to validate that the post start is executed. </ins> release: v1.9 file: test/e2e/common/node/lifecycle_hook.go - testname: Pod Lifecycle, prestop exec hook


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --k8s-e2e-ltsc2019-containerd-flannel-sdnoverlay-master [] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestart http hook properly [NodeConformance] [Conformance] [] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance] Note: the tests have been excluded until they are fixed. In the case of multinode clusters, the http server pod and the test cluster can spawn on different nodes, which can be problematic for poststart / prestop hooks, as they are executed by the kubelet itself, and the cross-node lifecycle hook might fail (according to the [Kubernetes network model], it is not mandatory for kubelet to be able to access pods on a different node). However, that is not always the case, different Networking setups allow cross-node communication between kubelet and pods, which means that the above tests are passing properly (for example: ), which begs the question if this behaviour should be universal or not, especially since the tests are marked as , which would imply that it should work in all cases. [Kubernetes network model]: links to appreciated links to specific failures in spyglass appreciated Slack: /sig testing /sig networking /sig windows <!-- Please see the deflaking doc () for more guidance! --
/sig network /sig node /area conformance /assign /priority critical-urgent /cc This is important since affects conformance and defines the networking behaviour for the poststart/ prestop hooks
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /sig node /sig conformance /assign /priority critical-urgent /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i'd personally let folks have first dibs on what we need to do here. Though from the name of the test itself, it seems like the test is NOT a sig-network test :)
Indeed, it does not have any tag in its name, but the tests rely on networking and on how CNIs work. As I've stated above, it's not a problem for flannel sdnbridge, but it is for flannel sdnoverlay; other CNIs might have a similar issue too.
because the network is not a thing :) let´s tag the sig owner then /sig node I think that the key here is to decide if kubelet should be able to access pod in other nodes, and based on the link Claudiu is part of the Kubernetes networking model. I saw users that doesn't want kubelet to access even the pods on same host, so I think that is legit to say kubelet MAY reach pods in other nodes but not MUST ... but I'm not familiar with the to fully understand the context. If kubelet "may" reach pods in the other nodes, the test must only test lifecycle hooks for pods in the same node as the kubelet under test, right?
This is correct. The model also says: "For those platforms that support Pods running in the host network (e.g. Linux): pods in the host network of a node can communicate with all pods on all nodes without NAT" I believe that Windows host network is not supported, right?
Right, and there are specific tests that checks that exact scenario you've mentioned: However, the flaky tests I've mentioned in the issue require kubelet to be able to dial-in into a webserver pod for a different pod's poststart / prestop HTTP lifecycle hooks. There are couple of things regarding this: of all, in order to ensure kubelet would be able to run the HTTP lifecycle hooks, we'd have to make sure both pods are on the same node, as that is a supported scenario, according to the Kubernetes Network Model. I have sent a PR for this above. Alternatively, a Cluster IP could probably work as well, but IMO, ensuring that both pods end up on the same node is simpler and easier to clean up. , it's interesting that those HTTP lifecycle hook tests work on different networking setups, without any pods having HostNetwork enabled, even though you've mentioned they should (e.g.: Linux / Windows with flannel l2bridge, and even Linux with flannel overlay). Now, my question is: is this behaviour a feature, or a bug, especially since the tests are marked as `?
oh, I missed that part, so in Linux kubelet must be able to communicate with pods on all nodes (becuase it is running in the hostnetwork) If that is correct, it seems that one solution is to keep the test for Linux and tag it as LinuxOnly, and a new test for WindowsOnly that only spawn the pods under test in the same node EDIT Discussed in sig-net meeting , no clear answer... can be kubelet considered as a hostnetwork pod?
For the purposes of testing, I don't mind forcing the pods onto the same node., but let's document the heck out of that at the code-site?
correct but it will be supported once HostProcess containers are merged in 1.22: Only in Overlay networks (ie Flannel where these tests are flaking) does the pod have to be on the same node. It doesn't require host network mode and these tests are . I would be more inclined to disable this test for Windows Overlay test setups and have a sig-windows test for the overlay functionality. I would also question if the lifecycle hooks only running on the same node is useful and if it should just be documented that it doesn't work in the Windows Overlay setup.
is looking at test PRs from SIG Node. I will let sig-net triage this issue
Next steps on this?
/triage accepted
We still haven't merged this: I also had this, but it has been decided to be blocked:
requires conformance approval, at this point I think that has to be postponed to 1.23 right, since that behavior is not required, we shouldn't test it, because is legit to fail and having a test for something that can fail can create confusion :smile:

hook should execute prestop http hook properly [NodeConformance] [Conformance]' description: When a pre-stop handler is specified in the container lifecycle using a 'HttpGet' action, then the handler MUST be invoked before the container is terminated. <del> A server pod is created that will serve http requests, create a second pod with a container lifecycle specifying a pre-stop that invokes the server pod to validate that the pre-stop is executed. </del> <ins> A server pod is created that will serve http requests, create a second pod on the same node with a container lifecycle specifying a pre-stop that invokes the server pod to validate that the pre-stop is executed. </ins> release: v1.9 file: test/e2e/common/node/lifecycle_hook.go - testname: Container Runtime, TerminationMessage, from log output of succeeding container


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --k8s-e2e-ltsc2019-containerd-flannel-sdnoverlay-master [] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestart http hook properly [NodeConformance] [Conformance] [] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance] Note: the tests have been excluded until they are fixed. In the case of multinode clusters, the http server pod and the test cluster can spawn on different nodes, which can be problematic for poststart / prestop hooks, as they are executed by the kubelet itself, and the cross-node lifecycle hook might fail (according to the [Kubernetes network model], it is not mandatory for kubelet to be able to access pods on a different node). However, that is not always the case, different Networking setups allow cross-node communication between kubelet and pods, which means that the above tests are passing properly (for example: ), which begs the question if this behaviour should be universal or not, especially since the tests are marked as , which would imply that it should work in all cases. [Kubernetes network model]: links to appreciated links to specific failures in spyglass appreciated Slack: /sig testing /sig networking /sig windows <!-- Please see the deflaking doc () for more guidance! --
/sig network /sig node /area conformance /assign /priority critical-urgent /cc This is important since affects conformance and defines the networking behaviour for the poststart/ prestop hooks
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /sig node /sig conformance /assign /priority critical-urgent /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i'd personally let folks have first dibs on what we need to do here. Though from the name of the test itself, it seems like the test is NOT a sig-network test :)
Indeed, it does not have any tag in its name, but the tests rely on networking and on how CNIs work. As I've stated above, it's not a problem for flannel sdnbridge, but it is for flannel sdnoverlay; other CNIs might have a similar issue too.
because the network is not a thing :) let´s tag the sig owner then /sig node I think that the key here is to decide if kubelet should be able to access pod in other nodes, and based on the link Claudiu is part of the Kubernetes networking model. I saw users that doesn't want kubelet to access even the pods on same host, so I think that is legit to say kubelet MAY reach pods in other nodes but not MUST ... but I'm not familiar with the to fully understand the context. If kubelet "may" reach pods in the other nodes, the test must only test lifecycle hooks for pods in the same node as the kubelet under test, right?
This is correct. The model also says: "For those platforms that support Pods running in the host network (e.g. Linux): pods in the host network of a node can communicate with all pods on all nodes without NAT" I believe that Windows host network is not supported, right?
Right, and there are specific tests that checks that exact scenario you've mentioned: However, the flaky tests I've mentioned in the issue require kubelet to be able to dial-in into a webserver pod for a different pod's poststart / prestop HTTP lifecycle hooks. There are couple of things regarding this: of all, in order to ensure kubelet would be able to run the HTTP lifecycle hooks, we'd have to make sure both pods are on the same node, as that is a supported scenario, according to the Kubernetes Network Model. I have sent a PR for this above. Alternatively, a Cluster IP could probably work as well, but IMO, ensuring that both pods end up on the same node is simpler and easier to clean up. , it's interesting that those HTTP lifecycle hook tests work on different networking setups, without any pods having HostNetwork enabled, even though you've mentioned they should (e.g.: Linux / Windows with flannel l2bridge, and even Linux with flannel overlay). Now, my question is: is this behaviour a feature, or a bug, especially since the tests are marked as `?
oh, I missed that part, so in Linux kubelet must be able to communicate with pods on all nodes (becuase it is running in the hostnetwork) If that is correct, it seems that one solution is to keep the test for Linux and tag it as LinuxOnly, and a new test for WindowsOnly that only spawn the pods under test in the same node EDIT Discussed in sig-net meeting , no clear answer... can be kubelet considered as a hostnetwork pod?
For the purposes of testing, I don't mind forcing the pods onto the same node., but let's document the heck out of that at the code-site?
correct but it will be supported once HostProcess containers are merged in 1.22: Only in Overlay networks (ie Flannel where these tests are flaking) does the pod have to be on the same node. It doesn't require host network mode and these tests are . I would be more inclined to disable this test for Windows Overlay test setups and have a sig-windows test for the overlay functionality. I would also question if the lifecycle hooks only running on the same node is useful and if it should just be documented that it doesn't work in the Windows Overlay setup.
is looking at test PRs from SIG Node. I will let sig-net triage this issue
Next steps on this?
/triage accepted
We still haven't merged this: I also had this, but it has been decided to be blocked:
requires conformance approval, at this point I think that has to be postponed to 1.23 right, since that behavior is not required, we shouldn't test it, because is legit to fail and having a test for something that can fail can create confusion :smile:

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/kubernetes/test/e2e/framework" <ins> e2enode "k8s.io/kubernetes/test/e2e/framework/node" </ins> e2epod "k8s.io/kubernetes/test/e2e/framework/pod" imageutils "k8s.io/kubernetes/test/utils/image"


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --k8s-e2e-ltsc2019-containerd-flannel-sdnoverlay-master [] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestart http hook properly [NodeConformance] [Conformance] [] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance] Note: the tests have been excluded until they are fixed. In the case of multinode clusters, the http server pod and the test cluster can spawn on different nodes, which can be problematic for poststart / prestop hooks, as they are executed by the kubelet itself, and the cross-node lifecycle hook might fail (according to the [Kubernetes network model], it is not mandatory for kubelet to be able to access pods on a different node). However, that is not always the case, different Networking setups allow cross-node communication between kubelet and pods, which means that the above tests are passing properly (for example: ), which begs the question if this behaviour should be universal or not, especially since the tests are marked as , which would imply that it should work in all cases. [Kubernetes network model]: links to appreciated links to specific failures in spyglass appreciated Slack: /sig testing /sig networking /sig windows <!-- Please see the deflaking doc () for more guidance! --
/sig network /sig node /area conformance /assign /priority critical-urgent /cc This is important since affects conformance and defines the networking behaviour for the poststart/ prestop hooks
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /sig node /sig conformance /assign /priority critical-urgent /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i'd personally let folks have first dibs on what we need to do here. Though from the name of the test itself, it seems like the test is NOT a sig-network test :)
Indeed, it does not have any tag in its name, but the tests rely on networking and on how CNIs work. As I've stated above, it's not a problem for flannel sdnbridge, but it is for flannel sdnoverlay; other CNIs might have a similar issue too.
because the network is not a thing :) let´s tag the sig owner then /sig node I think that the key here is to decide if kubelet should be able to access pod in other nodes, and based on the link Claudiu is part of the Kubernetes networking model. I saw users that doesn't want kubelet to access even the pods on same host, so I think that is legit to say kubelet MAY reach pods in other nodes but not MUST ... but I'm not familiar with the to fully understand the context. If kubelet "may" reach pods in the other nodes, the test must only test lifecycle hooks for pods in the same node as the kubelet under test, right?
This is correct. The model also says: "For those platforms that support Pods running in the host network (e.g. Linux): pods in the host network of a node can communicate with all pods on all nodes without NAT" I believe that Windows host network is not supported, right?
Right, and there are specific tests that checks that exact scenario you've mentioned: However, the flaky tests I've mentioned in the issue require kubelet to be able to dial-in into a webserver pod for a different pod's poststart / prestop HTTP lifecycle hooks. There are couple of things regarding this: of all, in order to ensure kubelet would be able to run the HTTP lifecycle hooks, we'd have to make sure both pods are on the same node, as that is a supported scenario, according to the Kubernetes Network Model. I have sent a PR for this above. Alternatively, a Cluster IP could probably work as well, but IMO, ensuring that both pods end up on the same node is simpler and easier to clean up. , it's interesting that those HTTP lifecycle hook tests work on different networking setups, without any pods having HostNetwork enabled, even though you've mentioned they should (e.g.: Linux / Windows with flannel l2bridge, and even Linux with flannel overlay). Now, my question is: is this behaviour a feature, or a bug, especially since the tests are marked as `?
oh, I missed that part, so in Linux kubelet must be able to communicate with pods on all nodes (becuase it is running in the hostnetwork) If that is correct, it seems that one solution is to keep the test for Linux and tag it as LinuxOnly, and a new test for WindowsOnly that only spawn the pods under test in the same node EDIT Discussed in sig-net meeting , no clear answer... can be kubelet considered as a hostnetwork pod?
For the purposes of testing, I don't mind forcing the pods onto the same node., but let's document the heck out of that at the code-site?
correct but it will be supported once HostProcess containers are merged in 1.22: Only in Overlay networks (ie Flannel where these tests are flaking) does the pod have to be on the same node. It doesn't require host network mode and these tests are . I would be more inclined to disable this test for Windows Overlay test setups and have a sig-windows test for the overlay functionality. I would also question if the lifecycle hooks only running on the same node is useful and if it should just be documented that it doesn't work in the Windows Overlay setup.
is looking at test PRs from SIG Node. I will let sig-net triage this issue
Next steps on this?
/triage accepted
We still haven't merged this: I also had this, but it has been decided to be blocked:
requires conformance approval, at this point I think that has to be postponed to 1.23 right, since that behavior is not required, we shouldn't test it, because is legit to fail and having a test for something that can fail can create confusion :smile:

preStopWaitTimeout   = 30 * time.Second ) ginkgo.Context("when create a pod with lifecycle hook", func() { <del> var targetIP, targetURL string </del> <ins> var targetIP, targetURL, targetNode string </ins> ports := []v1.ContainerPort{ { ContainerPort: 8080,


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --k8s-e2e-ltsc2019-containerd-flannel-sdnoverlay-master [] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestart http hook properly [NodeConformance] [Conformance] [] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance] Note: the tests have been excluded until they are fixed. In the case of multinode clusters, the http server pod and the test cluster can spawn on different nodes, which can be problematic for poststart / prestop hooks, as they are executed by the kubelet itself, and the cross-node lifecycle hook might fail (according to the [Kubernetes network model], it is not mandatory for kubelet to be able to access pods on a different node). However, that is not always the case, different Networking setups allow cross-node communication between kubelet and pods, which means that the above tests are passing properly (for example: ), which begs the question if this behaviour should be universal or not, especially since the tests are marked as , which would imply that it should work in all cases. [Kubernetes network model]: links to appreciated links to specific failures in spyglass appreciated Slack: /sig testing /sig networking /sig windows <!-- Please see the deflaking doc () for more guidance! --
/sig network /sig node /area conformance /assign /priority critical-urgent /cc This is important since affects conformance and defines the networking behaviour for the poststart/ prestop hooks
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /sig node /sig conformance /assign /priority critical-urgent /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i'd personally let folks have first dibs on what we need to do here. Though from the name of the test itself, it seems like the test is NOT a sig-network test :)
Indeed, it does not have any tag in its name, but the tests rely on networking and on how CNIs work. As I've stated above, it's not a problem for flannel sdnbridge, but it is for flannel sdnoverlay; other CNIs might have a similar issue too.
because the network is not a thing :) let´s tag the sig owner then /sig node I think that the key here is to decide if kubelet should be able to access pod in other nodes, and based on the link Claudiu is part of the Kubernetes networking model. I saw users that doesn't want kubelet to access even the pods on same host, so I think that is legit to say kubelet MAY reach pods in other nodes but not MUST ... but I'm not familiar with the to fully understand the context. If kubelet "may" reach pods in the other nodes, the test must only test lifecycle hooks for pods in the same node as the kubelet under test, right?
This is correct. The model also says: "For those platforms that support Pods running in the host network (e.g. Linux): pods in the host network of a node can communicate with all pods on all nodes without NAT" I believe that Windows host network is not supported, right?
Right, and there are specific tests that checks that exact scenario you've mentioned: However, the flaky tests I've mentioned in the issue require kubelet to be able to dial-in into a webserver pod for a different pod's poststart / prestop HTTP lifecycle hooks. There are couple of things regarding this: of all, in order to ensure kubelet would be able to run the HTTP lifecycle hooks, we'd have to make sure both pods are on the same node, as that is a supported scenario, according to the Kubernetes Network Model. I have sent a PR for this above. Alternatively, a Cluster IP could probably work as well, but IMO, ensuring that both pods end up on the same node is simpler and easier to clean up. , it's interesting that those HTTP lifecycle hook tests work on different networking setups, without any pods having HostNetwork enabled, even though you've mentioned they should (e.g.: Linux / Windows with flannel l2bridge, and even Linux with flannel overlay). Now, my question is: is this behaviour a feature, or a bug, especially since the tests are marked as `?
oh, I missed that part, so in Linux kubelet must be able to communicate with pods on all nodes (becuase it is running in the hostnetwork) If that is correct, it seems that one solution is to keep the test for Linux and tag it as LinuxOnly, and a new test for WindowsOnly that only spawn the pods under test in the same node EDIT Discussed in sig-net meeting , no clear answer... can be kubelet considered as a hostnetwork pod?
For the purposes of testing, I don't mind forcing the pods onto the same node., but let's document the heck out of that at the code-site?
correct but it will be supported once HostProcess containers are merged in 1.22: Only in Overlay networks (ie Flannel where these tests are flaking) does the pod have to be on the same node. It doesn't require host network mode and these tests are . I would be more inclined to disable this test for Windows Overlay test setups and have a sig-windows test for the overlay functionality. I would also question if the lifecycle hooks only running on the same node is useful and if it should just be documented that it doesn't work in the Windows Overlay setup.
is looking at test PRs from SIG Node. I will let sig-net triage this issue
Next steps on this?
/triage accepted
We still haven't merged this: I also had this, but it has been decided to be blocked:
requires conformance approval, at this point I think that has to be postponed to 1.23 right, since that behavior is not required, we shouldn't test it, because is legit to fail and having a test for something that can fail can create confusion :smile:

} podHandleHookRequest := e2epod.NewAgnhostPod("", "pod-handle-http-request", nil, nil, ports, "netexec") ginkgo.BeforeEach(func() { <ins> node, err := e2enode.GetRandomReadySchedulableNode(f.ClientSet) framework.ExpectNoError(err) targetNode = node.Name nodeSelection := e2epod.NodeSelection{} e2epod.SetAffinity(&nodeSelection, targetNode) e2epod.SetNodeSelection(&podHandleHookRequest.Spec, nodeSelection) </ins> podClient = f.PodClient() ginkgo.By("create the container to handle the HTTPGet hook request.") newPod := podClient.CreateSync(podHandleHookRequest)


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --k8s-e2e-ltsc2019-containerd-flannel-sdnoverlay-master [] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestart http hook properly [NodeConformance] [Conformance] [] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance] Note: the tests have been excluded until they are fixed. In the case of multinode clusters, the http server pod and the test cluster can spawn on different nodes, which can be problematic for poststart / prestop hooks, as they are executed by the kubelet itself, and the cross-node lifecycle hook might fail (according to the [Kubernetes network model], it is not mandatory for kubelet to be able to access pods on a different node). However, that is not always the case, different Networking setups allow cross-node communication between kubelet and pods, which means that the above tests are passing properly (for example: ), which begs the question if this behaviour should be universal or not, especially since the tests are marked as , which would imply that it should work in all cases. [Kubernetes network model]: links to appreciated links to specific failures in spyglass appreciated Slack: /sig testing /sig networking /sig windows <!-- Please see the deflaking doc () for more guidance! --
/sig network /sig node /area conformance /assign /priority critical-urgent /cc This is important since affects conformance and defines the networking behaviour for the poststart/ prestop hooks
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /sig node /sig conformance /assign /priority critical-urgent /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i'd personally let folks have first dibs on what we need to do here. Though from the name of the test itself, it seems like the test is NOT a sig-network test :)
Indeed, it does not have any tag in its name, but the tests rely on networking and on how CNIs work. As I've stated above, it's not a problem for flannel sdnbridge, but it is for flannel sdnoverlay; other CNIs might have a similar issue too.
because the network is not a thing :) let´s tag the sig owner then /sig node I think that the key here is to decide if kubelet should be able to access pod in other nodes, and based on the link Claudiu is part of the Kubernetes networking model. I saw users that doesn't want kubelet to access even the pods on same host, so I think that is legit to say kubelet MAY reach pods in other nodes but not MUST ... but I'm not familiar with the to fully understand the context. If kubelet "may" reach pods in the other nodes, the test must only test lifecycle hooks for pods in the same node as the kubelet under test, right?
This is correct. The model also says: "For those platforms that support Pods running in the host network (e.g. Linux): pods in the host network of a node can communicate with all pods on all nodes without NAT" I believe that Windows host network is not supported, right?
Right, and there are specific tests that checks that exact scenario you've mentioned: However, the flaky tests I've mentioned in the issue require kubelet to be able to dial-in into a webserver pod for a different pod's poststart / prestop HTTP lifecycle hooks. There are couple of things regarding this: of all, in order to ensure kubelet would be able to run the HTTP lifecycle hooks, we'd have to make sure both pods are on the same node, as that is a supported scenario, according to the Kubernetes Network Model. I have sent a PR for this above. Alternatively, a Cluster IP could probably work as well, but IMO, ensuring that both pods end up on the same node is simpler and easier to clean up. , it's interesting that those HTTP lifecycle hook tests work on different networking setups, without any pods having HostNetwork enabled, even though you've mentioned they should (e.g.: Linux / Windows with flannel l2bridge, and even Linux with flannel overlay). Now, my question is: is this behaviour a feature, or a bug, especially since the tests are marked as `?
oh, I missed that part, so in Linux kubelet must be able to communicate with pods on all nodes (becuase it is running in the hostnetwork) If that is correct, it seems that one solution is to keep the test for Linux and tag it as LinuxOnly, and a new test for WindowsOnly that only spawn the pods under test in the same node EDIT Discussed in sig-net meeting , no clear answer... can be kubelet considered as a hostnetwork pod?
For the purposes of testing, I don't mind forcing the pods onto the same node., but let's document the heck out of that at the code-site?
correct but it will be supported once HostProcess containers are merged in 1.22: Only in Overlay networks (ie Flannel where these tests are flaking) does the pod have to be on the same node. It doesn't require host network mode and these tests are . I would be more inclined to disable this test for Windows Overlay test setups and have a sig-windows test for the overlay functionality. I would also question if the lifecycle hooks only running on the same node is useful and if it should just be documented that it doesn't work in the Windows Overlay setup.
is looking at test PRs from SIG Node. I will let sig-net triage this issue
Next steps on this?
/triage accepted
We still haven't merged this: I also had this, but it has been decided to be blocked:
requires conformance approval, at this point I think that has to be postponed to 1.23 right, since that behavior is not required, we shouldn't test it, because is legit to fail and having a test for something that can fail can create confusion :smile:

}, } podWithHook := getPodWithHook("pod-with-poststart-exec-hook", imageutils.GetE2EImage(imageutils.Agnhost), lifecycle) testPodWithHook(podWithHook) }) /*