Question:
The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows

} } <ins> // Finished writing into the file, close it, so we can delete it later. err = file.Close() assert.NoErrorf(t, err, "could not close file.") </ins> time.Sleep(20 * time.Millisecond) // Make the function ReadLogs end. fakeRuntimeService.Lock()


Question:
The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows

<ins> //go:build windows // +build windows /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package logs import ( "os" "syscall" ) // Based on Windows implementation of Windows' syscall.Open // https://cs.opensource.google/go/go/+/refs/tags/go1.22.2:src/syscall/syscall_windows.go;l=342 // In addition to syscall.Open, this function also adds the syscall.FILE_SHARE_DELETE flag to sharemode, // which will allow us to read from the file without blocking the file from being deleted or renamed. // This is essential for Log Rotation which is done by renaming the open file. Without this, the file rename would fail. func openFileShareDelete(path string) (*os.File, error) { pathp, err := syscall.UTF16PtrFromString(path) if err != nil { return nil, err } var access uint32 = syscall.GENERIC_READ var sharemode uint32 = syscall.FILE_SHARE_READ | syscall.FILE_SHARE_WRITE | syscall.FILE_SHARE_DELETE var createmode uint32 = syscall.OPEN_EXISTING var attrs uint32 = syscall.FILE_ATTRIBUTE_NORMAL handle, err := syscall.CreateFile(pathp, access, sharemode, nil, createmode, attrs, 0) if err != nil { return nil, err } return os.NewFile(uintptr(handle), path), nil } </ins>


Question:
in This flaking test case was newly as part of the above PR Flake was reproducible with stress tool: [root kubernetes]# go version go version go1.22.2 linux/ppc64le [root kubernetes]# stress - ^TestLog /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Looks like in this test case is a concatenation of logs from Pod0 and Pod1, hence it is failing in some cases. I will add a PR to split it.

o.Prefix = true return o }, <del> expectedOutSubstrings: []string{"[pod/test-sts-0/test-container] test log content for pod test-sts-0n[pod/test-sts-1/test-container] test log content for pod test-sts-1n"}, </del> <ins> expectedOutSubstrings: []string{ "[pod/test-sts-0/test-container] test log content for pod test-sts-0n", "[pod/test-sts-1/test-container] test log content for pod test-sts-1n", }, </ins> }, { name: "pod logs with prefix: init container",


Question:
Currently, the scheduler honors Unschedulable(AndUnresolvable) or Pending only from PreFilter, Filter, Reserve, and Permit (WaitOnPermit). It doesn't expect other extension points to return those statuses and just regard all non-success status as errors, that is, immediately retrying scheduling of Pods. Actually returns Pending at PreBind, which is currently being ignored as described. So, we should expand UnschedulablePlugins/PendingPlugins to include PreBind plugins. (or maybe all extension points too?) /cc /sig scheduling /kind feature
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Both seems reasonable to me.
Because is not handled for , the following error is logged:
This patch gets rid of the error. I am uncertain whether it has other implications? Performance was the same with and without it.
What I imagined is like that. It shouldn't have any other impact because volumebinding, which is the only PreBind plugin apart from DRA in in-tree, only returns success or error status. We have to mention the change in the release note for custom plugin developers though.
I've turned this into a PR:

// latency for a pod. InitialAttemptTimestamp *time.Time // UnschedulablePlugins records the plugin names that the Pod failed with Unschedulable or UnschedulableAndUnresolvable status. <del> // It's registered only when the Pod is rejected in PreFilter, Filter, Reserve, or Permit (WaitOnPermit). </del> <ins> // It's registered only when the Pod is rejected in PreFilter, Filter, Reserve, PreBind or Permit (WaitOnPermit). </ins> UnschedulablePlugins sets.Set[string] // PendingPlugins records the plugin names that the Pod failed with Pending status. PendingPlugins sets.Set[string]


Question:
Currently, the scheduler honors Unschedulable(AndUnresolvable) or Pending only from PreFilter, Filter, Reserve, and Permit (WaitOnPermit). It doesn't expect other extension points to return those statuses and just regard all non-success status as errors, that is, immediately retrying scheduling of Pods. Actually returns Pending at PreBind, which is currently being ignored as described. So, we should expand UnschedulablePlugins/PendingPlugins to include PreBind plugins. (or maybe all extension points too?) /cc /sig scheduling /kind feature
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Both seems reasonable to me.
Because is not handled for , the following error is logged:
This patch gets rid of the error. I am uncertain whether it has other implications? Performance was the same with and without it.
What I imagined is like that. It shouldn't have any other impact because volumebinding, which is the only PreBind plugin apart from DRA in in-tree, only returns success or error status. We have to mention the change in the release note for custom plugin developers though.
I've turned this into a PR:

// Run "prebind" plugins. if status := fwk.RunPreBindPlugins(ctx, state, assumedPod, scheduleResult.SuggestedHost); !status.IsSuccess() { <ins> if status.IsRejected() { fitErr := &framework.FitError{ NumAllNodes: 1, Pod:         assumedPodInfo.Pod, Diagnosis: framework.Diagnosis{ NodeToStatusMap:      framework.NodeToStatusMap{scheduleResult.SuggestedHost: status}, UnschedulablePlugins: sets.New(status.Plugin()), }, } return framework.NewStatus(status.Code()).WithError(fitErr) } </ins> return status }


Question:
/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Failures look like started when landed

ginkgo.It("should reject a pod with an AppArmor profile", func(ctx context.Context) { status := runAppArmorTest(ctx, f, false, v1.DeprecatedAppArmorBetaProfileRuntimeDefault) <del> expectSoftRejection(status) </del> <ins> expectRejection(status) </ins> }) }) }


Question:
/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Failures look like started when landed

return e2epod.NewPodClient(f).Create(ctx, pod) } <del> func expectSoftRejection(status v1.PodStatus) { </del> <ins> func expectRejection(status v1.PodStatus) { </ins> args := []interface{}{"PodStatus: %+v", status} <del> gomega.Expect(status.Phase).To(gomega.Equal(v1.PodPending), args...) </del> <ins> gomega.Expect(status.Phase).To(gomega.Equal(v1.PodFailed), args...) </ins> gomega.Expect(status.Reason).To(gomega.Equal("AppArmor"), args...) gomega.Expect(status.Message).To(gomega.ContainSubstring("AppArmor"), args...) <del> gomega.Expect(status.ContainerStatuses[0].State.Waiting.Reason).To(gomega.Equal("Blocked"), args...) </del> } func isAppArmorEnabled() bool {


Question:
for a PodFailurePolicyRule. However, when trying to create a JobSet programmatically with a JobTemplate that defines a PodFailurePolicy rule for failing a job immediately if the failed pod had condition DisruptionTarget, I get a validation error indicating that cannot be null. I've tried using both and (with matching versions for package as well). Error: JobSet spec: JobTemplate passes validation. JobSet: applying example JobSet in the description of this issue. JobSet: and applying example JobSet in the description of this issue. No response 30 (tests on a kind cluster using kindest/node:v1.30.0) <details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
/assign
I confirmed the issue, moreover, I see that when the JobSet webhook is disabled then it works ok, I think it must be the JobSet webhook which adds the "onExitCodes" field during deserializion /serialization step. I can also see that when I rebuild JobSet using kubernetes field "onExitCodes" with "omitempty", then it the JobSet creates ok. I will open a PR.

// When specified, it should match one the container or initContainer // names in the pod template. // +optional <del> ContainerName *string `json:"containerName" protobuf:"bytes,1,opt,name=containerName"` </del> <ins> ContainerName *string `json:"containerName,omitempty" protobuf:"bytes,1,opt,name=containerName"` </ins> // Represents the relationship between the container exit code(s) and the // specified values. Containers completed with success (exit code 0) are


Question:
for a PodFailurePolicyRule. However, when trying to create a JobSet programmatically with a JobTemplate that defines a PodFailurePolicy rule for failing a job immediately if the failed pod had condition DisruptionTarget, I get a validation error indicating that cannot be null. I've tried using both and (with matching versions for package as well). Error: JobSet spec: JobTemplate passes validation. JobSet: applying example JobSet in the description of this issue. JobSet: and applying example JobSet in the description of this issue. No response 30 (tests on a kind cluster using kindest/node:v1.30.0) <details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
/assign
I confirmed the issue, moreover, I see that when the JobSet webhook is disabled then it works ok, I think it must be the JobSet webhook which adds the "onExitCodes" field during deserializion /serialization step. I can also see that when I rebuild JobSet using kubernetes field "onExitCodes" with "omitempty", then it the JobSet creates ok. I will open a PR.

// Represents the requirement on the container exit codes. // +optional <del> OnExitCodes *PodFailurePolicyOnExitCodesRequirement `json:"onExitCodes" protobuf:"bytes,2,opt,name=onExitCodes"` </del> <ins> OnExitCodes *PodFailurePolicyOnExitCodesRequirement `json:"onExitCodes,omitempty" protobuf:"bytes,2,opt,name=onExitCodes"` </ins> // Represents the requirement on the pod conditions. The requirement is represented // as a list of pod condition patterns. The requirement is satisfied if at // least one pattern matches an actual pod condition. At most 20 elements are allowed. // +listType=atomic // +optional <del> OnPodConditions []PodFailurePolicyOnPodConditionsPattern `json:"onPodConditions" protobuf:"bytes,3,opt,name=onPodConditions"` </del> <ins> OnPodConditions []PodFailurePolicyOnPodConditionsPattern `json:"onPodConditions,omitempty" protobuf:"bytes,3,opt,name=onPodConditions"` </ins> } // PodFailurePolicy describes how failed pods influence the backoffLimit.


Question:
After merged we are seeing the new test in fail since it was turned on be default now (it passed in the PR linked because it didn't have the alpha feature enabled). <br class="Apple-interchange-newline"<!--EndFragment--</body /sig windows /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
ptal

e2eoutput.TestContainerOutput(ctx, f, "check pod SecurityContext username", pod, 1, []string{"ContainerAdministrator"}) }) <del> ginkgo.It("should ignore SELinux Specific SecurityContext if set", func(ctx context.Context) { </del> <ins> ginkgo.It("should ignore Linux Specific SecurityContext if set", func(ctx context.Context) { </ins> ginkgo.By("Creating a pod with SELinux options") // It is sufficient to show that the pod comes up here. Since we're stripping the SELinux and other linux // security contexts in apiserver and not updating the pod object in the apiserver, we cannot validate the


Question:
After merged we are seeing the new test in fail since it was turned on be default now (it passed in the PR linked because it didn't have the alpha feature enabled). <br class="Apple-interchange-newline"<!--EndFragment--</body /sig windows /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
ptal

f.Namespace.Name), "failed to wait for pod %s to be running", windowsPodWithSELinux.Name) }) <del> ginkgo.It("should ignore ProcMount Specific SecurityContext if set", func(ctx context.Context) { ginkgo.By("Creating a pod with ProcMount options") // It is sufficient to show that the pod comes up here. Since we're stripping the SELinux and other linux // security contexts in apiserver and not updating the pod object in the apiserver, we cannot validate the // pod object to not have those security contexts. However the pod coming to running state is a sufficient // enough condition for us to validate since prior to https://github.com/kubernetes/kubernetes/pull/93475 // the pod would have failed to come up. windowsPodWithSELinux := createTestPod(f, imageutils.GetE2EImage(imageutils.Agnhost), windowsOS) windowsPodWithSELinux.Spec.Containers[0].Args = []string{"test-webserver-with-selinux"} windowsPodWithSELinux.Spec.SecurityContext = &v1.PodSecurityContext{} pmt := v1.UnmaskedProcMount containerUserName := "ContainerAdministrator" windowsPodWithSELinux.Spec.Containers[0].SecurityContext = &v1.SecurityContext{ ProcMount:      &pmt, WindowsOptions: &v1.WindowsSecurityContextOptions{RunAsUserName: &containerUserName}} windowsPodWithSELinux.Spec.Tolerations = []v1.Toleration{{Key: "os", Value: "Windows"}} windowsPodWithSELinux, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(ctx, windowsPodWithSELinux, metav1.CreateOptions{}) framework.ExpectNoError(err) framework.Logf("Created pod %v", windowsPodWithSELinux) framework.ExpectNoError(e2epod.WaitForPodNameRunningInNamespace(ctx, f.ClientSet, windowsPodWithSELinux.Name, f.Namespace.Name), "failed to wait for pod %s to be running", windowsPodWithSELinux.Name) }) </del> ginkgo.It("should not be able to create pods with containers running as ContainerAdministrator when runAsNonRoot is true", func(ctx context.Context) { ginkgo.By("Creating a pod")


Question:
July 24th. I think there is some suble way the test is named. Test is covered by and that works. I think this is mainly an issue where this test should not be run with these jobs. /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what is the correct way to label a job now? This is apparently wrong and causing the job to be picked up in the unlabeled name.
You still need to have a corresponding feature in because, as you noticed, there are jobs which don't support skipping based on the feature gate.
I created an issue in test-infra. once all tasks are completed, it's okay only to use framework.WithFeatureGate. There may be a lot of jobs that need to be updated. for now, you have to add a feature label or modify all the failing jobs.
/assign
xref

// TODO: document the feature (owning SIG, when to use this feature for a test) PodResources = framework.WithNodeFeature(framework.ValidNodeFeatures.Add("PodResources")) <ins> // ResourceHealthStatus (SIG-node, resource health Status for device plugins and DRA <https://kep.k8s.io/4680>) ResourceHealthStatus = framework.WithNodeFeature(framework.ValidNodeFeatures.Add("ResourceHealthStatus")) </ins> // TODO: document the feature (owning SIG, when to use this feature for a test) ProcMountType = framework.WithNodeFeature(framework.ValidNodeFeatures.Add("ProcMountType"))


Question:
July 24th. I think there is some suble way the test is named. Test is covered by and that works. I think this is mainly an issue where this test should not be run with these jobs. /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what is the correct way to label a job now? This is apparently wrong and causing the job to be picked up in the unlabeled name.
You still need to have a corresponding feature in because, as you noticed, there are jobs which don't support skipping based on the feature gate.
I created an issue in test-infra. once all tasks are completed, it's okay only to use framework.WithFeatureGate. There may be a lot of jobs that need to be updated. for now, you have to add a feature label or modify all the failing jobs.
/assign
xref

v1 "k8s.io/api/core/v1" kubeletdevicepluginv1beta1 "k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" <ins> "k8s.io/kubernetes/test/e2e/nodefeature" </ins> imageutils "k8s.io/kubernetes/test/utils/image" admissionapi "k8s.io/pod-security-admission/api" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid" <del> "k8s.io/kubernetes/pkg/features" </del> "k8s.io/kubernetes/test/e2e/framework" "k8s.io/kubernetes/test/e2e_node/testdeviceplugin" ) <del> var _ = SIGDescribe("Device Plugin Failures Pod Status:", framework.WithFeatureGate(features.ResourceHealthStatus), func() { </del> <ins> var _ = SIGDescribe("Device Plugin Failures Pod Status", nodefeature.ResourceHealthStatus, func() { </ins> f := framework.NewDefaultFramework("device-plugin-failures") f.NamespacePodSecurityLevel = admissionapi.LevelPrivileged


Question:
Related to and SIG storage meeting notes: From me: It is related to image volume plugin and selinux. Is it a known limitation of all the volume plugins when using selinux? I have a different opinion. If two pods don't share the same mountpoint (hostpath), they can run the same node when the selinux is enabled. the current implementation of cri-o is sharing the same hostpath when two pods use the same image volume. More details about my idea: From Sharing each volume between namespaces is kind of natural. Sharing PVs not possible but sharing image quite common. From Although for most volume plugins it is true that, they can't be mounted with 2 different selinux context on same node, we want to understand why is this not possible for Imagevolume source. For a traditional storage persistent storage plugins - this limitation is underlying limitation of storage backend/Linux kernel etc. For ImageVolume source, it seems like a simple solution which unpacks image to two different hostpath would work and not run into this limitation. can you specify, why this limitation exists for ImageVolume plugins? From we choose to keep it simple and follow the lead of other volume types. supporting multiple selinux types would mean making more complicated the way we mount the image into the container AFAIR. But agreed we can chat about this elsewhere, in the meantime for the purposes of this PR: this is consistent with behavior of other volume plugins and is expected behavior. Let's discuss the SELinux behavior for image volumes in this issue. cc /sig node /sig storage
In my opinion, from an end-user perspective, the image volume may act like a read-only with a pre-populated content via image. Two pods from different namespaces can share the same image and run on the same node.
Pasted from According to , the containerd doesn't have the same limitation.
Given that OCI/Image Volume Source support is in alpha, I'd prefer that we consider discussing this as part of the original feature. We can discuss this here but I don't know if we need a separate KEP for this yet.
Yes, so far, in our conversations at the containerd community meetings we decided to start this feature out using a single pod scope for the pod scoped image volume host path because in containerd each runtime handler may have a different snapshotter (possibly on a different fs), or at least that's the plan. Additionally, the pods may have different os versions on windows indicating a different platform can be selected on the same node and this sort of versioning may extend beyond to other os providers. We could ask Kirtana to confirm the versioning issue and status of platform version selection over the platform index manifest, if the snapshotter reason isn't enough to at least start with per pod scope. If this is insufficient support we can certainly discuss. Cheers, Mike
No need a separate KEP. We can update the original KEP once this issue is resolved.
I'm working on a technical solution (just a PoC) in , not sure how feasible that is as time of writing, though.
yeah I agree I don't think we need a different feature for this, especially because it's mostly an implementation detail in the runtime. would you open this as an issue in kubernetes/kubernetes instead of the enhancements repo, or even better in the cri-o repo?
/transfer kubernetes
I moved this issue into kubernetes repo.
/triage accepted /priority important-soon
thanks!
Copid from kep: As merged, do we need to change the KEPs to reflect the new behavior?
no, nothing needs to be changed from a Kubernetes any more :+1:
/remove-sig storage

"k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/kubelet/images" "k8s.io/kubernetes/test/e2e/framework" <ins> e2enode "k8s.io/kubernetes/test/e2e/framework/node" </ins> e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" "k8s.io/kubernetes/test/e2e/nodefeature"


Question:
Related to and SIG storage meeting notes: From me: It is related to image volume plugin and selinux. Is it a known limitation of all the volume plugins when using selinux? I have a different opinion. If two pods don't share the same mountpoint (hostpath), they can run the same node when the selinux is enabled. the current implementation of cri-o is sharing the same hostpath when two pods use the same image volume. More details about my idea: From Sharing each volume between namespaces is kind of natural. Sharing PVs not possible but sharing image quite common. From Although for most volume plugins it is true that, they can't be mounted with 2 different selinux context on same node, we want to understand why is this not possible for Imagevolume source. For a traditional storage persistent storage plugins - this limitation is underlying limitation of storage backend/Linux kernel etc. For ImageVolume source, it seems like a simple solution which unpacks image to two different hostpath would work and not run into this limitation. can you specify, why this limitation exists for ImageVolume plugins? From we choose to keep it simple and follow the lead of other volume types. supporting multiple selinux types would mean making more complicated the way we mount the image into the container AFAIR. But agreed we can chat about this elsewhere, in the meantime for the purposes of this PR: this is consistent with behavior of other volume plugins and is expected behavior. Let's discuss the SELinux behavior for image volumes in this issue. cc /sig node /sig storage
In my opinion, from an end-user perspective, the image volume may act like a read-only with a pre-populated content via image. Two pods from different namespaces can share the same image and run on the same node.
Pasted from According to , the containerd doesn't have the same limitation.
Given that OCI/Image Volume Source support is in alpha, I'd prefer that we consider discussing this as part of the original feature. We can discuss this here but I don't know if we need a separate KEP for this yet.
Yes, so far, in our conversations at the containerd community meetings we decided to start this feature out using a single pod scope for the pod scoped image volume host path because in containerd each runtime handler may have a different snapshotter (possibly on a different fs), or at least that's the plan. Additionally, the pods may have different os versions on windows indicating a different platform can be selected on the same node and this sort of versioning may extend beyond to other os providers. We could ask Kirtana to confirm the versioning issue and status of platform version selection over the platform index manifest, if the snapshotter reason isn't enough to at least start with per pod scope. If this is insufficient support we can certainly discuss. Cheers, Mike
No need a separate KEP. We can update the original KEP once this issue is resolved.
I'm working on a technical solution (just a PoC) in , not sure how feasible that is as time of writing, though.
yeah I agree I don't think we need a different feature for this, especially because it's mostly an implementation detail in the runtime. would you open this as an issue in kubernetes/kubernetes instead of the enhancements repo, or even better in the cri-o repo?
/transfer kubernetes
I moved this issue into kubernetes repo.
/triage accepted /priority important-soon
thanks!
Copid from kep: As merged, do we need to change the KEPs to reflect the new behavior?
no, nothing needs to be changed from a Kubernetes any more :+1:
/remove-sig storage

e2eskipper.SkipUnlessFeatureGateEnabled(features.ImageVolume) }) <del> createPod := func(ctx context.Context, volumes []v1.Volume, volumeMounts []v1.VolumeMount) { var selinuxOptions *v1.SELinuxOptions if selinux.GetEnabled() { selinuxOptions = &v1.SELinuxOptions{ User:  defaultSELinuxUser, Role:  defaultSELinuxRole, Type:  defaultSELinuxType, Level: defaultSELinuxLevel, } ginkgo.By(fmt.Sprintf("Using SELinux on pod: %v", selinuxOptions)) } </del> <ins> createPod := func(ctx context.Context, podName, nodeName string, volumes []v1.Volume, volumeMounts []v1.VolumeMount, selinuxOptions *v1.SELinuxOptions) { </ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      podName, Namespace: f.Namespace.Name, }, Spec: v1.PodSpec{ <ins> NodeName:      nodeName, </ins> RestartPolicy: v1.RestartPolicyAlways, SecurityContext: &v1.PodSecurityContext{ SELinuxOptions: selinuxOptions,


Question:
Related to and SIG storage meeting notes: From me: It is related to image volume plugin and selinux. Is it a known limitation of all the volume plugins when using selinux? I have a different opinion. If two pods don't share the same mountpoint (hostpath), they can run the same node when the selinux is enabled. the current implementation of cri-o is sharing the same hostpath when two pods use the same image volume. More details about my idea: From Sharing each volume between namespaces is kind of natural. Sharing PVs not possible but sharing image quite common. From Although for most volume plugins it is true that, they can't be mounted with 2 different selinux context on same node, we want to understand why is this not possible for Imagevolume source. For a traditional storage persistent storage plugins - this limitation is underlying limitation of storage backend/Linux kernel etc. For ImageVolume source, it seems like a simple solution which unpacks image to two different hostpath would work and not run into this limitation. can you specify, why this limitation exists for ImageVolume plugins? From we choose to keep it simple and follow the lead of other volume types. supporting multiple selinux types would mean making more complicated the way we mount the image into the container AFAIR. But agreed we can chat about this elsewhere, in the meantime for the purposes of this PR: this is consistent with behavior of other volume plugins and is expected behavior. Let's discuss the SELinux behavior for image volumes in this issue. cc /sig node /sig storage
In my opinion, from an end-user perspective, the image volume may act like a read-only with a pre-populated content via image. Two pods from different namespaces can share the same image and run on the same node.
Pasted from According to , the containerd doesn't have the same limitation.
Given that OCI/Image Volume Source support is in alpha, I'd prefer that we consider discussing this as part of the original feature. We can discuss this here but I don't know if we need a separate KEP for this yet.
Yes, so far, in our conversations at the containerd community meetings we decided to start this feature out using a single pod scope for the pod scoped image volume host path because in containerd each runtime handler may have a different snapshotter (possibly on a different fs), or at least that's the plan. Additionally, the pods may have different os versions on windows indicating a different platform can be selected on the same node and this sort of versioning may extend beyond to other os providers. We could ask Kirtana to confirm the versioning issue and status of platform version selection over the platform index manifest, if the snapshotter reason isn't enough to at least start with per pod scope. If this is insufficient support we can certainly discuss. Cheers, Mike
No need a separate KEP. We can update the original KEP once this issue is resolved.
I'm working on a technical solution (just a PoC) in , not sure how feasible that is as time of writing, though.
yeah I agree I don't think we need a different feature for this, especially because it's mostly an implementation detail in the runtime. would you open this as an issue in kubernetes/kubernetes instead of the enhancements repo, or even better in the cri-o repo?
/transfer kubernetes
I moved this issue into kubernetes repo.
/triage accepted /priority important-soon
thanks!
Copid from kep: As merged, do we need to change the KEPs to reflect the new behavior?
no, nothing needs to be changed from a Kubernetes any more :+1:
/remove-sig storage

}, } <del> ginkgo.By(fmt.Sprintf("Creating a pod (%v/%v)", f.Namespace.Name, podName)) </del> <ins> ginkgo.By(fmt.Sprintf("Creating a pod (%s/%s)", f.Namespace.Name, podName)) </ins> e2epod.NewPodClient(f).Create(ctx, pod) } f.It("should succeed with pod and pull policy of Always", func(ctx context.Context) { <ins> var selinuxOptions *v1.SELinuxOptions if selinux.GetEnabled() { selinuxOptions = &v1.SELinuxOptions{ User:  defaultSELinuxUser, Role:  defaultSELinuxRole, Type:  defaultSELinuxType, Level: defaultSELinuxLevel, } ginkgo.By(fmt.Sprintf("Using SELinux on pod: %v", selinuxOptions)) } </ins> createPod(ctx, <ins> podName, "", </ins> []v1.Volume{{Name: volumeName, VolumeSource: v1.VolumeSource{Image: &v1.ImageVolumeSource{Reference: validImageRef, PullPolicy: v1.PullAlways}}}}, []v1.VolumeMount{{Name: volumeName, MountPath: volumePathPrefix}}, <ins> selinuxOptions, </ins> ) <del> ginkgo.By(fmt.Sprintf("Waiting for the pod (%v/%v) to be running", f.Namespace.Name, podName)) </del> <ins> ginkgo.By(fmt.Sprintf("Waiting for the pod (%s/%s) to be running", f.Namespace.Name, podName)) </ins> err := e2epod.WaitForPodNameRunningInNamespace(ctx, f.ClientSet, podName, f.Namespace.Name) <del> framework.ExpectNoError(err, "Failed to await for the pod to be running: (%v/%v)", f.Namespace.Name, podName) </del> <ins> framework.ExpectNoError(err, "Failed to await for the pod to be running: (%s/%s)", f.Namespace.Name, podName) </ins> ginkgo.By(fmt.Sprintf("Verifying the volume mount contents for path: %s", volumePathPrefix))


Question:
Related to and SIG storage meeting notes: From me: It is related to image volume plugin and selinux. Is it a known limitation of all the volume plugins when using selinux? I have a different opinion. If two pods don't share the same mountpoint (hostpath), they can run the same node when the selinux is enabled. the current implementation of cri-o is sharing the same hostpath when two pods use the same image volume. More details about my idea: From Sharing each volume between namespaces is kind of natural. Sharing PVs not possible but sharing image quite common. From Although for most volume plugins it is true that, they can't be mounted with 2 different selinux context on same node, we want to understand why is this not possible for Imagevolume source. For a traditional storage persistent storage plugins - this limitation is underlying limitation of storage backend/Linux kernel etc. For ImageVolume source, it seems like a simple solution which unpacks image to two different hostpath would work and not run into this limitation. can you specify, why this limitation exists for ImageVolume plugins? From we choose to keep it simple and follow the lead of other volume types. supporting multiple selinux types would mean making more complicated the way we mount the image into the container AFAIR. But agreed we can chat about this elsewhere, in the meantime for the purposes of this PR: this is consistent with behavior of other volume plugins and is expected behavior. Let's discuss the SELinux behavior for image volumes in this issue. cc /sig node /sig storage
In my opinion, from an end-user perspective, the image volume may act like a read-only with a pre-populated content via image. Two pods from different namespaces can share the same image and run on the same node.
Pasted from According to , the containerd doesn't have the same limitation.
Given that OCI/Image Volume Source support is in alpha, I'd prefer that we consider discussing this as part of the original feature. We can discuss this here but I don't know if we need a separate KEP for this yet.
Yes, so far, in our conversations at the containerd community meetings we decided to start this feature out using a single pod scope for the pod scoped image volume host path because in containerd each runtime handler may have a different snapshotter (possibly on a different fs), or at least that's the plan. Additionally, the pods may have different os versions on windows indicating a different platform can be selected on the same node and this sort of versioning may extend beyond to other os providers. We could ask Kirtana to confirm the versioning issue and status of platform version selection over the platform index manifest, if the snapshotter reason isn't enough to at least start with per pod scope. If this is insufficient support we can certainly discuss. Cheers, Mike
No need a separate KEP. We can update the original KEP once this issue is resolved.
I'm working on a technical solution (just a PoC) in , not sure how feasible that is as time of writing, though.
yeah I agree I don't think we need a different feature for this, especially because it's mostly an implementation detail in the runtime. would you open this as an issue in kubernetes/kubernetes instead of the enhancements repo, or even better in the cri-o repo?
/transfer kubernetes
I moved this issue into kubernetes repo.
/triage accepted /priority important-soon
thanks!
Copid from kep: As merged, do we need to change the KEPs to reflect the new behavior?
no, nothing needs to be changed from a Kubernetes any more :+1:
/remove-sig storage

}) f.It("should succeed with pod and multiple volumes", func(ctx context.Context) { <ins> var selinuxOptions *v1.SELinuxOptions if selinux.GetEnabled() { selinuxOptions = &v1.SELinuxOptions{ User:  defaultSELinuxUser, Role:  defaultSELinuxRole, Type:  defaultSELinuxType, Level: defaultSELinuxLevel, } ginkgo.By(fmt.Sprintf("Using SELinux on pod: %v", selinuxOptions)) } </ins> createPod(ctx, <ins> podName, "", </ins> []v1.Volume{ {Name: volumeName + "-0", VolumeSource: v1.VolumeSource{Image: &v1.ImageVolumeSource{Reference: validImageRef}}}, {Name: volumeName + "-1", VolumeSource: v1.VolumeSource{Image: &v1.ImageVolumeSource{Reference: validImageRef}}},


Question:
Related to and SIG storage meeting notes: From me: It is related to image volume plugin and selinux. Is it a known limitation of all the volume plugins when using selinux? I have a different opinion. If two pods don't share the same mountpoint (hostpath), they can run the same node when the selinux is enabled. the current implementation of cri-o is sharing the same hostpath when two pods use the same image volume. More details about my idea: From Sharing each volume between namespaces is kind of natural. Sharing PVs not possible but sharing image quite common. From Although for most volume plugins it is true that, they can't be mounted with 2 different selinux context on same node, we want to understand why is this not possible for Imagevolume source. For a traditional storage persistent storage plugins - this limitation is underlying limitation of storage backend/Linux kernel etc. For ImageVolume source, it seems like a simple solution which unpacks image to two different hostpath would work and not run into this limitation. can you specify, why this limitation exists for ImageVolume plugins? From we choose to keep it simple and follow the lead of other volume types. supporting multiple selinux types would mean making more complicated the way we mount the image into the container AFAIR. But agreed we can chat about this elsewhere, in the meantime for the purposes of this PR: this is consistent with behavior of other volume plugins and is expected behavior. Let's discuss the SELinux behavior for image volumes in this issue. cc /sig node /sig storage
In my opinion, from an end-user perspective, the image volume may act like a read-only with a pre-populated content via image. Two pods from different namespaces can share the same image and run on the same node.
Pasted from According to , the containerd doesn't have the same limitation.
Given that OCI/Image Volume Source support is in alpha, I'd prefer that we consider discussing this as part of the original feature. We can discuss this here but I don't know if we need a separate KEP for this yet.
Yes, so far, in our conversations at the containerd community meetings we decided to start this feature out using a single pod scope for the pod scoped image volume host path because in containerd each runtime handler may have a different snapshotter (possibly on a different fs), or at least that's the plan. Additionally, the pods may have different os versions on windows indicating a different platform can be selected on the same node and this sort of versioning may extend beyond to other os providers. We could ask Kirtana to confirm the versioning issue and status of platform version selection over the platform index manifest, if the snapshotter reason isn't enough to at least start with per pod scope. If this is insufficient support we can certainly discuss. Cheers, Mike
No need a separate KEP. We can update the original KEP once this issue is resolved.
I'm working on a technical solution (just a PoC) in , not sure how feasible that is as time of writing, though.
yeah I agree I don't think we need a different feature for this, especially because it's mostly an implementation detail in the runtime. would you open this as an issue in kubernetes/kubernetes instead of the enhancements repo, or even better in the cri-o repo?
/transfer kubernetes
I moved this issue into kubernetes repo.
/triage accepted /priority important-soon
thanks!
Copid from kep: As merged, do we need to change the KEPs to reflect the new behavior?
no, nothing needs to be changed from a Kubernetes any more :+1:
/remove-sig storage

{Name: volumeName + "-0", MountPath: volumePathPrefix + "-0"}, {Name: volumeName + "-1", MountPath: volumePathPrefix + "-1"}, }, <ins> selinuxOptions, </ins> ) <del> ginkgo.By(fmt.Sprintf("Waiting for the pod (%v/%v) to be running", f.Namespace.Name, podName)) </del> <ins> ginkgo.By(fmt.Sprintf("Waiting for the pod (%s/%s) to be running", f.Namespace.Name, podName)) </ins> err := e2epod.WaitForPodNameRunningInNamespace(ctx, f.ClientSet, podName, f.Namespace.Name) <del> framework.ExpectNoError(err, "Failed to await for the pod to be running: (%v/%v)", f.Namespace.Name, podName) </del> <ins> framework.ExpectNoError(err, "Failed to await for the pod to be running: (%s/%s)", f.Namespace.Name, podName) </ins> for i := range 2 { volumePath := fmt.Sprintf("%s-%d", volumePathPrefix, i)


Question:
Related to and SIG storage meeting notes: From me: It is related to image volume plugin and selinux. Is it a known limitation of all the volume plugins when using selinux? I have a different opinion. If two pods don't share the same mountpoint (hostpath), they can run the same node when the selinux is enabled. the current implementation of cri-o is sharing the same hostpath when two pods use the same image volume. More details about my idea: From Sharing each volume between namespaces is kind of natural. Sharing PVs not possible but sharing image quite common. From Although for most volume plugins it is true that, they can't be mounted with 2 different selinux context on same node, we want to understand why is this not possible for Imagevolume source. For a traditional storage persistent storage plugins - this limitation is underlying limitation of storage backend/Linux kernel etc. For ImageVolume source, it seems like a simple solution which unpacks image to two different hostpath would work and not run into this limitation. can you specify, why this limitation exists for ImageVolume plugins? From we choose to keep it simple and follow the lead of other volume types. supporting multiple selinux types would mean making more complicated the way we mount the image into the container AFAIR. But agreed we can chat about this elsewhere, in the meantime for the purposes of this PR: this is consistent with behavior of other volume plugins and is expected behavior. Let's discuss the SELinux behavior for image volumes in this issue. cc /sig node /sig storage
In my opinion, from an end-user perspective, the image volume may act like a read-only with a pre-populated content via image. Two pods from different namespaces can share the same image and run on the same node.
Pasted from According to , the containerd doesn't have the same limitation.
Given that OCI/Image Volume Source support is in alpha, I'd prefer that we consider discussing this as part of the original feature. We can discuss this here but I don't know if we need a separate KEP for this yet.
Yes, so far, in our conversations at the containerd community meetings we decided to start this feature out using a single pod scope for the pod scoped image volume host path because in containerd each runtime handler may have a different snapshotter (possibly on a different fs), or at least that's the plan. Additionally, the pods may have different os versions on windows indicating a different platform can be selected on the same node and this sort of versioning may extend beyond to other os providers. We could ask Kirtana to confirm the versioning issue and status of platform version selection over the platform index manifest, if the snapshotter reason isn't enough to at least start with per pod scope. If this is insufficient support we can certainly discuss. Cheers, Mike
No need a separate KEP. We can update the original KEP once this issue is resolved.
I'm working on a technical solution (just a PoC) in , not sure how feasible that is as time of writing, though.
yeah I agree I don't think we need a different feature for this, especially because it's mostly an implementation detail in the runtime. would you open this as an issue in kubernetes/kubernetes instead of the enhancements repo, or even better in the cri-o repo?
/transfer kubernetes
I moved this issue into kubernetes repo.
/triage accepted /priority important-soon
thanks!
Copid from kep: As merged, do we need to change the KEPs to reflect the new behavior?
no, nothing needs to be changed from a Kubernetes any more :+1:
/remove-sig storage

}) f.It("should fail if image volume is not existing", func(ctx context.Context) { <ins> var selinuxOptions *v1.SELinuxOptions if selinux.GetEnabled() { selinuxOptions = &v1.SELinuxOptions{ User:  defaultSELinuxUser, Role:  defaultSELinuxRole, Type:  defaultSELinuxType, Level: defaultSELinuxLevel, } ginkgo.By(fmt.Sprintf("Using SELinux on pod: %v", selinuxOptions)) } </ins> createPod(ctx, <ins> podName, "", </ins> []v1.Volume{{Name: volumeName, VolumeSource: v1.VolumeSource{Image: &v1.ImageVolumeSource{Reference: invalidImageRef}}}}, []v1.VolumeMount{{Name: volumeName, MountPath: volumePathPrefix}}, <ins> selinuxOptions, </ins> ) <del> ginkgo.By(fmt.Sprintf("Waiting for the pod (%v/%v) to fail", f.Namespace.Name, podName)) </del> <ins> ginkgo.By(fmt.Sprintf("Waiting for the pod (%s/%s) to fail", f.Namespace.Name, podName)) </ins> err := e2epod.WaitForPodContainerToFail(ctx, f.ClientSet, f.Namespace.Name, podName, 0, images.ErrImagePullBackOff.Error(), time.Minute) <del> framework.ExpectNoError(err, "Failed to await for the pod to be running: (%v/%v)", f.Namespace.Name, podName) </del> <ins> framework.ExpectNoError(err, "Failed to await for the pod to be running: (%s/%s)", f.Namespace.Name, podName) </ins> }) f.It("should succeed if image volume is not existing but unused", func(ctx context.Context) { <ins> var selinuxOptions *v1.SELinuxOptions if selinux.GetEnabled() { selinuxOptions = &v1.SELinuxOptions{ User:  defaultSELinuxUser, Role:  defaultSELinuxRole, Type:  defaultSELinuxType, Level: defaultSELinuxLevel, } ginkgo.By(fmt.Sprintf("Using SELinux on pod: %v", selinuxOptions)) } </ins> createPod(ctx, <ins> podName, "", </ins> []v1.Volume{{Name: volumeName, VolumeSource: v1.VolumeSource{Image: &v1.ImageVolumeSource{Reference: invalidImageRef}}}}, nil, <ins> selinuxOptions, </ins> ) <del> ginkgo.By(fmt.Sprintf("Waiting for the pod (%v/%v) to be running", f.Namespace.Name, podName)) </del> <ins> ginkgo.By(fmt.Sprintf("Waiting for the pod (%s/%s) to be running", f.Namespace.Name, podName)) </ins> err := e2epod.WaitForPodNameRunningInNamespace(ctx, f.ClientSet, podName, f.Namespace.Name) <del> framework.ExpectNoError(err, "Failed to await for the pod to be running: (%v/%v)", f.Namespace.Name, podName) </del> <ins> framework.ExpectNoError(err, "Failed to await for the pod to be running: (%s/%s)", f.Namespace.Name, podName) </ins> ginkgo.By(fmt.Sprintf("Verifying the volume mount is not used for path: %s", volumePathPrefix)) output := e2epod.ExecCommandInContainer(f, podName, containerName, "/bin/ls", filepath.Dir(volumePathPrefix)) gomega.Expect(output).NotTo(gomega.ContainSubstring(strings.TrimPrefix(volumePathPrefix, "/"))) }) <ins> f.It("should succeed with multiple pods and same image on the same node", func(ctx context.Context) { node, err := e2enode.GetRandomReadySchedulableNode(ctx, f.ClientSet) framework.ExpectNoError(err, "Failed to get a ready schedulable node") baseName := "test-pod" anotherSELinuxLevel := "s0:c100,c200" for i := range 2 { podName := fmt.Sprintf("%s-%d", baseName, i) var selinuxOptions *v1.SELinuxOptions if selinux.GetEnabled() { if i == 0 { selinuxOptions = &v1.SELinuxOptions{ User:  defaultSELinuxUser, Role:  defaultSELinuxRole, Type:  defaultSELinuxType, Level: defaultSELinuxLevel, } } else { selinuxOptions = &v1.SELinuxOptions{ User:  defaultSELinuxUser, Role:  defaultSELinuxRole, Type:  defaultSELinuxType, Level: anotherSELinuxLevel, } } ginkgo.By(fmt.Sprintf("Using SELinux on pod %q: %v", podName, selinuxOptions)) } createPod(ctx, podName, node.Name, []v1.Volume{{Name: volumeName, VolumeSource: v1.VolumeSource{Image: &v1.ImageVolumeSource{Reference: validImageRef}}}}, []v1.VolumeMount{{Name: volumeName, MountPath: volumePathPrefix}}, selinuxOptions, ) ginkgo.By(fmt.Sprintf("Waiting for the pod (%s/%s) to be running", f.Namespace.Name, podName)) err := e2epod.WaitForPodNameRunningInNamespace(ctx, f.ClientSet, podName, f.Namespace.Name) framework.ExpectNoError(err, "Failed to await for the pod to be running: (%s/%s)", f.Namespace.Name, podName) ginkgo.By(fmt.Sprintf("Verifying the volume mount contents for path: %s", volumePathPrefix)) firstFileContents := e2epod.ExecCommandInContainer(f, podName, containerName, "/bin/cat", filepath.Join(volumePathPrefix, "dir", "file")) gomega.Expect(firstFileContents).To(gomega.Equal("1")) secondFileContents := e2epod.ExecCommandInContainer(f, podName, containerName, "/bin/cat", filepath.Join(volumePathPrefix, "file")) gomega.Expect(secondFileContents).To(gomega.Equal("2")) } podName := baseName + "-0" ginkgo.By(fmt.Sprintf("Rechecking the pod (%s/%s) after another pod is running as expected", f.Namespace.Name, podName)) err = e2epod.WaitForPodNameRunningInNamespace(ctx, f.ClientSet, podName, f.Namespace.Name) framework.ExpectNoError(err, "Failed to await for the pod to be running: (%s/%s)", f.Namespace.Name, podName) ginkgo.By(fmt.Sprintf("Verifying the volume mount contents for path: %s", volumePathPrefix)) firstFileContents := e2epod.ExecCommandInContainer(f, podName, containerName, "/bin/cat", filepath.Join(volumePathPrefix, "dir", "file")) gomega.Expect(firstFileContents).To(gomega.Equal("1")) secondFileContents := e2epod.ExecCommandInContainer(f, podName, containerName, "/bin/cat", filepath.Join(volumePathPrefix, "file")) gomega.Expect(secondFileContents).To(gomega.Equal("2")) }) </ins> })