Question: return nil } <del> // DeleteTCPLoadBalancer is an implementation of TCPLoadBalancer.DeleteTCPLoadBalancer. func (gce *GCECloud) DeleteTCPLoadBalancer(name, region string) error { </del> <ins> // EnsureTCPLoadBalancerDeleted is an implementation of TCPLoadBalancer.EnsureTCPLoadBalancerDeleted. func (gce *GCECloud) EnsureTCPLoadBalancerDeleted(name, region string) error { </ins> op, err := gce.service.ForwardingRules.Delete(gce.projectID, region, name).Do() if err != nil && isHTTPErrorCode(err, http.StatusNotFound) { glog.Infof("Forwarding rule %s already deleted. Continuing to delete target pool.", name)

The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.


Question: return nil } <del> // DeleteTCPLoadBalancer is an implementation of TCPLoadBalancer.DeleteTCPLoadBalancer. func (gce *GCECloud) DeleteTCPLoadBalancer(name, region string) error { </del> <ins> // EnsureTCPLoadBalancerDeleted is an implementation of TCPLoadBalancer.EnsureTCPLoadBalancerDeleted. func (gce *GCECloud) EnsureTCPLoadBalancerDeleted(name, region string) error { </ins> op, err := gce.service.ForwardingRules.Delete(gce.projectID, region, name).Do() if err != nil && isHTTPErrorCode(err, http.StatusNotFound) { glog.Infof("Forwarding rule %s already deleted. Continuing to delete target pool.", name)

I'll periodically check to see whether anything is leaked after this point. I apologize if the deletions affected any running tests - everything looked to only be from GKE and parallel e2e runs.


Question: MounterSetUpTests(t, false) }) } <ins> func TestMounterSetUpWithFSGroup(t *testing.T) { fakeClient := fakeclient.NewSimpleClientset() plug, tmpDir := newTestPlugin(t, fakeClient, nil) defer os.RemoveAll(tmpDir) testCases := []struct { name        string accessModes []api.PersistentVolumeAccessMode readOnly    bool fsType      string setFsGroup  bool fsGroup     int64 }{ { name: "default fstype, with no fsgroup (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, readOnly: false, fsType:   "", }, { name: "default fstype  with fsgroup (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, readOnly:   false, fsType:     "", setFsGroup: true, fsGroup:    3000, }, { name: "fstype, fsgroup, RWM, ROM provided (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteMany, api.ReadOnlyMany, }, fsType:     "ext4", setFsGroup: true, fsGroup:    3000, }, { name: "fstype, fsgroup, RWO, but readOnly (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, readOnly:   true, fsType:     "ext4", setFsGroup: true, fsGroup:    3000, }, { name: "fstype, fsgroup, RWO provided (should apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, fsType:     "ext4", setFsGroup: true, fsGroup:    3000, }, } for i, tc := range testCases { t.Logf("Running test %s", tc.name) volName := fmt.Sprintf("test-vol-%d", i) pv := makeTestPV("test-pv", 10, testDriver, volName) pv.Spec.AccessModes = tc.accessModes pvName := pv.GetName() spec := volume.NewSpecFromPersistentVolume(pv, tc.readOnly) if tc.fsType != "" { spec.PersistentVolume.Spec.CSI.FSType = tc.fsType } mounter, err := plug.NewMounter( spec, &api.Pod{ObjectMeta: meta.ObjectMeta{UID: testPodUID, Namespace: testns}}, volume.VolumeOptions{}, ) if err != nil { t.Fatalf("Failed to make a new Mounter: %v", err) } if mounter == nil { t.Fatal("failed to create CSI mounter") } csiMounter := mounter.(*csiMountMgr) csiMounter.csiClient = setupClient(t, true) attachID := getAttachmentName(csiMounter.volumeID, csiMounter.driverName, string(plug.host.GetNodeName())) attachment := makeTestAttachment(attachID, "test-node", pvName) _, err = csiMounter.k8s.StorageV1beta1().VolumeAttachments().Create(attachment) if err != nil { t.Errorf("failed to setup VolumeAttachment: %v", err) continue } // Mounter.SetUp() var fsGroupPtr *int64 if tc.setFsGroup { fsGroup := tc.fsGroup fsGroupPtr = &fsGroup } if err := csiMounter.SetUp(fsGroupPtr); err != nil { t.Fatalf("mounter.Setup failed: %v", err) } //Test the default value of file system type is not overridden if len(csiMounter.spec.PersistentVolume.Spec.CSI.FSType) != len(tc.fsType) { t.Errorf("file system type was overridden by type %s", csiMounter.spec.PersistentVolume.Spec.CSI.FSType) } // ensure call went all the way pubs := csiMounter.csiClient.(*fakeCsiDriverClient).nodeClient.GetNodePublishedVolumes() if pubs[csiMounter.volumeID].Path != csiMounter.GetPath() { t.Error("csi server may not have received NodePublishVolume call") } } } </ins> func TestUnmounterTeardown(t *testing.T) { plug, tmpDir := newTestPlugin(t, nil, nil)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: fsgroup is only intended for RWO volumes because it recursively changes the permissions of all the directories in the volume. This can cause problems for nfs-type volumes because multiple pods could access the same volume and have the permissions changed out from underneath. What you expected to happen: Determine whether or not the plugin supports fsgroup either via: access mode. This is still a bit ambiguous for nfs volume types, that can theoretically support rwo, especially since Kubernetes does not do any sort of access mode enforcement. a new csi capability
CC
Are you saying that CSI internal driver should check driver capability for RWO, if supported, only then apply value ? What would the be ?
Yes one possible way is to infer fsgroup is supported for all rwo volumes, which may not be true? Like a nfs volume technically supports rwo but not fsgroup. So the other option is to make it a new CSI capability. I'm not sure exactly sure how best to describe it though, maybe something about volume ownership management.
Will it work if is not set then don't chown the ownership? I would think that for shared file system types(nfs, glusterfs) - won't be set. For raw block storage too - I think we shouldn't have to do this.
what do you mean by ? Shouldnt have to solve this issue ?
I meant - which is obvious. I just meant that the entire mechanism of changing ownership of files should apply only to volume types with block storage filesystems. I am just wondering if we can get away with detecting fsType and not have to introduce new capability in CSI.
agreed, do not map well to current csi.Capabilities enums which makes it hard to determine RWO when mounting. Perhaps, instead of a new capability, we revisit the existing capabilities to see if there are any way RWO can be reliably inferred from returned capabilities.


Question: MounterSetUpTests(t, false) }) } <ins> func TestMounterSetUpWithFSGroup(t *testing.T) { fakeClient := fakeclient.NewSimpleClientset() plug, tmpDir := newTestPlugin(t, fakeClient, nil) defer os.RemoveAll(tmpDir) testCases := []struct { name        string accessModes []api.PersistentVolumeAccessMode readOnly    bool fsType      string setFsGroup  bool fsGroup     int64 }{ { name: "default fstype, with no fsgroup (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, readOnly: false, fsType:   "", }, { name: "default fstype  with fsgroup (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, readOnly:   false, fsType:     "", setFsGroup: true, fsGroup:    3000, }, { name: "fstype, fsgroup, RWM, ROM provided (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteMany, api.ReadOnlyMany, }, fsType:     "ext4", setFsGroup: true, fsGroup:    3000, }, { name: "fstype, fsgroup, RWO, but readOnly (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, readOnly:   true, fsType:     "ext4", setFsGroup: true, fsGroup:    3000, }, { name: "fstype, fsgroup, RWO provided (should apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, fsType:     "ext4", setFsGroup: true, fsGroup:    3000, }, } for i, tc := range testCases { t.Logf("Running test %s", tc.name) volName := fmt.Sprintf("test-vol-%d", i) pv := makeTestPV("test-pv", 10, testDriver, volName) pv.Spec.AccessModes = tc.accessModes pvName := pv.GetName() spec := volume.NewSpecFromPersistentVolume(pv, tc.readOnly) if tc.fsType != "" { spec.PersistentVolume.Spec.CSI.FSType = tc.fsType } mounter, err := plug.NewMounter( spec, &api.Pod{ObjectMeta: meta.ObjectMeta{UID: testPodUID, Namespace: testns}}, volume.VolumeOptions{}, ) if err != nil { t.Fatalf("Failed to make a new Mounter: %v", err) } if mounter == nil { t.Fatal("failed to create CSI mounter") } csiMounter := mounter.(*csiMountMgr) csiMounter.csiClient = setupClient(t, true) attachID := getAttachmentName(csiMounter.volumeID, csiMounter.driverName, string(plug.host.GetNodeName())) attachment := makeTestAttachment(attachID, "test-node", pvName) _, err = csiMounter.k8s.StorageV1beta1().VolumeAttachments().Create(attachment) if err != nil { t.Errorf("failed to setup VolumeAttachment: %v", err) continue } // Mounter.SetUp() var fsGroupPtr *int64 if tc.setFsGroup { fsGroup := tc.fsGroup fsGroupPtr = &fsGroup } if err := csiMounter.SetUp(fsGroupPtr); err != nil { t.Fatalf("mounter.Setup failed: %v", err) } //Test the default value of file system type is not overridden if len(csiMounter.spec.PersistentVolume.Spec.CSI.FSType) != len(tc.fsType) { t.Errorf("file system type was overridden by type %s", csiMounter.spec.PersistentVolume.Spec.CSI.FSType) } // ensure call went all the way pubs := csiMounter.csiClient.(*fakeCsiDriverClient).nodeClient.GetNodePublishedVolumes() if pubs[csiMounter.volumeID].Path != csiMounter.GetPath() { t.Error("csi server may not have received NodePublishVolume call") } } } </ins> func TestUnmounterTeardown(t *testing.T) { plug, tmpDir := newTestPlugin(t, nil, nil)

Currently, this is what the code is doing which we knew needed to be revisited:
Users may not fully understand what fsgroup is and set it anyway, or a default psp with some fsgroup may be defined in which it will automatically be set on all pods. The first case we can attribute to user error, but the second case may not have a good workaround and could cause permission issues on rwx volumes. So I think we do need to solve this.
Yes, agreed this has to be solved. As you pointed out user error or improperly configured PSP can set value that does not match the correct attribute, causing permission issues. Here is what I think can be done without new CSI capabilities: , then do not apply fsgroup then: , apply fsgroup , do not apply fsgroup to void stated issue then: , apply fsgroup I think that may help deduce when to apply fsgroup.
If mount reference count <= 1, apply fsgroup If mount reference count 1, do not apply fsgroup to void stated issue A RWM volume (say NFS) can be mounted on multiple nodes. Do you have reliable count of mounts across all nodes?
In addition, that would be a change in semantics to how we support rwx volumes today. If we don't have a new CSI capability, then I think the best course of action is to only apply fsgroup for RWO volumes. Some questions remain: Do all RWO volumes support fsgroup? RWX is a superset of RWO. So if a user requests a RWO volume, it could still be satisfied by a RWX volume type. Can we detect this and still not apply the fsgroup even though the volume capability will say RWO?
If there's not a reliable way to enumerate node mounts (thinking about it, for CSI external drivers, the answer is NO). Theres is no reliable way to have k8s autocorrect a bad fsgroup/PV.AccessMode combo. So we would have to rely on the config provided by user/admin. I have come to similar conclusion as -- mainly, apply fsgroup only to volumes with access modes. The other k8s specified access modes, , cannot be reliably applied.


Question: MounterSetUpTests(t, false) }) } <ins> func TestMounterSetUpWithFSGroup(t *testing.T) { fakeClient := fakeclient.NewSimpleClientset() plug, tmpDir := newTestPlugin(t, fakeClient, nil) defer os.RemoveAll(tmpDir) testCases := []struct { name        string accessModes []api.PersistentVolumeAccessMode readOnly    bool fsType      string setFsGroup  bool fsGroup     int64 }{ { name: "default fstype, with no fsgroup (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, readOnly: false, fsType:   "", }, { name: "default fstype  with fsgroup (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, readOnly:   false, fsType:     "", setFsGroup: true, fsGroup:    3000, }, { name: "fstype, fsgroup, RWM, ROM provided (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteMany, api.ReadOnlyMany, }, fsType:     "ext4", setFsGroup: true, fsGroup:    3000, }, { name: "fstype, fsgroup, RWO, but readOnly (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, readOnly:   true, fsType:     "ext4", setFsGroup: true, fsGroup:    3000, }, { name: "fstype, fsgroup, RWO provided (should apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, fsType:     "ext4", setFsGroup: true, fsGroup:    3000, }, } for i, tc := range testCases { t.Logf("Running test %s", tc.name) volName := fmt.Sprintf("test-vol-%d", i) pv := makeTestPV("test-pv", 10, testDriver, volName) pv.Spec.AccessModes = tc.accessModes pvName := pv.GetName() spec := volume.NewSpecFromPersistentVolume(pv, tc.readOnly) if tc.fsType != "" { spec.PersistentVolume.Spec.CSI.FSType = tc.fsType } mounter, err := plug.NewMounter( spec, &api.Pod{ObjectMeta: meta.ObjectMeta{UID: testPodUID, Namespace: testns}}, volume.VolumeOptions{}, ) if err != nil { t.Fatalf("Failed to make a new Mounter: %v", err) } if mounter == nil { t.Fatal("failed to create CSI mounter") } csiMounter := mounter.(*csiMountMgr) csiMounter.csiClient = setupClient(t, true) attachID := getAttachmentName(csiMounter.volumeID, csiMounter.driverName, string(plug.host.GetNodeName())) attachment := makeTestAttachment(attachID, "test-node", pvName) _, err = csiMounter.k8s.StorageV1beta1().VolumeAttachments().Create(attachment) if err != nil { t.Errorf("failed to setup VolumeAttachment: %v", err) continue } // Mounter.SetUp() var fsGroupPtr *int64 if tc.setFsGroup { fsGroup := tc.fsGroup fsGroupPtr = &fsGroup } if err := csiMounter.SetUp(fsGroupPtr); err != nil { t.Fatalf("mounter.Setup failed: %v", err) } //Test the default value of file system type is not overridden if len(csiMounter.spec.PersistentVolume.Spec.CSI.FSType) != len(tc.fsType) { t.Errorf("file system type was overridden by type %s", csiMounter.spec.PersistentVolume.Spec.CSI.FSType) } // ensure call went all the way pubs := csiMounter.csiClient.(*fakeCsiDriverClient).nodeClient.GetNodePublishedVolumes() if pubs[csiMounter.volumeID].Path != csiMounter.GetPath() { t.Error("csi server may not have received NodePublishVolume call") } } } </ins> func TestUnmounterTeardown(t *testing.T) { plug, tmpDir := newTestPlugin(t, nil, nil)

Even if we were to introduce additional plugin capabilities that the CO can query (from driver) to find out what modes can be applied, there would still be the possibility of user/admin mis-configuration. Probably not, Is this where you think an additional capability would help ( ) That is a good point. Can't think of a way to detect/guess RWX when RWO is specified, this without the driver providing that info. Can we adopt the rule until we decide what capabilities would look like ?
I will just try to summarize the conversation from Slack. We can safely assume that only applies to block storage type with file system on top of it. So for all - volumes that need to have called MUST have a file system() on them. Is that assumption correct? If it is - can we not use presence of field or we can query the local volume for fs type (like how we query selinux relabelling capability) and call only when - volume is RW and has a valid block storage file system type. Will this not solve our use case? EDIT: renamed to .
We'll need to make it clear that is only for block-based storage systems, and the field cannot be reused for some multi-protocol file server (ie, nfsv3, nfsv4, smb, etc). For that case, then the plugin will need to expose its own new storageclass parameter + volume attribute to specify protocol.
Can we not use to get fsType - and use that information for calling ? It may be problematic to rely entirely on present inside csi source.
the only problem with is that in CSI volume info is intentionally opaque to the CSI code inside k8s. All mount logic is delegated to the driver.
Instead of a new capability or trying to deduce how best to apply fsgroup, why not pass the to the driver and let it decide how to apply it (either as a attribute for the mount or top level request param)? Right now, all mount logic has been delegated to the external CSI driver except for applying permission. Thoughts ?
that is true but we can still inspect file system on the mounted path because by the time gets called volume is already mounted. This should be no different than how selinux support is determined. what you think?
is this approved for 1.12 milestone? Thank you for working on this,
still discussing how best to approach this.


Question: MounterSetUpTests(t, false) }) } <ins> func TestMounterSetUpWithFSGroup(t *testing.T) { fakeClient := fakeclient.NewSimpleClientset() plug, tmpDir := newTestPlugin(t, fakeClient, nil) defer os.RemoveAll(tmpDir) testCases := []struct { name        string accessModes []api.PersistentVolumeAccessMode readOnly    bool fsType      string setFsGroup  bool fsGroup     int64 }{ { name: "default fstype, with no fsgroup (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, readOnly: false, fsType:   "", }, { name: "default fstype  with fsgroup (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, readOnly:   false, fsType:     "", setFsGroup: true, fsGroup:    3000, }, { name: "fstype, fsgroup, RWM, ROM provided (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteMany, api.ReadOnlyMany, }, fsType:     "ext4", setFsGroup: true, fsGroup:    3000, }, { name: "fstype, fsgroup, RWO, but readOnly (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, readOnly:   true, fsType:     "ext4", setFsGroup: true, fsGroup:    3000, }, { name: "fstype, fsgroup, RWO provided (should apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, fsType:     "ext4", setFsGroup: true, fsGroup:    3000, }, } for i, tc := range testCases { t.Logf("Running test %s", tc.name) volName := fmt.Sprintf("test-vol-%d", i) pv := makeTestPV("test-pv", 10, testDriver, volName) pv.Spec.AccessModes = tc.accessModes pvName := pv.GetName() spec := volume.NewSpecFromPersistentVolume(pv, tc.readOnly) if tc.fsType != "" { spec.PersistentVolume.Spec.CSI.FSType = tc.fsType } mounter, err := plug.NewMounter( spec, &api.Pod{ObjectMeta: meta.ObjectMeta{UID: testPodUID, Namespace: testns}}, volume.VolumeOptions{}, ) if err != nil { t.Fatalf("Failed to make a new Mounter: %v", err) } if mounter == nil { t.Fatal("failed to create CSI mounter") } csiMounter := mounter.(*csiMountMgr) csiMounter.csiClient = setupClient(t, true) attachID := getAttachmentName(csiMounter.volumeID, csiMounter.driverName, string(plug.host.GetNodeName())) attachment := makeTestAttachment(attachID, "test-node", pvName) _, err = csiMounter.k8s.StorageV1beta1().VolumeAttachments().Create(attachment) if err != nil { t.Errorf("failed to setup VolumeAttachment: %v", err) continue } // Mounter.SetUp() var fsGroupPtr *int64 if tc.setFsGroup { fsGroup := tc.fsGroup fsGroupPtr = &fsGroup } if err := csiMounter.SetUp(fsGroupPtr); err != nil { t.Fatalf("mounter.Setup failed: %v", err) } //Test the default value of file system type is not overridden if len(csiMounter.spec.PersistentVolume.Spec.CSI.FSType) != len(tc.fsType) { t.Errorf("file system type was overridden by type %s", csiMounter.spec.PersistentVolume.Spec.CSI.FSType) } // ensure call went all the way pubs := csiMounter.csiClient.(*fakeCsiDriverClient).nodeClient.GetNodePublishedVolumes() if pubs[csiMounter.volumeID].Path != csiMounter.GetPath() { t.Error("csi server may not have received NodePublishVolume call") } } } </ins> func TestUnmounterTeardown(t *testing.T) { plug, tmpDir := newTestPlugin(t, nil, nil)

It would be a 1.12.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Approved for 1.12
Based on previous post, this is a summary: If then is not applied (because it could be an indication of a nonblock fs, or error) Done is provided if fstype is invalid Done if fstype is valid and then apply fsgroup logic else ignore fsGroup Done
- any updates on ?
Waiting for a review and approval.
you have a outstanding comment on the PR. Can you please answer that?
How does this approach relate to inline CSI volumes? There's no PersistentVolume involved, and hence no way to specify an access mode. , but my driver ignores it (always using tmpfs). I considered having my CSI driver read the mounting pod's spec to see if it has a security context with "fsGroup" set, since that detail is not supplied to the driver, as . My goal is to have the volume mounted with the files on it readable by the container's user, but there's no way for my driver to know who that user will be. Am I supposed to just use 0444 for the ownership bits, to allow everyone to read the files?


Question: package events <del> import ( "k8s.io/api/core/v1" clientv1 "k8s.io/api/core/v1" ) </del> const ( // Container event reason list CreatedContainer        = "Created"

Is this a BUG REPORT or FEATURE REQUEST?: What happened: Now the and has been consistent in by PR . There is no need to convert v1.ObjectReference to clientv1.ObjectReference in . What you expected to happen: Remove this function and cleanup the codes which call this function. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Will submit the cleanup after PR merged.
/sig api-machinery
/sub


Question: gomega.Expect(stdout).To(gomega.Equal(mountName), msg) } else { // We *expect* cat to return error here <del> gomega.Expect(err).To(gomega.HaveOccurred(), msg) </del> <ins> framework.ExpectError(err, msg) </ins> } } }

What happened: After fixing golint failures under (now still in progress), we need to put for each and . Then common error checking code has become like: We can replace this code with which is more readable and easy to be understood what the code does. Now we have following e2e test directories which can be replaced: [x] test/e2e/apimachinery/ () [x] test/e2e/apps/ () [x] test/e2e/auth/ () [x] test/e2e/common/ () [x] test/e2e/node/ () [x] test/e2e/storage/ () [x] () [x] () [x] () [x] () [x] () [x] () [x] [x] test/e2e/storage/persistentvolumes- [x] () [x] () [x] [x] [x] [x] [x] ()
/sig testing /kind cleanup
/assign PR will be submitted soon.
Fix the following in
Fix in
Fix in
Hi, maybe I could take files beneath test/e2e/storage/vsphere/
oops, I'm finished and prepare for PR.
Thank you That's OK
Fix the following in
/assign
All related PRs were merged. I think this ticket can be closed. BTW, should we also replace to ?
Thanks for your work. thing is managed with We are almost finishing the work also :-)
/remove-kind bug
I found in these files. I'm working on :
I can work to the following item.


Question: package net import ( <ins> "crypto/tls" </ins> "net" "net/http" "net/url" "os" "reflect" "testing" <ins> "k8s.io/kubernetes/pkg/util/sets" </ins> ) <ins> func TestCloneTLSConfig(t *testing.T) { expected := sets.NewString( // These fields are copied in CloneTLSConfig "Rand", "Time", "Certificates", "RootCAs", "NextProtos", "ServerName", "InsecureSkipVerify", "CipherSuites", "PreferServerCipherSuites", "MinVersion", "MaxVersion", "CurvePreferences", "NameToCertificate", "GetCertificate", "ClientAuth", "ClientCAs", "ClientSessionCache", // These fields are not copied "SessionTicketsDisabled", "SessionTicketKey", // These fields are unexported "serverInitOnce", "mutex", "sessionTicketKeys", ) fields := sets.NewString() structType := reflect.TypeOf(tls.Config{}) for i := 0; i < structType.NumField(); i++ { fields.Insert(structType.Field(i).Name) } if missing := expected.Difference(fields); len(missing) > 0 { t.Errorf("Expected fields that were not seen in http.Transport: %v", missing.List()) } if extra := fields.Difference(expected); len(extra) > 0 { t.Errorf("New fields seen in http.Transport: %vnAdd to CopyClientTLSConfig if client-relevant, then add to expected list in TestCopyClientTLSConfig", extra.List()) } } </ins> func TestGetClientIP(t *testing.T) { ipString := "10.0.0.1" ip := net.ParseIP(ipString)

When the master dials nodes, it is mutating the tlsConfig held by the master kubelet client, to add a server name. That means if two different nodes are dialed from a master started with , x509 verification will fail for the second one. Introduced in
this is a regression from 1.3 and breaks pod logs/exec/attach and node proxying on installations that use
We probably need to review that PR and look to see if any other deliberate copies were flagged incorrectly by govet.
Installations that depend on secure trust from externally provided kubelet server certs are the highest security rings. If we don't fix this we need to tell anyone using this flag not to upgrade until we release 1.4.1.
the rest of the changes in the govet cleanup PR looked fine to me. fix for this issue is in
Since we are telling folks they have to upgrade to kubectl 1.4.0 for rollingupdate to work it would definitely be a confusing message to say other folks should not upgrade to 1.4.0 at all.
If the fix is merged into master by this afternoon, lets discuss whether or not we should cherrypick it - e.g. risks involved, how we can manually qa to verify nothing else was broken, etc.
Risk involved: If you are using a secure cluster (defined as ensuring no one can MITM your nodes and masters) and you upgrade to 1.4.0 and it does not include this fix - you are broken.
What is the likelihood of interactions with other code? Is it possible this fix could introduce a breaking change to something else? Has the fix been manually verified? Are we confident that this will fully fixes the issue (vs other edge cases exist for secure clusters that this doesn't catch). Ideally I would like to see this cherrypicked into 1.4 this evening and then cut a new beta that includes it.
None. The single line that is currently broken was changed. The rest is a new function (with no side effects), and tests. No, it restores the original intent of the TLS config copy, and tests cover the cases we are using this with (secure, insecure, custom dialer) Yes. I swept for the same issue elsewhere and didn't find any issues. We test with a fully secured master-kubelet setup in Openshift and got a clean test run with this fix applied.
SGTM. Thanks for fixing this.


Question: vmPowerStateStopped      = "stopped" vmPowerStateDeallocated  = "deallocated" vmPowerStateDeallocating = "deallocating" <ins> // nodeNameEnvironmentName is the environment variable name for getting node name. // It is only used for out-of-tree cloud provider. nodeNameEnvironmentName = "NODE_NAME" </ins> ) var (

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If computer name prefix does not match VMSS name, then Azure cloud provider will use VMSS API instead of using metadata service data. This leads to throttling, because every affected kubelet is making API requests. What you expected to happen: Azure cloud provider should use metadata service data instead of calling VMSS API when instance is current. How to reproduce it (as minimally and precisely as possible): Set computer name prefix of the VMSS to something different than default. You should observe Azure provider making requests to the VMSS API rather than using metadata service, which leads to throttling if you have multiple kubelets with this bug. Anything else we need to know?: This is bug was introduced by - returns false even if instance is current, because it expects node name to match VMSS name + instance id, however node name is derived from the computer-name-prefix which can be different than VMSS name. Node name should be matched against , not against . Environment: Kubernetes version (use ): 1.18.x and higher Cloud provider or hardware configuration: azure /sig cloud-provider
/sig cloud-provider
/area provider/azure
CC
/remove-sig cloud-provider /sig cloud-provider (trying to remove the label)
/sig cloud-provider
Thanks for the reporting. Unfortunately, computer name prefix is not included in IMDS, so it can't be used without VMSS API for constructing the NodeName. Would check what could we do to workaround the issue.
Can you memorize the computer name prefix in the first request to the Azure API and then use it for the isCurrentInstance check?
opened to fix the issue. Would you like to have a look?
I can confirm that your change should fix the problem I reported. Do you know if your fix will be to 1.18.x?
is working on the cherry pick
Thank you for quick fix!


Question: "containers": [ { "name": "kubernetes-pause", <del> "image": "gcr.io/google_containers/pause-amd64:3.0" </del> <ins> "image": "gcr.io/google_containers/pause-amd64:3.1" </ins> } ], "restartPolicy": "Never",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the pause-amd64:3.0 image on has an empty schema2 manifest, which prevents is from being pulled by proxy/caches which only support V2 registries. What you expected to happen: Being able to pull ` using a V2 manifest compatible client. Anything else we need to know?: The proxy/cahce is requesting this
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After talking with , he mentionned the following
AFAIK, does not support well. After inspecting the image, you can find is still using manifest v1.
/cc FYI
Im not sure which exact version x schéma X is needed. What I know is that we can download through the proxy/cache all other Kubernetes kube-system images, only the pause image is giving us trouble. Therefore, I would say that having the image available using the same format as the other kube-system images would be the request here.
Yeah, we haven't updated the pause image in a while, since it doesn't really ever change. I'm not sure whether rebuilding + pushing over the existing tag is safe, or if we should bump to or something. thoughts?
If we do rebuilding/updating the image, that would help and a bunch of similar issues.
it's better/save to go with new tag
I didn't realize there were changes to the pause binary that were never actually deployed. It seems like we should probably just do that (and bump the tag).
that sounds great. Any idea when the existing 3.0 tag could be pushed again?
We shouldn't push over the 3.0 tag; we should push a 3.1 or 4.0 tag instead. I'm not sure who owns the pause container who should drive this. Maybe sig-node and/or
It seems like it should be sig-node. I'll look into it.
The new container has been built, can you test with
Its working! thanks! I was able to pull it through our artifactory proxy , Lee Verberne <:
excellent, thanks for checking. /assign


Question: "containers": [ { "name": "kubernetes-pause", <del> "image": "gcr.io/google_containers/pause-amd64:3.0" </del> <ins> "image": "gcr.io/google_containers/pause-amd64:3.1" </ins> } ], "restartPolicy": "Never",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: pause does not reap zombie processes using the default pause container provided by What you expected to happen: should include the up to date pause code to reap zombie processes when using docker = 1.13.1 found here How to reproduce it (as minimally and precisely as possible): a.out will leave a zombie process that is not cleaned up Anything else we need to know?: Building the pause container from master solves this issue Related: Environment: Kubernetes version (use ): 1.7.0+ Cloud provider or hardware configuration**: N/A OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 4.10.13+ Install tools: Others: docker = 1.13.1
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
i get a simliar problem when i run crond service in pod container, every day it makes some [anacron] zombie progresses UID PID PPID C STIME TTY TIME CMD root 1 0 0 Nov14 ? 00:00:00 /pause root 7 0 0 Nov14 ? 00:00:00 /usr/local/bin/dumb-init /usr/lo root 13 7 0 Nov14 ? 00:00:00 nginx: master process /usr/local nobody 21 13 0 Nov14 ? 00:00:01 nginx: worker process nobody 22 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 23 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 24 13 0 Nov14 ? 00:00:05 nginx: worker process root 77 1 0 Nov14 ? 00:00:00 [anacron] root 1 0 00:01 ? 00:00:00 [anacron] <defunctroot 1 0 01:01 ? 00:00:00 [anacron] <defunctroot 1 0 02:01 ?


Question: "containers": [ { "name": "kubernetes-pause", <del> "image": "gcr.io/google_containers/pause-amd64:3.0" </del> <ins> "image": "gcr.io/google_containers/pause-amd64:3.1" </ins> } ], "restartPolicy": "Never",

00:00:00 [anacron] <defunctroot 1 0 03:01 ? 00:00:00 [anacron] <defunctroot 0 0 11:49 ? 00:00:00 /bin/bash root 0 1 19:15 ? 00:00:00 /bin/bash root 0 19:15 ? 00:00:00 ps -ef !
Do we know why this was never resolved / was never deployed?
I just fought through this issue with a rabbitmq helm chart. I had to custom build a pause container from master to fix it. The rabbitmq helm chart uses for its liveness checks. It's a thin wrapper around a call to the rabbitmq-server daemon, which is not PID 1 (/pause is PID 1). The rabbitmq-server process forks some children to do the work and then they exit. Since the pause-amd64:3.0 is not reaping, those processes stick around as zombies. The liveness check runs every 5 seconds. After about 4 hours, there are about 27K zombies and the node begins failing to launch new containers because it's starved for resources. The pause container built from current master fixes that behavior, no zombies at all in the container.
The plan is to build a new version of pause-amd64 and then update the image referenced by Kubernetes. This means the default image will change in the next Kubernetes release. We can make the new image available once the freeze lifts and we can merge .
Looks like this made it into 1.10


Question: registry = "k8s.gcr.io/build-image", repository = "debian-iptables", # Ensure the digests above are updated to match a new tag <del> tag = "buster-v1.4.0",  # ignored, but kept here for documentation </del> <ins> tag = "buster-v1.5.0",  # ignored, but kept here for documentation </ins> ) def etcd_tarballs():

The based container images for are shipped with the wrong architecture within the manifest if not amd64: Refers to We updated the images a few minutes ago to buster-v1.3.0, which are now ready to be used as base for the multi-architecture images. If the images contain the right architectures, then it may magically happen that the kube-proxy images are fixed, too. This is because we already use the build syntax for it: /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release


Question: if err != nil { return probe.Unknown, err } <del> if strings.ToLower(string(data)) != defaultHealthyOutput { </del> <ins> if !strings.HasPrefix(strings.ToLower(string(data)), defaultHealthyOutput) { </ins> return probe.Failure, nil } return probe.Success, nil

As far as I can tell, container liveness probes of type exec flat out don't work against clusters built at head. I had the kubelet always log the output of the exec'ed command, and it's always empty no matter what I exec, causing the probe to fail since it requires the output to be equal to the string "ok". If you want to test it, try adding a liveness probe of this form to any container: I also tried these lovely forms, to more closely match the liveness example under examples/: On the plus side, this may be an opportunistic time to switch over to
The reason that the e2e test for exec liveness probes hasn't caught this is that it only tests that containers are restarted when they fail liveness probes. It doesn't test that containers aren't restarted when their liveness checks should be succeeding. The e2e test's exec'ed commands also only ever produced empty output, rather than the expected "fail" output.
Is it possible to include this in a new pre-release, since it fixes a bug in health checking? I see it merged in master and I'm wondering what the policy is of releasing new code in a separate (0.16.*) release.


Question: // Otherwise we will return an error. func findSecurityGroupForInstance(instance *ec2.Instance, taggedSecurityGroups map[string]*ec2.SecurityGroup) (*ec2.GroupIdentifier, error) { instanceID := aws.StringValue(instance.InstanceId) <del> var best *ec2.GroupIdentifier </del> <ins> var tagged []*ec2.GroupIdentifier var untagged []*ec2.GroupIdentifier </ins> for _, group := range instance.SecurityGroups { groupID := aws.StringValue(group.GroupId) if groupID == "" { glog.Warningf("Ignoring security group without id for instance %q: %v", instanceID, group) continue } <del> if best == nil { best = group continue </del> <ins> _, isTagged := taggedSecurityGroups[groupID] if isTagged { tagged = append(tagged, group) } else { untagged = append(untagged, group) </ins> } <ins> } </ins> <del> _, bestIsTagged := taggedSecurityGroups[*best.GroupId] _, groupIsTagged := taggedSecurityGroups[groupID] if bestIsTagged && !groupIsTagged { // best is still best } else if groupIsTagged && !bestIsTagged { best = group } else { // We create instances with one SG // If users create multiple SGs, they must tag one of them as being k8s owned return nil, fmt.Errorf("Multiple security groups found for instance (%s); ensure the k8s security group is tagged", instanceID) </del> <ins> if len(tagged) > 0 { // We create instances with one SG // If users create multiple SGs, they must tag one of them as being k8s owned if len(tagged) != 1 { return nil, fmt.Errorf("Multiple tagged security groups found for instance %s; ensure only the k8s security group is tagged", instanceID) </ins> } <ins> return tagged[0], nil </ins> } <del> if best == nil { glog.Warningf("No security group found for instance %q", instanceID) </del> <ins> if len(untagged) > 0 { // For back-compat, we will allow a single untagged SG if len(untagged) != 1 { return nil, fmt.Errorf("Multiple untagged security groups found for instance %s; ensure the k8s security group is tagged", instanceID) } return untagged[0], nil </ins> } <del> return best, nil </del> <ins> glog.Warningf("No security group found for instance %q", instanceID) return nil, nil </ins> } // Return all the security groups that are tagged as being part of our cluster

reported hitting this error: The instance had 4 security groups, and only one of them was tagged. However, the logic around picking the security group is incorrect.. if we get two non-tagged SGs before we see the first tagged SG, it throws an error. We'll want to cherry-pick this to 1.2
I think this one was a victim of flakiness in the auto-assigner when the author is the first-listed maintainer. Assigning to you, :-)


Question: } } <del> func TestSetDefaultReplicationControllerImagePullPolicy(t *testing.T) { containersWithoutPullPolicy, _ := json.Marshal([]map[string]interface{}{ { "name":  "install", "image": "busybox:latest", }, }) </del> <ins> type InitContainerValidator func(got, expected *v1.Container) error </ins> <del> containersWithPullPolicy, _ := json.Marshal([]map[string]interface{}{ { "name":            "install", "imagePullPolicy": "IfNotPresent", }, }) </del> <ins> func TestSetDefaultReplicationControllerInitContainers(t *testing.T) { assertEnvFieldRef := func(got, expected *v1.Container) error { if len(got.Env) != len(expected.Env) { return fmt.Errorf("different number of env: got <%v>, expected <%v>", len(got.Env), len(expected.Env)) } for j := range got.Env { ge := &got.Env[j] ee := &expected.Env[j] if ge.Name != ee.Name { return fmt.Errorf("different name of env: got <%v>, expected <%v>", ge.Name, ee.Name) } if ge.ValueFrom.FieldRef.APIVersion != ee.ValueFrom.FieldRef.APIVersion { return fmt.Errorf("different api version of FieldRef <%v>: got <%v>, expected <%v>", ge.Name, ge.ValueFrom.FieldRef.APIVersion, ee.ValueFrom.FieldRef.APIVersion) } } return nil } assertImagePullPolicy := func(got, expected *v1.Container) error { if got.ImagePullPolicy != expected.ImagePullPolicy { return fmt.Errorf("different image pull poicy: got <%v>, expected <%v>", got.ImagePullPolicy, expected.ImagePullPolicy) } return nil } assertContainerPort := func(got, expected *v1.Container) error { if len(got.Ports) != len(expected.Ports) { return fmt.Errorf("different number of ports: got <%v>, expected <%v>", len(got.Ports), len(expected.Ports)) } for i := range got.Ports { gp := &got.Ports[i] ep := &expected.Ports[i] if gp.Name != ep.Name { return fmt.Errorf("different name of port: got <%v>, expected <%v>", gp.Name, ep.Name) } if gp.Protocol != ep.Protocol { return fmt.Errorf("different port protocol <%v>: got <%v>, expected <%v>", gp.Name, gp.Protocol, ep.Protocol) } } return nil } assertResource := func(got, expected *v1.Container) error { if len(got.Resources.Limits) != len(expected.Resources.Limits) { return fmt.Errorf("different number of resources.Limits: got <%v>, expected <%v>", len(got.Resources.Limits), (expected.Resources.Limits)) } for k, v := range got.Resources.Limits { if ev, found := expected.Resources.Limits[v1.ResourceName(k)]; !found { return fmt.Errorf("failed to find resource <%v> in expected resources.Limits.", k) } else { if ev.Value() != v.Value() { return fmt.Errorf("different resource.Limits: got <%v>, expected <%v>.", v.Value(), ev.Value()) } } } if len(got.Resources.Requests) != len(expected.Resources.Requests) { return fmt.Errorf("different number of resources.Requests: got <%v>, expected <%v>", len(got.Resources.Requests), (expected.Resources.Requests)) } for k, v := range got.Resources.Requests { if ev, found := expected.Resources.Requests[v1.ResourceName(k)]; !found { return fmt.Errorf("failed to find resource <%v> in expected resources.Requests.", k) } else { if ev.Value() != v.Value() { return fmt.Errorf("different resource.Requests: got <%v>, expected <%v>.", v.Value(), ev.Value()) } } } return nil } assertProb := func(got, expected *v1.Container) error { // Assert LivenessProbe if got.LivenessProbe.Handler.HTTPGet.Path != expected.LivenessProbe.Handler.HTTPGet.Path || got.LivenessProbe.Handler.HTTPGet.Scheme != expected.LivenessProbe.Handler.HTTPGet.Scheme || got.LivenessProbe.FailureThreshold != expected.LivenessProbe.FailureThreshold || got.LivenessProbe.SuccessThreshold != expected.LivenessProbe.SuccessThreshold || got.LivenessProbe.PeriodSeconds != expected.LivenessProbe.PeriodSeconds || got.LivenessProbe.TimeoutSeconds != expected.LivenessProbe.TimeoutSeconds { return fmt.Errorf("different LivenessProbe: got <%v>, expected <%v>", got.LivenessProbe, expected.LivenessProbe) } // Assert ReadinessProbe if got.ReadinessProbe.Handler.HTTPGet.Path != expected.ReadinessProbe.Handler.HTTPGet.Path || got.ReadinessProbe.Handler.HTTPGet.Scheme != expected.ReadinessProbe.Handler.HTTPGet.Scheme || got.ReadinessProbe.FailureThreshold != expected.ReadinessProbe.FailureThreshold || got.ReadinessProbe.SuccessThreshold != expected.ReadinessProbe.SuccessThreshold || got.ReadinessProbe.PeriodSeconds != expected.ReadinessProbe.PeriodSeconds || got.ReadinessProbe.TimeoutSeconds != expected.ReadinessProbe.TimeoutSeconds { return fmt.Errorf("different ReadinessProbe: got <%v>, expected <%v>", got.ReadinessProbe, expected.ReadinessProbe) } return nil } assertLifeCycle := func(got, expected *v1.Container) error { if got.Lifecycle.PostStart.HTTPGet.Path != expected.Lifecycle.PostStart.HTTPGet.Path || got.Lifecycle.PostStart.HTTPGet.Scheme != expected.Lifecycle.PostStart.HTTPGet.Scheme { return fmt.Errorf("different LifeCycle: got <%v>, expected <%v>", got.Lifecycle, expected.Lifecycle) } return nil } cpu, _ := resource.ParseQuantity("100Gi") mem, _ := resource.ParseQuantity("100Mi") </ins> tests := []struct { <del> rc               v1.ReplicationController expectPullPolicy v1.PullPolicy </del> <ins> name       string rc         v1.ReplicationController expected   []v1.Container validators []InitContainerValidator </ins> }{ { <ins> name: "imagePullIPolicy", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "install", "image": "busybox" } ]`, }, }, }, }, }, expected: []v1.Container{ { ImagePullPolicy: v1.PullAlways, }, }, validators: []InitContainerValidator{assertImagePullPolicy}, }, { name: "FieldRef", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "env": [ { "name": "MY_POD_IP", "valueFrom": { "fieldRef": { "apiVersion": "", "fieldPath": "status.podIP" } } } ] } ]`, }, }, }, }, }, expected: []v1.Container{ { Env: []v1.EnvVar{ { Name: "MY_POD_IP", ValueFrom: &v1.EnvVarSource{ FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "status.podIP", }, }, }, }, }, }, validators: []InitContainerValidator{assertEnvFieldRef}, }, { name: "ContainerPort", </ins> rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ <del> "pod.beta.kubernetes.io/init-containers": string(containersWithoutPullPolicy), </del> <ins> "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "ports": [ { "name": "default" } ] } ]`, </ins> }, }, }, }, }, <del> expectPullPolicy: v1.PullAlways, </del> <ins> expected: []v1.Container{ { Ports: []v1.ContainerPort{ { Name:     "default", Protocol: v1.ProtocolTCP, }, }, }, }, validators: []InitContainerValidator{assertContainerPort}, </ins> }, { <ins> name: "Resources", </ins> rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ <del> "pod.beta.kubernetes.io/init-containers": string(containersWithPullPolicy), </del> <ins> "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "resources": { "limits": { "cpu": "100Gi", "memory": "100Mi" }, "requests": { "cpu": "100Gi", "memory": "100Mi" } } } ]`, </ins> }, }, }, }, }, <del> expectPullPolicy: v1.PullIfNotPresent, </del> <ins> expected: []v1.Container{ { Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceCPU:    cpu, v1.ResourceMemory: mem, }, Requests: v1.ResourceList{ v1.ResourceCPU:    cpu, v1.ResourceMemory: mem, }, }, }, }, validators: []InitContainerValidator{assertResource}, }, { name: "Prob", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "livenessProbe": { "httpGet": { "host": "localhost" } }, "readinessProbe": { "httpGet": { "host": "localhost" } } } ]`, }, }, }, }, }, expected: []v1.Container{ { LivenessProbe: &v1.Probe{ Handler: v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, TimeoutSeconds:   1, PeriodSeconds:    10, SuccessThreshold: 1, FailureThreshold: 3, }, ReadinessProbe: &v1.Probe{ Handler: v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, TimeoutSeconds:   1, PeriodSeconds:    10, SuccessThreshold: 1, FailureThreshold: 3, }, }, }, validators: []InitContainerValidator{assertProb}, }, { name: "LifeCycle", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "lifecycle": { "postStart": { "httpGet": { "host": "localhost" } }, "preStop": { "httpGet": { "host": "localhost" } } } } ]`, }, }, }, }, }, expected: []v1.Container{ { Lifecycle: &v1.Lifecycle{ PostStart: &v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, PreStop: &v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, }, }, }, validators: []InitContainerValidator{assertLifeCycle}, </ins> }, } <ins> assertInitContainers := func(got, expected []v1.Container, validators []InitContainerValidator) error { if len(got) != len(expected) { return fmt.Errorf("different number of init container: got <%d>, expected <%d>", len(got), len(expected)) } for i := range got { g := &got[i] e := &expected[i] for _, validator := range validators { if err := validator(g, e); err != nil { return err } } } return nil } </ins> for _, test := range tests { rc := &test.rc obj2 := roundTrip(t, runtime.Object(rc))

What happened: kubelet doesn't seem to be able to manage containers started with CRI-O any more I'm testing kubelet + CRI-O on an Ubuntu 18.04 server installation, inside a qemu/kvm VM managed by Vagrant (see ), in order to be able to test k8s + KubeVirt + KataContainers, which are all compatible with CRI-O kubelet will obsess on starting new containers, which looks like and endless process, and the cluster never gets ready. For instance here are some syslogs : What you expected to happen: It used to work (see asciinema recording at ) a few days ago, but I fear some recent changes broke it. How to reproduce it (as minimally and precisely as possible): Testing with vagrant up --provider=libvirt on Linux with a clone of that commit: vagrant ssh into the VM and ps -edf f or tail -f /var/log/syslog would do... Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: qemu/kvm VM +OS (e.g: ): Ubuntu 18.04.2 LTS Kernel (e.g. ): Linux kubernetes-vagrant-01 4.15.0-48-generic -Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm as in Network plugin and version (if this is a network-related bug): Others:
/sig node
I can confirm this. If I start cri-o from a systemd service it fails with the above mentioned error message. Starting crio with without systemd works just fine. systemd file:
Thanks for your detailed report :) Just want to do some quick version confirming - you've been using k8s 1.14.1 both when it did and did not work? Any other version changes?
Not sure for before, when it used to work, but according to the asciinema recording mentioned above (), it seems the same. However, not sure for CRI-O, underlying OS stuff (systemd, kernel, etc.). Sorry.
I was looking for some information for days. I have exactly the same issue when I use podman + systemd on 18.04.2.


Question: } } <del> func TestSetDefaultReplicationControllerImagePullPolicy(t *testing.T) { containersWithoutPullPolicy, _ := json.Marshal([]map[string]interface{}{ { "name":  "install", "image": "busybox:latest", }, }) </del> <ins> type InitContainerValidator func(got, expected *v1.Container) error </ins> <del> containersWithPullPolicy, _ := json.Marshal([]map[string]interface{}{ { "name":            "install", "imagePullPolicy": "IfNotPresent", }, }) </del> <ins> func TestSetDefaultReplicationControllerInitContainers(t *testing.T) { assertEnvFieldRef := func(got, expected *v1.Container) error { if len(got.Env) != len(expected.Env) { return fmt.Errorf("different number of env: got <%v>, expected <%v>", len(got.Env), len(expected.Env)) } for j := range got.Env { ge := &got.Env[j] ee := &expected.Env[j] if ge.Name != ee.Name { return fmt.Errorf("different name of env: got <%v>, expected <%v>", ge.Name, ee.Name) } if ge.ValueFrom.FieldRef.APIVersion != ee.ValueFrom.FieldRef.APIVersion { return fmt.Errorf("different api version of FieldRef <%v>: got <%v>, expected <%v>", ge.Name, ge.ValueFrom.FieldRef.APIVersion, ee.ValueFrom.FieldRef.APIVersion) } } return nil } assertImagePullPolicy := func(got, expected *v1.Container) error { if got.ImagePullPolicy != expected.ImagePullPolicy { return fmt.Errorf("different image pull poicy: got <%v>, expected <%v>", got.ImagePullPolicy, expected.ImagePullPolicy) } return nil } assertContainerPort := func(got, expected *v1.Container) error { if len(got.Ports) != len(expected.Ports) { return fmt.Errorf("different number of ports: got <%v>, expected <%v>", len(got.Ports), len(expected.Ports)) } for i := range got.Ports { gp := &got.Ports[i] ep := &expected.Ports[i] if gp.Name != ep.Name { return fmt.Errorf("different name of port: got <%v>, expected <%v>", gp.Name, ep.Name) } if gp.Protocol != ep.Protocol { return fmt.Errorf("different port protocol <%v>: got <%v>, expected <%v>", gp.Name, gp.Protocol, ep.Protocol) } } return nil } assertResource := func(got, expected *v1.Container) error { if len(got.Resources.Limits) != len(expected.Resources.Limits) { return fmt.Errorf("different number of resources.Limits: got <%v>, expected <%v>", len(got.Resources.Limits), (expected.Resources.Limits)) } for k, v := range got.Resources.Limits { if ev, found := expected.Resources.Limits[v1.ResourceName(k)]; !found { return fmt.Errorf("failed to find resource <%v> in expected resources.Limits.", k) } else { if ev.Value() != v.Value() { return fmt.Errorf("different resource.Limits: got <%v>, expected <%v>.", v.Value(), ev.Value()) } } } if len(got.Resources.Requests) != len(expected.Resources.Requests) { return fmt.Errorf("different number of resources.Requests: got <%v>, expected <%v>", len(got.Resources.Requests), (expected.Resources.Requests)) } for k, v := range got.Resources.Requests { if ev, found := expected.Resources.Requests[v1.ResourceName(k)]; !found { return fmt.Errorf("failed to find resource <%v> in expected resources.Requests.", k) } else { if ev.Value() != v.Value() { return fmt.Errorf("different resource.Requests: got <%v>, expected <%v>.", v.Value(), ev.Value()) } } } return nil } assertProb := func(got, expected *v1.Container) error { // Assert LivenessProbe if got.LivenessProbe.Handler.HTTPGet.Path != expected.LivenessProbe.Handler.HTTPGet.Path || got.LivenessProbe.Handler.HTTPGet.Scheme != expected.LivenessProbe.Handler.HTTPGet.Scheme || got.LivenessProbe.FailureThreshold != expected.LivenessProbe.FailureThreshold || got.LivenessProbe.SuccessThreshold != expected.LivenessProbe.SuccessThreshold || got.LivenessProbe.PeriodSeconds != expected.LivenessProbe.PeriodSeconds || got.LivenessProbe.TimeoutSeconds != expected.LivenessProbe.TimeoutSeconds { return fmt.Errorf("different LivenessProbe: got <%v>, expected <%v>", got.LivenessProbe, expected.LivenessProbe) } // Assert ReadinessProbe if got.ReadinessProbe.Handler.HTTPGet.Path != expected.ReadinessProbe.Handler.HTTPGet.Path || got.ReadinessProbe.Handler.HTTPGet.Scheme != expected.ReadinessProbe.Handler.HTTPGet.Scheme || got.ReadinessProbe.FailureThreshold != expected.ReadinessProbe.FailureThreshold || got.ReadinessProbe.SuccessThreshold != expected.ReadinessProbe.SuccessThreshold || got.ReadinessProbe.PeriodSeconds != expected.ReadinessProbe.PeriodSeconds || got.ReadinessProbe.TimeoutSeconds != expected.ReadinessProbe.TimeoutSeconds { return fmt.Errorf("different ReadinessProbe: got <%v>, expected <%v>", got.ReadinessProbe, expected.ReadinessProbe) } return nil } assertLifeCycle := func(got, expected *v1.Container) error { if got.Lifecycle.PostStart.HTTPGet.Path != expected.Lifecycle.PostStart.HTTPGet.Path || got.Lifecycle.PostStart.HTTPGet.Scheme != expected.Lifecycle.PostStart.HTTPGet.Scheme { return fmt.Errorf("different LifeCycle: got <%v>, expected <%v>", got.Lifecycle, expected.Lifecycle) } return nil } cpu, _ := resource.ParseQuantity("100Gi") mem, _ := resource.ParseQuantity("100Mi") </ins> tests := []struct { <del> rc               v1.ReplicationController expectPullPolicy v1.PullPolicy </del> <ins> name       string rc         v1.ReplicationController expected   []v1.Container validators []InitContainerValidator </ins> }{ { <ins> name: "imagePullIPolicy", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "install", "image": "busybox" } ]`, }, }, }, }, }, expected: []v1.Container{ { ImagePullPolicy: v1.PullAlways, }, }, validators: []InitContainerValidator{assertImagePullPolicy}, }, { name: "FieldRef", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "env": [ { "name": "MY_POD_IP", "valueFrom": { "fieldRef": { "apiVersion": "", "fieldPath": "status.podIP" } } } ] } ]`, }, }, }, }, }, expected: []v1.Container{ { Env: []v1.EnvVar{ { Name: "MY_POD_IP", ValueFrom: &v1.EnvVarSource{ FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "status.podIP", }, }, }, }, }, }, validators: []InitContainerValidator{assertEnvFieldRef}, }, { name: "ContainerPort", </ins> rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ <del> "pod.beta.kubernetes.io/init-containers": string(containersWithoutPullPolicy), </del> <ins> "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "ports": [ { "name": "default" } ] } ]`, </ins> }, }, }, }, }, <del> expectPullPolicy: v1.PullAlways, </del> <ins> expected: []v1.Container{ { Ports: []v1.ContainerPort{ { Name:     "default", Protocol: v1.ProtocolTCP, }, }, }, }, validators: []InitContainerValidator{assertContainerPort}, </ins> }, { <ins> name: "Resources", </ins> rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ <del> "pod.beta.kubernetes.io/init-containers": string(containersWithPullPolicy), </del> <ins> "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "resources": { "limits": { "cpu": "100Gi", "memory": "100Mi" }, "requests": { "cpu": "100Gi", "memory": "100Mi" } } } ]`, </ins> }, }, }, }, }, <del> expectPullPolicy: v1.PullIfNotPresent, </del> <ins> expected: []v1.Container{ { Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceCPU:    cpu, v1.ResourceMemory: mem, }, Requests: v1.ResourceList{ v1.ResourceCPU:    cpu, v1.ResourceMemory: mem, }, }, }, }, validators: []InitContainerValidator{assertResource}, }, { name: "Prob", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "livenessProbe": { "httpGet": { "host": "localhost" } }, "readinessProbe": { "httpGet": { "host": "localhost" } } } ]`, }, }, }, }, }, expected: []v1.Container{ { LivenessProbe: &v1.Probe{ Handler: v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, TimeoutSeconds:   1, PeriodSeconds:    10, SuccessThreshold: 1, FailureThreshold: 3, }, ReadinessProbe: &v1.Probe{ Handler: v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, TimeoutSeconds:   1, PeriodSeconds:    10, SuccessThreshold: 1, FailureThreshold: 3, }, }, }, validators: []InitContainerValidator{assertProb}, }, { name: "LifeCycle", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "lifecycle": { "postStart": { "httpGet": { "host": "localhost" } }, "preStop": { "httpGet": { "host": "localhost" } } } } ]`, }, }, }, }, }, expected: []v1.Container{ { Lifecycle: &v1.Lifecycle{ PostStart: &v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, PreStop: &v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, }, }, }, validators: []InitContainerValidator{assertLifeCycle}, </ins> }, } <ins> assertInitContainers := func(got, expected []v1.Container, validators []InitContainerValidator) error { if len(got) != len(expected) { return fmt.Errorf("different number of init container: got <%d>, expected <%d>", len(got), len(expected)) } for i := range got { g := &got[i] e := &expected[i] for _, validator := range validators { if err := validator(g, e); err != nil { return err } } } return nil } </ins> for _, test := range tests { rc := &test.rc obj2 := roundTrip(t, runtime.Object(rc))

It is the same for both 14.15 and 14.18 kernels. Podman without systemd runs fine. Podman with systemd fails at runc with the same error podman[]: open /dev/stderr: no such device or address podman[]: NAME: runc - Open Container Initiative runtime podman[]: runc is a command line client for running applications packaged according to ... podman[]: time="2019-05-11T13:04:59-07:00" level=error msg="open /dev/stderr: no such device or addressn" podman[]: open /dev/stderr: no such device or address
I've tried to reproduce it on 18.04.2 using Podman from but I am not able to reproduce it. Do you have any reproducer? How did you install Podman?
I used ceph unit files to run ceph in podman rather than docker. Systemd unit successfully starts docker but fails with podman. I can successfully run the same podman container run command from the command line. How were you trying to replicate this? Can you post your unit file?
Hi, I'am having the same problem with Kubernetes with CRI-O. After joining a second worker node it fails to start flannel. gives the following error: What I've tried: Installing previous version of Kubernetes (from to ) Installing previous version of crio (from to ) Building runc myself from their github page and selecting this in the Running it on Ubunut 18.04.2 Issue: (the one giuseppe linked)
I wonder if the issue is with systemd and runc. In both cases, crio and podman, I see runc help output which tells me that something is not being passed to runc that it needs.
I've only used so it is different than your test and you confirmed it works from the CLI for you as well. Could you share your service file so I test it?
I've tried systemd and cgroupfs but no luck so far.
so it seems that runc fails to re-open . That seems like a runc issue, as it doesn't really need to reopen it. To fix the failure, you could use something like:
Just an example.


Question: } } <del> func TestSetDefaultReplicationControllerImagePullPolicy(t *testing.T) { containersWithoutPullPolicy, _ := json.Marshal([]map[string]interface{}{ { "name":  "install", "image": "busybox:latest", }, }) </del> <ins> type InitContainerValidator func(got, expected *v1.Container) error </ins> <del> containersWithPullPolicy, _ := json.Marshal([]map[string]interface{}{ { "name":            "install", "imagePullPolicy": "IfNotPresent", }, }) </del> <ins> func TestSetDefaultReplicationControllerInitContainers(t *testing.T) { assertEnvFieldRef := func(got, expected *v1.Container) error { if len(got.Env) != len(expected.Env) { return fmt.Errorf("different number of env: got <%v>, expected <%v>", len(got.Env), len(expected.Env)) } for j := range got.Env { ge := &got.Env[j] ee := &expected.Env[j] if ge.Name != ee.Name { return fmt.Errorf("different name of env: got <%v>, expected <%v>", ge.Name, ee.Name) } if ge.ValueFrom.FieldRef.APIVersion != ee.ValueFrom.FieldRef.APIVersion { return fmt.Errorf("different api version of FieldRef <%v>: got <%v>, expected <%v>", ge.Name, ge.ValueFrom.FieldRef.APIVersion, ee.ValueFrom.FieldRef.APIVersion) } } return nil } assertImagePullPolicy := func(got, expected *v1.Container) error { if got.ImagePullPolicy != expected.ImagePullPolicy { return fmt.Errorf("different image pull poicy: got <%v>, expected <%v>", got.ImagePullPolicy, expected.ImagePullPolicy) } return nil } assertContainerPort := func(got, expected *v1.Container) error { if len(got.Ports) != len(expected.Ports) { return fmt.Errorf("different number of ports: got <%v>, expected <%v>", len(got.Ports), len(expected.Ports)) } for i := range got.Ports { gp := &got.Ports[i] ep := &expected.Ports[i] if gp.Name != ep.Name { return fmt.Errorf("different name of port: got <%v>, expected <%v>", gp.Name, ep.Name) } if gp.Protocol != ep.Protocol { return fmt.Errorf("different port protocol <%v>: got <%v>, expected <%v>", gp.Name, gp.Protocol, ep.Protocol) } } return nil } assertResource := func(got, expected *v1.Container) error { if len(got.Resources.Limits) != len(expected.Resources.Limits) { return fmt.Errorf("different number of resources.Limits: got <%v>, expected <%v>", len(got.Resources.Limits), (expected.Resources.Limits)) } for k, v := range got.Resources.Limits { if ev, found := expected.Resources.Limits[v1.ResourceName(k)]; !found { return fmt.Errorf("failed to find resource <%v> in expected resources.Limits.", k) } else { if ev.Value() != v.Value() { return fmt.Errorf("different resource.Limits: got <%v>, expected <%v>.", v.Value(), ev.Value()) } } } if len(got.Resources.Requests) != len(expected.Resources.Requests) { return fmt.Errorf("different number of resources.Requests: got <%v>, expected <%v>", len(got.Resources.Requests), (expected.Resources.Requests)) } for k, v := range got.Resources.Requests { if ev, found := expected.Resources.Requests[v1.ResourceName(k)]; !found { return fmt.Errorf("failed to find resource <%v> in expected resources.Requests.", k) } else { if ev.Value() != v.Value() { return fmt.Errorf("different resource.Requests: got <%v>, expected <%v>.", v.Value(), ev.Value()) } } } return nil } assertProb := func(got, expected *v1.Container) error { // Assert LivenessProbe if got.LivenessProbe.Handler.HTTPGet.Path != expected.LivenessProbe.Handler.HTTPGet.Path || got.LivenessProbe.Handler.HTTPGet.Scheme != expected.LivenessProbe.Handler.HTTPGet.Scheme || got.LivenessProbe.FailureThreshold != expected.LivenessProbe.FailureThreshold || got.LivenessProbe.SuccessThreshold != expected.LivenessProbe.SuccessThreshold || got.LivenessProbe.PeriodSeconds != expected.LivenessProbe.PeriodSeconds || got.LivenessProbe.TimeoutSeconds != expected.LivenessProbe.TimeoutSeconds { return fmt.Errorf("different LivenessProbe: got <%v>, expected <%v>", got.LivenessProbe, expected.LivenessProbe) } // Assert ReadinessProbe if got.ReadinessProbe.Handler.HTTPGet.Path != expected.ReadinessProbe.Handler.HTTPGet.Path || got.ReadinessProbe.Handler.HTTPGet.Scheme != expected.ReadinessProbe.Handler.HTTPGet.Scheme || got.ReadinessProbe.FailureThreshold != expected.ReadinessProbe.FailureThreshold || got.ReadinessProbe.SuccessThreshold != expected.ReadinessProbe.SuccessThreshold || got.ReadinessProbe.PeriodSeconds != expected.ReadinessProbe.PeriodSeconds || got.ReadinessProbe.TimeoutSeconds != expected.ReadinessProbe.TimeoutSeconds { return fmt.Errorf("different ReadinessProbe: got <%v>, expected <%v>", got.ReadinessProbe, expected.ReadinessProbe) } return nil } assertLifeCycle := func(got, expected *v1.Container) error { if got.Lifecycle.PostStart.HTTPGet.Path != expected.Lifecycle.PostStart.HTTPGet.Path || got.Lifecycle.PostStart.HTTPGet.Scheme != expected.Lifecycle.PostStart.HTTPGet.Scheme { return fmt.Errorf("different LifeCycle: got <%v>, expected <%v>", got.Lifecycle, expected.Lifecycle) } return nil } cpu, _ := resource.ParseQuantity("100Gi") mem, _ := resource.ParseQuantity("100Mi") </ins> tests := []struct { <del> rc               v1.ReplicationController expectPullPolicy v1.PullPolicy </del> <ins> name       string rc         v1.ReplicationController expected   []v1.Container validators []InitContainerValidator </ins> }{ { <ins> name: "imagePullIPolicy", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "install", "image": "busybox" } ]`, }, }, }, }, }, expected: []v1.Container{ { ImagePullPolicy: v1.PullAlways, }, }, validators: []InitContainerValidator{assertImagePullPolicy}, }, { name: "FieldRef", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "env": [ { "name": "MY_POD_IP", "valueFrom": { "fieldRef": { "apiVersion": "", "fieldPath": "status.podIP" } } } ] } ]`, }, }, }, }, }, expected: []v1.Container{ { Env: []v1.EnvVar{ { Name: "MY_POD_IP", ValueFrom: &v1.EnvVarSource{ FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "status.podIP", }, }, }, }, }, }, validators: []InitContainerValidator{assertEnvFieldRef}, }, { name: "ContainerPort", </ins> rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ <del> "pod.beta.kubernetes.io/init-containers": string(containersWithoutPullPolicy), </del> <ins> "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "ports": [ { "name": "default" } ] } ]`, </ins> }, }, }, }, }, <del> expectPullPolicy: v1.PullAlways, </del> <ins> expected: []v1.Container{ { Ports: []v1.ContainerPort{ { Name:     "default", Protocol: v1.ProtocolTCP, }, }, }, }, validators: []InitContainerValidator{assertContainerPort}, </ins> }, { <ins> name: "Resources", </ins> rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ <del> "pod.beta.kubernetes.io/init-containers": string(containersWithPullPolicy), </del> <ins> "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "resources": { "limits": { "cpu": "100Gi", "memory": "100Mi" }, "requests": { "cpu": "100Gi", "memory": "100Mi" } } } ]`, </ins> }, }, }, }, }, <del> expectPullPolicy: v1.PullIfNotPresent, </del> <ins> expected: []v1.Container{ { Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceCPU:    cpu, v1.ResourceMemory: mem, }, Requests: v1.ResourceList{ v1.ResourceCPU:    cpu, v1.ResourceMemory: mem, }, }, }, }, validators: []InitContainerValidator{assertResource}, }, { name: "Prob", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "livenessProbe": { "httpGet": { "host": "localhost" } }, "readinessProbe": { "httpGet": { "host": "localhost" } } } ]`, }, }, }, }, }, expected: []v1.Container{ { LivenessProbe: &v1.Probe{ Handler: v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, TimeoutSeconds:   1, PeriodSeconds:    10, SuccessThreshold: 1, FailureThreshold: 3, }, ReadinessProbe: &v1.Probe{ Handler: v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, TimeoutSeconds:   1, PeriodSeconds:    10, SuccessThreshold: 1, FailureThreshold: 3, }, }, }, validators: []InitContainerValidator{assertProb}, }, { name: "LifeCycle", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "lifecycle": { "postStart": { "httpGet": { "host": "localhost" } }, "preStop": { "httpGet": { "host": "localhost" } } } } ]`, }, }, }, }, }, expected: []v1.Container{ { Lifecycle: &v1.Lifecycle{ PostStart: &v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, PreStop: &v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, }, }, }, validators: []InitContainerValidator{assertLifeCycle}, </ins> }, } <ins> assertInitContainers := func(got, expected []v1.Container, validators []InitContainerValidator) error { if len(got) != len(expected) { return fmt.Errorf("different number of init container: got <%d>, expected <%d>", len(got), len(expected)) } for i := range got { g := &got[i] e := &expected[i] for _, validator := range validators { if err := validator(g, e); err != nil { return err } } } return nil } </ins> for _, test := range tests { rc := &test.rc obj2 := roundTrip(t, runtime.Object(rc))

I use ceph-ansible to install cluster so it may need to install additional daemons like mon before running mgr but the error, in this case, should be different, [Unit] Description=Ceph Manager [Service] EnvironmentFile=-/etc/environment ExecStartPre=-/usr/bin/podman stop ceph-mgr-storage2n1-la ExecStartPre=-/usr/bin/podman rm ceph-mgr-storage2n1-la ExecStart=/usr/bin/podman run --rm --net=host  --memory=16G  --cpus=4  -v /var/lib/ceph:/var/lib/ceph:z  -v /etc/ceph:/etc/ceph:z  -v /var/run/ceph:/var/run/ceph:z  -v /etc/localtime:/etc/localtime:ro  -e CLUSTER=ceph  -e CEPHDAEMON=MGR  -e CONTAINERIMAGE=  {{ cephmgrdockerextraenv }}  --name=ceph-mgr-storage2n1-la  ExecStopPost=-/usr/bin/podman stop ceph-mgr-storage2n1-la Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi- delighted <:
BTW, I projectatomic ppa and used apt-get -y podman to install. My podman version is 1.3.0 Sasha Litvak — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
Corrected file.


Question: } } <del> func TestSetDefaultReplicationControllerImagePullPolicy(t *testing.T) { containersWithoutPullPolicy, _ := json.Marshal([]map[string]interface{}{ { "name":  "install", "image": "busybox:latest", }, }) </del> <ins> type InitContainerValidator func(got, expected *v1.Container) error </ins> <del> containersWithPullPolicy, _ := json.Marshal([]map[string]interface{}{ { "name":            "install", "imagePullPolicy": "IfNotPresent", }, }) </del> <ins> func TestSetDefaultReplicationControllerInitContainers(t *testing.T) { assertEnvFieldRef := func(got, expected *v1.Container) error { if len(got.Env) != len(expected.Env) { return fmt.Errorf("different number of env: got <%v>, expected <%v>", len(got.Env), len(expected.Env)) } for j := range got.Env { ge := &got.Env[j] ee := &expected.Env[j] if ge.Name != ee.Name { return fmt.Errorf("different name of env: got <%v>, expected <%v>", ge.Name, ee.Name) } if ge.ValueFrom.FieldRef.APIVersion != ee.ValueFrom.FieldRef.APIVersion { return fmt.Errorf("different api version of FieldRef <%v>: got <%v>, expected <%v>", ge.Name, ge.ValueFrom.FieldRef.APIVersion, ee.ValueFrom.FieldRef.APIVersion) } } return nil } assertImagePullPolicy := func(got, expected *v1.Container) error { if got.ImagePullPolicy != expected.ImagePullPolicy { return fmt.Errorf("different image pull poicy: got <%v>, expected <%v>", got.ImagePullPolicy, expected.ImagePullPolicy) } return nil } assertContainerPort := func(got, expected *v1.Container) error { if len(got.Ports) != len(expected.Ports) { return fmt.Errorf("different number of ports: got <%v>, expected <%v>", len(got.Ports), len(expected.Ports)) } for i := range got.Ports { gp := &got.Ports[i] ep := &expected.Ports[i] if gp.Name != ep.Name { return fmt.Errorf("different name of port: got <%v>, expected <%v>", gp.Name, ep.Name) } if gp.Protocol != ep.Protocol { return fmt.Errorf("different port protocol <%v>: got <%v>, expected <%v>", gp.Name, gp.Protocol, ep.Protocol) } } return nil } assertResource := func(got, expected *v1.Container) error { if len(got.Resources.Limits) != len(expected.Resources.Limits) { return fmt.Errorf("different number of resources.Limits: got <%v>, expected <%v>", len(got.Resources.Limits), (expected.Resources.Limits)) } for k, v := range got.Resources.Limits { if ev, found := expected.Resources.Limits[v1.ResourceName(k)]; !found { return fmt.Errorf("failed to find resource <%v> in expected resources.Limits.", k) } else { if ev.Value() != v.Value() { return fmt.Errorf("different resource.Limits: got <%v>, expected <%v>.", v.Value(), ev.Value()) } } } if len(got.Resources.Requests) != len(expected.Resources.Requests) { return fmt.Errorf("different number of resources.Requests: got <%v>, expected <%v>", len(got.Resources.Requests), (expected.Resources.Requests)) } for k, v := range got.Resources.Requests { if ev, found := expected.Resources.Requests[v1.ResourceName(k)]; !found { return fmt.Errorf("failed to find resource <%v> in expected resources.Requests.", k) } else { if ev.Value() != v.Value() { return fmt.Errorf("different resource.Requests: got <%v>, expected <%v>.", v.Value(), ev.Value()) } } } return nil } assertProb := func(got, expected *v1.Container) error { // Assert LivenessProbe if got.LivenessProbe.Handler.HTTPGet.Path != expected.LivenessProbe.Handler.HTTPGet.Path || got.LivenessProbe.Handler.HTTPGet.Scheme != expected.LivenessProbe.Handler.HTTPGet.Scheme || got.LivenessProbe.FailureThreshold != expected.LivenessProbe.FailureThreshold || got.LivenessProbe.SuccessThreshold != expected.LivenessProbe.SuccessThreshold || got.LivenessProbe.PeriodSeconds != expected.LivenessProbe.PeriodSeconds || got.LivenessProbe.TimeoutSeconds != expected.LivenessProbe.TimeoutSeconds { return fmt.Errorf("different LivenessProbe: got <%v>, expected <%v>", got.LivenessProbe, expected.LivenessProbe) } // Assert ReadinessProbe if got.ReadinessProbe.Handler.HTTPGet.Path != expected.ReadinessProbe.Handler.HTTPGet.Path || got.ReadinessProbe.Handler.HTTPGet.Scheme != expected.ReadinessProbe.Handler.HTTPGet.Scheme || got.ReadinessProbe.FailureThreshold != expected.ReadinessProbe.FailureThreshold || got.ReadinessProbe.SuccessThreshold != expected.ReadinessProbe.SuccessThreshold || got.ReadinessProbe.PeriodSeconds != expected.ReadinessProbe.PeriodSeconds || got.ReadinessProbe.TimeoutSeconds != expected.ReadinessProbe.TimeoutSeconds { return fmt.Errorf("different ReadinessProbe: got <%v>, expected <%v>", got.ReadinessProbe, expected.ReadinessProbe) } return nil } assertLifeCycle := func(got, expected *v1.Container) error { if got.Lifecycle.PostStart.HTTPGet.Path != expected.Lifecycle.PostStart.HTTPGet.Path || got.Lifecycle.PostStart.HTTPGet.Scheme != expected.Lifecycle.PostStart.HTTPGet.Scheme { return fmt.Errorf("different LifeCycle: got <%v>, expected <%v>", got.Lifecycle, expected.Lifecycle) } return nil } cpu, _ := resource.ParseQuantity("100Gi") mem, _ := resource.ParseQuantity("100Mi") </ins> tests := []struct { <del> rc               v1.ReplicationController expectPullPolicy v1.PullPolicy </del> <ins> name       string rc         v1.ReplicationController expected   []v1.Container validators []InitContainerValidator </ins> }{ { <ins> name: "imagePullIPolicy", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "install", "image": "busybox" } ]`, }, }, }, }, }, expected: []v1.Container{ { ImagePullPolicy: v1.PullAlways, }, }, validators: []InitContainerValidator{assertImagePullPolicy}, }, { name: "FieldRef", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "env": [ { "name": "MY_POD_IP", "valueFrom": { "fieldRef": { "apiVersion": "", "fieldPath": "status.podIP" } } } ] } ]`, }, }, }, }, }, expected: []v1.Container{ { Env: []v1.EnvVar{ { Name: "MY_POD_IP", ValueFrom: &v1.EnvVarSource{ FieldRef: &v1.ObjectFieldSelector{ APIVersion: "v1", FieldPath:  "status.podIP", }, }, }, }, }, }, validators: []InitContainerValidator{assertEnvFieldRef}, }, { name: "ContainerPort", </ins> rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ <del> "pod.beta.kubernetes.io/init-containers": string(containersWithoutPullPolicy), </del> <ins> "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "ports": [ { "name": "default" } ] } ]`, </ins> }, }, }, }, }, <del> expectPullPolicy: v1.PullAlways, </del> <ins> expected: []v1.Container{ { Ports: []v1.ContainerPort{ { Name:     "default", Protocol: v1.ProtocolTCP, }, }, }, }, validators: []InitContainerValidator{assertContainerPort}, </ins> }, { <ins> name: "Resources", </ins> rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ <del> "pod.beta.kubernetes.io/init-containers": string(containersWithPullPolicy), </del> <ins> "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "resources": { "limits": { "cpu": "100Gi", "memory": "100Mi" }, "requests": { "cpu": "100Gi", "memory": "100Mi" } } } ]`, </ins> }, }, }, }, }, <del> expectPullPolicy: v1.PullIfNotPresent, </del> <ins> expected: []v1.Container{ { Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceCPU:    cpu, v1.ResourceMemory: mem, }, Requests: v1.ResourceList{ v1.ResourceCPU:    cpu, v1.ResourceMemory: mem, }, }, }, }, validators: []InitContainerValidator{assertResource}, }, { name: "Prob", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "livenessProbe": { "httpGet": { "host": "localhost" } }, "readinessProbe": { "httpGet": { "host": "localhost" } } } ]`, }, }, }, }, }, expected: []v1.Container{ { LivenessProbe: &v1.Probe{ Handler: v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, TimeoutSeconds:   1, PeriodSeconds:    10, SuccessThreshold: 1, FailureThreshold: 3, }, ReadinessProbe: &v1.Probe{ Handler: v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, TimeoutSeconds:   1, PeriodSeconds:    10, SuccessThreshold: 1, FailureThreshold: 3, }, }, }, validators: []InitContainerValidator{assertProb}, }, { name: "LifeCycle", rc: v1.ReplicationController{ Spec: v1.ReplicationControllerSpec{ Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "pod.beta.kubernetes.io/init-containers": ` [ { "name": "fun", "image": "alpine", "lifecycle": { "postStart": { "httpGet": { "host": "localhost" } }, "preStop": { "httpGet": { "host": "localhost" } } } } ]`, }, }, }, }, }, expected: []v1.Container{ { Lifecycle: &v1.Lifecycle{ PostStart: &v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, PreStop: &v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path:   "/", Scheme: v1.URISchemeHTTP, }, }, }, }, }, validators: []InitContainerValidator{assertLifeCycle}, </ins> }, } <ins> assertInitContainers := func(got, expected []v1.Container, validators []InitContainerValidator) error { if len(got) != len(expected) { return fmt.Errorf("different number of init container: got <%d>, expected <%d>", len(got), len(expected)) } for i := range got { g := &got[i] e := &expected[i] for _, validator := range validators { if err := validator(g, e); err != nil { return err } } } return nil } </ins> for _, test := range tests { rc := &test.rc obj2 := roundTrip(t, runtime.Object(rc))

[Unit] Description=Ceph Manager [Service] EnvironmentFile=-/etc/environment ExecStartPre=-/usr/bin/podman stop ceph-mgr-storage2n1-la ExecStartPre=-/usr/bin/podman rm ceph-mgr-storage2n1-la ExecStart=/usr/bin/podman run --rm --net=host  --memory=16G  --cpus=4  -v /var/lib/ceph:/var/lib/ceph:z  -v /etc/ceph:/etc/ceph:z  -v /var/run/ceph:/var/run/ceph:z  -v /etc/localtime:/etc/localtime:ro  -e CLUSTER=ceph  -e CEPHDAEMON=MGR  -e CONTAINERIMAGE=  --name=ceph-mgr-storage2n1-la  ExecStopPost=-/usr/bin/podman stop ceph-mgr-storage2n1-la Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi- Giuseppe Scrivano <:
Would it have an effect system wide? Can I add it to a unit file or unit override instead? Giuseppe Scrivano <:
yes, I think it is better to add it only to the unit file
Why would runc work outside of systemd? And if it needs stderr, giving it Null in settings will redirect stderr to Null I presume? On Tue, May 14, 2019, 9:46 AM Giuseppe Scrivano <:
Running into same issues on Ubuntu 18.04.2 using cri-o-1.13 via PPA.
PR here: opencontainers/runc
Until it is avaialble with project atomic and such, can I use a null workaround? Giuseppe Scrivano <:
yes, but you will lose anything written to stderr. You could probably also use a file, like
What about actually mounting dev and using /dev/stderr ? On Tue, May 14, 2019, 3:11 PM Giuseppe Scrivano <:
is a symlink to . It is present, the issue here is that is a socket. The fix for runc got merged, could you use an updated runc?
I tried apt update/upgrade and restarting crio, but I'm still getting the issue.