Question: github.com/mitchellh/go-wordwrap # github.com/mitchellh/mapstructure v1.1.2 => github.com/mitchellh/mapstructure v1.1.2 github.com/mitchellh/mapstructure <del> # github.com/moby/ipvs v1.0.0 => github.com/moby/ipvs v1.0.0 </del> <ins> # github.com/moby/ipvs v1.0.1 => github.com/moby/ipvs v1.0.1 </ins> github.com/moby/ipvs # github.com/moby/term v0.0.0-20200312100748-672ec06f55cd => github.com/moby/term v0.0.0-20200312100748-672ec06f55cd github.com/moby/term

I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO.


Question: github.com/mitchellh/go-wordwrap # github.com/mitchellh/mapstructure v1.1.2 => github.com/mitchellh/mapstructure v1.1.2 github.com/mitchellh/mapstructure <del> # github.com/moby/ipvs v1.0.0 => github.com/moby/ipvs v1.0.0 </del> <ins> # github.com/moby/ipvs v1.0.1 => github.com/moby/ipvs v1.0.1 </ins> github.com/moby/ipvs # github.com/moby/term v0.0.0-20200312100748-672ec06f55cd => github.com/moby/term v0.0.0-20200312100748-672ec06f55cd github.com/moby/term

However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).


Question: github.com/mitchellh/go-wordwrap # github.com/mitchellh/mapstructure v1.1.2 => github.com/mitchellh/mapstructure v1.1.2 github.com/mitchellh/mapstructure <del> # github.com/moby/ipvs v1.0.0 => github.com/moby/ipvs v1.0.0 </del> <ins> # github.com/moby/ipvs v1.0.1 => github.com/moby/ipvs v1.0.1 </ins> github.com/moby/ipvs # github.com/moby/term v0.0.0-20200312100748-672ec06f55cd => github.com/moby/term v0.0.0-20200312100748-672ec06f55cd github.com/moby/term

FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks


Question: "//pkg/cloudprovider/providers/vsphere:go_default_library", "//pkg/cloudprovider/providers/vsphere/vclib:go_default_library", "//pkg/kubelet/apis:go_default_library", <ins> "//pkg/kubelet/metrics:go_default_library", </ins> "//pkg/volume/util/volumehelper:go_default_library", "//test/e2e/framework:go_default_library", "//test/e2e/framework/metrics:go_default_library",

In 1.8, we a new feature to expose PVC information through summary API so user can check the volume belongs to which PVC. We should add e2e test in under test/e2enode.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
GitHub didn't allow me to assign the following users: vkamra. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Can we please move this out of 1.8 milestone?
- I'm looking into adding this but yes we should move this out of 1.8. I don't have permissions to do that though.
ack Can we please move this out of 1.8?
Sorry am not sure why was this removed from 1.8 milestone. Are we not aiming to add the tests in 1.8? Or Are we disabling the feature or are we fine shipping it without any e2e tests in 1.8?
The test is in . I think we can keep this feature in 1.8


Question: // If not found, returns (nil, nil) func (gce *GCECloud) findDiskByName(diskName string, zone string) (*GCEDisk, error) { mc := newDiskMetricContext("get", zone) <del> disk, err := gce.service.Disks.Get(gce.projectID, zone, diskName).Do() </del> <ins> disk, err := gce.manager.GetDisk(gce.projectID, zone, diskName) </ins> if err == nil { d := &GCEDisk{ Zone: lastComponent(disk.Zone),

Unit tests are needed for the GCE cloud provider disk interface.
Assigned to
/assign
GitHub didn't allow me to assign the following users: verult. Note that only can be assigned.. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details


Question: core "k8s.io/client-go/testing" "k8s.io/client-go/tools/record" featuregatetesting "k8s.io/component-base/featuregate/testing" <ins> "k8s.io/klog/v2" </ins> "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/kubelet/volumemanager/cache" "k8s.io/kubernetes/pkg/volume"

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --N/A Triage instead: Data race. links to appreciated links to specific failures in spyglass appreciated specific failure <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
previously:
The data race is for this time . fixed only by using a copy.
/assign
GitHub didn't allow me to assign the following users: Haleygo. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Seems noone work here, try to reproduce and fix. /assign
kubernetes# ./stress -p 20 - "TestRunPositive_VolumeMountControllerAttachEnabledRace" 15m25s: runs so far, 0 failures :( I cannot reproduce it, does anyone have better suggestions?
? I have no better idea.
-p 200 ,-p 300, I reproduced this fault (and another DATA RACE). I continued to test after adding deepCopy and found that the stress did not seem to recognize my modification, because the DATA RACE error line was the same as the original, I was a little bit confused....
confused: Use to compile stress, Add Info log to prove that the code has been modified: Call and fail the increased log is not printed (one timeout). As a result, I cannot verify whether I have fixed this problem!
I forgot to update , now the test continues. :-)


Question: apiServiceFailed := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "failed", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy"}, }, }

When installing the metrics-server to implement the API and the prometheus-adapter to implement the API, it is possible to make the OpenAPI handler fail to initialize with this error log in apiserver: This will result in kubectl discovery calls to /openapi/v2 to fail with a 503: The OpenAPI handler should not fail to initialize if both and are serving the same path for different API groups/versions. I can provide exact installation steps if needed, however, a high-level repro is: install metrics-server to serve install prometheus-adapter to serve ensure resourceRules is populated in the prometheus-adapter config so that resource metrics are also served by restarting apiserver will result in the OpenAPI failure and 503 responses to /openapi/v2 path No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
Please let me know if I should reopen the issue in
/sig api-machinery
/sig instrumentation [metrics apis]
Would you mind taking a look when have time? Thank you^^ /triage accepted
I have a question. should be serving under . Why does an apiservice serving another gvk register paths under ? cc we've made the assumption that an APIService (such as metrics-server or prometheus-adapter) serves only gvks under its own registered group version. I'll do some more investigation but just wondering if prometheus doesn't follow that assumption?
the prometheus-adapter can also be configured to serve and replace the metrics-server by enabling these metrics, However, if the prometheus-adapter fronts and metrics-server fronts , it does seem like the correct behavior would be for prometheus-adapter to serve until an operator updates the APIService to reference prometheus-adapter rather than metrics-server. We're seeing this issue occur for upgrades between 1.27 and 1.28 so it does look this path duplication did not cause the OpenAPI handler to fail on 1.27.
I just upgraded from 1.27 to 1.28 and running into this problem. Some kubectl apply operations fail, and ArgoCD cannot load its custom resources. Any suggested workaround? Would removing the from the prometheus configuration resolve it? Or perhaps disable metrics-server?
Simply looking at the code diff between 1.27 and 1.28 I can't pinpoint the cause.


Question: apiServiceFailed := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "failed", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy"}, }, }

Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question.


Question: apiServiceFailed := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "failed", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy"}, }, }

In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.


Question: apiServiceFailed := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "failed", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy"}, }, }

I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in


Question: "fmt" "k8s.io/api/core/v1" <ins> "k8s.io/kubernetes/pkg/kubelet/util/format" </ins> "k8s.io/kubernetes/pkg/securitycontext" )

when deploying a container that has runAsNonRoot but runs as root this error appears, but for pods with lots of containers it can be confusing which container failed, especially if there are 3-rd party webhooks that inject containers please add the container name to the message, making it
/sig api-machinery
/assign I can try this fix
I just submitted a pr, can you review it for me, is it as you expected?
thx, looks good!
/remove-sig api-machiner /sig node /area kubelet
Those labels are not set on the issue: <detailsIn response to : /remove-sig api-machiner /sig node /area kubelet Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig api-machinery


Question: func NewResourceExpirationEvaluator(currentVersion apimachineryversion.Info) (ResourceExpirationEvaluator, error) { ret := &resourceExpirationEvaluator{ <del> // TODO https://github.com/kubernetes/kubernetes/issues/111972 set this back to false after beta is tagged. strictRemovedHandlingInAlpha: true, </del> <ins> strictRemovedHandlingInAlpha: false, </ins> } if len(currentVersion.Major) > 0 { currentMajor64, err := strconv.ParseInt(currentVersion.Major, 10, 32)

once the we hit a beta release, we can reset the default value for strict handling to false to prep for 1.26. /kind cleanup /priority important-soon /milestone v1.26 /assign /cc
/sig api-machinery
/triage accepted
Looks like Nov 10 ( ) <img width="946" alt="image" src="https://user-"
/milestone v1.26
please don't drop the milestone from issues without asking first... this is why CI is failing on master currently (my fault for not remembering to resolve this between beta and rc, sorry)
I removed all with code freeze and the ones back that got approved as exception request / are big fixes. Missed this one; sorry.


Question: if len(a.GetSubresource()) != 0 { return nil } switch a.GetResource().GroupResource() { case podResource: if operation == admission.Create || operation == admission.Update {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the documentation at the , I tried enabling quota for a namespace other than but I was still not able to create a pod with priorityClass in my namespace with following error: Error from server (Forbidden): error when creating "": pods "memory-demo" is forbidden: pods with system-cluster-critical priorityClass is not permitted in my-namespace namespace What you expected to happen: Expected pod to be created with namespace where I allowed quota. I can see that both ResourceQuota and Priority Plugins are enabled from logs. I0408 17:15:08. ] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook. I0408 17:15:08. ] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. How to reproduce it (as minimally and precisely as possible): Follow the steps at Anything else we need to know?: I think, the issue is with priorityPlugin where we are rejecting instead of letting resourceQuota plugin to handle the admission when is enabled. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig scheduling /cc
Same happens to me. Correction, it does not allow to use the default permissions outside of kube-system
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
See also:
/remove-lifecycle stale
/assign Per offline discussion with
Same issue here.


Question: if len(a.GetSubresource()) != 0 { return nil } switch a.GetResource().GroupResource() { case podResource: if operation == admission.Create || operation == admission.Update {

We are using namespaces other than kube-system for system pods as it is getting too crowded, therefore we would like to run cluster-node-critical pods in other namespaces.
Waiting for this to be available myself. No reason my Node Exporter dataset has to run in the namespace instead of my namespace.


Question: t.Errorf("mismatch in expected bandwidth. expected %v got %v", expectedBandwidth, inputConfig.RuntimeConfig.Bandwidth) } <ins> expectedIpRange := [][]map[string]interface{}{ { {"subnet": "10.0.2.0/24"}, }, } if !reflect.DeepEqual(inputConfig.RuntimeConfig.IpRanges, expectedIpRange) { t.Errorf("mismatch in expected ipRange. expected %v got %v", expectedIpRange, inputConfig.RuntimeConfig.IpRanges) } </ins> // Get its IP address status, err := plug.GetPodNetworkStatus("podNamespace", "podName", containerID) if err != nil {

/kind feature The CNI "capability arguments" parameter - currently used to pass port mappings - has a few new capabilities: bandwidth limiting and network address management. The dockershim's networking driver should use them. The network address is just passing the PodCIDR dynamically to the plugin, so it no longer has to retrieve it from the APIServer. Bandwidth limiting is already implemented in Kubenet. It is configured via annotations. There is no upgrade risk in providing these values; non-compliant plugins will simply ignore them. Administrators need to explicitly enable each capability individually in their CNI configuration file. To avoid skew between runtimes, the other runtimes should also be upgraded to pass these. We may also wish to look in to a way to add this more directly to the CRI, so we don't have this skew problem, but that's a different issue.
/sig network
/assign
fyi.
Looks like this was never done for CRI?
cc


Question: PREFIX = gcr.io/google-containers IMAGE = fluentd-elasticsearch <del> TAG = v2.0.0 </del> <ins> TAG = v2.0.1 </ins> build: docker build --pull -t $(PREFIX)/$(IMAGE):$(TAG) .

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: [root EFK]# kubectl logs -f fluentd-es-jrsf7 -n kube-system /var/lib/gems/2.1.0/gems/ffi- block in ffilib': Could not open library '': cannot open shared object file: No such file or directory. (LoadError) Could not open library '': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': libsystemd- cannot open shared object file: No such file or directory from /var/lib/gems/2.1.0/gems/ffi-map' from /var/lib/gems/2.1.0/gems/ffi- ffilib' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Native' from /var/lib/gems/2.1.0/gems/systemd-journal- <module:Id128' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Systemd' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/systemd-journal-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from rescue in require' fromrequire' from /var/lib/gems/2.1.0/gems/fluent-plugin-systemd- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/fluentd-block in tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-tryloadplugin' from /var/lib/gems/2.1.


Question: PREFIX = gcr.io/google-containers IMAGE = fluentd-elasticsearch <del> TAG = v2.0.0 </del> <ins> TAG = v2.0.1 </ins> build: docker build --pull -t $(PREFIX)/$(IMAGE):$(TAG) .

0/gems/fluentd- newimpl' from /var/lib/gems/2.1.0/gems/fluentd-newinput' from /var/lib/gems/2.1.0/gems/fluentd- addsource' from /var/lib/gems/2.1.0/gems/fluentd-block in configure' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-configure' from /var/lib/gems/2.1.0/gems/fluentd- configure' from /var/lib/gems/2.1.0/gems/fluentd-runconfigure' from /var/lib/gems/2.1.0/gems/fluentd- runconfigure' from /var/lib/gems/2.1.0/gems/fluentd-block in start' from /var/lib/gems/2.1.0/gems/fluentd- call' from /var/lib/gems/2.1.0/gems/fluentd-mainprocess' from /var/lib/gems/2.1.0/gems/fluentd- start' from /var/lib/gems/2.1.0/gems/fluentd-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/fluentd-0.12.39/bin/fluentd:5:in <top (required)' from /usr/local/bin/fluentd:23:inload' from /usr/local/bin/fluentd:23:in <main' What you expected to happen: fluentd-es run success How to reproduce it (as minimally and precisely as possible): kubectl create -f fluentd-es- the image is google-containers/fluentd-elasticsearch:v2.0.0 Anything else we need to know?


Question: PREFIX = gcr.io/google-containers IMAGE = fluentd-elasticsearch <del> TAG = v2.0.0 </del> <ins> TAG = v2.0.1 </ins> build: docker build --pull -t $(PREFIX)/$(IMAGE):$(TAG) .

: Environment: Kubernetes version (use ):Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-21T08:23:22Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-21T08:08:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release):centos 7.3 Kernel (e.g. ):Linux node-21.46 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Yup, the situation with fluentd on the system with systemd is very sloppy. There's a hack to make it works sometimes, but it seems that it doesn't work. I'll look into it
/sig instrumentation /assign
Facing same issue.


Question: PREFIX = gcr.io/google-containers IMAGE = fluentd-elasticsearch <del> TAG = v2.0.0 </del> <ins> TAG = v2.0.1 </ins> build: docker build --pull -t $(PREFIX)/$(IMAGE):$(TAG) .

Environment Details:kubectl version Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} OS (e.g. from /etc/os-release):Centos 7 (Core) Kernel (e.g. uname -a): Linux xxxx 3.10.0-327.el7.x8664 Error for reference /var/lib/gems/2.1.0/gems/ffi- block in ffilib': Could not open library '': cannot open shared object file: No such file or directory. (LoadError) Could not open library '': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': libsystemd- cannot open shared object file: No such file or directory from /var/lib/gems/2.1.0/gems/ffi-map' from /var/lib/gems/2.1.0/gems/ffi- ffilib' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Native' from /var/lib/gems/2.1.0/gems/systemd-journal- <module:Id128' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Systemd' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/systemd-journal-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.


Question: PREFIX = gcr.io/google-containers IMAGE = fluentd-elasticsearch <del> TAG = v2.0.0 </del> <ins> TAG = v2.0.1 </ins> build: docker build --pull -t $(PREFIX)/$(IMAGE):$(TAG) .

0/gems/systemd-journal- <top (required)' fromrequire' from rescue in require' fromrequire' from /var/lib/gems/2.1.0/gems/fluent-plugin-systemd- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/fluentd-block in tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- newimpl' from /var/lib/gems/2.1.0/gems/fluentd-newinput' from /var/lib/gems/2.1.0/gems/fluentd- addsource' from /var/lib/gems/2.1.0/gems/fluentd-block in configure' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-configure' from /var/lib/gems/2.1.0/gems/fluentd- configure' from /var/lib/gems/2.1.0/gems/fluentd-runconfigure' from /var/lib/gems/2.1.0/gems/fluentd- runconfigure' from /var/lib/gems/2.1.0/gems/fluentd-block in start' from /var/lib/gems/2.1.0/gems/fluentd- call' from /var/lib/gems/2.1.0/gems/fluentd-mainprocess' from /var/lib/gems/2.1.0/gems/fluentd- start' from /var/lib/gems/2.1.0/gems/fluentd-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/fluentd-0.12.39/bin/fluentd:5:in <top (required)' from /usr/local/bin/fluentd:23:inload' from /usr/local/bin/fluentd:23:in <


Question: PREFIX = gcr.io/google-containers IMAGE = fluentd-elasticsearch <del> TAG = v2.0.0 </del> <ins> TAG = v2.0.1 </ins> build: docker build --pull -t $(PREFIX)/$(IMAGE):$(TAG) .

main'
You can solve it temporary by removing I'm working on the solution
A solution here might be to base fluentd-es off an image with journald supporting XZ and LZ4 compression. I'm looking :)
Thanks


Question: } func TestValidateValidHost(t *testing.T) { <del> v := &validator{ appArmorFS: "./testdata/", } </del> <ins> v := &validator{} </ins> tests := []struct { profile     string

As mentioned the return value of is not being used any longer and the entire method can be removed. The usage of was removed in Now there is no need to have the function and store it's result at all. /sig node /kind cleanup /good-first-issue /help-wanted KEP xref:
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : As mentioned the return value of is not being used any longer and the entire method can be removed. /kind cleanup /good-first-issue /help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/assign
I am having issues submitting the pull request. I removed that portion of the code under however this is my first time contributing and would like some help submitting this new updated version..
/assign
/assign
/assign
/assign


Question: nsenterPath string confDir     string binDirs     []string <ins> podCidr     string </ins> } type cniNetwork struct {


Question: return pl } <ins> type TestDuplicatePlugin struct { } func (dp *TestDuplicatePlugin) Name() string { return duplicatePluginName } func (dp *TestDuplicatePlugin) PreFilter(pc *PluginContext, p *v1.Pod) *Status { return nil } func (dp *TestDuplicatePlugin) Extensions() PreFilterExtensions { return nil } var _ PreFilterPlugin = &TestDuplicatePlugin{} func newDuplicatePlugin(_ *runtime.Unknown, _ FrameworkHandle) (Plugin, error) { return &TestDuplicatePlugin{}, nil } </ins> var registry Registry = func() Registry { r := make(Registry) r.Register(scoreWithNormalizePlugin1, newScoreWithNormalizePlugin1) r.Register(scoreWithNormalizePlugin2, newScoreWithNormalizePlugin2) r.Register(scorePlugin1, newScorePlugin1) r.Register(pluginNotImplementingScore, newPluginNotImplementingScore) <ins> r.Register(duplicatePluginName, newDuplicatePlugin) </ins> return r }()

There is no reason this should be allowed to happen, especially that we can't configure duplicate plugins differently since plugin configs is a separate list . /sig scheduling
/label help-wanted
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help-wanted
/help /good-first-issue
/assign
/remove-good-first-issue /remove-help


Question: }} } <ins> // NewGone returns an error indicating the item no longer available at the server and no forwarding address is known. func NewGone(message string) error { return &StatusError{unversioned.Status{ Status:  unversioned.StatusFailure, Code:    http.StatusGone, Reason:  unversioned.StatusReasonGone, Message: message, }} } </ins> // NewInvalid returns an error indicating the item is invalid and cannot be processed. func NewInvalid(kind, name string, errs fielderrors.ValidationErrorList) error { causes := make([]unversioned.StatusCause, 0, len(errs))

We sorta fixed this-- now it's a real status error, but we left the status code 500, so it still produces stack traces and potentially confuses clients. we should figure out the correct status code.
I think there is a lot more going on related to this in . Clients are re-listing after the error but not rewatching.
/sub
It turned out this was not causing job e2e flakes. Just sending 500s that were confusing. So lowering priority.
How about a NewBadRequest error?
that certainly sounds like an improvement.


Question: return semver.MustParse(version.APIVersion + ".0"), nil } <del> // isSharedPIDNamespaceEnabled returns true if the Docker version is 1.13.1+ </del> <ins> // isSharedPIDNamespaceSupported returns true if the Docker version is 1.13.1+ </ins> // (API version 1.26+), and false otherwise. <del> func isSharedPIDNamespaceEnabled() (bool, error) { </del> <ins> func isSharedPIDNamespaceSupported() (bool, error) { </ins> version, err := getDockerAPIVersion() if err != nil { return false, err

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --BUG REPORT: What happened: I'm running a simple Kubernetes cluster with master and one node. All basic PODs are deploying successfully (ie. weave-net) but when I try to deploy GlusterFS as a POD - container creation fails with error: This is a result of failing to perform from docker file. The weird part is that if I run this image directly on node through docker it runs smoothly. What you expected to happen: I'd expect to be able to deploy GlusterFS through Kubernetess properly. How to reproduce it (as minimally and precisely as possible): Setup minimal cluster and try to deploy the follwing POD Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: virtualbox OS (e.g. from /etc/os-release):Kernel (e.g. ):Docker versionInstall tools: vagrant Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
One update - I can deploy this POD with no problems on version of k8s.
I am hitting the same bug on a freshly installed k8s cluster running on installed via kubeadm plus weave-net with a currently unsupported docker version. For me, downgrading to a supported docker version helped to resolve this issue (v1.12):
- Thanks for the info! - Well we can confirm that kubernetes is not working properly with Docker 1.29 and 1.3
Reiterating the mentions to trigger a notification: <detailsIn response to : - Thanks for the info!


Question: return semver.MustParse(version.APIVersion + ".0"), nil } <del> // isSharedPIDNamespaceEnabled returns true if the Docker version is 1.13.1+ </del> <ins> // isSharedPIDNamespaceSupported returns true if the Docker version is 1.13.1+ </ins> // (API version 1.26+), and false otherwise. <del> func isSharedPIDNamespaceEnabled() (bool, error) { </del> <ins> func isSharedPIDNamespaceSupported() (bool, error) { </ins> version, err := getDockerAPIVersion() if err != nil { return false, err

- Well we can confirm that kubernetes is not working properly with Docker 1.29 and 1.3 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is not really a bug since kubernetes doesn't support the docker version you use yet, but feel free to send a patch if you find the problem.
I was thinking about your comment and I think it is still a bug since downgrading Kubernetes to 1.6 fixes the problem while having the same setup as described above. So something have changed in 1.7. But yeah. Will try to investigate on my own.
I'm actually getting this same error with 1.7.1 and docker 1.13.1. Anyone have other thoughts on what the issue might be here? I found this so question () but it looks like everything is configured correctly.
In kubernetes 1.7, under pid # 1, there is a "/ pause" process, and since init takes pid 1, it is no longer available. In earlier versions of kubernetes, pid # 1 was free . PID USER TIME COMMAND 1 root 0:00 /pause 7 root 3:41 /tiller Kubernetes now shares a single PID namespace among all containers in a pod when running with docker = 1.13.1. This means processes can now signal processes in other containers in a pod, but it also means that the pattern will cause the pod to be restarted rather than a single container.
I can work around the problem using the option in the kubelet, however this not the desired solution of course.
just FYI, looks like some images have to be run as PID 1.
In 1.7 release, we enabled shared pid namespace if the docker version is docker 1.13 and beyond for debug pod feature. Do anyone know why those images, such as GlusterFS have to run as PID 1? we need to figure out how to resolve this conflict with shared pid namespace. If we cannot, we have to disable that setting for 1.8 since we scheduled to support docker 1.13 in 1.8 timeframe. I am currently assigning you this bug to come up a solution.


Question: return semver.MustParse(version.APIVersion + ".0"), nil } <del> // isSharedPIDNamespaceEnabled returns true if the Docker version is 1.13.1+ </del> <ins> // isSharedPIDNamespaceSupported returns true if the Docker version is 1.13.1+ </ins> // (API version 1.26+), and false otherwise. <del> func isSharedPIDNamespaceEnabled() (bool, error) { </del> <ins> func isSharedPIDNamespaceSupported() (bool, error) { </ins> version, err := getDockerAPIVersion() if err != nil { return false, err

cc/
gluster/gluster-centos is a container image that runs systemd so that it can run ntpd, crond, gssproxy, glusterd & sshd inside a single container. This isn't how Kubernetes is intended to be used and I don't know how much we should go out of our way to support it (this is a question for & seems to be a good write up of the challenges of running systemd inside of docker. It also prevents Kubernetes from doing process management like: reporting state of processes signaling individual processes for graceful restart ingesting and display daemon output (to stdout/err) Ideally we could change this many-processes-in-a-container to be many-containers-in-a-pod, something like this: This doesn't need a privileged container and doesn't run sshd or ntpd, neither of which are needed in Kubernetes. This also doesn't run crond, but that may actually be needed by gluster. I've never used gluster. (I used flags so I didn't have to investigate how to make these processes log to stdout) If you must run systemd in a container, adding this to the container config will get systemd to run: This tells systemd to run in system mode even though it's not PID 1 and disables its chroot detection (which is unrelated to PID 1 but checked when invoked as "systemd", I guess).
afaict, running in a container is well recommended ( and ) at certain scenarios and a common practice. iic, there are enough containers who make use of as its . If shared namespace which introduced in 1.7 breaks this, I believe its a backward compatibility break :(. I would invite here to share his thoughts as well. :)
Those articles are about how to work around shortcomings in vanilla docker that Kubernetes solves natively: resource sharing via pods, containing via SELinux, reaping orphaned zombies, etc. My earlier question was along the lines of how much effort should go into supporting use cases that are not using Kubernetes as intended. (I'm actually asking, not trying to make a point.) Does Kubernetes commit to continuing support for every container image that's ever run on Kubernetes? For how long? Indefinitely? or can we change it over the course of multiple releases? Anyway, this change is compatible with running an init system in a container.


Question: return semver.MustParse(version.APIVersion + ".0"), nil } <del> // isSharedPIDNamespaceEnabled returns true if the Docker version is 1.13.1+ </del> <ins> // isSharedPIDNamespaceSupported returns true if the Docker version is 1.13.1+ </ins> // (API version 1.26+), and false otherwise. <del> func isSharedPIDNamespaceEnabled() (bool, error) { </del> <ins> func isSharedPIDNamespaceSupported() (bool, error) { </ins> version, err := getDockerAPIVersion() if err != nil { return false, err

The problem is with images that assume they will only ever have , which has so far been the case with docker due to a technical limitation, but it was never the case with other runtimes such as rkt, hyperv, etc. Brainstorming other options: shared pid namespace and require users of systemd to set an environment variable in their Kubernetes config (or port the multi-process container to a multi-container pod). shared pid namespace and enable it in a future release where we have the same problem. a solution where pid namespace is configurable per-pod, but not supported indefinitely. It would probably be implemented using annotations, and we will have to modify the Container Runtime Interface to support it. Over the course of multiple releases the default behavior would change but could be reverted based on the presence of an annotation. In a future release, users will have to change an annotation in their config to get the old behavior (rather than the environment variable). In a future future release, the annotation will stop working and users will have to set the environment variable. to supporting per-pod shared vs isolated pid namespaces indefinitely and adding it to the Kubernetes API. It doesn't make sense in some runtimes, so we probably can't require all runtimes to implement it, so it'll likely only see use with Docker. Last time we discussed it in SIG Node, and didn't want to support isolated pid namespaces indefinitely.
Yes running systemd in a container is a critical need for us. There are lots of situations where it is and will be used. Biggest use case if for people simply moving current workloads that run in a VM into a container. I have a service that runs fun under systemd in a VM. If I can simply take the init script and move it whole sale into a container, why wouldn't I want to do this. It's simple and does require a great deal of knowledge by the end user. service containers. Not every project can easily be moved into microservices. But they probably can run inside of containers. If we drop support for systemd running inside of containers, then we loose a big group of users. Getting access to journald information inside of the container. Applications that write to syslog and directly to the journal. Currently in container world without systemd, this information is dropped on the floor.


Question: return semver.MustParse(version.APIVersion + ".0"), nil } <del> // isSharedPIDNamespaceEnabled returns true if the Docker version is 1.13.1+ </del> <ins> // isSharedPIDNamespaceSupported returns true if the Docker version is 1.13.1+ </ins> // (API version 1.26+), and false otherwise. <del> func isSharedPIDNamespaceEnabled() (bool, error) { </del> <ins> func isSharedPIDNamespaceSupported() (bool, error) { </ins> version, err := getDockerAPIVersion() if err != nil { return false, err

if I run systemd and journald, I can actually retrieve and use this information.
If the intension is to force all containers inside of a POD to run with the same PID Namespace? I think this is a mistake since it eliminates another really cool use case. I would love to be able to run two containers inside of a pod, and INIT container with escallated privileges and a locked down container, with no privileges. I want to make sure the locked down container can not see the processes of the privileged container. An example of this, might be a container that loads a kernel module that is needed by the locked down container. Another use case would be a buildah container. The INIT container mounts up a COW file system onto a directory on a location that the locked down container can write, The locked down container writes out the data for his image, when the container completes, the INIT container commits the data to the repo. I believe allowing different containers inside of a POD to run with different security and isolated PIDs should not be removed.
none of the things you've mentioned are removed by this change, save using docker to hide other processes in the pod, and I'm not sure why that's a requirement. Is anyone currently using the "privileged sidecar" pattern you describe or is it hypothetical? It just occurred to me that an interesting compromise here might be to only enable shared pid for multi-container pods. This would enable pod semantics for PID when it would make a difference while drop-in docker monoliths keep the PID 1 behavior they're expecting in docker. I'd have to double check, but I'm pretty sure that behavior could be implemented entirely in the dockershim with no API or CRI changes.
Hypothetical, but I am pushing it for a solution for problems I hear from users fairly frequently, where then need big privileges at start and stop of the pod, but not for the time is is available to the network.
ok, thanks. ftr, I think we should continue to support this pattern as well, and I don't think a shared PID namespace prohibits it. I will bring up this issue at Tuesday's sig node meetup if you'd like to join us. Does the config I mentioned in fix the issue with systemd?


Question: return semver.MustParse(version.APIVersion + ".0"), nil } <del> // isSharedPIDNamespaceEnabled returns true if the Docker version is 1.13.1+ </del> <ins> // isSharedPIDNamespaceSupported returns true if the Docker version is 1.13.1+ </ins> // (API version 1.26+), and false otherwise. <del> func isSharedPIDNamespaceEnabled() (bool, error) { </del> <ins> func isSharedPIDNamespaceSupported() (bool, error) { </ins> version, err := getDockerAPIVersion() if err != nil { return false, err

Copying from a comment made elsewhere:
Update from SIG Node Meetup: Everyone wants to support these use cases, but we're not yet sure the requirements we want to support medium to long term. is going to sync up with and then we'll figure out what we want to support in a future release. For 1.8 the current plan is to leave docker shared pid namespace enabled by default. Cluster administrators can disable it per-node with the --docker-disable-shared-pid flag. I'll see about whether we should add something back to the release notes.
This seems somewhat backwards. Wouldn't it make more sense to go back to supporting systemd use cases by default NOW (by making the shared PID namespace behavior optional or something) until a final decision is reached and can be advertised to current users?
absolutely ! thank you very much
I think that is the best solution is to make the PID Namespace sharing optional.
- optional at what scope - cluster, node, pod. let's sync up before next sig-node and we can revisit if need be before 1.8 goes out. The main question right now is if pid namespace sharing is opt-in or opt-out. We need to weigh pro/cons of both.
I would expect it at the POD, since pods would have different requirements, not sure why you would want to have this at the cluster or node level.
yes it works :)
awesome, thanks! I'd like to better understand the requirements of people objecting to this change ( I think there's a bit of confusion about what has changed, so let's start with some facts: This container image runs on Kubernetes without changes to the image, but it requires (non-obvious) pod configuration. is nothing inherently incompatible between a shared pid namespace and a process manager like systemd. systemd and other init systems work with pid != 1, but may require special invocation. in this issue is specific to Kubernetes. That is, this container image wouldn't work with vanilla docker PID namespace sharing, either. change does not affect securityContext I can think of a few things that are objectionable about the change, but let's state them as requirements. For example, are these requirements for anyone? Kubernetes must run all containers published on docker hub with no additional pod configuration Kubernetes must run all systemd containers, regardless of how they are built, with zero additional pod configuration.


Question: return semver.MustParse(version.APIVersion + ".0"), nil } <del> // isSharedPIDNamespaceEnabled returns true if the Docker version is 1.13.1+ </del> <ins> // isSharedPIDNamespaceSupported returns true if the Docker version is 1.13.1+ </ins> // (API version 1.26+), and false otherwise. <del> func isSharedPIDNamespaceEnabled() (bool, error) { </del> <ins> func isSharedPIDNamespaceSupported() (bool, error) { </ins> version, err := getDockerAPIVersion() if err != nil { return false, err

Kubernetes must run this specific container image with zero additional configuration Kubernetes must officially support isolated PID namespaces between containers in a single pod across all runtimes (docker, rkt, hyperv, etc). Configuration changes are ok, but the flag to systemd cannot be used because the documentation says its not useful. I want to use systemd as intended. Rather than having Kubernetes work around this container image, should we patch the Dockerfile to make it compatible with even the native docker shared-pid, unrelated to Kubernetes?
Configuration changes are ok, but the --system flag to systemd cannot be used because the documentation says its not useful. I want to use systemd as intended. We can ask the systemd guys why they recommend never using the --system flag, but I would guess one of the reasons is that processes showing up that were not created in the ancestry of systemd, could cause confusion. I would also guess that running more then one container with systemd --system would not be supported. Kubernetes must officially support isolated PID namespaces between containers in a single pod. This is required so that you can run different containers with different security constraints in the same pod at the same time. Your last comment makes the seem like it is the default, when actually it is the exception. almost no containers run with shared-pid normally. I guess in a side car container, it makes sense to support this scenario, but forcing this as the ONLY way to run containers is limiting.
This is the part I don't understand. Why? What does PID visibility have to do with security constraints? What am I missing? nor do they run with a shared network namespace, but we impose that on docker containers because that's the pod model.
An update from the last SIG Node: we'll disable this by default in 1.8 and revisit for 1.9 I'll use to track that.
if I have a privileged container process in the same pid namespace as a pid in a non priv container, then the container process in the non priv can attack the priv process. Having them in separate pid namespaces gives you better security. Bottom line, you have a potential change where containers that expect to be run as PID 1, whether legitimately or not, could break when you make this change. Kubernetes in my opinion should make it easy to opt out of this behavior.


Question: return semver.MustParse(version.APIVersion + ".0"), nil } <del> // isSharedPIDNamespaceEnabled returns true if the Docker version is 1.13.1+ </del> <ins> // isSharedPIDNamespaceSupported returns true if the Docker version is 1.13.1+ </ins> // (API version 1.26+), and false otherwise. <del> func isSharedPIDNamespaceEnabled() (bool, error) { </del> <ins> func isSharedPIDNamespaceSupported() (bool, error) { </ins> version, err := getDockerAPIVersion() if err != nil { return false, err

/priority critical-urgent
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details
Update: is pending review and should be to the v1.8 Milestone to resolve this issue.


Question: "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/sets" <ins> utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/kubernetes/pkg/features" </ins> "k8s.io/kubernetes/pkg/volume" "k8s.io/kubernetes/pkg/volume/util" "k8s.io/legacy-cloud-providers/azure"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Azure is not listed as "migratable" in could refer to Why is this needed:
/sig azure /assign


Question: proxier.portsMap = replacementPortsMap } <ins> // Clear UDP conntrack for port or all conntrack entries when port equal zero. // When a packet arrives, it will not go through NAT table again, because it is not "the first" packet. // The solution is clearing the conntrack. Known issus: // https://github.com/docker/docker/issues/8795 // https://github.com/kubernetes/kubernetes/issues/31983 func (proxier *Proxier) clearUdpConntrackForPort(port int) { var err error = nil glog.V(2).Infof("Deleting conntrack entries for udp connections") if port > 0 { err = proxier.execConntrackTool("-D", "-p", "udp", "--dport", strconv.Itoa(port)) if err != nil && !strings.Contains(err.Error(), noConnectionToDelete) { glog.Errorf("conntrack return with error: %v", err) } } else { glog.Errorf("Wrong port number. The port number must be greater than zero") } } </ins> // Join all words with spaces, terminate with newline and write to buf. func writeLine(buf *bytes.Buffer, words ...string) { buf.WriteString(strings.Join(words, " ") + "n")

Hi, We have a device (router in this case) that sends long living stream of udp packets to our kubernetes cluster. We then create a service and pods that expose a node port to receive the incoming udp packets. However, packets were not reach our containers. Further digging and found that because UDP packets already stream into our host (therefore conntrack entry already created) BEFORE our service is created (iptable rules ), the iptable rule didn't take effect. Since our UDP stream is long living, the conntrack entry never get timeout neither. To workaround the issue, we have to stop the UDP stream for sometime after the service is created so that the conntrack entry times out. If we use "conntrack -D" to clear all entries after service is deployed also workaround the issue. Though it is the behavior of the linux iptable that new rules doesn't affect already existing connection, we are trying to see if kubernetes proxy have any config that can clear the related conntrack entries when a new service is deployed?
I am interested in your problem. Can you give some more details how did you add iptables rules before service started. Did you use the PREROUTING chain to forward port to pod ?
OK I was able to reproduce the problem
I think the issue is more problematic and complex. It is explained here:
Yes I think it is the same conntrack issue as in the one reported in docker issue. But I also think that the kubernetes might have a better control over it, since it knows the service up/down. I think k8s already deleting the conntrack entries when removing services. Just wondering if the same can apply to adding services?
Yes you are right. Kubernetes as a Docker manager should take care of this. Deleting the conntrack entries is done in very simple way: Just call the application
But there is another problem because I removed conntrack entries and then I created service and still have problem. I had to clear conntrack entries after I started my service. I will try add this to source code and then try it.
In code it looks much better. I assume you used for your service to use some particular port. The fix is in place when node port is binded. Do you think there is any other scenario where conntrack entries should be cleared?


Question: proxier.portsMap = replacementPortsMap } <ins> // Clear UDP conntrack for port or all conntrack entries when port equal zero. // When a packet arrives, it will not go through NAT table again, because it is not "the first" packet. // The solution is clearing the conntrack. Known issus: // https://github.com/docker/docker/issues/8795 // https://github.com/kubernetes/kubernetes/issues/31983 func (proxier *Proxier) clearUdpConntrackForPort(port int) { var err error = nil glog.V(2).Infof("Deleting conntrack entries for udp connections") if port > 0 { err = proxier.execConntrackTool("-D", "-p", "udp", "--dport", strconv.Itoa(port)) if err != nil && !strings.Contains(err.Error(), noConnectionToDelete) { glog.Errorf("conntrack return with error: %v", err) } } else { glog.Errorf("Wrong port number. The port number must be greater than zero") } } </ins> // Join all words with spaces, terminate with newline and write to buf. func writeLine(buf *bytes.Buffer, words ...string) { buf.WriteString(strings.Join(words, " ") + "n")

When I create PR I will let you know
The PR created: If you want verify it you can easily do it using local cluster Then clone kube-deploy project to create local k8s cluster with image you already created with fixes
Thanks for the fix. This is what we did manually to workaround so it should be good enough for us. It would be even better if you can clear the conntrack entry only for the affected entry, such as destination port, to minimizes the impact. Wondering if that possible? Will try out as well. Thanks.
I wanted do it like you mentioned but unfortunately conntract command doesn't give that possibilities. In parameters you can only pass the source and destination IP addresses.
Sorry didn't get back for a while. I tried the conntrack tool and seems like you can use --dport option: conntrack -D -s 172.27.209.165 -p udp --dport 53 udp 17 63 src=172.27.209.165 dst=171.70.168.183 sport= dport=53 src=171.70.168.183 dst=172.27.209.165 sport=53 dport= [ASSURED] mark=0 use=1 udp 17 33 src=172.27.209.165 dst=171.70.168.183 sport= dport=53 src=171.70.168.183 dst=172.27.209.165 sport=53 dport= [ASSURED] mark=0 use=1 conntrack v1.4.3 (conntrack-tools): 2 flow entries have been deleted.
Thanks for info I will try it
I've updated my PR for flag It works well
It's not clear to me why there would be a conntrack rule for your service port before we add any rules?
We have a router, which configured to export statistics using UDP to our host. That UDP stream is long live. If our host restarted, or we re-deploy our cluster (mainly during development), then udp packets received by host before K8S is provisioned.
But why do you have conntrack rules - simply receiving packets will not cause conntrack to create records.. , dachulin <:


Question: disablePreemption        bool percentageOfNodesToScore int32 enableNonPreempting      bool <del> lastProcessedNodeIndex   int </del> <ins> nextStartNodeIndex       int </ins> } // snapshot snapshots scheduler cache and node infos for all fit and priority

What would you like to be : A unit test that verifies the evaluation order of the nodes in the scheduler. Why is this needed: The scheduler evaluates the nodes in a specific order until it finds a pre-defined minimum number of feasible nodes. To give all nodes a fair chance of being evaluated, in each scheduling cycle, the scheduler starts evaluating from where it left off in the previous cycle. This is captured by the parameter in  We need to add a unit test that captures regressions related to this behavior to avoid bugs like /sig scheduling /priority important-soon /help
/assign
/remove-help


Question: shouldSchedule:        false, shouldContinueRunning: false, }, <ins> { predicateName: "InsufficientResourceError", podsOnNode: []*v1.Pod{ { Spec: v1.PodSpec{ Containers: []v1.Container{{ Ports: []v1.ContainerPort{{ HostPort: 666, }}, Resources: resourceContainerSpec("50M", "0.5"), }}, }, }, }, ds: &extensions.DaemonSet{ Spec: extensions.DaemonSetSpec{ Selector: &metav1.LabelSelector{MatchLabels: simpleDaemonSetLabel}, Template: v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: simpleDaemonSetLabel, }, Spec: resourcePodSpec("", "100M", "0.5"), }, }, }, wantToRun:             true, shouldSchedule:        false, shouldContinueRunning: true, }, { predicateName: "ShouldRunDaemonPod", podsOnNode: []*v1.Pod{ { Spec: v1.PodSpec{ Containers: []v1.Container{{ Ports: []v1.ContainerPort{{ HostPort: 666, }}, Resources: resourceContainerSpec("50M", "0.5"), }}, }, }, }, ds: &extensions.DaemonSet{ Spec: extensions.DaemonSetSpec{ Selector: &metav1.LabelSelector{MatchLabels: simpleDaemonSetLabel}, Template: v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: simpleDaemonSetLabel, }, Spec: resourcePodSpec("", "50M", "0.5"), }, }, }, wantToRun:             true, shouldSchedule:        true, shouldContinueRunning: true, }, { predicateName: "ErrNodeSelectorNotMatch", ds: &extensions.DaemonSet{ Spec: extensions.DaemonSetSpec{ Selector: &metav1.LabelSelector{MatchLabels: simpleDaemonSetLabel}, Template: v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: simpleDaemonSetLabel, }, Spec: v1.PodSpec{ NodeSelector: simpleDaemonSetLabel2, }, }, }, }, wantToRun:             false, shouldSchedule:        false, shouldContinueRunning: false, }, { predicateName: "ShouldRunDaemonPod", ds: &extensions.DaemonSet{ Spec: extensions.DaemonSetSpec{ Selector: &metav1.LabelSelector{MatchLabels: simpleDaemonSetLabel}, Template: v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: simpleDaemonSetLabel, }, Spec: v1.PodSpec{ NodeSelector: simpleDaemonSetLabel, }, }, }, }, wantToRun:             true, shouldSchedule:        true, shouldContinueRunning: true, }, { predicateName: "ErrPodAffinityNotMatch", ds: &extensions.DaemonSet{ Spec: extensions.DaemonSetSpec{ Selector: &metav1.LabelSelector{MatchLabels: simpleDaemonSetLabel}, Template: v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: simpleDaemonSetLabel, }, Spec: v1.PodSpec{ Affinity: &v1.Affinity{ NodeAffinity: &v1.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &v1.NodeSelector{ NodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{ { Key:      "type", Operator: v1.NodeSelectorOpIn, Values:   []string{"test"}, }, }, }, }, }, }, }, }, }, }, }, wantToRun:             false, shouldSchedule:        false, shouldContinueRunning: false, }, { predicateName: "ShouldRunDaemonPod", ds: &extensions.DaemonSet{ Spec: extensions.DaemonSetSpec{ Selector: &metav1.LabelSelector{MatchLabels: simpleDaemonSetLabel}, Template: v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: simpleDaemonSetLabel, }, Spec: v1.PodSpec{ Affinity: &v1.Affinity{ NodeAffinity: &v1.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &v1.NodeSelector{ NodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{ { Key:      "type", Operator: v1.NodeSelectorOpIn, Values:   []string{"production"}, }, }, }, }, }, }, }, }, }, }, }, wantToRun:             true, shouldSchedule:        true, shouldContinueRunning: true, }, </ins> } for i, c := range cases { for _, strategy := range updateStrategies() { <del> node := newNode("test-node", nil) </del> <ins> node := newNode("test-node", simpleDaemonSetLabel) </ins> node.Status.Conditions = append(node.Status.Conditions, c.nodeCondition...) node.Status.Allocatable = allocatableResources("100M", "1") manager, _, _ := newTestController()

Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement /sig apps Description: is a key function of DaemonSet; it's better to have more unit test case for it, e.g. node status, pod on the node. There's several test cases to the DaemonSet's finial decision, which are kind of integration test to me; prefer to provide more unit test case for this key function.
GitHub didn't allow me to assign the following users: mathspanda. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
do you have any specific cases or shall I figure out something? I can take a look at this issue if you don't mind.
, , thanks very much for our effort on this enhancement; as you known, DaemonSet will use scheduler's predicates to decide which node can place pods; there's several predicates errors , the purpose of this issue is to trigger every error in unit test for code coverage.
Sure, thanks. I'll cover these predicates.
Also I've found interesting issue when I wanted to add test for predicate. It looks like is not handled or maybe I did something wrong, I couldn't find check. According following comments it should fail because AWS EBS forbids any two pods mounting the same volume ID : Test case: Any thoughts? is it correct behavior?


Question: // Make an update of the DaemonSet which we know will create a hash collision when // the next ControllerRevision is created. <del> _, err = dsClient.Update(ds) if err != nil { t.Fatalf("Failed to update DaemonSet: %v", err) } </del> <ins> ds = updateDS(t, dsClient, ds.Name, func(updateDS *apps.DaemonSet) { updateDS.Spec.Template.Spec.TerminationGracePeriodSeconds = &one }) </ins> // Wait for any pod with the latest Spec to exist err = wait.PollImmediate(100*time.Millisecond, 10*time.Second, func() (bool, error) {

https://k8s- /sig apps /priority critical-urgent /assign PR that the test: /cc
GitHub didn't allow me to assign the following users: mortent. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : https://k8s- /priority critical-urgent /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake
I'm looking into this.
The test is still flaky, but with different symptom: An example flake: https://k8s- It's likely because the collision count was updated in a previous sync loop, but the in the cache store hasn't been updated and we used a stale collision count while calculating the new hash: /assign


Question: { Name: to.StringPtr("foo"), RegistryProperties: &containerregistry.RegistryProperties{ <del> LoginServer: to.StringPtr("foo-microsoft.azurecr.io"), </del> <ins> LoginServer: to.StringPtr("*.azurecr.io"), </ins> }, }, { Name: to.StringPtr("bar"), RegistryProperties: &containerregistry.RegistryProperties{ <del> LoginServer: to.StringPtr("bar-microsoft.azurecr.io"), </del> <ins> LoginServer: to.StringPtr("*.azurecr.cn"), </ins> }, }, { Name: to.StringPtr("baz"), RegistryProperties: &containerregistry.RegistryProperties{ <del> LoginServer: to.StringPtr("baz-microsoft.azurecr.io"), </del> <ins> LoginServer: to.StringPtr("*.azurecr.de"), }, }, { Name: to.StringPtr("bus"), RegistryProperties: &containerregistry.RegistryProperties{ LoginServer: to.StringPtr("*.azurecr.us"), </ins> }, }, }


Question: { Name: to.StringPtr("foo"), RegistryProperties: &containerregistry.RegistryProperties{ <del> LoginServer: to.StringPtr("foo-microsoft.azurecr.io"), </del> <ins> LoginServer: to.StringPtr("*.azurecr.io"), </ins> }, }, { Name: to.StringPtr("bar"), RegistryProperties: &containerregistry.RegistryProperties{ <del> LoginServer: to.StringPtr("bar-microsoft.azurecr.io"), </del> <ins> LoginServer: to.StringPtr("*.azurecr.cn"), </ins> }, }, { Name: to.StringPtr("baz"), RegistryProperties: &containerregistry.RegistryProperties{ <del> LoginServer: to.StringPtr("baz-microsoft.azurecr.io"), </del> <ins> LoginServer: to.StringPtr("*.azurecr.de"), }, }, { Name: to.StringPtr("bus"), RegistryProperties: &containerregistry.RegistryProperties{ LoginServer: to.StringPtr("*.azurecr.us"), </ins> }, }, }

After my repro and investigation(really spent a lot of time...), I found the issue lies here, would return empty even after granting sp access to azure acr, and after around 15min, it returns correct loginServer list: Looks like it's a SDK or acr list with specific sp issue, workaround is wait around 15min or more time, it could recover by itself. Paste my investigation logs here, I grant sp access to acr in around , you could see it returns empty registry list first and then after a few more minutes, it returns the correct list with loginServer:
BTW, in v1.12, acr list() uses new API, I will check whether in v1.12 would have same issue. Acutally I have already implemented LazyProvide func(check here: ), while it won't fix the original issue.
Update: After a few more tries, I found it's related to this code snippet, before , we need to generate a new ServicePrincipalToken every time: I will move this code snippet into Provide() func, and send out a PR! The failure recover time is within 2min now.


Question: } } <ins> func TestRegisterDuplicatePluginWouldFail(t *testing.T) { plugin := config.Plugin{Name: duplicatePluginName, Weight: 1} pluginSet := config.PluginSet{ Enabled: []config.Plugin{ plugin, plugin, }, } plugins := config.Plugins{} plugins.PreFilter = &pluginSet _, err := NewFramework(registry, &plugins, emptyArgs) if err == nil { t.Fatal("Framework initialization should fail") } if err != nil && !strings.Contains(err.Error(), "already registered") { t.Fatalf("Unexpected error, got %s, expect: plugin already registered", err.Error()) } } </ins> func TestRunScorePlugins(t *testing.T) { tests := []struct { name          string

There is no reason this should be allowed to happen, especially that we can't configure duplicate plugins differently since plugin configs is a separate list . /sig scheduling
/label help-wanted
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help-wanted
/help /good-first-issue
/assign
/remove-good-first-issue /remove-help


Question: EgressBurst int `json:"egressBurst,omitempty"` } <ins> // cniIpRange maps to the standard CNI ip range Capability type cniIpRange struct { Subnet string `json:"subnet"` } </ins> func SplitDirs(dirs string) []string { // Use comma rather than colon to work better with Windows too return strings.Split(dirs, ",")

/kind feature The CNI "capability arguments" parameter - currently used to pass port mappings - has a few new capabilities: bandwidth limiting and network address management. The dockershim's networking driver should use them. The network address is just passing the PodCIDR dynamically to the plugin, so it no longer has to retrieve it from the APIServer. Bandwidth limiting is already implemented in Kubenet. It is configured via annotations. There is no upgrade risk in providing these values; non-compliant plugins will simply ignore them. Administrators need to explicitly enable each capability individually in their CNI configuration file. To avoid skew between runtimes, the other runtimes should also be upgraded to pass these. We may also wish to look in to a way to add this more directly to the CRI, so we don't have this skew problem, but that's a different issue.
/sig network
/assign
fyi.
Looks like this was never done for CRI?
cc