Question: nodePortAddresses []string, kernelHandler KernelHandler, ) (*Proxier, error) { <del> // Set the route_localnet sysctl we need for if err := utilproxy.EnsureSysctl(sysctl, sysctlRouteLocalnet, 1); err != nil { return nil, err } </del> // Proxy needs br_netfilter and bridge-nf-call-iptables=1 when containers // are connected to a Linux bridge (but not SDN bridges).  Until most // plugins handle this, log when config is missing

Also, the vector scores are clickable and will show the CVSS scores too.
To be clear, this is fixed in v1.18.4+, v1.17.7+, and v1.16.11+ correct?
Right. Sorry if the extra paragraph in "Fixed Versions" made that unclear.
Got it. Thank you!
So just to be sure, this wouldn't be a problem in an EKS cluster running in its own VPC, right? (assuming no untrusted hosts on the same VPC)
Are all the conditions listed under , a logical AND ?
Edit: after more testing, I think is indeed required
The attack can be done from any pod with (this is still K8S default), so you might not be safe. a more accurate title: "Node setting allows to bypass localhost boundary"
We believe that of the four conditions, you might be affected if (1 and (2 or 3) and 4) are true for your cluster. Sorry that it's not super clear from the text. Trying and failing to convey info concisely and with precision makes me glad that I'm in software, not law.
Isn't this broadly fixed by ? We're discussing better answers and options for deployment, too, but I am curious what the close-condition for this bug is?
I made a typo in the issue description saying that this is addressed by instead of . I'll edit it now. Sorry about that. Yes, this is mostly fixed by . I marked this as closed due to that PR fixing the majority of security issues caused by the setting. We can re-open if people think that we should address the remaining case that talks about.
Sorry, folks. I thought that this issue was already closed. Maybe that makes my response above to less confusing.
Yes, this being open was one of the major points of confusion for us. Appreciate the clarification
/label official-cve-feed (Related to )


Question: "path/filepath" "reflect" goruntime "runtime" <ins> "sort" </ins> "sync" "sync/atomic" "testing"

TestUpdateAllocatedResourcesStatus Few days ago No response No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
I could reproduce this with in maybe 1/10 times.
cc I am unfamilar with this code so wanted to see if you have any ideas on this.
yes, I'll have a look ASAP
In the flake, the ResourceHealth array is with a different order. If this is acceptable, we should tune the test.
nice catch ! I think the best fix is ensure the ordering of statuses either in the production code or in the test code before to check Note: if we do this in the production code then we should go full throttle and offer a full API guarantee (docs, explicit tests) because users will depend on this
A quick fix is like if we want to ignore the order.


Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "io/ioutil" "math/rand"

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Unknown yet. Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated /sig scheduling /help
Can I give it a try?
Sure, feel free to take it. /assign ahmad-diaa /remove-help
I keep trying to get it fail locally by running the following command: but it never does, am I doing something wrong? Is there something else that I should be doing and would be more helpful?
Sometimes the flake is hard to reproduce in local env. You may have to read the error log of the flakes thoroughly to get some clues. BTW: you can also run the test multiple times using the directive:
After looking through the logs I came up with some information that might be useful As far as I understand, the reason for the flake is the of the medium priority pod not being cleared That is because the update request for removing returns the following error: So my current guess is that might be outdated in some cases As I am not really familiar with the logic, I would like to know whether or not those are valid points so that I could dig deeper. What do you think?
If that's the case, you can set here: You can raise a PR and manually try for ~20 times to see you can reproduce the flake.
found another issue tracking this too. it seems it has been flaky for a while.
Sorry, didn't mean to take this one for you. I worked the duplicate issue and submitted a PR. Feel free to review that PR, or submit your own if you have a different solution.
This sometimes happens. But we usually take the first PR unless it's clearly worked by someone. In terms of which style to resolve the Conflict error, I'm asking here:
Sorry, didn't mean to take this one for you. I worked the duplicate issue and submitted a PR. Feel free to review that PR, or submit your own if you have a different solution. No worries, I haven't started implementing yet and your PR seems to be following the correct approach as well according to this:


Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "io/ioutil" "math/rand"

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: [TODO] Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig node These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.


Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "io/ioutil" "math/rand"

Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }) }) <del> func hostCheck() (int, int) { </del> <ins> func hostCheck() (int, int, int) { </ins> // this is a very rough check. We just want to rule out system that does NOT have // multi-NUMA nodes or at least 4 cores

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node started to fail since 09-13.
/cc
/assign
/cc
/triage accepted /priority important-soon it's indeed failing too often
/reopen The test is still failing: Now timing out at 120s i.e. 2 minutes as we increased the default timeout . We need to understand the root cause and get to the bottom of this.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Snippet from the build log () corresponding from this failure We see from the above logs that: node has two numa nodes, 8 cpus per NUMA so the root cause of the failure is not because the tests are running on a node that doesn't have multi-numa support. is started and the health check fails a couple of times but eventually succeeds. pod is created with 9 cpus (cpusperNUMA +1 i.e. 8+1) which is expected to fail with Topology Affinity Error as we cannot allocate resources for such a pod from a single NUMAnode. We fail to get matching metrics. It is not clear if the check fails because we don't get kubelet metrics at all or the expected metrics but based on point 6 below it seems like it is because of latter. We can add additional logs for more clarity here. each is executed and Kubelet is restarted and health check is done. are collected from the namespace corresponding to the test pod which shows that the pod started even though it was expected to fail with Topology Affinity Error. We need to understand why the pod got successfully created even though it requested CPUs that couldn't be allocated from a single NUMAnode.


Question: // 5. create and start a scheduler with name "foo-scheduler" clientSet2 := clientset.NewForConfigOrDie(&restclient.Config{Host: context.httpServer.URL, ContentConfig: restclient.ContentConfig{GroupVersion: testapi.Groups[v1.GroupName].GroupVersion()}}) informerFactory2 := informers.NewSharedInformerFactory(context.clientSet, 0) <del> podInformer2 := factory.NewPodInformer(context.clientSet, 0, fooScheduler) </del> <ins> podInformer2 := factory.NewPodInformer(context.clientSet, 0) </ins> schedulerConfigFactory2 := createConfiguratorWithPodInformer(fooScheduler, clientSet2, podInformer2, informerFactory2) schedulerConfig2, err := schedulerConfigFactory2.Create()

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Scheduler's pod informer has a selector that . This is incorrect. Scheduler must watch all the pods of the cluster, including those scheduled by other schedulers in order to update its cache of the state of the cluster. Otherwise, it will miss important events and the amount of available resources, pods on nodes, ... which are needed for scheduler to work correctly, will not be updated properly. /sig scheduling /assign
cc
For updating it cache, shouldn't the default scheduler rely on node info/status? Also its not very clear, what information the default scheduler needs from other pods (not to be scheduled by it) that it makes decision on? Just trying to understand the issue more clearly.
Scheduler watches informer events to update its cache. Node status is not enough to update all the information that scheduler caches. Scheduler must know what pods are placed on what nodes. For example, in order to check inter-pod affinity/anti-affinity it needs to know the placement of all the pods, including those scheduled by other schedulers.
But this information is present here: So it seems that your PR will now make sure to get the scheduled pods' information directly from the pod informer.
So when I check the history, I see it was here:
have we seen any specific issues due to this change (like between default scheduler and DS controller)?
Yes, but scheduler relies on its pod informers to keep this location up to date. That's exactly why it is important for the scheduler to receive updates for all the pods in the cluster. Yes. We missed the problem at the time. This is problematic only in clusters that use multiple schedulers. I initially thought that DS controller could cause issues as well, but DS controller does not set , so it must not cause an issue. We have seen instances of scheduler's missing certain events in cluster and as a result scheduler's cache getting stale, but I doubt if it is related to this particular issue. This only affects clusters with more than one scheduler. If you run the test that I have written in with the head, it will fail. That tests simulate a multi-scheduler scenario.


Question: ) const ( <del> pluginDir  = "/flexvolume" driverName = "fake-driver" </del> <ins> pluginDir       = "/flexvolume" driverName      = "fake-driver" errorDriverName = "error-driver" </ins> ) func assertPathSuffix(t *testing.T, dir1 string, dir2 string) {

When kubelet is running if you create two volume plugin directories here: /usr/libexec/kubernetes/kubelet-plugins/volume/exec /usr/libexec/kubernetes/kubelet-plugins/volume/exec/cifs /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds You will see these expected logs: Now if you copy over the cifs plugin file to cifs directory and then copy the uds plugin the errors go away. The pods using the uds volume will function but any pods attempting to use the cifs volume will fail with the following error from kubelet: desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "pv" err=no volume plugin matched Restarting kubelet fixes this problem as then both plugins are present and properly . A functional volume plugin should be to kubelet even if another volume plugin is missing or invalid. Fixing the invalid volume plugin currently only adds the one that was invalid and fixed and other valid plugins that were previously present are skipped. Start kubelet service on a node. Create two directories for two flex volume plugins in the /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ directory. Eg. nodeagent~uds and cifs Kubelet logs will now start complaining that the plugins are missing. Add the files to the plugin folders and then start a pod using the plugin. Notice that only the last plugin file copied is available to be used by pods. The first plugin will generate this error if used by a volume "err=no volume plugin matched" Performing a touch on the folder of the missing plugin will cause kubelet to properly index it. <details</details<detailsLocal kube cluster </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig k8s-infra
/sig storage /remove-sig k8s-infra


Question: import ( "context" <ins> "errors" </ins> "fmt" "io" "strconv"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Conformance test failed: xref What you expected to happen: I expected this conformance test to pass. How to reproduce it (as minimally and precisely as possible): We have configured docker's logging driver to use journald, and then ran sonobouy tests against a k8s 1.16.3 cluster. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Red Hat Enterprise Linux Server release 7.7 (Maipo) Kernel (e.g. ): 5.3.11- SMP Tue Nov 12 14:45:59 EST 2019 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig cli
/assign
I have a working patch that fixes this, will push a PR shortly.


Question: "LOGS_DIR" = ${kube_env}['LOGS_DIR'] "MANIFESTS_DIR" = ${kube_env}['MANIFESTS_DIR'] "INFRA_CONTAINER" = ${kube_env}['WINDOWS_INFRA_CONTAINER'] <ins> "WINDOWS_ENABLE_PIGZ" = ${kube_env}['WINDOWS_ENABLE_PIGZ'] </ins> "Path" = ${env:Path} + ";" + ${kube_env}['NODE_DIR'] "KUBE_NETWORK" = "l2bridge".ToLower()

<!-- Please only use this template for submitting enhancement requests --What would you like to be : pigz for GCE Windows Why is this needed: Pigz improves the image layer extraction performance for multi-core systems. Pigz is usually preinstalled on Linux. This enhancement is scoped specifically for the GKE Windows node scripts. For Context:
/assign me
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
GitHub didn't allow me to assign the following users: me. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign me Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows /assign jeremyje
s/GKE/GCE/? as in kube-up?
Replaced but I don't understand the kube-up part.
is how one creates a GCE cluster from this repo. Otherwise we'd be talking about something that isn't part of kubernetes.


Question: // // In general, in order to preserve a well-defined interface between the // kubelet and the container runtime, annotations SHOULD NOT influence <del> // runtime behaviour. For legacy reasons, there are some annotations which // currently explicitly break this rule, listed below; in future versions // of the interface these will be promoted to typed features. </del> <ins> // runtime behaviour. </ins> // // Annotations can also be useful for runtime authors to experiment with // new features that are opaque to the Kubernetes APIs (both user-facing // and the CRI). Whenever possible, however, runtime authors SHOULD // consider proposing new typed fields for any new features instead. <del> // // 1. Seccomp // //      key: security.alpha.kubernetes.io/seccomp/pod //      description: the seccomp profile for the containers of an entire pod. //      value: see below. // //      key: security.alpha.kubernetes.io/seccomp/container/<container name> //      description: the seccomp profile for the container (overrides pod). //      value: see below // //      The value of seccomp is runtime agnostic: //      * runtime/default: the default profile for the container runtime //      * unconfined: unconfined profile, ie, no seccomp sandboxing //      * localhost/<profile-name>: the profile installed to the node's //        local seccomp profile root. Note that profile root is set in //        kubelet, and it is not passed in CRI yet, see https://issues.k8s.io/36997. // </del> map<string, string> annotations = 7; // Optional configurations specific to Linux hosts. LinuxPodSandboxConfig linux = 8;

CRI supports AppArmor, Seccomp and Sysctls via annotations , we should move them to typed and structured fields instead. (refer ) [x] Apparmor () [x] Sysctls () [x] Seccomp () cc
/cc
/sig node
Seccomp is addressed at .


Question: // // In general, in order to preserve a well-defined interface between the // kubelet and the container runtime, annotations SHOULD NOT influence <del> // runtime behaviour. For legacy reasons, there are some annotations which // currently explicitly break this rule, listed below; in future versions // of the interface these will be promoted to typed features. </del> <ins> // runtime behaviour. </ins> // // Annotations can also be useful for runtime authors to experiment with // new features that are opaque to the Kubernetes APIs (both user-facing // and the CRI). Whenever possible, however, runtime authors SHOULD // consider proposing new typed fields for any new features instead. <del> // // 1. Seccomp // //      key: security.alpha.kubernetes.io/seccomp/pod //      description: the seccomp profile for the containers of an entire pod. //      value: see below. // //      key: security.alpha.kubernetes.io/seccomp/container/<container name> //      description: the seccomp profile for the container (overrides pod). //      value: see below // //      The value of seccomp is runtime agnostic: //      * runtime/default: the default profile for the container runtime //      * unconfined: unconfined profile, ie, no seccomp sandboxing //      * localhost/<profile-name>: the profile installed to the node's //        local seccomp profile root. Note that profile root is set in //        kubelet, and it is not passed in CRI yet, see https://issues.k8s.io/36997. // </del> map<string, string> annotations = 7; // Optional configurations specific to Linux hosts. LinuxPodSandboxConfig linux = 8;

Seccomp profile is passed to runtime by in CRI: is relative to node's local seccomp profile root, it is defined in kubelet. But runtime doesn't know it. We should pass full profile path in CRI. cc/ Will file a PR to fix this later.
Add support for Seccomp sounds like a new feature. The milestone should be reset to , correct?
not really. The current implementation supports seccomp already. If it's missing critical information to function properly, this would be a bug.
are we blocking releases for CRI in 1.5? i was not sure, so marking this as non-release-blocker and leaving 1.5 milestone. please correct if i am mistaken.
I don't get why this is a 1.5 targetted thing myself. Yes we want CRI to be in a good state in 1.5, but afaik we have zero intention of anyone actually using (at the very least because of the lack of exec auth regressing kubelet security). why should this be part of 1.5?
First, this is a small enough change that allows non-docker integrations to properly support seccomp. Second, if "we have zero intention of anyone actually using" is a valid reason, we should never fix any bugs in CRI. We do want people to try using it (perhaps not in production yet). However, given that seccomp is only in alpha, I am ok with dropping this and just continue the docker support for now.
Perhaps I misunderstood the meaning of code freeze, but I thought it was only bug-fixes regardless of size of change (or exceptions that have gone through some process) What I meant is we don't intend this to be used generally as part of the 1.5 release. Because of that, I don't think we should consider bugs in CRI bugs for the purpose of the 1.5 release freeze. My statement is just that we shouldn't violate code-freeze for this. We should totally fix this right after the 1.5 freeze window ends, and saying that I'm arguing we should never fix anything is disingenuous and taking it to an absurd extreme.
the only thing we don't seem to agree on is whether seccomp support was considered to be supported for CRI in 1.5.


Question: // // In general, in order to preserve a well-defined interface between the // kubelet and the container runtime, annotations SHOULD NOT influence <del> // runtime behaviour. For legacy reasons, there are some annotations which // currently explicitly break this rule, listed below; in future versions // of the interface these will be promoted to typed features. </del> <ins> // runtime behaviour. </ins> // // Annotations can also be useful for runtime authors to experiment with // new features that are opaque to the Kubernetes APIs (both user-facing // and the CRI). Whenever possible, however, runtime authors SHOULD // consider proposing new typed fields for any new features instead. <del> // // 1. Seccomp // //      key: security.alpha.kubernetes.io/seccomp/pod //      description: the seccomp profile for the containers of an entire pod. //      value: see below. // //      key: security.alpha.kubernetes.io/seccomp/container/<container name> //      description: the seccomp profile for the container (overrides pod). //      value: see below // //      The value of seccomp is runtime agnostic: //      * runtime/default: the default profile for the container runtime //      * unconfined: unconfined profile, ie, no seccomp sandboxing //      * localhost/<profile-name>: the profile installed to the node's //        local seccomp profile root. Note that profile root is set in //        kubelet, and it is not passed in CRI yet, see https://issues.k8s.io/36997. // </del> map<string, string> annotations = 7; // Optional configurations specific to Linux hosts. LinuxPodSandboxConfig linux = 8;

My interpretation is that we assumed seccomp would be supported until we found out that it wouldn't for other runtimes due to the lack of profile path. That's the only reason why I considered this a bug. Of course we could say that this is not really an important feature in CRI, so we can drop it, but we can be more clear on that.
If there's a bug in a feature no one uses, does fixing it make a sound? I think what we're not agreeing on is the role of CRI in 1.5. I understood it as something that is not generally usable. We can't recommend a customer use it, even as an experimental feature, due to security aspects (if nothing else). Correct me if I'm wrong. It's a developer integration point, and developers integrating with it are doing so with the kubernetes master branch, not with some release. Those reasons together result in my opinion that CRI is still not really a customer/release feature, and thus bugs or features related to it should not be exceptions to the freeze window, but rather should targeted at master as soon as the freeze lifts. If this were already post-freeze, would we be cherry-picking this over to 1.5?
This is not true. cri-o is depending on this fix for apparmor support.
is it depending on the 1.5 release branch directly, or on master?
I think it's better to make this in 1.5, so cri-o could work with first release with experimental CRI. It helps to encourage people to use CRI and find features misses or bugs.
Modifying the user-specified annotation is a little bit unsettling. I think adding a new annotation explicitly for the path would be better. Or, we should just add an explicit field for this. I spoke to and separately. I still think this is a bug fix, but we may want to be more careful about the fix. Given the time constraint, let's just list this as a known issue and work on it for 1.6.
OK, let's move this to next release and think carefully about seccomp feature besides this: kubelet may just pass the profile as string via CRI; may assume the profile locate in a file in a known format and pass path to the runtime; or maybe some other solutions.


Question: // // In general, in order to preserve a well-defined interface between the // kubelet and the container runtime, annotations SHOULD NOT influence <del> // runtime behaviour. For legacy reasons, there are some annotations which // currently explicitly break this rule, listed below; in future versions // of the interface these will be promoted to typed features. </del> <ins> // runtime behaviour. </ins> // // Annotations can also be useful for runtime authors to experiment with // new features that are opaque to the Kubernetes APIs (both user-facing // and the CRI). Whenever possible, however, runtime authors SHOULD // consider proposing new typed fields for any new features instead. <del> // // 1. Seccomp // //      key: security.alpha.kubernetes.io/seccomp/pod //      description: the seccomp profile for the containers of an entire pod. //      value: see below. // //      key: security.alpha.kubernetes.io/seccomp/container/<container name> //      description: the seccomp profile for the container (overrides pod). //      value: see below // //      The value of seccomp is runtime agnostic: //      * runtime/default: the default profile for the container runtime //      * unconfined: unconfined profile, ie, no seccomp sandboxing //      * localhost/<profile-name>: the profile installed to the node's //        local seccomp profile root. Note that profile root is set in //        kubelet, and it is not passed in CRI yet, see https://issues.k8s.io/36997. // </del> map<string, string> annotations = 7; // Optional configurations specific to Linux hosts. LinuxPodSandboxConfig linux = 8;

We need to clearly define it. define profile format clearly
Please document known issues for 1.5 in
(copied from ) And let's move here to discuss how to pass profiles to runtime. Related with .
This is issue is related with two level APIs: CRI, since we may also want to support profiles from ConfigMaps, I think we should pass full profile contents instead of path to runtime. issues is also related with seccomp api in kube-apiserver, it is based on annotations today. We should define seccomp profile format clearly (see ) and decide whether to add struct types to pod spec or just continue the annotations way. I think it's more clear to add specific struct types to pod spec.
Agreed - the path stuff is really a crutch (and implementation-specific) - I think is the answer here.
I don't disagree, but it's not strictly a requirement for supporting ConfigMaps (kubelet could create a file with the contents). An open question is whether to pass it as a raw string, or wait for and pass the structured data. The annotations way is deprecated (). I think the bigger question is where/how to put the data. E.g. structured data in the API, unstructured (string type) data in the API, reference to another object (e.g. ConfigMap). If the Kubelet needs to parse the spec, I think that's a good argument for a structured data type (either a standalone object or embedded in the pod API).
Yes, already opened to replace annotations with typed structs. I personally prefer a structured data type in both kubernetes API and CRI. WDYT?
A counter argument is that as the spec grows or new features are , we need to keep updating the Kubernetes API to maintain parity, or update the code in each CRI implementation to translate the Kubernetes spec. OTOH, if we use unstructured data we can use 3rd party libraries to parse out the data we need (assuming the spec evolves in a backwards compatible way).
What spec?
Whichever seccomp profile spec is being implemented by runtimes. E.g. if we decide to base the Kubernetes seccomp profile API off the OCI seccomp profile spec, and OCI adds some new features, at some point we'd probably want to add those features to Kubernetes.


Question: // // In general, in order to preserve a well-defined interface between the // kubelet and the container runtime, annotations SHOULD NOT influence <del> // runtime behaviour. For legacy reasons, there are some annotations which // currently explicitly break this rule, listed below; in future versions // of the interface these will be promoted to typed features. </del> <ins> // runtime behaviour. </ins> // // Annotations can also be useful for runtime authors to experiment with // new features that are opaque to the Kubernetes APIs (both user-facing // and the CRI). Whenever possible, however, runtime authors SHOULD // consider proposing new typed fields for any new features instead. <del> // // 1. Seccomp // //      key: security.alpha.kubernetes.io/seccomp/pod //      description: the seccomp profile for the containers of an entire pod. //      value: see below. // //      key: security.alpha.kubernetes.io/seccomp/container/<container name> //      description: the seccomp profile for the container (overrides pod). //      value: see below // //      The value of seccomp is runtime agnostic: //      * runtime/default: the default profile for the container runtime //      * unconfined: unconfined profile, ie, no seccomp sandboxing //      * localhost/<profile-name>: the profile installed to the node's //        local seccomp profile root. Note that profile root is set in //        kubelet, and it is not passed in CRI yet, see https://issues.k8s.io/36997. // </del> map<string, string> annotations = 7; // Optional configurations specific to Linux hosts. LinuxPodSandboxConfig linux = 8;

It also means that if different runtimes support different feature sets, we'll be stuck either tracking only the common features, or specifying features that may be ignored by the runtime. This may be a non-issue if the seccomp feature set rarely changes.
Well, yes, but that applies to everything in the CRI, and is one of the tradeoffs we ma[d|k]e by introducing this abstraction in the first place.
My $0.02: Moving the existing logic/information for seccomp into a first-class field for CRI (as in ) is productive because of the known problems with the annotation approach. My initial proposal for seccomp support included a resource similar to what has proposed in kubernetes/community and we were unable to come to an agreement on the issue refers to here: I do not necessarily expect that we will be able to come to an agreement about the seccomp resource in time for 1.7, so I think we should proceed with and do backward compatible CRI changes later if we need to inject the content of the seccomp profile instead of the path. I think we would deeply regret embedding API surface to describe seccomp profiles in the Pod API :)


Question: assign bool) (assigned mountDevice, alreadyAttached bool, err error) { deviceMappings := map[mountDevice]EBSVolumeID{} <ins> volumeStatus := map[EBSVolumeID]string{} // for better logging of volume status </ins> for _, blockDevice := range info.BlockDeviceMappings { name := aws.StringValue(blockDevice.DeviceName) if strings.HasPrefix(name, "/dev/sd") {

E2e tests occasionally flake on AWS. It happens randomly to all tests that create a pod with a volume, write something to the volume, delete the pod and create a new one that reads from the volume. For example: /D controller detaches a volume from a node (= the old pod died) after that (under 1s) it tries to attach the same volume to the same node (=the new pod is scheduled and coincidentally it uses the same node). In most cases this works, however, few times per day it happens that the second attach (in Kubernetes terms, see below) does not succeed and the test times out. AWS cloud provider in Kubernetes heavily uses DescribeInstances and DescribeVolumes to check that a volume has been attached or detached: In step 1. above, Kubernetes issues AWS API call and polls until it reports that the volume has been detached. In step 2. above, Kubernetes issues AWS API call to check what devices are used on a node and then it tries to find a free device for the volume that should be attached. In this bug, in step 2 reports the volume as detaching, even that reported the volume as detached in step 1. There was no call in between! Kubernetes does not expect that this could ever happen and does not try attach the volume (it's not reported as detached, so why bother, right?). This is the code that finds out that the volume is not detached ("detaching" != "detached"): And few lines below we skip : We (OpenShift) run lot of Kubernetes e2e tests on AWS. We run them in a single AWS project and I guess we hit some timing issues / races on AWS side. We hit also AWS API quota pretty hard, but it does not seem to be related to this bug, I mention it only to illustrate the amount of things that run in parallel and that AWS may not like that load. Environment: Kubernetes v1.13.4 /kind bug /sig aws /sig storage
Flow from AWS API perspective: So far so good. = volume vol- is back detaching: All timing is related to a response received, processed and logged by Kubernetes, which may be different to time corresponding request was sent. Everything in this report happens in sequence, there is no pararellism for a single volume.


Question: assign bool) (assigned mountDevice, alreadyAttached bool, err error) { deviceMappings := map[mountDevice]EBSVolumeID{} <ins> volumeStatus := map[EBSVolumeID]string{} // for better logging of volume status </ins> for _, blockDevice := range info.BlockDeviceMappings { name := aws.StringValue(blockDevice.DeviceName) if strings.HasPrefix(name, "/dev/sd") {

(Other volumes may be and often are attached / detached in parallel to this flow).
this looks like a bug in AWS itself, seems to return old data randomly. Can you please check AWS side of things? And we probably need fix / workaround on Kubernetes side. How can we check that a volume has been 100% detached if returns inconsistent data? We can check if a volume is in and not report is as attached here: This will cause Kubernetes to re-use the same device name and call . How bulletproof is it going to be? And most importantly, what other parts of Kubernetes depend on being consistent?
I confirmed internally. This is expected behavior for and since ec2 API follows a where resource mutation might be not reflected immediately. This could explain why returns a volume in detaching state even after the previous call returns without the volume (detached) I think this won't work in the case when the volume is actually stuck in detaching state, since attaching a stuck volume could simply make it worse. Given that, how about, in call, we return error when notice the volume is still being returned from and we rely on the AD controller retry to check the state next time, and hopefully the state converges.
If can return stale data, depending on it in (or anywhere else) probably won't help, because we may again get new or stale data "randomly". We call in today to get fresh list of used/free devices. Depending on timing, two cases happen: Sometimes it returns that the volume is detaching and I can see in logs. Handling as you suggest would help here. Sometimes it returns that the volume is detached. We find a new device name and call AWS and start polling for finished attach. This then returns that the volume is detaching from the node with the old device name. So we're actually attaching a volume that's being detached and you wrote that's a bad thing. In my logs, I can see that Kubernetes recovers from this error: Volume plugin AttachVolume() fails with A/D controller re-tries AttachVolume() calls AttachDisk() that calls to find a free device. returns that the volume is actually being attached to the node (I can see in logs) AttachDisk() polls until the volume is attached. -it works, but to me it feels like it's just an accident. Checking can't help in this case.


Question: assign bool) (assigned mountDevice, alreadyAttached bool, err error) { deviceMappings := map[mountDevice]EBSVolumeID{} <ins> volumeStatus := map[EBSVolumeID]string{} // for better logging of volume status </ins> for _, blockDevice := range info.BlockDeviceMappings { name := aws.StringValue(blockDevice.DeviceName) if strings.HasPrefix(name, "/dev/sd") {

So, is there another way how to check that a volume is really detached? Or should we promote the error flow described above to official fix and add bunch of comments around so we won't remove / tweak the flow in future? What bad can happen when we call AWS on a volume that's detaching? AWS at least once reported that the volume is detached, so the volume probably is not be stuck.
I think it is only bad if the volume was previously stuck. But feels acceptable for me if it is being detached and the previous call in at least confirmed once. Confirming internally. But I suspect there isn't any. As long as the volume is not stuck (unmounted properly), we should be able to attach a detaching volume especially when it is at least confirmed being detached once from call in . Even if the first time it doesn't work, retry should help recover the situation. How do you think?
According pointed, AWS recommends two actions to manage the eventual consistency model (k8s already does 1, but AFAIK not 2, except when there's an exponential backoff among calls: 1) Run the appropriate call before modifying a resource 2) Add wait time between subsequent commands, even if returns a green response (this does infer that might return stale data) Currently, k8s assumes a state based on a single green response, e.g., is called repeatedly with an exponential backoff until it returns a given volume state. What if it required at least two positive responses instead? What are the chances that would return stale data twice? I'd assume that two calls returning a detached volume (with exponential backoff in between, which enforces point 2 above) would be reliable enough. The downside is that this would increase the API usage, which is already high.
do you know if the native AWS calls may be more stale-data-proof?
Looking at implementation, it tries with constant backoff of 15 seconds (which I consider pretty large) and it returns when the first returns that the volume is in use, so basically the same as what Kubernetes does.
From what I saw, sometimes it happens that reports a volume as detached, subsequent reports that the volume is not attached, succeeds and only after that shows that the volume is (from the old device name) instead of . So Kubernetes got at least two confirmations that the volume is detached.


Question: assign bool) (assigned mountDevice, alreadyAttached bool, err error) { deviceMappings := map[mountDevice]EBSVolumeID{} <ins> volumeStatus := map[EBSVolumeID]string{} // for better logging of volume status </ins> for _, blockDevice := range info.BlockDeviceMappings { name := aws.StringValue(blockDevice.DeviceName) if strings.HasPrefix(name, "/dev/sd") {

What would be the correct delay to wait for second (third, ...) confirmation that the volume is really detached? IMO it would hurt our performance a lot.
The Wait function simply calls DescribeXX under the hood and uses some hard coded timeout. So it's similar to what kubernetes does like said.


Question: if effectiveSc.RunAsUser != nil { if *effectiveSc.RunAsUser == 0 { <del> return fmt.Errorf("container's runAsUser breaks non-root policy") </del> <ins> return fmt.Errorf("container's runAsUser breaks non-root policy (pod: %q, container: %s)", format.Pod(pod), container.Name) </ins> } return nil } switch { case uid != nil && *uid == 0: <del> return fmt.Errorf("container has runAsNonRoot and image will run as root") </del> <ins> return fmt.Errorf("container has runAsNonRoot and image will run as root (pod: %q, container: %s)", format.Pod(pod), container.Name) </ins> case uid == nil && len(username) > 0: <del> return fmt.Errorf("container has runAsNonRoot and image has non-numeric user (%s), cannot verify user is non-root", username) </del> <ins> return fmt.Errorf("container has runAsNonRoot and image has non-numeric user (%s), cannot verify user is non-root (pod: %q, container: %s)", username, format.Pod(pod), container.Name) </ins> default: return nil }

when deploying a container that has runAsNonRoot but runs as root this error appears, but for pods with lots of containers it can be confusing which container failed, especially if there are 3-rd party webhooks that inject containers please add the container name to the message, making it
/sig api-machinery
/assign I can try this fix
I just submitted a pr, can you review it for me, is it as you expected?
thx, looks good!
/remove-sig api-machiner /sig node /area kubelet
Those labels are not set on the issue: <detailsIn response to : /remove-sig api-machiner /sig node /area kubelet Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig api-machinery


Question: policies.  An API call must be allowed by some policy in order to proceed. The following implementations are available, and are selected by flag: <del> - `--authoriation_mode=AlwaysDeny` - `--authoriation_mode=AlwaysAllow` - `--authoriation_mode=ABAC` </del> <ins> - `--authorization_mode=AlwaysDeny` - `--authorization_mode=AlwaysAllow` - `--authorization_mode=ABAC` </ins> `AlwaysDeny` blocks all requests (used in tests). `AlwaysAllow` allows all requests; use if you don't need authorization.

In CPU usage went up significantly compared to It consumes more than one core even when there's no significant logging load on the node. Two changes between those version: CC
Ruby by default is single-core, so anything over that is likely introduced by the changes. But we'd need to investigate to make sure.
Can you please disable monitoring and see whether this is the problem?
I meant can you ...
Was going to do this, yes
Built container manually without prometheus plugin, still consumes 1.7 cpu
Can you please find the difference then?
Used new image with the old configuration (), still 1.7 cores It seems like a problem with the new image for some reason. FYI
What can consume 1.7 cores with the same configuration on a different machine? How to debug this?
Does this mean all components are same, ruby version, plugins and configuration settings? What are the differences between old and new image? OS or other parts are also different?
There are at least the following changer Ubuntu Trusty -Debian Jessie td-agent package -manually installed embedded ruby -ruby 2.1 embedded gems -fixed version gems from Gemfile embedded libjemalloc - I understand that it's a lot of changes and I'm going to narrow the problem down, the main question is where to look and how to debug to speed this process up?
If this is not fixed before Monday we will revert . cc
Only and should be reverted, not
No, we want to revert migration to the new base image. The changes to extract configuration into env var/config map should preserve. This mean that the revert might not be straightforward.
I think our backup option is to switch back to using td-agent package, not reverting whole change, because is also about security and we don't want to revert that I will check whether that helps
Thanks for investigating this! I have a couple comments: The full package diff can be found here: The first thing I would look at is whether the ruby version makes a difference. I believe td-agent was using 2.3 The main benefit of is actually not using td-agent, since the td-agent package includes lots of stuff we don't use (e.g. rails). Maybe if we need to fallback we could manually remove those extra packages. I'll poke around in the container and see if anything jumps out.


Question: policies.  An API call must be allowed by some policy in order to proceed. The following implementations are available, and are selected by flag: <del> - `--authoriation_mode=AlwaysDeny` - `--authoriation_mode=AlwaysAllow` - `--authoriation_mode=ABAC` </del> <ins> - `--authorization_mode=AlwaysDeny` - `--authorization_mode=AlwaysAllow` - `--authorization_mode=ABAC` </ins> `AlwaysDeny` blocks all requests (used in tests). `AlwaysAllow` allows all requests; use if you don't need authorization.

Thanks for the comments, they totally make sense. Also, thanks for the diff. Thanks!
Actually td-agent was using ruby 2.1:
A manual search through commenting out parts of the config file revealed that removing the systemd config solves the issue: I'll play around with the systemd libraries and see if I can figure out what's going on here.
Still investigating, but I think this may be a regression in the fluent-plugin-systemd library between version 0.0.5 (old image) and 0.0.7 (new image). I realized the diff I linked above was not in sync with the version that was actually pushed (). That diff is here, and accurately reflects the change in the systemd library:
Confirmed this is a regression in (0.0.6 does not have the problem). Fortunately 0.0.7 only has a single commit, I'll file an issue on the plugin. In the meantime, I'll send a PR to pin fluent-plugin-systemd to 0.0.6.
Thanks a huge lot for investigating and fixing this!
Reopening to track the version bump.
Good investigation. Latest systemd plugin uses and it seems to cause busy loop like problem.
Hi I just released that should fix this issue.
Thanks! And thanks to who already included it into our image :)
Yeah thanks your too fast for me :)
Thank you a lot for the investigation.


Question: patch := getDaemonSetImagePatch(ds.Spec.Template.Spec.Containers[0].Name, RedisImage) ds, err = c.AppsV1().DaemonSets(ns).Patch(dsName, types.StrategicMergePatchType, []byte(patch)) Expect(err).NotTo(HaveOccurred()) <del> templateGeneration++ </del> By("Check that daemon pods images are updated.") err = wait.PollImmediate(dsRetryPeriod, dsRetryTimeout, checkDaemonPodsImageAndAvailability(c, ds, RedisImage, 1)) Expect(err).NotTo(HaveOccurred()) <del> By(fmt.Sprintf("Make sure all daemon pods have correct template generation %d", templateGeneration)) err = checkDaemonPodsTemplateGeneration(c, ns, label, fmt.Sprint(templateGeneration)) Expect(err).NotTo(HaveOccurred()) </del> By("Check that daemon pods are still running on every node of the cluster.") err = wait.PollImmediate(dsRetryPeriod, dsRetryTimeout, checkRunningOnAllNodes(f, ds)) Expect(err).NotTo(HaveOccurred(), "error waiting for daemon pod to start")

Please see : https://k8s- fails just after this line: Failure was introduced in: But this was not noticed until it started running in [Conformance] tests. You can see the same problem in other dashboards too: https://k8s- https://k8s-
cc /sig apps
/priority critical-urgent
/kind bug /priority failing-test
/milestone v1.10
Is this just an oversight of not removing the template generation when the API update was done?
/cc
triaged in progress
i have never touched this test before ... so i just threw in a WIP, feel free to use/discard it. there seems to be lot of methods in there that is no longer used, may be we can rip those out too. (see )
xref
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action Required: This issue has not been updated since Mar 5. Please provide an update. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: f := framework.NewDefaultFramework("metrics-grabber") var c, ec clientset.Interface var grabber *e2emetrics.Grabber <ins> var masterRegistered bool </ins> ginkgo.BeforeEach(func() { var err error c = f.ClientSet ec = f.KubemarkExternalClusterClientSet <ins> // Check if master Node is registered nodes, err := c.CoreV1().Nodes().List(context.TODO(), metav1.ListOptions{}) </ins> framework.ExpectNoError(err) <ins> for _, node := range nodes.Items { if strings.HasSuffix(node.Name, "master") { masterRegistered = true } } </ins> gomega.Eventually(func() error { grabber, err = e2emetrics.NewMetricsGrabber(c, ec, true, true, true, true, true) if err != nil { return fmt.Errorf("failed to create metrics grabber: %v", err) } <del> if !grabber.HasControlPlanePods() { </del> <ins> if masterRegistered && !grabber.HasControlPlanePods() { </ins> return fmt.Errorf("unable to get find control plane pods") } return nil

MetricsGrabber tests failing when scheduler & controller-manager not visible. In managed offerings, like GKE, scheduler and controller-manager are not visible causing BeforeSuite of MetricsGrabber to fail. Test is failing then control plane pods can't be found (as on GKE): HasControlPlanePods was in /assign
/sig instrumentation
Just removing this check allows tests to run without problems (some skipped, one passing).
/triage accepted


Question: proxyPort           int socket              proxySocket timeout             time.Duration <ins> activeClients       *clientCache </ins> nodePort            int loadBalancerStatus  api.LoadBalancerStatus sessionAffinityType api.ServiceAffinity

Depends on UDP clients have a timeout (so we don't leak connections forever), but the test does not cover this. To do this, we probably need to track active connections, and plumb that all the way down to the goroutines spawned by the ProxyLoop routines. This is probably something we want to do anyway to be able to monitor.


Question: nt.removeZone(zone) } klog.V(5).Infof("Removed node %v in group %v from NodeTree", n.Name, zone) <del> nt.NumNodes-- </del> <ins> nt.numNodes-- </ins> return nil } }

Description: , part of the is exporting a field which allows to modify its value externally. This also means that this field is not thread-safe. Solution: Expose a method that returns this value in a thread-safe way and do not export the field. /kind bug /sig scheduling


Question: <del> approvers: - luxas - timothysc - fabriziopandini reviewers: - luxas - timothysc - fabriziopandini labels: - kind/documentation </del>

Instead, have a presubmit test that verifies that the docs can be generated. This will reduce the number of conflicts & rebases, therefore increase speed through the queue. Confirmed with that nothing depends on these docs at the moment. fyi, some manual process might have to change once this is in.
Yes, we want to generate the docs into , anyway. cc
Excellent. One closer step to YAML generation, I hope. :)
was interested in working on this.
Want to get started on this?
is this still a goal of ours?
I've seen a thread this week-end about that. I think docs have moved but we still need to generate them there.
can this issue be closed then?
related to ?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen There's still cruft in the docs directory.
/remove-help I don't think this meets . Please add back via if I'm incorrect
/sig docs /sig release /remove-area test-infra /area build-release /area kubeadm /area kubectl All of the following dirs/files have the following content. Do we need them? This file is autogenerated, but we've stopped checking such files into the repository to reduce the need for rebases. Please run hack/generate- to populate this file. docs/admin: .md files for kube components and kubeadm commands man/man1: .1 files for same user-guide/kubectl: .md files for kubectl commands yaml/kubectl: .yaml files for kubectl commands
/help Now that we have /good-first-issue, I think this does meet the criteria for help wanted. Probably need to talk to someone from sig-docs and figure out if/how these are used.
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command.


Question: <del> approvers: - luxas - timothysc - fabriziopandini reviewers: - luxas - timothysc - fabriziopandini labels: - kind/documentation </del>

<detailsIn response to : /help Now that we have /good-first-issue, I think this does meet the criteria for help wanted. Probably need to talk to someone from sig-docs and figure out if/how these are used. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-priority important-soon /priority important-longterm This hasn't been committed to for any release cycle in a while. I hope someone does commit to it, but I personally cannot, so I'm bumping down to the "hopefully next time" priority. /kind cleanup in addition to docs This impacts merge velocity of any kubectl/kubeadm changes because the docs/OWNERS file is small. We're going to work around by adding OWNERS files for the relevant sigs to the relevant dirs. But if these files don't have to exist in the first place, that'd be even better.
as discovered today this would not work as OWNERS in sub-folders do not solve the problem with the /docs/.generated_docs file which is in the parent folder. i've been looking into a solution to remove the placeholders completely and make verify-docs only consist of kube::util::gen-docs. this will: ensure that godoc errors and such still block presubmits the contributors are not blocked having to commit placeholder files
/lifecycle active
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle active as PR is still in flight.


Question: import ( "fmt" <ins> appsv1 "k8s.io/api/apps/v1" </ins> appsv1beta2 "k8s.io/api/apps/v1beta2" extensionsv1beta1 "k8s.io/api/extensions/v1beta1" apiequality "k8s.io/apimachinery/pkg/api/equality"

/priority failing-test /priority critical-urgent /sig apps This test case has been failing for a while and affects a number of jobs: This is affecting multiple jobs on the testgrid dashboard, and prevents us from cutting 1.9.0-alpha.2 (kubernetes/sig-release). Is there work ongoing to bring this job back to green?
All test failures look similar: . ~Is it due to network issue instead of DaemonSet?~
This seems related to the new DaemonSet. The connection refused error is happening because kube-apiserver is crashing:
needs an update
Ah, well... I guess it's halfway working as intended. We put that there to make the tests fail if we forgot to update it when adding a new version. We had hoped it would fail during presubmit tests though. :(
Sending a fix now
I wonder why the test is not failing on submit queue tests. Do i need to fix deployment and replicaset?
The rest of the group is not merged, and the v1 PR takes the selector mutability into account.
Since this bug crashes apiserver I'd expect it to affect many other e2e tests.
Noted, we need to fix deployment and replicaset when the v1 PR is merged.
Not quite sure how this test isn't failing before but is now consistently failing. I'd expect the commit that triggers this to block the submit queue. The e2e test is still accessing DaemonSet through extensions/v1beta1 endpoints. From the above apiserver log it's hard to tell what updates DaemonSet via apps/v1 before the code panics. I tried to reproduce it locally, and found that garbage collector PUT apps/v1 DaemonSet:
When apps/v1 gets introduced, it may be ordered before apps/beta1 alphabetically. IIRC, there is a code change on the apps/v1 PR that changes the order of API versions such that apps/v1 is ranked before apps/v1beta2 and apps/v1beta2 before apps/v1beta1. Perhaps this problem can be fixed when the PR merges.
Since crashes apiserver and makes debugging harder, which defeats the purpose of using it, we should return error on unsupported versions instead.
The first run with this change is still in progress, but this test already passed: https://k8s- Thanks for the quick fix


Question: }, { evictionPriority: 0, <del> pod: diskConsumingPod("guaranteed-disk", 299 /* Mb */, nil, v1.ResourceRequirements{ </del> <ins> // Only require 99% accuracy (297/300 Mb) because on some OS distributions, the file itself (excluding contents), consumes disk space. pod: diskConsumingPod("guaranteed-disk", 297 /* Mb */, nil, v1.ResourceRequirements{ </ins> Requests: v1.ResourceList{ v1.ResourceEphemeralStorage: resource.MustParse("300Mi"), },

Is this a BUG REPORT or FEATURE REQUEST?: Bug Report /kind bug What happened: improved disk eviction testing to make it more precise. For example, it expects a pod which uses 299 Mb of disk not to be evicted if it sets its limit to 300 Mi. As shown on , this test passes for all OS distributions except CoreOS. I have ruled out an issue with the storage driver and backing filesystem since both COS and CoreOS use overlay backed by extfs. What you expected to happen: I expected the pod using 299Mb not to be evicted when its limit is 300 Mb. I expected not to see this event: How to reproduce it (as minimally and precisely as possible): The container runs this command to use 299Mb: Each iteration writes 1Mb to disk, producing a total of 299 Mb of disk usage. The pod has set to 300Mi. The test has the LocalStorageCapacityIsolation feature gate enabled. Anything else we need to know?: This may be a discrepancy in the (which we use to measure container disk usage) or , which is used by the test to write exact quantities of disk.
/sig node
cc can you take a look at this after thanksgiving?
I bumped the coreos image... going to see if it clears up the flake
It appears the root filesystem is being included in the disk space calculation . Do you think the coreos image might be slightly larger than the other images under test?
One more code snippet that is interesting:
thats the container's root filesystem
correct. the feature appears to add and compare to the quota: The reproducible steps do not take into account the rootfs space. If the quota is set to 300Mi, then the under loop count would have to take into account the rootfs space to not exceed the limit. I do not see anywhere in the tests or the reproduce steps this part of the calculation.
The refers only to the writable layer of the container, and excludes base (image) layers. With the provided command, the only files that are in the container's rootfs are the ones written by the command, and should amount to 299Mb. Additionally, the container does not write any logs (through stdout), so that value should be 0.


Question: }, { evictionPriority: 0, <del> pod: diskConsumingPod("guaranteed-disk", 299 /* Mb */, nil, v1.ResourceRequirements{ </del> <ins> // Only require 99% accuracy (297/300 Mb) because on some OS distributions, the file itself (excluding contents), consumes disk space. pod: diskConsumingPod("guaranteed-disk", 297 /* Mb */, nil, v1.ResourceRequirements{ </ins> Requests: v1.ResourceList{ v1.ResourceEphemeralStorage: resource.MustParse("300Mi"), },

I ran a quick test on a vm with the current image, and found that both and correctly wrote and measured disk space correctly. I also tested as it inside a container, and verified that we are measuring the container's writable layer correctly. My guess now is that it has something to do with how we calculate disk usage by logging with journald. I am looking into it
Actually, the previous comment was incorrect. When I ran the test, I found that each file was 1028 kb instead of 1024 kb, which resulted in the command writing 300.3 Mb instead of 299 Mb. I will change the test to require slightly less precision. This appears to be because an empty file in coreos takes up 4 kb. Test this by running . On other OS we test on, empty files use 0 kb. This is what accounts for the difference.
I tracked down an issue to the dd call. The stdout and stderr need a redirect to /dev/null to suppress the following messages: Problem Line: Fix:
I PRed that fix and changed it to stderr, since the other tests seem to use stderr.
leaving this open until we get a green run
I'll continue to test locally to see if I find anything else.
opened PR with the help of and CoreOS defaults to using the actual size instead of the apprent size.


Question: return f.waiter.destChan } <del> // Stop stops the timer and returns true if the timer has not yet fired, or false otherwise. </del> <ins> // Stop conditionally stops the timer.  If the timer has neither fired // nor been stopped then this call stops the timer and returns true, // otherwise this call returns false.  This is like time.Timer::Stop. </ins> func (f *fakeTimer) Stop() bool { f.fakeClock.lock.Lock() defer f.fakeClock.lock.Unlock() <del> newWaiters := make([]fakeClockWaiter, 0, len(f.fakeClock.waiters)) for i := range f.fakeClock.waiters { w := &f.fakeClock.waiters[i] if w != &f.waiter { newWaiters = append(newWaiters, *w) </del> <ins> // The timer has already fired or been stopped, unless it is found // among the clock's waiters. stopped := false oldWaiters := f.fakeClock.waiters newWaiters := make([]fakeClockWaiter, 0, len(oldWaiters)) seekChan := f.waiter.destChan for i := range oldWaiters { // Identify the timer's fakeClockWaiter by the identity of the // destination channel, nothing else is necessarily unique and // constant since the timer's creation. if oldWaiters[i].destChan == seekChan { stopped = true } else { newWaiters = append(newWaiters, oldWaiters[i]) </ins> } } f.fakeClock.waiters = newWaiters <del> return !f.waiter.fired </del> <ins> return stopped </ins> } <del> // Reset resets the timer to the fake clock's "now" + d. It returns true if the timer has not yet // fired, or false otherwise. </del> <ins> // Reset conditionally updates the firing time of the timer.  If the // timer has neither fired nor been stopped then this call resets the // timer to the fake clock's "now" + d and returns true, otherwise // this call returns false.  This is like time.Timer::Reset. </ins> func (f *fakeTimer) Reset(d time.Duration) bool { f.fakeClock.lock.Lock() defer f.fakeClock.lock.Unlock() <del> active := !f.waiter.fired f.waiter.fired = false f.waiter.targetTime = f.fakeClock.time.Add(d) return active </del> <ins> waiters := f.fakeClock.waiters seekChan := f.waiter.destChan for i := range waiters { if waiters[i].destChan == seekChan { waiters[i].targetTime = f.fakeClock.time.Add(d) return true } } return false </ins> } type Ticker interface {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I was writing a variant of clock.FakeClock, and noticed a couple of bugs in . Look at the pointer comparison in --- the two pointers will never be equal, but the author clearly expected equality when is the index of the fakeClockWaiter corresponding to the fakeTimer. The two pointers will never be equal because one is a pointer into the fakeClock's slice while the other pointer is to a field of the . The consequence of this bug is that the Stop function is ineffective. Also, determines the returned value based on the original copy of the waiter, which --- being a copy --- did not get updated if/when the waiter fired. Note also that does not exercise FakeClock::NewTimer at all. What you expected to happen: I expected FakeClock Timers to work properly. How to reproduce it (as minimally and precisely as possible): Use the TestFakeTimer func introduced in Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
The fake timer's Reset() method also does not work correctly.


Question: "//vendor/github.com/aws/aws-sdk-go/aws:go_default_library", "//vendor/github.com/aws/aws-sdk-go/aws/session:go_default_library", "//vendor/github.com/aws/aws-sdk-go/service/ec2:go_default_library", <ins> "//vendor/github.com/ghodss/yaml:go_default_library", </ins> "//vendor/github.com/onsi/ginkgo:go_default_library", "//vendor/github.com/onsi/gomega:go_default_library", "//vendor/google.golang.org/api/googleapi:go_default_library",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The local-volume bootstrapper/provisioner configuration format changed in kubernetes-incubator/external-storage, and its related e2e tests will fail when the provisioner & bootstrapper images are updated to v2.0.0. What you expected to happen: I'd expect the e2e tests to continue to work properly with the new version of the provisioner and boostrapper. How to reproduce it (as minimally and precisely as possible): Update local-volume provisioner and boostrapper images to v2.0.0, and then the provisioner e2e test will fail.
/sig storage
/assign


Question: ## kubectl The ```kubectl``` command provides command line access to the kubernetes API. <del> ### Commands #### version Print the version of the client and server. Usage: ``` kubectl version [flags] Available Flags: --api-version="v1beta1": The version of the API to use against the server -a, --auth-path="/Users/bburns/.kubernetes_auth": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a certificate file for the certificate authority -c, --client=false: Client version only (no server required) --client-certificate="": Path to a client certificate for TLS. --client-key="": Path to a client key file for TLS. -h, --help=false: help for version --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/Users/bburns/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": Kubernetes apiserver to connect to Use "kubectl help [command]" for more information about that command. ``` #### proxy Create a local proxy to the API server Usage: ``` kubectl proxy [flags] Available Flags: --api-version="v1beta1": The API version to use when talking to the server -a, --auth-path="/Users/bburns/.kubernetes_auth": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority=: Path to a cert. file for the certificate authority. --client-certificate=: Path to a client key file for TLS. --client-key=: Path to a client key file for TLS. -h, --help=false: help for proxy --insecure-skip-tls-verify=%!s(bool=false): If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/Users/bburns/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -p, --port=8001: The port on which to run the proxy -s, --server="": The address of the Kubernetes API server --token=: Bearer token for authentication to the API server. --validate=false: If true, use a schema to validate the input before sending it -w, --www="": Also serve static files from the given directory under the prefix /static ``` #### get Display one or more resources Possible resources include pods (po), replication controllers (rc), services (se), minions (mi), or events (ev). If you specify a Go template, you can use any fields defined for the API version you are connecting to the server with. Examples: ```sh $ kubectl get pods <list all pods in ps output format> $ kubectl get replicationController 1234-56-7890-234234-456456 <list single replication controller in ps output format> $ kubectl get -o json pod 1234-56-7890-234234-456456 <list single pod in json output format> ``` Usage: ``` kubectl get [(-o|--output=)json|yaml|...] <resource> [<id>] [flags] Available Flags: --api-version="v1beta1": The API version to use when talking to the server -a, --auth-path="/Users/bburns/.kubernetes_auth": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority=: Path to a cert. file for the certificate authority. --client-certificate=: Path to a client key file for TLS. --client-key=: Path to a client key file for TLS. -h, --help=false: help for get --insecure-skip-tls-verify=%!s(bool=false): If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --no-headers=false: When using the default output, don't print headers --ns-path="/Users/bburns/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -o, --output="": Output format: json|yaml|template|templatefile --output-version="": Output the formatted object with the given version (default api-version) -l, --selector="": Selector (label query) to filter on -s, --server="": The address of the Kubernetes API server -t, --template="": Template string or path to template file to use when --output=template or --output=templatefile --token=: Bearer token for authentication to the API server. --validate=false: If true, use a schema to validate the input before sending it -w, --watch=false: After listing/getting the requested object, watch for changes. --watch-only=false: Watch for changes to the requseted object(s), without listing/getting first. ``` #### describe Show details of a specific resource. This command joins many API calls together to form a detailed description of a given resource. Usage: ``` kubectl describe <resource> <id> [flags] Available Flags: --api-version="v1beta1": The API version to use when talking to the server -a, --auth-path="/Users/bburns/.kubernetes_auth": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority=: Path to a cert. file for the certificate authority. --client-certificate=: Path to a client key file for TLS. --client-key=: Path to a client key file for TLS. -h, --help=false: help for describe --insecure-skip-tls-verify=%!s(bool=false): If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/Users/bburns/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --token=: Bearer token for authentication to the API server. --validate=false: If true, use a schema to validate the input before sending it ``` ### run-container Easily run one or more replicas of an image. Creates a replica controller running the specified image, with one or more replicas. Examples: ```sh $ kubectl run-container nginx --image=dockerfile/nginx <starts a single instance of nginx> $ kubectl run-container nginx --image=dockerfile/nginx --replicas=5 <starts a replicated instance of nginx> $ kubectl run-container nginx --image=dockerfile/nginx --dry-run <just print the corresponding API objects, don't actually send them to the apiserver> Usage: kubectl run-container <name> --image=<image> [--replicas=replicas] [--dry-run=<bool>] [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use --dry-run=false: If true, only print the object that would be sent, don't actually do anything --generator="run-container-controller-v1": The name of the api generator that you want to use.  Default 'run-container-controller-v1' -h, --help=false: help for run-container --image="": The image for the container you wish to run. --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": Path to the kubeconfig file to use for CLI requests. -l, --labels="": Labels to apply to the pod(s) created by this call to run. --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --no-headers=false: When using the default output, don't print headers --ns-path="/Users/bburns/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -o, --output="": Output format: json|yaml|template|templatefile --output-version="": Output the formatted object with the given version (default api-version) -r, --replicas=1: Number of replicas to create for this container. Default 1 -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr -t, --template="": Template string or path to template file to use when -o=template or -o=templatefile. --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging  $ kubectl run nginx dockerfile/nginx <starts a single instance of nginx> ``` #### create Create a resource by filename or stdin. JSON and YAML formats are accepted. Examples: ```sh $ kubectl create -f pod.json <create a pod using the data in pod.json> $ cat pod.json | kubectl create -f - <create a pod based on the json passed into stdin> ``` Usage: ``` kubectl create -f filename [flags] Available Flags: --api-version="v1beta1": The API version to use when talking to the server -a, --auth-path="/Users/bburns/.kubernetes_auth": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority=: Path to a cert. file for the certificate authority. --client-certificate=: Path to a client key file for TLS. --client-key=: Path to a client key file for TLS. -f, --filename="": Filename or URL to file to use to create the resource -h, --help=false: help for create --insecure-skip-tls-verify=%!s(bool=false): If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/Users/bburns/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --token=: Bearer token for authentication to the API server. --validate=false: If true, use a schema to validate the input before sending it ``` #### createall Create all resources contained in JSON file specified in a directory, filename or stdin JSON and YAML formats are accepted. Examples: ```sh $ kubectl createall -d configs/ <creates all resources listed in JSON or YAML files, found recursively under the configs directory> $ kubectl createall -f config.json <creates all resources listed in config.json> $ cat config.json | kubectl apply -f - <creates all resources listed in config.json> ``` Usage: ``` kubectl createall [-d directory] [-f filename] [flags] Available Flags: --api-version="v1beta1": The API version to use when talking to the server -a, --auth-path="/Users/bburns/.kubernetes_auth": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority=: Path to a cert. file for the certificate authority. --client-certificate=: Path to a client key file for TLS. --client-key=: Path to a client key file for TLS. -d, --directory="": Directory of JSON or YAML files to use to update the resource -f, --filename="": Filename or URL to file to use to update the resource -h, --help=false: help for createall --insecure-skip-tls-verify=%!s(bool=false): If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/Users/bburns/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --token=: Bearer token for authentication to the API server. --validate=false: If true, use a schema to validate the input before sending it ``` #### update Update a resource by filename or stdin. JSON and YAML formats are accepted. Examples: ```sh $ kubectl update -f pod.json <update a pod using the data in pod.json> $ cat pod.json | kubectl update -f - <update a pod based on the json passed into stdin> ``` Usage: ``` kubectl update -f filename [flags] Available Flags: --api-version="v1beta1": The API version to use when talking to the server -a, --auth-path="/Users/bburns/.kubernetes_auth": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority=: Path to a cert. file for the certificate authority. --client-certificate=: Path to a client key file for TLS. --client-key=: Path to a client key file for TLS. -f, --filename="": Filename or URL to file to use to update the resource -h, --help=false: help for update --insecure-skip-tls-verify=%!s(bool=false): If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/Users/bburns/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --token=: Bearer token for authentication to the API server. --validate=false: If true, use a schema to validate the input before sending it ``` #### delete Delete a resource by filename, stdin, resource and id or by resources and label selector. JSON and YAML formats are accepted. If both a filename and command line arguments are passed, the command line arguments are used and the filename is ignored. Note that the delete command does NOT do resource version checks, so if someone submits an update to a resource right when you submit a delete, their update will be lost along with the rest of the resource. Examples: ```sh $ kubectl delete -f pod.json <delete a pod using the type and id pod.json> $ cat pod.json | kubectl delete -f - <delete a pod based on the type and id in the json passed into stdin> $ kubectl delete pods,services -l name=myLabel <delete pods and services with label name=myLabel> $ kubectl delete pod 1234-56-7890-234234-456456 <delete a pod with ID 1234-56-7890-234234-456456> ``` Usage: ``` kubectl delete ([-f filename] | (<resource> [(<id> | -l <label>)] [flags] Available Flags: --api-version="v1beta1": The API version to use when talking to the server -a, --auth-path="/Users/bburns/.kubernetes_auth": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority=: Path to a cert. file for the certificate authority. --client-certificate=: Path to a client key file for TLS. --client-key=: Path to a client key file for TLS. -f, --filename="": Filename or URL to file to use to delete the resource -h, --help=false: help for delete --insecure-skip-tls-verify=%!s(bool=false): If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/Users/bburns/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -l, --selector="": Selector (label query) to filter on -s, --server="": The address of the Kubernetes API server --token=: Bearer token for authentication to the API server. --validate=false: If true, use a schema to validate the input before sending it ``` #### logs Print the logs for a container in a pod. If the pod has only one container, the container name is optional Examples: ```sh $ kubectl log 123456-7890 ruby-container <returns snapshot of ruby-container logs from pod 123456-7890> $ kubectl log -f 123456-7890 ruby-container <starts streaming of ruby-container logs from pod 123456-7890> ``` Usage: ``` kubectl log [-f] <pod> [<container>] [flags] Available Flags: --api-version="v1beta1": The API version to use when talking to the server -a, --auth-path="/Users/bburns/.kubernetes_auth": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority=: Path to a cert. file for the certificate authority. --client-certificate=: Path to a client key file for TLS. --client-key=: Path to a client key file for TLS. -f, --follow=false: Specify if the logs should be streamed. -h, --help=false: help for log --insecure-skip-tls-verify=%!s(bool=false): If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/Users/bburns/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --token=: Bearer token for authentication to the API server. --validate=false: If true, use a schema to validate the input before sending it ``` ### Usage ``` Usage: kubectl [flags] kubectl [command] Available Commands: version                                                   Print version of client and server proxy                                                     Run a proxy to the Kubernetes API server get [(-o|--output=)json|yaml|...] <resource> [<id>]       Display one or many resources describe <resource> <id>                                  Show details of a specific resource create -f filename                                        Create a resource by filename or stdin createall [-d directory] [-f filename]                    Create all resources specified in a directory, filename or stdin update -f filename                                        Update a resource by filename or stdin delete ([-f filename] | (<resource> [(<id> | -l <label>)] Delete a resource by filename, stdin or resource and id namespace [<namespace>]                                   Set and view the current Kubernetes namespace log [-f] <pod> [<container>]                              Print the logs for a container in a pod. help [command]                                            Help about any command Available Flags: --api-version="v1beta1": The API version to use when talking to the server -a, --auth-path="/Users/bburns/.kubernetes_auth": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority=: Path to a cert. file for the certificate authority. --client-certificate=: Path to a client key file for TLS. --client-key=: Path to a client key file for TLS. -h, --help=false: help for kubectl --insecure-skip-tls-verify=%!s(bool=false): If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/Users/bburns/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --token=: Bearer token for authentication to the API server. --validate=false: If true, use a schema to validate the input before sending it ``` </del> <ins> See [kubectl documentation](kubectl.md) for details. </ins> ## kubecfg is deprecated.  Please use kubectl! ## kubecfg command line interface

Manually updated docs have a tendency to drift from actual implementation. It shouldn't be too hard to add a script to generate the markdown from a . We could then run the script manually or as a commit hook when changes are made to pkg/kubectl.
A big chunk of that is cut/paste from the help output anyway.. Brendan On Jan 9, 2015 4:20 PM, "Jeff Lowdermilk" wrote:
Why are we including the help output? Seems like cobra should do that, or we should just tell the user to ask for help if they want it.
It's useful to see generated markdown of help output for the full command tree, e.g. if you want to link to the syntax on irc or something. Anyway, this was quite easy to do, I'll have a PR shortly.


Question: # Conversions # local input_pkgs=() <del> while read -r file; do dir="$(dirname "${file}")" </del> <ins> while read -r dir; do </ins> pkg="$(cd "${dir}" && GO111MODULE=on go list -find .)" input_pkgs+=("${pkg}") done < <( <del> ( kube::codegen::internal::git_grep -l  </del> <ins> ( kube::codegen::internal::git_grep -l --null  </ins> -e '+k8s:conversion-gen='  ":(glob)${root}"/'**/*.go'  || true  <del> ) | LC_ALL=C sort -u </del> <ins> ) | xargs -0 -n1 dirname  | LC_ALL=C sort -u </ins> ) if [ "${#input_pkgs[@]}" != 0 ]; then

function in script has following issues: function only supports file but we have multiple types file with name as in . Hence this function is not working in our case. For the mitigation there is already an open . I have asked for including our changes as well. We have multiple files under a for example and . Because of this for appears twice and it generates duplicate informers We only need unique elements before passing it to . I followed this approach for unique elements: We have packages under . Earlier we were generating client only for and . But with the new I am not able to do that. I am using the following call to function: It results in following flags to As there is no directory as , it doesnot generate anything. Is it still possible to call on specific folder as it was done with earlier scripts? One mitigation I can think of is to call function on whole and then remove the marker from the types for which genclient is not needed. But the issue is we won't be able to pass the flag for each pkg. function in script is not working as expected.
/sig api-machinery
/cc
/triage accepted
Hi . I knew that there were all sorts of clients out there doing all sorts of things with these scripts, so I am not surprised to find you. :) Do you have a PR which attempts to use kube_codegen, so I can repro your issues directly?
I think you have one too many in there Aside from that, the real problem here is that these old codegen tools make some deeeeeeep assumptions about the ability to string-join various arguments and get the right results. It's awful. I was trying to make it simpler in this script, which I did for all of the in-tree consumers, but out of tree people apparently don't always follow the same conventions. For example, we generate clients for every directory that requests it. For a reason I do not know, you have files like which say but you don't actually want a client? If you specify then it t
So this code expects that you generate an omnibus client. What if I add a flag, like which you can use to call it for each API to make distinct clients? In truth the codegen tools need to be fixed, but that has SO MUCH ripple I just can't right now.


Question: # Conversions # local input_pkgs=() <del> while read -r file; do dir="$(dirname "${file}")" </del> <ins> while read -r dir; do </ins> pkg="$(cd "${dir}" && GO111MODULE=on go list -find .)" input_pkgs+=("${pkg}") done < <( <del> ( kube::codegen::internal::git_grep -l  </del> <ins> ( kube::codegen::internal::git_grep -l --null  </ins> -e '+k8s:conversion-gen='  ":(glob)${root}"/'**/*.go'  || true  <del> ) | LC_ALL=C sort -u </del> <ins> ) | xargs -0 -n1 dirname  | LC_ALL=C sort -u </ins> ) if [ "${#input_pkgs[@]}" != 0 ]; then

As the codegen was breaking us at several step, hence didn't prepared the PR.
Thanks for the PR. I have one more concern, is it possible to call and client-gen on the same pkg? With the current script, that is not possible because of the line It removes all the generated files first and then calls the . Hence the later clientset gen will remove the earlier generated clients.
Why are you generating a client for an internal version that isn't served?
My mistake. We need to stop using clients for internal versions.


Question: github.com/urfave/negroni v1.0.0 // indirect github.com/vishvananda/netlink v0.0.0-20171020171820-b2de5d10e38e github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 // indirect <del> github.com/vmware/govmomi v0.20.0 </del> <ins> github.com/vmware/govmomi v0.20.1 </ins> github.com/vmware/photon-controller-go-sdk v0.0.0-20170310013346-4a435daef6cc github.com/xanzy/go-cloudstack v0.0.0-20160728180336-1e2cbf647e57 github.com/xlab/handysort v0.0.0-20150421192137-fb3537ed64a1 // indirect

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The vSphere cloud provider is unable to fetch the node failure domain labels when using VCP with SAML auth because of an authentication failure. For context, the SAML token auth is described here - SAML auth support for the zones api was introduced in this PR - , which was cherry-picked to 1.14.1 here - I've tested the credentials I'm using with govc to ensure that I'm able to get a token: The credentials also work fine when using VCP without the node labels feature. Relevant kubelet logs: What you expected to happen: VCP should add failure domain labels to the node object. How to reproduce it (as minimally and precisely as possible): Setup SAML auth for VCP with zone labeling enabled. Anything else we need to know?: There was a previous issue that was raised () but that was before SAML auth support for zones was implemented. This could potentially be an issue with govc handling of SAML tokens since even with the Administrator role assigned to the user, I'm still unable to list tags. The role and keys were generated with the following: Environment: Kubernetes version (use ): 14.1 Cloud provider or hardware configuration: vSphere OS (e.g: ): Linux & Windows
/sig vmware
Would you mind taking a look at this? I can provide more info if needed.
govc's command only works with the SOAP endpoint currently. The REST endpoint has its own session manager and requires a separate login. At the moment, govc's tags commands login/logout when each command is run and don't use a persisted session (which are saved to ~/.govmomi/sessions). However, you can use a token returned by to test SAML auth with the tags commands. For example: Can you give that try? There are some TODOs around this btw:
Those commands work for me. So that would imply the solution user, role, and permissions are set up correctly? Would you expect the to also work?


Question: github.com/urfave/negroni v1.0.0 // indirect github.com/vishvananda/netlink v0.0.0-20171020171820-b2de5d10e38e github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 // indirect <del> github.com/vmware/govmomi v0.20.0 </del> <ins> github.com/vmware/govmomi v0.20.1 </ins> github.com/vmware/photon-controller-go-sdk v0.0.0-20170310013346-4a435daef6cc github.com/xanzy/go-cloudstack v0.0.0-20160728180336-1e2cbf647e57 github.com/xlab/handysort v0.0.0-20150421192137-fb3537ed64a1 // indirect

I seem to be getting a 404 when using with the flag - Does seem to work when specifying a tag name: Are there any other govc commands I should check / verify?
yes, the solution user setup looks right. Not sure what the issue is with . Can you share your vCenter build number ()? I'll try to reproduce the issue.
$ govc about -u vcdo Name: VMware vCenter Server Vendor: VMware, Inc. Version: 6.7.0 Build: OS type: linux-x64 API type: VirtualCenter API version: 6.7.2 Product ID: vpx UUID: -55ed-4737-9cc4-
I can reproduce this, but don't know what the issue is yet. The version of Go and vSphere have changed since , but if it was one of those was the cause, I'd expect SAML auth to fail with the govc tags commands too. All I have to go on for now is the vCenter log message below. I'll update when I have more info.
I have it working with the change below, but need to figure out a few things first: Why this is the case when run inside kubelet and not govc (note that the port change to 443 breaks govc) A good way to determine which port to use (depends on the answer to the above) If this is behavior is specific to 6.7U2 or something triggered by a change in K8s I suppose a quick fix would change to try port 443 if 80 fails, but would prefer to avoid that as the failure generates stack traces in the
/assign
This is obvious in retrospect, but the issue is the message signature needs to use the same port as the original request URL. However, in the govc case there was no port set in the request URL. The client defaults to port when connecting (based on the scheme), but does not include any port with the header. In the case, the default is on the server side. This can be reproduced with if includes the port , rather than just as we were using. The VCP tests that use token auth for tags lookup just don't validate the signature, so this wasn't a change related to 6.7U2.


Question: github.com/urfave/negroni v1.0.0 // indirect github.com/vishvananda/netlink v0.0.0-20171020171820-b2de5d10e38e github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 // indirect <del> github.com/vmware/govmomi v0.20.0 </del> <ins> github.com/vmware/govmomi v0.20.1 </ins> github.com/vmware/photon-controller-go-sdk v0.0.0-20170310013346-4a435daef6cc github.com/xanzy/go-cloudstack v0.0.0-20160728180336-1e2cbf647e57 github.com/xlab/handysort v0.0.0-20150421192137-fb3537ed64a1 // indirect

I'll open a K8s PR to pull in the fix:
Updating k/vendor to govmomi master adds almost 6K lines for a 3 line fix. I may just create a govmomi branch off of v0.20 to reduce churn in the K8s release branches for this fix, may have to wait until tomorrow.
Really appreciate the insight into this issue. Can confirm that specifying the port (443) in the GOVC_URL reproduces the issue. Thank you!
Unfortunately, I'm seeing the same issue with v1.14.3: cloud_config Works fine when I use the username and password of a vsphere admin, but using cert and key still has authentication issues.
any related errors in vCenter's ?
Seeing a lot of this exception:
whoa! that stack trace is like punch in the gut (brings back memories!) :)
indeed!! apologies, I must have had the change for PR locally, but failed to include it as I had focused on the signature issue and updating vendor/