Question: Then the API server will program in the new RESTful resource path: * `/apis/stable.example.com/v1/namespaces/<namespace>/crontabs/...` <ins> **Note: This may take a while before RESTful resource path registration happen, please always check this before you create resource instances.** </ins> Now that this schema has been created, a user can `POST`:

Using this script: I'm seeing some random errors. 1 - running the above script it fails for me on the near the bottom when it tries to create . Notice that I had already successfully created which of the same type: Sometimes if I wait a while (like over a minute) then the two commands at the bottom work. 2 - At the top, if I comment out the commands and uncomment the commands, it fails with: Which is interesting, because when I give it a totally unknown type I get this: which leads me to think that it does know about the Org1Type but fails late in the processing for some reason. I said these errors we random, because there have been times when I would run this script and then go back later and try some of the failing commands manually and they would work - almost like there's a timing/async issue going on - but since I couldn't reliably reproduce those all I can do is just mention it but can't provide a solid testcase to show it, yet. Additionally, I've seen cases where the (for one of the instances) would fail and then I would try again and it would complain that the resource I'm trying to create already exists. I'm wondering if this is due to the original create actually working but then failing on some kind of verification step afterwards and that's the error being shown - but that's just a guess. I'm running these tests with PR but I saw seeing similar results even from master. ping
Any chance addresses this?
I'm pretty sure I tried that PR w/o any luck but let me try again just to be
nope, still fails for me using . On a related question... is it normal for thirdPartyResource types to not be visible right away? I find that I have to add a in there to wait for these new types to be known. I understand that some things in kube need/should-be async, but I'm not sure why the creation of these would be.
Hm, I'd have to look at the code, I thought it should be synchronous, but maybe not? might know.
It is not synchronous. It is picked up by the thirdparty controller. It can take a little while.
is something that can be changed or does it need to be this way?
That would take a lot more effort I'd say.


Question: Then the API server will program in the new RESTful resource path: * `/apis/stable.example.com/v1/namespaces/<namespace>/crontabs/...` <ins> **Note: This may take a while before RESTful resource path registration happen, please always check this before you create resource instances.** </ins> Now that this schema has been created, a user can `POST`:

Cannot comment about the feasibility, as I have not looked into that option yet.
Also note that if you're in a HA system, you may need to wait until it shows up on all apiserver replicas before using it... , Uday Ruddarraju wrote:
Is there a way we can tell if that has happened? It seems like we need some better way to check than "hit it until it works, and even then, maybe it does not".
An option is that we can wait until that really happen.
Yes I would prefer for certain operations to wait until they're really done before they return but as asked, how can we know for sure in this case?
I see some options: wait to return until it's ready to be called. return some value on where to check for it to be ready. resource is stubbed out immediately, and returns some 'not yet ready' status until it is. knows enough to 'keep trying' until some timeout. I'm not sure what the intention is going forward if in fact core k8s APIs are going to be switched to using this mechanism. It can't be left in such an inconsistent state without specification. This behavior is not in the design.
IMO, kubectl can try to check some condition until some timeout, but we still need to make sure what's the condition, I would hava to take a look at the thirdparty resource code and see what we can do.
I think doing it on the kubectl side would be a mistake because then the REST API clients would need to replicate that logic. If there are certain operations (like, IMO, creating a new 3rd party resource) that should be blocked then I'd prefer if the POST itself blocked until it was done.
The logical here is that when you create a new ThirdPartyResource resource, it will return just after the resource is stored in etcd, but the process to register a new RESTful resource path and add a new api group is asynchronous, this is what thirdparty_controller will do. Basically put this wait in kubectl is wrong, this makes kubectl create complicate. As for real use case, users can use discovery client to verify that the api group is registered.
Third party resource installation is not meant to be flipped on and off often. The only use case for that is testing your third party bring up.


Question: Then the API server will program in the new RESTful resource path: * `/apis/stable.example.com/v1/namespaces/<namespace>/crontabs/...` <ins> **Note: This may take a while before RESTful resource path registration happen, please always check this before you create resource instances.** </ins> Now that this schema has been created, a user can `POST`:

And there it's sufficient to just keep listing the third party data path until it returns a success value. For extra certainty, look up the endpoints of the kubernetes service and make sure it appears on every endpoint. The add is an async operation. It returns success because the 3rd party resource spec has been saved. I definitely think we should document the need for this wait better but I don't think we should actually make the control plane do the wait for you.
That's just what I mean, we should document this
fwiw, the async nature is intentional. the system is purposely de-coupled, there are lots of other examples of this in the system (e.g. creating a replication controller doesn't create pods immediately) It's much more robust this way (of course it does put a little more burden on the client side) I could see adding a utility function somewhere to wait, but honestly it's basically
yep, I get that and for lots of things the async nature makes sense. But I'm not sure it makes sense for all. For example, I think people would understand that it takes a while to spin up containers, but I'm not so sure the delay for creating a new resource type (which is mainly updating etcd) is so easy to understand. I think, at an abstract level, a lot of this goes to the UX. Sure, every client could do that 404-loop you wrote, but is that really necessary in all cases and is that the best UX we can offer? From a UX perspective I think there are going to be some operations that should appear to be synchronous even if under the covers its async and the server does the 404-loop for you (or at least by default and have a flag to disable it). And let's not forget there's nothing stopping a client from doing the async work on their side by running their cmd in the background, on a new thread/process, and then checking for that thread to complete - if they really didn't want to wait around. So, net, I think we need to look at this from a user's perspective and not a Kube-developer's, who understands what's going on under the covers. From a user's perspective, for operation "X" would I normally expect that to take some time? If so, then an async UX is ok.


Question: Then the API server will program in the new RESTful resource path: * `/apis/stable.example.com/v1/namespaces/<namespace>/crontabs/...` <ins> **Note: This may take a while before RESTful resource path registration happen, please always check this before you create resource instances.** </ins> Now that this schema has been created, a user can `POST`:

But if I would expect "X" to be immediate (meaning once I get back the 200 I should be able to take the next action on that resource because I should have gotten back a non-200 immediately if there was an error) then exposing it as an async op might not be the best UX. IMO, creating a TPR falls into that category.
The fact that there can be multiple apiservers means it really is an async operation: you need to wait for each apiserver to get the watch notification and install the handlers. Operations with that sort of dependency are never synchronous in our api. Is it problematic to just do the wait loop, or just wait 10 seconds or something? How often are you adding new 3rd party resources? , Doug Davis wrote:
(I'm not arguing that we shouldn't make things easy for users, rather that we are working on other things that are even worse for users and this is less important since it should be used infrequently.) , Daniel Smith wrote:
Just as background on my current experimentation with TPRs: I'm using a single hyperkube env, which I think has just a single API server. In this work, which is actually part of an education class, we're walking thru a script that registers some TPRs and then creates instances of them to show students about the concept and how to use them. I'm finding at after each I have to add some kind of or , that's a pretty poor UX. I can certainly understand in a multi-api-server env there may be more complex issues involved, but if we can at least improve it for the simple case I think that would help a lot. I would also note that even a !404-loop isn't sufficient in the HA case since you may be getting luck w.r.t. which apiServer you're hitting, right? And yes, I know there are a ton of bigger fish to fry and I do appreciate the discussion. :-)
It's unfortunately not really easier to do it for the single-master case, since we have to support the multi-master case. If we add an exception for the single master case that adds complexity and actually makes the problem a bit worse for the multi-master case.


Question: Then the API server will program in the new RESTful resource path: * `/apis/stable.example.com/v1/namespaces/<namespace>/crontabs/...` <ins> **Note: This may take a while before RESTful resource path registration happen, please always check this before you create resource instances.** </ins> Now that this schema has been created, a user can `POST`:

The "Kubernetes-y" thing to do here would be to add a status component to the 3rd party resource object and have each apiserver report when they've applied the change, then you'd wait for that instead of your current 404-loop. Having the POST method itself do the waiting is going to be a really tough sell. Watch is intended to handle all corner cases involved when clients are waiting on information from the server, POST is not equipped to do that. (e.g., what happens if the apiserver you POSTed to crashes while waiting? You wouldn't know if you should re-post or not and you wouldn't have any way to figure out the status at that point.) , Doug Davis wrote:
I wouldn't argue for a single-master exception, nor would I argue for a POST that waits. I am unfamiliar with how everything works, so I am having a little trouble parsing some of these responses. I see mentioned a and . What are those? Are they applicable here? Is there a way to watch for this change to be made that is built in, or is the way to go for now?
Why was this closed? is a doc issue, I don't think this is a doc problem.
I am not so concerned about the UX issue if the delay really is isolated to system start-up (as if there is such a thing). What about this scenario: operator registers a TPR, then waits for all the apiservers running at the time to fully react, and thereafter nothing is prepared for the TPR to be absent. Some time later another apiserver is , or an existing one crashes and is re-started. Will there be an interim during which the new(ly started) apiserver is not ready for clients to use the TPR?
would know better, but I'd expect the readiness check to fail until the initial sync of registered apiservice objects completed, preventing the new API server from making it into the LB pool
will report unhealthy until the apiservices are synced, but TPR was based on an unusual polling technique that can take about 10 seconds. I'm not certain when it will be ready. CRD is based on the state of an informer's list/watch, so its much more current.
This consistently works well for CRDs. modified script to use CRDs:
Closing since this works for CRDs. /close


Question: { Object: map[string]interface{}{ "apiVersion": "example.com/v2", <ins> "metadata":   map[string]interface{}{}, </ins> "kind":       "foo", "other":      "data", },

If a CR converter is wired in and it modifies the input list, the delegating CR converter does not protect against this. A converter cannot effect the input list Not really possible externally - it's a coding issue that the current converters don't hit. No response <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery /assign
/triage accepted


Question: # gazelle:exclude _gopath # gazelle:exclude _output # gazelle:exclude _tmp <ins> # gazelle:prefix k8s.io/kubernetes </ins> package(default_visibility = ["//visibility:public"])

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: I expected the update to run successfully. How to reproduce it (as minimally and precisely as possible): Checkout. . Anything else we need to know?: I get this in both master and release-1.9. I tested most carefully in release-1.9 (because I am working on a cherry-pick right now). In particular, I blew away the container and image I had used earlier for building. Before running I established this state: Then I ran --- which took a while but eventually completed. It left me this state: After that is when I ran the that is exhibited at the start of this issue. Environment: Kubernetes version (use ): Commit (current release-1.9) Cloud provider or hardware configuration: local VM OS (e.g. from /etc/os-release): Ubuntu 17.10 Kernel (e.g. ): Linux ubu17dev 4.13.0-21-generic -Ubuntu SMP Mon Dec 18 17:29:16 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others: Docker 17.12.0-ce
Ah, there's another bug for this, this is being fixed. (same issue, verify uses update internally)
Any workaround?
/reopen It looks like the 1.8 and 1.9 release branches are broken for the same reason, but the fix is a bit less trivial, since we can't just use the latest gazelle, since this would require a huge mass of changes to the BUILD files. (Luckily, 1.7 doesn't use gazelle, so it's unaffected.) Some additional background: gazelle doesn't vendor bazelbuild/buildtools, though it does have a bazel dependency on a particular version. As a result, is always grabbing the latest version of bazelbuild/buildtools, even with our hacks to check out an older version of gazelle. An additional complication for the 1.8 and 1.9 branches is that we're installing gazelle from bazelbuild/rulesgo instead of bazelbuild/bazel-gazelle.


Question: # gazelle:exclude _gopath # gazelle:exclude _output # gazelle:exclude _tmp <ins> # gazelle:prefix k8s.io/kubernetes </ins> package(default_visibility = ["//visibility:public"])

Various options under consideration to fix these release branches: vendor gazelle, similar to (but not a straight cherrypick, since versions are different) the bazelbuild/rulesgo repo somewhere (where?) and then explicitly vendor and older version of bazelbuild/buildtools, and then use that fork instead of the upstream fork the hack in to also check out a particular bazelbuild/buildtools revision (so hacky) Anyone have a better idea?
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Fully vendor gazelle, similar to (but not a straight cherrypick, since versions are different) Clone the bazelbuild/rulesgo repo somewhere (where?) and then explicitly vendor and older version of bazelbuild/buildtools, and then use that fork instead of the upstream fork Extend the hack in to also check out a particular bazelbuild/buildtools revision (so hacky) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /assign /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think we should start with 2) to get things fixed and then solve 1) for the current branch and come back to the release branches if we think we still need that.
can you describe 1) and 2) here?
I ended up implementing option 3 in ; it wasn't quite as gross as I thought it'd be. LMK what you think.
1) and 2) from the bullets in
ahh, thanks.
many thanks... would the same fix work in release-1.7 and release-1.8? trying to get those in shape as well
PRs are out for 1.8 and 1.9. The same thing would work for 1.7 AFAIK Edit: see below, I am wrong WRT 1.7
1.7 is unaffected because it doesn't use gazelle.


Question: # gazelle:exclude _gopath # gazelle:exclude _output # gazelle:exclude _tmp <ins> # gazelle:prefix k8s.io/kubernetes </ins> package(default_visibility = ["//visibility:public"])

I just opened with the same hacky fix for 1.8. apologies for all these breakages. my technical debt caught up to me. :(
/close this should be fixed everywhere, for now.


Question: # gazelle:exclude _gopath # gazelle:exclude _output # gazelle:exclude _tmp <ins> # gazelle:prefix k8s.io/kubernetes </ins> package(default_visibility = ["//visibility:public"])

http://k8s- looks like something is indeed busted :- /priority failing-test /kind bug /status approved-for-milestone /sig testing cc /assign
/assign looks like Mike is working on the fix


Question: )), # Since the multi_arch_container macro replaces the {ARCH} format string, # we need to escape the stamping vars. <del> # Also see comment above about why the push tags use ARCH while the # non-push tags do not. </del> docker_push_tags = ["{{STABLE_DOCKER_PUSH_REGISTRY}}/%s-{ARCH}:{{STABLE_DOCKER_TAG}}" % binary], <del> docker_tags = ["{{STABLE_DOCKER_REGISTRY}}/%s:{{STABLE_DOCKER_TAG}}" % binary], </del> <ins> docker_tags = ["{{STABLE_DOCKER_REGISTRY}}/%s-{ARCH}:{{STABLE_DOCKER_TAG}}" % binary], </ins> stamp = True, symlinks = { # Some cluster startup scripts expect to find the binaries in /usr/local/bin,

What happened: When we build with make to build k8s, it will use 'amd64' in the container names. This is not the case for bazel builds, which are still using something like 'kube-apiserver' instead of 'kube-apiserver-amd64' What you expected to happen: Images build with make or bazel should have 'amd64' in the container name How to reproduce it (as minimally and precisely as possible): make quick-release cat inside tar "RepoTags": ["" make bazel-release cat inside tar "RepoTags": [""
/sig cluster-lifecycle /assign
+1 we need to standardize both builds to generate the same. +1 to add to the bazel build
FYI
+1, cc
bazel / docker build parity SGTM.
+1. Perhaps we should document a standard, like: All docker images that are arch specific should have the arch in the name Arch is only able to be omitted when it's a manifest-based tag (multi-arch) Document legacy exceptions, and hopefully bring them into compliance /cc
i like it. where is the best spot to doc this?
-- please doc as updates to:
Created


Question: SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ <del> "/apis/apiservicegroup/v1": {}, </del> <ins> "/apis/apiservicegroup/v1/path1": {}, </ins> }, }, },

When installing the metrics-server to implement the API and the prometheus-adapter to implement the API, it is possible to make the OpenAPI handler fail to initialize with this error log in apiserver: This will result in kubectl discovery calls to /openapi/v2 to fail with a 503: The OpenAPI handler should not fail to initialize if both and are serving the same path for different API groups/versions. I can provide exact installation steps if needed, however, a high-level repro is: install metrics-server to serve install prometheus-adapter to serve ensure resourceRules is populated in the prometheus-adapter config so that resource metrics are also served by restarting apiserver will result in the OpenAPI failure and 503 responses to /openapi/v2 path No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
Please let me know if I should reopen the issue in
/sig api-machinery
/sig instrumentation [metrics apis]
Would you mind taking a look when have time? Thank you^^ /triage accepted
I have a question. should be serving under . Why does an apiservice serving another gvk register paths under ? cc we've made the assumption that an APIService (such as metrics-server or prometheus-adapter) serves only gvks under its own registered group version. I'll do some more investigation but just wondering if prometheus doesn't follow that assumption?
the prometheus-adapter can also be configured to serve and replace the metrics-server by enabling these metrics, However, if the prometheus-adapter fronts and metrics-server fronts , it does seem like the correct behavior would be for prometheus-adapter to serve until an operator updates the APIService to reference prometheus-adapter rather than metrics-server. We're seeing this issue occur for upgrades between 1.27 and 1.28 so it does look this path duplication did not cause the OpenAPI handler to fail on 1.27.
I just upgraded from 1.27 to 1.28 and running into this problem. Some kubectl apply operations fail, and ArgoCD cannot load its custom resources. Any suggested workaround? Would removing the from the prometheus configuration resolve it? Or perhaps disable metrics-server?
Simply looking at the code diff between 1.27 and 1.28 I can't pinpoint the cause.


Question: SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ <del> "/apis/apiservicegroup/v1": {}, </del> <ins> "/apis/apiservicegroup/v1/path1": {}, </ins> }, }, },

Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question.


Question: SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ <del> "/apis/apiservicegroup/v1": {}, </del> <ins> "/apis/apiservicegroup/v1/path1": {}, </ins> }, }, },

In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.


Question: SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ <del> "/apis/apiservicegroup/v1": {}, </del> <ins> "/apis/apiservicegroup/v1/path1": {}, </ins> }, }, },

I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in


Question: <ins> vagrant: user.present: - optional_groups: - docker - remove_groups: False - require: - pkg: docker-io </ins>

As a sysadmin When using the Vagrant cluster provided And logged in as the vagrant user to the master or minions I would like to be able to call docker commands wihout requiring the 'sudo' prefix. For Fedora =19 and RHEL=7 this can be enabled by adding the vagrant user to the docker group in /etc/groups
This should be trivial, but its tricker to do because of the whole salt configuration process as the docker group is only created on each kubernetes-minion once docker is installed by the salt-minion. To do this properly, we would need a salt state change for this, and new flag in to let you specify additional users that should be in the docker group.


Question: go test -c  ${goflags:+"${goflags[@]}"}  -gcflags="${gogcflags}"  <del> -asmflags="${goasmflags}"  </del> -ldflags="${goldflags}"  -tags="${gotags:-}"  -o "${outfile}"

While working on for , I found that our kubernetes1.24 package stopped to produce bit-identical results on 2 fresh builds. The first chunk is likely just the hash of the remainder, so can be ignored. The other chunks indicate that a random build temporary path is used for Sources should be able to reproduce bit-identical binaries e.g. for SLSA level4 On Debian or openSUSE do No response <detailsWARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-05-05T00:00:00Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 console $ cat /etc/os-release $ uname -a Linux bmwiedemann-factory 5.18.6-1-default SMP PREEMPTDYNAMIC Thu Jun 23 05:46:18 UTC 2022 () x8664 x8664 x86_64 GNU/Linux </details<detailsgo1.18 </details<details</details<details</details
/sig release /area build-release
can you please post a link to the full logs from the build?
any news on this? Haven't heard from (context: we worked to fix this a couple of times in )
Unfortunately not, is this something we can fix on the OBS side? If the go build timestamp () is the main issue here then we can approach a fix in our build pipeline.
It would be good if would not be embedded in the first place. It might be possible to replace it somehow in our builds, but all other people wanting reproducible builds, would need to replicate that.
full logs from the build: from kubernetes-1.24.3 built 2022-08-18
can you please try setting GOCACHE=off when you run the build? I am suspecting that the temp directory is being generated here: (or add an export in for that env var)
I to but still got these diffs.


Question: go test -c  ${goflags:+"${goflags[@]}"}  -gcflags="${gogcflags}"  <del> -asmflags="${goasmflags}"  </del> -ldflags="${goldflags}"  -tags="${gotags:-}"  -o "${outfile}"

So i built golang from source, peppered it with changes to track down where this is getting created and found that it is coming from: I don't know yet how to fix it. will dig more when i can (travelling for the next few days)
Would it work to override the , I see that they use the : Ref: https://reproducible- Edit: Probably not, we may make it configurable in golang :thinking:
this is so tricky :( will dig more when i get back from my event this week
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned".


Question: go test -c  ${goflags:+"${goflags[@]}"}  -gcflags="${gogcflags}"  <del> -asmflags="${goasmflags}"  </del> -ldflags="${goldflags}"  -tags="${gotags:-}"  -o "${outfile}"

<detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is still relevant even with 1.27. It might be worth revisiting this issue again. /reopen /triage accepted
Reopened this issue. <detailsIn response to : This is still relevant even with 1.27. It might be worth revisiting this issue again. /reopen /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
latest run -
I found that our package has the same issue, so it probably it is a toolchain issue in golang - and much faster to debug outside of the slow kubernetes build. =filed
AIUI, the conclusion by the golang people seems to be that commit is not the right approach and one should use with the go call instead. Not sure how that interacts with other parts of your build.
cc (please see referenced commit above!)
I see. I inherited the gcflags before so I plead innocence. It seems plausible to change it from to - I don't know when I will have time to look at that, though.


Question: go test -c  ${goflags:+"${goflags[@]}"}  -gcflags="${gogcflags}"  <del> -asmflags="${goasmflags}"  </del> -ldflags="${goldflags}"  -tags="${gotags:-}"  -o "${outfile}"

My patch doesn't seem to fix the whole problem: thockin-glaptop4 kubernetes buildtrimpath /$ make kubelet +++ [1114 19:10:12] Building go targets for linux/amd64 (non-static) thockin-glaptop4 kubernetes buildtrimpath /$ md5sum output/bin/kubelet output/bin/kubelet thockin-glaptop4 kubernetes buildtrimpath /$ make kubelet +++ [1114 19:12:40] Building go targets for linux/amd64 (non-static) thockin-glaptop4 kubernetes buildtrimpath /$ md5sum output/bin/kubelet output/bin/kubelet thockin-glaptop4 kubernetes build_trimpath /$
diff --- a 2023-11-14 19:13:45. -0800 +++ b 2023-11-14 19:13:56. -0800 -1,4 +1,4 -38rZHdQckYF9EEzmndR/BAXOwzCcq0Mk1UHT62ha/QLiRgXP4oojWh3oWP9/TWG7KW90IL8vKt1UOEPR +LeLArvl7sgWAd55ROKN2/BAXOwzCcq0Mk1UHT62ha/QLiRgXP4oojWh3oWP9/cYb5ZGQwa6js077YiAvK /lib64/ld-linux-x86- AUATUSH []AA] -,8 +,8 go1.21.3 runtime/cgo: call frame too large -2023-11-15T03:12:40Z -2023-11-15T03:12:40Z +2023-11-15T03:13:47Z +2023-11-15T03:13:47Z pthreadcreate failed: %s bad stack bounds: lo=%p hi=%p expand 32-byte kexpand 32-byte k
That seems unrelated. I tested with and it made our openSUSE package fully reproducible. Your diff contains a timestamp - maybe would get rid of that. The top part of the diff is just the go build-id hash of the remainder and can be ignored.


Question: go test -c  ${goflags:+"${goflags[@]}"}  -gcflags="${gogcflags}"  <del> -asmflags="${goasmflags}"  </del> -ldflags="${goldflags}"  -tags="${gotags:-}"  -o "${outfile}"

Thanks for confirming
Will there be backports to 1.24++ ? I found it applies cleanly to 1.26++ and works with some small tweaks in 1.24 and 1.25. If not, I'd submit the patch for our openSUSE packages.
1.24 and 1.25 reached EOL and we don't support these branches any longer. We can cherry-pick this to 1.26+ if folks ( agree
Cherry-picks: 1.29 1.28 1.27 1.26


Question: w.Write(LEVEL_0, "Concurrency Policy:t%sn", cronJob.Spec.ConcurrencyPolicy) w.Write(LEVEL_0, "Suspend:t%sn", printBoolPtr(cronJob.Spec.Suspend)) if cronJob.Spec.SuccessfulJobsHistoryLimit != nil { <del> w.Write(LEVEL_0, "Successful Job History Limit:t%dn", cronJob.Spec.SuccessfulJobsHistoryLimit) </del> <ins> w.Write(LEVEL_0, "Successful Job History Limit:t%dn", *cronJob.Spec.SuccessfulJobsHistoryLimit) </ins> } else { w.Write(LEVEL_0, "Successful Job History Limit:t<unset>n") }

What happened: The history limit in the describe display has the incorrect number What you expected to happen: The history limit should show the default limit (3) How to reproduce it (as minimally and precisely as possible): Add a cronjob with default history limit settings and run Anything else we need to know?: Environment: Kubernetes version (use ): $ kubectl version Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Brightbox OS (e.g: ): Ubuntu 18.04.2 Kernel (e.g. ): Linux bionic-vm 4.15.0-48-generic -Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeam Network plugin and version (if this is a network-related bug): Others:
/sig cli
/assign


Question: t.Errorf("Expected binder %q but got %q", expectedBinderType, binderType) } } <ins> func TestNodeAllocatableChanged(t *testing.T) { newQuantity := func(value int64) resource.Quantity { return *resource.NewQuantity(value, resource.BinarySI) } for _, c := range []struct { Name           string Changed        bool OldAllocatable v1.ResourceList NewAllocatable v1.ResourceList }{ { Name:           "no allocatable resources changed", Changed:        false, OldAllocatable: v1.ResourceList{v1.ResourceMemory: newQuantity(1024)}, NewAllocatable: v1.ResourceList{v1.ResourceMemory: newQuantity(1024)}, }, { Name:           "new node has more allocatable resources", Changed:        true, OldAllocatable: v1.ResourceList{v1.ResourceMemory: newQuantity(1024)}, NewAllocatable: v1.ResourceList{v1.ResourceMemory: newQuantity(1024), v1.ResourceStorage: newQuantity(1024)}, }, } { oldNode := &v1.Node{Status: v1.NodeStatus{Allocatable: c.OldAllocatable}} newNode := &v1.Node{Status: v1.NodeStatus{Allocatable: c.NewAllocatable}} changed := nodeAllocatableChanged(newNode, oldNode) if changed != c.Changed { t.Errorf("nodeAllocatableChanged should be %t, got %t", c.Changed, changed) } } } func TestNodeLabelsChanged(t *testing.T) { for _, c := range []struct { Name      string Changed   bool OldLabels map[string]string NewLabels map[string]string }{ { Name:      "no labels changed", Changed:   false, OldLabels: map[string]string{"foo": "bar"}, NewLabels: map[string]string{"foo": "bar"}, }, // Labels changed. { Name:      "new node has more labels", Changed:   true, OldLabels: map[string]string{"foo": "bar"}, NewLabels: map[string]string{"foo": "bar", "test": "value"}, }, } { oldNode := &v1.Node{ObjectMeta: metav1.ObjectMeta{Labels: c.OldLabels}} newNode := &v1.Node{ObjectMeta: metav1.ObjectMeta{Labels: c.NewLabels}} changed := nodeLabelsChanged(newNode, oldNode) if changed != c.Changed { t.Errorf("Test case %q failed: should be %t, got %t", c.Name, c.Changed, changed) } } } func TestNodeTaintsChanged(t *testing.T) { for _, c := range []struct { Name      string Changed   bool OldTaints []v1.Taint NewTaints []v1.Taint }{ { Name:      "no taint changed", Changed:   false, OldTaints: []v1.Taint{{Key: "key", Value: "value"}}, NewTaints: []v1.Taint{{Key: "key", Value: "value"}}, }, { Name:      "taint value changed", Changed:   true, OldTaints: []v1.Taint{{Key: "key", Value: "value1"}}, NewTaints: []v1.Taint{{Key: "key", Value: "value2"}}, }, } { oldNode := &v1.Node{Spec: v1.NodeSpec{Taints: c.OldTaints}} newNode := &v1.Node{Spec: v1.NodeSpec{Taints: c.NewTaints}} changed := nodeTaintsChanged(newNode, oldNode) if changed != c.Changed { t.Errorf("Test case %q failed: should be %t, not %t", c.Name, c.Changed, changed) } } } func TestNodeConditionsChanged(t *testing.T) { nodeConditionType := reflect.TypeOf(v1.NodeCondition{}) if nodeConditionType.NumField() != 6 { t.Errorf("NodeCondition type has changed. The nodeConditionsChanged() function must be reevaluated.") } for _, c := range []struct { Name          string Changed       bool OldConditions []v1.NodeCondition NewConditions []v1.NodeCondition }{ { Name:          "no condition changed", Changed:       false, OldConditions: []v1.NodeCondition{{Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue}}, NewConditions: []v1.NodeCondition{{Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue}}, }, { Name:          "only LastHeartbeatTime changed", Changed:       false, OldConditions: []v1.NodeCondition{{Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue, LastHeartbeatTime: metav1.Unix(1, 0)}}, NewConditions: []v1.NodeCondition{{Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue, LastHeartbeatTime: metav1.Unix(2, 0)}}, }, { Name:          "new node has more healthy conditions", Changed:       true, OldConditions: []v1.NodeCondition{}, NewConditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}, }, { Name:          "new node has less unhealthy conditions", Changed:       true, OldConditions: []v1.NodeCondition{{Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue}}, NewConditions: []v1.NodeCondition{}, }, { Name:          "condition status changed", Changed:       true, OldConditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionFalse}}, NewConditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}, }, } { oldNode := &v1.Node{Status: v1.NodeStatus{Conditions: c.OldConditions}} newNode := &v1.Node{Status: v1.NodeStatus{Conditions: c.NewConditions}} changed := nodeConditionsChanged(newNode, oldNode) if changed != c.Changed { t.Errorf("Test case %q failed: should be %t, got %t", c.Name, c.Changed, changed) } } } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Today, the scheduler re-processes unschedulable nodes . Node updates are sent by Kubelet frequently (every 10 seconds) and often times they don't change any of the node properties that could potentially make them feasible. We should be smarter about these updates and move pods to the schedulable queue only when certain properties of nodes change that could make them feasible. These properties include node allocatable, node conditions, taints, labels, .... Why is this needed: Improves performance of the scheduler and avoids doing unnecessary attempts to schedule pods. This is particularly helpful when a cluster has a lot of unschedulable pods. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature This is not a feature really. It is performance optimization, but in the absence of such a label, I need to use "feature" here. /sig scheduling
I can help to fix it. /assign mlmhl
The main reason for this problem is / update their status too frequently and most of these updates are heartbeats without any meaningful change. Recently sig-node introduces a new to alleviate this problem. With this mechanism, kubelet will split heartbeat and node status updates into two independent operations, and reduce the frequency of status updates to once per 1 minute(or more). In the long run this mechanism can indeed alleviate our problems, but at present it does not help us much as this is still an alpha feature and the node status updates frequency is still a bit high(per 1 minute right now).
Maybe we can try comparing old and new nodes after stripping their heartbeat info, like this:
I think that had sent earlier is more efficient. However, that PR does not filter out . In Wei's code, the heartbeat time is removed correctly. I think we should fix this issue and resubmit that PR.
could you resubmit a PR to: disregard in conditions comparison: and (optionally) disregard timestamp in taints comparison:
Thanks for all your advice, a new PR is submitted: This PR does not include your second suggestion(disregard timestamp in taints comparison) as I'm not sure if can be changed when node updating.


Question: t.Errorf("Expected binder %q but got %q", expectedBinderType, binderType) } } <ins> func TestNodeAllocatableChanged(t *testing.T) { newQuantity := func(value int64) resource.Quantity { return *resource.NewQuantity(value, resource.BinarySI) } for _, c := range []struct { Name           string Changed        bool OldAllocatable v1.ResourceList NewAllocatable v1.ResourceList }{ { Name:           "no allocatable resources changed", Changed:        false, OldAllocatable: v1.ResourceList{v1.ResourceMemory: newQuantity(1024)}, NewAllocatable: v1.ResourceList{v1.ResourceMemory: newQuantity(1024)}, }, { Name:           "new node has more allocatable resources", Changed:        true, OldAllocatable: v1.ResourceList{v1.ResourceMemory: newQuantity(1024)}, NewAllocatable: v1.ResourceList{v1.ResourceMemory: newQuantity(1024), v1.ResourceStorage: newQuantity(1024)}, }, } { oldNode := &v1.Node{Status: v1.NodeStatus{Allocatable: c.OldAllocatable}} newNode := &v1.Node{Status: v1.NodeStatus{Allocatable: c.NewAllocatable}} changed := nodeAllocatableChanged(newNode, oldNode) if changed != c.Changed { t.Errorf("nodeAllocatableChanged should be %t, got %t", c.Changed, changed) } } } func TestNodeLabelsChanged(t *testing.T) { for _, c := range []struct { Name      string Changed   bool OldLabels map[string]string NewLabels map[string]string }{ { Name:      "no labels changed", Changed:   false, OldLabels: map[string]string{"foo": "bar"}, NewLabels: map[string]string{"foo": "bar"}, }, // Labels changed. { Name:      "new node has more labels", Changed:   true, OldLabels: map[string]string{"foo": "bar"}, NewLabels: map[string]string{"foo": "bar", "test": "value"}, }, } { oldNode := &v1.Node{ObjectMeta: metav1.ObjectMeta{Labels: c.OldLabels}} newNode := &v1.Node{ObjectMeta: metav1.ObjectMeta{Labels: c.NewLabels}} changed := nodeLabelsChanged(newNode, oldNode) if changed != c.Changed { t.Errorf("Test case %q failed: should be %t, got %t", c.Name, c.Changed, changed) } } } func TestNodeTaintsChanged(t *testing.T) { for _, c := range []struct { Name      string Changed   bool OldTaints []v1.Taint NewTaints []v1.Taint }{ { Name:      "no taint changed", Changed:   false, OldTaints: []v1.Taint{{Key: "key", Value: "value"}}, NewTaints: []v1.Taint{{Key: "key", Value: "value"}}, }, { Name:      "taint value changed", Changed:   true, OldTaints: []v1.Taint{{Key: "key", Value: "value1"}}, NewTaints: []v1.Taint{{Key: "key", Value: "value2"}}, }, } { oldNode := &v1.Node{Spec: v1.NodeSpec{Taints: c.OldTaints}} newNode := &v1.Node{Spec: v1.NodeSpec{Taints: c.NewTaints}} changed := nodeTaintsChanged(newNode, oldNode) if changed != c.Changed { t.Errorf("Test case %q failed: should be %t, not %t", c.Name, c.Changed, changed) } } } func TestNodeConditionsChanged(t *testing.T) { nodeConditionType := reflect.TypeOf(v1.NodeCondition{}) if nodeConditionType.NumField() != 6 { t.Errorf("NodeCondition type has changed. The nodeConditionsChanged() function must be reevaluated.") } for _, c := range []struct { Name          string Changed       bool OldConditions []v1.NodeCondition NewConditions []v1.NodeCondition }{ { Name:          "no condition changed", Changed:       false, OldConditions: []v1.NodeCondition{{Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue}}, NewConditions: []v1.NodeCondition{{Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue}}, }, { Name:          "only LastHeartbeatTime changed", Changed:       false, OldConditions: []v1.NodeCondition{{Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue, LastHeartbeatTime: metav1.Unix(1, 0)}}, NewConditions: []v1.NodeCondition{{Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue, LastHeartbeatTime: metav1.Unix(2, 0)}}, }, { Name:          "new node has more healthy conditions", Changed:       true, OldConditions: []v1.NodeCondition{}, NewConditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}, }, { Name:          "new node has less unhealthy conditions", Changed:       true, OldConditions: []v1.NodeCondition{{Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue}}, NewConditions: []v1.NodeCondition{}, }, { Name:          "condition status changed", Changed:       true, OldConditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionFalse}}, NewConditions: []v1.NodeCondition{{Type: v1.NodeReady, Status: v1.ConditionTrue}}, }, } { oldNode := &v1.Node{Status: v1.NodeStatus{Conditions: c.OldConditions}} newNode := &v1.Node{Status: v1.NodeStatus{Conditions: c.NewConditions}} changed := nodeConditionsChanged(newNode, oldNode) if changed != c.Changed { t.Errorf("Test case %q failed: should be %t, got %t", c.Name, c.Changed, changed) } } } </ins>

If it can be changed, I will add this filter later, otherwise I think we can keep the current implementation to make the compare operation simpler and faster. What's more, just as I said in the PR's description, I'm not sure how much side effects this change has on scheduler's resource consumption, as it always copy node's conditions before comparison.
I am thinking whether we could move this logic(whether node became more schedulable) to kubelet, kubelet reports it through heartbeats.


Question: return w, nil }) <del> // Create second pod which should preempt first pod. </del> <ins> // Create third pod which should preempt other pods. </ins> preemptorPod, err := createPausePod(testCtx.ClientSet, <del> initPausePod(&pausePodConfig{Name: "preemptor-pod", Namespace: testCtx.NS.Name, Priority: &highPriority, Resources: &resourceRequest})) </del> <ins> initPausePod(&pausePodConfig{Name: "preemptor-pod", Namespace: testCtx.NS.Name, Priority: &highPriority, Resources: &preemptorResourceRequest})) </ins> if err != nil { t.Errorf("Error while creating the preemptor pod: %v", err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via -- We may delete podS though they may be waiting in Do not delete waiting pods if they are not scheduled. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc WDYT ?
It sounds to me that you prefer to prioritize waiting Pod over other assigned Pods. In that case, you may want to exclude waiting Pods in the earlier Filtering/Sorting Victims step, instead of simply disregard the deletion in PrepareCandidate(). In general, when Permit plugins work along with PostFilter (preemption), the logic could be complicated. There may not be a general preemption implementation (probably in the future we can exercise pluginArgs to provide different options). So for now the best bet is to implement your own PostFilter plugin.
Got it! But for the default preemption plugin, we delete the victims to free resources. If pod has assigned, we should delete it. If pod has not assigned, we just need reject it in WaitingPod because the pod just exists in the cache.
Oh, your point was to verify if it's a waiting pod first, and then delete it? If so, then it makes sense to me. Feel free to raise a PR. /assign
Yes, I will open a PR to do this


Question: for _, item := range testCases { stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) <del> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{node: item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) controller.NodeUpdated(item.oldNode, item.newNode)

What happened: A Pod was scheduled for deletion because a taint w/ effect was observed in the Node the Pod was assigned to, and after the taint is removed from the Node, the Pod scheduled deletion isn't canceled. After scheduling pod deletion, the only cancels pod deletion if all taints are removed from the node, including any taints that the user relies on for their use-cases (example below). What you expected to happen: After the taint that triggered scheduling pod deletion is removed, cancels the scheduled pod deletion. How to reproduce it (as minimally and precisely as possible): A node is tainted w/ , so any workloads already assigned to it which don't tolerate this taint are evicted. A pod is created w/ tolerations: w/ unspecified and w/ . is assigned to . At some point in time after the previous steps took place, is tainted as . gets an update event for and observes two taints. It proceeds to calculate the minimum time (in seconds) that tolerates for the two taints which at this point is 300 seconds, and marks for deletion in ~300 seconds - this happens in-memory, in a timed-queue. At some point in time after the previous steps took place, and before the 300 seconds trigger happen, taint is removed from . gets an update event for and observes only one taint, . It proceeds to calculate the minimum time (in seconds) that tolerates for the this taint which is infinity, and returns, not canceling the previous deletion. It completely ignores the fact that the taint that triggered deletion is no longer observed. Anything else we need to know?: Worked w/ to figure this out after all pods in a node got evicted without apparent reason. Environment: Kubernetes version (use ): 1.17.3 (but looking at the code seems to affect all of 1.17.x and 1.18.x and master. Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): 5.x Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A cc (owners of this code)
There seems to be at least .


Question: for _, item := range testCases { stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) <del> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{node: item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) controller.NodeUpdated(item.oldNode, item.newNode)

You can sort of get a feel for what's going on here by looking at these two code snippets: This is what happens in when the node does not have any NoExecute taints: If you have any NoExecute taints on the node processPodOnNode gets called and this routine is really, really eager to evict pods. So : We are currently working around this issue by changing over our NoExecute taints to NoSchedule.
/sig scheduling node
/cc
/remove-sig scheduling scheduler isn't involved in this case.
Which SIG owns the controller-manager or is the node lifecycle manager (and its NoExecuteTaintManager) owned by sig-node alone?
I think node lifecycle is jointly owned by sig-node and sig-cloud-provider
API machinery owns the mechanics of the controller manager (controller loop setup and management), but the individual controllers are SIG-specific
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Looks like we lost some momentum here and I'd like to avoid this falling into the abyss of bugs that never get fixed. When this bug is triggered the outcome is pretty terrible and once it does happen it is very difficult to dig into and understand what actually happened. Did we ever get this routed to the right owner?
adding sig-node leads for triage/routing. this seems like a clearly reproducible bug with significant correctness issues. /assign
thanks for the detailed report and reproducer. to help accelerate this, do you have a form of your reproducer we could add in the e2e suite (even if its failing initially)?
custom taints are considered disruptive, so it couldn't run in the main suite
It's been a while since I wrote tests for the e2e suite but if you are available over slack in case guidance is needed, I'm willing to put the effort.
Do all suites need to pass for a fix to be merged? I'd assume it is the case, otherwise how would a regression be prevented after the fix lands?


Question: for _, item := range testCases { stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) <del> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{node: item.newNode}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) controller.NodeUpdated(item.oldNode, item.newNode)

i was more inquiring if there was a test that could automate the reproduction, so even just a linked commit may be helpful for someone that wants to pick this up. i am happy to help if you want to reach out over slack later this week.
looks like a unit test could reproduce the issue, opened a WIP fix in
our code is not publicly accessible, I'm sorry. PR linked by seems to cover both the fix and the test that proves it.
/remove-lifecycle frozen
frozen keeps the bot from auto-closing
Ah, sorry then.


Question: rm.Stop() }) Describe("regular resource usage tracking", func() { <ins> // We assume that the scheduler will make reasonable scheduling choices // and assign ~N pods on the node. // Although we want to track N pods per node, there are N + add-on pods // in the cluster. The cluster add-on pods can be distributed unevenly // among the nodes because they are created during the cluster // initialization. This *noise* is obvious when N is small. We // deliberately set higher resource usage limits to account for the // noise. </ins> rTests := []resourceTest{ {podsPerNode: 0, limits: containersCPUSummary{ <del> "/kubelet":       {0.50: 0.05, 0.95: 0.15}, "/docker-daemon": {0.50: 0.03, 0.95: 0.06}, </del> <ins> "/kubelet":       {0.50: 0.06, 0.95: 0.08}, "/docker-daemon": {0.50: 0.05, 0.95: 0.06}, </ins> }, }, {podsPerNode: 35, limits: containersCPUSummary{ <del> "/kubelet":       {0.50: 0.15, 0.95: 0.35}, "/docker-daemon": {0.50: 0.06, 0.95: 0.30}, </del> <ins> "/kubelet":       {0.50: 0.12, 0.95: 0.14}, "/docker-daemon": {0.50: 0.06, 0.95: 0.08}, </ins> }, }, }

It fails all the time in . cc
I suspect the node was in a bad state. After deploying the new build, the test hasn't failed at all this week. Closing this and will re-open if it hits again.
It's failing again: and so on.
cc
The 0 pods per node test is more sensitive to noise caused by uneven distribution of cluster add-on pods. I adjusted the limits in to reflect the current kubelet status and also to account for the noise. Ideally we should disable the cluster add-ons in the cluster, but there is no easy way to do this in the current test framework. I think we should consider moving this test with hard limits to the node conformance test suite (where the addons are not enabled). We can keep this test here for monitoring purpose, but don't fail the build explicitly. We can integrate with other monitoring service and send alerts if necessary.
- actually it is easy to do during cluster startup. We can create a dedicated Node suite and run Node tests there (if it's not done already).
we actually just started node conformance tests in , which one of the v1.2 targets. It seems like this test might be a good fit there. cc will know more about porting/migrating tests to the node conformance suites.
Yes, one of the reasons we create a separate node test suite is moving this kind of test over. Let's try to move all node level resource tracking tests over to node test suite.
failing in https://k8s- https://k8s-
[FLAKE-PING] This flaky-test issue would love to have more attention.
This issue is not intended for GCI nor upgrade suites. Filing a new issue might be better. GCI has had problems with the docker memory usage for a while, as you can see in https://k8s- I think they fixed it in the latest dev image, but I am not sure if that was used in the upgrade tests.


Question: //       Above tests are performed by doSheduletTests() and doPodResizeResourceQuotaTests() //       in test/node/pod_resize_test.go <del> var _ = SIGDescribe("Pod InPlace Resize Container", framework.WithSerial(), feature.InPlacePodVerticalScaling, func() { </del> <ins> var _ = SIGDescribe("Pod InPlace Resize Container", framework.WithSerial(), feature.InPlacePodVerticalScaling, "[NodeAlphaFeature:InPlacePodVerticalScaling]", func() { </ins> if !podOnCgroupv2Node { cgroupMemLimit = CgroupMemLimit cgroupCPULimit = CgroupCPUQuota

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
could be related to
/assign
Yes, tests should be skipped with Feature:InPlacePodVerticalScaling on those CIs , it fails because InPlacePodVerticalScaling is not enabled. Will update tomorrow morning, checking more closely how to fix in a smooth way
Thanks , It was failing in 11 jobs for two reasons: In 7 jobs because InPlacePodVerticalScaling was not supported and was missed to identify, attempt to fix by because all impacted CI jobs will skip tests with labels with prefix NodeAlphaFeatures In 4 jobs because Arm64 was not checked in one test, attempt to fix by


Question: } bhash := sha256.New().Sum(body) <del> // Port in the signature must be that of the reverse proxy port, vCenter's default is port 80 port := "80" // TODO: get from lookup service </del> <ins> port := req.URL.Port() if port == "" { port = "80" // Default port for the "Host" header on the server side } </ins> var buf bytes.Buffer msg := []string{ nonce,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The vSphere cloud provider is unable to fetch the node failure domain labels when using VCP with SAML auth because of an authentication failure. For context, the SAML token auth is described here - SAML auth support for the zones api was introduced in this PR - , which was cherry-picked to 1.14.1 here - I've tested the credentials I'm using with govc to ensure that I'm able to get a token: The credentials also work fine when using VCP without the node labels feature. Relevant kubelet logs: What you expected to happen: VCP should add failure domain labels to the node object. How to reproduce it (as minimally and precisely as possible): Setup SAML auth for VCP with zone labeling enabled. Anything else we need to know?: There was a previous issue that was raised () but that was before SAML auth support for zones was implemented. This could potentially be an issue with govc handling of SAML tokens since even with the Administrator role assigned to the user, I'm still unable to list tags. The role and keys were generated with the following: Environment: Kubernetes version (use ): 14.1 Cloud provider or hardware configuration: vSphere OS (e.g: ): Linux & Windows
/sig vmware
Would you mind taking a look at this? I can provide more info if needed.
govc's command only works with the SOAP endpoint currently. The REST endpoint has its own session manager and requires a separate login. At the moment, govc's tags commands login/logout when each command is run and don't use a persisted session (which are saved to ~/.govmomi/sessions). However, you can use a token returned by to test SAML auth with the tags commands. For example: Can you give that try? There are some TODOs around this btw:
Those commands work for me. So that would imply the solution user, role, and permissions are set up correctly? Would you expect the to also work?


Question: } bhash := sha256.New().Sum(body) <del> // Port in the signature must be that of the reverse proxy port, vCenter's default is port 80 port := "80" // TODO: get from lookup service </del> <ins> port := req.URL.Port() if port == "" { port = "80" // Default port for the "Host" header on the server side } </ins> var buf bytes.Buffer msg := []string{ nonce,

I seem to be getting a 404 when using with the flag - Does seem to work when specifying a tag name: Are there any other govc commands I should check / verify?
yes, the solution user setup looks right. Not sure what the issue is with . Can you share your vCenter build number ()? I'll try to reproduce the issue.
$ govc about -u vcdo Name: VMware vCenter Server Vendor: VMware, Inc. Version: 6.7.0 Build: OS type: linux-x64 API type: VirtualCenter API version: 6.7.2 Product ID: vpx UUID: -55ed-4737-9cc4-
I can reproduce this, but don't know what the issue is yet. The version of Go and vSphere have changed since , but if it was one of those was the cause, I'd expect SAML auth to fail with the govc tags commands too. All I have to go on for now is the vCenter log message below. I'll update when I have more info.
I have it working with the change below, but need to figure out a few things first: Why this is the case when run inside kubelet and not govc (note that the port change to 443 breaks govc) A good way to determine which port to use (depends on the answer to the above) If this is behavior is specific to 6.7U2 or something triggered by a change in K8s I suppose a quick fix would change to try port 443 if 80 fails, but would prefer to avoid that as the failure generates stack traces in the
/assign
This is obvious in retrospect, but the issue is the message signature needs to use the same port as the original request URL. However, in the govc case there was no port set in the request URL. The client defaults to port when connecting (based on the scheme), but does not include any port with the header. In the case, the default is on the server side. This can be reproduced with if includes the port , rather than just as we were using. The VCP tests that use token auth for tags lookup just don't validate the signature, so this wasn't a change related to 6.7U2.


Question: } bhash := sha256.New().Sum(body) <del> // Port in the signature must be that of the reverse proxy port, vCenter's default is port 80 port := "80" // TODO: get from lookup service </del> <ins> port := req.URL.Port() if port == "" { port = "80" // Default port for the "Host" header on the server side } </ins> var buf bytes.Buffer msg := []string{ nonce,

I'll open a K8s PR to pull in the fix:
Updating k/vendor to govmomi master adds almost 6K lines for a 3 line fix. I may just create a govmomi branch off of v0.20 to reduce churn in the K8s release branches for this fix, may have to wait until tomorrow.
Really appreciate the insight into this issue. Can confirm that specifying the port (443) in the GOVC_URL reproduces the issue. Thank you!
Unfortunately, I'm seeing the same issue with v1.14.3: cloud_config Works fine when I use the username and password of a vsphere admin, but using cert and key still has authentication issues.
any related errors in vCenter's ?
Seeing a lot of this exception:
whoa! that stack trace is like punch in the gut (brings back memories!) :)
indeed!! apologies, I must have had the change for PR locally, but failed to include it as I had focused on the signature issue and updating vendor/


Question: See the [limitRange design doc](../design/admission_control_limit_range.md) and the [example of Limit Range](limitrange/) for more details. <del> ### NamespaceExists </del> <ins> ### NamespaceExists (deprecated) </ins> This plug-in will observe all incoming requests that attempt to create a resource in a Kubernetes `Namespace` and reject the request if the `Namespace` was not previously created.  We strongly recommend running this plug-in to ensure integrity of your data. <ins> The functionality of this admission controller has been merged into `NamespaceLifecycle` </ins> ### NamespaceAutoProvision (deprecated) This plug-in will observe all incoming requests that attempt to create a resource in a Kubernetes `Namespace` and create a new `Namespace` if one did not already exist previously. <del> We strongly recommend `NamespaceExists` over `NamespaceAutoProvision`. </del> <ins> We strongly recommend `NamespaceLifecycle` over `NamespaceAutoProvision`. </ins> ### NamespaceLifecycle <del> This plug-in enforces that a `Namespace` that is undergoing termination cannot have new objects created in it. </del> <ins> This plug-in enforces that a `Namespace` that is undergoing termination cannot have new objects created in it, and ensures that requests in a non-existant `Namespace` are rejected. </ins> A `Namespace` deletion kicks off a sequence of operations that remove all objects (pods, services, etc.) in that namespace.  In order to enforce integrity of that process, we strongly recommend running this plug-in. <del> Once `NamespaceAutoProvision` is deprecated, we anticipate `NamespaceLifecycle` and `NamespaceExists` will be merged into a single plug-in that enforces the life-cycle of a `Namespace` in Kubernetes. </del> ## Is there a recommended set of plug-ins to use? Yes.

We should have a single NamespaceLifecycle plugin that enforces the Namespace rules now that all providers are off NamespaceAutoProvision. We may also want to make this no longer a user choice to configure and hard-wire the server to always run this check first to simplify configuration errors. For an example


Question: "net.ipv4.ip_local_port_range", "net.ipv4.tcp_syncookies", "net.ipv4.ping_group_range", <ins> "net.ipv4.ip_unprivileged_port_start", </ins> } }

Is the sysctl namespaced? If so, can we include this in the set of "safe" sysctls? If we allow this, we would: add to permit in PodSecurity baseline profile cc So that containers can bind to low ports without needing to bake CAPNETBINDSERVICE into the files in the image doesn't help hostNetwork pods since they don't have their own namespace and you likely wouldn't want to change the system-wide sysctl also, pods that schedule to arbitrary nodes likely shouldn't start taking advantage of this until 2+ releases after it merges into the kubelet, because of potential node skew
/sig network security node auth
xref: for actually potentially setting this by default even.
/assign
Yes, it is network-namespaced and :+1: to adding it to safe list. From
/triage accepted


Question: "//staging/src/k8s.io/apimachinery/pkg/api/resource:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library", <ins> "//staging/src/k8s.io/component-base/featuregate:go_default_library", </ins> "//staging/src/k8s.io/component-base/featuregate/testing:go_default_library", "//vendor/github.com/stretchr/testify/assert:go_default_library", ],

What happened: In a v1.18 cluster with the LocalStorageCapacityIsolation feature gate disabled setting an ephemeral-storage request or limit causes the pod to be unschedulable: If the ephemeral-storage request is removed the pod is scheduled. This behavior exists in v1.19 and a recent v1.20 build. What you expected to happen: In previous releases such pods are scheduled. I expect that any ephemeral-storage limit is not enforced. How to reproduce it (as minimally and precisely as possible): Disable the LocalStorageCapacityIsolation feature guide by adding to the kube-scheduler. Create a pod like this: Anything else we need to know?: The problem was originally seen after moving from IBM Cloud Red Hat OpenShift v4.4 to v4.5. Pods that worked in 4.4 (Kubernetes v1.17) remained in a Pending state in OpenShift v4.5 (Kubernetes 1.18). We discovered that the IBM Cloud OpenShift service has the LocalStorageCapacityIsolation feature-gate disabled while Red Hat has enabled the feature. We then replicated the behavior with a plain Kubernetes cluster. Environment: Kubernetes version (use ): v1.18.10, v1.19.3, v1.20.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-scheduling
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
I can confirm it's a bug. Will work on it.


Question: "//staging/src/k8s.io/apimachinery/pkg/api/resource:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library", <ins> "//staging/src/k8s.io/component-base/featuregate:go_default_library", </ins> "//staging/src/k8s.io/component-base/featuregate/testing:go_default_library", "//vendor/github.com/stretchr/testify/assert:go_default_library", ],

/assign
Clarifying the recreate steps: I disabled the LocalStorageCapacityIsolation feature guide by adding to the kube-scheduler since this behavior seems to be within the scheduler.
Thanks for confirming!
This bug has been fixed in master branch, and will be backported to [1.18] and [1.19] soon. [1.18]: [1.19]:


Question: update-generated-docs update-generated-swagger-docs update-openapi-spec <del> update-bazel </del> update-gofmt" for t in ${BASH_TARGETS}; do

This is intended to be removed once there are no active branches with bazel. 1.19 was still supported last I checked. We have some other cleanup pending the final turndown of bazel branches.
19 just had it's last patch release a week ago , these are removed now.


Question: # TODO: Re-enable race detection when we switch to a thread-safe etcd client # KUBE_RACE="-race" make -C "${KUBE_ROOT}" test  <del> WHAT="$(kube::test::find_integration_test_dirs | paste -sd' ')"  </del> <ins> WHAT="$(kube::test::find_integration_test_dirs | paste -sd' ' -)"  </ins> KUBE_GOFLAGS="${KUBE_GOFLAGS:-} -tags 'integration no-docker'"  KUBE_RACE=""  KUBE_TIMEOUT="${KUBE_TIMEOUT}"

I don't know where the at the beginning of the target comes from, but probably from assumptions about sed version that are wrong. I had to update to gnu-sed 4.2.2 to build.
Also ootb on a Mac doesn't work with the rules specified (on el capitan):
does work
I take it back - updating sed did not appear to work. EDIT: I take that back, I had to clean after fixing paste.
most recent OSX version is UNIX certified, a POSIX requires as file argument when reading input from stdin. For , is only guaranteed to work in GNU sed, you need to insert literal newline and tab character there for portability.
Can someone with a mac send a patch? I don't have a mac...
It's update and every should end in instead of having no arguments. , Tim Hockin wrote:
(user should update sed, make needs changes to paste) , Clayton Coleman wrote:
I think we should stick to portable way for , it's too risky for relying on GNUism. If you don't mind, I will make a PR.
Sure, thanks , Cuong Manh Le wrote:
Are you sure the worked? I pull the latest from master, then run the same command above, I got: File is empty.
Make sure you are in your GOPATH in the correct spot for Kube On Jul 14, 2016, at 2:47 AM, Cuong Manh Le wrote: Are you sure the Makefile worked? I pull the latest from master, then run the same command above, I got: F0714 13:36:50. ] Error: Failed making a parser: unable to add directory "": unable to import "": expected 'package', found 'EOF' make: * [Makefile.generatedfiles:201: gendeepcopy] Error 255 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
If you get the "expected 'package', found 'EOF'" message it is usually because something somewhere failed, and left an empty .go file. make cleangenerated , Cuong Manh Le wrote:
From BSD sed man page: Maybe try that?
It's not about ERE ( enables it), it's about expanding escape sequences in RHS of search and replace .


Question: } // check if deployment's has finished the rollout <del> status, done, err := statusViewer.Status(cmdNamespace, info.Name, revision) </del> <ins> status, done, err := statusViewer.Status(info.Namespace, info.Name, revision) </ins> if err != nil { return err }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): rollout status namespace Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Macbook Pro 2015 OS (e.g. from /etc/os-release): Mac OSX Sierra Kernel (e.g. ): Install tools: Others: What happened: , where does the following: Describes a Deployment object Sets a non-default namespace in the metadata . An error message says the resource was not found . The response shows the status of the rollout. What you expected to happen: I expected to work as well by using the namespace that is set inside for the rollout status request. How to reproduce it (as minimally and precisely as possible): the following file as (taken from the but updated to add the namespace): Returns Returns Anything else we need to know: We've seen this issue on 1.5.1 as well, and also on Ubuntu. The recreate instructions above were performed on a fresh minikube setup.
I also noticed this today. The node is using version 1.5.2 and we are using latest gcloud SDK components (v146). At the moment I have to use
This issue also exists on kubectl built from HEAD.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: } // check if deployment's has finished the rollout <del> status, done, err := statusViewer.Status(cmdNamespace, info.Name, revision) </del> <ins> status, done, err := statusViewer.Status(info.Namespace, info.Name, revision) </ins> if err != nil { return err }

Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: {% if grains.get('cloud', '') == 'gce' and grains.get('os_family', '') == 'Debian' and grains.get('oscodename', '') == 'wheezy' -%} <del> {% set docker_pkg_name='' %} {% set override_deb='' %} {% set override_deb_sha1='' %} {% set override_docker_ver='' %} </del> <ins> {% set docker_pkg_name='docker-engine' %} {% set override_deb='docker-engine_1.9.1-0~wheezy_amd64.deb' %} {% set override_deb_sha1='d682e2f0545e21f2d7309c2d87826aa566cc4af0' %} {% set override_docker_ver='1.9.1' %} </ins> # Ubuntu presents as os_family=Debian, osfullname=Ubuntu {% elif grains.get('cloud', '') == 'aws' and grains.get('os_family', '') == 'Debian'

.
is this your turn?
Sure :) I will work on this! , Dawn Chen wrote:
Update: I'm testing v1.9.0~rc4. I'm running the following suite of tests (based on ) - , , .
Update: No tests failed in suite. This release has been looking good so far.
Update: No tests failed in suite. :smiley:
Update: tests passed.
Note, I'm think we want to set the flag from in
What dns options have to be set?
I would ask but I assume he the flag for a reason ;)
we can use dns-opt when we know we don't have to fall-back on older docker versions. We current have a (very clever) hack that works. We set the ndots option. , Brendan Burns wrote:
Docker has just released v1.9, so I think v1.9 should be validated on high priority.
yep, validation is underway by See above.
I have tested against most of the test suites. As of now there are no known issues. I'm planning on running soak tests next.
I haven't been able to find any issues. I will update HEAD to use docker v1.9 soon.
FYI: (Docker 1.9 serious performance issues)
Are the new /etc/hosts changes going to be an issue?
AFAIK, we want docker to not touch once the container is created.
re:/etc/hosts - we have our workaround which should dodge it entirely. , Vish Kannan wrote:
+1, why is K8S still on docker 1.7.1?
Docker v1.9.1 was released a few weeks ago.
it might be useful to run your mircobenchmark for docker v1.9 before we try to upgrade.
There are still some performance issues with v1.9.1. As of now I'm updating HEAD to run docker v1.9.1 for now to help identify other potential issues. , Yu-Ju Hong wrote:
Can we make sure v1.9.1 can pass e2e tests reliably before upgrading? I am dreading the endless test failures that we'll have to triage at the end of the day. The performance benchmark, e.g., at least can give us some idea about how this is going to affect kubelet.
Yes.


Question: {% if grains.get('cloud', '') == 'gce' and grains.get('os_family', '') == 'Debian' and grains.get('oscodename', '') == 'wheezy' -%} <del> {% set docker_pkg_name='' %} {% set override_deb='' %} {% set override_deb_sha1='' %} {% set override_docker_ver='' %} </del> <ins> {% set docker_pkg_name='docker-engine' %} {% set override_deb='docker-engine_1.9.1-0~wheezy_amd64.deb' %} {% set override_deb_sha1='d682e2f0545e21f2d7309c2d87826aa566cc4af0' %} {% set override_docker_ver='1.9.1' %} </ins> # Ubuntu presents as os_family=Debian, osfullname=Ubuntu {% elif grains.get('cloud', '') == 'aws' and grains.get('os_family', '') == 'Debian'

Of course! , Yu-Ju Hong wrote:
I got the result I post before with docker 1.9: Go Version: 1.5.1 Docker Version: Client Version: 1.9.0, API version: 1.21 Server Version: 1.9.0, API version: 1.21
We are now running docker 1.9.1 on the Mesosphere K8s CI. However, this is a little ambiguous, because the containers that run the kubelets still use docker 1.8.2. This upgrade did, however, fix a regular problem we were having in our CI scripts on docker 1.7.1 where layer pulls would block and/or stall (without any other concurrent docker use).
Besides the performance issue I brought up at , there are a network related issue in docker 1.9.1 should affect k8s: cc/ & We should wait for 1.9.2 for whatever reason.
xref: , Dawn Chen wrote:
Introduction I benchmarked and with the following 3 experiments: periodically and , gradually increase the dead container number and alive container number. periodically and , gradually increase the operation interval. periodically and , gradually increase the number of goroutines which doing and . The metrics we measured: CPU usage. latency. latency. Benchmark Environment Cloud Provider: GCE VM Instance: n1-standard-2 (2 vCPUs, 7.5 GB memory) OS: Debian GNU/Linux 8.3 (jessie) Docker version: Experiment 1 Workflow: start -create containers -periodically -periodically -periodically -create containers -... -finish Environment Variable Operation interval: 200ms Container Number ! Latency Docker 1.8.3 ! Docker 1.9.1 ! CPU Usage Docker 1.8.3 ! Docker 1.9.1 ! Experiment 2 Workflow: start -periodically with increasing interval -periodically with increasing interval -periodically with increasing interval -finish Environment Variable Dead container number: 600 Alive container number: 200 Interval ! [all=true] Latency Docker 1.8.3 ! Docker 1.9.1 ! [all=false] Latency Docker 1.8.3 ! Docker 1.9.1 !


Question: {% if grains.get('cloud', '') == 'gce' and grains.get('os_family', '') == 'Debian' and grains.get('oscodename', '') == 'wheezy' -%} <del> {% set docker_pkg_name='' %} {% set override_deb='' %} {% set override_deb_sha1='' %} {% set override_docker_ver='' %} </del> <ins> {% set docker_pkg_name='docker-engine' %} {% set override_deb='docker-engine_1.9.1-0~wheezy_amd64.deb' %} {% set override_deb_sha1='d682e2f0545e21f2d7309c2d87826aa566cc4af0' %} {% set override_docker_ver='1.9.1' %} </ins> # Ubuntu presents as os_family=Debian, osfullname=Ubuntu {% elif grains.get('cloud', '') == 'aws' and grains.get('os_family', '') == 'Debian'

Interval ! Latency Docker 1.8.3 ! Docker 1.9.1 ! CPU Usage Docker 1.8.3 ! Docker 1.9.1 ! Experiment 3 Result Workflow: start-periodically with increasing groutines number -periodically with increasing groutines number -finish Environment Variable Dead container number: 600 Alive container number: 200 interval: 1s interval: 0.5s Routine Number ! The change rule of routine number is the same for and . [all=true] Latency Docker 1.8.3 ! Docker 1.9.1 ! Latency Docker 1.8.3 ! Docker 1.9.1 ! CPU Usage Docker 1.8.3 ! Docker 1.9.1 ! Conclusion There is no significant performance difference for and between docker 1.8.3 and docker 1.9.1. Benchmark Tool /cc


Question: - [SecurityContextDeny](#securitycontextdeny) - [ResourceQuota](#resourcequota) - [LimitRanger](#limitranger) <del> - [NamespaceExists](#namespaceexists) </del> <ins> - [NamespaceExists (deprecated)](#namespaceexists-deprecated) </ins> - [NamespaceAutoProvision (deprecated)](#namespaceautoprovision-deprecated) - [NamespaceLifecycle](#namespacelifecycle) - [Is there a recommended set of plug-ins to use?](#is-there-a-recommended-set-of-plug-ins-to-use)


Question: group_versions+=("${leaf2}/${leaf}") fi done < <( <del> ( kube::codegen::internal::git_grep -l  </del> <ins> ( kube::codegen::internal::git_grep -l --null  </ins> -e '+genclient'  <del> ":(glob)${in_root}"/'**/types.go'  </del> <ins> ":(glob)${in_root}${one_input_api}"/'**/*.go'  </ins> || true  <del> ) | LC_ALL=C sort -u </del> <ins> ) | xargs -0 -n1 dirname  | LC_ALL=C sort -u </ins> ) if [ "${#group_versions[@]}" == 0 ]; then

function in script has following issues: function only supports file but we have multiple types file with name as in . Hence this function is not working in our case. For the mitigation there is already an open . I have asked for including our changes as well. We have multiple files under a for example and . Because of this for appears twice and it generates duplicate informers We only need unique elements before passing it to . I followed this approach for unique elements: We have packages under . Earlier we were generating client only for and . But with the new I am not able to do that. I am using the following call to function: It results in following flags to As there is no directory as , it doesnot generate anything. Is it still possible to call on specific folder as it was done with earlier scripts? One mitigation I can think of is to call function on whole and then remove the marker from the types for which genclient is not needed. But the issue is we won't be able to pass the flag for each pkg. function in script is not working as expected.
/sig api-machinery
/cc
/triage accepted
Hi . I knew that there were all sorts of clients out there doing all sorts of things with these scripts, so I am not surprised to find you. :) Do you have a PR which attempts to use kube_codegen, so I can repro your issues directly?
I think you have one too many in there Aside from that, the real problem here is that these old codegen tools make some deeeeeeep assumptions about the ability to string-join various arguments and get the right results. It's awful. I was trying to make it simpler in this script, which I did for all of the in-tree consumers, but out of tree people apparently don't always follow the same conventions. For example, we generate clients for every directory that requests it. For a reason I do not know, you have files like which say but you don't actually want a client? If you specify then it t
So this code expects that you generate an omnibus client. What if I add a flag, like which you can use to call it for each API to make distinct clients? In truth the codegen tools need to be fixed, but that has SO MUCH ripple I just can't right now.


Question: group_versions+=("${leaf2}/${leaf}") fi done < <( <del> ( kube::codegen::internal::git_grep -l  </del> <ins> ( kube::codegen::internal::git_grep -l --null  </ins> -e '+genclient'  <del> ":(glob)${in_root}"/'**/types.go'  </del> <ins> ":(glob)${in_root}${one_input_api}"/'**/*.go'  </ins> || true  <del> ) | LC_ALL=C sort -u </del> <ins> ) | xargs -0 -n1 dirname  | LC_ALL=C sort -u </ins> ) if [ "${#group_versions[@]}" == 0 ]; then

As the codegen was breaking us at several step, hence didn't prepared the PR.
Thanks for the PR. I have one more concern, is it possible to call and client-gen on the same pkg? With the current script, that is not possible because of the line It removes all the generated files first and then calls the . Hence the later clientset gen will remove the earlier generated clients.
Why are you generating a client for an internal version that isn't served?
My mistake. We need to stop using clients for internal versions.


Question: fw := compute.Firewall{} fw.Name = gceController.GetFirewallRuleName() <del> fw.SourceRanges = []string{GCEL7SrcRange} </del> <ins> fw.SourceRanges = gcecloud.LoadBalancerSrcRanges() </ins> fw.TargetTags = nodeTags.Items fw.Allowed = []*compute.FirewallAllowed{ {

From GCE email today: Current behavior: You are not required to add firewall rules for Network LB health checks today Action required: We are introducing three new ranges for health checking: 209.85.152.0/22, 209.85.204.0/22, and 35.191.0.0/16. You need to ensure that your firewall allows traffic from 209.85.152.0/22, 209.85.204.0/22, and 35.191.0.0/16. The health check probes to your backend instances will come from one of these ranges. We have until Jan 31, 2018 - 1 year.
/assign this issue explains the confusion I had several weeks ago when speaking with and you about L4 health checks. I didn't realize a firewall rule isn't currently required.
/assign
Is this done now?
Not yet, there is a WIP on .
/close


Question: "net", "--runner", "nat-closewait-server", "--options", <del> fmt.Sprintf(`{"LocalAddr":"0.0.0.0:%v", "PostFindTimeoutSeconds":%v}`, </del> <ins> fmt.Sprintf(`{"LocalAddr":":%v", "PostFinTimeoutSeconds":%v}`, </ins> testDaemonTCPPort, postFinTimeoutSeconds), },

Which jobs are failing: kops-aws and many other jobs Which test(s) are failing: This isn't the actual test, but the text we're consistently seeing is Since when has it been failing: Testgrid links: (ref: ) Reason for failure: TBD Anything else we need to know: Very rarely tests pass when one of the tests passes: Seems related to , that was merged about the same time:
/sig cluster-lifecycle /sig aws
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cluster-lifecycle /sig aws Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent this is merge blocking
/assign /cc /cc /cc
The PR we think we want to revert
is also failing, I don't think this is aws-specific
/cc
/assign
it failed with kind too :/ with the same message
Opened to revert
/unassign /sig network /sig testing /remove-sig cluster-lifecycle
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This may be the actual cause PR to address But we're going with revert first to be sure we've identified the offending PR
I think I've identified the root cause
Revert brought us back to green: Awaiting root cause fix
/remove-triage unresolved
/reopen merged but we're now seeing multiple tests fail since it merged Looking at it went red between these commits:
Reopened this issue. <detailsIn response to : /reopen merged but we're now seeing multiple tests fail since it merged Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is an attempt to fix
reverting in
is the failure cluster I suspect will go down once the revert goes in
/close Failure cluster has gone down, calling this done <img width="998" alt="Screen Shot 2020-01-10 at 9 56 17 AM" src="https://user-"
Closing this issue.


Question: "net", "--runner", "nat-closewait-server", "--options", <del> fmt.Sprintf(`{"LocalAddr":"0.0.0.0:%v", "PostFindTimeoutSeconds":%v}`, </del> <ins> fmt.Sprintf(`{"LocalAddr":":%v", "PostFinTimeoutSeconds":%v}`, </ins> testDaemonTCPPort, postFinTimeoutSeconds), },

<detailsIn response to : /close Failure cluster has gone down, calling this done <img width="998" alt="Screen Shot 2020-01-10 at 9 56 17 AM" src="https://user-"</details


Question: // +groupName=wardle.k8s.io // Package api is the internal version of the API. <del> package wardle </del> <ins> package wardle // import "k8s.io/sample-apiserver/pkg/apis/wardle" </ins>

Look at one of the directories under (Please don't pick more than one directory for one PR as it will cause issues for reviewers) Find all the files named (example find -name ) See the following link it has a this tells tools how this package needs to be imported. So we should ensure that each has an import Add a description appropriate to the package (similar to the one below) Look through sub directories and see if there are any other important directories that are likely to be used by outside users and add a Why are we doing this? We perfer to use vanity urls instead of directly using to refer to code, so this helps ensure that tools honor our wish. See It would be good to have some package level documentation since these directories under staging/ are meant to be for use by folks outside of kubernetes github org. /good-first-issue
/kind documentation /sig architecture
currently working my through Not going to use / assign since there could be a number of PRs for this issue.
/lifecycle active /good-first-issue edit: hmm..
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /lifecycle active /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
May I try on
IIUC this means that there needs to be a even at the root of the repo? Then how does work? :thinking:
Hi , can I make a PR for this directory ?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .