Question: return nil } <del> var redactedBytes []byte </del> <ins> var ( redactedBytes    []byte dataOmittedBytes []byte ) </ins> // Flatten redacts raw data entries from the config object for a human-readable view. func ShortenConfig(config *Config) {

Is this a BUG REPORT or FEATURE REQUEST?: What happened: shows: The use of "REDACTED" for a CA here is misleading, since it's a public certificate that is not a secret. What you expected to happen: I expected to see the full CA in pem format (like when you cat ) or something like "SUPPRESSED" or "TRUNCATED" How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Nope Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g. from /etc/os-release):Kernel (e.g. ):
(for thoughts)
Reiterating the mentions to trigger a notification: <detailsIn response to : (for thoughts) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
use you can see the complete certificate authority data
Hey Thank you - I have tracking that . The point of this issue is that at best it's a bad UX; at worst it's a miseducation on pki. "REDACTED" isn't the right word here
If there are multiple clusters displaying certificates, the configuration will be too long, so this switch is turned off by default. But we can add an example to help use. thanks
Hey I don't personally view truncation the responsibility of the CLI... users can pipe things to a pager if they really care. That being said, I'm not concerned about the length. The issue is purely about the word choice of "REDACTED". The term means: The use of the term "REDACTED" in the output implies that a CA public key is remove for security when (as you've pointed out) it is removed for truncation. Truncation and redaction are fundamentally different things, and k8s is misusing the word redacted for this use case.
Not quite a bug, but I think you're right a better term could be used there as it is a public key, not a private key. /remove-kind bug
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: return nil } <del> var redactedBytes []byte </del> <ins> var ( redactedBytes    []byte dataOmittedBytes []byte ) </ins> // Flatten redacts raw data entries from the config object for a human-readable view. func ShortenConfig(config *Config) {

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale I'm not sure what I need to do here. This was marked as a feature. Do you need something more from me?
We are waiting for someone to fix this.
/remove-good-first-issue I don't think this fits our definition of a :
So I have to keep replying to the comment bot every 30 days so it doesn't get auto-closed? That feels... odd
thanks for the education :) unfortunately other issues are higher priority and no one has stepped up to fix this. The autocloser is a form of bug bankruptcy that keeps our issue load in check. The contribex special interest group is definitely open to feedback on the bot.
The biggest hurdle in this is that Kubernetes is using some base64 trickery to display the string (basically setting the certificate data to the decoded bytes from ). We can change this to or but not (it has to be a valid base64 encoded string). I tried exploring multiple options for the text, but I couldn't find a suitable one. Given that we already print the user tokens and other credentials this is a bit weird. Should we redact those too? Is it okay if we print the CA certificate instead of truncating it?
I can't help but laugh at: I'm not wed to any particular name. I also have no problem displaying the entire certificate. The crux of the issue is that a CA public key is not secret or sensitive. It's a public key
I agree. The issue is with the constraints imposed by how this is implemented currently. I tried to come up with a suitable string (given the constraint) and couldn't find anything acceptable... The only suitable option that worked for me is . I personally prefer to print the CA certificate since we are already printing user tokens and client certs that are actual secrets.
I agree. Who do we need to get in agreement that printing the full CA is the best path forward?
the data is not omitted for security purposes, but to make the output more readable. see REDACTED is misleading... it should probably be "elided" or "..."
you can run to view full content
that's the crux of this issue - REDACTED is misleading.


Question: return nil } <del> var redactedBytes []byte </del> <ins> var ( redactedBytes    []byte dataOmittedBytes []byte ) </ins> // Flatten redacts raw data entries from the config object for a human-readable view. func ShortenConfig(config *Config) {

However, pointed out that other "long" fields are still completely printed in plain text such as client certs, so I think we need to revisit the "readability" argument. What do you think?
they are not... client cert/key are elided in the same way for the same reason: something like "++data+omitted++" could be better If the intent of config view without --raw specified is to be readable (which it currently is), printing many extremely long fields isn't great
Hmm what about: I'm pretty sure that would give us a valid b64 string (8+12=20%4=0) and it would subtly hint that the data isn't a secret.
Only valid characters are . And adding a prefix from the certificate only complicates things. is a suitable usable option. I'll submit a PR to continue the discussion there.


Question: // When specified, it should match one the container or initContainer // names in the pod template. // +optional <del> ContainerName *string `json:"containerName" protobuf:"bytes,1,opt,name=containerName"` </del> <ins> ContainerName *string `json:"containerName,omitempty" protobuf:"bytes,1,opt,name=containerName"` </ins> // Represents the relationship between the container exit code(s) and the // specified values. Containers completed with success (exit code 0) are

for a PodFailurePolicyRule. However, when trying to create a JobSet programmatically with a JobTemplate that defines a PodFailurePolicy rule for failing a job immediately if the failed pod had condition DisruptionTarget, I get a validation error indicating that cannot be null. I've tried using both and (with matching versions for package as well). Error: JobSet spec: JobTemplate passes validation. JobSet: applying example JobSet in the description of this issue. JobSet: and applying example JobSet in the description of this issue. No response 30 (tests on a kind cluster using kindest/node:v1.30.0) <details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
/assign
I confirmed the issue, moreover, I see that when the JobSet webhook is disabled then it works ok, I think it must be the JobSet webhook which adds the "onExitCodes" field during deserializion /serialization step. I can also see that when I rebuild JobSet using kubernetes field "onExitCodes" with "omitempty", then it the JobSet creates ok. I will open a PR.


Question: name = "go_default_test", size = "large", srcs = [ <ins> "etcd_cross_group_test.go", </ins> "etcd_storage_path_test.go", "main_test.go", ],

What happened: Set up a HA cluster with 1.9 and 1.10 API servers (occurs on any apiserver skew boundary where a cohabitating resource's storage version changed groups) Observed errors when modifying workload objects: when modified via the 1.10 server, saw this error in the 1.9 server: when modified via the 1.9 server, saw this error in the 1.10 server: When this error was encountered, all watchers for the resource in question would be dropped and have to re-establish, and the internal watchers feeding the watch cache would drop and re-establish What you expected to happen: Watch events from all supported storage versions are processed correctly How to reproduce it (as minimally and precisely as possible): Start a server from master (for inspectability, store in json). With Create a deployment (e.g. kube-dns in kube-system) Establish a watch via extensions (kubectl get deployments.extensions --all-namespaces -w) Establish a watch via apps (kubectl get --all-namespaces -w) Tail the kube-apiserver log Change the apps/v1 deployment in etcd to extensions/v1beta1: Observer client watches drop and server log errors: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig api-machinery /cc /priority critical-urgent
error is triggered by this returning false: constructed here: the info about expected cohabitating groups is not included in the list of acceptable inputs


Question: Expect(curHistory).NotTo(BeNil()) return curHistory } <ins> func waitFailedDaemonPodDeleted(c clientset.Interface, pod *v1.Pod) func() (bool, error) { return func() (bool, error) { if _, err := c.CoreV1().Pods(pod.Namespace).Get(pod.Name, metav1.GetOptions{}); err != nil { if errors.IsNotFound(err) { return true, nil } return false, fmt.Errorf("failed to get failed daemon pod %q: %v", pod.Name, err) } return false, nil } } </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: DaemonSet has an e2e test running that has dubious purpose. What you expected to happen: The e2e test should be refactored and commented clearer as it will be part of the conformance.
To elaborate, we're concerned that the following may not actually ensure that DaemonSet recreates Failed Pods: For example, it seems possible that the Node might simply overwrite the Pod status with its own information if it hasn't yet seen the change to the Phase. Ideally, we should verify that the Pod UID we force to Failed no longer shows up in the DaemonSet's list of Pods after the DaemonSet becomes happy again.


Question: func init() { sDec, _ := base64.StdEncoding.DecodeString("REDACTED+") redactedBytes = []byte(string(sDec)) <ins> sDec, _ = base64.StdEncoding.DecodeString("DATA+OMITTED") dataOmittedBytes = []byte(string(sDec)) </ins> } // IsConfigEmpty returns true if the config is empty.


Question: func init() { sDec, _ := base64.StdEncoding.DecodeString("REDACTED+") redactedBytes = []byte(string(sDec)) <ins> sDec, _ = base64.StdEncoding.DecodeString("DATA+OMITTED") dataOmittedBytes = []byte(string(sDec)) </ins> } // IsConfigEmpty returns true if the config is empty.


Question: func init() { sDec, _ := base64.StdEncoding.DecodeString("REDACTED+") redactedBytes = []byte(string(sDec)) <ins> sDec, _ = base64.StdEncoding.DecodeString("DATA+OMITTED") dataOmittedBytes = []byte(string(sDec)) </ins> } // IsConfigEmpty returns true if the config is empty.


Question: cachedService.service = service s.cache.set(namespacedName.String(), cachedService) case cache.Deleted: <del> err := s.ensureLBDeleted(service) </del> <ins> err := s.balancer.EnsureTCPLoadBalancerDeleted(s.loadBalancerName(service), s.zone.Region) </ins> if err != nil { return err, retryable }

All of our external load balancer tests failed for the better part of a day due to the GCE project that our tests run in hitting its quota for number of target pools. While the target pools have been deleted, the fact that we hit our quota indicates that there must be a leak somewhere, and we need to plug it.
For reference, the previous issue was
Could the leakage be related to ? The quota exhaustion occurred about 22 hrs after that.
I'd guess that this is due to GetTCPLoadBalancer returning that the load balancer doesn't exist if the forwarding rule is gone, ignoring the target pool, while the service controller considers its deleting done if the cloud provider says that the load balancer doesn't exist. I that logic to the cloud provider because the other implementations of DeleteTCPLoadBalancer weren't/won't be idempotent in the sense of returning ok if the LB doesn't exist. To fix it, we'll either need to make sure all Delete implementations are idempotent and remove the Get call in the controller, or add a method along the lines of IsTCPLoadBalancerDeleted to ask the cloud provider whether all pieces of it have been completely cleaned up. We can't make GetTCPLoadBalancer check that because it's relied on to tell us whether or not the balancer is fully created.
Copying over my commentary from on why the GKE e2e tests are leaking so many forwarding rules and target pools vs the GCE tests:
Is there a way for the services test to wait for the service controller to clean up the external resources before finishing (without just doing a sleep)?
Not without polling GCE directly for whether the resources still exist, although that may be worth it as a workaround. What would be perfect is that clean-up-all-cloud-provider-resources API method that we talked about a couple months ago.
Can we not perhaps use GCE namespaces or projects as a cheap way of getting this here?
I don't believe so. We can't use projects due to the fact that projects can't be programatically created (they're intended to be long-lived). I'm not aware of anything along the lines of a GCE namespace other than networks, which won't work because target pools and forwarding rules unfortunately don't seem to be scoped to networks.


Question: cachedService.service = service s.cache.set(namespacedName.String(), cachedService) case cache.Deleted: <del> err := s.ensureLBDeleted(service) </del> <ins> err := s.balancer.EnsureTCPLoadBalancerDeleted(s.loadBalancerName(service), s.zone.Region) </ins> if err != nil { return err, retryable }

The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.


Question: cachedService.service = service s.cache.set(namespacedName.String(), cachedService) case cache.Deleted: <del> err := s.ensureLBDeleted(service) </del> <ins> err := s.balancer.EnsureTCPLoadBalancerDeleted(s.loadBalancerName(service), s.zone.Region) </ins> if err != nil { return err, retryable }

I'll periodically check to see whether anything is leaked after this point. I apologize if the deletions affected any running tests - everything looked to only be from GKE and parallel e2e runs.


Question: sleep 5 <del> POD_LIST_1=$($KUBECFG '-template={{range.Items}}{{.Meta}} {{end}}' list pods) </del> <ins> POD_LIST_1=$($KUBECFG '-template={{range.Items}}{{.Name}} {{end}}' list pods) </ins> echo "Pods running: ${POD_LIST_1}" $KUBECFG stop redisSlaveController

What happened: is a subdirectory of the btrfs subvolume . kubelet fails to start with the following error: What you expected to happen: kubelet starts normally and doesn't fail with an error message How to reproduce it (as minimally and precisely as possible): Install a system using btrfs as file system for / Create a sub-volume for /var Start kubelet (even in disconnected mode) Anything else we need to know?: This is a follow-up issue of and has been promoted to fatal from onward, before it was a warning PR to fix the issue is Environment: Kubernetes version (use ): forward Cloud provider or hardware configuration: both bare metal and virtual OS (e.g. from /etc/os-release): SUSE 12+ or openSUSE (Tumbleweed/Leap) Kernel (e.g. ): Install tools: SUSE/openSUSE rpm Others:
/sig storage
fixed by /close
If I may - adding some notes - it might be helpful for others, until I get the time to do some more detailed tests and report a bug. I still had issues with kubelet not starting, with the same error - (version 1.13.4) My btrfs setup is that btrfs root is mounted at /, then I have also a subvolume /var and in it a subvolume /var/lib. The issue for me I think comes from the fact that I do not have a flat btrfs setup () but a nested one, and as such, I have as a mountpoint ONLY the root directory. Manually adding a mount entry for /var (heh - I should probably have mounted /var/lib) in /etc/fstab made it work: /var btrfs rw,relatime,ssd,space_cache,subvolid=259,subvol=/var 0 0` Also, even without this additional "intermediate" mount point I think it should have been working. I wonder if in , the loop 'for !found {' isn't exiting a tiny bit too fast (without testing for the root mountpoint itself) . Maybe it should be something like:
Has something changed in the code? Kubelet from kubernetes 1.20.1 is failing. Doing a mount as I described above fixes it.
see (open) and (closed) - apparently this was recently fixed (but isn't released yet).


Question: delete(objs, namespacedName) for _, w := range t.getWatches(gvr, ns) { <del> w.Delete(obj) </del> <ins> w.Delete(obj.DeepCopyObject()) </ins> } return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Hello, Found a potential data race in testing. There is a test If we update it like this (we actually can do it ): If watcher updates received object we get a data race. No data race My suggestion: send copy of object here:
/sig api-machinery
/cc /triage accepted Do you remember if we fixed similar issues before?
Hi ,The code avoiding data race seems to already exist If you commented the code on above, the following test case will pass (because of data race), so I think there is no data race in the code : )
Hi Did you run my test with flag? Of course if I comment there will be data race. But let's take a look at: We store in and do with the same . So, watcher receives that is stored in map. we call we take the same from : Therefore, if watcher modifies received in another goroutine we have data race. Output:
Hi You're right ,thank you for solving my confusion. One solution could be to copy of , I will try to fix it.


Question: }) It("should write files of various sizes, verify size, validate content", func() { <del> fileSizes := []int64{1 * framework.MiB, 100 * framework.MiB} </del> <ins> fileSizes := []int64{fileSizeSmall, fileSizeMedium} </ins> err := testVolumeIO(f, cs, config, volSource, nil /*no secContext*/, testFile, fileSizes) Expect(err).NotTo(HaveOccurred()) })

The "Volume plugin streaming [Slow] NFS should write files of various sizes, verify size, validate content" e2e test is failing on our 2k-node gce clusters (https://k8s-) with the following error: cc
Could you take a look into it or reassign as apt? This is part of the correctness suite and large cluster tests are release-blocking.
/assign
is this run using the kubemark environment? I'll also take a look. It could be that the commands this test needs to execute may not work with kubemark. /assign
cc
No, this is not kubemark. This is on a real 2000-node cluster.
Also all other volume-related e2e tests are passing, except this one.
Looking at the log I see that the nfs test fails trying to verify the test file's content. The test exec's into the nfs client pod and executes: The return code from above is 127. The test then deletes the nfs client pod calling which works. The nfs-server pod is deleted in the AfterEach. For some reason the reported error is 137 but the returned err 127. The same command is exec'd for all of the tested storage plugins so not sure why it's failing just for nfs. Error 127 might be "command not found".
The other strange thing is that it's only failing in this suite and not the others.
When I look at the kubelet log, I see the exec command coming through: I also see these suspicious kubelet soon afterwards: So I think the grep command is causing the node to run out of memory and get evicted. I think we can fix this either by: Making the grep command more memory efficient. I'm guessing it's reading the whole file into memory or, Make this test Serial so that it doesn't run in parallel with other pods, since it consumes a lot of memory.
An unrelated issue, is that the node allocatable feature didn't actually evict the pod. Instead this pod got a system OOM kill. /cc
related to the memcg notifications, not node allocatable. Thanks!
any theories why we see this only for the nfs-io tests, and not the glusterfs-io test for instance?
the gluster test didn't run on this in this suite.


Question: }) It("should write files of various sizes, verify size, validate content", func() { <del> fileSizes := []int64{1 * framework.MiB, 100 * framework.MiB} </del> <ins> fileSizes := []int64{fileSizeSmall, fileSizeMedium} </ins> err := testVolumeIO(f, cs, config, volSource, nil /*no secContext*/, testFile, fileSizes) Expect(err).NotTo(HaveOccurred()) })

I'm suspecting that the reason why we didn't see a problem with this test on other suites is that the load could be higher here. I've asked to see if we can reduce the memory usage of the grep call.
pointed out that there are no newlines in the generated file, so when grep is called it could buffer the entire file in memory. Adding a newline at the end of the dd input file should do it.
/priority critical-urgent
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/scalability sig/storage: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details
It seems like even if the generated file is broken up by newlines, grep could still use on the order of hundreds of MB of memory for a 100MiB file (even with the -m1 and -F options). We could check the file line by line, but it would take a long time. A better solution would be to check the MD5 file hash. It's fast and uses little memory. The expected hashes will be hard-coded into the test, and if different file sizes are in the future, a hash needs to be computed and as well.
Thanks a lot for fixing this!
Ref


Question: <ins> /* Copyright 2016 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package codeinspector import ( "fmt" "go/ast" "go/parser" "go/token" "io/ioutil" "strings" "unicode" ) // GetPublicFunctions lists all public functions (not methods) from a golang source file. func GetPublicFunctions(filePath string) ([]string, error) { var functionNames []string // Create the AST by parsing src. fset := token.NewFileSet() // positions are relative to fset f, err := parser.ParseFile(fset, filePath, nil, 0) if err != nil { return nil, fmt.Errorf("failed parse file to list functions: %v", err) } // Inspect the AST and print all identifiers and literals. ast.Inspect(f, func(n ast.Node) bool { var s string switch x := n.(type) { case *ast.FuncDecl: s = x.Name.Name // It's a function (not method), and is public, record it. if x.Recv == nil && isPublic(s) { functionNames = append(functionNames, s) } } return true }) return functionNames, nil } // isPublic checks if a given string is a public function name. func isPublic(myString string) bool { a := []rune(myString) a[0] = unicode.ToUpper(a[0]) return myString == string(a) } // GetSourceCodeFiles lists golang source code files from directory, excluding sub-directory and tests files. func GetSourceCodeFiles(dir string) ([]string, error) { files, err := ioutil.ReadDir(dir) if err != nil { return nil, err } var filenames []string for _, file := range files { if strings.HasSuffix(file.Name(), ".go") && !strings.HasSuffix(file.Name(), "_test.go") { filenames = append(filenames, file.Name()) } } return filenames, nil } </ins>

I just noticed you forgot to register the image locality priority function in in I also missed it when I reviewed your PR. I have fixed this in . But it would be great if you could figure out a mechanism that would ensure that all predicate/priority functions that are tested (you can assume they will always live under ) are registered, and fail the test if not. That will allow us to prevent this mistake in the future.
Thanks! Will fix this soon!
Seems need to wait for in before I send PR? Otherwise this test will always fail ...


Question: podSpec.EphemeralContainers = nil } <del> if (!utilfeature.DefaultFeatureGate.Enabled(features.VolumeSubpath) || !utilfeature.DefaultFeatureGate.Enabled(features.VolumeSubpathEnvExpansion)) && !subpathExprInUse(oldPodSpec) { // drop subpath env expansion from the pod if either of the subpath features is disabled and the old spec did not specify subpath env expansion </del> <ins> if !utilfeature.DefaultFeatureGate.Enabled(features.VolumeSubpath) && !subpathExprInUse(oldPodSpec) { // drop subpath env expansion from the pod if subpath feature is disabled and the old spec did not specify subpath env expansion </ins> VisitContainers(podSpec, AllContainers, func(c *api.Container, containerType ContainerType) bool { for i := range c.VolumeMounts { c.VolumeMounts[i].SubPathExpr = ""

/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc


Question: }, { "ImportPath": "github.com/json-iterator/go", <del> "Rev": "f2b4162afba35581b6d4a50d3b8f34e33c144682" </del> <ins> "Rev": "ab8a2e0c74be9d3be70b3184d9acc634935ded82" </ins> }, { "ImportPath": "github.com/mailru/easyjson/buffer",

is a library yet is is instead of release tag. Please consider vendoring by tag whenever possible. Thanks.
We vendored it that way because the commit that fixed a bug (which we needed to be fixed and vendored back in) was This was merged on June 13. The release 1.1.4 () was released on July 1. If we want to update it to 1.1.4, I believe it should be fine. Between the commit we have vendored right now and the latest release, there are only three commits: adds tests merges (?) the case-sensitive fix - I'm not really sure what this commit is doing. fixes renaming when naming strategy is set (looks fine). Also, there are also no licensing changes. opinions?
/sig api-machinery
+1 to what said, it was a fire drill and 1.11.0 release was in trouble. Though in hind sight, we should have opened a issue to track that. something better to do next time. cc
I'm indifferent to updating this. I'm okay with moving up if it doesn't break anything, or I'm fine with leaving things as is. We don't require deps to match semver. We can version out any commit in a deps tree, as we rely on our vendored copy.
It is always a good idea to match semver and it should be a default policy. Even if it doesn't matter to you, it should, because it is a best practice. Moreover downstream it matters much more as dependencies may be un-vendored in favour of tested re-usable packages.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Someone please take care of this or bot will eventually close this valid issue without action...
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: }, { "ImportPath": "github.com/json-iterator/go", <del> "Rev": "f2b4162afba35581b6d4a50d3b8f34e33c144682" </del> <ins> "Rev": "ab8a2e0c74be9d3be70b3184d9acc634935ded82" </ins> }, { "ImportPath": "github.com/mailru/easyjson/buffer",

/lifecycle rotten
/lifecycle rotten
/remove-lifecycle rotten PR open (thanks,
oops! sorry, I meant to remove it. thanks!
Thanks.
please do feel free to open issues/prs of this sort early in the release cycle, so we can work through things needed well before the milestones/freeze etc. we are getting some heat about updating dependencies and we kind of settled on some guidelines here: if you think that needs updating, we could definitely debate that as well :) (on a PR against that doc)
it have nothing to do with release cycle or justification for upgrade if/when dependency has been vendored incorrectly. It is basically a vendoring bug. No debate is necessary although that kind of issues usually meet some resistance and even hostility here... Bugs are filed when they are discovered, regardless of release cycle. It would be wrong to sit on the bug without reporting it in order to synchronise with release cycle.


Question: } } <ins> // estimateMaxStringLengthPerRequest estimates the maximum string length (in characters) // that has a set of enum values. // The result of the estimation is the length of the longest possible value. func estimateMaxStringEnumLength(s Schema) int64 { var maxLength int64 for _, v := range s.Enum() { if s, ok := v.(string); ok && int64(len(s)) > maxLength { maxLength = int64(len(s)) } } return maxLength } </ins> // estimateMaxArrayItemsPerRequest estimates the maximum number of array items with // the provided minimum serialized size that can fit into a single request. func estimateMaxArrayItemsFromMinSize(minSize int64) int64 {

pointed out to me that a CEL x-kubernetes-validations rule in a CRD was hitting the estimated cost limit when performing a simple "string has prefix" check on an en enum field. Putting a limit in the OpenAPI of the enum string field fixed the problem, but having to add a limit like this is counter-intuitive and unnecessary, since the allowed enum values already make the max allowed length obvious. The apiserver automatically computes the max length of an enum as the length of the longest allowed enum value. I believe we already refuse evaluate CEL rules on values that fail OpenAPI value validation, since we rely in the value validations to establish all length limits. I haven't tested it locally, but something like this should work (given enough nested arrays): As a workaround, one can manually set on the string enum, e.g.: 25+ All No response No response No response No response
/sig api-machinery /triage accepted /assign
/assign
I wasn't sure if we special-cased the maxLength / maxItems / maxProperties failures, or if all openapi validation had to pass before we started messing with CEL evaluation
it looks like we currently special-case specific openapi errors that short-circuit CEL validation, I'm not sure an enum validation failure would currently prevent evaluation:
The enum err is converted to () error which is not short-circuit here. But for defaulting case, all error would be short-circuit. I would suggest to add error into the blockingErr list to short-circuit CEL validation. Would this make sense?
yeah, that sounds good if we're going to rely on it to catch things that are longer than the maximum length enum value


Question: All               bool Record            bool ChangeCause       string <del> Local             bool </del> Cmd               *cobra.Command Limits               string

from https://k8s-
Really suspect this is
This is putting us into test-go post-commit run is broken
cc:
marked this p0 - if a fix isn't easy let's try to identify what introduced this and revert.
This flake on: 2016-10-30 07:19 PDT First instance: : It is flaking frequently, blocking PRs from getting merged, and slowing the submit queue ( at 75%). As on-call, I'm going to try and identify the cluprit and revert. Can use all help in doing so.
can you investigate whether your additional tests could cause this (by breaking the deployment of those pods, for instance) , Saad Ali wrote:
PR looks most suspect to me. It mucked with , and the errors we are seeing in each of the failures is related to : That PR was merged at Oct 30, 2016, 6:18 AM PDT and the first flake in was at Oct 30, 2016, 7:19 AM PDT. Thoughts? CC
I'm going to initiate a revert of to see if that fixes the issue
IMO we can just change the flaky test to not get pods but some other resources, such as deployments. We can't really expect to clean up pods immediately.
I'm going to proceed with revert (). Please feel free to send a follow up PR to handle this more gracefully.
Going to leave this opened until we have a few hours of runs with the revert. Want to verify, with some degree of certainty, that the revert took care of this flake.
We had 15 green runs of in a row. Considering this flake fixed.


Question: w.Flush() if trackingWriter.Written == 0 && !o.IgnoreNotFound && len(allErrs) == 0 { // if we wrote no output, and had no errors, and are not ignoring NotFound, be sure we output something <del> noResourcesFoundFormat := fmt.Sprintf("No resources found in %s namespace.", o.Namespace) fmt.Fprintln(o.ErrOut, noResourcesFoundFormat) </del> <ins> if !o.AllNamespaces { fmt.Fprintln(o.ErrOut, fmt.Sprintf("No resources found in %s namespace.", o.Namespace)) } else { fmt.Fprintln(o.ErrOut, fmt.Sprintf("No resources found")) } </ins> } return utilerrors.NewAggregate(allErrs) }

What happened: When running and no resource is found, kubectl returns , which is confusing, since we're trying to list instances of a cluster-scoped resource. What you expected to happen: When listing cluster-scoped resources, the message should either be or How to reproduce it (as minimally and precisely as possible): (assuming there are no ; can also try or similar) Environment: Kubernetes version (use ):
/sig cli
/reopen Reopening, because did not fix this issue. It only fixed , which also produced a misleading message, but that's not what this issue is about. This issue is about cluster-scoped resources, not about listing namespaced resources in all namespaces. Output of kubectl get with the changes from included:
Reopened this issue. <detailsIn response to : /reopen </details
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen This seems to be an error in the get code for kubectl, it looks like it tries to figure out if all the requested resources are namespaced and only then return specify the namespace in the output. This seems to be happening when we have only cluster-wide resources.


Question: w.Flush() if trackingWriter.Written == 0 && !o.IgnoreNotFound && len(allErrs) == 0 { // if we wrote no output, and had no errors, and are not ignoring NotFound, be sure we output something <del> noResourcesFoundFormat := fmt.Sprintf("No resources found in %s namespace.", o.Namespace) fmt.Fprintln(o.ErrOut, noResourcesFoundFormat) </del> <ins> if !o.AllNamespaces { fmt.Fprintln(o.ErrOut, fmt.Sprintf("No resources found in %s namespace.", o.Namespace)) } else { fmt.Fprintln(o.ErrOut, fmt.Sprintf("No resources found")) } </ins> } return utilerrors.NewAggregate(allErrs) }

You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: podCopy := pod.DeepCopy() // NominatedNodeName is updated only if we are trying to set it, and the value is // different from the existing one. <del> if !podutil.UpdatePodCondition(&podCopy.Status, condition) && pod.Status.NominatedNodeName == nominatedNode { </del> <ins> if !podutil.UpdatePodCondition(&podCopy.Status, condition) && (len(nominatedNode) == 0 || pod.Status.NominatedNodeName == nominatedNode) { </ins> return nil } <del> podCopy.Status.NominatedNodeName = nominatedNode </del> <ins> if nominatedNode != "" { podCopy.Status.NominatedNodeName = nominatedNode } </ins> return patchPod(client, pod, podCopy) }

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Conformance - GCE - master (ci-kubernetes-gce-conformance-latest) Which test(s) are failing: [sig-scheduling] SchedulerPreemption [Serial] PreemptionExecutionPath runs ReplicaSets to verify preemption running path [Conformance]Changes Since when has it been failing: 2020-06-08 22:27 PDT Testgrid link: Reason for failure: Regression by Anything else we need to know: /sig scheduling /priority critical-urgent
cc/ /assign


Question:
PLEASE NOTE: This document applies to the HEAD of the source tree
 If you are using a released version of Kubernetes, you should refer to the docs that go with that version.  The latest 1.0.x release of this document can be found [here](http://releases.k8s.io/release-1.0/docs/user-guide/kubectl-overview.md). Documentation for other releases can be found at [releases.k8s.io](http://releases.k8s.io).  --   # kubectl overview **Table of Contents** - [kubectl overview](#kubectl-overview) - [Overview](#overview) - [Common Operations](#common-operations) - [Kubectl Operations](#kubectl-operations) - [Resource Types](#resource-types) This overview is intended for anyone who wants to use `kubectl` command line tool to interact with Kubernetes cluster. Please remember that it is built for quick started with `kubectl`; for complete and detailed information, please refer to [kubectl](kubectl/kubectl.md). TODO: auto-generate this file to stay up with `kubectl` changes. Please see [#14177](https://github.com/kubernetes/kubernetes/pull/14177). ## Overview `kubectl` controls the Kubernetes cluster manager. The synopsis is: ``` kubectl [command] [TYPE] [NAME] [flags] ``` This specifies: - `command` is a certain operation performed on a given resource(s), such as `create`, `get`, `describe`, `delete` etc. - `TYPE` is the type of resource(s). Both singular and plural forms are accepted. For example, `node(s)`, `namespace(s)`, `pod(s)`, `replicationcontroller(s)`, `service(s)` etc. - `NAME` is the name of resource(s). `TYPE NAME` can be specified as `TYPE name1 name2` or `TYPE/name1 TYPE/name2`. `TYPE NAME` can also be specified by one or more file arguments: `-f file1 -f file2 ...`, [use YAML rather than JSON](config-best-practices.md) since YAML tends to be more user-friendly for config. - `flags` are used to provide more control information when running a command. For example, you can use `-s` or `--server` to specify the address and port of the Kubernetes API server. Command line flags override their corresponding default values and environment variables. [Use short flags sparingly, only for the most frequently used options](../devel/kubectl-conventions.md). Please use `kubectl help [command]` for detailed information about a command. Please refer to [kubectl](kubectl/kubectl.md) for a complete list of available commands and flags. ## Common Operations For explanation, here I gave some mostly often used `kubectl` command examples. Please replace sample names with actual values if you would like to try these commands. 1. `kubectl create` - Create a resource by filename or stdin // Create a service using the data in example-service.yaml. $ kubectl create -f example-service.yaml // Create a replication controller using the data in example-controller.yaml. $ kubectl create -f example-controller.yaml // Create objects whose definitions are in a directory. This looks for config objects in all .yaml, .yml, and .json files in <directory> and passes them to create. $ kubectl create -f <directory> 2. `kubectl get` - Display one or many resources // List all pods in ps output format. $ kubectl get pods // List all pods in ps output format with more information (such as node name). $ kubectl get pods -o wide // List a single replication controller with specified name in ps output format. You can use the alias 'rc' instead of 'replicationcontroller'. $ kubectl get replicationcontroller <rc-name> // List all replication controllers and services together in ps output format. $ kubectl get rc,services 3. `kubectl describe` - Show details of a specific resource or group of resources // Describe a node $ kubectl describe nodes <node-name> // Describe a pod $ kubectl describe pods/<pod-name> // Describe all pods managed by the replication controller <rc-name> // (rc-created pods get the name of the rc as a prefix in the pod the name). $ kubectl describe pods <rc-name> 4. `kubectl delete` - Delete resources by filenames, stdin, resources and names, or by resources and label selector // Delete a pod using the type and name specified in pod.yaml. $ kubectl delete -f pod.yaml // Delete pods and services with label name=. $ kubectl delete pods,services -l name= // Delete all pods $ kubectl delete pods --all 5. `kubectl exec` - Execute a command in a container // Get output from running 'date' from pod <pod-name>, using the first container by default. $ kubectl exec <pod-name> date // Get output from running 'date' in <container-name> from pod <pod-name>. $ kubectl exec <pod-name> -c <container-name> date // Get an interactive tty and run /bin/bash from pod <pod-name>, using the first container by default. $ kubectl exec -ti <pod-name> /bin/bash 6. `kubectl logs` - Print the logs for a container in a pod. // Returns snapshot of logs from pod <pod-name>. $ kubectl logs <pod-name> // Starts streaming of logs from pod <pod-name>, it is something like 'tail -f'. $ kubectl logs -f <pod-name> ## Kubectl Operations The following table describes all `kubectl` operations and their general synopsis: Operation       | Synopsis	|       Description -------------------- | -------------------- | -------------------- annotate	| `kubectl annotate [--overwrite] (-f FILENAME | TYPE NAME) KEY_1=VAL_1 ... KEY_N=VAL_N [--resource-version=version]` | Update the annotations on a resource api-versions	| `kubectl api-versions` | Print available API versions attach		| `kubectl attach POD -c CONTAINER` | Attach to a running container cluster-info	| `kubectl cluster-info` | Display cluster info config		| `kubectl config SUBCOMMAND` | Modifies kubeconfig files create		| `kubectl create -f FILENAME` | Create a resource by filename or stdin delete		| `kubectl delete ([-f FILENAME] | TYPE [(NAME | -l label | --all)])` | Delete resources by filenames, stdin, resources and names, or by resources and label selector describe	| `kubectl describe (-f FILENAME | TYPE [NAME_PREFIX | -l label] | TYPE/NAME)` | Show details of a specific resource or group of resources edit		| `kubectl edit (RESOURCE/NAME | -f FILENAME)` | Edit a resource on the server exec		| `kubectl exec POD [-c CONTAINER] -- COMMAND [args...]` | Execute a command in a container expose		| `kubectl expose (-f FILENAME | TYPE NAME) [--port=port] [--protocol=TCP|UDP] [--target-port=number-or-name] [--name=name] [----external-ip=external-ip-of-service] [--type=type]` | Take a replication controller, service or pod and expose it as a new Kubernetes Service get		| `kubectl get [(-o|--output=)json|yaml|wide|go-template=...|go-template-file=...|jsonpath=...|jsonpath-file=...] (TYPE [NAME | -l label] | TYPE/NAME ...) [flags]` | Display one or many resources label		| `kubectl label [--overwrite] (-f FILENAME | TYPE NAME) KEY_1=VAL_1 ... KEY_N=VAL_N [--resource-version=version]` | Update the labels on a resource logs		| `kubectl logs [-f] [-p] POD [-c CONTAINER]` | Print the logs for a container in a pod namespace	| `kubectl namespace [namespace]` | SUPERSEDED: Set and view the current Kubernetes namespace patch		| `kubectl patch (-f FILENAME | TYPE NAME) -p PATCH` | Update field(s) of a resource by stdin port-forward	| `kubectl port-forward POD [LOCAL_PORT:]REMOTE_PORT [...[LOCAL_PORT_N:]REMOTE_PORT_N]` | Forward one or more local ports to a pod proxy		| `kubectl proxy [--port=PORT] [--www=static-dir] [--www-prefix=prefix] [--api-prefix=prefix]` | Run a proxy to the Kubernetes API server replace		| `kubectl replace -f FILENAME` | Replace a resource by filename or stdin rolling-update	| `kubectl rolling-update OLD_CONTROLLER_NAME ([NEW_CONTROLLER_NAME] --image=NEW_CONTAINER_IMAGE | -f NEW_CONTROLLER_SPEC)` | Perform a rolling update of the given ReplicationController run		| `kubectl run NAME --image=image [--env="key=value"] [--port=port] [--replicas=replicas] [--dry-run=bool] [--overrides=inline-json]` | Run a particular image on the cluster scale		| `kubectl scale [--resource-version=version] [--current-replicas=count] --replicas=COUNT (-f FILENAME | TYPE NAME)` | Set a new size for a Replication Controller stop		| `kubectl stop (-f FILENAME | TYPE (NAME | -l label | --all))` | Deprecated: Gracefully shut down a resource by name or filename version		| `kubectl version` | Print the client and server version information ## Resource Types The `kubectl` supports the following resource types, and their abbreviated aliases: Resource Type	| Abbreviated Alias -------------------- | -------------------- componentstatuses	|	cs events	|	ev endpoints	|	ep horizontalpodautoscalers	|	hpa limitranges	|	limits nodes	|	no namespaces	|	ns pods	|	po persistentvolumes	|	pv persistentvolumeclaims	|	pvc resourcequotas	|	quota replicationcontrollers	|	rc daemonsets	|	ds services	|	svc  [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/user-guide/kubectl-overview.md?pixel)]()

Something like: cc
cc
- Can you please look at this issue since you're recently working on documentation?
thanks for the reminder. I will take care of it.


Question: memoryUsage, err := strconv.ParseFloat(lines[0], 64) framework.ExpectNoError(err) <ins> var rssToken, inactiveFileToken string if isCgroupV2 { // Use Anon memory for RSS as cAdvisor on cgroupv2 // see https://github.com/google/cadvisor/blob/a9858972e75642c2b1914c8d5428e33e6392c08a/container/libcontainer/handler.go#L799 rssToken = "anon" inactiveFileToken = "inactive_file" } else { rssToken = "total_rss" inactiveFileToken = "total_inactive_file" } </ins> var totalInactiveFile float64 for _, line := range lines[1:] { tokens := strings.Split(line, " ") <del> if tokens[0] == "total_rss" { </del> <ins> if tokens[0] == rssToken { </ins> rss, err = strconv.ParseFloat(tokens[1], 64) framework.ExpectNoError(err) } <del> if tokens[0] == "total_inactive_file" { </del> <ins> if tokens[0] == inactiveFileToken { </ins> totalInactiveFile, err = strconv.ParseFloat(tokens[1], 64) framework.ExpectNoError(err) }

N/A [sig-node] NodeProblemDetector: N/A https://prow- Not cgroupv2 compatible when access cpu and memory data
/sig node
Hi, can i work on this ?
Sure. Thanks
/assign
/triage accepted
/priority important-soon
/cc
are you still working on this? Were you able to setup environment to repro the test failure? Let us know if you need any help. Thanks!
Sorry for the delay, I followed the wrong guide and ended up getting stuck. But now I have managed to set it up. Please give me few more days to work on it.
Thanks Feel free to reach out here or on slack if you need any help, thanks for working on this!
/retitle Make nodeproblemdetector test cgroupv2 compatible
Failing job:
Testgrid:


Question: // How long to wait for a log pod to be displayed const podLogTimeout = 45 * time.Second <del> // utility function for gomega Eventually func getPodLogs(c *client.Client, namespace, podName, containerName string) (string, error) { logs, err := c.Get().Resource("pods").Namespace(namespace).Name(podName).SubResource("log").Param("container", containerName).Do().Raw() if err != nil { return "", err } if err == nil && strings.Contains(string(logs), "Internal Error") { return "", fmt.Errorf("Internal Error") } return string(logs), err } </del> var _ = Describe("Downward API volume", func() { f := NewFramework("downward-api") It("should provide podname only [Conformance]", func() {

We are running heapster, dns server, kube-proxy etc. in kube-system namespace. Some e2e tests try to make sure all those kube-system pods are at ready state before firing the real tests, others just simply ingore them. But both cases might cause e2e flakiness when some of those kube-system containers are in crashloop, for example, for heapster crashloop, for influxdb crashloop, etc. When such cases happen, the developer doesn't have much information on why since the nodes might be teared down. We should watch those kube-system pods, and onContainerFailure, we should collect the related logs. One simple way is using kubectl logs <Pod<Container--previous cc/
is this related enough to and that you can bundle it in?
The logs from the pods should be retrieved via the api server. This is different from the kubelet logs. I'll upload a PR for this.
Great job It will be very helpful for debugging.


Question: ], library = ":go_default_library", deps = [ <ins> "//federation/apis/federation:go_default_library", </ins> "//federation/apis/federation/v1beta1:go_default_library", "//federation/client/clientset_generated/federation_clientset/fake:go_default_library", "//pkg/api:go_default_library",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After labeling clusters in a Federation, when you type the field is empty. What you expected to happen: The fiels should contain information about the labels in the cluster How to reproduce it (as minimally and precisely as possible): a federation control pane in any Cloud Provider and any DNS provider your cluster: info aboyut the cluster with the flag: Note: if you describe it, you see the labels being : Environment: Kubernetes version (use ): 1.8.0 Cloud provider or hardware configuration**: anyone OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux Install tools: curl Others:
/sig multicluster
I'm working on it.


Question: glog.Infof("Running tests for APIVersion: %s", apiVersion) <del> manifestURL := ServeCachedManifestFile() apiServerURL, configFilePath := startComponents(manifestURL, apiVersion) </del> <ins> firstManifestURL := ServeCachedManifestFile(testPodSpecFile) secondManifestURL := ServeCachedManifestFile(testManifestFile) apiServerURL, configFilePath := startComponents(firstManifestURL, secondManifestURL, apiVersion) </ins> // Ok. we're good to go. glog.Infof("API Server started on %s", apiServerURL)

Currently it assumes it can read ContainerManifest - it should try both
Why not hold on this until we have Kubelet-read-Pod in?
Also the HTTP source has the same problem.
It should read Pod. Changed the title.
Ideally, I'd like all Pods created directly via Kubelet to be to etcd and surfaced via the API. If updates are made via the API, I'd like those to propagate back to the node. We can discuss whether the config file should be updated or ignored. At some point, I'd like Kubelet to persistently cache pods received from the apiserver, perhaps by just writing them all out as config files.
is the issue to sync local pods back to apiserver.
this is somewhat related to other issues we talked about.
Detail: We want locally specified pods to be able to use YAML config files, but we also want to rip YAML out of the API server . The config-file behavior should be as similar to the kubectl experience. Kubelet could use parts of the kubectl library to translate YAML to JSON. Vision: Eventually, I'd like Kubelet to use the apiserver library and support exactly the same Pod API as the master, at which time I'd like kubectl be able to create pods directly on Kubelets, as well as reading from config files. I also expect to have a per-node controller . I'd like to be able to use the per-node controller to update pods started via config files (or direct kubectl-kubelet pods), overriding those original files. For availability and self-hosting, we'll eventually need local checkpointing , ideally just using serialized JSON for each object. If Kubelet used apiserver, this could be done using a custom registry implementation.
I think that is a prerequisite for it.
If noone picks it up, I'm going to start working on it tomorrow.
thanks, wojtek-t
fixed it for http channel


Question: // buildELBSecurityGroupList returns list of SecurityGroups which should be // attached to ELB created by a service. List always consist of at least <del> // 1 member which is an SG created for this service. Extra groups can be </del> <ins> // 1 member which is an SG created for this service or a SG from the Global config. Extra groups can be </ins> // specified via annotation func (c *Cloud) buildELBSecurityGroupList(serviceName types.NamespacedName, loadBalancerName, annotation string) ([]string, error) { var err error

/kind bug What happened: When setting an elbSecurityGroup in for AWS, and deploying multiple services with differing ports, then the security group rules flap in AWS, causing services to become inaccessible (only one will be correctly configured at a time). What you expected to happen: I expect that the configured elbSecurityGroup's rules will not be touched at all. It must be pre-configured to permit access. Alternatively, all ports for all services are queried and managed together. How to reproduce it (as minimally and precisely as possible): See above. Anything else we need to know?: Most people are not running large clusters which may hit the AWS limits, so they won't use this option, nor see this issue. Environment: Kubernetes version (use ): 1.7.1 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 4.10.13-1-ARCH Install tools: Custom Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig aws
This is fixed by


Question: if !apierrors.IsBadRequest(err) { t.Errorf("expected HTTP status: BadRequest, got: %#v", apierrors.ReasonForError(err)) } <del> if err.Error() != expectedError { </del> <ins> if !strings.Contains(err.Error(), expectedError) { </ins> t.Errorf("expected %#v, got %#v", expectedError, err.Error()) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Below k8s UT fail when run with master-golang. TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON error:=== RUN TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON I1124 05:38:14. ] [cpumanager] initializing new in-memory state store unexpected error while creatng checkpointState: could not restore state from checkpoint: json: unexpected end of JSON input, please drain this node and delete the CPU manager checkpoint file "/tmp/cpumanagerstatetest479660663/cpumanagercheckpointtest" before restarting Kubelet --- FAIL: TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON (0.00s) TestStrategicMergePatchInvalid error:=== RUN TestStrategicMergePatchInvalid expected "invalid character 'b' looking for beginning of value", got "json: invalid character 'b' looking for beginning of value" --- FAIL: TestStrategicMergePatchInvalid (0.00s) TestJSONPatch error:=== RUN TestJSONPatch invalid-syntax: expected error invalid character 'i' looking for beginning of value, but got json: invalid character 'i' looking for beginning of value --- FAIL: TestJSONPatch (0.00s) TestV4ErrorDecoder expandless error:=== RUN TestV4ErrorDecoder wrong error for message "{": want="error stream protocol error: unexpected end of JSON input in "{"", got="error stream protocol error: json: unexpected end of JSON input in "{"" --- FAIL: TestV4ErrorDecoder (0.00s What you expected to happen: The tests should pass with no error. How to reproduce it (as minimally and precisely as possible): a stable version of golang installed master go from upstream git clone cd go/src cd ../.. cp -r ./go /usr/local export PATH=/usr/local/go/bin:$PATH kubernetes repo and run the unit tests. () Anything else we need to know?: These failures are occurring due to mismatch of error messages between upstream go and the k8s tests.


Question: if !apierrors.IsBadRequest(err) { t.Errorf("expected HTTP status: BadRequest, got: %#v", apierrors.ReasonForError(err)) } <del> if err.Error() != expectedError { </del> <ins> if !strings.Contains(err.Error(), expectedError) { </ins> t.Errorf("expected %#v, got %#v", expectedError, err.Error()) } }

There is an addition "json:" prefix to SyntaxError messages after this below commit: Environment: Kubernetes version (use ): latest source code from upstream Cloud provider or hardware configuration: OS (e.g: ): "CentOS Linux 8 (Core)" Kernel (e.g. ): Linux rajalakshmi-workspace 4.18.0-193.19.1.el82.ppc64le SMP Mon Sep 14 14:48:04 UTC 2020 ppc64le ppc64le ppc64le GNU/Linux Install tools: [root go]# go version go version devel + Fri Nov 20 00:54:43 2020 +0000 linux/ppc64le [root go]# Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
will fix it in Go 1.16 as release-blocker.
I would fix the test here to be more tolerant. exact string matches on error messages aren't really an API… I would not expect that to block the golang release
Agree too, we can think about having a substring match to avoid these issues instead!
/assign Lets see if we can fix this in the k8s tests


Question: func mountToRootfs(m *configs.Mount, c *mountConfig) error { rootfs := c.root <del> mountLabel := c.label mountFd := c.fd dest, err := securejoin.SecureJoin(rootfs, m.Destination) if err != nil { return err } </del> <ins> // procfs and sysfs are special because we need to ensure they are actually // mounted on a specific path in a container without any funny business. </ins> switch m.Device { case "proc", "sysfs": // If the destination already exists and is not a directory, we bail <del> // out This is to avoid mounting through a symlink or similar -- which </del> <ins> // out. This is to avoid mounting through a symlink or similar -- which </ins> // has been a "fun" attack scenario in the past. // TODO: This won't be necessary once we switch to libpathrs and we can //       stop all of these symlink-exchange attacks. <ins> dest := filepath.Clean(m.Destination) if !strings.HasPrefix(dest, rootfs) { // Do not use securejoin as it resolves symlinks. dest = filepath.Join(rootfs, dest) } </ins> if fi, err := os.Lstat(dest); err != nil { if !os.IsNotExist(err) { return err } <del> } else if fi.Mode()&os.ModeDir == 0 { </del> <ins> } else if !fi.IsDir() { </ins> return fmt.Errorf("filesystem %q must be mounted on ordinary directory", m.Device) } if err := os.MkdirAll(dest, 0o755); err != nil { return err } <del> // Selinux kernels do not support labeling of /proc or /sys </del> <ins> // Selinux kernels do not support labeling of /proc or /sys. </ins> return mountPropagate(m, rootfs, "", nil) <ins> } mountLabel := c.label mountFd := c.fd dest, err := securejoin.SecureJoin(rootfs, m.Destination) if err != nil { return err } switch m.Device { </ins> case "mqueue": if err := os.MkdirAll(dest, 0o755); err != nil { return err

opencontainers/runc has fixed three CVEs: Latest runc is pulled by k/k Check runc version in file No response n/a n/a <details</details<details</details<details</details<details</details
/assign /sig security architecture release
/triage accepted


Question: } } <ins> type buffer struct { b  bytes.Buffer rw sync.RWMutex } func (b *buffer) String() string { b.rw.RLock() defer b.rw.RUnlock() return b.b.String() } func (b *buffer) Write(p []byte) (n int, err error) { b.rw.Lock() defer b.rw.Unlock() return b.b.Write(p) } </ins> func TestSecretUpdated(t *testing.T) { datacenter := "0.0.0.0" secretName := "vccreds"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: No data races How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
for the record as we tend to lose the logs in the gcs bucket
/sig instrumentation


Question: // stopKubelet will kill the running kubelet, and returns a func that will restart the process again func stopKubelet() func() { kubeletServiceName := findRunningKubletServiceName() <del> stdout, err := exec.Command("sudo", "systemctl", "kill", kubeletServiceName).CombinedOutput() </del> <ins> // reset the kubelet service start-limit-hit stdout, err := exec.Command("sudo", "systemctl", "reset-failed", kubeletServiceName).CombinedOutput() framework.ExpectNoError(err, "Failed to reset kubelet start-limit-hit with systemctl: %v, %v", err, stdout) stdout, err = exec.Command("sudo", "systemctl", "kill", kubeletServiceName).CombinedOutput() </ins> framework.ExpectNoError(err, "Failed to stop kubelet with systemctl: %v, %v", err, stdout) return func() { stdout, err := exec.Command("sudo", "systemctl", "start", kubeletServiceName).CombinedOutput() framework.ExpectNoError(err, "Failed to restart kubelet with systemctl: %v, %v", err, stdout) } } <ins> func kubeletHealthCheck(url string) bool { insecureTransport := http.DefaultTransport.(*http.Transport).Clone() insecureTransport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true} insecureHTTPClient := &http.Client{ Transport: insecureTransport, } req, err := http.NewRequest("HEAD", url, nil) if err != nil { return false } req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", framework.TestContext.BearerToken)) resp, err := insecureHTTPClient.Do(req) if err != nil { klog.Warningf("Health check on %q failed, error=%v", url, err) } else if resp.StatusCode != http.StatusOK { klog.Warningf("Health check on %q failed, status=%d", url, resp.StatusCode) } return err == nil && resp.StatusCode == http.StatusOK } </ins> func toCgroupFsName(cgroupName cm.CgroupName) string { if framework.TestContext.KubeletConfig.CgroupDriver == "systemd" { return cgroupName.ToSystemd()

ci-kubernetes-node-kubelet-serial-hugepages when start the pod with the resources requests that contain multiple hugepages resources should set correct hugetlb mount and limit under the container cgroup Unexpected error: "pod ran to completion", Multiple sizes huge pages is going to be graduated to GA, so having this test job fixed is quite important.
/sig node
/assign
/triage accepted
/cc
I did a small analysis and I can see that the pod failed to start, I can not find any failure messages under the kubelet log, so it really hard to say why it failed.
were you able to reproduce it locally? btw, there is a "right way" to fix this - to fix dynamic update of a hugepage resources. This would eliminate need in a kubelet restart. There is even a PR for that: , we just need to make it working again.
after conversation with on slack I'd say we should stick to the current procedure - restarting kubelet when hugepages allocation changes. So, we should investigate why kubelet doesn't restart properly in this test case and fix it.
Nope, I failed to reproduce it locally, also under the CI, it does not appear often. The best thing will be to describe the pod in case if it failed, so we will see the reason.
under the code we have so from the log , I can see that the kubelet received the pod before the kubelet stopped, so the problem that pod dropped to Failed state because of the kubelet stop.
So some additional details, the one who responsible to restart the kubelet again, is the server service: restart the kubelet service are waiting for hugepages to be available are starting the pod server service restart the kubelet again because it was not healthy for too long - failed to start because of the kubelet restart
great findings! Should we just stop the kubelet and let the server service to start it again? We can also check if the kubelet is restarted successfully before starting the pod.
I tried it under the PR , also I unsure how can we check if the kubelet restarted successfully. All tests that I wrote before I checked if the node has NotReady state and waited until it will be Ready, but in case of the restart probably the node will always have Ready state because the time period is very short. I will continue to investigate and will check locally how can I fix it.


Question: // stopKubelet will kill the running kubelet, and returns a func that will restart the process again func stopKubelet() func() { kubeletServiceName := findRunningKubletServiceName() <del> stdout, err := exec.Command("sudo", "systemctl", "kill", kubeletServiceName).CombinedOutput() </del> <ins> // reset the kubelet service start-limit-hit stdout, err := exec.Command("sudo", "systemctl", "reset-failed", kubeletServiceName).CombinedOutput() framework.ExpectNoError(err, "Failed to reset kubelet start-limit-hit with systemctl: %v, %v", err, stdout) stdout, err = exec.Command("sudo", "systemctl", "kill", kubeletServiceName).CombinedOutput() </ins> framework.ExpectNoError(err, "Failed to stop kubelet with systemctl: %v, %v", err, stdout) return func() { stdout, err := exec.Command("sudo", "systemctl", "start", kubeletServiceName).CombinedOutput() framework.ExpectNoError(err, "Failed to restart kubelet with systemctl: %v, %v", err, stdout) } } <ins> func kubeletHealthCheck(url string) bool { insecureTransport := http.DefaultTransport.(*http.Transport).Clone() insecureTransport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true} insecureHTTPClient := &http.Client{ Transport: insecureTransport, } req, err := http.NewRequest("HEAD", url, nil) if err != nil { return false } req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", framework.TestContext.BearerToken)) resp, err := insecureHTTPClient.Do(req) if err != nil { klog.Warningf("Health check on %q failed, error=%v", url, err) } else if resp.StatusCode != http.StatusOK { klog.Warningf("Health check on %q failed, status=%d", url, resp.StatusCode) } return err == nil && resp.StatusCode == http.StatusOK } </ins> func toCgroupFsName(cgroupName cm.CgroupName) string { if framework.TestContext.KubeletConfig.CgroupDriver == "systemd" { return cgroupName.ToSystemd()

So the only reliable solution that I found, it just trying to start the pod number of times. I tested it 5 times and the test passed. Can you please review
Can we access kubelet service from hugepage test cases? If we can we can use its APIs like or to check if kubelet is ready after the restart. Overall I have a feeling of duplication test/e2enode/services functionality in This is a main reason for this and similar bugs I believe. It would be nice to get rid of this duplication if possible. That would nicely isolate kubelet service controls in one place and eliminate conflicts like this one. wdyt?
not really, the used mostly to start all needed components to emulate the cluster(not only the kubelet) and the test does not have access to it. I think you can separate the readiness check under the importable method, but it worth the separate PR.
I updated the PR to check the healthiness of the kubelet and looks like it works just fine.
thank you. it looks good to me so far. I'll add lgtm label when is merged and tests are all green.
the job is still flaky. can you reopen this issue?
I opened a different one - , the issue can affect any job, not only the hugepages one.
I can open a fix to W/A an issue(start the pod until it will succeed), it should fix all problems, WDYT?
i think it's ok as a workaround, until is fixed
would it be better to wait until api server reports correct resource value before running the pod?
We are waiting for it, the problem that the scheduler and the kubelet see a different picture regarding resources requested by pods.
ok. In this case restarting the pod until it succeeds seems reasonable to me to fix the flakiness until is fixed.
any progress on the workaround fix?


Question: handler = genericfilters.WithCORS(handler, c.CorsAllowedOriginList, nil, nil, nil, "true") <ins> // WithWarningRecorder must be wrapped by the timeout handler // to make the addition of warning headers threadsafe handler = genericapifilters.WithWarningRecorder(handler) </ins> // WithTimeoutForNonLongRunningRequests will call the rest of the request handling in a go-routine with the // context with deadline. The go-routine can keep running, while the timeout logic will return a timeout to the client. handler = genericfilters.WithTimeoutForNonLongRunningRequests(handler, c.LongRunningFunc)

I am doing v1.28.3 kube-apiserver testing in the test environment, and I often encounter kube-apiserver panic, especially when kube-apiserver call webhook fails. The following are the captured panic logs: and: kube-apiserver does not panic even if calling webhook times out or fails Deploy a webhook, remove the service corresponding to the webhook, and then create pods in batches to trigger webhook calls No response <details</details<detailsprivate cloud </details<details</details<details</details<details</details<details</details
/sig api-machinery
somewhat related to
The scenarios that trigger panic are similar(the apiserver fails to call webhook), but the log shows that the location of panic is different.
/assign For context of  Thank you. /triage accepted
again, full stack logs:
I'll try to test the mr:
Let me try to explain the panic - both goroutines are trying to write to the of the first goroutine: This request has timed out, and consequently the filter is trying to send a response to the caller. This is the client facing goroutine from the timeout filter. Note that the call to is protected by a mutex, that means that if the other goroutine is using the delegated object via the adapter, then this panic should not happen. second goroutine: let's check the other goroutine, it is also trying to write to the of the object: Now uses an unprotected version of the ResponseWriter object: The filter precedes the filter in the execution order, so the in is not using the thread safe implementation. This is the order today: one quick resolution is to move the after the timeout filter: , but there could be code that precedes that needs to write warning? Also note that the panic mentioned in is not as severe, it does not cause the apiserver to crash. For a more proper solution, i think we would want any thoughts?


Question: handler = genericfilters.WithCORS(handler, c.CorsAllowedOriginList, nil, nil, nil, "true") <ins> // WithWarningRecorder must be wrapped by the timeout handler // to make the addition of warning headers threadsafe handler = genericapifilters.WithWarningRecorder(handler) </ins> // WithTimeoutForNonLongRunningRequests will call the rest of the request handling in a go-routine with the // context with deadline. The go-routine can keep running, while the timeout logic will return a timeout to the client. handler = genericfilters.WithTimeoutForNonLongRunningRequests(handler, c.LongRunningFunc)

/cc
I tend to remove the timeout filter, the sooner the better, and I think the Apiserver panic is the P0-level bug, directly affecting the availability of the cluster I have cherry-picked and tested it in my cluster, and I still need to continue to observe for 1-2 days
I opened a minimal WIP fix for the warning writer in since the timeout filter removal is a much bigger change to reason through
thanks a lot, I will try to test and cherry-pick the above minimal WIP fix in
sorry, I completely overlooked , I only saw and that looked too big to take safely right now I prefer to
Can we backport the fix to 1.28?
Why 1.28 specifically? I'm not aware of anything that changed in 1.28 that made this more or less prevalent. If this meets backport criteria, going to all active branches could make sense (1.27+)


Question: testRootDir := makeTempDirOrDie("kubelet_integ_1.", "") configFilePath := makeTempDirOrDie("config", testRootDir) glog.Infof("Using %s as root dir for kubelet #1", testRootDir) <del> kcfg := kubeletapp.SimpleKubelet(cl, &fakeDocker1, machineList[0], testRootDir, manifestURL, "127.0.0.1", 10250, api.NamespaceDefault, empty_dir.ProbeVolumePlugins(), nil, cadvisorInterface, configFilePath) </del> <ins> kcfg := kubeletapp.SimpleKubelet(cl, &fakeDocker1, machineList[0], testRootDir, firstManifestURL, "127.0.0.1", 10250, api.NamespaceDefault, empty_dir.ProbeVolumePlugins(), nil, cadvisorInterface, configFilePath) </ins> kcfg.PodStatusUpdateFrequency = 1 * time.Second kubeletapp.RunKubelet(kcfg) // Kubelet (machine)


Question: "regexp" "strings" "syscall" <ins> "github.com/golang/glog" </ins> ) const MOUNT_MS_BIND = syscall.MS_BIND

07-02 and 13-02 No response No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
I might be related to the npd.
The same test failures happening again on 25.02:
/sig node
/sig node /cc The restart of caused the flake. I also opened to track it in NPD repo.
I missed this log. It is caused by oom. Adding some memory may help. I submitted to fix it. Does mean killing metrics-server?
/remove-sig node Sorry, I misread the log. The NPD found the metric server is OOM. It is about the metrics server, not the NPD. Sorry for disturb. I may add some memory for metric-server and check why metric server memory changed. I heard that metrics server recently released a new version. I will check if it is related.
/sig node
OOM metrics shows the metric server is OOMed. This is odd, that only metrics-server-nanny is with a memory limit. It uses whose command is . metrics-server container has no memory limit, but it got killed by OOM. Does this indicate that the node is OOM? (edited)
The last run failed in node-problem-detector ContainerStateTerminated. I0227 05:03:31. ] PodStartupLatency: perc50: 14m29.018886827s, perc90: 28m44.625518807s, perc99: 32m2.19303496s; threshold 1h0m0s I0227 05:03:32. ]


Question: "regexp" "strings" "syscall" <ins> "github.com/golang/glog" </ins> ) const MOUNT_MS_BIND = syscall.MS_BIND

Loaded restart count threshold overrides: map[coredns:2 fluentd-gcp:999 konnectivity-agent:2 kube-controller-manager:1 kube-proxy:2 kube-scheduler:1 l7-lb-controller:1 metadata-proxy:2 prometheus-to-sd-exporter:2] I0227 05:03:32. ] ClusterOOMsTracker: gathering cluster OOMs tracking measurement E0227 05:03:32. ] TestMetrics: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0] I0227 05:03:32. ] Step "[step: 29] gathering measurements" ended W0227 05:03:32. ] Got errors during step execution: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] I0227 05:03:32. ] Waiting for the chaos monkey subroutine to end... I0227 05:03:32. ] Chaos monkey ended. I0227 05:03:32. ] I0227 05:03:32. ] Stopping DnsLookupLatency probe... I0227 05:03:47. ] Stopping InClusterNetworkLatency probe... I0227 05:04:22. ] Resources cleanup time: 50.539700215s E0227 05:04:22. ]
E0227 05:04:22. ] Test Finished E0227 05:04:22. ] Test: E0227 05:04:22. ] Status: Fail E0227 05:04:22. ]


Question: "regexp" "strings" "syscall" <ins> "github.com/golang/glog" </ins> ) const MOUNT_MS_BIND = syscall.MS_BIND

Errors: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] E0227 05:04:22. ]
I0227 05:04:22. ]
I0227 05:04:22. ] Running testing/huge- I0227 05:04:22. ]


Question: { cfg: &kubeadmapi.MasterConfiguration{}, expected: api.Volume{ <del> Name: "pki", </del> <ins> Name: "k8s", </ins> VolumeSource: api.VolumeSource{ HostPath: &api.HostPathVolumeSource{ Path: kubeadmapi.GlobalEnvParams.KubernetesDir},

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? controller /etc/ssl/certs kubeadm /etc/ssl/certs Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 5 custom built master I also use a custom built hyperkube image instead of the single app images for apiserver, controller, ... Environment: Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): CentOS 7.2 Kernel (e.g. ): Linux ma-kub8ms0 3.10.0-327.36.3.el7.x8664 SMP Mon Oct 24 16:09:20 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: deployed with Azure RM template and ansible What happened: controller is not able to do https, including calls to the Azure RM API What you expected to happen: https should work as usual How to reproduce it (as minimally and precisely as possible): Use kubeadm to install kubernetes on CentOS exec into the controller and try "curl " Anything else do we need to know: It looks like caused this issue. On CentOS, I see these files on the host OS: I suspect the symlinks to /etc/pki/... being the problem here, as they are not available inside the hyperkube Debian image. I also tried to verify this with It seems to not be based on Debian and thus gives a different error: I see 2 possible solutions: of host mounting /etc/ssl/certs into the controller pod, the ca certificates could be installed into the image.


Question: { cfg: &kubeadmapi.MasterConfiguration{}, expected: api.Volume{ <del> Name: "pki", </del> <ins> Name: "k8s", </ins> VolumeSource: api.VolumeSource{ HostPath: &api.HostPathVolumeSource{ Path: kubeadmapi.GlobalEnvParams.KubernetesDir},

This is already done for hyperkube and would work out of the box when the host mount would be omitted. could try to figure out if /etc/pki should also be host mounted and add this to the controller manifest if required. I'd be happy to provide a fix by myself, I just need to know which solution you would prefer.
Same problem seems to apply to the apiserver. Not sure however if it actually does https calls.
implements solution 2.
This issue was moved to kubernetes/kubeadm


Question: go_test( name = "go_default_test", srcs = [ <ins> "labels_test.go", </ins> "pod_update_test.go", "types_test.go", ],

This is a sub issue for
cc can you please assign this to me? Thanks.
, please create PR directly and cc to Tim for review.
/sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: // corresponding requests. func newCacheRoundTripper(cacheDir string, rt http.RoundTripper) http.RoundTripper { d := diskv.New(diskv.Options{ <ins> PathPerm: os.FileMode(0750), FilePerm: os.FileMode(0660), </ins> BasePath: cacheDir, TempDir:  filepath.Join(cacheDir, ".diskv-temp"), })

In kubectl v1.8.0+, schema info is cached in the location specified by (defaulting to ), written with world-writeable permissions (rw-rw-rw-). If is specified and pointed at a different location accessible to other users/groups, the written files may be modified by other users/groups and disrupt the kubectl invocation. CVSS score: CVSS:3.0/AV:L/AC:H/PR:L/UI:R/S:U/C:L/I:L/A:N (3.3, low) What versions are affected? kubectl v1.8.0+ What configurations are affected? Invocations that point at world-writeable locations Impact Malformed responses written to the cache directory can disrupt the kubectl invocation Workaround Use the default location in the $HOME directory or point it at a directory that is only accessible to desired users/groups. (original description follows) ==== What happened: The files inside of ".kube/http-cache" are world writeable (rw-rw-rw-). While the default for these files appears to be the home directory, using the "--cache-dir" flag could put these files into a place where world writeable files would allow any user / process to modify the cache files. Modification of the cache files could influence the kubectl utility in a negative way for other users. What you expected to happen: Apply stricter file permissions to the http-cache files. How to reproduce it (as minimally and precisely as possible): Run any generic kubectl command which is successful and then list the cache directory $ kubectl get pods --all-namespaces $ ls -la Anything else we need to know?


Question: // corresponding requests. func newCacheRoundTripper(cacheDir string, rt http.RoundTripper) http.RoundTripper { d := diskv.New(diskv.Options{ <ins> PathPerm: os.FileMode(0750), FilePerm: os.FileMode(0660), </ins> BasePath: cacheDir, TempDir:  filepath.Join(cacheDir, ".diskv-temp"), })

: I estimate this is a low severity security issue with a CVSS score of "3.3 / CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:L/A:N" - Environment: Linux Kubernetes version (use kubectl version):Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS. Running kube api server in hyperkube. OS (e.g: cat /etc/os-release): NAME="CentOS Linux" VERSION="7.1808 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="7" PRETTYNAME="CentOS Linux 7.1808 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:7" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-7" CENTOSMANTISBTPROJECTVERSION="7" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="7" OSTREEVERSION=7.1808 Kernel (e.g. uname -a): Linux hackit.internal 3.10.0-862.11.6.el7.x8664 SMP Tue Aug 14 21:49:04 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Manual installation. Others: n/a
/sig product-security
Closing this issue. <detailsIn response to : if you think this is serious, send an email thanks!


Question: // corresponding requests. func newCacheRoundTripper(cacheDir string, rt http.RoundTripper) http.RoundTripper { d := diskv.New(diskv.Options{ <ins> PathPerm: os.FileMode(0750), FilePerm: os.FileMode(0660), </ins> BasePath: cacheDir, TempDir:  filepath.Join(cacheDir, ".diskv-temp"), })

</details
Adjusted the CVSS score to to indicate user interaction is required (since they must specify a non-default cache option pointed at a shared writeable location)
There are two places this should be done: PathPerm and FilePerm should be set here: the temp file and final file should have stricter permissions (probably 0660) here:
/cc
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9.
GitHub didn't allow me to assign the following users: yuchengwu. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
help fixing this is welcome, though it would only be fixed in releases 1.12+, per
/label official-cve-feed (Related to kubernetes/sig-security)


Question: // Create a usage object that is based on the quota resource version that will handle updates // by default, we preserve the past usage observation, and set hard to the current spec <del> usage := v1.ResourceQuota{ ObjectMeta: metav1.ObjectMeta{ Name:            resourceQuota.Name, Namespace:       resourceQuota.Namespace, ResourceVersion: resourceQuota.ResourceVersion, Labels:          resourceQuota.Labels, Annotations:     resourceQuota.Annotations}, Status: v1.ResourceQuotaStatus{ Hard: hardLimits, Used: used, }, </del> <ins> usage := resourceQuota.DeepCopy() usage.Status = v1.ResourceQuotaStatus{ Hard: hardLimits, Used: used, </ins> } dirty = dirty || !quota.Equals(usage.Status.Used, resourceQuota.Status.Used) // there was a change observed by this controller that requires we update quota if dirty { <del> _, err = rq.rqClient.ResourceQuotas(usage.Namespace).UpdateStatus(&usage) </del> <ins> _, err = rq.rqClient.ResourceQuotas(usage.Namespace).UpdateStatus(usage) </ins> return err } return nil

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The section for my was stripped, but left for all other resource types What you expected to happen: should put my changes into the cluster as requested How to reproduce it (as minimally and precisely as possible): Here are the resources (Note you will need to change the ownerReference UID and name to some object that exists in your cluster) Observe applying misses out the for the : Observe doing this again updates it correctly: Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: minikube
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Turns out, shutting down stops this from happening. After creating the object, it has the right ownerReference. On starting , it immediately removed the
/assign
fixed in
That was impressively fast! Thanks for the fix :)
cc


Question: attachdetachMutex.LockKey(string(nodeName)) defer attachdetachMutex.UnlockKey(string(nodeName)) <del> if err := detacher.vsphereVolumes.DetachDisk(volPath, nodeName); err != nil { </del> <ins> if err := detacher.vsphereVolumes.DetachDisk(newVolumePath, nodeName); err != nil { </ins> klog.Errorf("Error detaching volume %q: %v", volPath, err) return err }

When user creates a static PV using folder name in path: on vSAN backed datastore then detach disk does not work because check fails - The code expects folder name to be UUID but it does not and hence it determines disk is not attached to the node and hence no detach is ever performed. /sig storage cc


Question: registerMetrics.Do(func() { legacyregistry.MustRegister(etcdRequestLatency) legacyregistry.MustRegister(objectCounts) <ins> legacyregistry.MustRegister(etcdObjectCounts) </ins> legacyregistry.MustRegister(dbTotalSize) legacyregistry.MustRegister(etcdBookmarkCounts) legacyregistry.MustRegister(etcdLeaseObjectCounts) }) } <del> // UpdateObjectCount sets the etcd_object_counts metric. </del> <ins> // UpdateObjectCount sets the apiserver_storage_object_counts and etcd_object_counts (deprecated) metric. </ins> func UpdateObjectCount(resourcePrefix string, count int64) { objectCounts.WithLabelValues(resourcePrefix).Set(float64(count)) <ins> etcdObjectCounts.WithLabelValues(resourcePrefix).Set(float64(count)) </ins> } // RecordEtcdRequestLatency sets the etcd_request_duration_seconds metrics.

What would you like to be : We want to start migrating metrics which people depend on for operations to stable status. etcdobjectcounts was mentioned in a recent SIG meeting as a viable candidate. However, the 'etcd' prefix is not in line with stated metric conventions. SIG consensus determined that creating a new version of the metric with the metric name 'apiserverstorageobjectcounts' would be sufficient allow this metric to be promoted. We'd want to simultaneously deprecate 'etcdobjectcounts' (since it would be a duplicate) and remove it after the deprecation grace period window. /assign /cc
GitHub didn't allow me to assign the following users: erain. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : What would you like to be : /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig instrumentation api-machinery /cc
Ack. I am going to work on this.
/assign /triage accepted
In the SIG api-machinery meeting today, we agreed that the stable version of this metric should be named 'apiserverstorageobject_count'
Thanks and will use in the PR. (And sorry for missing today's meeting - due to a conflict)


Question: return len(c1map) == 0 } <ins> func shouldIgnoreNodeUpdate(oldNode, curNode v1.Node) bool { if !nodeInSameCondition(oldNode.Status.Conditions, curNode.Status.Conditions) { return false } oldNode.ResourceVersion = curNode.ResourceVersion oldNode.Status.Conditions = curNode.Status.Conditions return apiequality.Semantic.DeepEqual(oldNode, curNode) } </ins> func (dsc *DaemonSetsController) updateNode(old, cur interface{}) { oldNode := old.(*v1.Node) curNode := cur.(*v1.Node) <del> if reflect.DeepEqual(oldNode.Labels, curNode.Labels) && reflect.DeepEqual(oldNode.Spec.Taints, curNode.Spec.Taints) && nodeInSameCondition(oldNode.Status.Conditions, curNode.Status.Conditions) { // If node labels, taints and condition didn't change, we can ignore this update. </del> <ins> if shouldIgnoreNodeUpdate(*oldNode, *curNode) { </ins> return }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: a cluster, And node1 with resource 1 DaemonSet, request resource 4 run , There is no pod created. resource to the node: run , There is still no pod created. What you expected to happen: In step 5, There is one pod created. How to reproduce it (as minimally and precisely as possible): see above description Anything else we need to know?: Environment: Kubernetes version (use ):1.10.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release):CentOS7 Kernel (e.g. ): Install tools: Others: /sig app
xref
Thanks! is not work here! Because: DaemonSetsController didn't watch the changes.(we use to updateNode.) we restart kubelet, The DaemonSetsController watch the changes, but it seems that the node info is not updated, and still failed to create pod:
I found it is a bug here, ignore the changes. I made a fix , PTAL! Thanks!
Any workarounds for old cluster versions? Up: looks like in my case helped but after some delay.
, which version of k8s are you using? If it's sunset, you have to merge the pr yourself :). I think restart kubelet trigger delete/add to make sure it sync .
Sorry, I didn't say
I think we still support 1.12 for now :) let's cherry pick it if not and wait for next 1.12 release.


Question: timeout := time.Duration(3 * time.Second) ctx, cancel := context.WithTimeout(context.Background(), timeout) defer cancel() <del> scheme := runtime.NewScheme() </del> <ins> scheme := fake.NewTestScheme() </ins> metav1.AddMetaToScheme(scheme) informerReciveObjectCh := make(chan *metav1.PartialObjectMetadata, 1) objs := []runtime.Object{}

Because of this scheme mutation, this leads to concurrent use of the same scheme to cause the program to panic. It would be ideal to do this in some concurrent safe manner or add a warning or similar
/sig api-machinery
/assign
/assign /triage accepted