Question: masqueradeValue := 1 << uint(masqueradeBit) masqueradeMark := fmt.Sprintf("%#08x/%#08x", masqueradeValue, masqueradeValue) <del> if nodeIP == nil { klog.Warning("invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP") nodeIP = net.ParseIP("127.0.0.1") } </del> if len(clusterCIDR) == 0 { klog.Warning("clusterCIDR not specified, unable to distinguish between internal and external traffic") } else if utilnet.IsIPv6CIDRString(clusterCIDR) != ipt.IsIpv6() {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using Kubernetes 1.16.1, I started kube-proxy in iptables mode as a static pod on a master node. The master node was intentionally (and permanently) not available in the API as a Node object. kube-proxy started up and terminated soon after because it failed to (which is supposed to function as the bound address) and eventually returned an error: What you expected to happen: I expected the same behavior as in Kubernetes 1.15 where failure to look up the Node in the API would result in . How to reproduce it (as minimally and precisely as possible): Configure kube-proxy with iptables mode on Kubernetes 1.16 without the hosting Node being available in the Kubernetes API (under the OS hostname). Anything else we need to know?: PR (Fix for kube-proxy to wait for some duration for the node to be defined) introduced an additional . I believe this is the part that is causing the change in behavior: in Kubernetes 1.15, the check would be omitted and cause to set the default bind address to 127.0.0.1 (when iptables mode is set; I haven't checked other modes). The PR's primary purpose seems to be a different one (retrying node IP retrieval from the API), so my understanding is that the check was not really needed; unless I'm misunderstanding things which I don't want to rule out at all. The quick workaround was to explicitly configure kube-proxy with . However, I'm not entirely sure if that's a good solution long-term compared to letting Kubernetes do The Right Thing, which is what I'd like to clarify through this issue. Environment: Kubernetes version (use ): v1.16.1 Cloud provider or hardware configuration: Kubernetes running on DigitalOcean VMs (droplets) OS (e.g: ): Debian GNU/Linux 9 (stretch) Kernel (e.g.


Question: masqueradeValue := 1 << uint(masqueradeBit) masqueradeMark := fmt.Sprintf("%#08x/%#08x", masqueradeValue, masqueradeValue) <del> if nodeIP == nil { klog.Warning("invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP") nodeIP = net.ParseIP("127.0.0.1") } </del> if len(clusterCIDR) == 0 { klog.Warning("clusterCIDR not specified, unable to distinguish between internal and external traffic") } else if utilnet.IsIPv6CIDRString(clusterCIDR) != ipt.IsIpv6() {

): Linux master-k8s-1-16-test1 4.19.0-0.bpo.6-amd64 SMP Debian 4.19.67-2+deb10u1~bpo9+1 (2019-09-30) x86_64 GNU/Linux /sig network /cc who were involved in the referenced PR.
I made a similar comment in , wondering if this is a breaking change in v1.16 that we should revert? Previously if is nil, kube-proxy defaults to but now kube-proxy always exits with error instead of defaulting to local host. I think a lot of external cloud providers relied on this behavior because they needed to talk to API server prior to node addresses being set by the provider. Thoughts /assign
Opened to preserve behavior before v1.16 where node IP defaults to 127.0.01 is none is found. PTAL


Question: "path" "time" <ins> "github.com/onsi/ginkgo" "github.com/onsi/gomega" </ins> v1 "k8s.io/api/core/v1" <ins> rbacv1 "k8s.io/api/rbac/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels"

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Last successful run (23 Oct 2020, 10:07 PDT): First unsuccessful run (23 Oct 2020, 12:08 PDT): Testgrid link: Reason for failure: Anything else we need to know: presubmit job in repository also started to fail in the same fashion. Attempted to debug it myself, found out the following stuff: last successful run was accessing the apiserver via HTTP: all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail: perused the history of to find out that the PR is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests): first, it coincides with the start of the failures of , second, we changed the default hostname of the apiserver in node tests in that PR. Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a flag to the of these jobs. /assign /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Opened for a mitigation that would make those jobs stop failing.
This won't work, unfortunately. Copying response:
I have filed to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
Unfortunately I don't :/. My PR was also a wild guess. could you take a look at the proposed fix? Thanks once again for your help - really appreciated :).
the submission of hasn't fixed the problems, unfortunately :/. Sample failing run: We still have some peculiar problem related to health checking: and now NPD panics, too:
Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. /reopen
Reopened this issue.


Question: "path" "time" <ins> "github.com/onsi/ginkgo" "github.com/onsi/gomega" </ins> v1 "k8s.io/api/core/v1" <ins> rbacv1 "k8s.io/api/rbac/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels"

<detailsIn response to : Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
filed . I hope we could completely address this issue this time.
/milestone v1.20


Question: klog.Infof("Request Headers:") for key, values := range reqInfo.RequestHeaders { for _, value := range values { <ins> value = maskValue(key, value) </ins> klog.Infof("    %s: %s", key, value) } }

This issue was reported in the Description Kubernetes requires an authentication mechanism to enforce users’ privileges. One method of authentication, bearer tokens, are opaque strings used to associate a user with their having successfully authenticated previously. Any user with possession of this token may masquerade as the original user (the “bearer”) without further authentication. Within Kubernetes, the bearer token is captured within the hyperkube kube-apiserver system logs at high verbosity levels (--v 10). A malicious user with access to the system logs on such a system could masquerade as any user who has previously logged into the system. Exploit Scenario Alice logs into a Kubernetes cluster and is issued a Bearer token. The system logs her token. Eve, who has access to the logs but not the production Kubernetes cluster, replays Alice’s Bearer token, and can masquerade as Alice to the cluster. Recommendation Short term, remove the Bearer token from the log. Do not log any authentication credentials within the system, including tokens, private keys, or passwords that may be used to authenticate to the production Kubernetes cluster, regardless of the logging level. Long term, either implement policies that enforce code review to ensure that sensitive data is not exposed in logs, or implement logging filters that check for sensitive data and remove it prior to outputting the log. In either case, ensure that sensitive data cannot be trivially stored in logs. Anything else we need to know?: See for current status of all issues created from these findings. The vendor gave this issue an ID of TOB-K8S-001 and it was finding 6 of the report. The vendor considers this issue Medium Severity. To view the original finding, begin on page 31 of the Environment: Kubernetes version: 1.13.4
/sig auth /sig api-machinery
/remove-kind feature /kind bug
Should remote the "Authorization" header here?
for authorization headers, everything after the type should be masked (and potentially, only recognized auth types should be logged). Something like:
I think people would be unpleasantly surprised to see credentials in a logfile after turning up logging to debug an issue. /priority important-soon
Does either of you intend to createe PR to remove the "Authorization" header ?


Question: klog.Infof("Request Headers:") for key, values := range reqInfo.RequestHeaders { for _, value := range values { <ins> value = maskValue(key, value) </ins> klog.Infof("    %s: %s", key, value) } }

This issue seems CVE-worthy to me: loss of confidentiality (bearer tokens) + cross of privilege boundary (debug log level could be set by admin, then logs read later by low privileged user). My draft cvss would be: CVSS:3.0/AV:N/AC:H/PR:L/UI:N/S:U/C:H/I:N/A:N (5.3)
Two potential vectors: local user with access to system logs (local access, low privilege) (4.7) remote user with access to system logs (network access, high privilege): (4.4) CVE-2019- has been assigned to this issue.
If you want to, go ahead
- should this have the official CVE feed label ( retrospectively, I mean)?
probably? I have no idea how that label is supposed to get applied, but there appear to be several CVE issues that don't have it -
I think you're likely to know who to highlight this to. Sorry to mention you individually; please feel free to point out docs if there are any.
/label official-cve-feed thanks for the tag, the label now!
thanks for the useful filter. Currently in-scope issues are issues that were closed as fixed by code changes to k/k. So with that in mind, I have the labels to some of the closed issues too that conform to the scope defined here:
I would like to reply nearly four years later: There are valid cases to want to see the token, for example my current case, trying to debug token validation between the API server and other services.


Question: return intr.Run(func() error { _, err := watch.Until(0, w, func(e watch.Event) (bool, error) { // print deployment's status <del> status, done, err := statusViewer.Status(cmdNamespace, info.Name, revision) </del> <ins> status, done, err := statusViewer.Status(info.Namespace, info.Name, revision) </ins> if err != nil { return false, err }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): rollout status namespace Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Macbook Pro 2015 OS (e.g. from /etc/os-release): Mac OSX Sierra Kernel (e.g. ): Install tools: Others: What happened: , where does the following: Describes a Deployment object Sets a non-default namespace in the metadata . An error message says the resource was not found . The response shows the status of the rollout. What you expected to happen: I expected to work as well by using the namespace that is set inside for the rollout status request. How to reproduce it (as minimally and precisely as possible): the following file as (taken from the but updated to add the namespace): Returns Returns Anything else we need to know: We've seen this issue on 1.5.1 as well, and also on Ubuntu. The recreate instructions above were performed on a fresh minikube setup.
I also noticed this today. The node is using version 1.5.2 and we are using latest gcloud SDK components (v146). At the moment I have to use
This issue also exists on kubectl built from HEAD.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: return intr.Run(func() error { _, err := watch.Until(0, w, func(e watch.Event) (bool, error) { // print deployment's status <del> status, done, err := statusViewer.Status(cmdNamespace, info.Name, revision) </del> <ins> status, done, err := statusViewer.Status(info.Namespace, info.Name, revision) </ins> if err != nil { return false, err }

Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: "sync" "time" <ins> cadvisormemory "github.com/google/cadvisor/cache/memory" </ins> cadvisorfs "github.com/google/cadvisor/fs" cadvisorapiv2 "github.com/google/cadvisor/info/v2" "google.golang.org/grpc/codes"

What happened: New overlay filesystems created for Pods after kubelet start trigger repeated error messages. For example: Error messages are logged every 10 seconds for every filesystem created after kubelet start. For large kubelet nodes, this issue creates an excessive amount of error logs. What you expected to happen: New overlay filesystems created for Pods after kubelet start should not log error messages. How to reproduce it (as minimally and precisely as possible): Use cristatsprovider with cri-o and configure overlay storage. UsingLegacyCadvisorStats() needs to return false per for cristatsprovider to be used. Start with an empty /var/lib/containers/storage and schedule Pods on the node. Anything else we need to know?: As part of , cadvisor.GetFsInfoByFsUUID() was changed to cadvisor.GetDirFsInfo() in getFsInfo(); however, the test for cadvisorfs.ErrNoSuchDevice was not modified, changing how getFsInfo() handles a cache miss from cadvisor. cadvisor caches UUIDs/partitions/mounts on kubelet start and the cache is never refreshed. Given that it's expected that getFsInfo() will fail to retrieve data from cadvisor (see comments in ImageFsStats() and ), this condition should not be logged as an error. Alternatively, cadvisor fs could be modified to refresh on cache miss enabling getFsInfo() to return data on these filesystems. Environment: Kubernetes version (use ): 1.18.6 CRI: CRI-O
/sig node cc
ptal
thanks for reporting the issue. I will take a look at in upcoming week.
/assign
Sorry this is taking slightly longer. Currently held up in something little urgent, but as soon as that's over I will look into this one.
Do you have the time to look at this issue ? I can help with testing / debugging if required. Environment Kubernetes 1.19.3 Cri-o 1.19.0
Sorry I haven't gotten a chance to look into it yet, I would certainly appreciate any help in debugging. Thanks.
Right now I am thinking of either handling the cache miss better, or returning the even if the fs type is not in cadvisor, or both.
Issues go stale after 90d of inactivity. Mark the issue as fresh with .


Question: "sync" "time" <ins> cadvisormemory "github.com/google/cadvisor/cache/memory" </ins> cadvisorfs "github.com/google/cadvisor/fs" cadvisorapiv2 "github.com/google/cadvisor/info/v2" "google.golang.org/grpc/codes"

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
kindly ping - has this already been implemented ?
/assign
Fix for k/k in and cadvisor in
/remove-lifecycle rotten
hi do you know when the fixes will be released ? Thanks!
I don't, unfortunately.
the fix has made it into , so I imagine the change will come in by 1.22
/triage accepted /priority important-soon
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
So v1.22 is out for a while now. Is there someone who has already upgraded to v1.22 and can confirm that the getFsInfo errors are fixed?
haircommander mentioned that the fix made it into cadvisor 0.40.0-rc.1. But cadvisor version bump did not happen in k8s yet, probably due to issues mentioned here: iirc kind did not work with image kindest/node v1.22.1, I did not look into logs however. Will update post once I have access to the affected device. Edit:
I still have these logs with version 1.22.2
I have got the same error log about kubelet kubernetes version 1.22.2 containerd: 1.4.11 Is this problem have been resolve?


Question: "sync" "time" <ins> cadvisormemory "github.com/google/cadvisor/cache/memory" </ins> cadvisorfs "github.com/google/cadvisor/fs" cadvisorapiv2 "github.com/google/cadvisor/info/v2" "google.golang.org/grpc/codes"

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
I still have these logs with version 1.23.1
turns out we ended up being blocked on , so the fix has yet to be merged. let's push for it in 1.24
I'll wait for 1.24. With 1.22.5 My cluster has some of these messages but is stable. But with 1.23.x I get disk pressure because my sys logs get spammed quickly. I hoped going away fomr cadvisor to CRI stats woudl help, but instead of stat errors I now have these FS errors.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.


Question: "sync" "time" <ins> cadvisormemory "github.com/google/cadvisor/cache/memory" </ins> cadvisorfs "github.com/google/cadvisor/fs" cadvisorapiv2 "github.com/google/cadvisor/info/v2" "google.golang.org/grpc/codes"

</details
This is still relevant and requires a fix with
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale Related PR still not merged.
SnoozeThis is useful if we don't want to keep polling until that PR is closed.
() I will wait until is merged and then add a comment.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
Hi folks, I still have this issue in most recent Kubernetes 1.24.6. cAdvisor 0.44.0 that fix went into that version, is now a long time ago already in 1.24.0. Dep bump to runc 1.1.0, cadvisor 0.44.0 Automated cherry pick of : Bump cAdvisor to v0.44.1 28.4.2022 So is there possibly any chance to progress with a review of this PR please?
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: "sync" "time" <ins> cadvisormemory "github.com/google/cadvisor/cache/memory" </ins> cadvisorfs "github.com/google/cadvisor/fs" cadvisorapiv2 "github.com/google/cadvisor/info/v2" "google.golang.org/grpc/codes"

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle stale
Can we have any news on that topic please?
/remove-lifecycle rotten
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
Resolved.


Question: nt.zones = append(nt.zones, zone) nt.tree[zone] = &nodeArray{nodes: []string{n.Name}, lastIndex: 0} } <del> klog.V(5).Infof("Added node %v in group %v to NodeTree", n.Name, zone) </del> <ins> klog.V(2).Infof("Added node %q in group %q to NodeTree", n.Name, zone) </ins> nt.numNodes++ }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) Increase the log level of info logs to "v=2" when the scheduler or a node from its cache. 2) Increase the log level of info logs to "v=2" when the scheduler can't schedule a pod , , and add a log message for 2) report memory/cpu status of . Why is this needed: With cluster autoscaler now being used in production by some vendors, we want to have more visibility to events that could be related to the auto scaler to help debug customer issues. /sig scheduling /priority important-soon /assign
GitHub didn't allow me to assign the following users: congliu01. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : <!-- Please only use this template for submitting enhancement requests --1) Increase the log level of info logs to "v=2" when the scheduler or a node from its cache. 2) report memory/cpu status of . Why is this needed: With cluster autoscaler now being used in production by some vendors, we want to have more visibility to events that could be related to the auto scaler to help debug customer issues. /priority important-soon /assign </details
will consider this assigned to you.
/assign


Question: const pluginName = "kubernetes.io/flocker" <del> func newInitializedVolumePlugMgr() volume.VolumePluginMgr { </del> <ins> func newInitializedVolumePlugMgr(t *testing.T) (volume.VolumePluginMgr, string) { </ins> plugMgr := volume.VolumePluginMgr{} <del> plugMgr.InitPlugins(ProbeVolumePlugins(), volume.NewFakeVolumeHost("/foo/bar", nil, nil)) return plugMgr </del> <ins> dir, err := ioutil.TempDir("", "flocker") assert.NoError(t, err) plugMgr.InitPlugins(ProbeVolumePlugins(), volume.NewFakeVolumeHost(dir, nil, nil)) return plugMgr, dir </ins> } func TestGetByName(t *testing.T) { assert := assert.New(t) <del> plugMgr := newInitializedVolumePlugMgr() </del> <ins> plugMgr, _ := newInitializedVolumePlugMgr(t) </ins> plug, err := plugMgr.FindPluginByName(pluginName) assert.NotNil(plug, "Can't find the plugin by name")

Unit tests on our CI have apparently been failing for 2 weeks and no one noticed (oops!)... The PR that seems to have caused the failures is . The failing test is - . Full Test Log: I'm not sure why this is failing on our CI and not on shippable. Has anyone else seen something similar? That test only seems to fail on the second run through with . Our unit tests run in a container built specifically for testing k8s-mesos: The Dockerfile for the env: Is anyone else able to repro with the above env or their own?
/cc
that test was around a week ago, it's doesn't seem possible to me that it was failing since 2 weeks ago. Anyway, we have few duplications of this issue: this one And the PR where is working on: It's a little bit difficult to me to follow all the conversations in all the differente places about the same bug, could it be possible to unificate some? Since you provided the environment I will try and take a look to reproduce it locally. Thanks!
I think I've got a working patch in
Closing in favor of &


Question: const pluginName = "kubernetes.io/flocker" <del> func newInitializedVolumePlugMgr() volume.VolumePluginMgr { </del> <ins> func newInitializedVolumePlugMgr(t *testing.T) (volume.VolumePluginMgr, string) { </ins> plugMgr := volume.VolumePluginMgr{} <del> plugMgr.InitPlugins(ProbeVolumePlugins(), volume.NewFakeVolumeHost("/foo/bar", nil, nil)) return plugMgr </del> <ins> dir, err := ioutil.TempDir("", "flocker") assert.NoError(t, err) plugMgr.InitPlugins(ProbeVolumePlugins(), volume.NewFakeVolumeHost(dir, nil, nil)) return plugMgr, dir </ins> } func TestGetByName(t *testing.T) { assert := assert.New(t) <del> plugMgr := newInitializedVolumePlugMgr() </del> <ins> plugMgr, _ := newInitializedVolumePlugMgr(t) </ins> plug, err := plugMgr.FindPluginByName(pluginName) assert.NotNil(plug, "Can't find the plugin by name")

reports the following failure after syncing with latest code: --- FAIL: TestSetUpAtInternal (0.00s) Error Trace: Error: Not equal: "expected-to-be-set-properly" (expected) != "" (actual) FAIL FAIL 0.067s cc/
CC
Taking a look. What do you mean with failed sometimes? Is it a random failure?
Can you please provide more information. This test seems to be consistently green in our CI runs.
The test is failing on this assertion: but as you can see on the previous line I am checking that there was no error on that call. So the function is returning for the error checking but it's not setting . Only place where it can happen is here: Is the not been clean between runs?
When I said "the only place where it can happen" it was because the other s are being returned after a check. I think that there is no other way to get there.
PS: I run the test suite 100 times and I didn't get any error: (That branch is based on the branch)
can you test to see if the patch in fixes things for you?
See for reproducible test environment and failure logs.
Based on the solution in it sounds like this test creates files the first time through which causes the tests to fail on the second time through.


Question: sched := scheduler.NewFromConfig(schedulerConfig) // Prepare the event broadcaster. <del> if c.Broadcaster != nil && c.EventClient != nil { c.Broadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: c.EventClient.Events("")}) </del> <ins> if cc.Broadcaster != nil && cc.EventClient != nil { cc.Broadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: cc.EventClient.Events("")}) </ins> } // Start up the healthz server. <del> if c.InsecureServing != nil { separateMetrics := c.InsecureMetricsServing != nil handler := buildHandlerChain(newHealthzHandler(&c.ComponentConfig, separateMetrics), nil, nil) if err := c.InsecureServing.Serve(handler, 0, stopCh); err != nil { </del> <ins> if cc.InsecureServing != nil { separateMetrics := cc.InsecureMetricsServing != nil handler := buildHandlerChain(newHealthzHandler(&cc.ComponentConfig, separateMetrics), nil, nil) if err := cc.InsecureServing.Serve(handler, 0, stopCh); err != nil { </ins> return fmt.Errorf("failed to start healthz server: %v", err) } } <del> if c.InsecureMetricsServing != nil { handler := buildHandlerChain(newMetricsHandler(&c.ComponentConfig), nil, nil) if err := c.InsecureMetricsServing.Serve(handler, 0, stopCh); err != nil { </del> <ins> if cc.InsecureMetricsServing != nil { handler := buildHandlerChain(newMetricsHandler(&cc.ComponentConfig), nil, nil) if err := cc.InsecureMetricsServing.Serve(handler, 0, stopCh); err != nil { </ins> return fmt.Errorf("failed to start metrics server: %v", err) } } <del> if c.SecureServing != nil { handler := buildHandlerChain(newHealthzHandler(&c.ComponentConfig, false), c.Authentication.Authenticator, c.Authorization.Authorizer) if err := c.SecureServing.Serve(handler, 0, stopCh); err != nil { </del> <ins> if cc.SecureServing != nil { handler := buildHandlerChain(newHealthzHandler(&cc.ComponentConfig, false), cc.Authentication.Authenticator, cc.Authorization.Authorizer) if err := cc.SecureServing.Serve(handler, 0, stopCh); err != nil { </ins> // fail early for secure handlers, removing the old error loop from above return fmt.Errorf("failed to start healthz server: %v", err) } } // Start all informers. <del> go c.PodInformer.Informer().Run(stopCh) c.InformerFactory.Start(stopCh) </del> <ins> go cc.PodInformer.Informer().Run(stopCh) cc.InformerFactory.Start(stopCh) </ins> // Wait for all caches to sync before scheduling. <del> c.InformerFactory.WaitForCacheSync(stopCh) controller.WaitForCacheSync("scheduler", stopCh, c.PodInformer.Informer().HasSynced) </del> <ins> cc.InformerFactory.WaitForCacheSync(stopCh) controller.WaitForCacheSync("scheduler", stopCh, cc.PodInformer.Informer().HasSynced) </ins> // Prepare a reusable run function. run := func(ctx context.Context) {

PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. Main tracking issue: In the function creates a object and populates the field with an anonymous function. At the end of that function, is called, which contains the rest of our startup logic. Rather than split this across two functions, we should put all the startup logic in one named function called (not ) so that constructing the object is much more plain. /sig scheduling /kind cleanup /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. Main tracking issue: Rather than split this across two functions, we should put all the startup logic in one named function called (not ) so that constructing the object is much more plain. /kind cleanup /help /good-first-issue </details
I'll pick this up! /assign


Question: go aggregatedManager.Run(testCtx.Done()) <del> require.True(t, waitForEmptyQueue(testCtx.Done(), aggregatedManager)) </del> <ins> require.True(t, waitForQueueComplete(testCtx.Done(), aggregatedManager)) </ins> response, _, parsed := fetchPath(aggregatedResourceManager, "") if response.StatusCode != 200 {

pull-kubernetes-unit sometimes fails with data race. TestDirty I think the test was 3 months ago with the flaking state. There are two issues. First, updating and reading a bool-type variable is not thread-safe. and there is also another problem checking the state before completely syncing it. No response /sig api-machinery
/assign
/triage accepted


Question: // Enables the AWS EBS in-tree driver to AWS EBS CSI Driver migration feature. CSIMigrationAWS featuregate.Feature = "CSIMigrationAWS" <ins> // owner: @andyzhangx // alpha: v1.15 // // Enables the Azure Disk in-tree driver to Azure Disk Driver migration feature. CSIMigrationAzureDisk featuregate.Feature = "CSIMigrationAzureDisk" </ins> // owner: @RobertKrawitz // beta: v1.15 //

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Azure is not listed as "migratable" in could refer to Why is this needed:
/sig azure /assign


Question: // Check that this is a rule for the correct chain, and that it has // the correct number of argument (+2 for "-A <chain name>") <del> if !strings.HasPrefix(line, fmt.Sprintf("-A %s", string(chain))) || len(fields) != len(args)+2 { </del> <ins> if !strings.HasPrefix(line, fmt.Sprintf("-A %s", string(chain))) || len(fields) != len(argsCopy)+2 { </ins> continue } // Sadly, iptables has inconsistent quoting rules for comments. <del> // Just unquote any arg that is wrapped in quotes. </del> <ins> // Just remove all quotes. </ins> for i := range fields { <del> unquote(&fields[i]) </del> <ins> fields[i] = strings.Trim(fields[i], """) </ins> } // TODO: This misses reorderings e.g. "-x foo ! -y bar" will match "! -x foo -y bar"

I am running kubernetes 0.5.2 + and I've noticed that the number of entries in iptables increases every few seconds. I started with a clean install, no pods or replicationControllers, there is now two services by default (kubernetes-ro and kubernetes). I ran and got . Next, I turned up 7 services and 3 replicationControllers which resulted in 3 pods also being turned up. I started a watch... I saw "38", "43, "50, "64", "66", "68", "70", ..., "115" and on until I stopped . It appears the proxy continues to add REDIRECT's for the services, for example the lines that keep being look like: A similar line for each service gets every second or so, it appears to all be in the table. I'm thinking the newly function might be the culprit, since calls and in the log I see every few seconds (looks like does a sync every 5 seconds).
I noticed that the 0.5.3 and 0.5.4 releases don't include the , though it's in master. Perhaps the issue I've filed is already known and I missed it when searching the issues.
I have not heard this issue. Can you paste a listing of iptables-save -t nat when you are seeing this problem? Alternatively, I can cut you a branch with some extra debug instrumentation if you're willing to test it out today...
I'm not able to test it out today. As far as the iptables output, I'd have to type it in (can't copy/paste -- different physical network). However, the output has lines, like the one given in the issue, which are every 5 seconds to the "nat" table. It seems like the check to see if the rule already exists is perhaps not functioning properly. I am working on a fix for (because it uses the iptables -C flag which isn't supported in older iptables), in that fix I created a function to check if an existing rule already exists: Example call: Using may not be ideal, perhaps breaking up the output of the iptables command into a slice and checking each line would be better(?) sure yet.
Fix is pending - would be great if someone could verify it against an actually old iptables (I just faked the version string).


Question: // Check that this is a rule for the correct chain, and that it has // the correct number of argument (+2 for "-A <chain name>") <del> if !strings.HasPrefix(line, fmt.Sprintf("-A %s", string(chain))) || len(fields) != len(args)+2 { </del> <ins> if !strings.HasPrefix(line, fmt.Sprintf("-A %s", string(chain))) || len(fields) != len(argsCopy)+2 { </ins> continue } // Sadly, iptables has inconsistent quoting rules for comments. <del> // Just unquote any arg that is wrapped in quotes. </del> <ins> // Just remove all quotes. </ins> for i := range fields { <del> unquote(&fields[i]) </del> <ins> fields[i] = strings.Trim(fields[i], """) </ins> } // TODO: This misses reorderings e.g. "-x foo ! -y bar" will match "! -x foo -y bar"

should be able to try it out friday, maybe monday
Feel free to rip off our checks with out echeck function :) On Nov 26, 2014 7:36 PM, "Craig Wickesser" wrote:
that it actually works. On Nov 26, 2014 7:40 PM, "Tim Hockin" wrote:
I fired up kuberenetes 0.5.2 (plus + your changes) and things looked good initially (when starting up kubernetes, no pods/services, etc). The number of entries iptables-save remained constant. I then turned up some services and watched the length of iptables-save increase. I turned down the services and the number of entries in iptables-save continues to increase. At the moment, every 5 seconds it's back to adding two entries (one for the kubernetes service and one for the kubernetes-ro service). With log level set to 0, I see "Opened iptables portal for service.." messages in the proxy log every 5 seconds ().
Are you sure you have the right proxy version running? I could repro the initial problem, but not this - it works just fine. To be clear when you say it increases after creating a service - do you mean it increases by 1 rule or that it increases continually? , Craig Wickesser wrote:
On Monday/Tuesday I'll try the latest from master, perhaps I messed something up. To clarify, the increase was continuous. When I initially setup kubernetes (with a fresh/clean etcd), things looked normal with regards to iptables. I then deployed some pods and services and kept watching iptables-save. The number of entries increased every 5 seconds. I deleted pods, replication controllers and services and the number of entries continued to increase, but only by two (for the kubernetes and kubernetes-ro services) every 5 seconds.
Did you paste the output of iptables-save? Its possible that it is slightly different from my version. This is why parsing text is no fun :( On Nov 28, 2014 3:42 PM, "Craig Wickesser" wrote:
I did not include the output of iptables-save. I'll see if I can get additional output besides the line included in the initial issue.
Just a couple lines so I can see the format.


Question: // Check that this is a rule for the correct chain, and that it has // the correct number of argument (+2 for "-A <chain name>") <del> if !strings.HasPrefix(line, fmt.Sprintf("-A %s", string(chain))) || len(fields) != len(args)+2 { </del> <ins> if !strings.HasPrefix(line, fmt.Sprintf("-A %s", string(chain))) || len(fields) != len(argsCopy)+2 { </ins> continue } // Sadly, iptables has inconsistent quoting rules for comments. <del> // Just unquote any arg that is wrapped in quotes. </del> <ins> // Just remove all quotes. </ins> for i := range fields { <del> unquote(&fields[i]) </del> <ins> fields[i] = strings.Trim(fields[i], """) </ins> } // TODO: This misses reorderings e.g. "-x foo ! -y bar" will match "! -x foo -y bar"

On Nov 28, 2014 7:32 PM, "Craig Wickesser" wrote:
here's a few lines from : All of the lines follow the exact same format. Looks like the proxier (to fit with the output I provided) would need to look like this:
FWIW - I'm using CentOS 6.5 with iptables version 1.4.7.
I recompiled with changes to proxier as mentioned earlier (changed the order of args) and did not resolve the issue. I continue seeing (in the proxy log):
I've some additional logging to log the and the . I am logging them using these two lines: When I reduce the output between the and the (when it should return "true", that fields is a superset), the result looks like: Notice has quotes around it in the set, and the does not? I think perhaps that's the issue and that's why the check is false.
I just tried changing to: and that's working. Basically, strip out quotes from the line. There might be a better alternative.
Uggh, seriously? Your iptables add quotes around the comment, mine omits the quotes for single tokens. I guess I can force it to not be a single word, something like "service: " service name. What a hack. Any better ideas? , Craig Wickesser wrote:
Hmm, it seems worse than that. If I feed in a comment with quotes, I get ""foo"", so I can't do that to be compatible with your implementation. Can you try something? iptables -t nat -I PREROUTING -d 1.2.3.4 -m comment --comment "foo1" iptables -t nat -I PREROUTING -d 1.2.3.4 -m comment --comment '"foo2"' iptables-save | grep foo -A PREROUTING -d 1.2.3.4/32 -m comment --comment ""foo2"" -A PREROUTING -d 1.2.3.4/32 -m comment --comment foo1 , Tim Hockin wrote:
i'll try that in about 8 hours :)
here's what I got:
ugh, that is SO BROKEN. I'm going to have to do something really unpleasant.


Question: // Check that this is a rule for the correct chain, and that it has // the correct number of argument (+2 for "-A <chain name>") <del> if !strings.HasPrefix(line, fmt.Sprintf("-A %s", string(chain))) || len(fields) != len(args)+2 { </del> <ins> if !strings.HasPrefix(line, fmt.Sprintf("-A %s", string(chain))) || len(fields) != len(argsCopy)+2 { </ins> continue } // Sadly, iptables has inconsistent quoting rules for comments. <del> // Just unquote any arg that is wrapped in quotes. </del> <ins> // Just remove all quotes. </ins> for i := range fields { <del> unquote(&fields[i]) </del> <ins> fields[i] = strings.Trim(fields[i], """) </ins> } // TODO: This misses reorderings e.g. "-x foo ! -y bar" will match "! -x foo -y bar"

, Craig Wickesser wrote:
I have a potential fix, but airport wifi prevents me from pushing git <shrug. will publish it later.
sounds good, I appreciate the effort.
PR posted - please test before we commit.
I've tried the PR and a comment to it.
got the same problem here using Kubernetes v0.6.0-12-gd927031aecdeb6 on redhat 6.5
It looks like 0.6 was cut before - - can we patch that in, or should we wait of 0.7? , polariss0i wrote:
got the same problem on kubernetes v1.0.3 and iptables-v1.4.7. Should I file a new issue?


Question: var defaultReplicas int32 = 1 period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug The change in a new field () to the Pod spec that is set to true by default. This is causing problems for DaemonSets in some situations. DaemonSets are using ControllerRevisions to keep track of previous configurations and each revision will contain among other things the Pod spec for the DaemonSet. If a DaemonSet is created with the enableServiceLinks property is set to true, the ControllerRevision will also contain this field. If the cluster is rolled back to a previous version of kubernetes, the DeamonSet struct does not have this field and so the ControllerRevision and the DaemonSet will no longer match. This leads to a new ControllerRevision being created and a rolling update of all pods in the DaemonSet. Would it be a better solution to rename the field to and enable service links whenever the field is not set? This would fix the problem with the DaemonSets unless the field is explicitly set before the rollback. This is causing the master upgrade tests to fail as reported in /sig apps /cc /cc /cc
I guess the change in podSpec and the defaulting will break all the workload resources as they are based on comparing the template's podSpec. ~Is the new value defaulted to true also for the DaemonSet's podSpec template? If not, controller might go hot loop like with mutating admission controllers.~ I don't like us making such changes to v1 podSpec. The other way around seems more sensible. /cc
Why is this a bad thing? We will continue to add fields to podSpec - the issue isn't the defaulting (this would happen regardless of which way it defaulted), it's that the controller can't really be version aware. The new value would default to true in the daemonset as well.
I'm not sure that a rollback requiring daemonsets to recreate is a bug. I think that's potentially "the best possible thing that we can do in that case".
/cc SIG Release blocking dashboard with upgrade test failures - https://k8s-
So the problem isn't the fact that a field has been . It the way in which the field was .


Question: var defaultReplicas int32 = 1 period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either ca So the problem isn't the fact that a field has been . It the way in which the field was . enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either case), you won't trigger the rolling restart on rollback or controller manager failoverse), you won't trigger the rolling restart on rollback or controller manager failover unless you explicitly opt out of the behavior. I don't see why the former behavior is more desirable than the latter.
For background/context, the choice between enable/disable was initially discussed here:
Are you saying that the issue is that default value is not the same as the Go "zero value" ? The concept of zero value is interesting but it forces APIs to bend into unnatural shapes to respect it. Since forever Kubernetes has explicitly said that we have version-specific default values which ARE NOT governed by zero values, but instead are governed by the API version. It is safe to assume that if a field is documented as having a default in a version, you can rely on that field being populated with that default, with a few exceptions.


Question: var defaultReplicas int32 = 1 period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

We generally ask for boolean fields to be "in the affirmative direction", as in this case. Now, we could revisit this. It would be a significant change in direction, but it's not impossible. But I don't think that solves your problem in general -- if this field were optional string, with a default value of "Hello there", you would be in the same boat right?
So the underlying issue is that we are defaulting into the PodSpec which gets adopted into all of the PodTemplateSpecs of every object. That sort of defaults the users declared intent. Why not just default the value into the Pod itself . Is the intention really to default this in all Templates or to make it optional in the Template and default it in the Pod itself?
/milestone v1.13
I can't speak to the ideology of whether to default the field in all PodSpecs or just the Pod, but your suggestion makes sense to me. I'm not sure of the past reasoning for adding other defaulted fields, most recently SchedulerName in Jan. 2017 (), to all PodSpecs instead of just to the Pod object. Did we not have these same tests in place then? This would effectively be saying we should never add new defaulted fields to PodSpec. If this is the plan, I can create a PR for the fix and I'll add a big warning comment to SetDefaults_PodSpec.
As an example of where we don't default the template we have a CronJob that embeds JobTemplate, we never default that template, only an actual Job gets defaulting. I agree with that not defaulting any templates does make sense.
Agree to only default this in pods. Agree to adding the comment. Most important defaulting happens in pods to keep higher level pod specs simpler. We may need to raise the general pattern in a discussion with api review in general, since this is a change from previous strategy, but the previous strategy was not explicitly chosen (to my recollection at least)
Hi all do we have an agreed path forward for fixing this issue in 1.13? if so can we have an ETA for the fix to land? the release team would like for the tests to get passing in the blocking suites in an effort to unmask any other failures that might show up once these are fixed.


Question: var defaultReplicas int32 = 1 period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

/assign
We have a PR with a fix that will hopefully merge today or tomorrow. I will keep an eye on the dashboard to make sure it fixes the failing daemonset tests.
Thanks Can we link the PR here plz. /cc
Reopening, because of the auto-close. Tests are not passing yet. /reopen
Reopening this issue. <detailsIn response to : Reopening, because of the auto-close. Tests are not passing yet. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
PR that should (hopefully) close this issue is here:
DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- if it doesn't pop up again - /close
Closing this issue. <detailsIn response to : DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: writeLine(proxier.natRules, append(dstLocalOnlyArgs, "-j", "ACCEPT")...) } <del> // mark masq for KUBE-NODE-PORT writeLine(proxier.natRules, []string{ "-A", string(KubeNodePortChain), "-j", string(KubeMarkMasqChain), }...) </del> <ins> // -A KUBE-SERVICES  -m addrtype  --dst-type LOCAL -j KUBE-NODE-PORT args = append(args[:0], "-A", string(kubeServicesChain), "-m", "addrtype", "--dst-type", "LOCAL", ) writeLine(proxier.natRules, append(args, "-j", string(KubeNodePortChain))...) </ins> // mark drop for KUBE-LOAD-BALANCER writeLine(proxier.natRules, []string{

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /assign I'll make the pull request tomorrow. What happened: start kube-proxy in ipvs mode, create some NodePort services. Then look at the iptables rules. match-set of KUBE-NODE-PORT-TCP is inadequate to be qualified as NodePort traffic. "--dst-type LOCAL" is needed. What you expected to happen: since we have also KUBE-NODE-PORT-UDP, another way would be like this: How to reproduce it (as minimally and precisely as possible): start kube-proxy in ipvs mode, create some NodePort services. Then look at the iptables rules. Anything else we need to know?: Environment: Kubernetes version (use ): commit Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS Linux release 7.2.1511 (Core) Kernel (e.g. ): Linux VM34centos 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
/sig network /area kube-proxy /area ipvs /assign
/cc i remember this bug fixed, that's right?
, Thanks, seems like necessary. But you have any use case that ipvs proxier will have bugs in current mach rules?
It's already fixed by ? Can we close this issue now?
I don't see how fixed this issue. How about this scenario: k8s cluster A uses node port for service A k8s cluster B uses node port for service B pod (or host) in cluster A tries to access node port on k8s cluster B local node port rule matches the traffic and NATs it to the local node port, pod (or host) cannot reach cluster B's node port. Also, the remote node port could be replaced by any non-k8s host that happens to use port .
Then I think can solve this problem.


Question: CreateOption: "attach", }) } <del> vm.StorageProfile.DataDisks = &disks </del> <ins> newVM := compute.VirtualMachineScaleSetVM{ Sku:      vm.Sku, Location: vm.Location, VirtualMachineScaleSetVMProperties: &compute.VirtualMachineScaleSetVMProperties{ HardwareProfile: vm.HardwareProfile, StorageProfile: &compute.StorageProfile{ OsDisk:    vm.StorageProfile.OsDisk, DataDisks: &disks, }, }, } </ins> ctx, cancel := getContextWithCancel() defer cancel()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When attach/detach azure disk in vmss, it updates whole VM info, including: networkProfile, storageProfile, while this would lead to race condition when there is also network related setting changing in k8s cloud provider in the mean time. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig azure /assign


Question: { Name:    name, Image:   image, <del> Command: []string{"sh", "-c", "touch " + logFile + " && /node-problem-detector --logtostderr --system-log-monitors=" + configFile + fmt.Sprintf(" --apiserver-override=%s?inClusterConfig=false", framework.TestContext.Host)}, </del> <ins> Command: []string{"sh", "-c", "touch " + logFile + " && /node-problem-detector --logtostderr --system-log-monitors=" + configFile + fmt.Sprintf(" --apiserver-override=%s?inClusterConfig=true", framework.TestContext.Host)}, </ins> Env: []v1.EnvVar{ { Name: "NODE_NAME",

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Last successful run (23 Oct 2020, 10:07 PDT): First unsuccessful run (23 Oct 2020, 12:08 PDT): Testgrid link: Reason for failure: Anything else we need to know: presubmit job in repository also started to fail in the same fashion. Attempted to debug it myself, found out the following stuff: last successful run was accessing the apiserver via HTTP: all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail: perused the history of to find out that the PR is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests): first, it coincides with the start of the failures of , second, we changed the default hostname of the apiserver in node tests in that PR. Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a flag to the of these jobs. /assign /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Opened for a mitigation that would make those jobs stop failing.
This won't work, unfortunately. Copying response:
I have filed to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
Unfortunately I don't :/. My PR was also a wild guess. could you take a look at the proposed fix? Thanks once again for your help - really appreciated :).
the submission of hasn't fixed the problems, unfortunately :/. Sample failing run: We still have some peculiar problem related to health checking: and now NPD panics, too:
Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. /reopen
Reopened this issue.


Question: { Name:    name, Image:   image, <del> Command: []string{"sh", "-c", "touch " + logFile + " && /node-problem-detector --logtostderr --system-log-monitors=" + configFile + fmt.Sprintf(" --apiserver-override=%s?inClusterConfig=false", framework.TestContext.Host)}, </del> <ins> Command: []string{"sh", "-c", "touch " + logFile + " && /node-problem-detector --logtostderr --system-log-monitors=" + configFile + fmt.Sprintf(" --apiserver-override=%s?inClusterConfig=true", framework.TestContext.Host)}, </ins> Env: []v1.EnvVar{ { Name: "NODE_NAME",

<detailsIn response to : Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
filed . I hope we could completely address this issue this time.
/milestone v1.20


Question: // Get returns the election record from a Lease spec func (ll *LeaseLock) Get(ctx context.Context) (*LeaderElectionRecord, []byte, error) { <del> var err error ll.lease, err = ll.Client.Leases(ll.LeaseMeta.Namespace).Get(ctx, ll.LeaseMeta.Name, metav1.GetOptions{}) </del> <ins> lease, err := ll.Client.Leases(ll.LeaseMeta.Namespace).Get(ctx, ll.LeaseMeta.Name, metav1.GetOptions{}) </ins> if err != nil { return nil, nil, err } <ins> ll.lease = lease </ins> record := LeaseSpecToLeaderElectionRecord(&ll.lease.Spec) recordByte, err := json.Marshal(*record) if err != nil {

The lock acquired by tryAcquireOrRenew is released when the leader ends leadership. However, due to the cancellation of the context, the lock may be set as an empty lock, so the Update cannot be run normally, resulting in a failure to release the lock. The lock can be released every time the leader ends. run this: and then you can got a log like these: and you when you watch the leases at the same time: you can see that was failed to release because of No response <details</details<details</details<details</details<details</details<details</detailsrelated packages: v1.3.0 v0.25.0 v0.25.0 v2.70.1
/sig api-machinery
/triage accepted thanks!
/unassign
/assign
/unassign
/unassign me


Question: Namespace: namespace, Subsystem: subsystem, Name:      fmt.Sprintf("%s_admission_latencies_seconds", name), <del> Help:      fmt.Sprintf(helpTemplate, "latency histogram"), </del> <ins> Help:      fmt.Sprintf(helpTemplate, "latency histogram in seconds"), Buckets:   latencyBuckets, }, labels, ), deprecatedLatencies: prometheus.NewHistogramVec( prometheus.HistogramOpts{ Namespace: namespace, Subsystem: subsystem, Name:      fmt.Sprintf("%s_admission_latencies_milliseconds", name), Help:      fmt.Sprintf("(Deprecated) "+helpTemplate, "latency histogram in milliseconds"), </ins> Buckets:   latencyBuckets, }, labels, ), <del> latenciesSummary: summary, </del> <ins> latenciesSummary:           summary, deprecatedLatenciesSummary: deprecatedSummary, </ins> } } // MustRegister registers all the prometheus metrics in the metricSet. func (m *metricSet) mustRegister() { prometheus.MustRegister(m.latencies) <ins> prometheus.MustRegister(m.deprecatedLatencies) </ins> if m.latenciesSummary != nil { prometheus.MustRegister(m.latenciesSummary) } <ins> if m.deprecatedLatenciesSummary != nil { prometheus.MustRegister(m.deprecatedLatenciesSummary) } </ins> } // Reset resets all the prometheus metrics in the metricSet. func (m *metricSet) reset() { m.latencies.Reset() <ins> m.deprecatedLatencies.Reset() </ins> if m.latenciesSummary != nil { m.latenciesSummary.Reset() } <ins> if m.deprecatedLatenciesSummary != nil { m.deprecatedLatenciesSummary.Reset() } </ins> } // Observe records an observed admission event to all metrics in the metricSet. func (m *metricSet) observe(elapsed time.Duration, labels ...string) { <ins> elapsedSeconds := elapsed.Seconds() </ins> elapsedMicroseconds := float64(elapsed / time.Microsecond) <del> m.latencies.WithLabelValues(labels...).Observe(elapsedMicroseconds) </del> <ins> m.latencies.WithLabelValues(labels...).Observe(elapsedSeconds) m.deprecatedLatencies.WithLabelValues(labels...).Observe(elapsedMicroseconds) </ins> if m.latenciesSummary != nil { <del> m.latenciesSummary.WithLabelValues(labels...).Observe(elapsedMicroseconds) </del> <ins> m.latenciesSummary.WithLabelValues(labels...).Observe(elapsedSeconds) } if m.deprecatedLatenciesSummary != nil { m.deprecatedLatenciesSummary.WithLabelValues(labels...).Observe(elapsedMicroseconds) </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Admission metrics name is and , the units from metrics name are , but actually the return metrics are in . What you expected to happen: Return metrics in true units. How to reproduce it (as minimally and precisely as possible): curl apiserver:8080/metrics Anything else we need to know?: Environment: Kubernetes version (use ): master branch Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign
/sig api-machinery
/cc
What about these? - -
metrics had been renamed to . And these unit are all fixed to seconds by .


Question: } share := fileClient.GetShareReference(name) share.Properties.Quota = sizeGiB <del> if err = share.Create(nil); err != nil { </del> <ins> newlyCreated, err := share.CreateIfNotExists(nil) if err != nil { </ins> return fmt.Errorf("failed to create file share, err: %v", err) } <ins> if !newlyCreated { klog.V(2).Infof("file share(%s) under account(%s) already exists", name, accountName) } </ins> return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: use an existing azure file share name should be possible in azure file plugin What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig azure /assign


Question: "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", "//staging/src/k8s.io/client-go/util/testing:go_default_library", "//vendor/github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-03-01/compute:go_default_library", <del> "//vendor/github.com/Azure/go-autorest/autorest/to:go_default_library", </del> "//vendor/github.com/stretchr/testify/assert:go_default_library", ], )

<!-- Please only use this template for submitting enhancement requests -- What would you like to be : related to Why is this needed: /assign /sig azure
here is the way about how to get MaxDataDiskCount: and then build kubelet, copy kubelet to agent node and then get MaxDataDiskCount data: There are 207 VM sizes totally.
Update: this command could get VM size list easily:


Question: package dockershim import ( <del> "encoding/base64" </del> "fmt" "io/ioutil" "os"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: We have a private registry configured (where all images including the sandbox one), are hosted. The registry credentials are being pulled by one of k8s credentials providers () With Kubernetes 1.7 sandbox image failed to be pulled with the following error message: Looks like with this only creds from docker config are being evaluated on the sandbox image pull What you expected to happen: Sandbox image pull should be attempted using credentials returned by not just the basic docker credential provider, but other credentials providers registered in the system. How to reproduce it (as minimally and precisely as possible): 1) Register private repo credentials with one of the k8s credential providers (we've tried with rancher). Configure the to use the image from the the private repo. 2) Make sure you are not logged in to this private repo on the kubelet host 3) Try to create a pod. Sandbox image pull will fail with an error mentioned above. Anything else we need to know?: Non-sandbox images from the same private repo were able to download correctly using private repo credentials provider. Environment: Kubernetes version (use ): 1.7.2 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node Just tested the same use case against k8s 1.5.x, it worked fine there and sandbox image was pulled using the credentials passed by the credential provider, not from docker config file. So it might have regressed in 1.7.x (haven't tested it on 1.6, so can't tell for sure)
Confirmed a bug introduced in dockershim. Will file a PR to fix this.


Question: package dockershim import ( <del> "encoding/base64" </del> "fmt" "io/ioutil" "os"

Could you also verify whether fixed your problem?
tested against your commits cherry-picked on top of 1.8 beta, the fix seems to work
thanks.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: return result, nil } <del> func rewriteResolvFile(resolvFilePath string, dns []string, dnsSearch []string, useClusterFirstPolicy bool) error { </del> <ins> // RewriteResolvFile rewrites resolv.conf file generated by docker. // Exported for reusing in dockershim. func RewriteResolvFile(resolvFilePath string, dns []string, dnsSearch []string, useClusterFirstPolicy bool) error { </ins> if len(resolvFilePath) == 0 { glog.Errorf("ResolvConfPath is empty.") return nil

Kubernetes version (use ): HEAD (~v1.6.0-beta.4) Environment: any What happened: I want to use ClusterFirstWithHostNet which was implemented in , but only dockertools (legacy/non-CRI) was updated to support this feature. The CRI dockershim should be modified to support this rewriting the file of Pods with HostNetwork=true as well. This is a bug in v1.6, I hope it can be prioritized to be fixed in v1.6.1, since it's blocking me from using CRI. Since CRI is enabled by default, it's expected that it should have feature parity with the earlier implementation.
Related
- I guess I didn't realize these crossed paths. Do you have time to take a look?
Will take a look
Thanks! Ping me directly on this.
Filed for supporting it in CRI.


Question: for _, test := range tests { klog.V(4).Infof("starting test %q", test.name) ctrl := NewPlugin() <ins> ctrl.resourceQuotaFeatureGateEnabled = true ctrl.nonPreemptingPriority = true </ins> // Add existing priority classes. if err := addPriorityClasses(ctrl, test.existingClasses); err != nil { t.Errorf("Test %q: unable to add object to informer: %v", test.name, err)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the documentation at the , I tried enabling quota for a namespace other than but I was still not able to create a pod with priorityClass in my namespace with following error: Error from server (Forbidden): error when creating "": pods "memory-demo" is forbidden: pods with system-cluster-critical priorityClass is not permitted in my-namespace namespace What you expected to happen: Expected pod to be created with namespace where I allowed quota. I can see that both ResourceQuota and Priority Plugins are enabled from logs. I0408 17:15:08. ] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook. I0408 17:15:08. ] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. How to reproduce it (as minimally and precisely as possible): Follow the steps at Anything else we need to know?: I think, the issue is with priorityPlugin where we are rejecting instead of letting resourceQuota plugin to handle the admission when is enabled. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig scheduling /cc
Same happens to me. Correction, it does not allow to use the default permissions outside of kube-system
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
See also:
/remove-lifecycle stale
/assign Per offline discussion with
Same issue here.


Question: for _, test := range tests { klog.V(4).Infof("starting test %q", test.name) ctrl := NewPlugin() <ins> ctrl.resourceQuotaFeatureGateEnabled = true ctrl.nonPreemptingPriority = true </ins> // Add existing priority classes. if err := addPriorityClasses(ctrl, test.existingClasses); err != nil { t.Errorf("Test %q: unable to add object to informer: %v", test.name, err)

We are using namespaces other than kube-system for system pods as it is getting too crowded, therefore we would like to run cluster-node-critical pods in other namespaces.
Waiting for this to be available myself. No reason my Node Exporter dataset has to run in the namespace instead of my namespace.


Question: // Let the controller-manager allocate Node CIDRs for the Pod network. // Each node will get a subspace of the address CIDR provided with --pod-network-cidr. if cfg.Networking.PodSubnet != "" { <del> // TODO(Arvinderpal): Needs to be fixed once PR #73977 lands. Should be a list of maskSizes. maskSize := calcNodeCidrSize(cfg.Networking.PodSubnet) </del> defaultArguments["allocate-node-cidrs"] = "true" defaultArguments["cluster-cidr"] = cfg.Networking.PodSubnet <del> defaultArguments["node-cidr-mask-size"] = maskSize </del> if cfg.Networking.ServiceSubnet != "" { defaultArguments["service-cluster-ip-range"] = cfg.Networking.ServiceSubnet }

What happened: I use the following configuration file the last generation of ,There are three parameters in yaml which looks like default. The result is that the kube-controller cannot start. When I delete it, the kube-controller will work fine What you expected to happen: In this case, the default parameter should not be Anything else we need to know?: kubeadm version kubectl version
/sig cluster-lifecycle /area kubeadm /milestone v1.17


Question: "description": "ServicePort contains information on service's port.", "properties": { "name": { <del> "description": "The name of this port within the service. This must be a DNS_LABEL. All ports within a ServiceSpec must have unique names. This maps to the 'Name' field in EndpointPort objects. Optional if only one ServicePort is defined on this service.", </del> <ins> "description": "The name of this port within the service. This must be a DNS_LABEL. All ports within a ServiceSpec must have unique names. When considering the endpoints for a Service, this must match the 'name' field in the EndpointPort. Optional if only one ServicePort is defined on this service.", </ins> "type": "string" }, "nodePort": {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When running a cluster service and manually created endpoints, the port name set on the cluster definition acts as an implicit selector - suitable end-points do not get mapped to the cluster service. What you expected to happen: Port name shouldn't be an implicit selector. How to reproduce it (as minimally and precisely as possible): Using the following spec file "": kubectl create -f , then describe the service. Output shows the endpoint as being correctly mapped Now run kubectl delete -f the and uncomment the name: foobar Re-run the kubectl create -f command. Now, checking the service gives the output below (end point isn't mapped): Anything else we need to know?: Adding the name: foobar to the endpoint definition makes the end point to be mapped again into the cluster. I.e. there is an implicit matching of the end-point and cluster port definitions. This is either a bug or a very surprising and undocumented feature Environment: kubectl version Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-03-25T02:52:13Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-01-16T18:14:49Z", GoVersion:"go1.10.7", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): root cat /etc/os-release NAME="Ubuntu" VERSION="18.04.1 LTS (Bionic Beaver)" Kernel (e.g.


Question: "description": "ServicePort contains information on service's port.", "properties": { "name": { <del> "description": "The name of this port within the service. This must be a DNS_LABEL. All ports within a ServiceSpec must have unique names. This maps to the 'Name' field in EndpointPort objects. Optional if only one ServicePort is defined on this service.", </del> <ins> "description": "The name of this port within the service. This must be a DNS_LABEL. All ports within a ServiceSpec must have unique names. When considering the endpoints for a Service, this must match the 'name' field in the EndpointPort. Optional if only one ServicePort is defined on this service.", </ins> "type": "string" }, "nodePort": {

): uname -a Linux 4.15.0-51-generic -Ubuntu SMP Wed May 15 14:27:21 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/remove-sig api-machinery /sig network The name mapping to the endpoints is intentional, but needs documentation
/triage unresolved
/assign
It may be under-documented, but it is not UNdocmented.
Also
working on a PR to fix doc


These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions


Question: return false, false, errors.NewConflict(schema.GroupResource{Group: gvk.Group, Resource: gvk.Kind}, objectMeta.GetName(), fmt.Errorf("the ResourceVersion in the precondition (%s) does not match the ResourceVersion in record (%s). The object might have been modified", *options.Preconditions.ResourceVersion, objectMeta.GetResourceVersion())) } } <ins> // Negative values will be treated as the value `1s` on the delete path. if gracePeriodSeconds := options.GracePeriodSeconds; gracePeriodSeconds != nil && *gracePeriodSeconds < 0 { options.GracePeriodSeconds = utilpointer.Int64(1) } if deletionGracePeriodSeconds := objectMeta.GetDeletionGracePeriodSeconds(); deletionGracePeriodSeconds != nil && *deletionGracePeriodSeconds < 0 { objectMeta.SetDeletionGracePeriodSeconds(utilpointer.Int64(1)) } </ins> gracefulStrategy, ok := strategy.(RESTGracefulDeleteStrategy) if !ok { // If we're not deleting gracefully there's no point in updating Generation, as we won't update

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Some k8s client tried to evict a pod by setting negative value for deletionGracePeriodSeconds. But all eventual delete calls (and force-delete calls) made by kubelet, pod-garbage-collector, etc were not able to delete that pod, because apiserver doesnt allow deletionGracePeriodSeconds to be increased once it is set. So force-delete (which set it to 0) doesn't take effect when it's already negative. Not even kubectl is able to call delete with negative grace-period. But curl with an even more negative grace-period value worked. What you expected to happen: force-delete works. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I think the issue is in the code below As the comment suggest, in phase 1, the was set to a negative value (meaning was in the past), and in phase 2, has to be set even lower, otherwise, it will be considered as pending graceful deletion and not performing the immediate delete. I think there're two possible fixes. not allowing setting in the past ( should be non-negative). In that case, we could convert to 0 if negative. that could be set in the past. In that case, in phase 2, we treat negative value of the same as 0, and allow immediate delete. Thoughts? Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/cc /triage accepted
No component in the system should ever treat the timestamp of DeletionTimestamp as authoritative.


Question: return false, false, errors.NewConflict(schema.GroupResource{Group: gvk.Group, Resource: gvk.Kind}, objectMeta.GetName(), fmt.Errorf("the ResourceVersion in the precondition (%s) does not match the ResourceVersion in record (%s). The object might have been modified", *options.Preconditions.ResourceVersion, objectMeta.GetResourceVersion())) } } <ins> // Negative values will be treated as the value `1s` on the delete path. if gracePeriodSeconds := options.GracePeriodSeconds; gracePeriodSeconds != nil && *gracePeriodSeconds < 0 { options.GracePeriodSeconds = utilpointer.Int64(1) } if deletionGracePeriodSeconds := objectMeta.GetDeletionGracePeriodSeconds(); deletionGracePeriodSeconds != nil && *deletionGracePeriodSeconds < 0 { objectMeta.SetDeletionGracePeriodSeconds(utilpointer.Int64(1)) } </ins> gracefulStrategy, ok := strategy.(RESTGracefulDeleteStrategy) if !ok { // If we're not deleting gracefully there's no point in updating Generation, as we won't update

In Kube, components are required to calculate the time period from when they observe the event in their local clock (i.e. 30s from now), not use the deletion timestamp. A negative value of deletion grace period should be rejected by the apiserver as an invalid value (bad request / invalid field).
attempts to take a missing or positive deletion grace period negative should be rejected, I agree. if a negative value is already persisted, I'd be inclined to allow a negative value in an update and clip to 0
As I mentioned in the PR, eviction is passing negatives through to the store: In kubectl, we specify the default value for GracePeriod is -1. In the case of delete, we simply don't set the value in the deleteOptions we send to the API. In the case of eviction, we send the -1 value, and that gets passed along to the store.
I don't see the -1 getting sent to the eviction API. It looks like values < 0 result in a nil value being sent to the API: I would not expect the API to accept a negative value as-is.
Looks like you are correct. Do the deletionOptions go through their own validation or through some kind of validation for the evict API? The original bug report mentions 'evict' though it's not clear if the eviction API was used or not.
ValidateDeleteOptions is called inside pod deletion, but did not check that the field was non-negative. I don't see any eviction-specific validation.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/lifecycle rotten
Im investigating the and . Surprisingly see you here...


Question: func (rc *reconciler) updateReconstructedDevicePaths() { klog.V(4).InfoS("Updating reconstructed devicePaths") <ins> if rc.kubeClient == nil { // Skip reconstructing devicePath from node objects if kubelet is in standalone mode. // Such kubelet is not expected to mount any attachable volume or Secrets / ConfigMap. klog.V(2).InfoS("Skipped reconstruction of DevicePaths from node.status in standalone mode") rc.volumesNeedDevicePath = nil return } </ins> node, fetchErr := rc.kubeClient.CoreV1().Nodes().Get(context.TODO(), string(rc.nodeName), metav1.GetOptions{}) if fetchErr != nil { // This may repeat few times per second until kubelet is able to read its own status for the first time.

Must be a result of When running kubelet in standalone mode got this panic: No panic run kubelet in standalone mode No response master branch <detailsN/A </details<detailsN/A </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /sig storage /priority critical-urgent
/assign
I saw this as well. It is another instance of nil client pointer access.
Here's the old reconstruction logic with standalone kubelet handling:
Reading scattered docs, "standalone mode" is a kubelet without and it won't ever get any connection to the API server, right? Is there an e2e ~test~ job that runs kubelet in standalone mode?
Fix: I haven't found a presubmit job with a standalone kubelet.
I don't believe we have a presubmit for this (yet). I can repro this in local cluster with standalone kubelet with a small code change: I tried your diff and it works. Please feel free to try the above for double-verification.


Question: func TestCanSupport(t *testing.T) { assert := assert.New(t) <del> plugMgr := newInitializedVolumePlugMgr() </del> <ins> plugMgr, _ := newInitializedVolumePlugMgr(t) </ins> plug, err := plugMgr.FindPluginByName(pluginName) assert.NoError(err)

Unit tests on our CI have apparently been failing for 2 weeks and no one noticed (oops!)... The PR that seems to have caused the failures is . The failing test is - . Full Test Log: I'm not sure why this is failing on our CI and not on shippable. Has anyone else seen something similar? That test only seems to fail on the second run through with . Our unit tests run in a container built specifically for testing k8s-mesos: The Dockerfile for the env: Is anyone else able to repro with the above env or their own?
/cc
that test was around a week ago, it's doesn't seem possible to me that it was failing since 2 weeks ago. Anyway, we have few duplications of this issue: this one And the PR where is working on: It's a little bit difficult to me to follow all the conversations in all the differente places about the same bug, could it be possible to unificate some? Since you provided the environment I will try and take a look to reproduce it locally. Thanks!
I think I've got a working patch in
Closing in favor of &


Question: func TestCanSupport(t *testing.T) { assert := assert.New(t) <del> plugMgr := newInitializedVolumePlugMgr() </del> <ins> plugMgr, _ := newInitializedVolumePlugMgr(t) </ins> plug, err := plugMgr.FindPluginByName(pluginName) assert.NoError(err)

reports the following failure after syncing with latest code: --- FAIL: TestSetUpAtInternal (0.00s) Error Trace: Error: Not equal: "expected-to-be-set-properly" (expected) != "" (actual) FAIL FAIL 0.067s cc/
CC
Taking a look. What do you mean with failed sometimes? Is it a random failure?
Can you please provide more information. This test seems to be consistently green in our CI runs.
The test is failing on this assertion: but as you can see on the previous line I am checking that there was no error on that call. So the function is returning for the error checking but it's not setting . Only place where it can happen is here: Is the not been clean between runs?
When I said "the only place where it can happen" it was because the other s are being returned after a check. I think that there is no other way to get there.
PS: I run the test suite 100 times and I didn't get any error: (That branch is based on the branch)
can you test to see if the patch in fixes things for you?
See for reproducible test environment and failure logs.
Based on the solution in it sounds like this test creates files the first time through which causes the tests to fail on the second time through.


Question: "k8s.io/kubernetes/pkg/scheduler/factory" labelsutil "k8s.io/kubernetes/pkg/util/labels" "k8s.io/kubernetes/pkg/util/metrics" <ins> "k8s.io/kubernetes/staging/src/k8s.io/client-go/util/retry" </ins> "k8s.io/kubernetes/test/integration/framework" )

https://k8s- /sig apps /priority critical-urgent /assign PR that the test: /cc
GitHub didn't allow me to assign the following users: mortent. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : https://k8s- /priority critical-urgent /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake
I'm looking into this.
The test is still flaky, but with different symptom: An example flake: https://k8s- It's likely because the collision count was updated in a previous sync loop, but the in the cache store hasn't been updated and we used a stale collision count while calculating the new hash: /assign


Question: "Failed to set statusUpdateNeeded to needed %t because nodeName=%q  does not exist", needed, nodeName) <ins> return </ins> } nodeToUpdate.statusUpdateNeeded = needed

Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: $ kubectl describe pods test-ipj93 Name: test-ipj93 Namespace: default Node: kubernetes-minion-group-yxra/10.128.0.3 Start Time: Mon, 28 Nov 2016 14:39:28 -0800 Labels: <noneStatus: Pending IP: Controllers: <noneContainers: busybox: Container ID: Image: busybox Image ID: Port: Command: sh -c while true; do echo (hostname)-(cat /proc/stat cut -f 4 -d" "); sleep 30; done QoS Tier: cpu: Burstable memory: BestEffort Requests: cpu: 100m State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test ReadOnly: false default-token-kjoqg: Type: Secret (a volume populated by a Secret) SecretName: default-token-kjoqg Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 57m 57m 1 {default-scheduler } Normal Scheduled Successfully assigned test-ipj93 to kubernetes-minion-group-yxra 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulling pulling image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulled Successfully pulled image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Created Created container with docker id ; Security:[seccomp=unconfined] 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Started Started container with docker id 47m 47m 6 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.


Question: "Failed to set statusUpdateNeeded to needed %t because nodeName=%q  does not exist", needed, nodeName) <ins> return </ins> } nodeToUpdate.statusUpdateNeeded = needed

SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig.


Question: "Failed to set statusUpdateNeeded to needed %t because nodeName=%q  does not exist", needed, nodeName) <ins> return </ins> } nodeToUpdate.statusUpdateNeeded = needed

This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes