Question: } func (mounter *Mounter) IsNotMountPoint(dir string) (bool, error) { <del> return IsNotMountPoint(mounter, dir) </del> <ins> return isNotMountPoint(mounter, dir) </ins> } func (mounter *Mounter) IsLikelyNotMountPoint(file string) (bool, error) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There's , and . The former is used by local and nfs volume plugins, while the latter is used by , which is also used by local and nfs volume plugins. What you expected to happen: Can we consolidate these? This is really confusing and could easily be the source of bugs. <!-- DO NOT EDIT BELOW THIS LINE --/kind cleanup
cc
/assign I'd like to clean up code.


Question: <ins> load("@io_bazel_rules_go//go:def.bzl", "go_library") go_library( name = "go_default_library", srcs = ["fuzzer.go"], importpath = "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/fuzzer", visibility = ["//visibility:public"], deps = [ "//pkg/kubelet/apis/kubeletconfig:go_default_library", "//pkg/kubelet/apis/kubeletconfig/v1alpha1:go_default_library", "//pkg/kubelet/qos:go_default_library", "//pkg/kubelet/types:go_default_library", "//pkg/master/ports:go_default_library", "//vendor/github.com/google/gofuzz:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/runtime/serializer:go_default_library", ], ) filegroup( name = "package-srcs", srcs = glob(["**"]), tags = ["automanaged"], visibility = ["//visibility:private"], ) filegroup( name = "all-srcs", srcs = [":package-srcs"], tags = ["automanaged"], visibility = ["//visibility:public"], ) </ins>

Solution: I will add one.
/cc
xref
Thanks for sending a PR so quickly!
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code slush, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: If this issue is not resolved or labeled as by Wed, Nov 22 it will be moved out of the v1.9 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: Name: "foo", }, Data: map[string][]byte{ <del> v1.DockerConfigKey: secretData, </del> <ins> v1.DockerConfigJsonKey: secretData, </ins> }, <del> Type: v1.SecretTypeDockercfg, </del> <ins> Type: v1.SecretTypeDockerConfigJson, </ins> }, expectErr: false, },

/kind bug We're having trouble retrieving images from a private registry on DockerCloud. For nearly a year the following has worked flawlessly. Last Friday I pulled down the 1.9.0 CLI (via brew) and since then have not been able to access any image in a private registry on DockerCloud. I've tried this on 10+ brand new clusters created with kops v1.8.0. The Kubernetes cluster version have been 1.8.0, 1.8.4, 1.8.5 and 1.9.0 (which is not supported by kops yet) and all exhibit the same problem. We create the secret as such: And verify it using: Which yields: And create a test service with with these contents: The pod fails immediatly with: I've reviewed the 1.9.0 release notes and nothing jumps out as the root cause. For reference I've tried putting quotes around the value when creating the secret to no avail. I've also tried prefixing the value with , so the full value would read: also to no avail. Environment: Kubernetes version: Cloud provider and installation info: AWS cluster built with Kops 1.8.0, using (to isolate this issue) nearly the default settings for everything.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have same problem, and I "solved" it creating a docker-registry secret in an old kubernetes version and pasting it on new kubernetes
I'm recently (I know it worked perfectly about last month) facing issues with as well. For me it ended in a similar manner saying: I was able to confirm, that this is a kubernetes CLI related issue and not end user. Based on and this I was able to create properly formatted auth information looking like this: . When encoding this one-liner using (MacOS workstation) and apply like this works fine and images can be pulled from private repository again.


Question: Name: "foo", }, Data: map[string][]byte{ <del> v1.DockerConfigKey: secretData, </del> <ins> v1.DockerConfigJsonKey: secretData, </ins> }, <del> Type: v1.SecretTypeDockercfg, </del> <ins> Type: v1.SecretTypeDockerConfigJson, </ins> }, expectErr: false, },

However this breaks all automation and can't be considered as solution Confirmed working with kubectl 1.8.5 and server 1.8.4 (guess 1.7.9 as well)
looks like there's an extra "auths" element when adding the secret using 1.9 kubectl
Added in /assign
Many thanks for the quick fix on this. Any idea when this might get rolled out? Also, it might be a good idea if this was to the known issue list for v1.9.0. It would have saved us a bunch of time trying to track down the root cause. Just trying to make sure no one else needs to do the same.
linking to the "v1.9.0 known issues" list
ack, asking Mehdy since he is the patch manager for v1.9 branch. Mehdy, any idea when we can ship 1.9.1 with this fix?
workarounds for folks until 1.9.1 is released: if you have a .dockerconfigjson for your private registry already, you can manually specify the type and data key (e.g. ) if you don't have a .dockerconfigjson file already, you can fix up the secret produced by manually: the type from to the data key from to the modified secret with
1 will be released tomorrow
I can confirm solution works, thanks so much!
I am still having issues in 1.9.1. This worked in the 1.8.x branches: results in the following error:
you have to remove the gitlab-registry secret before you can apply an update... type is immutable, and the fix in 1.9.1 required changing the type of the secret


Question: /* Release : v1.23 Testname: Pod Lifecycle, prestop https hook <del> Description: When a pre-stop handler is specified in the container lifecycle using a 'HttpGet' action, then the handler MUST be invoked before the container is terminated. A server pod is created that will serve https requests, create a second pod with a container lifecycle specifying a pre-stop that invokes the server pod to validate that the pre-stop is executed. </del> <ins> Description: When a pre-stop handler is specified in the container lifecycle using a 'HttpGet' action, then the handler MUST be invoked before the container is terminated. A server pod is created that will serve https requests, create a second pod on the same node with a container lifecycle specifying a pre-stop that invokes the server pod to validate that the pre-stop is executed. </ins> */ ginkgo.It("should execute prestop https hook properly [MinimumKubeletVersion:1.23] [NodeConformance]", func() { lifecycle := &v1.Lifecycle{

release-master-informing: capz-windows-containerd-masterKubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] Kubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] Both the tests have been failing during most of the runs since 20th October Kubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] | 1m6s -- | -- {Oct 31 18:42:02.425: Timed out after 30.001s. Expected <errors.errorString | 0xc0015283b0: { s: "failed to match regexp "GET /echo?msg=prestop" in output "I1031 18:41:06. 8988 ] Started HTTP server on port 9090nI1031 18:41:06. 8988 ] Started UDP server on port 9091n"", } to be nil failed (0xc000f1d000) +0x469 () +0x335} Kubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] | 5m11s -- | -- {Oct 31 19:05:50.205: timed out while waiting for pod container-lifecycle-hook-6892/pod-with-poststart-https-hook to be running and ready failed (PodClient).CreateSync(0xc000eb5920, 0x22?) +0x94 (0xc0045e8c00) +0x73 () +0x325} No response_ /sig windows cc
/triage accepted /cc
These failures happen also with Kubernetes master + containerd master with Flannel CNI. Testgrid link:
I figured out why these new failures appear.


Question: /* Release : v1.23 Testname: Pod Lifecycle, prestop https hook <del> Description: When a pre-stop handler is specified in the container lifecycle using a 'HttpGet' action, then the handler MUST be invoked before the container is terminated. A server pod is created that will serve https requests, create a second pod with a container lifecycle specifying a pre-stop that invokes the server pod to validate that the pre-stop is executed. </del> <ins> Description: When a pre-stop handler is specified in the container lifecycle using a 'HttpGet' action, then the handler MUST be invoked before the container is terminated. A server pod is created that will serve https requests, create a second pod on the same node with a container lifecycle specifying a pre-stop that invokes the server pod to validate that the pre-stop is executed. </ins> */ ginkgo.It("should execute prestop https hook properly [MinimumKubeletVersion:1.23] [NodeConformance]", func() { lifecycle := &v1.Lifecycle{

I've submitted the fix via PR
This issue is also affecting 1.25 tests: If possible, we should cherry-pick to the release-1.25 branch once it's merged on master.


Question: return } <ins> func getProxyEnvVars() []api.EnvVar { envs := []api.EnvVar{} for _, env := range os.Environ() { pos := strings.Index(env, "=") if pos == -1 { // malformed environment variable, skip it. continue } name := env[:pos] value := env[pos+1:] if strings.HasSuffix(strings.ToLower(name), "_proxy") && value != "" { envVar := api.EnvVar{Name: name, Value: value} envs = append(envs, envVar) } } return envs } </ins>

Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? http proxy, httpproxy, httpsproxy, AWS, kubeadm, kubectl, cloud-provider Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): v1.4.4 Environment: Cloud provider or hardware configuration: kubeadm on EC2 instances on AWS OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Linux ip-10-0-133-99 4.4.0-45-generic -Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: httpproxy, httpsproxy and noproxy set in /etc/environment, Docker config as well on startup scripts that run What happened: won't complete and gets stuck on it's first deployment. When one looks through the logs it shows that the pods are downloading fine but we are stuck due to the kube-controller-manager continually crashing as it can't contact AWS. Elsewhere in kubernetes I see references to and if the AWS cloud provider could use that it would be able to connect as well. From the POD if I manually set httpproxy and httpsproxy and no_proxy I am able to use the AWS CLI without any issues. What you expected to happen: I would expect to successfully complete and use the environment proxies set up on the system so as not to depend on a default Internet Gateway being connected to the VPC just for AWS access. How to reproduce it (as minimally and precisely as possible): Set up a public VPC with Squid set up as a proxy Set up a private VPC peered to the public VPC Set up a master node and run Set up a regular node and run will never fully initialize. If you remove the --cloud-provider aws` then it will complete and nodes will register. Anything else do we need to know:
what do you have in proxy variables ? (env | grep -i proxy) it tries to connect to (tcp 176.32.118.53:443) directly. You're expecting to connect there via proxy ? then do you have something like in your no_proxy ?


Question: return } <ins> func getProxyEnvVars() []api.EnvVar { envs := []api.EnvVar{} for _, env := range os.Environ() { pos := strings.Index(env, "=") if pos == -1 { // malformed environment variable, skip it. continue } name := env[:pos] value := env[pos+1:] if strings.HasSuffix(strings.ToLower(name), "_proxy") && value != "" { envVar := api.EnvVar{Name: name, Value: value} envs = append(envs, envVar) } } return envs } </ins>

Unfortunately on the pod itself for the the environment variables aren't passed through. As this is created using I don't have control over the pod definition. So basically none of the *_proxy variables are set inside the pod in the docker image, however if I and export the variables and try to wget that URL it works. Basically I'm hoping we can update to respect and pass the environment variables down to the system pods if they exist, eg, add: Without it I can't stand up a cluster with AWS integration on a completely private VPC and for such a small change it would make a rather big difference.
ok, so request is actually to pass proxy variables into containers. interesting. I'll check how it can be done. BTW, for noproxy value in your example: things like 10.0.0.0/8 would not work on 99% of software that respects noproxy variable. it must be always exact IP match.
you're right that what I'm actually asking for is to pass proxy variables through either from a commandline setting on kubeadm or pick it up from /etc/environment or shell environment that launches kubeadm. I didn't realize it till I dug in to try and think about how one might go about fixing this. At first I thought we would just pick up the variables from the system environment but then I realized kube-controller-manager is deployed in a container and so we'd need to update the yaml file that launches it to contain the environment variables. As I'd like to go to production with this soon hopefully this is a small change that can be .
This issue was moved to kubernetes/kubeadm


Question: } newNodes, err := e2enode.CheckReady(c, len(origNodes.Items)-1, 5*time.Minute) <del> framework.ExpectEqual(err, nil) </del> <ins> framework.ExpectNoError(err) </ins> framework.ExpectEqual(len(newNodes), len(origNodes.Items)-1) _, err = c.CoreV1().Nodes().Get(nodeToDelete.Name, metav1.GetOptions{})

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: kubernetes/node-problem-detector/node-problem-detector-ci: ci-npd-e2e-node See the Which test(s) are failing: [] NodeProblemDetector [NodeFeature:NodeProblemDetector] [] SystemLogMonitor should generate node condition and events for corresponding errors Since when has it been failing: Dec 03. was at 19:50 PST, was at 21:51 PST. Testgrid link: https://k8s- Reason for failure: Apparently is the culprit. Anything else we need to know: This is likely caused by some library upgrade. Nothing big.
/sig node
/assign
Seems like the failure is caused by . /cc /cc /cc /cc
/priority critical-urgent
The particular bug I care about has been fixed in . I'm gonna still work on submitting , as the same error should also be affecting a few other tests.


Question: //  * Multiple consumers and producers. In particular, it is allowed for an //      item to be reenqueued while it is being processed. //  * Shutdown notifications. <del> package workqueue </del> <ins> package workqueue // import "k8s.io/client-go/util/workqueue" </ins>

Look at one of the directories under (Please don't pick more than one directory for one PR as it will cause issues for reviewers) Find all the files named (example find -name ) See the following link it has a this tells tools how this package needs to be imported. So we should ensure that each has an import Add a description appropriate to the package (similar to the one below) Look through sub directories and see if there are any other important directories that are likely to be used by outside users and add a Why are we doing this? We perfer to use vanity urls instead of directly using to refer to code, so this helps ensure that tools honor our wish. See It would be good to have some package level documentation since these directories under staging/ are meant to be for use by folks outside of kubernetes github org. /good-first-issue
/kind documentation /sig architecture
currently working my through Not going to use / assign since there could be a number of PRs for this issue.
/lifecycle active /good-first-issue edit: hmm..
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /lifecycle active /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
May I try on
IIUC this means that there needs to be a even at the root of the repo? Then how does work? :thinking:
Hi , can I make a PR for this directory ?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: //  * Multiple consumers and producers. In particular, it is allowed for an //      item to be reenqueued while it is being processed. //  * Shutdown notifications. <del> package workqueue </del> <ins> package workqueue // import "k8s.io/client-go/util/workqueue" </ins>

/lifecycle rotten
I guess this is still not done, was supposed to partially fix this issue. /reopen
Reopened this issue. <detailsIn response to : I guess this is still not done, was supposed to partially fix this issue. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /area code-organization


Question: failedGVs := map[schema.GroupVersion]error{} var resourcesByGV map[schema.GroupVersion]*metav1.APIResourceList // Switch on content-type server responded with: aggregated or unaggregated. <del> switch responseContentType { case AcceptV1: err = json.Unmarshal(body, apiGroupList) if err != nil { return nil, nil, nil, err } case AcceptV2Beta1: </del> <ins> switch { case isV2Beta1ContentType(responseContentType): </ins> var aggregatedDiscovery apidiscovery.APIGroupDiscoveryList err = json.Unmarshal(body, &aggregatedDiscovery) if err != nil {

We are using (and ) to interact with Kubernetes clusters behind a couple of proxy layers. One of these proxies completes the headers to include character encoding details. This was working fine until recently, when client-go/kubectl have started to fail when interacting with the discovery API. It seems that a recent change has made kubectl more restrictive of the content-type that it expects. I suspect that it is related to this commit: For example, this is an error from a kubectl command: We can of course try to fix it on our end, removing that charset information. However, it is a valid content-type and the way kubectl is handling the content-type there seems like it is relying on implementation details of kube-apiserver (that it does not add metadata to the content-type). I propose to fix this issue in , by properly parsing and handling the content-type.
cc /kind regression /kind bug /sig api-machinery /priority important-soon looks like the switch statement in downloadLegacy() needs to be more tolerant of the variety of content-types it can get from a server (including text/plain or no content-type if old servers could have returned that...)
/transfer kubernetes/kubernetes
Something went wrong or the destination repo kubernetes/kubernetes/kubernetes does not exist. <detailsIn response to : /transfer kubernetes/kubernetes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes
Yep, the string comparison for the content-type is flimsy, let's address that. Thanks for the report!
/assign
/triage accepted
/cc
The fix is being cherry-picked back to: : :


Question: Reason: startWaitingReason, }}, }, <del> map[string]api.ContainerState{ </del> <ins> expectedInitState: map[string]api.ContainerState{ "without-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: initWaitingReason, }}, "with-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: initWaitingReason, }}, }, expectedLastTerminationState: map[string]api.ContainerState{ </ins> "with-old-record": {Terminated: &api.ContainerStateTerminated{}}, }, }, // For running container, State should be Running, LastTerminationState should be retrieved from latest terminated status. { <del> []api.Container{{Name: "running"}}, []*kubecontainer.ContainerStatus{ </del> <ins> containers: []api.Container{{Name: "running"}}, statuses: []*kubecontainer.ContainerStatus{ </ins> { Name:      "running", State:     kubecontainer.ContainerStateRunning,

is this a Node bug? cc as next oncall to follow up on this.
This is a node bug due to init containers
I may need help debugging - I see:
So I think the problem is in the error case - we're not properly recording that the start failed, and the next time we come in we somehow think the first container succeeded. I'll add some debugging to see if we can't recreate it.
Let's discuss on instead.


Question: Reason: startWaitingReason, }}, }, <del> map[string]api.ContainerState{ </del> <ins> expectedInitState: map[string]api.ContainerState{ "without-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: initWaitingReason, }}, "with-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: initWaitingReason, }}, }, expectedLastTerminationState: map[string]api.ContainerState{ </ins> "with-old-record": {Terminated: &api.ContainerStateTerminated{}}, }, }, // For running container, State should be Running, LastTerminationState should be retrieved from latest terminated status. { <del> []api.Container{{Name: "running"}}, []*kubecontainer.ContainerStatus{ </del> <ins> containers: []api.Container{{Name: "running"}}, statuses: []*kubecontainer.ContainerStatus{ </ins> { Name:      "running", State:     kubecontainer.ContainerStateRunning,

Kubelet issues a docker inspect call after creating a container, but for very short lived containers there is a window in which they could exit before being inspected in the docker manager. If the container PID is 0, we should check the ExitCode and other relevant fields and handle them appropriately. [1] This is the suspected cause of several flakes: /cc
looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: I'd expect the second one to not have run in this case? Node logs:
Init containers have an implicit assumption that container statuses include results of previous SyncPod invocations. If that's not the case we may need to fix that (or at least mitigate it). On May 18, 2016, at 8:28 PM, Prashanth B wrote: looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: May 18 15:58:33.380: INFO: pod-init--1d4b-11e6-80a4- started at 2016-05-18T15:58:04-07:00 (2+1 container statuses recorded) May 18 15:58:33.380: INFO: Init container init1 ready: false, restart count 0 May 18 15:58:33.380: INFO: Init container init2 ready: true, restart count 0 I'd expect the second one to not have run in this case? Node logs: — You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub
This should be true. Is there any evidence that suggests the opposite? only shows conflicting view of whether init1 has run.
If the container exits already and it's not an infra container, kubelet should not error out. Although even if it does, I failed to see how this could affect the readiness of the init container without checking the code further.


Question: Reason: startWaitingReason, }}, }, <del> map[string]api.ContainerState{ </del> <ins> expectedInitState: map[string]api.ContainerState{ "without-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: initWaitingReason, }}, "with-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: initWaitingReason, }}, }, expectedLastTerminationState: map[string]api.ContainerState{ </ins> "with-old-record": {Terminated: &api.ContainerStateTerminated{}}, }, }, // For running container, State should be Running, LastTerminationState should be retrieved from latest terminated status. { <del> []api.Container{{Name: "running"}}, []*kubecontainer.ContainerStatus{ </del> <ins> containers: []api.Container{{Name: "running"}}, statuses: []*kubecontainer.ContainerStatus{ </ins> { Name:      "running", State:     kubecontainer.ContainerStateRunning,

Each syncPod invocation, we inspect the list of init container in order looking for lastTerminationState. There could be a bug in that logic, or if we didn't get the full status, we might make a bad decision. When determining which container to run next, we always look for the last init container in the chain that has success, then run the next container (unless the pod is RestartNever). , Yu-Ju Hong wrote:
Saw this flake again. How do we verify that container status is not lost between syn pods?
... between calls to syncPods
The second sync clearly saw the init1 container, and decided to launch init2. See "InitContainersToKeep" in the message below.
containers to keep can include failed containers. , Yu-Ju Hong wrote:
Discussed on slack. In summary, the problem has to do with how we convert the kubecontainer.PodStatus to the api.PodStatus. If a container is terminated, the function makes the decisions: the pod restart policy is "Always", and there is a "failure reason" recorded for the given container, this means the pod worker tried to (re)start the container but failed. Set the state to "Waiting" and populate the reason. , the container state is simply "terminated". Init containers don't obey the restart policy like regular containers, and the logic will need to be amended to work with them. is working on a PR. In addition to that, kubelet should not error out if a user container exited before it could apply the oom score (which is best effort in this case). fixes that.
Ahh, thanks for digging into the issue. This is the exact the concern I had when I reviewed the initial implementation of init container, which blocks pr for a while. I even asked to help me verify this case, unfortunately we didn't reproduce the issue back then. :-(
I wish I'd seen the flakes in the runs prior to merge.
We didn't notice this until we started running the tests against docker v1.11.1, which is much faster and hit the race condition differently.
I suppose that's a good thing...
It is hard to reproduce this.


Question: Reason: startWaitingReason, }}, }, <del> map[string]api.ContainerState{ </del> <ins> expectedInitState: map[string]api.ContainerState{ "without-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: initWaitingReason, }}, "with-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: initWaitingReason, }}, }, expectedLastTerminationState: map[string]api.ContainerState{ </ins> "with-old-record": {Terminated: &api.ContainerStateTerminated{}}, }, }, // For running container, State should be Running, LastTerminationState should be retrieved from latest terminated status. { <del> []api.Container{{Name: "running"}}, []*kubecontainer.ContainerStatus{ </del> <ins> containers: []api.Container{{Name: "running"}}, statuses: []*kubecontainer.ContainerStatus{ </ins> { Name:      "running", State:     kubecontainer.ContainerStateRunning,

The extra check of last terminated container in your initial code makes me nervous, but we couldn't reproduce it due to that small race windows until we have a faster docker running on the node. :-)
I got "Pods should invoke init containers on a RestartAlways pod" flake (referenced from ): Logs:
All the fixes went in. Please reopen this issue if you see any more flakes.


Question: framework.ExpectNoError(waitErr, "some pods failed to complete within %v", completeTimeout) }) }) <del> ginkgo.Context("Pods sharing a single local PV [Serial]", func() { var ( pv *v1.PersistentVolume ) ginkgo.BeforeEach(func(ctx context.Context) { localVolume := &localTestVolume{ ltr: &utils.LocalTestResource{ Node: config.randomNode, Path: "/tmp", }, localVolumeType: DirectoryLocalVolumeType, } pvConfig := makeLocalPVConfig(config, localVolume) var err error pv, err = e2epv.CreatePV(ctx, config.client, f.Timeouts, e2epv.MakePersistentVolume(pvConfig)) framework.ExpectNoError(err) }) ginkgo.AfterEach(func(ctx context.Context) { if pv == nil { return } ginkgo.By(fmt.Sprintf("Clean PV %s", pv.Name)) err := config.client.CoreV1().PersistentVolumes().Delete(ctx, pv.Name, metav1.DeleteOptions{}) framework.ExpectNoError(err) }) ginkgo.It("all pods should be running", func(ctx context.Context) { var ( pvc   *v1.PersistentVolumeClaim pods  = map[string]*v1.Pod{} count = 2 err   error ) pvc = e2epv.MakePersistentVolumeClaim(makeLocalPVCConfig(config, DirectoryLocalVolumeType), config.ns) ginkgo.By(fmt.Sprintf("Create a PVC %s", pvc.Name)) pvc, err = e2epv.CreatePVC(ctx, config.client, config.ns, pvc) framework.ExpectNoError(err) ginkgo.By(fmt.Sprintf("Create %d pods to use this PVC", count)) podConfig := e2epod.Config{ NS:           config.ns, PVCs:         []*v1.PersistentVolumeClaim{pvc}, SeLinuxLabel: selinuxLabel, } for i := 0; i < count; i++ { pod, err := e2epod.MakeSecPod(&podConfig) framework.ExpectNoError(err) pod, err = config.client.CoreV1().Pods(config.ns).Create(ctx, pod, metav1.CreateOptions{}) framework.ExpectNoError(err) pods[pod.Name] = pod } ginkgo.By("Wait for all pods are running") const runningTimeout = 5 * time.Minute waitErr := wait.PollImmediate(time.Second, runningTimeout, func() (done bool, err error) { podsList, err := config.client.CoreV1().Pods(config.ns).List(ctx, metav1.ListOptions{}) if err != nil { return false, err } runningPods := 0 for _, pod := range podsList.Items { switch pod.Status.Phase { case v1.PodRunning: runningPods++ } } return runningPods == count, nil }) framework.ExpectNoError(waitErr, "Some pods are not running within %v", runningTimeout) }) }) </del> }) func deletePodAndPVCs(ctx context.Context, config *localTestConfig, pod *v1.Pod) error {

During OPNFV test execution we noticed the folder selinux context changed. e2e test log How the folder changed after the test executed. I expected the TC won't change the selinux context. The question here, should this sigstorageserial TC set work on an selinux enabled K8S cluster or not? This question valid for the another sigstorage TC set with several test case, where selinux block a lot of TC too. See above. No response_ <details</details<detailsOpenStack </details<details</details<details</details<details</details<details</details
/sig testing
/sig storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/assign
Reproducer with simple kubernetes/kubernetes artifacts:
/triage accepted


Question: ds:            newDaemonSet("ds"), shouldEnqueue: true, }, <ins> { test:    "Node Allocatable changed", oldNode: newNode("node1", nil), newNode: func() *v1.Node { node := newNode("node1", nil) node.Status.Allocatable = allocatableResources("200M", "200m") return node }(), ds: func() *apps.DaemonSet { ds := newDaemonSet("ds") ds.Spec.Template.Spec = resourcePodSpecWithoutNodeName("200M", "200m") return ds }(), expectedEventsFunc: func(strategyType apps.DaemonSetUpdateStrategyType) int { switch strategyType { case apps.OnDeleteDaemonSetStrategyType: return 2 case apps.RollingUpdateDaemonSetStrategyType: return 3 default: t.Fatalf("unexpected UpdateStrategy %+v", strategyType) } return 0 }, shouldEnqueue: true, }, </ins> } for _, c := range cases { for _, strategy := range updateStrategies() {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: a cluster, And node1 with resource 1 DaemonSet, request resource 4 run , There is no pod created. resource to the node: run , There is still no pod created. What you expected to happen: In step 5, There is one pod created. How to reproduce it (as minimally and precisely as possible): see above description Anything else we need to know?: Environment: Kubernetes version (use ):1.10.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release):CentOS7 Kernel (e.g. ): Install tools: Others: /sig app
xref
Thanks! is not work here! Because: DaemonSetsController didn't watch the changes.(we use to updateNode.) we restart kubelet, The DaemonSetsController watch the changes, but it seems that the node info is not updated, and still failed to create pod:
I found it is a bug here, ignore the changes. I made a fix , PTAL! Thanks!
Any workarounds for old cluster versions? Up: looks like in my case helped but after some delay.
, which version of k8s are you using? If it's sunset, you have to merge the pr yourself :). I think restart kubelet trigger delete/add to make sure it sync .
Sorry, I didn't say
I think we still support 1.12 for now :) let's cherry pick it if not and wait for next 1.12 release.


Question: "//pkg/api/unversioned:go_default_library", "//pkg/client/cache:go_default_library", "//pkg/client/clientset_generated/internalclientset:go_default_library", <ins> "//pkg/kubelet:go_default_library", </ins> "//pkg/kubelet/api/v1alpha1/stats:go_default_library", "//pkg/kubelet/cm:go_default_library", <ins> "//pkg/kubelet/container:go_default_library", </ins> "//pkg/kubelet/dockertools:go_default_library", "//pkg/kubelet/images:go_default_library", "//pkg/kubelet/metrics:go_default_library",

Cluster logging assumes kubelet stores container log at a specific directory with specific name. Currently the path is: After the CRI work, the path will be: However, we don't have test coverage for this. We don't even know whether the log files are there or not. We should add a node e2e test for this, basically: 1) Create a container which prints logs 2) Check whether the expected log file is created. 3) Check whether the logs are collected in the file as expected. And it would be better to launch a pod to do the check instead of accessing local file system directly, which makes containerizing easier. /cc because you are collecting missing test. /cc
I will try to add a node-e2e test there
I think it's ok to add this for CRI. But there maybe problems on today's interface for rkt. If I'm not wrong, rkt does't store logs at same location with docker. cc/ Is this true?
Thanks! :)
After gets merged, we need to update the test to verify the cri log path.


Question: return nil, fmt.Errorf("failed to run in container - Exec setup failed - %v", err) } var buf bytes.Buffer <del> wrBuf := bufio.NewWriter(&buf) </del> startOpts := docker.StartExecOptions{ Detach:       false, Tty:          false, <del> OutputStream: wrBuf, ErrorStream:  wrBuf, </del> <ins> OutputStream: &buf, ErrorStream:  &buf, </ins> RawTerminal:  false, } <del> errChan := make(chan error, 1) go func() { errChan <- dm.client.StartExec(execObj.ID, startOpts) }() wrBuf.Flush() return buf.Bytes(), <-errChan </del> <ins> err = dm.client.StartExec(execObj.ID, startOpts) return buf.Bytes(), err </ins> } // ExecInContainer uses nsenter to run the command inside the container identified by containerID.

As far as I can tell, container liveness probes of type exec flat out don't work against clusters built at head. I had the kubelet always log the output of the exec'ed command, and it's always empty no matter what I exec, causing the probe to fail since it requires the output to be equal to the string "ok". If you want to test it, try adding a liveness probe of this form to any container: I also tried these lovely forms, to more closely match the liveness example under examples/: On the plus side, this may be an opportunistic time to switch over to
The reason that the e2e test for exec liveness probes hasn't caught this is that it only tests that containers are restarted when they fail liveness probes. It doesn't test that containers aren't restarted when their liveness checks should be succeeding. The e2e test's exec'ed commands also only ever produced empty output, rather than the expected "fail" output.
Is it possible to include this in a new pre-release, since it fixes a bug in health checking? I see it merged in master and I'm wondering what the policy is of releasing new code in a separate (0.16.*) release.


Question: "testing" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/validation/field" </ins> "k8s.io/kubernetes/federation/apis/federation" "k8s.io/kubernetes/pkg/api" ) <ins> func TestValidateClusterSpec(t *testing.T) { type validateClusterSpecTest struct { testName string spec     *federation.ClusterSpec path     *field.Path } successCases := []validateClusterSpecTest{ { testName: "normal CIDR", spec: &federation.ClusterSpec{ ServerAddressByClientCIDRs: []federation.ServerAddressByClientCIDR{ { ClientCIDR:    "0.0.0.0/0", ServerAddress: "localhost:8888", }, }, }, path: field.NewPath("spec"), }, { testName: "missing CIDR", spec: &federation.ClusterSpec{ ServerAddressByClientCIDRs: []federation.ServerAddressByClientCIDR{ { ClientCIDR:    "", ServerAddress: "localhost:8888", }, }, }, path: field.NewPath("spec"), }, { testName: "no host in CIDR", spec: &federation.ClusterSpec{ ServerAddressByClientCIDRs: []federation.ServerAddressByClientCIDR{ { ClientCIDR:    "0.0.0.0/32", ServerAddress: "localhost:8888", }, }, }, path: field.NewPath("spec"), }, } for _, successCase := range successCases { errs := ValidateClusterSpec(successCase.spec, successCase.path) if len(errs) != 0 { t.Errorf("expect success for testname: %q  but got: %v", successCase.testName, errs) } } errorCases := []validateClusterSpecTest{ { testName: "invalid CIDR : network missing", spec: &federation.ClusterSpec{ ServerAddressByClientCIDRs: []federation.ServerAddressByClientCIDR{ { ClientCIDR:    "0.0.0.0", ServerAddress: "localhost:8888", }, }, }, path: field.NewPath("spec"), }, { testName: "invalid CIDR : invalid address value", spec: &federation.ClusterSpec{ ServerAddressByClientCIDRs: []federation.ServerAddressByClientCIDR{ { ClientCIDR:    "256.0.0.0/16", ServerAddress: "localhost:8888", }, }, }, path: field.NewPath("spec"), }, { testName: "invalid CIDR : invalid address formation", spec: &federation.ClusterSpec{ ServerAddressByClientCIDRs: []federation.ServerAddressByClientCIDR{ { ClientCIDR:    "0.0.0/16", ServerAddress: "localhost:8888", }, }, }, path: field.NewPath("spec"), }, { testName: "invalid CIDR : invalid network num", spec: &federation.ClusterSpec{ ServerAddressByClientCIDRs: []federation.ServerAddressByClientCIDR{ { ClientCIDR:    "0.0.0.0/33", ServerAddress: "localhost:8888", }, }, }, path: field.NewPath("spec"), }, } for _, errorCase := range errorCases { errs := ValidateClusterSpec(errorCase.spec, errorCase.path) if len(errs) == 0 { t.Errorf("expect failure for testname : %q", errorCase.testName) } } } </ins> func TestValidateCluster(t *testing.T) { successCases := []federation.Cluster{ {

Right now it is possible to add a cluster for which it won't be possible to create a client. For example client-related code does: while validation checks only: cc:
switching this to 1.6 as it's too late for 1.5. ok? (please switch it right back if you disagree)
Not a release blocker for 1.7.
are you interested in fixing this?
of course, will do that.
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.8 milestone for more than 7 days: kind: Must specify exactly one of [, , ]. priority: Must specify exactly one of [, , ]. Removing it from the milestone. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: result := int32(0) for i := range pods { po := pods[i] <del> if po.Status.Phase != v1.PodRunning { continue } for j := range po.Status.InitContainerStatuses { stat := po.Status.InitContainerStatuses[j] result += stat.RestartCount } for j := range po.Status.ContainerStatuses { stat := po.Status.ContainerStatuses[j] result += stat.RestartCount </del> <ins> if po.Status.Phase == v1.PodRunning || po.Status.Phase == v1.PodPending { for j := range po.Status.InitContainerStatuses { stat := po.Status.InitContainerStatuses[j] result += stat.RestartCount } for j := range po.Status.ContainerStatuses { stat := po.Status.ContainerStatuses[j] result += stat.RestartCount } </ins> } } if *job.Spec.BackoffLimit == 0 {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I create a job with initcontainer with restartPolicy : when the initContainer exec failed and it will be restarted, the pod Phase will be pending all the time. but in JobController, the only care the running pod. So this pod will be always restarted. What you expected to happen: when restart count to backoffLimit, the job failed. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig apps
/assign


Question: }, }, }, <ins> Required: []string{"name"}, </ins> }, }, Dependencies: []string{},

FYI. Currently the field of TPR is marked as which is wrong, actually if you try to create a TPR with empty versions, it will cause api server panic. Versions should be necessary field of TPR, we should validate this somehow.


Question: key = "oops something went wrong with the key" } <del> // Record some output when items are deleted. outputSetLock.Lock() defer outputSetLock.Unlock() outputSet.Insert(key) </del> <ins> // Report this deletion. deletionCounter <- key </ins> }, }, ) // Run the controller and run it until we close stop. stop := make(chan struct{}) <ins> defer close(stop) </ins> go controller.Run(stop) // Let's add a few objects to the source. <del> for _, name := range []string{"a-hello", "b-controller", "c-framework"} { </del> <ins> testIDs := []string{"a-hello", "b-controller", "c-framework"} for _, name := range testIDs { </ins> // Note that these pods are not valid-- the fake source doesn't <del> // call validation or perform any other checking. </del> <ins> // call validation or anything. </ins> source.Add(&api.Pod{ObjectMeta: api.ObjectMeta{Name: name}}) } // Let's wait for the controller to process the things we just added. <del> time.Sleep(500 * time.Millisecond) close(stop) </del> <ins> outputSet := util.StringSet{} for i := 0; i < len(testIDs); i++ { outputSet.Insert(<-deletionCounter) } </ins> <del> outputSetLock.Lock() </del> for _, key := range outputSet.List() { fmt.Println(key) }

From shippable, I noticed there are several failed cases, for example, E0413 17:34:38. ] watch of *api.Pod ended with: Stop requested --- FAIL: Example (0.61s) got: want: a-hello b-controller c-framework FAIL coverage: 85.9% of statements FAIL 7.652s cc/
another example:
ah-- this is a different thing. I will fix this, too.
I'm seeing this from an unrelated change
Yeah I think that's the one I expected to see.


Question: }, { // PV requires external deleter <del> name:            "8-10 - external deleter", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </del> <ins> name:            "8-10-1 - external deleter when volume is dynamic provisioning", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </ins> initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents,

In my CSI controller I have enabled the feature gate HonorPVReclaimPolicy for csi-provisioner. Based on the documentation Finalizers can be on a PersistentVolume to ensure that PersistentVolumes having Delete reclaim policy are deleted only after the backing storage are deleted. The newly introduced finalizers and external- are only to dynamically provisioned volumes. I can see the finalizer set on the PV while creating static Persistent Volume. PV deletion is stuck due to this finalizer apiVersion: v1 kind: PersistentVolume metadata: creationTimestamp: "2023-11-08T11:39:24Z" deletionGracePeriodSeconds: 0 deletionTimestamp: "2023-11-08T11:54:31Z" finalizers: external- name: csi-infoscale-pv PV should be deleted when PVC is deleted. Persistent Volume yaml apiVersion: v1 kind: PersistentVolume metadata: name: csi-infoscale-pv spec: storageClassName: csi-infoscale-sc-prod persistentVolumeReclaimPolicy: Delete capacity: storage: 2Gi accessModes: ReadWriteOnce csi: driver: org.veritas.infoscale # Please provide pre-provisioned Infoscale volume name. volumeHandle: clust21432/vrtskube_dg-/vol1 fsType: vxfs Persistent Volume Claim apiVersion: v1 kind: PersistentVolumeClaim metadata: name: csi-infoscale-pvc-static spec: accessModes: ReadWriteOnce resources: requests: storage: 2Gi storageClassName: csi-infoscale-sc-prod volumeName: csi-infoscale-pv Same thing works fine on OCP. WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.


Question: }, { // PV requires external deleter <del> name:            "8-10 - external deleter", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </del> <ins> name:            "8-10-1 - external deleter when volume is dynamic provisioning", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </ins> initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents,

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-08-24T00:48:26Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v5.0.1 Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-08-24T00:42:11Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"} Vanilla K8s setup <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage maybe similar issues should go in
Opened similar issue in external-provisioner repo also There is no reply on that as well. Can you please help understand the behavior of the feature HonorPVReclaimPolicy.
If you want Persistent Volume (PV) to be deleted when the associated Persistent Volume Claim (PVC) is deleted, you donot need to use finalizers explicitly. Kubernetes handles this behavior by default. It is ok that if you use to get the desire output using Finalizer we can do it.
As per my understanding Please follow the steps : Add finalizer in metadata of as you did above need to make change in the controller which is handling PV deletion in .go file. See whether the controller administrator is working or not. I think it will work.
Finalizer is because the feature gate HonorPVReclaimPolicy for csi-provisioner is enabled. We are not adding the finalizer explicitly. From the documentation, "The newly introduced finalizers and external- are only to dynamically provisioned volumes." Not sure why is the finalizer getting set.


Question: }, { // PV requires external deleter <del> name:            "8-10 - external deleter", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </del> <ins> name:            "8-10-1 - external deleter when volume is dynamic provisioning", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </ins> initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents,

Also, for deleting PV it requires the annotation "" but this annotation is set in case of dynamic PVs. Should it be set explicitly for static PV?
This feature is alpha so the feature gate is not enabled by default. Did you enable the feature gate on kube-controller-manager?
Yes, i have enabled this feature.
external- is also applied to statically provisioned volumes. FYI
Hi Which version of the external provisioner is the driver using? Does the driver meet the requirements of the CSI spec on the DeleteVolume PRC? FYI: Is this problem occasional or permanently reproducible? Can you provide the provisioner logs at that time?
/triage needs-information
cc
cc
External provisioner used is : "csi-provisioner:v3.5.0" This problem is always reproducible Provisioner logs: I1123 12:22:38. 1 ] handleProtectionFinalizer Volume : &PersistentVolume{ObjectMeta:{csi-infoscale-pv -420a-4e9e-be82- 0 2023-11-23 12:21:18 +0000 UTC <nil<nilmap[] map[{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"csi-infoscale-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"2Gi"},"csi":{"driver":"org.veritas.infoscale","fsType":"vxfs","volumeHandle":"clust40559/vrtskubedg-/vol1"},"persistentVolumeReclaimPolicy":"Delete","storageClassName":"csi-infoscale-sc-prod"}} ] [] [ external-] [{kubectl-client-side-apply Update v1 2023-11-23 12:21:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"":{}}},"f:spec":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:csi":{".


Question: }, { // PV requires external deleter <del> name:            "8-10 - external deleter", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </del> <ins> name:            "8-10-1 - external deleter when volume is dynamic provisioning", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </ins> initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents,

":{},"f:driver":{},"f:fsType":{},"f:volumeHandle":{}},"f:persistentVolumeReclaimPolicy":{},"f:storageClassName":{},"f:volumeMode":{}}} } {csi-provisioner Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{"v:"external-"":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"":{}}},"f:spec":{"f:claimRef":{".


Question: }, { // PV requires external deleter <del> name:            "8-10 - external deleter", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </del> <ins> name:            "8-10-1 - external deleter when volume is dynamic provisioning", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </ins> initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents,

":{},"f:apiVersion":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:22:38 +0000 UTC FieldsV1 {"f:status":{"f:phase":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{ 0} {<nil} 2Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:org.veritas.infoscale,VolumeHandle:clust40559/vrtskubedg-/vol1,ReadOnly:false,FSType:vxfs,VolumeAttributes:map[string]string{},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:csi-infoscale-pvc-static,UID:-59a5-4f02-83ac-,APIVersion:v1,ResourceVersion:,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:csi-infoscale-sc-prod,MountOptions:[],VolumeMode:Filesystem,NodeAffinity:nil,},Status:PersistentVolumeStatus{Phase:Released,Message:,Reason:,},} I1123 12:22:38. 1 ] delete "csi-infoscale-pv": started I1123 12:22:38. 1 ] GRPC call: /csi.v1.Controller/DeleteVolume I1123 12:22:38. 1 ] GRPC request: {"volumeid":"clust40559/vrtskubedg-/vol1"} I1123 12:22:39.


Question: }, { // PV requires external deleter <del> name:            "8-10 - external deleter", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </del> <ins> name:            "8-10-1 - external deleter when volume is dynamic provisioning", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </ins> initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents,

1 ] GRPC response: {} I1123 12:22:39. 1 ] GRPC error: <nilI1123 12:22:39. 1 ] delete "csi-infoscale-pv": volume deleted I1123 12:22:39. 1 ] delete "csi-infoscale-pv": failed to remove finalizer for persistentvolume: Operation cannot be fulfilled on persistentvolumes "csi-infoscale-pv": the object has been modified; please apply your changes to the latest version and try again W1123 12:22:39. 1 ] Retrying syncing volume "csi-infoscale-pv", failure 0 E1123 12:22:39. 1 ] error syncing volume "csi-infoscale-pv": Operation cannot be fulfilled on persistentvolumes "csi-infoscale-pv": the object has been modified; please apply your changes to the latest version and try again I1123 12:22:39. 1 ] handleProtectionFinalizer Volume : &PersistentVolume{ObjectMeta:{csi-infoscale-pv -420a-4e9e-be82- 0 2023-11-23 12:21:18 +0000 UTC 2023-11-23 12:22:38 +0000 UTC 0xc0007f63d0 map[] map[{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"csi-infoscale-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"2Gi"},"csi":{"driver":"org.veritas.infoscale","fsType":"vxfs","volumeHandle":"clust40559/vrtskubedg-/vol1"},"persistentVolumeReclaimPolicy":"Delete","storageClassName":"csi-infoscale-sc-prod"}} ] [] [external-] [{kubectl-client-side-apply Update v1 2023-11-23 12:21:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"":{}}},"f:spec":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:csi":{".


Question: }, { // PV requires external deleter <del> name:            "8-10 - external deleter", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </del> <ins> name:            "8-10-1 - external deleter when volume is dynamic provisioning", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </ins> initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents,


Question: }, { // PV requires external deleter <del> name:            "8-10 - external deleter", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </del> <ins> name:            "8-10-1 - external deleter when volume is dynamic provisioning", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </ins> initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents,

":{},"f:apiVersion":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:22:38 +0000 UTC FieldsV1 {"f:status":{"f:message":{},"f:phase":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{ 0} {<nil} 2Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:org.veritas.infoscale,VolumeHandle:clust40559/vrtskubedg-/vol1,ReadOnly:false,FSType:vxfs,VolumeAttributes:map[string]string{},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:csi-infoscale-pvc-static,UID:-59a5-4f02-83ac-,APIVersion:v1,ResourceVersion:,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:csi-infoscale-sc-prod,MountOptions:[],VolumeMode:Filesystem,NodeAffinity:nil,},Status:PersistentVolumeStatus{Phase:Failed,Message:error getting deleter volume plugin for volume "csi-infoscale-pv": no deletable volume plugin matched,Reason:,},} I1123 12:22:40.


Question: }, { // PV requires external deleter <del> name:            "8-10 - external deleter", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </del> <ins> name:            "8-10-1 - external deleter when volume is dynamic provisioning", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </ins> initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents,

1 ] handleProtectionFinalizer Volume : &PersistentVolume{ObjectMeta:{csi-infoscale-pv -420a-4e9e-be82- 0 2023-11-23 12:21:18 +0000 UTC 2023-11-23 12:22:38 +0000 UTC 0xc0007f63d0 map[] map[{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"csi-infoscale-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"2Gi"},"csi":{"driver":"org.veritas.infoscale","fsType":"vxfs","volumeHandle":"clust40559/vrtskubedg-/vol1"},"persistentVolumeReclaimPolicy":"Delete","storageClassName":"csi-infoscale-sc-prod"}} ] [] [external-] [{kubectl-client-side-apply Update v1 2023-11-23 12:21:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"":{}}},"f:spec":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:csi":{".":{},"f:driver":{},"f:fsType":{},"f:volumeHandle":{}},"f:persistentVolumeReclaimPolicy":{},"f:storageClassName":{},"f:volumeMode":{}}} } {csi-provisioner Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{"v:"external-"":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"":{}}},"f:spec":{"f:claimRef":{".


Question: }, { // PV requires external deleter <del> name:            "8-10 - external deleter", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10", "1Gi", "uid10-1", "claim10-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </del> <ins> name:            "8-10-1 - external deleter when volume is dynamic provisioning", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-1", "1Gi", "uid10-1-1", "claim10-1-1", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, </ins> initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents,

":{},"f:apiVersion":{},"f:kind":{},"f:name":{},"f:namespace":{},"f:resourceVersion":{},"f:uid":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:22:38 +0000 UTC FieldsV1 {"f:status":{"f:message":{},"f:phase":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{ 0} {<nil} 2Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:org.veritas.infoscale,VolumeHandle:clust40559/vrtskubedg-/vol1,ReadOnly:false,FSType:vxfs,VolumeAttributes:map[string]string{},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:csi-infoscale-pvc-static,UID:-59a5-4f02-83ac-,APIVersion:v1,ResourceVersion:,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:csi-infoscale-sc-prod,MountOptions:[],VolumeMode:*Filesystem,NodeAffinity:nil,},Status:PersistentVolumeStatus{Phase:Failed,Message:error getting deleter volume plugin for volume "csi-infoscale-pv": no deletable volume plugin matched,Reason:,},}
Thank you I will try to fix it. /remove-triage needs-information /assign


Question: namespace: kube-system labels: k8s-app: fluentd-gcp-scaler <del> version: v0.1.0 </del> <ins> version: v0.2.0 </ins> addonmanager.kubernetes.io/mode: Reconcile spec: selector:

Forking from To summarize, here's what we observed: daemonset calls coming every minute from both fluentd-scaler and addon-manager (verified this by individually turning them on/off). Things we need to understand here: Is the daemonset object continuously toggling b/w 2 states? (we know here that it's RV is increasing continuously) If yes, what field(s) in the object are changing? IIRC value of some label/annotation (I think 'UpdatedPodsScheduled') is changing (probably related to 2 below) Also, why/should the fluentd-scaler send any api request if the resources are already set to the right value? pods are getting deleted and recreated by daemonset-controller when the scaler is enabled (as was also seen in ). Why this is happening? One thing to note here - all those delete calls are preceded by calls from respective kubelets (but maybe that's expected). cc /priority critical-urgent /assign
Marked this as release-blocker as at least some part of it seems like a correctness issue (i.e fluentd pods dying and getting recreated). Note also that this is causing increased mem-usage with daemonset-controller (depicted here - )
Thanks for filing this! The only thing that was changing (apart from ds/fluentd-gcp status) was the resource version.
I modified fluentd scaler to use --dry-run in to check whether there would be a change, but this still keeps on updating, even though there is no change. Apparently, the logic inside kubectl is broken...
Btw, addon manager sends tons of PATCH requests for all resources every minute anyway, so even though I will fix fluent-scaler not to send them, it won't solve everything.
Should this issue actually be closed? It was auto-closed by the bot, but nobody has confirmed that the actual issue is fixed.
I'll let to confirm (looks like he'll be back on Monday)
/status in-progress
Yes, we should keep this issue open until we fully understand what's going wrong and fix the root-cause. I believe much of the discussion wrt this is happening under related issue
ACK. In progress ETA: 20/03/2018 Risks: May delay the release if not satisfactorily resolved
Need a number of successive green runs, then close.


Question: namespace: kube-system labels: k8s-app: fluentd-gcp-scaler <del> version: v0.1.0 </del> <ins> version: v0.2.0 </ins> addonmanager.kubernetes.io/mode: Reconcile spec: selector:

So I took a look and seems like the fluentd-scaler version bump (which was supposed to fix) was merged b/w runs and of our . However, I don't see any visible improvement in controller-manager memory usage around that time: !
Also, I'm noticing such lines occasionally (once every 5-10 secs in the start and once every ~1min later on) from of latest run: And back when the fluentd-scaler wasn't doing anything useful (i.e before ), the had such lines only sparingly.
Now it isn't even sending requests (other than GET) to the apiserver. How do you generate that graph? Is it 99th percentile of memory usage over time? Or what exactly? One difference after introducing fluentd-gcp-scaler is that now there is a change occurring on the DaemonSet (once, but still). Also, what test is failing because of this? I'd like to take a look into the logs.
Answering your questions: I fetched the ResourceUsageSummaries for density test from historical runs (e.g ) and then obtained the 99% mem-usage for controller-manager for each run. Yes. More precisely, we're collecting mem-usage from controller-manager once every 1 min (or sth like that) and then taking 99th %ile of the resulting sample. Density and load tests. Where we're checking that mem usage of controller-manager is less than the given threshold (i.e 500MB). Ref: The tests are occasionally failing due to just crossing the threshold. Overall, the mem-usage had seen an increase back during the scaler change (see graph above). The job is this - https://k8s- Inside , you'll find all necessary logs (also a file named which has the resource-usage data).
The fluentd-scaler update eliminated fluentd restarts problem. Verified from controller-manager logs that we no longer see such logs that we were : This is great, and I believe that moves this out of release-blocker as we no longer face problems I mentioned in
That said, and me will continue looking into increase in mem-usage (which is not that serious). We'll start by verifying that turning the scaler off improves mem-usage.
Feel free to override me if you have a good reason :)
I agree that this doesn't sounds to be release-blocker.


Question: namespace: kube-system labels: k8s-app: fluentd-gcp-scaler <del> version: v0.1.0 </del> <ins> version: v0.2.0 </ins> addonmanager.kubernetes.io/mode: Reconcile spec: selector:

But I would like to see follow-ups at least for issues like: does this memory increase come from? what was claimed to be "no-op" changes for daemon set were causing problems [is that issues in kubectl, apiserver, somewhere else?] Something like that may hit us again in the (near?) future if we won't understand it now. The current solution is really volatile [if there is a bug somewhere we shouldn't workaround it on our side, but instead have this fixed] at the cluster start (maybe it's fine to leave as is, but I would like to see a discussion of that and a conscious decision being made here)
Definitely agree those are questions we need to answer.
Yup. 1) is tested in . 2) I believe apiserver sends watch events even when only resource version changes, which causes the daemonset controller to recreate the pods. So either: resource version shouldn't change when there is no change anywhere else in the resource apiserver shouldn't send watch events when only resource version is changing daemonset should check whether watch event it got is actually changing anything 3) Isn't that an inherent part of the way addon manager works? It creates all sorts of resources, some dependent on the other (e.g. cluster roles and cluster role bindings), all at once. If some dependency is not satisfied, it will get fixed during the next run.
This is the code that is supposed to protect us from situation that something must change when we write object to etcd (resource version is not stored in etcd): So there should never be a situation that only resource version is changing. If it is, there is bug and need to be fixed. - FYI Reconciliation is a different thing that creating something in one state only to change that to a different state by a different component. I'm not saying this is for sure wrong approach. But I would like to see a conscious decision process discussing pros-and-cons of that.
That's what I'd expect as well. This I'm not sure if should be correct behaviour as IIUC watchers need to receive every RV update for correctness reasons ( ) Maybe. But are we sure if RV is just increasing without any object change? For creation of addons, yes (that race will likely last only for starting few seconds).


Question: namespace: kube-system labels: k8s-app: fluentd-gcp-scaler <del> version: v0.1.0 </del> <ins> version: v0.2.0 </ins> addonmanager.kubernetes.io/mode: Reconcile spec: selector:

But this is wrt addon changing cluster resources, which can potentially create race at arbitrary time after cluster start (though your fix may have gotten rid of this).
Ok.. Seems like I missed Wojtek's comment above.
So I just checked the ds-controller code and it seems like we're updating the ds status only if this condition is being violated: Which means one of those fields is probably mismatching. I'll try to add some logs and run the test at --v=4 to get more info.
anything is possible, but we do have tests ensuring this works properly (no-op updates do not actually result in an update+resourceVersion bump in etcd):
So I ran some 100-node experiments against with more debug logs and I think I now understand the reason for this. The controller-manager logs and mainly show the following difference: This logs show-up when the scaler is active. And these pod deletions are happening as part of rolling-upgrade of daemonset to newer version (the one having resources set). You can see they're getting deleted one-by-one, which also explains why it takes so long (and so are happening even 30 mins after the cluster creation). And the reason why they're deleted one-by-one is because ds-controller has a rolling-upgrade strategy where by default. Proof: And the increased mem-usage FMU is because of the rolling-upgrade performed by ds-controller (over the prolonged period) - as it's the only difference I observe from logs Further, the reason why the RV was increasing was because of few fields 'NumUnavailable', 'NumAvailable', etc fluctuating while the rolling-upgrade
I spoke offline with and suggested that we should increase maxUnavailable for the fluentd daemonset to a large enough value so that we're not bottlenecked by it. My reasoning is: fluentd is not some user workload which will have some service downtime in case there are many unavailable at once fluentds on nodes are independent of each other (i.e one fluentd shouldn't need to wait for another node's fluentd to come up) we should try to make the scaling happen as fast as possible - otherwise it may be too late before they cope-up with increased log-load in the cluster I'm going to make that change and test it against my PR (thanks for pointing that we can change maxUnavailable directly in ds config).


Question: namespace: kube-system labels: k8s-app: fluentd-gcp-scaler <del> version: v0.1.0 </del> <ins> version: v0.2.0 </ins> addonmanager.kubernetes.io/mode: Reconcile spec: selector:

So increasing maxUnavailable indeed seems to help (ref: ): All the 101 replicas were deleted at once instead of one-by-one and happened pretty much within 2 mins of the original fluentd-ds seen by the controller: We should get that PR into 1.10.
Thanks for confirming this! Did increasing maxUnavailable also reduce c-m memory usage?
So the run was still over the threshold, but I was expecting it as I ran c-m with increased verbosity (which ). That said, I'll observe how the usages are once that PR gets in - and FMU it's still possible to have higher values due to the rolling-upgrade potentially happening during the test.
Is there anything else to do here or should this be closed now?
Let's close it once we're sure there are no more such failures. We may also need to bump the threshold. On Mon, Mar 26, 2018, 9:46 AM Kubernetes Submit Queue < :
So bumping maxUnavailable for fluentd rolling-upgrade in is causing couple of issues: Our latest 5k-node performance failed as part of BeforeSuite where we're checking that ~all kube-system pods are running or ready. The reason is because quite some fluentd pods were killed as part of the rolling-upgrade and we timed out before all those pods could come up Our 100-node performance runs due to high CPU/mem-usage of c-m observed during density test, in cases when the fluentd upgrade happens during the test - which is kind of expected So at this point, we might want to choose among one of the 2 options: Rollback , which comes at the cost of tradeoffs I explained in Increase our CPU/mem thresholds to eliminate those flakes in 100-node job and increase the "max-allowed-not-ready-pods" in kube-system ns to a higher value, so that our 5k-node test doesn't fail in BeforeSuite Thoughts?
TBH, I don't know why it was merged without waiting for opinions... I really vote for reverting that. We don't want this - this may mean making cluster "not fully usable" from user perspective.Which is not really what we want to test.
SGTM. Let's revert it and increase our thresholds for the time-being. The discussion about changing fluentd rolling-upgrades can continue to happen in the background.


Question: namespace: kube-system labels: k8s-app: fluentd-gcp-scaler <del> version: v0.1.0 </del> <ins> version: v0.2.0 </ins> addonmanager.kubernetes.io/mode: Reconcile spec: selector:

[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 5 days, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
This issue is now fixed after my above PRs. We're having continuous for the job


Question: if limitedMode { limitedNum-- } <ins> if len(msg.log) > 0 { isNewLine = msg.log[len(msg.log)-1] == eol[0] } else { isNewLine = true } </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Docker has a hardcoded 16K limit on log line length, which splits lines longer than that into multiple messages in the log file. in , however, doesn't account for this when is set to and prints the timestamp in front of every entry in the log file instead of every logical line. This leads to random timestamps appearing in the middle of log lines, potentially breaking the applications that parse these logs. What you expected to happen: inserts timestamps once per logical line, so original log data is preserved. Alternatively, allow API to request the logs in a sane format like JSON, which would allow merging the lines back and combining them with the timestamps more intelligently on the application side. How to reproduce it (as minimally and precisely as possible): Create : apiVersion: v1 kind: Pod metadata: name: log-lines spec: containers: name: main image: busybox command: ['sh', '-c', 'while sleep 1; do echo -n ; for i in $(seq 1 ); do echo -n A; done; echo ; done'] Start the pod with Fetch the logs with , or using the API () Observe random timestamps inserted in the middle of the log lines. Environment: Kubernetes version (use ):
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I will work on it. /sig cli /remove-sig node
The logs are being corrupted by the API endpoint in kubelet, so shouldn't it be in scope of SIG node?
The log is not corrupted by k8s. The same log could be got using "docker container logs -t CONTAINERID" I think you should post your requirement to Docker.


Question: if limitedMode { limitedNum-- } <ins> if len(msg.log) > 0 { isNewLine = msg.log[len(msg.log)-1] == eol[0] } else { isNewLine = true } </ins> } }

Please refer following code Line 50 ~ 52 if config.Timestamps { logLine = append([]byte(msg.Timestamp.Format(jsonmessage.RFC3339NanoFixed)+" "), logLine...) }
kubelet doesn't rely on Docker when reading logs, it reads the files itself (see the code linked in the issue). I don't think Docker would change the forced line splitting (see for example), so this should be fixed in Kubernetes.
Your are right and kubelet read log file from host such as /var/log/pods/pods/NAMESPACEPOD-NAMExxx/....(JSON format) I will post a PR for it. /remove-sig cli /sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .


Question: if limitedMode { limitedNum-- } <ins> if len(msg.log) > 0 { isNewLine = msg.log[len(msg.log)-1] == eol[0] } else { isNewLine = true } </ins> } }

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. are you or anybody on this thread are still interested to work on this? Please re-open if so. Meanwhile closing this ahead of bot doing it because the inactivity. /close
Closing this issue. <detailsIn response to : I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. Instructions for interacting with me using PR comments are available .


Question: if limitedMode { limitedNum-- } <ins> if len(msg.log) > 0 { isNewLine = msg.log[len(msg.log)-1] == eol[0] } else { isNewLine = true } </ins> } }

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen will fix it.
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: if limitedMode { limitedNum-- } <ins> if len(msg.log) > 0 { isNewLine = msg.log[len(msg.log)-1] == eol[0] } else { isNewLine = true } </ins> } }

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .


Question: if limitedMode { limitedNum-- } <ins> if len(msg.log) > 0 { isNewLine = msg.log[len(msg.log)-1] == eol[0] } else { isNewLine = true } </ins> } }

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I received a bug report about this issue. The root cause is that the buffer reads up to a new line. If the line is greater than 4096 bytes and '--timestamps' is turrned on the kubelet will write the timestamp and the partial log line. We will need to refactor the ReadLogs function to allow for a partial line read.


Question: func exportCustomMetricFromService(f *framework.Framework, consumerName string, metricValue int) *common.ResourceConsumer { serviceAnnotations := map[string]string{ "prometheus.io/scrape": "true", <del> "prometheus.io/path":   "/Metrics", </del> <ins> "prometheus.io/path":   "/metrics", </ins> "prometheus.io/port":   "8080", } return common.NewMetricExporter(consumerName, f.Namespace.Name, nil, serviceAnnotations, metricValue, f.ClientSet, f.InternalClientset, f.ScalesGetter)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Used the endpoint /BumpMetric to create fake metrics for Prometheus. When I tried to curl /metrics I got the following response: Bad request. Not a POST request What you expected to happen: I expected to see something like this: # HELP info message. # TYPE gauge 0. How to reproduce it (as minimally and precisely as possible): Following the docs, after having it running inside kubernetes: Anything else we need to know?: Looks like introduced this by mistake. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig testing


Question: args: args, // PodStart is used as default because waiting for a pod is the // most common operation. <del> timeout:  TestContext.timeouts.PodStart, interval: TestContext.timeouts.Poll, </del> <ins> timeout:      TestContext.timeouts.PodStart, interval:     TestContext.timeouts.Poll, consistently: consistently, </ins> } }

In and , I started seeing these flakes. The job in question has not been very flaky before so this is odd. In and , I started seeing these flakes. The job "Job should not create pods when created in suspend state" is the one that is flaking. Triage says a few days ago. No response No response /sig apps
Did some digging and I did not see any changes that could account for an increase of flakiness in the job controller code. There was a change made to the e2e tests: The logs say: The code is pretty simple so I wonder if we are hitting a case where sometimes the suspend condition is not set and we fail. Maybe we could add a consistent call here?
/triage accepted /priority important-soon
/cc
/reopen /assign do you mind monitoring the CI state for 1 or 2 days to confirm your PR really fixed it ?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It makes sense to me.
cc I think this issue has been fixed. Can we close it?
/close Thanks for the fix. The test is not flaky now.
Closing this issue. <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details