Question: // zero capacity, and the default labels. func (kl *Kubelet) getNodeAnyWay() (*v1.Node, error) { if kl.kubeClient != nil { <del> if n, err := kl.nodeLister.Get(string(kl.nodeName)); err == nil { </del> <ins> if n, err := kl.GetNode(); err == nil { </ins> return n, nil } }

What happened: A deployment is created that uses a after the cluster master is created but before the cluster worker nodes are fully initialized. During the initialization of the worker nodes, the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector. Depending on how long the nodes are available for scheduling without the necessary node labels for the deployment, the scheduler will start to spam with cluster with pods in status. This spamming stops once the worker nodes are fully initialized and the pods are scheduled successful. The exact timing of all of this is TBD. What you expected to happen: Pods in status are annoying and require manual cleanup. It would be preferred that the scheduler avoid this an simply issue warning events. How to reproduce it (as minimally and precisely as possible): See issue description. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.18.3 Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 4.15.0-96-generic Install tools: Managed service install Network plugin and version (if this is a network-related bug): N/A Others: N/A
Related to and
/sig scheduling
by spamming are you referring to scheduler logs? also, why are the nodes not labelled from the get go?
Just to confirm, all the pods in question are still in Pending state, yes? If the pods are in pending state, all of them will get scheduled eventually once the nodes get their labels.
The pods are not in state. They are actually in state. The pods eventually get scheduled but we are left with anywhere from a few to dozens of these state pods. Also, we are looking at labeling the nodes via kubelet flags to see if that helps.
/remove-kind bug
After we add some retry logic and wait time in this problem can be solved: As you can see this function returns initial node as the first request failed.
I also get hit by this, when single controller node gets rebooted and scheduler starts as static container before kubelet becomes ready.


Question: // zero capacity, and the default labels. func (kl *Kubelet) getNodeAnyWay() (*v1.Node, error) { if kl.kubeClient != nil { <del> if n, err := kl.nodeLister.Get(string(kl.nodeName)); err == nil { </del> <ins> if n, err := kl.GetNode(); err == nil { </ins> return n, nil } }

Can be cleaned up with:
/sig node
the spam cycle is this loop: "pod create, pod schedule, worker reject, pod failed" -- we don't have a lot of protection in higher level workload controllers to handle pods that fail at kubelet admission time versus pods that just dont create due to quota/admission. in general, is the spam causing any other stability issues? is pod gc pruning the pods?
I wonder if we can do a trick in node lister to mitigate any race condition here...
we could maybe do something similar to to ensure node lister has synced...
hacking on option to ensure if kubelet has a node lister with a valid kube client, we wait for it to sync at least once here which should mitigate this issue. see:
We noticed the problem on the OLM pod which is managed by the OLM operator. Right or wrong, the operator does not like seeing the pod go to status and no longer reconciles the pod. The end result is that manual intervention (i.e. pod deletion) is required to fix the problem.
sure... i would have thought the pod is failed in that instance and reached a terminal state. either way, i am looking at what i can do to mitigate this.
The problem persists in our environment as well, and the workaround cited above will not do anymore. Is there something in the works for this one?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: var k8sBinDir = flag.String("k8s-bin-dir", "", "Directory containing k8s kubelet and kube-apiserver binaries.") <del> var buildTargets = []string{ "cmd/kubelet", "cmd/kube-apiserver", "test/e2e_node/e2e_node.test", } </del> func buildGo() { glog.Infof("Building k8s binaries...") k8sRoot, err := getK8sRootDir() if err != nil { glog.Fatalf("Failed to locate kubernetes root directory %v.", err) } <del> cmd := exec.Command(filepath.Join(k8sRoot, "hack/build-go.sh"), buildTargets...) </del> <ins> cmd := exec.Command(filepath.Join(k8sRoot, "hack/build-go.sh")) </ins> cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr err = cmd.Run()

is failing with: Examples:
This is the failing line of code...
PASS: FAIL: Seems like this disappeared: Also this is flipped: we run the following
my leading theory is this is something in
The issue is that the above PR changed a bunch of to be and thus we no longer wait for getArchive() to complete before starting tests. And unfortunately the previous PR cannot be reverted
After merging, kubelet-gce-e2e-ci still failed with the same issue. reopen the issue. Currently the failure blocked the merge-bot.
did stop the bleeding. Now kubelet-gce-e2e-ci is green: and merge-bot is unblocked.
I think the "go arc" piece was a red herring. I suspect the issue is that the same PR tries to copy the binary into the archive directory, but PR stops building ginkgo, so the copy fails. That is why this line then fails...


Question: } } } <ins> func hasJump(rules []iptablestest.Rule, destChain, ipSet string) bool { match := false for _, r := range rules { if r[iptablestest.Jump] == destChain { match = true if ipSet != "" { if strings.Contains(r[iptablestest.MatchSet], ipSet) { return true } match = false } } } return match } </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Service loadbalancer source range don't work in ipvs proxy mode. Which result in gci-gce-ipvs CI failed. http://k8s- How to reproduce it (as minimally and precisely as possible): In gce environment. a service with backend. service type to service util service external IP is specified. the service from pod inside and pod out of the . Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): Install tools: cluster/kube- Others:
/kind bug /sig network /label ipvs
/area ipvs
The test seems to be flaky now. Can you take a look please?
I don't know why, but I think this flaky have the similiar reason with other normal service-test cases, like Those flaky are pretty annoying, problem is, I have run those cases in my own environment many times, never fail... : ( But still, I'll do my best to find the reason.


Question: deps = [ "//cmd/kubeadm/app/apis/kubeadm:go_default_library", "//cmd/kubeadm/app/constants:go_default_library", <ins> "//cmd/kubeadm/app/features:go_default_library", </ins> "//cmd/kubeadm/app/util:go_default_library", "//cmd/kubeadm/app/util/apiclient:go_default_library", "//pkg/api/legacyscheme:go_default_library",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently, IPVS proxier does not run any e2e test cases. What you expected to happen: I think it's important to create test suite for IPVS in upstream CI which will help a lot in moving IPVS existing alpha. Anything else we need to know?: Let's work together. /sig network
Self /assign
There are to ways to do this work: , but doesn't support any kube-proxy config right now, so I had this pr , and also, need support it too, see this pr: kubernetes/kubernetes-anywhere build cluster. See this pr: kubernetes/test-infra The second way is much easier and faster, but I was told they gonna deprecated the , so in the long term, we still need and 's support. If anybody have better way or have some suggestion about this, please let me know, I'll be really appreciate, Thanks.


Question: "terminationMessagePath":   "Optional: Path at which the file to which the container's termination message will be written is mounted into the container's filesystem. Message written is intended to be brief final status, such as an assertion failure message. Will be truncated by the node if greater than 4096 bytes. The total message length across all containers will be limited to 12kb. Defaults to /dev/termination-log. Cannot be updated.", "terminationMessagePolicy": "Indicate how the termination message should be populated. File will use the contents of terminationMessagePath to populate the container status message on both success and failure. FallbackToLogsOnError will use the last chunk of container log output if the termination message file is empty and the container exited with an error. The log output is limited to 2048 bytes or 80 lines, whichever is smaller. Defaults to File. Cannot be updated.", "imagePullPolicy":          "Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images", <del> "securityContext":          "Security options the pod should run with. More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/", </del> <ins> "securityContext":          "SecurityContext defines the security options the container should be run with. If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext. More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/", </ins> "stdin":                    "Whether this container should allocate a buffer for stdin in the container runtime. If this is not set, reads from stdin in the container will always result in EOF. Default is false.", "stdinOnce":                "Whether the container runtime should close the stdin channel after it has been opened by a single attach. When stdin is true the stdin stream will remain open across multiple attach sessions. If stdinOnce is set to true, stdin is opened on container start, is empty until the first client attaches to stdin, and then remains open and accepts data until the client disconnects, at which time stdin is closed and remains closed until the container is restarted. If this flag is false, a container processes that reads from stdin will never receive an EOF. Default is false", "tty":                      "Whether this container should allocate a TTY for itself, also requires 'stdin' to be true. Default is false.",

What happened: states that Container.SecurityContext applies to a Pod, which can be misleading since we already have PodSecurityContext. What you expected to happen: Documentation states that it applies to container How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment:
/sig api-machinery /kind documentation
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Not stale.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle rotten
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: apps "k8s.io/api/apps/v1" "k8s.io/api/core/v1" <ins> "k8s.io/apimachinery/pkg/api/errors" </ins> apierrs "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: DaemonSet has an e2e test running that has dubious purpose. What you expected to happen: The e2e test should be refactored and commented clearer as it will be part of the conformance.
To elaborate, we're concerned that the following may not actually ensure that DaemonSet recreates Failed Pods: For example, it seems possible that the Node might simply overwrite the Pod status with its own information if it hasn't yet seen the change to the Phase. Ideally, we should verify that the Pod UID we force to Failed no longer shows up in the DaemonSet's list of Pods after the DaemonSet becomes happy again.


Question: if itemOut.Status != api.StatusWorking || itemOut.Details == nil || itemOut.Details.ID == "" { t.Errorf("Unexpected status: %#v (%s)", itemOut, string(body)) } <ins> wait.Done() </ins> } func TestCreateNotFound(t *testing.T) {

Intermittent failures on Travis.
Hrm, this passes 100/100 locally for me. I'll look at the code and see if I see anything. --brendan


Question: // MinionList is a list of minions. type MinionList struct { JSONBase `json:",inline" yaml:",inline"` <del> Items    []Minion `json:"minions,omitempty" yaml:"minions,omitempty"` </del> <ins> // DEPRECATED: the below Minions is due to a naming mistake and // will be replaced with Items in the future. Minions []Minion `json:"minions,omitempty" yaml:"minions,omitempty"` Items   []Minion `json:"items,omitempty" yaml:"items,omitempty"` </ins> } // Binding is written by a scheduler to cause a pod to be bound to a host.

should have the JSON attribute name "items", not "minions"
Doh! Fix now and break clients?
Define Items and Minions and use Convertors to resolve it?
Sure. I'll make a fix.


Question: burst, containerLogsDir, osInterface, networkPlugin, runtimeHelper, httpClient, &NativeExecHandler{}, fakeOOMAdjuster, fakeProcFs, false, imageBackOff, false, false, true) dm.dockerPuller = &FakeDockerPuller{} <del> dm.versionCache = cache.NewVersionCache(func() (kubecontainer.Version, kubecontainer.Version, error) { return dm.getVersionInfo() }) </del> return dm }

The recently merged PR a version cache in dockertools, which makes the test  flaky. The reason is that injected error into fakedockerclient and then expected to see the error from right after that (See ). However, the error was consumed by the goroutine which periodically updates the version cache (See ). The docker daemon version is rarely updated, and we only need version once for each container creating, during which we also do container create, start, inspect #etc. 1) Do we care about the cost of the call so much that we have to introduce the complexity? 2) Kubelet periodically calls , can we rely on that to update the cache? Anyhow, at least, we should disable the cache or have a fake one in the unit test. :) /cc
I once considered move this cache to runtime level, while thus may need to change runtime interface. What if only cache apiversion?
We can only get via , so there will still be the data race. :) One option is to update the version cache in , now kubelet periodically call to check the runtime status. Another better option is to cache the version in layer, now is under our control, we can refactor it with an additional argument indicating whether to use cached value or get directly from the daemon. If we cache the version only for performance, I prefer the second option which is cleaner and more efficient. :)
OK, will look into this tonight
The unit test itself should not start a goroutine to update the cache (unless you're actually testing the goroutine). We should disable it anyway, as we often do in our unit tests. It also sounds like we don't need a goroutine dedicated for this. If we can tolerate the additional overhead from time to time, we can set a expiration time (e.g., 1 minute), and only call the runtime if the entry expires.
Totally agree. :) Maybe a new function in wrapping the update logic, or wrapping the logic in the .
expiration time sounds good in this case
This is happening quite frequently - increasing priority to P0.


Question: }) /* <del> Testname: allowPrivilegeEscalation=true. Description: Configuring the allowPrivilegeEscalation to true, allows the privilege escalation operation. A container is configured with allowPrivilegeEscalation=true and a given uid (1000) which is not 0. When the container is run, the container is run using uid=0 (making use of the privilege escalation). This test is marked LinuxOnly since Windows does not support running as UID / GID. </del> <ins> Release : v1.15 Testname: Security Context, allowPrivilegeEscalation=true. Description: Configuring the allowPrivilegeEscalation to true, allows the privilege escalation operation. A container is configured with allowPrivilegeEscalation=true and a given uid (1000) which is not 0. When the container is run, container's output MUST match with expected output verifying container ran with uid=0 (making use of the privilege escalation). This e2e Can not be promoted to Conformance as it is Container Runtime dependent and runtime may not allow to run. [LinuxOnly]: This test is marked LinuxOnly since Windows does not support running as UID / GID. </ins> */ It("should allow privilege escalation when true [LinuxOnly] [NodeConformance]", func() { podName := "alpine-nnp-true-" + string(uuid.NewUUID())

/kind cleanup /area conformance /area test /sig node /sig testing /sig architecture /assign /assign We have a number of tests that only have the tag that may make sense as tests, we should identify whether these tests cases meet (or could meet) the . We should open PR's to promote these if so. I would advise starting with the test/common cases
GitHub didn't allow me to assign the following users: mgdevstack, mkurylec. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /kind cleanup /area conformance /area test /sig node /sig testing /sig architecture /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The tests in test/e2enode don't actually get run as part of Conformance as near as I can tell. If we want to run them as part of conformance, they need to be migrated into test/e2e/common. Some of the tests in e2enode may not be able to be ported over because they assume a level access we don't have in a conformance environment.
regarding this testcase: [sig-storage] HostPath should support r/w [NodeConformance]: It uses hostPath and privileged=true, to read/write from /tmp folder. Which would you suggest is the best orientation to overcome the privilege mode setting and allow its promotion?
/assign
Regarding , the suite runs the test only if the kubelet configuration was required to have the QOS (cgroups-per-qos) enabled. This configuration for the kubelet is selected at the time the cluster is set up. Looking for where this might be configured found: where is set to true. For other settings it is also configured to true. where is set to true. BUT also found where it is mentioned that the flag is no longer available. Questions: would it be ok to promote this test to be conformance, even it can be skipped based on configuration of the cluster? And not supported in the containerized kubelet?
I would consider cgroups to be an implementation detail that Conformance shouldn't be verifying
It uses hostPath and privileged=true, to read/write from /tmp folder.


Question: }) /* <del> Testname: allowPrivilegeEscalation=true. Description: Configuring the allowPrivilegeEscalation to true, allows the privilege escalation operation. A container is configured with allowPrivilegeEscalation=true and a given uid (1000) which is not 0. When the container is run, the container is run using uid=0 (making use of the privilege escalation). This test is marked LinuxOnly since Windows does not support running as UID / GID. </del> <ins> Release : v1.15 Testname: Security Context, allowPrivilegeEscalation=true. Description: Configuring the allowPrivilegeEscalation to true, allows the privilege escalation operation. A container is configured with allowPrivilegeEscalation=true and a given uid (1000) which is not 0. When the container is run, container's output MUST match with expected output verifying container ran with uid=0 (making use of the privilege escalation). This e2e Can not be promoted to Conformance as it is Container Runtime dependent and runtime may not allow to run. [LinuxOnly]: This test is marked LinuxOnly since Windows does not support running as UID / GID. </ins> */ It("should allow privilege escalation when true [LinuxOnly] [NodeConformance]", func() { podName := "alpine-nnp-true-" + string(uuid.NewUUID())

Which would you suggest is the best orientation to overcome the privilege mode setting and allow its promotion? Michelle Au If my understanding is correct, on some platforms, writing to /tmp requires privileged. I don't see a way to to remove that requirement.
If my understanding is correct, on some platforms, writing to /tmp requires privileged. I don't see a way to to remove that requirement. Yeah writing to host doesn't seem like a thing we can expect of conformance tests. I wonder if it makes sense to tag this test with something to denote the requirement, ie: I see some tests in test/e2e_node use
The hostpath tests in already have a comment explaining this. Maybe the comment should be updated to be more clear that that is the reason why we can't add it to conformance.
That's /tmp on the host or in the container filesystem?
/tmp on the host
ping what else remains to be done before we close this out? I am guessing that we want to close , is there anything beyond that?
Once tests under (issue- ) and are promoted we are good to close this issue. Short update for reference- None of [NodeConformance] tests from "test/e2e_node" seems eligible for [Conformance] promotion at this moment due to node level dependencies. Remaining [NodeConformance] tests from "test/e2e" has following status - Waiting for to be closed and then below tests would be promoted to Conformance Needs to be Promoted Can not be promoted to Conformance as //Slow by design Can not be promoted due to Node level dependency
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen
/assign


Question: }) } <ins> func getParams(featureList map[string]bool) string { if features.Enabled(featureList, features.SupportIPVSProxyMode) { return "- --proxy-mode=ipvsn        - --feature-gates=SupportIPVSProxyMode=true" } return "" } </ins> func getClusterCIDR(podsubnet string) string { if len(podsubnet) == 0 { return ""


Question: glog.Infof("Starting APIServiceRegistrationController") defer glog.Infof("Shutting down APIServiceRegistrationController") <del> if !controllers.WaitForCacheSync("APIServiceRegistrationController", stopCh, c.apiServiceSynced, c.servicesSynced) { </del> <ins> if !controllers.WaitForCacheSync("APIServiceRegistrationController", stopCh, c.apiServiceSynced) { </ins> return }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Currently APIServiceRegistrationController watch service change and find the related APIServices related if any. Then it AddAPIService/RemoveAPIService for APIAggregator. I dig into the kube-aggregator and find it no need totally, because update APIServices status when related service or endpoint change. So if does not watch services, it will not miss any event for APIServices. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/cc
/kind enhancement /sig api-machinery
/assign
that does not follow. AvailableConditionController modifies the APIService object. APIServiceRegistrationController must still watch the APIService objects to observe those changes.
oh, I misread the description. You're saying APIServiceRegistrationController doesn't need to watch v1.Service objects


Question: var warnOnce sync.Once func newGCPAuthProvider(_ string, gcpConfig map[string]string, persister restclient.AuthProviderConfigPersister) (restclient.AuthProvider, error) { <del> // deprecated in v1.22, remove in v1.25 // this should be updated to use klog.Warningf in v1.24 to more actively warn consumers </del> warnOnce.Do(func() { <del> klog.V(1).Infof(`WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead. To learn more, consult https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins`) </del> <ins> klog.Warningf(`WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead. To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke`) </ins> }) ts, err := tokenSource(isCmdTokenSource(gcpConfig), gcpConfig)

We should drop the bit once in favor of a once gcp has a replacement. xref: /assign /milestone v1.24 /triage accepted /sig auth
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten /lifecycle frozen
I'm in the release 1.24 bug triage shadow. Does this issue still target release 1.24? If yes, would you please update the kind and priority fields for us to track it better?
Yes, definitely targeting 1.24 /priority important-soon /kind cleanup deprecation
would you please link the PR that solves this issue and update the status?
There is none yet. Waiting on GKE folks.
addresses this issue.


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package testutil import ( "strings" "testing" "k8s.io/component-base/metrics" ) func TestNewFakeKubeRegistry(t *testing.T) { registryVersion := "1.18.0" counter := metrics.NewCounter( &metrics.CounterOpts{ Name: "test_counter_name", Help: "counter help", }, ) deprecatedCounter := metrics.NewCounter( &metrics.CounterOpts{ Name:              "test_deprecated_counter", Help:              "counter help", DeprecatedVersion: "1.18.0", }, ) hiddenCounter := metrics.NewCounter( &metrics.CounterOpts{ Name:              "test_hidden_counter", Help:              "counter help", DeprecatedVersion: "1.17.0", }, ) var tests = []struct { name     string metric   *metrics.Counter expected string }{ { name:   "normal", metric: counter, expected: ` # HELP test_counter_name [ALPHA] counter help # TYPE test_counter_name counter test_counter_name 0 `, }, { name:   "deprecated", metric: deprecatedCounter, expected: ` # HELP test_deprecated_counter [ALPHA] (Deprecated since 1.18.0) counter help # TYPE test_deprecated_counter counter test_deprecated_counter 0 `, }, { name:     "hidden", metric:   hiddenCounter, expected: ``, }, } for _, test := range tests { tc := test t.Run(tc.name, func(t *testing.T) { registry := NewFakeKubeRegistry(registryVersion) registry.MustRegister(tc.metric) if err := GatherAndCompare(registry, strings.NewReader(tc.expected), tc.metric.FQName()); err != nil { t.Fatal(err) } }) } } </ins>

What would you like to be : Once a metric be deprecated, we should update the unit test accordingly if there is any. We need to work out a way for a unit test freely select codebase version, and also should make sure the unit test won't fail in the subsequent release. Why is this needed: Deprecated also needs to be protected. /sig instrumentation /assign


Question: } if status == health.Healthy { result = append(result, minion) <ins> } else { glog.Errorf("%s failed a health check, ignoring.", minion) </ins> } } return result, nil

Should be able to tell from logs if a minion isn't present because cloud provider didn't tell us about it, or if it's failing health checks.


Question: go_library( name = "go_default_library", srcs = [ <ins> "conntrack.go", </ins> "dns.go", "dns_common.go", "dns_configmap.go",

Which jobs are flaking: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are flaking: [sig-network] Services should be able to preserve UDP traffic when server pod cycles for a NodePort service Testgrid link: https://k8s- Reason for failure: /sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
CRI-O CI is also hitting this failure. We have been unable to reproduce locally. our CI runs on aws
hmm, I only see 1 failure out of 4 and seems that the connectivity didn't work at all
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/assign
GitHub didn't allow me to assign the following users: JacobTanenbaum. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Saw the test flaking recently: Wondering if this is just a timeout issue.


Question: go_library( name = "go_default_library", srcs = [ <ins> "conntrack.go", </ins> "dns.go", "dns_common.go", "dns_configmap.go",

Maybe we can increase the timeout and/or switch to a TCP based test:
the test should use UDP, and UDP is unreliable by definition ... however is curious that most of the failures happen because it is not able to communicate with the first pod I think that is something that should be investigated, I've analyzed some occurences and it works fine for the second pod, however, the UDP losses should be totally random
Hi everyone, yeah so.. collecting some data on this... using a two node kind cluster, locally, with the antrea plugin, it passes using a 4 node AWS cluster with the antrea CNI plugin, it failed using a two node kidn cluster locally with calico, it passes So I guess the best reproducer is to run these tests, for now, in the cloud.... Now that I have a reproducer for this maybe can live debug it some time next wk with one of yall :)
bear in mind that this test is connecting to a nodePort in the node IP, the test is functionally correct, it may be flaky due to the unreliable UDP nature or due to the pod scheduling, but if is failing constantly it has to be environmental.
Second failure after two weeks of green 5k runs:
yep, :/ https://k8s-|UDP /assign
/reopen I'm afraid this is going to be challenging :smile: The difference now is that we are testing NodePort and ClusterIP If both fail, we can blame the test for sure, If only NodePort fails, we need to check the job, if GCE or KIND If only ClusterIP fails, ... :thinking:
Reopened this issue. <detailsIn response to : /reopen I'm afraid this is going to be challenging :smile: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're hitting this really frequently in 5k-node tests. So I started looking a little bit into that.


Question: go_library( name = "go_default_library", srcs = [ <ins> "conntrack.go", </ins> "dns.go", "dns_common.go", "dns_configmap.go",

I don't have full understanding of what exactly is happening, but I already see a couple things that may be causing issues: when creating the first pod, you wait for corresponding endpoint for that being created; but that doesn't mean that kube-proxy was actually programmed correctly at that time (in general it isn't); given iptables programming is rate-limitted, it may easily happen that actually programming happens 10s+ later That explains some delays, but it doesn't explain e.g. this run: where iptables were programmed at but the test didn't respond until If someone more familiar with networking stuff could look into it, it would be great. - can you help triaging ^^ for deletions it's even worse, because you don't even wait for endpoints to be there (i.e. Endpoints can still contain the old pod when the test start pinging to look for the second one) So I claim that timeouts are visibly to low in this test, but I suspect it's not everything. I will send out the first PR that tries to improve it in couple minutes.
thanks wojtek, have been slammed but Ill try to collect more data on it this wknd as well.
It seems that my PR didn't really help much. We should dig deeper, but I would need someone with more networking expertise to look a bit into
Actually - it seems only failures from one group are on non . So maybe that's actually related. The non-kind related ones look like:
OK - for this one, I actually know what is happening - I will send out a PR with a fix for this too.
for the KIND jobs already increased the resource requests on some of the jobs, there are issues on KIND jobs with kube-proxy not able to take the iptables lock and not being able to create the iptables rules for the services until the next resync
is to fix the flakes I mentioned above; all other flakes seem to be kind-oriented to me (which I'm not super familiar with).
We did not adjust the CPU on all jobs yet, pending how to manage that on prow ... I do see just a few other flakes at the bottom of the triage dashboard that look like timeouts instead.
Actually - almost all failures are effectively timeout. The top one that is is actually also a timeout. Didn't look into kind failures, but this may easily be related to iptables issues.


Question: go_library( name = "go_default_library", srcs = [ <ins> "conntrack.go", </ins> "dns.go", "dns_common.go", "dns_configmap.go",

After my second deflaking PR, we seem to be in the stage where: tests are failing on dozens of different things (probably iptables-issues related) tests are failing only on NodePort tests on "Failed to connecto to backend 1" which it would be great if someone else could look into (kube-proxy seemed to be programmed correctly based on logs)
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
This has failed again - Linking to specific part of log where timeout occurred .... we see this
So reviewing the above logs note how in STEP: checking client pod connected to the backend 1 on Node IP 10.138.0.5 28 attempts are made all in the 25th second of the 2 minute at 11am Should this Step back off and wait a few seconds in between tries? So looking at this code we see the following
From reading ... the intention expressed in the test code looks good to me. It is not clear to me why there is a mismatch between the 28 tries in the POD Client Logs and the expressed intention running the logContainsFn(podBackend1) every 5 seconds still digging ....
we are trying to catch one failure in this PR that has additional debugging, it dumps the conntrack table of the nodes on a failure
for those following the issue, we've extra debugging to the test so it dumps the whole conntrack table once it finish. We should look that there is NO stale entries, connections from the pod client to the deleted endpoint) and there is conntrack entries to the new endpoint from the pod client.
- we have runs with the conntrack dump now, e.g. this one: would you be able to take a look into it?
not yet :/ , but I think tomorrow can have some time
cc
For anyone reproducing this issue btw, you need a multinode k8s cluster (it isnt doable on kind i guess bc less likeliehood of network failures when running local) -
Curious about two potential hypotheses to fix this, have we looked into either of these? 1) can we confirm that is turned on/off? 2) is it possible that scaling the number of coredns replicas could help ?


Question: go_library( name = "go_default_library", srcs = [ <ins> "conntrack.go", </ins> "dns.go", "dns_common.go", "dns_configmap.go",

3) Would simulating this in kind by overloading coredns be a possibility as a way to make a more reproducible bug model?
regarding 1) kubeadm does not ship nodeLocalDNSCache I don't think GCE kube-up does by default either
ok, so now im curious about (2) - i.e. what if we just bumped coredns replica count?
interesting, :thinking: why do you think is a DNS thing? the netcat command that polls the services use the IP addresses not the hostname. I think that it has to be related to some iptables/conntrack race or similar,at least in KIND jobs we know that we have an iptables problem that affects network tests, that has to be fixed before this because it affect this test You can instrument the test locally with this and dump the state of iptables and conntrack on failure if you are able to reproduce it
i figured DNS bc i believe its known that conntrack and DNS at scale can get very messy, right ?
one other curious thing about this test is that we only create 2 pods in the recycling event for the svc. that likely means that any real error is hidden by chance..... If we created a 3rd pod, then we might potentially double the failure rate thus making the flakyness go away and expose the real underlying issue .
There's an invocation of dumpConntrack(cs) here [sig-network] Conntrack should be able to preserve UDP traffic when server pod cycles for a ClusterIP service
is updating the logging to run in an infinite loop ..
This is improving in KIND with the new changes we rolled out yesterday, let's see how it evolves this week
- which changes ? Are you saying this is no longer reproducible on Kind?
there was a lot of noise in the CI signal, especially in KIND jobs due to the CNI portmap bug we finally solved yesterday now we should have a better signal and we can go to the bottom because probably the failure will be "legit" and not a consequence, so this comment should apply
Hi ! I ran this a few times on kind and it passed, but I see , so I'm going to try the loop approach you suggested. Here's what I'm using to test:
quick note, <- potential motivation to possibly have runnable on some subset of tests in upstream eventually, i.e.


Question: go_library( name = "go_default_library", srcs = [ <ins> "conntrack.go", </ins> "dns.go", "dns_common.go", "dns_configmap.go",

so we can do things like to reproduce this sort of thing. matt did you get around to the 'try-till-it-fails' scenario?
I haven't been able to get it to fail on kind yet, -- 100 cycles and 0 failures on a three-node kind cluster! I'm letting the loop run for a few hours, will check back in when I have that data. Is it worth running this test on a non-kind environment?
I ran for several hours on my mac, through 1144 iterations no error yet. My conclusion: this test isn't flaky on KinD. Possible next step: run a large number of iterations of the same test on a GKE cluster.
Flakes on kind are also happening:
not many since the portmap fix on the 14th though ( / ) <img width="1111" alt="Screen Shot 2020-07-23 at 11 45 46 PM" src="https://user-"
Agree - it's much better than it was before.
I was analyzing this failure, the problem is that kind jobs does not have verbosity enough in kube-proxy We can see that the packet was never replied, this is the NAT entry from client -nodePort in the node with the client This is the same flow, but in the node with the server and the pod-server-1 seems to be healthy we need more logs :eyeglasses:
We have an open ticket to bump those in CI, it's just been lost in the noise. We should still do that. Antonio Ojea <:
If you are busy just assign it to me, I'll need this verbosity soon :)
It's in the kind repo, shouldn't be too hard to find, we have ~60 open atm. Antonio Ojea <:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I made another attempt to reproduce these failures locally on KinD, but still no success. Here's what I did: create a KinD cluster run two instances of sonobuoy at the same time This took ~4 hours to run; all the kube tests passed and sonobuoy reported all successes.


Question: go_library( name = "go_default_library", srcs = [ <ins> "conntrack.go", </ins> "dns.go", "dns_common.go", "dns_configmap.go",

Would love to help out in reproing this, I'm at a loss for good next step so any suggestions welcome!
that's the problem with these flakes, they are not easy to repro, and with this low rate of failures it may be caused for combination of running tests, environmental problems (cpu, memory, ...) , ... I always start for checking the current failures and try to find patterns
Let's take a look to one occurrence The tests fails because can't connect to the first pod If we check the code of the tests, we can see that it always uses the port as source port, we can use that to identify the nodePort assigned interesting thing here, we are source nating the pod ip, and we don't see any other conntrack entry The test verifies it has endpoints however, kube-proxy is using endpointlsices since this version but it has no entries so the iptables rules to forward the traffic are not there Seems we have to adapt this function in the e2e framework to use endpointslices cc: EDIT seems that this function is being used by all the network tests that can cause flakes if the endpoint is updated before the endpointslice (we switched recently kube-proxy to use the later)
Great catch. Ideally this function should validate if both Endpoint and appropriate EndpointSlice objects(s) exist.
Actually - let me take it and send a PR for it.
is out for review
Interesting failures here It fails to connect to the first pod
Hmm - I took a quick look at kube-proxy logs and these look fine to me....
I clicked too fast, sorry ... I was looking at the wrong service and I though I've found something
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.


Question: go_library( name = "go_default_library", srcs = [ <ins> "conntrack.go", </ins> "dns.go", "dns_common.go", "dns_configmap.go",

just saw it as well
can you please add links to the job with the failures, I just merged new code to improve the logging 2 days ago
ok, here there is a new occurrence and with the new debug logs is easier to follow has unreplied conntrack entries: looking at the other side The iptables rules are there but no conntrack entries related to 10.244.2.32
In I revert PR with debug info as I think that it likely caused significant memory growth in scalability tests, but we have one run that has finished without being killed by OOMKiller (I increased memory limit for that jobs as a mitigation). Maybe logs from this run will be useful for debugging:
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Two more flakes today:
It seems that the packet never gets DNATed in the receiving host, so it does not reach the pod in the node. There is the same test that is using ClusterIP instead of NodePort that is more reliable. This is being a tough one
/triage accepted
Yesterday's flake - I don't think I'm seeing the log messages from the latest log PR you made ? Or are the prow logs not logging at a high-enough level?
Logs are there, please bear with me: The error on the test is: click on the job failure and you'll see a larger output where you can see the server pod name and get the node where the pod is running: and the client-pod name and its node: and the flow we are looking for: with the node-name we can get the kube-proxy logs there, clicking in artifacts: and you can see the "new" conntrack log entries : and in the client: wow, there is a kernel bug there , can it be related? :thinking:
wohoo - some progress :) I'm not an expert here, but I think it can be? Adding here - they may be able to confirm/reject the hypothesis.
This kernel panic may be a coincidence, maybe this is the reason
Adding some recent failures:
Spotted another flake today:
Another flake in the 5k job:


Question: // Create a deployment to delete nginx pods and instead bring up redis pods. deploymentName := "redis-deployment" Logf("Creating deployment %s", deploymentName) <del> _, err = c.Deployments(ns).Create(newDeployment(deploymentName, replicas, deploymentPodLabels, "redis", "redis")) </del> <ins> _, err = c.Deployments(ns).Create(newDeployment(deploymentName, replicas, deploymentPodLabels, "redis", "redis", extensions.RollingUpdateDeploymentStrategyType)) </ins> Expect(err).NotTo(HaveOccurred()) defer func() { deployment, err := c.Deployments(ns).Get(deploymentName)

There's some practical limit on the number of concurrent access tokens that can be used in a Kubernetes cluster before it begins to spend all of its time in the authn stack verifying them. Absent fixes to it was < 500; with the fixes linked there it's still not infinitely scalable-- I still hit (what I believe to be) with ~ 5k unique tokens @ ~700QPS. This request is to this dimension an official name & definition, and list it with others, e.g. here: it it have a useful value (I think it should be 10k) /kind bug /sig scalability
/cc
oh and 4. add tests which verify it is as high as believed.
cc since projected tokens will increase the number of unique tokens
/sig auth
LRUs are not good at scaling and sharding has some major pitfalls as pointed out. I would suggest that we try to drop the LRU cache for successful authentication results in the token cache. The main benefit of using an LRU here is bound cache size. Entries in the token cache have a fast expiration independent of the LRU policy. The records probably take up O(2kb) (guesstimate) of memory. Given that these are successful attempts with presumably valid tokens, I don't think we are getting any benefit from the LRU. If we don't have a constraint on fixed cache size, we can back the token cache with a sync.Map and run a single background goroutine to periodically walk the map and remove expired entries. This would fix the problematic locking and eviction due to nonuniform distribution of tokens over shards. The failure cache is less important and we don't cache failures by default. If we consider this the "slowpath", we can probably leave it as. If that's not good enough, we have a couple options (with various tradeoffs) to improve it.
If off by default is not enough, more reasoning why the failed cache is not super useful: it might help with an accidental DoS but it doesn't prevent anything actively malicious DoS. I think we can focus our improvements on the success cache.
/assign He is already working on adding an explicit test for it.
has taken it over and is actively working on this. Unfortunately, I cannot assign this issue to him as he's not a k8s member yet
It's not closed - all testes must be submitted and appropriate documentation must be
Thanks everyone!!


Question: t *testing.T, expectedDetachCallCount int, fakePlugin *volumetesting.FakeVolumePlugin) { <del> if len(fakePlugin.Detachers) == 0 && expectedDetachCallCount == 0 { </del> <ins> if len(fakePlugin.GetDetachers()) == 0 && expectedDetachCallCount == 0 { </ins> return } err := retryWithExponentialBackOff( time.Duration(5*time.Millisecond), func() (bool, error) { <del> for i, detacher := range fakePlugin.Detachers { </del> <ins> for i, detacher := range fakePlugin.GetDetachers() { </ins> actualCallCount := detacher.GetDetachCallCount() if actualCallCount == expectedDetachCallCount { return true, nil

Looks like similar issue with
That's the PV controller this is the attach controller, thus separate issues.


Question: return 1 fi <ins> # provide `--pull` argument to `docker build` if `KUBE_BUILD_PULL_LATEST_IMAGES` # is set to y or Y; otherwise try to build the image without forcefully # pulling the latest base image. local DOCKER_BUILD_OPTS=() if [[ "${KUBE_BUILD_PULL_LATEST_IMAGES}" =~ [yY] ]]; then DOCKER_BUILD_OPTS+=("--pull") fi local -r docker_build_opts="${DOCKER_BUILD_OPTS[@]}" </ins> for wrappable in "${binaries[@]}"; do local oldifs=$IFS

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: when I run , found bound variable err. What you expected to happen: Build success How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ):Linux paasn1 4.14.78-300.el7.Centos.x8664 SMP Sat Jan 5 15:49:43 CST 2019 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others: GNU bash version 4.2.46(2)-release (x8664-redhat-linux-gnu) Copyright (C) 2011 Free Software Foundation, Inc.
/sig release


Question: input MicroTime }{ {MicroTime{}}, <del> {DateMicro(1998, time.May, 5, 1, 5, 5, 50, time.Local)}, </del> <ins> {DateMicro(1998, time.May, 5, 1, 5, 5, 1000, time.Local)}, </ins> {DateMicro(1998, time.May, 5, 5, 5, 5, 0, time.Local)}, }

Found some error logs when sending series events to apiserver: seems the later eventTime differed from the first one in series events After investigation, I found the problem was in the marshal. MicroTime is supposed to work on microsecond level precision, but init with nanosecond level precision, in this case, generates microsecond level precision data and generates nanosecond level precision, so when unmarshalled, the new time is not considered the exactly same as the original time (they are different at nanosecond level) MicroTime marshal should work at same precision for json and proto, so unmarshal can generate the exactly same MicroTime result: This causes the series events won't be updated on api-server after created master <details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign
Good catch... we'd probably need to truncate to microseconds when marshaling and unmarshaling proto to ensure any already persisted data is fixed up when read. Something like this:
Thanks for advice, going to send a PR soon


Question: Hard_rename log message [FILTER] <ins> Name        modify Match       winevt.raw Hard_rename Message message [FILTER] </ins> Name         parser Match        kube_* Key_Name     message

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --On Windows 20H2 we are observing crashloops with fluent-bit due to a malformed Windows Event Log entry in the Security log. Fluent-bit to not crash loop. Bring up a GCE Windows cluster on Windows 20H2. You may see a loss of logs. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted


Question: return &os.PathError{Op: "fstat", Path: file.Name(), Err: err} } <del> // Skip chown if uid is already the one we want. if int(s.Uid) == u.Uid { </del> <ins> // Skip chown if uid is already the one we want or any of the STDIO descriptors // were redirected to /dev/null. if int(s.Uid) == u.Uid || s.Rdev == null.Rdev { </ins> continue }

opencontainers/runc has fixed three CVEs: Latest runc is pulled by k/k Check runc version in file No response n/a n/a <details</details<details</details<details</details<details</details
/assign /sig security architecture release
/triage accepted


Question: err = gce.waitForRegionOp(op, region) if err != nil { glog.Warningf("Failed waiting for Forwarding Rule %s to be deleted: got error %s.", name, err.Error()) <ins> return err </ins> } } op, err = gce.service.TargetPools.Delete(gce.projectID, region, name).Do()

All of our external load balancer tests failed for the better part of a day due to the GCE project that our tests run in hitting its quota for number of target pools. While the target pools have been deleted, the fact that we hit our quota indicates that there must be a leak somewhere, and we need to plug it.
For reference, the previous issue was
Could the leakage be related to ? The quota exhaustion occurred about 22 hrs after that.
I'd guess that this is due to GetTCPLoadBalancer returning that the load balancer doesn't exist if the forwarding rule is gone, ignoring the target pool, while the service controller considers its deleting done if the cloud provider says that the load balancer doesn't exist. I that logic to the cloud provider because the other implementations of DeleteTCPLoadBalancer weren't/won't be idempotent in the sense of returning ok if the LB doesn't exist. To fix it, we'll either need to make sure all Delete implementations are idempotent and remove the Get call in the controller, or add a method along the lines of IsTCPLoadBalancerDeleted to ask the cloud provider whether all pieces of it have been completely cleaned up. We can't make GetTCPLoadBalancer check that because it's relied on to tell us whether or not the balancer is fully created.
Copying over my commentary from on why the GKE e2e tests are leaking so many forwarding rules and target pools vs the GCE tests:
Is there a way for the services test to wait for the service controller to clean up the external resources before finishing (without just doing a sleep)?
Not without polling GCE directly for whether the resources still exist, although that may be worth it as a workaround. What would be perfect is that clean-up-all-cloud-provider-resources API method that we talked about a couple months ago.
Can we not perhaps use GCE namespaces or projects as a cheap way of getting this here?
I don't believe so. We can't use projects due to the fact that projects can't be programatically created (they're intended to be long-lived). I'm not aware of anything along the lines of a GCE namespace other than networks, which won't work because target pools and forwarding rules unfortunately don't seem to be scoped to networks.


Question: err = gce.waitForRegionOp(op, region) if err != nil { glog.Warningf("Failed waiting for Forwarding Rule %s to be deleted: got error %s.", name, err.Error()) <ins> return err </ins> } } op, err = gce.service.TargetPools.Delete(gce.projectID, region, name).Do()

The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.


Question: err = gce.waitForRegionOp(op, region) if err != nil { glog.Warningf("Failed waiting for Forwarding Rule %s to be deleted: got error %s.", name, err.Error()) <ins> return err </ins> } } op, err = gce.service.TargetPools.Delete(gce.projectID, region, name).Do()

I'll periodically check to see whether anything is leaked after this point. I apologize if the deletions affected any running tests - everything looked to only be from GKE and parallel e2e runs.


Question: <ins> /* Copyright 2016 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package core import ( "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/runtime" ) func addDeepCopyFuncs(scheme *runtime.Scheme) { if err := scheme.AddGeneratedDeepCopyFuncs( api.DeepCopy_api_DeleteOptions, api.DeepCopy_api_ExportOptions, api.DeepCopy_api_List, api.DeepCopy_api_ListOptions, api.DeepCopy_api_ObjectMeta, api.DeepCopy_api_ObjectReference, api.DeepCopy_api_OwnerReference, api.DeepCopy_api_Service, api.DeepCopy_api_ServiceList, api.DeepCopy_api_ServicePort, api.DeepCopy_api_ServiceSpec, api.DeepCopy_api_ServiceStatus, ); err != nil { // if one of the deep copy functions is malformed, detect it immediately. panic(err) } } </ins>

I started the federation-apiserver and went to and got the following output: Expected: I should have got an empty v1.ServiceList Any idea on what we are doing wrong? Have you guys seen this error?
I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote:
Deep copy generation was slightly broken as well - you may want to double check that they are registered. On May 23, 2016, at 5:50 PM, Daniel Smith wrote: I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote: — You are receiving this because you are on a team that was mentioned. Reply to this email directly or view it on GitHub
I got the same error with latest branch, as I was swamping on service controller, so I use a previous version and worked around it.
deep copies were indeed missing, but adding them hasnt fixed the issue. Will debug more. is my PR to add the deep copies cc Please feel free to file issues for any such problem you face. We need to fix all these issues. Its good if we know about them early. Thanks!
fixes this


Question: require ( github.com/gogo/protobuf v1.3.2 google.golang.org/grpc v1.51.0 <ins> k8s.io/klog/v2 v2.80.1 </ins> ) require ( <ins> github.com/go-logr/logr v1.2.3 // indirect </ins> github.com/golang/protobuf v1.5.2 // indirect github.com/google/go-cmp v0.5.9 // indirect golang.org/x/net v0.4.0 // indirect

generation for local KEK based on counter; also rotate once a week even if the counter does not hit 2 million? using AES-GCM encryption of local KEK ref: The implementation would be to the directory.
/sig auth /triage accepted
/assign
GitHub didn't allow me to assign the following users: ibihim. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
thank you !
/milestone v1.26
Hi bug-triage shadow team member for 1.26 here. Looks like this issue is in the 1.26 milestone, but has been moved to 1.27. Is there something else or should this issue go to 1.27 as well? Thank you!
I just forgot to move this issue /milestone v1.27
/reopen is the partial implementation.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
My suggestion on the reference implementation, after reading 's comments is like so... The GRPC server, which implements the expects an implementation that satisfies . The local key logic can be a middleware. So it expects a it can fallback to, if it doesn't know a key or needs to encrypt its own local KEK. So the local implementation would be need to: have a storage of s have a reference to the current in use (if we put it into the cache, it might vanish) have a reference to another , which should be the remote KMS. So the sub-stories could look like: write Middleware write Transformers that can get exhausted (by time, by usage) write 's storage write some dummy for remote KMS? I have some sketches of what I am talking about .


Question: require ( github.com/gogo/protobuf v1.3.2 google.golang.org/grpc v1.51.0 <ins> k8s.io/klog/v2 v2.80.1 </ins> ) require ( <ins> github.com/go-logr/logr v1.2.3 // indirect </ins> github.com/golang/protobuf v1.5.2 // indirect github.com/google/go-cmp v0.5.9 // indirect golang.org/x/net v0.4.0 // indirect

I would implement that in terms of the interface so that it can be used both in testing (via a local key) and in real code (by delegating to a HSM).
Hi bug-triage team member for 1.27 here. This issue is still open but has had no recent activity, and no active PRs. Is everything on track for the Code Freeze in March 14th? Thank you!
/milestone clear
Could be closed, right? :)
We still need metrics and tracing support for the reference implementation (though it is all a lot less important now).
/retitle [KMSv2] Add metrics and tracing in for plugins
This is duplicate of /close
Closing this issue. <detailsIn response to : This is duplicate of Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <del> apiVersion: v1 kind: ReplicationController </del> <ins> apiVersion: extensions/v1beta1 kind: Deployment </ins> metadata: labels: name: openshift name: openshift spec: replicas: 1 <del> selector: name: openshift </del> <ins> selector: matchLabels: name: openshift </ins> template: metadata: labels:

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Vagrant OS (e.g. from /etc/os-release): CoreOS Kernel (e.g. ):Install tools: Vagrant & Others: What happened: What you expected to happen: Success How to reproduce it (as minimally and precisely as possible): Followed instructions at Tried to run Received: Anything else do we need to know: From googling, it seems several of the fields in the config are for when using OpenShift origin. But creating the replication controller is a prerequisite to launching OpenShift Origin. Chicken & egg problem.


Question: } } <ins> // genEnumWithRuleAndValues creates a function that accepts an optional maxLength // with given validation rule and a set of enum values, following the convention of existing tests. // The test has two checks, first with maxLength unset to check if maxLength can be concluded from enums, // second with maxLength set to ensure it takes precedence. func genEnumWithRuleAndValues(rule string, values ...string) func(maxLength *int64) *schema.Structural { enums := make([]schema.JSON, 0, len(values)) for _, v := range values { enums = append(enums, schema.JSON{Object: v}) } return func(maxLength *int64) *schema.Structural { return &schema.Structural{ Generic: schema.Generic{ Type: "string", }, ValueValidation: &schema.ValueValidation{ MaxLength: maxLength, Enum:      enums, }, Extensions: schema.Extensions{ XValidations: apiextensions.ValidationRules{ { Rule: rule, }, }, }, } } } </ins> func genBytesWithRule(rule string) func(maxLength *int64) *schema.Structural { return func(maxLength *int64) *schema.Structural { return &schema.Structural{

pointed out to me that a CEL x-kubernetes-validations rule in a CRD was hitting the estimated cost limit when performing a simple "string has prefix" check on an en enum field. Putting a limit in the OpenAPI of the enum string field fixed the problem, but having to add a limit like this is counter-intuitive and unnecessary, since the allowed enum values already make the max allowed length obvious. The apiserver automatically computes the max length of an enum as the length of the longest allowed enum value. I believe we already refuse evaluate CEL rules on values that fail OpenAPI value validation, since we rely in the value validations to establish all length limits. I haven't tested it locally, but something like this should work (given enough nested arrays): As a workaround, one can manually set on the string enum, e.g.: 25+ All No response No response No response No response
/sig api-machinery /triage accepted /assign
/assign
I wasn't sure if we special-cased the maxLength / maxItems / maxProperties failures, or if all openapi validation had to pass before we started messing with CEL evaluation
it looks like we currently special-case specific openapi errors that short-circuit CEL validation, I'm not sure an enum validation failure would currently prevent evaluation:
The enum err is converted to () error which is not short-circuit here. But for defaulting case, all error would be short-circuit. I would suggest to add error into the blockingErr list to short-circuit CEL validation. Would this make sense?
yeah, that sounds good if we're going to rely on it to catch things that are longer than the maximum length enum value


Question: buf.Reset() } } <ins> func TestPrintCluster(t *testing.T) { tests := []struct { cluster    federation.Cluster expect     []metav1alpha1.TableRow showLabels bool }{ { federation.Cluster{ ObjectMeta: metav1.ObjectMeta{ Name:              "cluster1", CreationTimestamp: metav1.Time{Time: time.Now().Add(1.9e9)}, }, }, []metav1alpha1.TableRow{{Cells: []interface{}{"cluster1", "Unknown", "0s"}}}, false, }, { federation.Cluster{ ObjectMeta: metav1.ObjectMeta{ Name:              "cluster2", CreationTimestamp: metav1.Time{Time: time.Now().Add(1.9e9)}, Labels:            map[string]string{"label1": "", "label2": "cluster"}, }, Status: federation.ClusterStatus{ Conditions: []federation.ClusterCondition{ { Status: api.ConditionTrue, Type:   federation.ClusterReady, }, { Status: api.ConditionFalse, Type:   federation.ClusterOffline, }, }, }, }, []metav1alpha1.TableRow{{Cells: []interface{}{"cluster2", "Ready,NotOffline", "0s", "label1=,label2=cluster"}}}, true, }, } for i, test := range tests { rows, err := printCluster(&test.cluster, printers.PrintOptions{ShowLabels: test.showLabels}) if err != nil { t.Fatal(err) } for i := range rows { rows[i].Object.Object = nil } if !reflect.DeepEqual(test.expect, rows) { t.Errorf("%d mismatch: %s", i, diff.ObjectReflectDiff(test.expect, rows)) } } } func TestPrintClusterList(t *testing.T) { tests := []struct { clusters   federation.ClusterList expect     []metav1alpha1.TableRow showLabels bool }{ // Test podList's pod: name, num of containers, restarts, container ready status { federation.ClusterList{ Items: []federation.Cluster{ { ObjectMeta: metav1.ObjectMeta{ Name:              "cluster1", CreationTimestamp: metav1.Time{Time: time.Now().Add(1.9e9)}, }, Status: federation.ClusterStatus{ Conditions: []federation.ClusterCondition{ { Status: api.ConditionTrue, Type:   federation.ClusterReady, }, }, }, }, { ObjectMeta: metav1.ObjectMeta{ Name:              "cluster2", CreationTimestamp: metav1.Time{Time: time.Now().Add(1.9e9)}, Labels:            map[string]string{"label1": "", "label2": "cluster2"}, }, Status: federation.ClusterStatus{ Conditions: []federation.ClusterCondition{ { Status: api.ConditionTrue, Type:   federation.ClusterReady, }, { Status: api.ConditionFalse, Type:   federation.ClusterOffline, }, }, }, }, }, }, []metav1alpha1.TableRow{{Cells: []interface{}{"cluster1", "Ready", "0s", "<none>"}}, {Cells: []interface{}{"cluster2", "Ready,NotOffline", "0s", "label1=,label2=cluster2"}}}, true, }, } for _, test := range tests { rows, err := printClusterList(&test.clusters, printers.PrintOptions{ShowLabels: test.showLabels}) if err != nil { t.Fatal(err) } for i := range rows { rows[i].Object.Object = nil } if !reflect.DeepEqual(test.expect, rows) { t.Errorf("mismatch: %s", diff.ObjectReflectDiff(test.expect, rows)) } } } </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After labeling clusters in a Federation, when you type the field is empty. What you expected to happen: The fiels should contain information about the labels in the cluster How to reproduce it (as minimally and precisely as possible): a federation control pane in any Cloud Provider and any DNS provider your cluster: info aboyut the cluster with the flag: Note: if you describe it, you see the labels being : Environment: Kubernetes version (use ): 1.8.0 Cloud provider or hardware configuration**: anyone OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux Install tools: curl Others:
/sig multicluster
I'm working on it.


Question: node := newNode("node1", nil) node.Status.Conditions = []v1.NodeCondition{ {Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue}, <del> {Type: v1.NodeInodePressure, Status: v1.ConditionFalse}, </del> } return node }(),

Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: It seems kubelet did not update inode pressure; and only test codes are using . So prefer to deprecate or remove it directly. (seems no node-misc group :). )


Question: // utils.go contains functions used across test suites. const ( <del> cniVersion       = "v0.8.7" </del> <ins> cniVersion       = "v0.9.1" </ins> cniArch          = "amd64" cniDirectory     = "cni/bin" // The CNI tarball places binaries under directory under "cni/bin". cniConfDirectory = "cni/net.d"

The latest release of CNI plugins is , but we still have . I'll attempt to create a PR for this based on the .
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
- assigning to you :)


Question: func TestPriorityQueue_PendingPods(t *testing.T) { q := NewPriorityQueue(nil) q.Add(&medPriorityPod) <del> q.unschedulableQ.addOrUpdate(&unschedulablePod) q.unschedulableQ.addOrUpdate(&highPriorityPod) </del> <ins> addOrUpdateUnschedulablePod(q, &unschedulablePod) addOrUpdateUnschedulablePod(q, &highPriorityPod) </ins> expectedList := []*v1.Pod{&medPriorityPod, &unschedulablePod, &highPriorityPod} if !reflect.DeepEqual(expectedList, q.PendingPods()) { t.Error("Unexpected list of pending Pods for node.")

Which jobs are failing: pull-kubernetes-bazel-test Which test(s) are failing: bazel test //pkg/scheduler/internal/queue:godefaulttest Since when has it been failing: Testgrid link: Reason for failure: WARNING: DATA RACE Read at 0x00c000230db0 by goroutine 47: runtime.mapiterinit() +0x0 (PriorityQueue).flushUnschedulableQLeftover() +0x17a (PriorityQueue).flushUnschedulableQLeftover-fm() +0x41 () +0x61 () +0xce () +0x5a Previous write at 0x00c000230db0 by goroutine 45: runtime.mapassignfaststr() +0x0 (UnschedulablePodsMap).addOrUpdate() +0x30e () +0x601 testing.tRunner() +0x162 Goroutine 47 (running) created at: (PriorityQueue).run() +0x172 () +0xe0b () +0x50 () +0x5a5 testing.tRunner() +0x162 Goroutine 45 (running) created at: testing.(T).Run() +0x659 testing.runTests.func1() +0xa8 testing.tRunner() +0x162 testing.runTests() +0x4ee testing.(M).Run() +0x2ee () bazel-out/k8- +0x2fb Anything else we need to know: I'm very sorry about that. UnschedulablePodsMap is not thread-safe. In , lock will be acquired before operating UnschedulablePodsMap. But in , lock isn‘t be acquired. In the previous case, only a single thread runs so it's ok. But after add goroutine , there will be data race. So I need to acquire lock before operating UnschedulablePodsMap.
/sig testing /sig scheduling


Question: return true, errors.New("not implemented") } <ins> func (mounter *Mounter) EvalHostSymlinks(pathname string) (string, error) { return "", unsupportedErr } </ins> func (mounter *Mounter) PrepareSafeSubpath(subPath Subpath) (newHostPath string, cleanupAction func(), err error) { return subPath.Path, nil, unsupportedErr }

/kind bug Using , if a block device is symlinked to the discovery directory e.g. , then MapVolume will fail with since the containerized kubelet cannot see the path on host. is mounted by containerized kubelet, so if before we try to map the device we set devicePath to the evaluation of symlink and end up with a devicePath of , containerized kubelet will see the device and successfully map the device. Are there any issues with this solution? This is on Kubernetes 1.10
/sig storage
ah yes containerized kubelet has many issues when dealing with symlinks. Check out Mounter.GetMode(), it calls a nsenter EvalSymlinks which should be able to do what you need. But actually, one thing to consider, is do you need to do the losetup on the host, or does inside the container work?
Yes it appears to work from inside the container, on the host I run which disappears after I delete the pod using the device. My understanding of running losetup in MapVolume is not that great but I suppose that as long as that /dev/loop0 is created we are fine. Thanks for the quick response, I'll create a PR using that function and tag you
Does it create it on the host or in the containerized kubelet? What happens when containerized kubelet is restarted? The purpose of losetup is to do "refcounting" on the block device to prevent possible naming collisions when devices are attached/detached under unhealthy conditions. Ie, if a device showed up as /dev/sdb and you gave that to a pod, and then for some reason it become unhealthy and disappeared (while the pod is still running), another device (maybe even another volume type) could get attached to the node and reuse /dev/sdb. Now the pod has access to a device it's not supposed to. With filesystem volumes, the refcounting is done with mount points, but for block device, loopback is the equivalent.
It is created on the host (and also visible in the container), and stays around through restart. I understand now, thank you.
Actually after I restart it becomes /vdb instead of /dev/vdb in the losetup output, idk what that is about.


Question: return true, errors.New("not implemented") } <ins> func (mounter *Mounter) EvalHostSymlinks(pathname string) (string, error) { return "", unsupportedErr } </ins> func (mounter *Mounter) PrepareSafeSubpath(subPath Subpath) (newHostPath string, cleanupAction func(), err error) { return subPath.Path, nil, unsupportedErr }

There is no such thing as /vdb and /dev/vdb is still intact of course. Seems inconsequential since it is disappearing after all pods using the device are deleted, i.e. it's still serving as a refcounter


Question: // Verify scheduler metrics. // TODO: Reset metrics at the beginning of the test. // We should do something similar to how we do it for APIserver. <del> framework.ExpectNoError(framework.VerifySchedulerLatency(c)) </del> <ins> if err = framework.VerifySchedulerLatency(c); err != nil { framework.Logf("Warning: Scheduler latency not calculated, %v", err) } </ins> }) // Explicitly put here, to delete namespace at the end of the test

Problem Running the e2e density tests on a local cluster or any other cluster where ssh to master doesn't work fails, even though enough pods are created, because of SSH attempts. Specifically "should allow starting 30" is the argument to reproduce. Solution Allow density tests to pass even if some meta test operations fail after the fact. cc


Question: return errRequeue } <del> // Check parameters. claimParameters, classParameters, err := ctrl.getParameters(ctx, claim, class) </del> <ins> // Check parameters. Do not record event to Claim if its parameters are invalid, // syncKey will record the error. claimParameters, classParameters, err := ctrl.getParameters(ctx, claim, class, false) </ins> if err != nil { return err }

When ResourceClaimParameters or ResourceClassParameters fail validation in resource driver during PodSchedulingContext sync loop, the error is visible only in the events of the PodSchedulingContext. The error should be visible on the object that fails validation as well, ResourceClaim or ResourceClass respectively. Creating ResourceClaim with unsupported parameters' values or unsupported No response <details</details<details</details<details</details<details</details<details</details<details</details
/assign
/sig node
/triage accepted
KEP:


Question: kubectl get replicationcontrollers "${kube_flags[@]}" kubectl describe replicationcontroller frontend-controller "${kube_flags[@]}" | grep -q 'Replicas:.*3 desired' kubectl delete rc frontend-controller "${kube_flags[@]}" <ins> rcsbefore="$(kubectl get replicationcontrollers  -o template -t "{{ len .items }}" "${kube_flags[@]}")" kubectl create -f examples/guestbook/frontend-controller.json "${kube_flags[@]}" kubectl create -f examples/guestbook/redis-slave-controller.json "${kube_flags[@]}" kubectl delete rc frontend-controller redis-slave-controller "${kube_flags[@]}" # delete multiple controllers at once rcsafter="$(kubectl get replicationcontrollers  -o template -t "{{ len .items }}" "${kube_flags[@]}")" [ "$((${rcsafter} - ${rcsbefore}))" -eq 0 ] </ins> kube::log::status "Testing kubectl(${version}:nodes)" kubectl get nodes "${kube_flags[@]}"

Currently, it could not delete multiple pods at once, need to delete them one by one, it's very inconvenient if I have tons of pods to delete. Unlike docker, docker can multiple stop/rm containers/images at once.
This is possible using label selectors as of , which is better if you have tons of pods.
I agree that using label selector is better to delete all of pods under the label. But to delete multile pods at once is still usable: I just want to delete 1/2 pods under the label, label selector will not work selector could not span over other selectors or pods Another reason is docker rm/stop x y z works, I will wonder why kubectl cannot. This is not urgent, but I hope it can be implemented.
I agree. Thanks for the suggestion.
Would this be a suitable task for someone new to k8s development? If so, I'd be willing to take a look at this, if someone could provide a couple of hints where to start.
Further to my last message, I have a functional work in progress I would like to share, if anyone is interested?
PRs are welcome. See
Yes, this is a reasonable place to start. Since its in the client, you won't get much orientation to the core server code, but it will be a good place to orient to the client side API. There us also prior art that you can model off of. The kubectl create ... command already supports multiple resources. A good command to adapter would be the 'label' command. You can find it in Many thanks in advance! Brendan On Feb 19, 2015 8:21 AM, "Eric Tune" wrote:


Question: currentRevision *apps.ControllerRevision, updateRevision *apps.ControllerRevision, collisionCount int32, <del> pods []*v1.Pod) (*apps.StatefulSetStatus, error) { </del> <ins> pods []*v1.Pod) (statefulSetStatus *apps.StatefulSetStatus, updateErr error) { </ins> // get the current and update revisions of the set. currentSet, err := ApplyRevision(set, currentRevision) if err != nil {

I create a ResourceQuota with two pods limit in my namespace. Then, I create a statefuleSet with 3 replicas. As expected, two pods are created successfully, and the third pod failed to create. But, the ReadyReplicas is not 2 ,but 1. The ReadyReplicas is correct 1Create a ResourceQuota with two pods limit in a new namespace. 2Create a statefulSet with 3 replicas 3Check if the ReadyReplicas of statefulSet matchs the number of stateful running pods. 1According to the logic of the statefulSet controller, new pod created in current sync is not be immediately counted into the ReadyReplicas. 2Failure to create pod will cause to return error, then skip updating status. <details</details<details</details<detailskind v1.23.3 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign


Question: }, }, { <ins> // PV requires external deleter name:            "8-10-2 - external deleter when volume is static provisioning", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-2", "1Gi", "uid10-1-2", "claim10-1-2", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-2", "1Gi", "uid10-1-2", "claim10-1-2", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents, errors:          noerrors, test:            testSyncVolume, }, { // PV requires external deleter name:            "8-10-3 - external deleter when volume is migrated", initialVolumes:  []*v1.PersistentVolume{volumeWithAnnotation(volume.AnnMigratedTo, "pd.csi.storage.gke.io", volumeWithAnnotation(volume.AnnDynamicallyProvisioned, "kubernetes.io/gce-pd", newVolume("volume8-10-3", "1Gi", "uid10-1-3", "claim10-1-3", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, volume.AnnDynamicallyProvisioned)))}, expectedVolumes: []*v1.PersistentVolume{volumeWithAnnotation(volume.AnnMigratedTo, "pd.csi.storage.gke.io", volumeWithAnnotation(volume.AnnDynamicallyProvisioned, "kubernetes.io/gce-pd", newVolume("volume8-10-3", "1Gi", "uid10-1-3", "claim10-1-3", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, volume.AnnDynamicallyProvisioned)))}, initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents, errors:          noerrors, test:            testSyncVolume, }, { </ins> // delete success - two PVs are provisioned for a single claim. // One of the PVs is deleted. name: "8-11 - two PVs provisioned for a single claim",

In my CSI controller I have enabled the feature gate HonorPVReclaimPolicy for csi-provisioner. Based on the documentation Finalizers can be on a PersistentVolume to ensure that PersistentVolumes having Delete reclaim policy are deleted only after the backing storage are deleted. The newly introduced finalizers and external- are only to dynamically provisioned volumes. I can see the finalizer set on the PV while creating static Persistent Volume. PV deletion is stuck due to this finalizer apiVersion: v1 kind: PersistentVolume metadata: creationTimestamp: "2023-11-08T11:39:24Z" deletionGracePeriodSeconds: 0 deletionTimestamp: "2023-11-08T11:54:31Z" finalizers: external- name: csi-infoscale-pv PV should be deleted when PVC is deleted. Persistent Volume yaml apiVersion: v1 kind: PersistentVolume metadata: name: csi-infoscale-pv spec: storageClassName: csi-infoscale-sc-prod persistentVolumeReclaimPolicy: Delete capacity: storage: 2Gi accessModes: ReadWriteOnce csi: driver: org.veritas.infoscale # Please provide pre-provisioned Infoscale volume name. volumeHandle: clust21432/vrtskube_dg-/vol1 fsType: vxfs Persistent Volume Claim apiVersion: v1 kind: PersistentVolumeClaim metadata: name: csi-infoscale-pvc-static spec: accessModes: ReadWriteOnce resources: requests: storage: 2Gi storageClassName: csi-infoscale-sc-prod volumeName: csi-infoscale-pv Same thing works fine on OCP. WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.


Question: }, }, { <ins> // PV requires external deleter name:            "8-10-2 - external deleter when volume is static provisioning", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-2", "1Gi", "uid10-1-2", "claim10-1-2", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-2", "1Gi", "uid10-1-2", "claim10-1-2", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents, errors:          noerrors, test:            testSyncVolume, }, { // PV requires external deleter name:            "8-10-3 - external deleter when volume is migrated", initialVolumes:  []*v1.PersistentVolume{volumeWithAnnotation(volume.AnnMigratedTo, "pd.csi.storage.gke.io", volumeWithAnnotation(volume.AnnDynamicallyProvisioned, "kubernetes.io/gce-pd", newVolume("volume8-10-3", "1Gi", "uid10-1-3", "claim10-1-3", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, volume.AnnDynamicallyProvisioned)))}, expectedVolumes: []*v1.PersistentVolume{volumeWithAnnotation(volume.AnnMigratedTo, "pd.csi.storage.gke.io", volumeWithAnnotation(volume.AnnDynamicallyProvisioned, "kubernetes.io/gce-pd", newVolume("volume8-10-3", "1Gi", "uid10-1-3", "claim10-1-3", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, volume.AnnDynamicallyProvisioned)))}, initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents, errors:          noerrors, test:            testSyncVolume, }, { </ins> // delete success - two PVs are provisioned for a single claim. // One of the PVs is deleted. name: "8-11 - two PVs provisioned for a single claim",

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-08-24T00:48:26Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v5.0.1 Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-08-24T00:42:11Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"} Vanilla K8s setup <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage maybe similar issues should go in
Opened similar issue in external-provisioner repo also There is no reply on that as well. Can you please help understand the behavior of the feature HonorPVReclaimPolicy.
If you want Persistent Volume (PV) to be deleted when the associated Persistent Volume Claim (PVC) is deleted, you donot need to use finalizers explicitly. Kubernetes handles this behavior by default. It is ok that if you use to get the desire output using Finalizer we can do it.
As per my understanding Please follow the steps : Add finalizer in metadata of as you did above need to make change in the controller which is handling PV deletion in .go file. See whether the controller administrator is working or not. I think it will work.
Finalizer is because the feature gate HonorPVReclaimPolicy for csi-provisioner is enabled. We are not adding the finalizer explicitly. From the documentation, "The newly introduced finalizers and external- are only to dynamically provisioned volumes." Not sure why is the finalizer getting set.


Question: }, }, { <ins> // PV requires external deleter name:            "8-10-2 - external deleter when volume is static provisioning", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-2", "1Gi", "uid10-1-2", "claim10-1-2", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-2", "1Gi", "uid10-1-2", "claim10-1-2", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents, errors:          noerrors, test:            testSyncVolume, }, { // PV requires external deleter name:            "8-10-3 - external deleter when volume is migrated", initialVolumes:  []*v1.PersistentVolume{volumeWithAnnotation(volume.AnnMigratedTo, "pd.csi.storage.gke.io", volumeWithAnnotation(volume.AnnDynamicallyProvisioned, "kubernetes.io/gce-pd", newVolume("volume8-10-3", "1Gi", "uid10-1-3", "claim10-1-3", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, volume.AnnDynamicallyProvisioned)))}, expectedVolumes: []*v1.PersistentVolume{volumeWithAnnotation(volume.AnnMigratedTo, "pd.csi.storage.gke.io", volumeWithAnnotation(volume.AnnDynamicallyProvisioned, "kubernetes.io/gce-pd", newVolume("volume8-10-3", "1Gi", "uid10-1-3", "claim10-1-3", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, volume.AnnDynamicallyProvisioned)))}, initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents, errors:          noerrors, test:            testSyncVolume, }, { </ins> // delete success - two PVs are provisioned for a single claim. // One of the PVs is deleted. name: "8-11 - two PVs provisioned for a single claim",

Also, for deleting PV it requires the annotation "" but this annotation is set in case of dynamic PVs. Should it be set explicitly for static PV?
This feature is alpha so the feature gate is not enabled by default. Did you enable the feature gate on kube-controller-manager?
Yes, i have enabled this feature.
external- is also applied to statically provisioned volumes. FYI
Hi Which version of the external provisioner is the driver using? Does the driver meet the requirements of the CSI spec on the DeleteVolume PRC? FYI: Is this problem occasional or permanently reproducible? Can you provide the provisioner logs at that time?
/triage needs-information
cc
cc
External provisioner used is : "csi-provisioner:v3.5.0" This problem is always reproducible Provisioner logs: I1123 12:22:38. 1 ] handleProtectionFinalizer Volume : &PersistentVolume{ObjectMeta:{csi-infoscale-pv -420a-4e9e-be82- 0 2023-11-23 12:21:18 +0000 UTC <nil<nilmap[] map[{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"csi-infoscale-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"2Gi"},"csi":{"driver":"org.veritas.infoscale","fsType":"vxfs","volumeHandle":"clust40559/vrtskubedg-/vol1"},"persistentVolumeReclaimPolicy":"Delete","storageClassName":"csi-infoscale-sc-prod"}} ] [] [ external-] [{kubectl-client-side-apply Update v1 2023-11-23 12:21:18 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"":{}}},"f:spec":{"f:accessModes":{},"f:capacity":{".":{},"f:storage":{}},"f:csi":{".


Question: }, }, { <ins> // PV requires external deleter name:            "8-10-2 - external deleter when volume is static provisioning", initialVolumes:  []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-2", "1Gi", "uid10-1-2", "claim10-1-2", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, expectedVolumes: []*v1.PersistentVolume{newExternalProvisionedVolume("volume8-10-2", "1Gi", "uid10-1-2", "claim10-1-2", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, gceDriver, nil, volume.AnnBoundByController)}, initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents, errors:          noerrors, test:            testSyncVolume, }, { // PV requires external deleter name:            "8-10-3 - external deleter when volume is migrated", initialVolumes:  []*v1.PersistentVolume{volumeWithAnnotation(volume.AnnMigratedTo, "pd.csi.storage.gke.io", volumeWithAnnotation(volume.AnnDynamicallyProvisioned, "kubernetes.io/gce-pd", newVolume("volume8-10-3", "1Gi", "uid10-1-3", "claim10-1-3", v1.VolumeBound, v1.PersistentVolumeReclaimDelete, classEmpty, volume.AnnDynamicallyProvisioned)))}, expectedVolumes: []*v1.PersistentVolume{volumeWithAnnotation(volume.AnnMigratedTo, "pd.csi.storage.gke.io", volumeWithAnnotation(volume.AnnDynamicallyProvisioned, "kubernetes.io/gce-pd", newVolume("volume8-10-3", "1Gi", "uid10-1-3", "claim10-1-3", v1.VolumeReleased, v1.PersistentVolumeReclaimDelete, classEmpty, volume.AnnDynamicallyProvisioned)))}, initialClaims:   noclaims, expectedClaims:  noclaims, expectedEvents:  noevents, errors:          noerrors, test:            testSyncVolume, }, { </ins> // delete success - two PVs are provisioned for a single claim. // One of the PVs is deleted. name: "8-11 - two PVs provisioned for a single claim",

":{},"f:driver":{},"f:fsType":{},"f:volumeHandle":{}},"f:persistentVolumeReclaimPolicy":{},"f:storageClassName":{},"f:volumeMode":{}}} } {csi-provisioner Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{"v:"external-"":{}}}} } {kube-controller-manager Update v1 2023-11-23 12:21:22 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{"":{}}},"f:spec":{"f:claimRef":{".