Question: kubetypes.HTTPSource, &api.Pod{ ObjectMeta: api.ObjectMeta{ <del> UID:       "111", Name:      "foo" + "-" + hostname, Namespace: "default", SelfLink: getSelfLink("foo-"+hostname, kubetypes.NamespaceDefault), </del> <ins> UID:         "111", Name:        "foo" + "-" + hostname, Namespace:   "default", Annotations: map[string]string{kubetypes.ConfigHashAnnotationKey: "111"}, SelfLink:    getSelfLink("foo-"+hostname, kubetypes.NamespaceDefault), </ins> }, Spec: api.PodSpec{ NodeName:                      hostname,

See for more context. Currently, kubelet compares the spec of a mirror pod (created via apiserver) with that of a static pod (config stored as a file) to decide whether the two are considered identical. If they differ, kubelet would delete the mirror pod and re-create it with the correct spec. Due to the fact that apiserver may set default values for some fields in the spec of the mirror pod, the specs could deviate undesirably. To avoid problems like this, we can store the config of a static pod as an annotation in the mirror pod, and compare the static pod with the annotation.
Would a hash of the config be sufficient if we're only going to compare? The whole spec would generally be more useful, but I wonder if that is too much data to put in an annotation.
This seems totally reasonable to me, but I'm still desperately curious why you can't just bypass defaulting when you generate mirror pod from static pod? (i.e. just use the static pod's configuration verbatim)
It's not kubelet who applies the default values here. The admission plugin was only run at the apiserver, and to bypass that, you have to ask apiserver to treat mirror pods differently and potentially at multiple places.
On May 22, 2015 10:38 AM, "Yu-Ju Hong" wrote: Understood, but is it really multiple places? If so then I agree it's a bad idea. Otherwise it might be worth considering as an alternative.
It's untenable for Kubelet to assume to know/circumvent all fields that will be set by other entities: admission control, defaulting, late initialization, auto-something, whatever. It needs to obey the same rules as other clients.
I don't understand why you would ever want to apply any of the things you mentioned to static pods. Static pods should be copied verbatim into the mirror pods, right? That's what I'm proposing.
What if the master is updated before the kubelet, and the master adds a new API field with a default value?
That case should be discussed in
I agree that mirror pod should reflect the true status of the static pod running on the node, and no default values should be set in this case.


Question: kubetypes.HTTPSource, &api.Pod{ ObjectMeta: api.ObjectMeta{ <del> UID:       "111", Name:      "foo" + "-" + hostname, Namespace: "default", SelfLink: getSelfLink("foo-"+hostname, kubetypes.NamespaceDefault), </del> <ins> UID:         "111", Name:        "foo" + "-" + hostname, Namespace:   "default", Annotations: map[string]string{kubetypes.ConfigHashAnnotationKey: "111"}, SelfLink:    getSelfLink("foo-"+hostname, kubetypes.NamespaceDefault), </ins> }, Spec: api.PodSpec{ NodeName:                      hostname,

However, I think it'd be harder to implement this way since, as stated above, that there are many things to circumvent in the apiserver itself. It could also be harder to keep this special case from being broken by a yet another new feature/plugin. Keeping the hash of the config in the annotation seems like a good compromise even though it may not reflect the true status 100% accurately. I think this is acceptable. What do you think?
Sure, if you think the annotation approach is easier, I would go with that. I guess in the case Brian mentoined (master adds a new API field with a default value), you would not update the mirror pod, because the static pod definition has not changed?
Yes, a hash would be sufficient.
It shouldn't be necessary to update the mirror pod in the case that a new default field was updated, because that field should be populated automatically by the apiserver on decode from etcd.
We should fix this so we don't have this problem again in 1.2.


Question: test.TestDynamicProvisioning() } <del> // Run the last test with storage.k8s.io/v1beta1 on pvc if betaTest != nil { ginkgo.By("Testing " + betaTest.Name + " with beta volume provisioning") betaClass := newBetaStorageClass(*betaTest, "beta") // create beta class manually betaClass, err := c.StorageV1beta1().StorageClasses().Create(context.TODO(), betaClass, metav1.CreateOptions{}) framework.ExpectNoError(err) defer deleteStorageClass(c, betaClass.Name) // fetch V1beta1 StorageClass as V1 object for the test class, err := c.StorageV1().StorageClasses().Get(context.TODO(), betaClass.Name, metav1.GetOptions{}) framework.ExpectNoError(err) betaTest.Client = c betaTest.Class = class betaTest.Claim = e2epv.MakePersistentVolumeClaim(e2epv.PersistentVolumeClaimConfig{ ClaimSize:        betaTest.ClaimSize, StorageClassName: &class.Name, VolumeMode:       &betaTest.VolumeMode, }, ns) betaTest.Claim.Spec.StorageClassName = &(class.Name) (*betaTest).TestDynamicProvisioning() } </del> }) ginkgo.It("should provision storage with non-default reclaim policy Retain", func() {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-slow [sig-storage] Dynamic Provisioning DynamicProvisioner [Slow] should provision storage with different parameters 5/13 https://k8s- v1beta1.StorageClass got removed in We should remove the v1beta1 test.
/assign can you help take a look?
/triage accepted
/sig storage
I run the test with the latest from master without removing the beta test and it failed with a different error before getting to the beta test part: Do you have any idea what would've happened in my setup? I redeployed k/k using master HEAD
For this testcase: I saw that the provisioned PD has a size of 1.5Gi according to kubernetes: And in gcloud It's really interesting that the test was expecting 2Gi to be created, is this something that was changed recently?
The logs you posted are showing pd csi as the provisioner. Were you testing with the intree driver or csi driver? The CSI driver should to the nearest Gb. I believe intree does the .
I think I did intree because I didn't send as an argument in my command: It's interesting that the PVC has this annotation too:
mmm something interesting about my setup is that I created the cluster with CSIMigration turned on Do you think that might be forcing the usage of ?
Yes likely. Try with CSIMigration off and see if that helps. If so, then this potentially seems like a rounding bug with the csi driver
Thanks I tried with and it did indeed round up to 2Gi.


Question: IP address. release: v1.9 file: test/e2e/common/pods.go <ins> - testname: Pods, completes the lifecycle of a Pod and the PodStatus codename: '[k8s.io] Pods should run through the lifecycle of Pods and PodStatus [Conformance]' description: A Pod is created with a static label which MUST succeed. It MUST succeed when patching the label and the pod data. When checking and replacing the PodStatus it MUST succeed. It MUST succeed when deleting the Pod. release: v1.20 file: test/e2e/common/pods.go </ins> - testname: Pods, remote command execution over websocket codename: '[k8s.io] Pods should support remote command execution over websockets [NodeConformance] [Conformance]'

[x] Original approval issue: - Reverted due to flakes [x] Two weeks soak end date: 20 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Pod endpoints which are untested. a Pod with a static label the Pod with a new Label and updated data the Pod to ensure it's patched the Pod's status the Pod's status Ready condition to False the Pod's status to check if Ready is False all Pods in all Namespaces find the Pod(1) ensure that the Pod is found and is patched Namespaced Pod(1) via a Collection with a LabelSelector the Pod to check that it's deleted Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/reopen Test PR Merge Promotion PR to be created next
Reopened this issue. <detailsIn response to : /reopen Test PR Merge Promotion PR to be created next Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
reverted by due to flaky test
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/reopen PR on test grid for 2 weeks before promotion to conformance
Reopened this issue. <detailsIn response to : /reopen PR on test grid for 2 weeks before promotion to conformance Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: fi } <del> function kube::build::probe_address { # Apple has an ancient version of netcat with custom timeout flags.  This is # the best way I (jbeda) could find to test for that. local netcat if nc 2>&1 | grep -e 'apple' >/dev/null ; then netcat="nc -G 1" else netcat="nc -w 1" fi </del> <ins> function kube::build::rsync_probe { </ins> # Wait unil rsync is up and running. <del> if ! which nc >/dev/null ; then V=6 kube::log::info "netcat not installed, waiting for 1s" sleep 1 return 0 fi local tries=10 </del> <ins> local tries=20 </ins> while (( ${tries} > 0 )) ; do <del> if ${netcat} -z "$1" "$2" 2> /dev/null ; then </del> <ins> if rsync "rsync://k8s@${1}:${2}/"  --password-file="${LOCAL_OUTPUT_BUILD_CONTEXT}/rsyncd.password"  &> /dev/null ; then </ins> return 0 fi tries=$(( ${tries} - 1))

https://k8s-
I have a similar problem when I try to make release on my local desktop:
Related to .
Similar problem for :
Can you open a new issue for your problem with information about your set up. Mac/Linux? Docker version? etc.
Please add details to (to be opened) issue. I suspect that is a different issue.
New issue:


Question: echo "ELASTICSEARCH_LOGGING_REPLICAS='${ELASTICSEARCH_LOGGING_REPLICAS:-1}'" echo "ENABLE_NODE_LOGGING='${ENABLE_NODE_LOGGING:-false}'" echo "ENABLE_CLUSTER_UI='${ENABLE_CLUSTER_UI}'" <ins> echo "ENABLE_HOSTPATH_PROVISIONER='${ENABLE_HOSTPATH_PROVISIONER:-false}'" </ins> echo "LOGGING_DESTINATION='${LOGGING_DESTINATION:-}'" echo "ENABLE_CLUSTER_DNS='${ENABLE_CLUSTER_DNS:-false}'" echo "DNS_SERVER_IP='${DNS_SERVER_IP:-}'"

feature request Enables easier testing for pv/pvc, and all derivatives like pet sets. The only alternative is to use manually created pv of local type. I have time to work on it myself and created this feature to collect feedback/ideas on what provider should be enabled by default.
Since this was merged should we close this issue?


Question: // Never send a bookmark event if we did not see an event here, this is fine // because we don't provide any guarantees on sending bookmarks. if lastProcessedResourceVersion == 0 { <ins> // pop expired watchers in case there has been no update c.bookmarkWatchers.popExpiredWatchers() </ins> continue } bookmarkEvent := &watchCacheEvent{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I try to backport bookmark feature to our internal branch. When I enable bookmark feature, apiserver leaks memory. I spend some time to dig the root cause and find the bookmarkWatchers of service cacher is keep increasing. I am not sure if it exists at upstream code. Please let me know if it is ok at latest code. What you expected to happen: Avoid memory leaking How to reproduce it (as minimally and precisely as possible): a cluster. not update service. will find kube-apiserver's memory keep increasing. Anything else we need to know?: Kubelet will watch service with bookmark, and add a watcher to bookmarkWatchers. and service init resource version is 0 Without updating service, we cannot update "lastProcessedResourceVersion", it will keep 0, bookmarkTimer branch will always be continued. As a result, it cannot trigger popExpiredWatchers, so the service watcher will keep increasing every watching. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scalability PLAT :)
How about making this change ?
- can you also please take a look?
Agree with u. That's same way I fix in our branch. But I wanna hear the suggestion of and .
! With the same bugfix wrote above, we can see the memory not increasing any more.
Given that watch-bookmarks doesn't create any additional go-routines (well, creates 1, but cannot leak them), it has to leak the data. The only data that can be leaked is exactly bookmarkWatchers. So I looked into the code and got into exactly the same conclusion - that we must be leaking watchers for resources that aren't changing. The fix as above makes sense (though we also need a unit test for it).
Thanks a lot.
has the fix by
Thanks for reporting the bug, I will look at this day later. In addition, also need to think carefully about whether need to send a bookmark when there are no events at all.
cc


Question: "where N means number of retries allowed for kubelet to post node status.") fs.Float32Var(&o.NodeEvictionRate, "node-eviction-rate", 0.1, "Number of nodes per second on which pods are deleted in case of node failure when a zone is healthy (see --unhealthy-zone-threshold for definition of healthy/unhealthy). Zone refers to entire cluster in non-multizone clusters.") fs.Float32Var(&o.SecondaryNodeEvictionRate, "secondary-node-eviction-rate", 0.01, "Number of nodes per second on which pods are deleted in case of node failure when a zone is unhealthy (see --unhealthy-zone-threshold for definition of healthy/unhealthy). Zone refers to entire cluster in non-multizone clusters. This value is implicitly overridden to 0 if the cluster size is smaller than --large-cluster-size-threshold.") <del> fs.Int32Var(&o.LargeClusterSizeThreshold, "large-cluster-size-threshold", 50, fmt.Sprintf("Number of nodes from which %s treats the cluster as large for the eviction logic purposes. --secondary-node-eviction-rate is implicitly overridden to 0 for clusters this size or smaller.", names.NodeLifecycleController)) </del> <ins> fs.Int32Var(&o.LargeClusterSizeThreshold, "large-cluster-size-threshold", 50, fmt.Sprintf("Number of nodes from which %s treats the cluster as large for the eviction logic purposes. --secondary-node-eviction-rate is implicitly overridden to 0 for clusters this size or smaller. Notice: If nodes reside in multiple zones, this threshold will be considered as zone node size threshold for each zone to determine node eviction rate independently.", names.NodeLifecycleController)) </ins> fs.Float32Var(&o.UnhealthyZoneThreshold, "unhealthy-zone-threshold", 0.55, "Fraction of Nodes in a zone which needs to be not Ready (minimum 3) for zone to be treated as unhealthy. ") }

The arg large-cluster-size-threshold for kube-controller-manager works as zone size threshold when determining node eviction rate. If there are multiple zones, the description for arg stating "cluster" threshold would be missleading. ARG: Matched arg description Multiple zones cluster No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig docs


Question: // +optional optional string workingDir = 5; <del> // List of ports to expose from the container. Exposing a port here gives // the system additional information about the network connections a // container uses, but is primarily informational. Not specifying a port here </del> <ins> // List of ports to expose from the container. Not specifying a port here </ins> // DOES NOT prevent that port from being exposed. Any port which is // listening on the default "0.0.0.0" address inside a container will be // accessible from the network. <ins> // Modifying this array with strategic merge patch may corrupt the data. // For more information See https://github.com/kubernetes/kubernetes/issues/108255. </ins> // Cannot be updated. // +optional // +patchMergeKey=containerPort

The comments in says that : However there are several cases where this is false : When using a Service with a targetPort that references a name in the ports arrays of the pod. Some software (ab)uses this API to get which ports are used (See Although unfixable, my goal is to fix the documentation so other people don't have the same problem). The Azure Load Balancer implementation does this for instance. Some other network/security software might use this information. Also, is it possible to document/reference the above issue in the comment ? Since it is described as informational I expected for the network to work without this information. But it's definitely not the case. Not needed since it's mostly a doc issue. I'm interested in contributing this patch but would like some feedbacks before doing it. :) <details</details<details</details<details</details<details</details<details</details<details</details
/sig network
/triage accepted -- thoughts? I believe the doc are due for a refresh.
I'd take this one. /assign
I didn't see that you wanted to do this patch, I am very sorry and now I unassign it. /unassign
No worries But I'm still not sure where / what to add. Should I only fix the part saying it is only informational ? Is there anything else ? (cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: } return ret } <del> func getPodLabelName(pod *v1.Pod) string { podName := pod.Name if pod.GenerateName != "" { podNameSlice := strings.Split(pod.Name, "-") podName = strings.Join(podNameSlice[:len(podNameSlice)-1], "-") if label, ok := pod.GetLabels()[apps.DefaultDeploymentUniqueLabelKey]; ok { podName = strings.ReplaceAll(podName, fmt.Sprintf("-%s", label), "") } } return podName } </del>

The metric used to report acutal pod in the label , but now I see the podname prefix only. The full pod name in the label, so I can trace the metric to the failing pod. Check raw api and verify the actual pod name with uids. <details</detailsAzure AKS <details</detailsNo response No response No response_
/sig node
/triage accepted /priority important-soon /cc As discussed this is a change in functionaloty that wasn't highlighted in release notes. Need to decide on what do do with it. I understand it's a change of functionality. what is the side effect of this for you? Does the label without last part not informational enough?
I've been using the metric in my custom grafana dashboard that shows CPU, RAM, IO usage of pods with graph of increases in probes. If the apps are stable, the probe graph looks steadily. I can see if some irregularities in system metrics are influencing probes of respective pod. Also, if readiness probe is failing, the pod is removed from service, but is not restarted so I must have knowledge of pods with failing readiness probes. I attach the screenshot of my graph. ! So I'd welcome the possibility to pair the probes with exact pod and thank you for looking into it.
are we thinking that this resulted in an undesirable change? Spologies for being a bit late to reply, I'm returning from a brief paternity leave staycation.
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
Well this is awkward... I still believe the actual behavior does not respect the intended idea...
This issue makes it hard to link probe metrics with right pods. The value of pod label in and are also inconsistent with how it is used in other kubelet metrics, see: vs Definitely a bug, regression since 1.25.
I can't get anyone to look at the fix/revert: you may have better luck
/milestone v1.28
discussion in
You've pushed it through, congrats!


Question: } // If the container logs directory does not exist, create it. <del> if _, err := os.Stat(containerLogsDir); err != nil { if err := kl.os.MkdirAll(containerLogsDir, 0755); err != nil { glog.Errorf("Failed to create directory %q: %v", containerLogsDir, err) </del> <ins> if _, err := os.Stat(ContainerLogsDir); err != nil { if err := kl.os.MkdirAll(ContainerLogsDir, 0755); err != nil { glog.Errorf("Failed to create directory %q: %v", ContainerLogsDir, err) </ins> } }

Cluster logging assumes kubelet stores container log at a specific directory with specific name. Currently the path is: After the CRI work, the path will be: However, we don't have test coverage for this. We don't even know whether the log files are there or not. We should add a node e2e test for this, basically: 1) Create a container which prints logs 2) Check whether the expected log file is created. 3) Check whether the logs are collected in the file as expected. And it would be better to launch a pod to do the check instead of accessing local file system directly, which makes containerizing easier. /cc because you are collecting missing test. /cc
I will try to add a node-e2e test there
I think it's ok to add this for CRI. But there maybe problems on today's interface for rkt. If I'm not wrong, rkt does't store logs at same location with docker. cc/ Is this true?
Thanks! :)
After gets merged, we need to update the test to verify the cri log path.


Question: // wait until new webhook is called the first time if err := wait.PollImmediate(time.Millisecond*5, wait.ForeverTestTimeout, func() (bool, error) { <del> _, err = client.CoreV1().Pods("default").Patch(reinvocationMarkerFixture.Name, types.JSONPatchType, []byte("[]")) </del> <ins> _, err = client.CoreV1().Pods(markerNs).Patch(marker.Name, types.JSONPatchType, []byte("[]")) </ins> select { case <-upCh: return true, nil

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig api-machinery /assign
/cc
want to take a look at the audit event issue first? I can dig in after that if issues remain.
yes, I'm looking at the audit event issue now
actually the failure message suggests that the missing audit log is caused by unexpected admission invocation: I think it's unrelated to auditing Note that there are two tests named "no reinvocation of webhook B when in-tree or prior webhook mutations" :( and from the audit log (), it also suggests that the test is hitting a wrong endpoint
Thanks I'll dig in.


Question: <ins> # Development Guide # Releases and Official Builds Official releases are built in Docker containers.  Details are [here](build/README.md).  You can do simple builds and development with just a local Docker installation.  If want to build go locally outside of docker, please continue below. ## Go development environment Kubernetes is written in [Go](http://golang.org) programming language. If you haven't set up Go development environment, please follow [this instruction](http://golang.org/doc/code.html) to install go tool and set up GOPATH. Ensure your version of Go is at least 1.3. ## Put kubernetes into GOPATH We highly recommend to put kubernetes' code into your GOPATH. For example, the following commands will download kubernetes' code under the current user's GOPATH (Assuming there's only one directory in GOPATH.): ``` $ echo $GOPATH /home/user/goproj $ mkdir -p $GOPATH/src/github.com/GoogleCloudPlatform/ $ cd $GOPATH/src/github.com/GoogleCloudPlatform/ $ git clone git@github.com:GoogleCloudPlatform/kubernetes.git ``` The commands above will not work if there are more than one directory in ``$GOPATH``. (Obviously, clone your own fork of Kubernetes if you plan to do development.) ## godep and dependency management Kubernetes uses [godep](https://github.com/tools/godep) to manage dependencies. It is not strictly required for building Kubernetes but it is required when managing dependencies under the Godeps/ tree, and is required by a number of the build and test scripts. Please make sure that ``godep`` is installed and in your ``$PATH``. ### Installing godep There are many ways to build and host go binaries. Here is an easy way to get utilities like ```godep``` installed: 1. Ensure that [mercurial](http://mercurial.selenic.com/wiki/Download) is installed on your system. (some of godep's dependencies use the mercurial source control system).  Use ```apt-get install mercurial``` or ```yum install mercurial``` on Linux, or [brew.sh](http://brew.sh) on OS X, or download directly from mercurial. 2. Create a new GOPATH for your tools and install godep: ``` export GOPATH=$HOME/go-tools mkdir -p $GOPATH go get github.com/tools/godep ``` 3. Add $GOPATH/bin to your path. Typically you'd add this to your ~/.profile: ``` export GOPATH=$HOME/go-tools export PATH=$PATH:$GOPATH/bin ``` ### Using godep Here is a quick summary of `godep`.  `godep` helps manage third party dependencies by copying known versions into Godeps/_workspace.  You can use `godep` in three ways: 1. Use `godep` to call your `go` commands.  For example: `godep go test ./...` 2. Use `godep` to modify your `$GOPATH` so that other tools know where to find the dependencies.  Specifically: `export GOPATH=$GOPATH:$(godep path)` 3. Use `godep` to copy the saved versions of packages into your `$GOPATH`.  This is done with `godep restore`. We recommend using options #1 or #2. ## Hooks Before committing any changes, please link/copy these hooks into your .git directory. This will keep you from accidentally committing non-gofmt'd go code. ``` cd kubernetes ln -s hooks/prepare-commit-msg .git/hooks/prepare-commit-msg ln -s hooks/commit-msg .git/hooks/commit-msg ``` ## Unit tests ``` cd kubernetes hack/test-go.sh ``` Alternatively, you could also run: ``` cd kubernetes godep go test ./... ``` If you only want to run unit tests in one package, you could run ``godep go test`` under the package directory. For example, the following commands will run all unit tests in package kubelet: ``` $ cd kubernetes # step into kubernetes' directory. $ cd pkg/kubelet $ godep go test # some output from unit tests PASS ok      github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet   0.317s ``` ## Coverage ``` cd kubernetes godep go tool cover -html=target/c.out ``` ## Integration tests You need an etcd somewhere in your PATH. To install etcd, run: ``` cd kubernetes hack/install-etcd.sh sudo ln -s $(pwd)/third_party/etcd/bin/etcd /usr/bin/etcd ``` ``` cd kubernetes hack/test-integration.sh ``` ## End-to-End tests You can run an end-to-end test which will bring up a master and two minions, perform some tests, and then tear everything down. Make sure you have followed the getting started steps for your chosen cloud platform (which might involve changing the `KUBERNETES_PROVIDER` environment variable to something other than "gce". ``` cd kubernetes hack/e2e-test.sh ``` Pressing control-C should result in an orderly shutdown but if something goes wrong and you still have some VMs running you can force a cleanup with the magical incantation: ``` hack/e2e-test.sh 1 1 1 ``` ## Testing out flaky tests [Instructions here](docs/devel/flaky-tests.md) ## Add/Update dependencies Kubernetes uses [godep](https://github.com/tools/godep) to manage dependencies. To add or update a package, please follow the instructions on [godep's document](https://github.com/tools/godep). To add a new package ``foo/bar``: - Make sure the kubernetes' root directory is in $GOPATH/github.com/GoogleCloudPlatform/kubernetes - Run ``godep restore`` to make sure you have all dependancies pulled. - Download foo/bar into the first directory in GOPATH: ``go get foo/bar``. - Change code in kubernetes to use ``foo/bar``. - Run ``godep save ./...`` under kubernetes' root directory. To update a package ``foo/bar``: - Make sure the kubernetes' root directory is in $GOPATH/github.com/GoogleCloudPlatform/kubernetes - Run ``godep restore`` to make sure you have all dependancies pulled. - Update the package with ``go get -u foo/bar``. - Change code in kubernetes accordingly if necessary. - Run ``godep update foo/bar`` under kubernetes' root directory. ## Keeping your development fork in sync One time after cloning your forked repo: ``` git remote add upstream https://github.com/GoogleCloudPlatform/kubernetes.git ``` Then each time you want to sync to upstream: ``` git fetch upstream git rebase upstream/master ``` ## Regenerating the API documentation ``` cd kubernetes/api sudo docker build -t kubernetes/raml2html . sudo docker run --name="docgen" kubernetes/raml2html sudo docker cp docgen:/data/kubernetes.html . ``` View the API documentation using htmlpreview (works on your fork, too): ``` http://htmlpreview.github.io/?https://github.com/GoogleCloudPlatform/kubernetes/blob/master/api/kubernetes.html ``` </ins>

Right now, we have a mixture of user documentation and developer documentation in the top-level directory. We should make it easier to find a just user documentation. Step 1 would be to move developer documentation into a subdir like docs/development or something like that. User docs would stay in the top-level docs dir.
move to


Question: if err != nil { return false, fmt.Errorf("got error while getting pod events: %s", err) } <del> if len(events.Items) == 0 { return false, nil // no events have occurred yet </del> <ins> for _, event := range events.Items { if strings.Contains(event.Message, msg) { return true, nil } </ins> } <del> return strings.Contains(events.Items[0].Message, msg), nil </del> <ins> return false, nil </ins> } }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-new-master-upgrade-cluster-new-parallel Which test(s) are failing: Varies, but all subpath failures, including: [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath with backstepping is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] ... and pretty much every other subpath test, but never all of them at once. There's also a few other storage tests failing, such as: [sig-storage] Volume expand [Slow] Verify if editing PVC allows resize [sig-storage] Detaching volumes should not work when mount is in progress Since when has it been failing: 11/22 Testgrid link: https://k8s- Reason for failure: These flakes started around the time that merged, but doesn't match up with the exact merge stamp, so it's probably coincidental. The subpath test failures seem to be mostly timeouts: ... so possibly this is just GCE fail. Anything else we need to know: This test job has always been flaky, with around a 40% failure rate. <!-- DO NOT EDIT BELOW THIS LINE --/kind flake /sig storage /priority important-soon cc
can you please help investigate this. It looks to me like the tests are flaking for and . I wonder if the v0 tests are racing with the non-v0 tests for something in this test?
The other tests including for the other CSI driver and non-CSI in-tree plugins are solid green which also makes me think this is a issue with the hostpath tests or driver.


Question: if err != nil { return false, fmt.Errorf("got error while getting pod events: %s", err) } <del> if len(events.Items) == 0 { return false, nil // no events have occurred yet </del> <ins> for _, event := range events.Items { if strings.Contains(event.Message, msg) { return true, nil } </ins> } <del> return strings.Contains(events.Items[0].Message, msg), nil </del> <ins> return false, nil </ins> } }

(just adding additional similar flakes here for tracking) Similar flakes in :
adding to the milestone for triage since the flakes look new
These all 4 subpath tests call and it fails in finding a specific failed event by using . As uses to check if the specific error event happens, and , I guess that tests may flake if other failed event happens before the expected one. Also, already wait for the event, we might be able to delete  in . I will create a PR to fix above.
/priority critical-urgent /remove-priority important-soon /milestone v1.13
while we should update the test with your PR, I'm not sure if it's going to fix the failure. Looking at one failing test log, these are the events I see on the failing pod: So it seems to be failing before even reaching the subpath logic.
Timeline of events: Test pod started getting processed by kubelet: It took a minute after the test pod was created for the csi driver to finish registering and mount to succeed, and then subpath event to trigger: So should address the issue. I was initially confused because the test logs didn't show the subpath event for some reason
I opened up to investigate if 1 minute for csi plugin registration is expected
I've a tracker issue for the complete list of storage flakes: This is so that we can track whether we're addressing most of the flakes with the subpath fixes, or if additional fixes are required.
keeping open until flakes are verified cleaned up
/reopen Reopening this issue until we can verify the fix in CI
As of this morning we are seeing CSI "subpath should fail" flakes in the following job Multiple "subpath should fail" flakes in this slow job alone Multiple "subpath should fail" flakes in the 1.13 slow job, similar to master. The following 2 tests are flaky only in this upgrade job can you help us determine if these are cases are flaky tests that you are fixing or is there an underlying CSI bug that needs addressing?
root cause of this is a delay in an external sidecar containers and the tests not waiting long enough to accommodate the delay. There are no changes required in (outside of test) to handle this. These should not be 1.13 blocking. We'll continue to look in to getting them non-flaky today.


Question: if err != nil { return false, fmt.Errorf("got error while getting pod events: %s", err) } <del> if len(events.Items) == 0 { return false, nil // no events have occurred yet </del> <ins> for _, event := range events.Items { if strings.Contains(event.Message, msg) { return true, nil } </ins> } <del> return strings.Contains(events.Items[0].Message, msg), nil </del> <ins> return false, nil </ins> } }

In the meantime, we can mark these tests as flaky to reduce the noise, if you'd like. looks like the same issue () is still being hit. See : And logs from (): We need to make the tests handle this condition more gracefully. The suggestion here is one possible solution.
Would be good to capture this as a known issue in release notes.
to followup on capturing this in Release notes
per , can you create a targeted fix increasing the timeout? Looks like marking these tests as flaky will be difficult for 1.13, but if that doesn't fix it we will need to do that.
Short term fix to bump the test timeout:
As per discussion in release burndown meeting, this is not a 1.13 blocker. please move this issue to 1.14 if there are no pending issues you are concerned about
Ok looks like the test still flaked. Looking at , the mount succeeded after a minute and proceeded to subpath processing, which (correctly) failed. However, for some resaon, the test is not seeing the subpath event and still failed after 4 minutes. On kubelet, subpath processing failed after a minute and generated the event: But the test doesn't see it for some reason, and still times out after 4 more minutes:
I saw the same on CSI volume mount succeeded at: Kubelet correctly failed the pod because of subpath But test still failed: Because the test failed to find the pod event. This looks like a test issue. Not a 1.13 blocker.
can you PTAL at this
Is there a way to check if events are overloaded and not able to handle new events? The MountVolume retries while the driver is not registered is very frequent and could have overloaded the event server.
Checking for the existence of an event seems like the wrong way to test this. Would checking pod status alone be insufficient? Ultimately that is what we really care about.
Checking for the specific event guarantees that we are failing with the correct error. But perhaps events are not reliable enough to depend on. Checking if the Pod never came up running would probably reduce the flakes but could potentially mask false negatives. cc for any thoughts.


Question: if err != nil { return false, fmt.Errorf("got error while getting pod events: %s", err) } <del> if len(events.Items) == 0 { return false, nil // no events have occurred yet </del> <ins> for _, event := range events.Items { if strings.Contains(event.Message, msg) { return true, nil } </ins> } <del> return strings.Contains(events.Items[0].Message, msg), nil </del> <ins> return false, nil </ins> } }

Tests are still flaking and actually it seems like in general storage flakes in slow are flaking more since 11-21 or so: https://k8s- My theory is that the driver registration delay is causing a flood of events, which causes tests that check for events to fail. I think this is revealing multiple issues in both the tests and product, and we should work on fixing them all, although any one of the fixes should mitigate the test flakes: Driver registrar can sometimes take a minute to come up: Volume manager operation executor doesn't seem to be throttling retries or backing off: Tests should not rely on events: Tests should wait for CSI driver to be registered on all nodes before creating test pods:
and have been merged and will address the biggest issues. has been backported to 1.13 already, backport is still pending.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "bytes" <ins> "errors" </ins> "fmt" "math/rand" "net"

introduced somethng that really should be e2e tested.
Would be nice to get this in in the 2 weeks we have after 1.2 branch cut, if no more flakes crop up (but I'm not too optimistic). Bumping to next-release.
/sig network I'm assuming sig-network owns kube-proxy
I thought I noted a change related to this recently, but I may be mistaken.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/lifecycle frozen /remove-lifecycle stale
/remove-lifecycle rotten
Going to try to take this on.
thanks!
I've had some trouble constructing a test for this. should I determine the timeout? There doesn't appear to be an existing mechanism for waiting for service creation. seem to time out rather than be refused - I'm confused by this, as it seems to be the opposite of the intended behavior.
can look at the endpoints + wait for longer than kube-proxy sync time. refused means packets got to their destination but there was nothing listening there. If traffic is blocked, then you will just get a timeout.
I created a service with no matching pods to test manually outside of the e2e suite. # curl test-api curl: (7) Failed to connect to athena-api port 80: Connection timed out # curl --connect-timeout 8 test-api curl: (28) Connection timed out after 8000 milliseconds Looking back at original example in I see that I'm getting the "before" behaviour. An iptables REJECT should send a denial, not a timeout. Am I looking at the wrong thing here? To me this looks like a regression, but I don't think that's likely.


Question: import ( "bytes" <ins> "errors" </ins> "fmt" "math/rand" "net"

I have a cluster from HEAD as of 2 weeks ago: So indeed it seems to not work. If I hit that same service IP from the VM itself I immediately get curl: (7) Failed to connect to 10.0.199.54 port 80: Connection refused. If I insert an iptables jump to KUBE-SERVICES at the front of FORWARD, it correctly rejects. This seems in accordance with the iptables flowchart. I'll do some spelunking to see if there's a reason NOT to reject in this case. Vallery Lancey <:
Can you check IPVS mode?
This is why we need tests. Sigh.
Proposed fix is - still needs your e2e :)
Thanks Tim, if I have time I’ll get a PR up tonight.
Sure will check IPVS mode.
Base on my test, IPVS has no such issue.


Question: <ins> //go:build !providerless // +build !providerless /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package vsphere import ( "testing" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes/fake" "k8s.io/legacy-cloud-providers/vsphere/vclib" ) // Annotation used to distinguish nodes in node cache / informer / API server const nodeAnnotation = "test" func getNode(annotation string) *v1.Node { return &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name: "node1", Annotations: map[string]string{ nodeAnnotation: annotation, }, }, } } func TestGetNode(t *testing.T) { tests := []struct { name           string cachedNodes    []*v1.Node informerNodes  []*v1.Node // "nil" means that the NodeManager has no nodeLister apiServerNodes []*v1.Node // "nil" means that the NodeManager has no nodeGetter expectedNodeAnnotation string expectNotFound         bool }{ { name:           "No cached node anywhere", cachedNodes:    []*v1.Node{}, informerNodes:  []*v1.Node{}, apiServerNodes: []*v1.Node{}, expectNotFound: true, }, { name:           "No lister & getter", cachedNodes:    []*v1.Node{}, informerNodes:  nil, apiServerNodes: nil, expectNotFound: true, }, { name:                   "cache is used first", cachedNodes:            []*v1.Node{getNode("cache")}, informerNodes:          []*v1.Node{getNode("informer")}, apiServerNodes:         []*v1.Node{getNode("apiserver")}, expectedNodeAnnotation: "cache", }, { name:                   "informer is used second", cachedNodes:            []*v1.Node{}, informerNodes:          []*v1.Node{getNode("informer")}, apiServerNodes:         []*v1.Node{getNode("apiserver")}, expectedNodeAnnotation: "informer", }, { name:                   "API server is used last", cachedNodes:            []*v1.Node{}, informerNodes:          []*v1.Node{}, apiServerNodes:         []*v1.Node{getNode("apiserver")}, expectedNodeAnnotation: "apiserver", }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { // local NodeManager cache cache := make(map[string]*v1.Node) for _, node := range test.cachedNodes { cache[node.Name] = node } // Client with apiServerNodes objs := []runtime.Object{} for _, node := range test.apiServerNodes { objs = append(objs, node) } client := fake.NewSimpleClientset(objs...) nodeGetter := client.CoreV1() // Informer + nodeLister. Despite the client already has apiServerNodes, they won't appear in the // nodeLister, because the informer is never started. factory := informers.NewSharedInformerFactory(client, 0 /* no resync */) nodeInformer := factory.Core().V1().Nodes() for _, node := range test.informerNodes { nodeInformer.Informer().GetStore().Add(node) } nodeLister := nodeInformer.Lister() nodeManager := NodeManager{ registeredNodes: cache, } if test.informerNodes != nil { nodeManager.SetNodeLister(nodeLister) } if test.apiServerNodes != nil { nodeManager.SetNodeGetter(nodeGetter) } node, err := nodeManager.GetNode("node1") if test.expectNotFound && err != vclib.ErrNoVMFound { t.Errorf("Expected NotFound error, got: %v", err) } if !test.expectNotFound && err != nil { t.Errorf("Unexpected error: %s", err) } if test.expectedNodeAnnotation != "" { if node.Annotations == nil { t.Errorf("Expected node with annotation %q, got nil", test.expectedNodeAnnotation) } else { if ann := node.Annotations[nodeAnnotation]; ann != test.expectedNodeAnnotation { t.Errorf("Expected node with annotation %q, got %q", test.expectedNodeAnnotation, ann) } } } }) } } </ins>

I am enabling CSI migration on a vSphere cluster. A node was fully drained, all its Pods are deleted, but just before KCM detaches in-tree vSphere volume volumes from it, KCM restarts. A new KCM starts, sees attached volumes in that are not needed by any Pod on the node and calls vSphere in-tree volume plugin . The in-tree vSphere volume plugin + cloud provider marks the volume as detached right away, without reaching to vCenter API. The volume remains attached forever. I don't think it's related to CSI migration, IMO a regular node drain + KCM restart at the same time can cause the same issue. KCM logs: The in-tree cloud provider has a and from it. It apparently has not processed all nodes at the time A/D controller tries to detach the node. The in-tree vSphere volume plugin + cloud provider detaches the volume in vCenter. That's really tricky, it's very hard / impossible to reproduce on a real cluster. I had to inject errors to Detach() in-tree volume call, so they fail until KCM is restarted. Newly started KCM would detach the volume just fine (no error would be injected there), but marks the volume as detached right away. For the record (and search-ability), the CSI driver that tries to attach the volume to another node complains either: Or: cc It could be somewhat related to , however, not all cases of can be related to KCM restart. As I mentioned above, it's very hard/impossible to synchronize KCM restart with draining a node in a real cluster. <details</details<detailsvSphere 7.0.3 </details<details</details<detailsOpenShift </details<detailsCRI-O </details<details</details
/sig storage
Tried just moving a Pod between nodes without CSI migration involved + restarting KCM when the volume is still attached to the old node. In-tree volume plugin on the new node says:
/assign
What CSI driver version and K8s version are you testing with?
Kubernetes is 1.25 and CSI driver is 2.7, still, I think it's not releated to the CSI driver, it's KCM who marks the in-tree volume as detached due to not populated node cache.


Question: <ins> //go:build !providerless // +build !providerless /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package vsphere import ( "testing" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes/fake" "k8s.io/legacy-cloud-providers/vsphere/vclib" ) // Annotation used to distinguish nodes in node cache / informer / API server const nodeAnnotation = "test" func getNode(annotation string) *v1.Node { return &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name: "node1", Annotations: map[string]string{ nodeAnnotation: annotation, }, }, } } func TestGetNode(t *testing.T) { tests := []struct { name           string cachedNodes    []*v1.Node informerNodes  []*v1.Node // "nil" means that the NodeManager has no nodeLister apiServerNodes []*v1.Node // "nil" means that the NodeManager has no nodeGetter expectedNodeAnnotation string expectNotFound         bool }{ { name:           "No cached node anywhere", cachedNodes:    []*v1.Node{}, informerNodes:  []*v1.Node{}, apiServerNodes: []*v1.Node{}, expectNotFound: true, }, { name:           "No lister & getter", cachedNodes:    []*v1.Node{}, informerNodes:  nil, apiServerNodes: nil, expectNotFound: true, }, { name:                   "cache is used first", cachedNodes:            []*v1.Node{getNode("cache")}, informerNodes:          []*v1.Node{getNode("informer")}, apiServerNodes:         []*v1.Node{getNode("apiserver")}, expectedNodeAnnotation: "cache", }, { name:                   "informer is used second", cachedNodes:            []*v1.Node{}, informerNodes:          []*v1.Node{getNode("informer")}, apiServerNodes:         []*v1.Node{getNode("apiserver")}, expectedNodeAnnotation: "informer", }, { name:                   "API server is used last", cachedNodes:            []*v1.Node{}, informerNodes:          []*v1.Node{}, apiServerNodes:         []*v1.Node{getNode("apiserver")}, expectedNodeAnnotation: "apiserver", }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { // local NodeManager cache cache := make(map[string]*v1.Node) for _, node := range test.cachedNodes { cache[node.Name] = node } // Client with apiServerNodes objs := []runtime.Object{} for _, node := range test.apiServerNodes { objs = append(objs, node) } client := fake.NewSimpleClientset(objs...) nodeGetter := client.CoreV1() // Informer + nodeLister. Despite the client already has apiServerNodes, they won't appear in the // nodeLister, because the informer is never started. factory := informers.NewSharedInformerFactory(client, 0 /* no resync */) nodeInformer := factory.Core().V1().Nodes() for _, node := range test.informerNodes { nodeInformer.Informer().GetStore().Add(node) } nodeLister := nodeInformer.Lister() nodeManager := NodeManager{ registeredNodes: cache, } if test.informerNodes != nil { nodeManager.SetNodeLister(nodeLister) } if test.apiServerNodes != nil { nodeManager.SetNodeGetter(nodeGetter) } node, err := nodeManager.GetNode("node1") if test.expectNotFound && err != vclib.ErrNoVMFound { t.Errorf("Expected NotFound error, got: %v", err) } if !test.expectNotFound && err != nil { t.Errorf("Unexpected error: %s", err) } if test.expectedNodeAnnotation != "" { if node.Annotations == nil { t.Errorf("Expected node with annotation %q, got nil", test.expectedNodeAnnotation) } else { if ann := node.Annotations[nodeAnnotation]; ann != test.expectedNodeAnnotation { t.Errorf("Expected node with annotation %q, got %q", test.expectedNodeAnnotation, ann) } } } }) } } </ins>

/assign
/triage accepted
I found an easier way how to reproduce this without extra code in KCM: a StatefulSet with 1 Pod that uses 1 PVC. the node where the Pod runs. down kube-controller-manager to zero (not really possible for me) or kill all kube-controller-manager Pods. This will cause new KCM processes to wait for leader election. In my env. this gives me ~30 seconds for the next step. Delete the Pod. A new one will be scheduled on a different node (because the current node is cordoned) + there is no KCM running at this time that would detach the volume from the old node. KCM wins its leader election, sees a volume attached to a node where it is not needed -it calls of the volume plugin. VSphere cloud provider does not have its caches populated yet (= this bug) -the volume is not detached, but returns success.
dangling attach does not help here. The volume is still attached to the old node even after ~10 minutes. fails with and not with . I can see there is a lot of dangling attach code in the cloud provider, but I did not debug further.


Question: nodeName k8stypes.NodeName, isAttached bool, asw cache.ActualStateOfWorld, <ins> timeout time.Duration, </ins> ) { <del> result := false volumes := asw.GetVolumesToReportAttached(logger) for _, volume := range volumes[nodeName] { if volume.Name == volumeName { result = true </del> <ins> var result bool var lastErr error err := wait.PollUntilContextTimeout(context.TODO(), 50*time.Millisecond, timeout, false, func(context.Context) (done bool, err error) { volumes := asw.GetVolumesToReportAttached(logger) for _, volume := range volumes[nodeName] { if volume.Name == volumeName { result = true } </ins> } <del> } if result == isAttached { t.Logf("Volume <%v> is reported as attached to node <%v>: %v", volumeName, nodeName, result) return </del> <ins> if result == isAttached { t.Logf("Volume <%v> is reported as attached to node <%v>: %v", volumeName, nodeName, result) return true, nil } lastErr = fmt.Errorf("Check volume <%v> is reported as attached to node <%v>, got %v, expected %v", volumeName, nodeName, result, isAttached) return false, nil }) if err != nil { t.Fatalf("last error: %q, wait timeout: %q", lastErr, err.Error()) </ins> } <del> t.Fatalf("Check volume <%v> is reported as attached to node <%v>, got %v, expected %v", volumeName, nodeName, result, isAttached) </del> }

TestRunOneVolumeDetachFailNodeWithReadWriteOnce 4 times since 03-19 23:00 PDT, intermittently ! https://k8s- Still investigating whether this is a tolerable flake or not with more history than testgrid view, or otherwise would appreciate input from anyone with expertise in this test, especially as regards to whether this should be a blocker for tomorrows beta cut. maybe: /sig storage /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is already open for this, looks like it is attempting to fix it
thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. /close
Closing this issue. <detailsIn response to : thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: nodeName k8stypes.NodeName, isAttached bool, asw cache.ActualStateOfWorld, <ins> timeout time.Duration, </ins> ) { <del> result := false volumes := asw.GetVolumesToReportAttached(logger) for _, volume := range volumes[nodeName] { if volume.Name == volumeName { result = true </del> <ins> var result bool var lastErr error err := wait.PollUntilContextTimeout(context.TODO(), 50*time.Millisecond, timeout, false, func(context.Context) (done bool, err error) { volumes := asw.GetVolumesToReportAttached(logger) for _, volume := range volumes[nodeName] { if volume.Name == volumeName { result = true } </ins> } <del> } if result == isAttached { t.Logf("Volume <%v> is reported as attached to node <%v>: %v", volumeName, nodeName, result) return </del> <ins> if result == isAttached { t.Logf("Volume <%v> is reported as attached to node <%v>: %v", volumeName, nodeName, result) return true, nil } lastErr = fmt.Errorf("Check volume <%v> is reported as attached to node <%v>, got %v, expected %v", volumeName, nodeName, result, isAttached) return false, nil }) if err != nil { t.Fatalf("last error: %q, wait timeout: %q", lastErr, err.Error()) </ins> } <del> t.Fatalf("Check volume <%v> is reported as attached to node <%v>, got %v, expected %v", volumeName, nodeName, result, isAttached) </del> }

pull-kubernetes-unit TestRunOneVolumeDetachFailNodeWithReadWriteOnce Jan 7th, 2022 Check volume <fake-plugin/volume-nameis reported as attached to node <fail-detach-node, got false, expected true No response_ /sig storage
/cc
this is the top unit test flake according to
cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
I think this test is no longer flaky by checking the test grid, close it now.
Unfortunately this does still seem to be flaky: Here was the output from : /reopen
Reopened this issue. <detailsIn response to : Unfortunately this does still seem to be flaky: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
still the case
How about adding some logging to help troubleshoot this? I looked at it a little, but it is hard to tell which part of the test is failing. PR
/
This is still happening
/triage accepted
From the logs, in the OneVolumeDetachFailNodeWithReadWriteOnce test, succeeded, but the next verification failed. In verifyVolumeAttachedToNode, the attachState is from calling and within asw, it calls . In verifyVolumeReportedAsAttachedToNode, the volumes are obtained from calling , it seems that the node is not in asw.nodesToUpdateStatusFor at this step. Cannot reproduce locally. I tried to run the test 100 times locally with debug.Stack()... 3 * 100 times all passed.
Tried to run stress test on my linux machine as well but still cannot reproduce. Command: Result: Also tried running n process in parallel. Command: Result:
It seems like it happens more frequently.


Question: nodeName k8stypes.NodeName, isAttached bool, asw cache.ActualStateOfWorld, <ins> timeout time.Duration, </ins> ) { <del> result := false volumes := asw.GetVolumesToReportAttached(logger) for _, volume := range volumes[nodeName] { if volume.Name == volumeName { result = true </del> <ins> var result bool var lastErr error err := wait.PollUntilContextTimeout(context.TODO(), 50*time.Millisecond, timeout, false, func(context.Context) (done bool, err error) { volumes := asw.GetVolumesToReportAttached(logger) for _, volume := range volumes[nodeName] { if volume.Name == volumeName { result = true } </ins> } <del> } if result == isAttached { t.Logf("Volume <%v> is reported as attached to node <%v>: %v", volumeName, nodeName, result) return </del> <ins> if result == isAttached { t.Logf("Volume <%v> is reported as attached to node <%v>: %v", volumeName, nodeName, result) return true, nil } lastErr = fmt.Errorf("Check volume <%v> is reported as attached to node <%v>, got %v, expected %v", volumeName, nodeName, result, isAttached) return false, nil }) if err != nil { t.Fatalf("last error: %q, wait timeout: %q", lastErr, err.Error()) </ins> } <del> t.Fatalf("Check volume <%v> is reported as attached to node <%v>, got %v, expected %v", volumeName, nodeName, result, isAttached) </del> }

But I may have no time to dig into the cause. Let's see.
cc as the test author.
The problem is that this test has a lot of dependency on the sleeps that are hardcoded with a very narrow scope, but our CI seems to be able to stress it much more. As soon as you slightly change one of the sleep duration it starts to flake locally The test can not make an assumption that timing is perfect, even more in the sub second range can someone in sig-storage review this test and make it less dependant on those tight sleeps?
According to most failures, it failed at Line 660. Let me run some stress locally. with 250ms sleep with 100ms sleep
I agree this test case is too brittle due to its specific timing dependencies. To fix this, I suggest reworking the test case to: 1) Add retry with timeout to wait for final conditions 2) Remove the intermediate timing-based state transition checks. Instead, we can keep track of state transitions in the Fakes and validate those stats. wdyt?
Friendly ping from 1.27 CI Signal that this test is still a noisy flake on the master-blocking dashboard as CI Signal has flagged it in and


Question: "github.com/google/go-cmp/cmp" "k8s.io/apimachinery/pkg/util/json" "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/apimachinery/pkg/util/waitgroup" auditinternal "k8s.io/apiserver/pkg/apis/audit" "k8s.io/apiserver/pkg/audit"

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated [Will update shortly] links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Which jobs are flaking: bazel-test-master Which test(s) are flaking: gotest: Testgrid link: Reason for failure: links to appreciatedlinks to specific failures in spyglass appreciated /sig api-machinary Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/assign /triage accepted
Flaked twice ~2 weeks ago but now looking good: Since it's been relatively quiet, I'm going to go ahead and close this for now. /close
Closing this issue. <detailsIn response to : Flaked twice ~2 weeks ago but now looking good: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign I meet this test failure in And I can reproduce it with local stress testing. The reason is the same with
Reopened this issue. <detailsIn response to : /reopen /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // ExcludeMasterFromStandardLB excludes master nodes from standard load balancer. // If not set, it will be default to true. ExcludeMasterFromStandardLB *bool `json:"excludeMasterFromStandardLB" yaml:"excludeMasterFromStandardLB"` <ins> // DisableOutboundSNAT disables the outbound SNAT for public load balancer rules. // It should only be set when loadBalancerSku is standard. If not set, it will be default to false. DisableOutboundSNAT *bool `json:"disableOutboundSNAT" yaml:"disableOutboundSNAT"` </ins> // Maximum allowed LoadBalancer Rule Count is the limit enforced by Azure Load balancer MaximumLoadBalancerRuleCount int `json:"maximumLoadBalancerRuleCount" yaml:"maximumLoadBalancerRuleCount"`

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: is the recommended way to configure the outbound networks together with Azure standard loadbalancer (SLB). However, for load balancer rules are disabled by default. That means all frontend IPs would be used as the SNAT IPs for virtual machines, according to . This would introduce troubles for various security scenarios, as there's no way to get a stable list of egress IPs. Azure cloud provider should support this scenario, and disable the outbound SNAT when outbound rules are configured by provisioning tools. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /kind bug /sig azure /priority critical-urgent


Question: # Clean up kubectl delete jobs pi "${kube_flags[@]}" # Post-condition: no pods exist. <del> kube::test::get_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' </del> <ins> kube::test::wait_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' </ins> # Pre-Condition: no Deployment exists kube::test::get_object_assert deployment "{{range.items}}{{$id_field}}:{{end}}" ''

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: test-cmd runkubectlrun_tests Since when has it been failing: According to triage, the first failure was 2019-02-21 12:00 UTC Testgrid link: Reason for failure: Unknown Anything else we need to know: This may be a flake? I can't tell yet. The testgrid link is not a continuous failure, but it could be that not enough PR's have whatever change is causing the failure Triage link: It looks like it may be happening in CI as well
/sig cli /priority critical-urgent /milestone v1.14
/remove-kind failing-test /kind flake I can't see it consistently happening in CI, so it must be a flake?
What I got when running locally...
And I see the same thing in the build- for
Trying to track down the first occurrence. I think there was an XML parsing error that was obscuring this that was fixed recently. Backtracking through pull-kubernetes-integration failures and visually inspecting them. - runkubectlruntests - 2019-02-22 15:45 PST - runkubectlruntests - 2019-02-21 01:13 PST - runkubectlrun_tests - 2019-02-20 16:50 PST ... and then I don't see this particular test failing during 02-19 or 02-18
So going by that, let's cast a net in commits that landed just prior to the earliest PR failure, and N hours prior
Suspects - ~only because it touches nope, rolled back to right after this landed, no repro after 30 attempts - ~enables server side apply <-- this is my guess~ nope, rolled back to right after this landed, no repro after 30 attempts - reduce lock contention in watchcache
the test might be racy in the first place, maybe enabling the feature changes the time profile of the test.
Welp, now that I have what I think is a local repro loop for this, and I'm bisecting, it's looking like the whole commit range I suspected is fine Looking closer, this particular test case is overly optimistic, it deletes a job and immediately gets pods expecting there to be none.


Question: # Clean up kubectl delete jobs pi "${kube_flags[@]}" # Post-condition: no pods exist. <del> kube::test::get_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' </del> <ins> kube::test::wait_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' </ins> # Pre-Condition: no Deployment exists kube::test::get_object_assert deployment "{{range.items}}{{$id_field}}:{{end}}" ''

I'd still like to nail down what timing change surface this before I go change the to a or using a different set of flags for the call
Well, for some reason seems to be when the flakes started showing up
I realize that the flake is hard to reproduce especially since it's hard to run specific tests in . You can comment a lot of the tests by playing with some of these shell scripts so that they mostly just run the test you want to run. if you do so, you might be able to run 10 or 20 times the test in a row at this PR and right before. I'm curious what the difference would be in term of flake rate.
Yeah that's what I've been doing. It's and specifically looks to be . Which has nothing directly to do with pods, so I guess it's just a matter of this slowing the apiserver down just enough to start tripping a poorly written test.
FYI for the above. My plan for now is just to rewrite the test so it's more forgiving. But I wanted to confirm that it's expected / known that apiserver performance changed a bit as a result of this.
/assign /assign
What metric are you seeing suffer with that commit? There should be no impact.
A racy test goes from flaking 0% of the time to flaking 10-20% of the time Jordan Liggitt <:
flakes: It should probably use instead. The thing is, I can't get it to flake before. I hacked up test/cmd/legacy- to run just this, and none of the other test cases
It definitely should. I'll take a look at job-pod deletion propagation time before/after that commit, but there's nothing I can think of in that path a new networking API group would impact.


Question: for _, testcase := range testcases { t.Run(testcase.name, func(t *testing.T) { <del> buf := new(bytes.Buffer) errBuf := new(bytes.Buffer) </del> <ins> buf := new(buffer) errBuf := new(buffer) </ins> klog.SetOutputBySeverity("INFO", buf) klog.SetOutputBySeverity("WARNING", errBuf)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: No data races How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
for the record as we tend to lose the logs in the gcs bucket
/sig instrumentation


Question: // Flaky operation failures in an e2e test can be captured through this. flakeReport *FlakeReport <del> // To make sure that this framework cleans up after itself, no matter what, // we install a Cleanup action before each test and clear it after.  If we // should abort, the AfterSuite hook should run all Cleanup actions. cleanupHandle CleanupActionHandle </del> // afterEaches is a map of name to function to be called after each test.  These are not // cleared.  The call order is randomized so that no dependencies can grow between // the various afterEaches

Here is a list of TODOs that could be fixed for Ginkgo migration from v1 to v2. [x] Lost the validation on the SIGDescribe: [x] Ginkgo V2 currently cannot get the number of specs that will run in , this might be fixed in the Ginkgo's minor release of V2 in the future. [x] <delGinkgo dry-run mode has dropped the test spec in V2 (), consider to switch back to use to generate test spec when the feature is ready in . </del--- This has been fixed in Ginkgo 2.1.4 [x] <delMigrate all the custom reporter in , should be lightweight and not sure where those custom reporters are used. </delFixed here: [x] Replace all the camel case flags to kebab case flags, e.g. to . [x] <delRemove the legacy e2e test binary's parameter and in 1.28, those parameters were used to manage report, this is no longer needed for Ginkgo v2. </delno longer needed. [x] upgrade vendor when is migrated to ginkgo v2, this will help to remove the dependency of - [x] <delrevisit to explore the possibility to cover the test in Ginkgo V2. </delfixed by [x] remove the workaround introduced in Ginkgo V1 ( is not called after hitting ^c): partially addressed by: , some cleanup needed for each sig before removing the hook entirely. I think has the pr to fix this entirely, but I cannot get the link for that, mark it as resolved. : [x] New work item (?): capture only a single output text output stream () in JUnit (this issue was resolved along with another one in ) [ ] Due to the junit format change, some tool which use to grab "value" directly cannot get the failure reason as what it does in v1. [x] Test failures are missing information since GinkgoV2 migration [ ] is clustering stack traces, not failure text () Address those leftovers if there is any update from Ginkgo in the future. see the relevant link on the each of the issues.


Question: // Flaky operation failures in an e2e test can be captured through this. flakeReport *FlakeReport <del> // To make sure that this framework cleans up after itself, no matter what, // we install a Cleanup action before each test and clear it after.  If we // should abort, the AfterSuite hook should run all Cleanup actions. cleanupHandle CleanupActionHandle </del> // afterEaches is a map of name to function to be called after each test.  These are not // cleared.  The call order is randomized so that no dependencies can grow between // the various afterEaches

No response_ <details</details<detailsall </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/cc
/cc
hello! I would like to work on this issues
I think we cannot do anything now except the last two, and those two should be based on top of the:
I would like to work on task 5 of this issue
no problem, pls wait for an while until that PR got merged, then you can create a pr to fix those deprecated parameters.
Regarding : searching for points out the places where the workaround is used and needs to be replaced once is solved.
I just noticed this comment in
seems like is not used in k/k, but in other sigs.
hmm... I just grepped the test code, will push a pr to fix this.
/open
/re-open
okay, this is not resolved actually, but I have no way to re-open it.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
~ it seems that some more jobs needs to be adapted to not use parallel~ [repo][0]=kubernetes/test-infra~ ~ACTION REQUIRED: Ginkgo V2 doesn't accept go test's flags to parallelize Ginkgo specs, please switch to use or instead.~ Sorry for the noise, the flag is passed to kubetest, and it correctly translate it to , it is not the issue here
np, thanks for the update!
New work item (?): capture only a single output text output stream () in JUnit.
updated the issue, haven't got a chance to take a close look at the item though.
This should be resolved by "open stdout" now should only appear if a test intentionally records SpecReports.
mark it as resolved.
is this complete now?


Question: // Flaky operation failures in an e2e test can be captured through this. flakeReport *FlakeReport <del> // To make sure that this framework cleans up after itself, no matter what, // we install a Cleanup action before each test and clear it after.  If we // should abort, the AfterSuite hook should run all Cleanup actions. cleanupHandle CleanupActionHandle </del> // afterEaches is a map of name to function to be called after each test.  These are not // cleared.  The call order is randomized so that no dependencies can grow between // the various afterEaches

, Only two of them left, I need to follow this one, here is the issue for this one opened in the Ginkgo: I think this should be fixed in the third party tools instead of Kubernetes, if we agree with that, we can mark it as completed as well. For the first one, I will continue to follow until there is a update from Ginkgo's side, for the later one, there is no consensus yet IIUC, we can open a separate issue for that, I am gonna to close this one since there is no much thing we can do now. /close
We agreed to fix the tool, but had no issue open for it. I just created one:
Another issue for


Question: if err != nil { t.Fatalf("Failed to create scheduler: %v", err) } <ins> // Profiles </ins> profiles := make([]string, 0, len(s.Profiles)) for name := range s.Profiles { profiles = append(profiles, name)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --here is supposed to be set through , but is not called from anywhere except tests, so always returns . should be set through here. Am I missing something? N/A None Kubernetes version (use ): master () Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: None
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
Thanks for pointing that out. The plumbing of extenders to the framework impact calling extenders on preemption. That seems to be a path not covered by tests end to end. Can you submit a PR to fix this? as you mentioned we need to pass the extenders on profile creation. We need to patch the fix back to v1.21 and v1.20 (the release where this was refactored). /cc
I have submitted a PR for a fix.


Question: defaultPodResourcesPath    = "/var/lib/kubelet/pod-resources" defaultPodResourcesTimeout = 10 * time.Second defaultPodResourcesMaxSize = 1024 * 1024 * 16 // 16 Mb <ins> kubeletReadOnlyPort        = "10255" kubeletHealthCheckURL      = "http://127.0.0.1:" + kubeletReadOnlyPort + "/healthz" </ins> ) func getNodeSummary() (*stats.Summary, error) {

ci-kubernetes-node-kubelet-serial-hugepages when start the pod with the resources requests that contain multiple hugepages resources should set correct hugetlb mount and limit under the container cgroup Unexpected error: "pod ran to completion", Multiple sizes huge pages is going to be graduated to GA, so having this test job fixed is quite important.
/sig node
/assign
/triage accepted
/cc
I did a small analysis and I can see that the pod failed to start, I can not find any failure messages under the kubelet log, so it really hard to say why it failed.
were you able to reproduce it locally? btw, there is a "right way" to fix this - to fix dynamic update of a hugepage resources. This would eliminate need in a kubelet restart. There is even a PR for that: , we just need to make it working again.
after conversation with on slack I'd say we should stick to the current procedure - restarting kubelet when hugepages allocation changes. So, we should investigate why kubelet doesn't restart properly in this test case and fix it.
Nope, I failed to reproduce it locally, also under the CI, it does not appear often. The best thing will be to describe the pod in case if it failed, so we will see the reason.
under the code we have so from the log , I can see that the kubelet received the pod before the kubelet stopped, so the problem that pod dropped to Failed state because of the kubelet stop.
So some additional details, the one who responsible to restart the kubelet again, is the server service: restart the kubelet service are waiting for hugepages to be available are starting the pod server service restart the kubelet again because it was not healthy for too long - failed to start because of the kubelet restart
great findings! Should we just stop the kubelet and let the server service to start it again? We can also check if the kubelet is restarted successfully before starting the pod.
I tried it under the PR , also I unsure how can we check if the kubelet restarted successfully. All tests that I wrote before I checked if the node has NotReady state and waited until it will be Ready, but in case of the restart probably the node will always have Ready state because the time period is very short. I will continue to investigate and will check locally how can I fix it.


Question: defaultPodResourcesPath    = "/var/lib/kubelet/pod-resources" defaultPodResourcesTimeout = 10 * time.Second defaultPodResourcesMaxSize = 1024 * 1024 * 16 // 16 Mb <ins> kubeletReadOnlyPort        = "10255" kubeletHealthCheckURL      = "http://127.0.0.1:" + kubeletReadOnlyPort + "/healthz" </ins> ) func getNodeSummary() (*stats.Summary, error) {

So the only reliable solution that I found, it just trying to start the pod number of times. I tested it 5 times and the test passed. Can you please review
Can we access kubelet service from hugepage test cases? If we can we can use its APIs like or to check if kubelet is ready after the restart. Overall I have a feeling of duplication test/e2enode/services functionality in This is a main reason for this and similar bugs I believe. It would be nice to get rid of this duplication if possible. That would nicely isolate kubelet service controls in one place and eliminate conflicts like this one. wdyt?
not really, the used mostly to start all needed components to emulate the cluster(not only the kubelet) and the test does not have access to it. I think you can separate the readiness check under the importable method, but it worth the separate PR.
I updated the PR to check the healthiness of the kubelet and looks like it works just fine.
thank you. it looks good to me so far. I'll add lgtm label when is merged and tests are all green.
the job is still flaky. can you reopen this issue?
I opened a different one - , the issue can affect any job, not only the hugepages one.
I can open a fix to W/A an issue(start the pod until it will succeed), it should fix all problems, WDYT?
i think it's ok as a workaround, until is fixed
would it be better to wait until api server reports correct resource value before running the pod?
We are waiting for it, the problem that the scheduler and the kubelet see a different picture regarding resources requested by pods.
ok. In this case restarting the pod until it succeeds seems reasonable to me to fix the flakiness until is fixed.
any progress on the workaround fix?


Question: } // if we're trying to hit the kube-apiserver and there wasn't an explicit config, use the in-cluster config <del> if server == "kubernetes.default.svc" { </del> <ins> if target == "kubernetes.default.svc" { </ins> // if we can find an in-cluster-config use that.  If we can't, fall through. inClusterConfig, err := rest.InClusterConfig() if err == nil {

We are getting the following error when the apiserver connects the webhook on localhost (configured via URL). We should only use the SSL tunnel for the connections to nodes when the webhooks are running as services. /sig api-machinery /kind bug /assign
/close


Question: } } <ins> // printPodsMultiline prints multiple pods with a proper alignment. func printPodsMultiline(w PrefixWriter, title string, pods []api.Pod) { printPodsMultilineWithIndent(w, "", title, "t", pods) } // printPodsMultilineWithIndent prints multiple pods with a user-defined alignment. func printPodsMultilineWithIndent(w PrefixWriter, initialIndent, title, innerIndent string, pods []api.Pod) { w.Write(LEVEL_0, "%s%s:%s", initialIndent, title, innerIndent) if pods == nil || len(pods) == 0 { w.WriteLine("<none>") return } // to print pods in the sorted order sort.Slice(pods, func(i, j int) bool { cmpKey := func(pod api.Pod) string { return pod.Name } return cmpKey(pods[i]) < cmpKey(pods[j]) }) for i, pod := range pods { if i != 0 { w.Write(LEVEL_0, "%s", initialIndent) w.Write(LEVEL_0, "%s", innerIndent) } w.Write(LEVEL_0, "%sn", pod.Name) } } </ins> // printPodTolerationsMultiline prints multiple tolerations with a proper alignment. func printPodTolerationsMultiline(w PrefixWriter, title string, tolerations []api.Toleration) { printTolerationsMultilineWithIndent(w, "", title, "t", tolerations)

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: I need to know to which Pod(s) a PVC is being mounted to, without having to go through each Pod one by one checking the Volume mounts and the PersistentVolumeClaim. The kubectl describe pvc and get pvc, only tell you to which Persistent Volume the PVC is bound to, but not if is being used by a Pod(s) What you expected to happen: Add a new field in the describe or get that lists the pods that are using the PVC as a volume mount How to reproduce it (as minimally and precisely as possible): Have a PV/PVC bound and create a pod using a volume mount Anything else we need to know?: Not sure if this is only for Azure Disk, which is the Storage Class I am using. Environment: Kubernetes version (use ): 7 Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4 Install tools: Azure Kubernetes Services Others: -n/a
/sig storage
/sig cli
I think the kubectl describe can just construct this on the fly. Another angle is should we add this information to PVC object itself
That will be great, as right now the only way to correlate a PVC to a Pod is via a "describe" on the Pod.
I can work on a PR for this!
GitHub didn't allow me to assign the following users: clandry94. Note that only and repo collaborators can be assigned. <detailsIn response to : I can work on this! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
+1 for the feature. cc
How to find "what pods are using particular PVC through command line"
+1 for the feature.
this feature will be in Kubernetes 1.12
ok kubernetes 1.12 released, how to? which command?
Can someone point to the documentation or paste command to get the pods which have the PVC in use
see the associated PR to see an example usage of this The line in should show it.
Workaround for the meantime:
Thanks , this workaround is great for those people who still use old kubernetes version.


Question: } } <ins> // printPodsMultiline prints multiple pods with a proper alignment. func printPodsMultiline(w PrefixWriter, title string, pods []api.Pod) { printPodsMultilineWithIndent(w, "", title, "t", pods) } // printPodsMultilineWithIndent prints multiple pods with a user-defined alignment. func printPodsMultilineWithIndent(w PrefixWriter, initialIndent, title, innerIndent string, pods []api.Pod) { w.Write(LEVEL_0, "%s%s:%s", initialIndent, title, innerIndent) if pods == nil || len(pods) == 0 { w.WriteLine("<none>") return } // to print pods in the sorted order sort.Slice(pods, func(i, j int) bool { cmpKey := func(pod api.Pod) string { return pod.Name } return cmpKey(pods[i]) < cmpKey(pods[j]) }) for i, pod := range pods { if i != 0 { w.Write(LEVEL_0, "%s", initialIndent) w.Write(LEVEL_0, "%s", innerIndent) } w.Write(LEVEL_0, "%sn", pod.Name) } } </ins> // printPodTolerationsMultiline prints multiple tolerations with a proper alignment. func printPodTolerationsMultiline(w PrefixWriter, title string, tolerations []api.Toleration) { printTolerationsMultilineWithIndent(w, "", title, "t", tolerations)

However it will meet an error if the pod have no volumes, so I improves it by adding a to check whether the volumes exists or not.
python script to list pods with pvc in namespace(s): Is a part of a cli to set pods with velero/restic annotations. Hope it helps someone :)
For posterity... There is a Used By: field in the output of describe.
client-go didn't give this api... ,PodPVCIndex can solve the problem
Hi thanks worked for me ;) but I want to make it reverse my workarounds with adding after didn't worked do you have any ideas
$ kubectl describe pvc -A Used" Name: ocs-deviceset-lso-volumeset-0-data-1b45dn Namespace: openshift-storage StorageClass: lso-volumeset Used By: rook-ceph-osd-1--8fwpd Also useful ^
the reason it didn't work is because each return does, in point of fact, have a key defined for claimName. That key is just set to null value. So, what you'd actually want to do is test for nulls. As nice as answer is, it's a PITA to parse unstructured human-readable command output, as opposed to command output in a markup language that well known shell parsers like jq and yq can work with. Towards that end, because I had use of this, I came up with the following: That simple not null test at the end plucks out all your nulls. Obviously invert the last conditional if you want all the nulls, but you'll get all sorts of temp things that way that you probably won't care about.


Question: NoSnatTestProxy          = ImageConfig{e2eRegistry, "no-snat-test-proxy", "1.0", true} NWayHTTP                 = ImageConfig{e2eRegistry, "n-way-http", "1.0", true} // When these values are updated, also update cmd/kubelet/app/options/options.go <del> Pause               = ImageConfig{gcRegistry, "pause", "3.0", false} </del> <ins> Pause               = ImageConfig{gcRegistry, "pause", "3.0", true} </ins> Porter              = ImageConfig{e2eRegistry, "porter", "1.0", true} PortForwardTester   = ImageConfig{e2eRegistry, "port-forward-tester", "1.0", true} Redis               = ImageConfig{e2eRegistry, "redis", "1.0", true}

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Failures - https://k8s- Part of pause is not used properly for tests for other arches than x86. As the current pause image is not a manifest image, so we need to use a different image for different archs like for , for etc.. With this, I'm using logs-generator from repo What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig testing
/assign


Question: import ( "context" "fmt" <del> "strconv" </del> <ins> "os" </ins> "strings" v1 "k8s.io/api/core/v1"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If computer name prefix does not match VMSS name, then Azure cloud provider will use VMSS API instead of using metadata service data. This leads to throttling, because every affected kubelet is making API requests. What you expected to happen: Azure cloud provider should use metadata service data instead of calling VMSS API when instance is current. How to reproduce it (as minimally and precisely as possible): Set computer name prefix of the VMSS to something different than default. You should observe Azure provider making requests to the VMSS API rather than using metadata service, which leads to throttling if you have multiple kubelets with this bug. Anything else we need to know?: This is bug was introduced by - returns false even if instance is current, because it expects node name to match VMSS name + instance id, however node name is derived from the computer-name-prefix which can be different than VMSS name. Node name should be matched against , not against . Environment: Kubernetes version (use ): 1.18.x and higher Cloud provider or hardware configuration: azure /sig cloud-provider
/sig cloud-provider
/area provider/azure
CC
/remove-sig cloud-provider /sig cloud-provider (trying to remove the label)
/sig cloud-provider
Thanks for the reporting. Unfortunately, computer name prefix is not included in IMDS, so it can't be used without VMSS API for constructing the NodeName. Would check what could we do to workaround the issue.
Can you memorize the computer name prefix in the first request to the Azure API and then use it for the isCurrentInstance check?
opened to fix the issue. Would you like to have a look?
I can confirm that your change should fix the problem I reported. Do you know if your fix will be to 1.18.x?
is working on the cherry pick
Thank you for quick fix!


Question: // list all pods to include the pods that don't match the rc's selector // anymore but has the stale controller ref. // TODO: Do the List and Filter in a single pass, or use an index. <del> pods, err := rm.podLister.Pods(rc.Namespace).List(labels.Everything()) </del> <ins> allPods, err := rm.podLister.Pods(rc.Namespace).List(labels.Everything()) </ins> if err != nil { return err } // Ignore inactive pods. var filteredPods []*v1.Pod <del> for _, pod := range pods { </del> <ins> for _, pod := range allPods { </ins> if controller.IsPodActive(pod) { filteredPods = append(filteredPods, pod) }

https://k8s- Failed: [] Network Partition [Disruptive] [Slow] [] [ReplicationController] should eagerly create replacement pod during network partition when termination grace is non-zero {Kubernetes e2e suite}
https://k8s- seems to be the first failure here. It seems to have gone in with a bunch of RC/RS changes () cc/


Question: "time" v1 "k8s.io/api/core/v1" <ins> apierrors "k8s.io/apimachinery/pkg/api/errors" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" utilfeature "k8s.io/apiserver/pkg/util/feature"

When using a generic ephemeral volume, the volume binder in kube-scheduler typically processes a new pod before the generic volume controller in kube-controller-manager has a chance to create the PVC. This leads to events for pods using generic ephemeral volumes: It would be nice to suppress these warnings for a short while (duration to be determined) after the pod creation. Later they should be emitted because then they indicate that the generic volume controller isn't working properly. This makes nicer to use.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage


Question: <ins> # See the OWNERS docs at https://go.k8s.io/owners approvers: - lavalamp - smarterclayton - deads2k - sttts - liggitt - caesarxuchao reviewers: - thockin - lavalamp - smarterclayton - wojtek-t - deads2k - derekwaynecarr - caesarxuchao - cheftako - mikedanese - liggitt - gmarek - sttts - ncdc - logicalhan - tallclair labels: - sig/api-machinery </ins>  No newline at end of file

This issue proposes the writing of a new test for the API endpoint with the intention of promoting it to Conformance. This issue contains a very masic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this query, there is only one API endpoint which is for Namespace that isn't being covered. We selected patchCoreV1Namespace to write a test for. Dependencies: The mock test: If a test with these calls is build and merged, conformance will go up by 1 endpoint.
/sig testing /sig architecture /area conformance