Question: parameters[k] = v.(string) } } <ins> if snapshotProvider, ok := snapshotClass.Object["driver"]; ok { snapshotter = snapshotProvider.(string) } </ins> case d.SnapshotClass.FromFile != "": snapshotClass, err := loadSnapshotClass(d.SnapshotClass.FromFile) framework.ExpectNoError(err, "load snapshot class from %s", d.SnapshotClass.FromFile)

Making it depend on the driver name was probably unnecessary. A single flag for all incarnations of that test probably would have been sufficient and thus better ("keep it simple, stupid!" - the stupid being me here...). It also shouldn't have used the global flag set. In other words:
/reopen I don't really want to get into the habit of defining per test case flags in There are thousands of test cases and it would be unwieldy to discover all the flags available and understand how to use them. What factors would influence how many volumes you want the test to create? Can we just hardcode a value in the test case?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The defaults must be usable so that users do not need to understand them. The flags can be discovered with - not ideal, but at least they show up. That's effectively what the default value for the flag does. This is a way to to run a custom stress test: the default is so low, it just tests that the feature works. By increasing it as high as the underlying CSI driver and storage system supports, one can run this (and potentially other tests) as part of stress testing. Not sure whether it's worth supporting this. It seemed worthwhile at the time, and better than hard-coding just one value.
The bigger issue is: how do users of the test suite actually learn what tests they run and what those tests do? I suspect the answer to that is simply "read the source", in which case they'll also find the knobs that can be used to tweak testing.
Since increasing the parameter depends on the csi driver/storage system, then I would recommending splitting this up into 2 test cases: 1) test basic functionality 2) test stress scenario, and add the parameter to the driver test config similarly to other stress tests:
At least for the storage tests, because we are intending that they be run externally, having all the tunables in one place I think makes it easier to discover.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: parameters[k] = v.(string) } } <ins> if snapshotProvider, ok := snapshotClass.Object["driver"]; ok { snapshotter = snapshotProvider.(string) } </ins> case d.SnapshotClass.FromFile != "": snapshotClass, err := loadSnapshotClass(d.SnapshotClass.FromFile) framework.ExpectNoError(err, "load snapshot class from %s", d.SnapshotClass.FromFile)

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle stale /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
I saw that the flag was removed in , I think we can close this one if doesn't apply anymore
/close The flag was removed in #diff-20517805986874f69e3254bed93d59996d5a6fd571a70ab8120736ded5aafa24
You can't close an active issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /close The flag was removed in #diff-20517805986874f69e3254bed93d59996d5a6fd571a70ab8120736ded5aafa24 </details
/close
Closing this issue.


Question: parameters[k] = v.(string) } } <ins> if snapshotProvider, ok := snapshotClass.Object["driver"]; ok { snapshotter = snapshotProvider.(string) } </ins> case d.SnapshotClass.FromFile != "": snapshotClass, err := loadSnapshotClass(d.SnapshotClass.FromFile) framework.ExpectNoError(err, "load snapshot class from %s", d.SnapshotClass.FromFile)

<detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: There are a few things to note in this description.  First is that we are running the ```kubernetes/cassandra``` image.  This is a standard Cassandra installation on top of Debian.  However it also adds a custom [```SeedProvider```](https://svn.apache.org/repos/asf/cassandra/trunk/src/java/org/apache/cassandra/locator/SeedProvider.java) to Cassandra.  In Cassandra, a ```SeedProvider``` bootstraps the gossip protocol that Cassandra uses to find other nodes.  The ```KubernetesSeedProvider``` discovers the Kubernetes API Server using the built in Kubernetes discovery service, and then uses the Kubernetes API to find new nodes (more on this later) <del> You may also note that we are setting some Cassandra parameters (```MAX_HEAP_SIZE``` and ```HEAP_NEWSIZE```).  We also tell Kubernetes that the container exposes both the ```CQL``` and ```Thrift``` API ports.  Finally, we tell the cluster manager that we need 0.5 cpu (0.5 core). </del> <ins> You may also note that we are setting some Cassandra parameters (```MAX_HEAP_SIZE``` and ```HEAP_NEWSIZE```) and adding information about the [namespace](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/namespaces.md).  We also tell Kubernetes that the container exposes both the ```CQL``` and ```Thrift``` API ports.  Finally, we tell the cluster manager that we need 0.5 cpu (0.5 core). </ins> Given this configuration, we can create the pod from a file specification as follows

The namespace seems to be hard coded :
cc:
Thanks. We can improve it after v1 release.


Question: manager.podStoreSynced = alwaysReady syncAndValidateDaemonSets(t, manager, ds, podControl, 1, 0) } <ins> func TestDSManagerInit(t *testing.T) { // Insert a stable daemon set and make sure we don't create an extra pod // for the one node which already has a daemon after a simulated restart. ds := newDaemonSet("test") ds.Status = extensions.DaemonSetStatus{ CurrentNumberScheduled: 1, NumberMisscheduled:     0, DesiredNumberScheduled: 1, } nodeName := "only-node" podList := &api.PodList{ Items: []api.Pod{ *newPod("podname", nodeName, simpleDaemonSetLabel), }} response := runtime.EncodeOrDie(testapi.Default.Codec(), podList) fakeHandler := utiltesting.FakeHandler{ StatusCode:   200, ResponseBody: response, } testServer := httptest.NewServer(&fakeHandler) // TODO: Uncomment when fix #19254 // defer testServer.Close() client := client.NewOrDie(&client.Config{Host: testServer.URL, GroupVersion: testapi.Default.GroupVersion()}) manager := NewDaemonSetsController(client, controller.NoResyncPeriodFunc) manager.dsStore.Add(ds) manager.nodeStore.Add(newNode(nodeName, nil)) manager.podStoreSynced = alwaysReady controller.SyncAllPodsWithStore(manager.kubeClient, manager.podStore.Store) fakePodControl := &controller.FakePodControl{} manager.podControl = fakePodControl manager.syncHandler(getKey(ds, t)) validateSyncDaemonSets(t, fakePodControl, 0, 0) } </ins>

The test is in flaky suite and no one seems to care. We need an owner for this test. cc Last failure log:
do you think you can take a look at this?
This is happening because of a bad assumption. The assumption is: if the reflector's store is consistent with etcd, the store given back to clients of the framework.Controller is too. I'd really like a general fix applicable to all controllers for this problem, but fixing it just for the RCs is pretty easy. In more detail: This function reports when the reflector has listed from apiserver. But the store given to the reflector is the fifo of the framework.Controller. The store given back to the caller is still not consistent, and will not be till the framework.Controller puts all the objects from the fifo into the client state store (). The rc specific fix: Before starting the rc manager, seed the expectations store with an expectation like {rc:foo, adds: foo.Status.Replicas, deletes:0}. This will wait for status.Replicas number of ADD events till a timeout. The general fix: Somehow expose this through framework.Controller.HasSynced (clients don't care about the fifo anyway, shouldn't it just be an implementation detail?) ideas? cc'ing people who have written controllers that might be interested I don't think this (and generally anything flaky for 1 release) is a p0, I can fix it if you don't mind dealing with the flakes till I've cleared things off my plate. I'd be more than happy to review a fix.
Well, it's a P0 because test is flaky. E.g last run:
- did you have chance to look into it?
The "DaemonRestart" tests are still labeled at . I will send a PR to promote them.
are you still planning to send aPR to promote the DaemonRestart tests out of flaky? I think this is the right thing to do.
Ack; sorry, I lost this in the muck.


Question: return fmt.Errorf("waiting for apiserver timed out") } <del> // WaitForApiserverRestarted waits until apiserver's restart count increased. func WaitForApiserverRestarted(c clientset.Interface, initialRestartCount int32) error { </del> <ins> // waitForApiserverRestarted waits until apiserver's restart count increased. func waitForApiserverRestarted(c clientset.Interface, initialRestartCount int32) error { </ins> for start := time.Now(); time.Since(start) < time.Minute; time.Sleep(5 * time.Second) { <del> restartCount, err := GetApiserverRestartCount(c) </del> <ins> restartCount, err := getApiserverRestartCount(c) </ins> if err != nil { Logf("Failed to get apiserver's restart count: %v", err) continue

This test is failing in both gce serial suite: http://k8s- /sig network /priority failing-test /priority critical-urgent /kind bug /status approved-for-milestone cc /assign PTAL
xref
/milestone v1.10
this also fails in http://k8s-
Failing to connect to Services at all. I can't even ping pod IPs:
So I can ping pods same-node and cross-node right up until it kills the apiserver. Rebooting fixes a given node. The apiserver demonstrably comes back - I can query it for info...
- I could use help - have to shift attention soon.
FWIW, I am testing on a Calico-enabled node
I'll take a look at this as well.
Weird, now even after a reboot, this condition is sticky. I have a cluster in this state, if you want to poke.
ugh, now it is resolved.
I just repro'ed without the e2e - have a cluster, proved ping works, killed apiserver ping fails.
Interesting - docker shows the pods not running, but apiserver says they are alive.
Update: docker sees no running pods. API server thinks kubelet is running them. kubelet is getting "" from apiserver, even after kubelet restart.
Update and recap: Bring up a cluster Run some pods Prove that pods are running and services work pkill -9 apiserver Wait. Watch kubelets "lose" pods that were assigned to them Observe that pods are not running and services do not work Observe that apiserver says the pods are running pkill -9 kubelet Observe that it is still broken kubectl delete one pod Observe that kubelet is fixed for that one pod, but not others Q: Why is kubelet not getting a full listing when it restarts?
What exactly does it mean for it to be fixed for a deleted pod?
Kubelet started running the pod that was deleted - it got the update for that one.
The cluster I was testing was a week or 2 old. I built a cluster from HEAD and I can not repro - said the same. Is this test running against head of master?
I also ran the test manually against my new cluster and it passed
FWIW I was also unable to reproduce this yesterday on a cluster from HEAD.


Question: return fmt.Errorf("waiting for apiserver timed out") } <del> // WaitForApiserverRestarted waits until apiserver's restart count increased. func WaitForApiserverRestarted(c clientset.Interface, initialRestartCount int32) error { </del> <ins> // waitForApiserverRestarted waits until apiserver's restart count increased. func waitForApiserverRestarted(c clientset.Interface, initialRestartCount int32) error { </ins> for start := time.Now(); time.Since(start) < time.Minute; time.Sleep(5 * time.Second) { <del> restartCount, err := GetApiserverRestartCount(c) </del> <ins> restartCount, err := getApiserverRestartCount(c) </ins> if err != nil { Logf("Failed to get apiserver's restart count: %v", err) continue

Did some investigations and had some findings for this specific failure: Traced down a bit on the history, the first occurrence was https://k8s- From the , seems suspicious because it modified . Also found reported a relevant symptom: The fix () for that was merged a day ago and may explained why we can't repro from HEAD anymore.
nice, is this issue reproducible without the fix ()?
Sorry should have had confirmed this, bringing up a cluster now.
Quick update: I was able to repro the same failure () from HEAD with reverted. Though this test is now flaky on some other errors: https://k8s- https://k8s- They all seem to failed while communicating with apiserver.
Thanks ! So fixed the issue and I am curious about what introduced the issue. Do you happen to know when did this test start to fail? Just trying to find out what went wrong in between. We have some other possibly related issues so finding out the root cause of this one would be super helpful.
/assign
The first occurrence (for that exact failure) I noticed was https://k8s-, which happened on 2018-02-15 09:49 PST.
You all rock! I see a couple greens on it now, but still mostly red?
The test is now flaky mostly on creating an exec pod after restarting apiserver. Some sample failures: To elaborate a bit more, a snippet of the test procedure is: ssh to master and (sending a SIGTERM) Keep polling on /healthz on apiserver until 200 is returned. Create an exec pod for later steps (and failed on various erros --- EOF, connection refused, apiserver terminating). According to above test failures, my suspicion is that the test is assuming the apiserver is restarted too soon. It is possible that apiserver captured SIGTERM and did a graceful termination, so that polling on /healthz may succeed even in the termination phase. I'm not surprised trying to create a pod via a terminating apiserver will hit those errors. Could anyone confirme the SIGTERM theory?
Spoke with folks, sounds like after SIGTERM apiserver does wait a bit for remaining requests to finish. I will take a stab to refine the test. cc
Sent .
Test is now passing on gce but failing on gke.


Question: return fmt.Errorf("waiting for apiserver timed out") } <del> // WaitForApiserverRestarted waits until apiserver's restart count increased. func WaitForApiserverRestarted(c clientset.Interface, initialRestartCount int32) error { </del> <ins> // waitForApiserverRestarted waits until apiserver's restart count increased. func waitForApiserverRestarted(c clientset.Interface, initialRestartCount int32) error { </ins> for start := time.Now(); time.Since(start) < time.Minute; time.Sleep(5 * time.Second) { <del> restartCount, err := GetApiserverRestartCount(c) </del> <ins> restartCount, err := getApiserverRestartCount(c) </ins> if err != nil { Logf("Failed to get apiserver's restart count: %v", err) continue

Apparently we don't have a way to trigger apiserver restart on gke at this moment. Sending a PR to skip this test on gke.
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Commented too quick. On gke we do a same-version master upgrade. Will send another fix :/


Question: var policy Policy switch topologyPolicyName { <del> case PolicyNone: policy = NewNonePolicy() </del> case PolicyBestEffort: policy = NewBestEffortPolicy(numaInfo, opts)

Follow up from: See details there. When is , there is no need to discover the NUMA topology. /sig node /priority backlog /kind feature /help-wanted
/cc
/triage accepted
I would like to take this up also can I get more brief information about it? code review comments from asking that inside the NewManager() implementation, we need to skip NewPolicyOptions and NewNUMAInfo but when the NumaTopology comes kubelet issue also comes. I need some more information to start with this!
/cc
When there is no topology manger policy specified, NewManager should be a no-op. Just skip all checks.
Exactly. Just to add: we may need few iterations over the change to get a clean flow in the call.
/assign


Question: # Base images - name: "k8s.gcr.io/debian-base: dependents" <del> version: buster-v1.3.0 </del> <ins> version: buster-v1.4.0 </ins> refPaths: - path: build/workspace.bzl match: tag =

The based container images for are shipped with the wrong architecture within the manifest if not amd64: Refers to We updated the images a few minutes ago to buster-v1.3.0, which are now ready to be used as base for the multi-architecture images. If the images contain the right architectures, then it may magically happen that the kube-proxy images are fixed, too. This is because we already use the build syntax for it: /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release


Question: - name: ssl-certs mountPath: /etc/ssl/certs readOnly: true <del> - image: gcr.io/google_containers/addon-resizer:1.5 </del> <ins> - image: gcr.io/google_containers/addon-resizer:1.6 </ins> name: heapster-nanny resources: limits:

HPA tests are currently failing in gke-large cluster suite: I did some debugging and the problem seems to be related to pod-nanny. Looking into logs from the last run: was first scheduled on the large node dedicated to it pod-nany updated heapster deployment then heapster was scheduled on the normal node (when it actually shouldn't fit) that point it was never updated by the pod nanny. that point, heapster started crashlooping (OOMing) [which in fact was expected in too small machine] So it's pretty clear that that pod-nanny changed the amount of resources to something smaller, which in fact it shouldn't.


Question: } // returns the item and true if it is found and not expired, otherwise nil and false. <ins> // If this returns false, it has locked c.inFlightLock and it is caller's responsibility // to unlock that. </ins> func (c *timeCache) get(key string) (T, bool) { c.lock.RLock() defer c.lock.RUnlock() data, ok := c.cache[key] now := c.clock.Now() if !ok || now.Sub(data.lastUpdate) > c.ttl { <ins> // We must lock this while we hold c.lock-- otherwise, a writer could // write to c.cache and remove the channel from c.inFlight before we // manage to read c.inFlight. c.inFlightLock.Lock() </ins> return nil, false } return data.item, true } <ins> // c.inFlightLock MUST be locked before calling this. fillOrWait will unlock it. </ins> func (c *timeCache) fillOrWait(key string) chan T { <del> c.inFlightLock.Lock() </del> defer c.inFlightLock.Unlock() // Already a call in progress?

Failed due to an apparently unrelated PR: https://travis-
Problem here is that I left a small chance of a double-call in my parallelized cache. Didn't realize this was being tested. Will fix.


Question: if err != nil { t.Fatalf(err.Error()) } test.prepare(base) pathToCreate := filepath.Join(base, test.path) fd, err := doSafeOpen(pathToCreate, base)

Subpath unix socket tests can flake because of long socket path. On Linux the path limit should be 108 chars or less. /sig storage


Question: "type": "boolean" }, "storageCapacity": { <del> "description": "If set to true, storageCapacity indicates that the CSI volume driver wants pod scheduling to consider the storage capacity that the driver deployment will report by creating CSIStorageCapacity objects with capacity information.nnThe check can be enabled immediately when deploying a driver. In that case, provisioning new volumes with late binding will pause until the driver deployment has published some suitable CSIStorageCapacity object.nnAlternatively, the driver can be deployed with the field unset or false and it can be flipped later when storage capacity information has been published.nnThis field is immutable.nnThis is a beta field and only available when the CSIStorageCapacity feature is enabled. The default is false.", </del> <ins> "description": "If set to true, storageCapacity indicates that the CSI volume driver wants pod scheduling to consider the storage capacity that the driver deployment will report by creating CSIStorageCapacity objects with capacity information.nnThe check can be enabled immediately when deploying a driver. In that case, provisioning new volumes with late binding will pause until the driver deployment has published some suitable CSIStorageCapacity object.nnAlternatively, the driver can be deployed with the field unset or false and it can be flipped later when storage capacity information has been published.nnThis field was immutable in Kubernetes <= 1.22 and now is mutable.nnThis is a beta field and only available when the CSIStorageCapacity feature is enabled. The default is false.", </ins> "type": "boolean" }, "tokenRequests": {

Up to 1.21, that field is immutable. That means in practice, a CSIDriver object has to be deleted and then re-created with a different value instead. Mutation via delete/created is unnecessarily complex and as the only consumer of that field (the kube-scheduler) can handle mutation just fine (it'll always ask the informer cache for the current value), it's better to allow mutation.
/sig storage /assign
This is independent of allowing other fields to become mutable, but that might also make sense.
/triage accepted


Question: Metrics:   usage, Available: availableResources[m.Name], }) <ins> delete(availableResources, m.Name) } // print lines for nodes of which the metrics is unreachable. for nodeName := range availableResources { printMissingMetricsNodeLine(w, nodeName) </ins> } return nil }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: kubectl top nodes didn't report one of my nodes. I have a cluster of 8 and only 7 printed.


Question: Metrics:   usage, Available: availableResources[m.Name], }) <ins> delete(availableResources, m.Name) } // print lines for nodes of which the metrics is unreachable. for nodeName := range availableResources { printMissingMetricsNodeLine(w, nodeName) </ins> } return nil }

<pre:::: fang 10:04:02 (cgthayer-pchat) 0 platform; kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-charles-test-cluster-default-pool--jkrj 311m 33% 2470Mi 93% gke-charles-test-cluster-default-pool--krsh 154m 16% 1763Mi 66% gke-charles-test-cluster-default-pool--vhgq 265m 28% 1954Mi 73% gke-charles-test-cluster-default-pool--x3bs 186m 19% 1990Mi 75% gke-charles-test-cluster-default-pool--073f 270m 28% 1729Mi 65% gke-charles-test-cluster-default-pool--09zf 403m 42% 2698Mi 102% gke-charles-test-cluster-default-pool--p7h1 191m 20% 1582Mi 59% :::: fang 10:04:07 (cgthayer-pchat) 0 platform; kubectl get nodes NAME STATUS ROLES AGE VERSION gke-charles-test-cluster-default-pool--073f Ready <none7d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--09zf Ready <none7d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--cl65 Ready <none6d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--jkrj Ready <none8d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--krsh Ready <none8d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--p7h1 Ready <none7d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--vhgq Ready <none8d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--x3bs Ready <none8d v1.7.12-gke.1 <preWhat you expected to happen: I expected all nodes to be printed.


Question: Metrics:   usage, Available: availableResources[m.Name], }) <ins> delete(availableResources, m.Name) } // print lines for nodes of which the metrics is unreachable. for nodeName := range availableResources { printMissingMetricsNodeLine(w, nodeName) </ins> } return nil }

Since the missing node might have bad data I might expect it to have the same number of columns with "-" values and a note in a new 6th column saying something like "unavailabe" "unresponsive" "stale-data" etc. e.g. <pregke-charles-test-cluster-default-pool--p7h1 - - - - unreachable </preHow to reproduce it (as minimally and precisely as possible): Not sure. Perhaps run a pod where you can intentionally trigger it to use too much cpu or memory. Anything else we need to know?: Environment: Kubernetes version (use ): kubectl version Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-12-21T06:34:11Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.8-gke.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-16T18:26:58Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind feature
I raise a PR to print when the metrics is missing. As for the 6th column you mentioned, I think it could be debatable for its necessity. :)


Question: } } <del> if klog.V(4).Enabled() { logPluginScores(nodes, scoresMap, pod) } </del> if len(g.extenders) != 0 && nodes != nil { var mu sync.Mutex var wg sync.WaitGroup

Scheduler throughput drops significantly in http://perf- ! After , seems to be the curlspirit.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent /sig scheduling /sig scalability


Question: // Not local instance, get addresses from Azure ARM API. if !isLocalInstance { <del> return addressGetter(name) </del> <ins> if az.vmSet != nil { return addressGetter(name) } // vmSet == nil indicates credentials are not provided. return nil, fmt.Errorf("no credentials provided for Azure cloud provider") </ins> } if len(metadata.Network.Interface) == 0 {

Currently there is no way to run without any identity (be it service principal or E/MSI). In reality code running on a worker node can use via - and only via - without the need to access ARM. Here is the proposed change a new mode authN = this mode, is basically telling the cloud provider to work any identity, this means only IMDS calls is allowed. This mode can be detected from if no SPN set and both MSI settings are set to . to use a config map within the cluster. This config map can be loaded on the cluster statically. or created - if not existent - in that then only is used in The net new result of the above is users bootstrapping clusters can configure the worker node to use without any identity on said worker nodes.
/sig azure These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by
In-tree AzureDisk volume plugin is also depending on Azure credential to fetch lun id. we should figure out how to solve it without credential
fetch disk lun id is also a dependency in kubelet code now, after my investigation, it could be removed on Linux node, while it cannot be removed on Windows node, need to change the k/k upstream code, there is a discussion on going here: , let' wait for the discussion result.
For Linux: 1- use IMDS my understanding is new api versions will have the disk/lun pairs. 2- don't use arm for the return of is the second parameter passed to . For Windows: I don't understand the dep on ARM for this. The above still apply, we just need to find a way to encode the value as ut return from that allow us to find the disk by on a windows machine. might be a good source for this. Ideally speaking. The value should be the same. But i will leave it for you to decide.
could be a good idea since currently it's a LUN num returned by Attach() and then it could be changed to the windows disk num, I will make the code change on v1.15 later, thanks for discussion.
Update: PR would remove the VM API calls when getting the LUN number for AzureDisk.


Question: } } <ins> func TestPriorityQueue_MoveAllToActiveOrBackoffQueueWithOutQueueingHint(t *testing.T) { c := testingclock.NewFakeClock(time.Now()) logger, ctx := ktesting.NewTestContext(t) ctx, cancel := context.WithCancel(ctx) defer cancel() m := makeEmptyQueueingHintMapPerProfile() defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.SchedulerQueueingHints, false)() m[""][NodeAdd] = []*QueueingHintFunction{ { PluginName:     "fooPlugin", QueueingHintFn: queueHintReturnQueue, }, } q := NewTestQueue(ctx, newDefaultQueueSort(), WithClock(c), WithQueueingHintMapPerProfile(m)) // To simulate the pod is failed in scheduling in the real world, Pop() the pod from activeQ before AddUnschedulableIfNotPresent()s below. if err := q.Add(logger, medPriorityPodInfo.Pod); err != nil { t.Errorf("add failed: %v", err) } err := q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(unschedulablePodInfo.Pod, "fooPlugin"), q.SchedulingCycle()) if err != nil { t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err) } err = q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(highPriorityPodInfo.Pod, "fooPlugin"), q.SchedulingCycle()) if err != nil { t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err) } // Construct a Pod, but don't associate its scheduler failure to any plugin hpp1 := clonePod(highPriorityPodInfo.Pod, "hpp1") // This Pod will go to backoffQ because no failure plugin is associated with it. err = q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(hpp1), q.SchedulingCycle()) if err != nil { t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err) } // Construct another Pod, and associate its scheduler failure to plugin "barPlugin". hpp2 := clonePod(highPriorityPodInfo.Pod, "hpp2") // This Pod will go to the unschedulable Pod pool. err = q.AddUnschedulableIfNotPresent(logger, q.newQueuedPodInfo(hpp2, "barPlugin"), q.SchedulingCycle()) if err != nil { t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err) } // This NodeAdd event moves unschedulablePodInfo and highPriorityPodInfo to the backoffQ, // because of the queueing hint function registered for NodeAdd/fooPlugin. q.MoveAllToActiveOrBackoffQueue(logger, NodeAdd, nil, nil, nil) if q.activeQ.Len() != 1 { t.Errorf("Expected 1 item to be in activeQ, but got: %v", q.activeQ.Len()) } // Pop out the medPriorityPodInfo in activeQ. if p, err := q.Pop(logger); err != nil || p.Pod != medPriorityPodInfo.Pod { t.Errorf("Expected: %v after Pop, but got: %v", medPriorityPodInfo.Pod, p.Pod.Name) } // hpp2 won't be moved. if q.podBackoffQ.Len() != 3 { t.Fatalf("Expected 3 items to be in podBackoffQ, but got: %v", q.podBackoffQ.Len()) } // pop out the pods in the backoffQ. // This doesn't make them in-flight pods. for q.podBackoffQ.Len() != 0 { _, err = q.podBackoffQ.Pop() if err != nil { t.Errorf("pop failed: %v", err) } } q.schedulingCycle++ unschedulableQueuedPodInfo := q.newQueuedPodInfo(unschedulablePodInfo.Pod, "fooPlugin") highPriorityQueuedPodInfo := q.newQueuedPodInfo(highPriorityPodInfo.Pod, "fooPlugin") hpp1QueuedPodInfo := q.newQueuedPodInfo(hpp1) err = q.AddUnschedulableIfNotPresent(logger, unschedulableQueuedPodInfo, q.SchedulingCycle()) if err != nil { t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err) } err = q.AddUnschedulableIfNotPresent(logger, highPriorityQueuedPodInfo, q.SchedulingCycle()) if err != nil { t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err) } err = q.AddUnschedulableIfNotPresent(logger, hpp1QueuedPodInfo, q.SchedulingCycle()) if err != nil { t.Fatalf("unexpected error from AddUnschedulableIfNotPresent: %v", err) } if err = q.Add(logger, medPriorityPodInfo.Pod); err != nil { t.Errorf("add failed: %v", err) } // hpp1 will go to backoffQ because no failure plugin is associated with it. // All plugins other than hpp1 are enqueued to the unschedulable Pod pool. for _, pod := range []*v1.Pod{unschedulablePodInfo.Pod, highPriorityPodInfo.Pod, hpp2} { if q.unschedulablePods.get(pod) == nil { t.Errorf("Expected %v in the unschedulablePods", pod.Name) } } if _, ok, _ := q.podBackoffQ.Get(hpp1QueuedPodInfo); !ok { t.Errorf("Expected %v in the podBackoffQ", hpp1.Name) } // Move clock by podInitialBackoffDuration, so that pods in the unschedulablePods would pass the backing off, // and the pods will be moved into activeQ. c.Step(q.podInitialBackoffDuration) q.flushBackoffQCompleted(logger) // flush the completed backoffQ to move hpp1 to activeQ. q.MoveAllToActiveOrBackoffQueue(logger, NodeAdd, nil, nil, nil) if q.activeQ.Len() != 4 { t.Errorf("Expected 4 items to be in activeQ, but got: %v", q.activeQ.Len()) } if q.podBackoffQ.Len() != 0 { t.Errorf("Expected 0 item to be in podBackoffQ, but got: %v", q.podBackoffQ.Len()) } if len(q.unschedulablePods.podInfoMap) != 1 { // hpp2 won't be moved regardless of its backoff timer. t.Errorf("Expected 1 item to be in unschedulablePods, but got: %v", len(q.unschedulablePods.podInfoMap)) } } </ins> func clonePod(pod *v1.Pod, newName string) *v1.Pod { pod = pod.DeepCopy() pod.Name = newName

ref: /sig scheduling /kind cleanup Part of: The unit test in the scheduling queue only assumes the case where the queueing hint is enabled. We should make another unit test to confirm works without QHint. should have been existing since before we introduced QHint. So, just bring the old one back should be fine.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
/assign


Question: *out = new(PluginSet) (*in).DeepCopyInto(*out) } <del> if in.NormalizeScore != nil { in, out := &in.NormalizeScore, &out.NormalizeScore *out = new(PluginSet) (*in).DeepCopyInto(*out) } </del> if in.Reserve != nil { in, out := &in.Reserve, &out.Reserve *out = new(PluginSet)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Consolidate ) into Score plugin set in config API. Why is this needed: NormalizeScore extension point is implemented in . We decided to treat NormalizeScore as an "optional" step of Score plugin, see . However, we deferred the removal of due to API review delays. Now that PR was merged. We can start the actual removal from the API and start the API review process. /sig scheduling
/assign


Question: return utilfile.FileExists(kubeletpath) } <ins> func (mounter *NsenterMounter) EvalHostSymlinks(pathname string) (string, error) { return mounter.ne.EvalSymlinks(pathname, true) } </ins> func (mounter *NsenterMounter) CleanSubPaths(podDir string, volumeName string) error { return doCleanSubPaths(mounter, podDir, volumeName) }

/kind bug Using , if a block device is symlinked to the discovery directory e.g. , then MapVolume will fail with since the containerized kubelet cannot see the path on host. is mounted by containerized kubelet, so if before we try to map the device we set devicePath to the evaluation of symlink and end up with a devicePath of , containerized kubelet will see the device and successfully map the device. Are there any issues with this solution? This is on Kubernetes 1.10
/sig storage
ah yes containerized kubelet has many issues when dealing with symlinks. Check out Mounter.GetMode(), it calls a nsenter EvalSymlinks which should be able to do what you need. But actually, one thing to consider, is do you need to do the losetup on the host, or does inside the container work?
Yes it appears to work from inside the container, on the host I run which disappears after I delete the pod using the device. My understanding of running losetup in MapVolume is not that great but I suppose that as long as that /dev/loop0 is created we are fine. Thanks for the quick response, I'll create a PR using that function and tag you
Does it create it on the host or in the containerized kubelet? What happens when containerized kubelet is restarted? The purpose of losetup is to do "refcounting" on the block device to prevent possible naming collisions when devices are attached/detached under unhealthy conditions. Ie, if a device showed up as /dev/sdb and you gave that to a pod, and then for some reason it become unhealthy and disappeared (while the pod is still running), another device (maybe even another volume type) could get attached to the node and reuse /dev/sdb. Now the pod has access to a device it's not supposed to. With filesystem volumes, the refcounting is done with mount points, but for block device, loopback is the equivalent.
It is created on the host (and also visible in the container), and stays around through restart. I understand now, thank you.
Actually after I restart it becomes /vdb instead of /dev/vdb in the losetup output, idk what that is about.


Question: return utilfile.FileExists(kubeletpath) } <ins> func (mounter *NsenterMounter) EvalHostSymlinks(pathname string) (string, error) { return mounter.ne.EvalSymlinks(pathname, true) } </ins> func (mounter *NsenterMounter) CleanSubPaths(podDir string, volumeName string) error { return doCleanSubPaths(mounter, podDir, volumeName) }

There is no such thing as /vdb and /dev/vdb is still intact of course. Seems inconsequential since it is disappearing after all pods using the device are deleted, i.e. it's still serving as a refcounter


Question: // Max amount of time to wait for the container runtime to come up. maxWaitForContainerRuntime = 30 * time.Second <ins> // Max amount of time to wait for node list/watch to initially sync maxWaitForAPIServerSync = 10 * time.Second </ins> // nodeStatusUpdateRetry specifies how many times kubelet retries when posting node status failed. nodeStatusUpdateRetry = 5

What happened: A deployment is created that uses a after the cluster master is created but before the cluster worker nodes are fully initialized. During the initialization of the worker nodes, the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector. Depending on how long the nodes are available for scheduling without the necessary node labels for the deployment, the scheduler will start to spam with cluster with pods in status. This spamming stops once the worker nodes are fully initialized and the pods are scheduled successful. The exact timing of all of this is TBD. What you expected to happen: Pods in status are annoying and require manual cleanup. It would be preferred that the scheduler avoid this an simply issue warning events. How to reproduce it (as minimally and precisely as possible): See issue description. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.18.3 Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 4.15.0-96-generic Install tools: Managed service install Network plugin and version (if this is a network-related bug): N/A Others: N/A
Related to and
/sig scheduling
by spamming are you referring to scheduler logs? also, why are the nodes not labelled from the get go?
Just to confirm, all the pods in question are still in Pending state, yes? If the pods are in pending state, all of them will get scheduled eventually once the nodes get their labels.
The pods are not in state. They are actually in state. The pods eventually get scheduled but we are left with anywhere from a few to dozens of these state pods. Also, we are looking at labeling the nodes via kubelet flags to see if that helps.
/remove-kind bug
After we add some retry logic and wait time in this problem can be solved: As you can see this function returns initial node as the first request failed.
I also get hit by this, when single controller node gets rebooted and scheduler starts as static container before kubelet becomes ready.


Question: // Max amount of time to wait for the container runtime to come up. maxWaitForContainerRuntime = 30 * time.Second <ins> // Max amount of time to wait for node list/watch to initially sync maxWaitForAPIServerSync = 10 * time.Second </ins> // nodeStatusUpdateRetry specifies how many times kubelet retries when posting node status failed. nodeStatusUpdateRetry = 5

Can be cleaned up with:
/sig node
the spam cycle is this loop: "pod create, pod schedule, worker reject, pod failed" -- we don't have a lot of protection in higher level workload controllers to handle pods that fail at kubelet admission time versus pods that just dont create due to quota/admission. in general, is the spam causing any other stability issues? is pod gc pruning the pods?
I wonder if we can do a trick in node lister to mitigate any race condition here...
we could maybe do something similar to to ensure node lister has synced...
hacking on option to ensure if kubelet has a node lister with a valid kube client, we wait for it to sync at least once here which should mitigate this issue. see:
We noticed the problem on the OLM pod which is managed by the OLM operator. Right or wrong, the operator does not like seeing the pod go to status and no longer reconciles the pod. The end result is that manual intervention (i.e. pod deletion) is required to fix the problem.
sure... i would have thought the pod is failed in that instance and reached a terminal state. either way, i am looking at what i can do to mitigate this.
The problem persists in our environment as well, and the workaround cited above will not do anymore. Is there something in the works for this one?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: return true, nil } <del> // nextRotationDeadline returns a value for the threshold at which the // current certificate should be rotated, 80%+/-10% of the expiration of the // certificate. func (m *manager) nextRotationDeadline() time.Time { // forceRotation is not protected by locks if m.forceRotation { m.forceRotation = false return time.Now() } m.certAccessLock.RLock() defer m.certAccessLock.RUnlock() </del> <ins> // Check that the current certificate on disk satisfies the requests from the // current template. // // Note that extra items in the certificate's SAN or orgs that don't exist in // the template will not trigger a renewal. // // Requires certAccessLock to be locked. func (m *manager) certSatisfiesTemplateLocked() bool { </ins> if m.cert == nil { <del> return time.Now() </del> <ins> return false </ins> } <del> // Ensure the currently held certificate satisfies the requested subject CN and SANs </del> if template := m.getTemplate(); template != nil { if template.Subject.CommonName != m.cert.Leaf.Subject.CommonName { <del> glog.V(2).Infof("Current certificate CN (%s) does not match requested CN (%s), rotating now", m.cert.Leaf.Subject.CommonName, template.Subject.CommonName) return time.Now() </del> <ins> glog.V(2).Infof("Current certificate CN (%s) does not match requested CN (%s)", m.cert.Leaf.Subject.CommonName, template.Subject.CommonName) return false </ins> } currentDNSNames := sets.NewString(m.cert.Leaf.DNSNames...) desiredDNSNames := sets.NewString(template.DNSNames...) missingDNSNames := desiredDNSNames.Difference(currentDNSNames) if len(missingDNSNames) > 0 { <del> glog.V(2).Infof("Current certificate is missing requested DNS names %v, rotating now", missingDNSNames.List()) return time.Now() </del> <ins> glog.V(2).Infof("Current certificate is missing requested DNS names %v", missingDNSNames.List()) return false </ins> } currentIPs := sets.NewString()

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I'm testing the new option in kubernetes 1.12 and it seems that kubelet always requests the signing of a new CSR and tries to rotate the certificates when it is started. This is happening even though there's a valid (brand new) certificate on disk issued by kube-controller-manager. From what I can tell, this is happening because kubelet implements and is set after a successful certificate rotation, but is never persisted to disk. If kubelet is restarted, then the last requested CSR is nil. What you expected to happen: Ideally, kubelet should only request the signing of a new CSR if it doesn't already have a serving certificate or it's about to expire. How to reproduce it (as minimally and precisely as possible): Restart a kubelet instance that had it's serving certificate issued by kube-controller-manager Check and see that it requested the signing of a new CSR Anything else we need to know?: kubelet is being run with . , , and are all omitted to use the default values. The TLS bootstrapping for client certificates is currently not being used--just the bootstrapping for serving certificates. Environment: Kubernetes version: v1.12.0 - built from source at the tag () Cloud provider or hardware configuration: Bare metal OS: CentOS 7.5.1804 Kernel: Install tools: Custom RPMs built from this git repo with no additional patches Others:
/sig node
/sig auth
Not sure if this is the ideal fix, but I've patched it so that it compares the current CSR template to the last requested template only if it doesn't already match the current cert.
cc
thanks for the proposed fix. I think it looks good. Could you send a PR? We should also remove entirely and only look at current cert and template. does that sound reasonable?
Sure thing, I'd be happy to do that. EDIT: I will submit a PR after approval from my employer.
I think that's probably ok, as long as it checks everything we include in the template and we consider a certificate missing anything we request to be cause for re-requesting another cert (an issuer couldn't choose to issue a cert that drops some SANs and us not request another cert) left some comments in
This regression is a frequent flyer.


Question: return true, nil } <del> // nextRotationDeadline returns a value for the threshold at which the // current certificate should be rotated, 80%+/-10% of the expiration of the // certificate. func (m *manager) nextRotationDeadline() time.Time { // forceRotation is not protected by locks if m.forceRotation { m.forceRotation = false return time.Now() } m.certAccessLock.RLock() defer m.certAccessLock.RUnlock() </del> <ins> // Check that the current certificate on disk satisfies the requests from the // current template. // // Note that extra items in the certificate's SAN or orgs that don't exist in // the template will not trigger a renewal. // // Requires certAccessLock to be locked. func (m *manager) certSatisfiesTemplateLocked() bool { </ins> if m.cert == nil { <del> return time.Now() </del> <ins> return false </ins> } <del> // Ensure the currently held certificate satisfies the requested subject CN and SANs </del> if template := m.getTemplate(); template != nil { if template.Subject.CommonName != m.cert.Leaf.Subject.CommonName { <del> glog.V(2).Infof("Current certificate CN (%s) does not match requested CN (%s), rotating now", m.cert.Leaf.Subject.CommonName, template.Subject.CommonName) return time.Now() </del> <ins> glog.V(2).Infof("Current certificate CN (%s) does not match requested CN (%s)", m.cert.Leaf.Subject.CommonName, template.Subject.CommonName) return false </ins> } currentDNSNames := sets.NewString(m.cert.Leaf.DNSNames...) desiredDNSNames := sets.NewString(template.DNSNames...) missingDNSNames := desiredDNSNames.Difference(currentDNSNames) if len(missingDNSNames) > 0 { <del> glog.V(2).Infof("Current certificate is missing requested DNS names %v, rotating now", missingDNSNames.List()) return time.Now() </del> <ins> glog.V(2).Infof("Current certificate is missing requested DNS names %v", missingDNSNames.List()) return false </ins> } currentIPs := sets.NewString()

I'd love to see an integration test that would catch this (but that shouldn't block a fix).
Thanks for the comments! Now that I have the CLA signed, I'll address those and submit a PR.
Just submitted the PR:


Question: Expect(events.Items[1].Message).Should(Equal(fmt.Sprintf("Scaled down rc %s to 0", rcName))) } <ins> func testRecreateDeployment(f *Framework) { ns := f.Namespace.Name c := f.Client // Create nginx pods. deploymentPodLabels := map[string]string{"name": "sample-pod-3"} rcPodLabels := map[string]string{ "name": "sample-pod-3", "pod":  "nginx", } rcName := "nginx-controller" replicas := 3 _, err := c.ReplicationControllers(ns).Create(newRC(rcName, replicas, rcPodLabels, "nginx", "nginx")) Expect(err).NotTo(HaveOccurred()) defer func() { Logf("deleting replication controller %s", rcName) Expect(c.ReplicationControllers(ns).Delete(rcName)).NotTo(HaveOccurred()) }() // Verify that the required pods have come up. err = verifyPods(c, ns, "sample-pod-3", false, 3) if err != nil { Logf("error in waiting for pods to come up: %s", err) Expect(err).NotTo(HaveOccurred()) } // Create a deployment to delete nginx pods and instead bring up redis pods. deploymentName := "redis-deployment-3" Logf("Creating deployment %s", deploymentName) _, err = c.Deployments(ns).Create(newDeployment(deploymentName, replicas, deploymentPodLabels, "redis", "redis", extensions.RecreateDeploymentStrategyType)) Expect(err).NotTo(HaveOccurred()) defer func() { deployment, err := c.Deployments(ns).Get(deploymentName) Expect(err).NotTo(HaveOccurred()) Logf("deleting deployment %s", deploymentName) Expect(c.Deployments(ns).Delete(deploymentName, nil)).NotTo(HaveOccurred()) // TODO: remove this once we can delete rcs with deployment newRC, err := deploymentutil.GetNewRC(*deployment, c) Expect(err).NotTo(HaveOccurred()) Expect(c.ReplicationControllers(ns).Delete(newRC.Name)).NotTo(HaveOccurred()) }() err = waitForDeploymentStatus(c, ns, deploymentName, replicas, 0, replicas, 0) Expect(err).NotTo(HaveOccurred()) // Verify that the pods were scaled up and down as expected. We use events to verify that. deployment, err := c.Deployments(ns).Get(deploymentName) Expect(err).NotTo(HaveOccurred()) waitForEvents(c, ns, deployment, 2) events, err := c.Events(ns).Search(deployment) if err != nil { Logf("error in listing events: %s", err) Expect(err).NotTo(HaveOccurred()) } // There should be 2 events, one to scale up the new RC and then to scale down the old RC. Expect(len(events.Items)).Should(Equal(2)) newRC, err := deploymentutil.GetNewRC(*deployment, c) Expect(err).NotTo(HaveOccurred()) Expect(newRC).NotTo(Equal(nil)) Expect(events.Items[0].Message).Should(Equal(fmt.Sprintf("Scaled down rc %s to 0", rcName))) Expect(events.Items[1].Message).Should(Equal(fmt.Sprintf("Scaled up rc %s to 3", newRC.Name))) } </ins> // testRolloverDeployment tests that deployment supports rollover. // i.e. we can change desired state and kick off rolling update, then change desired state again before it finishes. func testRolloverDeployment(f *Framework) {

There's some practical limit on the number of concurrent access tokens that can be used in a Kubernetes cluster before it begins to spend all of its time in the authn stack verifying them. Absent fixes to it was < 500; with the fixes linked there it's still not infinitely scalable-- I still hit (what I believe to be) with ~ 5k unique tokens @ ~700QPS. This request is to this dimension an official name & definition, and list it with others, e.g. here: it it have a useful value (I think it should be 10k) /kind bug /sig scalability
/cc
oh and 4. add tests which verify it is as high as believed.
cc since projected tokens will increase the number of unique tokens
/sig auth
LRUs are not good at scaling and sharding has some major pitfalls as pointed out. I would suggest that we try to drop the LRU cache for successful authentication results in the token cache. The main benefit of using an LRU here is bound cache size. Entries in the token cache have a fast expiration independent of the LRU policy. The records probably take up O(2kb) (guesstimate) of memory. Given that these are successful attempts with presumably valid tokens, I don't think we are getting any benefit from the LRU. If we don't have a constraint on fixed cache size, we can back the token cache with a sync.Map and run a single background goroutine to periodically walk the map and remove expired entries. This would fix the problematic locking and eviction due to nonuniform distribution of tokens over shards. The failure cache is less important and we don't cache failures by default. If we consider this the "slowpath", we can probably leave it as. If that's not good enough, we have a couple options (with various tradeoffs) to improve it.
If off by default is not enough, more reasoning why the failed cache is not super useful: it might help with an accidental DoS but it doesn't prevent anything actively malicious DoS. I think we can focus our improvements on the success cache.
/assign He is already working on adding an explicit test for it.
has taken it over and is actively working on this. Unfortunately, I cannot assign this issue to him as he's not a k8s member yet
It's not closed - all testes must be submitted and appropriate documentation must be
Thanks everyone!!


Question: Create-Directories Download-HelperScripts <del> Install-LoggingAgent Configure-LoggingAgent Restart-LoggingAgent </del> <ins> # Even if Stackdriver is already installed, the function will still [re]start the service. if (IsLoggingEnabled $kube_env) { Install-LoggingAgent Configure-LoggingAgent Restart-LoggingAgent } </ins> Create-DockerRegistryKey Configure-Dockerd

What happened: The stackdriver logging agent is installed and start running no matter logging is enabled/disabled in the cluster or not. What you expected to happen: Do not install and start the logging agent if logging is disabled.
/sig windows
cc


Question: BeforeEach(func() { podClient = f.PodClient() }) <del> // TODO: Fix Flaky issue #68066 and then re-add this back into Conformance Suite </del> /* <del> Release : v1.9 </del> <ins> Release : v1.15 </ins> Testname: Pods, delete grace period <del> Description: Create a pod, make sure it is running, create a watch to observe Pod creation. Create a 'kubectl local proxy', capture the port the proxy is listening. Using the http client send a ‘delete’ with gracePeriodSeconds=30. Pod SHOULD get deleted within 30 seconds. </del> <ins> Description: Create a pod, make sure it is running. Create a 'kubectl local proxy', capture the port the proxy is listening. Using the http client send a ‘delete’ with gracePeriodSeconds=30. Pod SHOULD get deleted within 30 seconds. </ins> */ <del> It("should be submitted and removed  [Flaky]", func() { </del> <ins> framework.ConformanceIt("should be submitted and removed", func() { </ins> By("creating the pod") name := "pod-submit-remove-" + string(uuid.NewUUID()) value := strconv.Itoa(time.Now().Nanosecond())

We need to fix this potentially flaky test and add it back to conformance suite. Please see discussion in Also waiting on:
cc /sig testing
/area conformance
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
can we convert it to conformance now??
Once 1.15 opens up


Question: // be made to pass by removing pods, or you change an existing predicate so that // it can never be made to pass by removing pods, you need to add the predicate // failure error in nodesWherePreemptionMightHelp() in scheduler/core/generic_scheduler.go <del> ErrDiskConflict                          = newPredicateFailureError("NoDiskConflict") ErrVolumeZoneConflict                    = newPredicateFailureError("NoVolumeZoneConflict") ErrNodeSelectorNotMatch                  = newPredicateFailureError("MatchNodeSelector") ErrPodAffinityNotMatch                   = newPredicateFailureError("MatchInterPodAffinity") ErrPodAffinityRulesNotMatch              = newPredicateFailureError("PodAffinityRulesNotMatch") ErrPodAntiAffinityRulesNotMatch          = newPredicateFailureError("PodAntiAffinityRulesNotMatch") ErrExistingPodsAntiAffinityRulesNotMatch = newPredicateFailureError("ExistingPodsAntiAffinityRulesNotMatch") ErrTaintsTolerationsNotMatch             = newPredicateFailureError("PodToleratesNodeTaints") ErrPodNotMatchHostName                   = newPredicateFailureError("HostName") ErrPodNotFitsHostPorts                   = newPredicateFailureError("PodFitsHostPorts") ErrNodeLabelPresenceViolated             = newPredicateFailureError("CheckNodeLabelPresence") ErrServiceAffinityViolated               = newPredicateFailureError("CheckServiceAffinity") ErrMaxVolumeCountExceeded                = newPredicateFailureError("MaxVolumeCount") ErrNodeUnderMemoryPressure               = newPredicateFailureError("NodeUnderMemoryPressure") ErrNodeUnderDiskPressure                 = newPredicateFailureError("NodeUnderDiskPressure") ErrNodeOutOfDisk                         = newPredicateFailureError("NodeOutOfDisk") ErrNodeNotReady                          = newPredicateFailureError("NodeNotReady") ErrNodeNetworkUnavailable                = newPredicateFailureError("NodeNetworkUnavailable") ErrNodeUnschedulable                     = newPredicateFailureError("NodeUnschedulable") ErrNodeUnknownCondition                  = newPredicateFailureError("NodeUnknownCondition") ErrVolumeNodeConflict                    = newPredicateFailureError("VolumeNodeAffinityConflict") ErrVolumeBindConflict                    = newPredicateFailureError("VolumeBindingNoMatch") </del> <ins> ErrDiskConflict                          = newPredicateFailureError("NoDiskConflict", "node(s) had no available disk") ErrVolumeZoneConflict                    = newPredicateFailureError("NoVolumeZoneConflict", "node(s) had no available volume zone") ErrNodeSelectorNotMatch                  = newPredicateFailureError("MatchNodeSelector", "node(s) didn't match node selector") ErrPodAffinityNotMatch                   = newPredicateFailureError("MatchInterPodAffinity", "node(s) didn't match pod affinity/anti-affinity") ErrPodAffinityRulesNotMatch              = newPredicateFailureError("PodAffinityRulesNotMatch", "node(s) didn't match pod affinity rules") ErrPodAntiAffinityRulesNotMatch          = newPredicateFailureError("PodAntiAffinityRulesNotMatch", "node(s) didn't match pod anti-affinity rules") ErrExistingPodsAntiAffinityRulesNotMatch = newPredicateFailureError("ExistingPodsAntiAffinityRulesNotMatch", "node(s) didn't satisfy existing pods anti-affinity rules") ErrTaintsTolerationsNotMatch             = newPredicateFailureError("PodToleratesNodeTaints", "node(s) had taints that the pod didn't tolerate") ErrPodNotMatchHostName                   = newPredicateFailureError("HostName", "node(s) didn't match the requested hostname") ErrPodNotFitsHostPorts                   = newPredicateFailureError("PodFitsHostPorts", "node(s) didn't have free ports for the requested pod ports") ErrNodeLabelPresenceViolated             = newPredicateFailureError("CheckNodeLabelPresence", "node(s) didn't have the requested labels") ErrServiceAffinityViolated               = newPredicateFailureError("CheckServiceAffinity", "node(s) didn't match service affinity") ErrMaxVolumeCountExceeded                = newPredicateFailureError("MaxVolumeCount", "node(s) exceed max volume count") ErrNodeUnderMemoryPressure               = newPredicateFailureError("NodeUnderMemoryPressure", "node(s) had memory pressure") ErrNodeUnderDiskPressure                 = newPredicateFailureError("NodeUnderDiskPressure", "node(s) had disk pressure") ErrNodeOutOfDisk                         = newPredicateFailureError("NodeOutOfDisk", "node(s) were out of disk space") ErrNodeNotReady                          = newPredicateFailureError("NodeNotReady", "node(s) were not ready") ErrNodeNetworkUnavailable                = newPredicateFailureError("NodeNetworkUnavailable", "node(s) had unavailable network") ErrNodeUnschedulable                     = newPredicateFailureError("NodeUnschedulable", "node(s) were unschedulable") ErrNodeUnknownCondition                  = newPredicateFailureError("NodeUnknownCondition", "node(s) had unknown conditions") ErrVolumeNodeConflict                    = newPredicateFailureError("VolumeNodeAffinityConflict", "node(s) had volume node affinity conflict") ErrVolumeBindConflict                    = newPredicateFailureError("VolumeBindingNoMatch", "node(s) didn't find available persistent volumes to bind") </ins> // ErrFakePredicate is used for test only. The fake predicates returning false also returns error // as ErrFakePredicate. <del> ErrFakePredicate = newPredicateFailureError("FakePredicateError") </del> <ins> ErrFakePredicate = newPredicateFailureError("FakePredicateError", "Nodes failed the fake predicate") </ins> ) // InsufficientResourceError is an error type that indicates what kind of resource limit is

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature The is not user friendly. There has been to make it better, but we still print in the pending message. We should change these errors to a more human readable message. For example, should be changed to something like . /sig scheduling


Question: // ValidateCSINodeDrivers tests that the specified CSINodeDrivers have valid data. func validateCSINodeDrivers(drivers []storage.CSINodeDriver, fldPath *field.Path, validationOpts CSINodeValidationOptions) field.ErrorList { allErrs := field.ErrorList{} <del> driverNamesInSpecs := make(sets.String) </del> <ins> driverNamesInSpecs := sets.New[string]() </ins> for i, driver := range drivers { idxPath := fldPath.Index(i) allErrs = append(allErrs, validateCSINodeDriver(driver, driverNamesInSpecs, idxPath, validationOpts)...)

It seems to be valid to add here. Would you like to open a pull request for it? this is like a cleanup, as the set insert will skip and the validation will do twice. is used to track bugs. I suggest you open a PR for that if you like.
/sig storage
I think adding ` == true and we didn't skip, it will insert set twice I believe? wasn't notice that at first glance. However, I can file a PR now
/triage accepted
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
Please feel free to submit a PR.


Question: } func (a genericAccessor) SetAnnotations(annotations map[string]string) { <ins> if a.annotations == nil { emptyAnnotations := make(map[string]string) a.annotations = &emptyAnnotations } </ins> *a.annotations = annotations }

When trying to edit multiple resources, for example: will panic. Ref as the possible cause cc
if was the cause, any fix needs to happen in 1.1 as well, was already cherry picked there
Also shows that we need a test for this case.
is the cause. When we edit multiple resources (i.e., list-type objects), the list's Accessor.annotations (type ) is nil and will panic when UpdateApplyAnnotation tries to SetAnnotations ().


Question: maxBackOffTolerance  = time.Duration(1.3 * float64(kubelet.MaxContainerBackOff)) podRetryPeriod       = 1 * time.Second podRetryTimeout      = 1 * time.Minute <ins> podReadyTimeout      = 1 * time.Minute </ins> ) // testHostIP tests that a pod gets a host IP

[x] Original approval issue: - Reverted due to flakes [x] Two weeks soak end date: 20 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Pod endpoints which are untested. a Pod with a static label the Pod with a new Label and updated data the Pod to ensure it's patched the Pod's status the Pod's status Ready condition to False the Pod's status to check if Ready is False all Pods in all Namespaces find the Pod(1) ensure that the Pod is found and is patched Namespaced Pod(1) via a Collection with a LabelSelector the Pod to check that it's deleted Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/reopen Test PR Merge Promotion PR to be created next
Reopened this issue. <detailsIn response to : /reopen Test PR Merge Promotion PR to be created next Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
reverted by due to flaky test
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/reopen PR on test grid for 2 weeks before promotion to conformance
Reopened this issue. <detailsIn response to : /reopen PR on test grid for 2 weeks before promotion to conformance Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: scheme := runtime.NewScheme() scheme.AddKnownTypes(cmResource.GroupVersion(), &v1.ConfigMap{}) codecs := serializer.NewCodecFactory(scheme) <del> o := NewFieldManagedObjectTracker(scheme, codecs.UniversalDecoder(), fakeTypeConverter) </del> <ins> o := NewFieldManagedObjectTracker(scheme, codecs.UniversalDecoder(), configMapTypeConverter(scheme)) </ins> reaction := ObjectReaction(o) patch := []byte(`{"apiVersion": "v1", "kind": "ConfigMap", "metadata": {"name": "cm-1"}, "data": {"k": "v"}}`)

Smoke tests run by publishing-bot on client-go master are failing Ref for failure logs: Tests defined for client-go, after publishing, Since commit No response As part of , when module is used from within the staging directory, can be used by traversing to the parent directory. But once the module is made as a separate repository as part of publishing, then traversing to repository using path will not work. No response /sig-api-machinery
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looking
I'll remove the swagger file dependency, I don't think we strictly need it.
/triage accepted


Question: newDevicePath := "" <del> err = wait.Poll(1*time.Second, timeout, func() (bool, error) { </del> <ins> err = wait.PollImmediate(1*time.Second, timeout, func() (bool, error) { </ins> if newDevicePath, err = findDiskByLun(int(lun), io, exec); err != nil { return false, fmt.Errorf("azureDisk - WaitForAttach ticker failed node (%s) disk (%s) lun(%v) err(%s)", nodeName, diskName, lun, err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When starting pods with AFS volumes, there is an initial delay of five seconds until the pod is transitioning from the "Scheduled" state. The reason for this is that currently the volume mounting happens inside a wait.Poll which will initially wait a specified interval (currently 5 seconds) before execution. Event log for comparison with and without AFS volume attached (seconds since event): What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign /priority important-soon /sig cloud-provider /area provider/azure
see


Question: } } <del> // OpenAPIv3 type/maxLength/maxItems/MaxProperties/required/wrong type field validation failures are viewed as blocking err for CEL validation </del> <ins> // OpenAPIv3 type/maxLength/maxItems/MaxProperties/required/enum violation/wrong type field validation failures are viewed as blocking err for CEL validation </ins> func hasBlockingErr(errs field.ErrorList) (bool, *field.Error) { for _, err := range errs { <del> if err.Type == field.ErrorTypeRequired || err.Type == field.ErrorTypeTooLong || err.Type == field.ErrorTypeTooMany || err.Type == field.ErrorTypeTypeInvalid { </del> <ins> if err.Type == field.ErrorTypeNotSupported || err.Type == field.ErrorTypeRequired || err.Type == field.ErrorTypeTooLong || err.Type == field.ErrorTypeTooMany || err.Type == field.ErrorTypeTypeInvalid { </ins> return true, field.Invalid(nil, nil, "some validation rules were not checked because the object was invalid; correct the existing errors to complete validation") } }

pointed out to me that a CEL x-kubernetes-validations rule in a CRD was hitting the estimated cost limit when performing a simple "string has prefix" check on an en enum field. Putting a limit in the OpenAPI of the enum string field fixed the problem, but having to add a limit like this is counter-intuitive and unnecessary, since the allowed enum values already make the max allowed length obvious. The apiserver automatically computes the max length of an enum as the length of the longest allowed enum value. I believe we already refuse evaluate CEL rules on values that fail OpenAPI value validation, since we rely in the value validations to establish all length limits. I haven't tested it locally, but something like this should work (given enough nested arrays): As a workaround, one can manually set on the string enum, e.g.: 25+ All No response No response No response No response
/sig api-machinery /triage accepted /assign
/assign
I wasn't sure if we special-cased the maxLength / maxItems / maxProperties failures, or if all openapi validation had to pass before we started messing with CEL evaluation
it looks like we currently special-case specific openapi errors that short-circuit CEL validation, I'm not sure an enum validation failure would currently prevent evaluation:
The enum err is converted to () error which is not short-circuit here. But for defaulting case, all error would be short-circuit. I would suggest to add error into the blockingErr list to short-circuit CEL validation. Would this make sense?
yeah, that sounds good if we're going to rely on it to catch things that are longer than the maximum length enum value


Question: // Removes SCSI controller which is latest attached to VM. func cleanUpController(newSCSIController types.BaseVirtualDevice, vmDevices object.VirtualDeviceList, vm *object.VirtualMachine, ctx context.Context) error { <ins> if newSCSIController == nil || vmDevices == nil || vm == nil { return nil } </ins> ctls := vmDevices.SelectByType(newSCSIController) if len(ctls) < 1 { return ErrNoDevicesFound

Kubernetes 1.3.0. During attach disk to VM, call to cleanUpController dereferenced a null pointer. Stack trace:
The seems points to a comment instead of a cleanUpCotroller function in v1.3.0. There might be some modifications later which make the line number inaccurate. Would you please double check which line it really should point to? Thanks.
that is correct, this was a build from v1.3.0 with changes related to this: therefore the line numbers do no line up with v1.3.0. In either case, the panic happened inside
Hi Davide, I saw your PR for this issue and I have another question. Why cleanUpController could be called with an empty newSCSIController? According to the context, an empty newSCSIController should not even be returned by CreateSCSIController rt? Thanks.
it doesn't look like it should happen, but it did and there wasn't any checks for nil pointers in our code or the Govmomi library.


Question: // rest.Config generated by the resolver. type AuthenticationInfoResolverWrapper func(AuthenticationInfoResolver) AuthenticationInfoResolver <del> // AuthenticationInfoResolver builds rest.Config base on the server name. </del> <ins> // AuthenticationInfoResolver builds rest.Config base on the server or service // name and service namespace. </ins> type AuthenticationInfoResolver interface { <ins> // ClientConfigFor builds rest.Config based on the server. </ins> ClientConfigFor(server string) (*rest.Config, error) <ins> // ClientConfigForService builds rest.Config based on the serviceName and // serviceNamespace. ClientConfigForService(serviceName, serviceNamespace string) (*rest.Config, error) </ins> } <del> // AuthenticationInfoResolverFunc implements AuthenticationInfoResolver. type AuthenticationInfoResolverFunc func(server string) (*rest.Config, error) </del> <ins> // AuthenticationInfoResolverDelegator implements AuthenticationInfoResolver. type AuthenticationInfoResolverDelegator struct { ClientConfigForFunc        func(server string) (*rest.Config, error) ClientConfigForServiceFunc func(serviceName, serviceNamespace string) (*rest.Config, error) } func (a *AuthenticationInfoResolverDelegator) ClientConfigFor(server string) (*rest.Config, error) { return a.ClientConfigForFunc(server) } </ins> <del> //ClientConfigFor implements AuthenticationInfoResolver. func (a AuthenticationInfoResolverFunc) ClientConfigFor(server string) (*rest.Config, error) { return a(server) </del> <ins> func (a *AuthenticationInfoResolverDelegator) ClientConfigForService(serviceName, serviceNamespace string) (*rest.Config, error) { return a.ClientConfigForServiceFunc(serviceName, serviceNamespace) </ins> } type defaultAuthenticationInfoResolver struct {

We are getting the following error when the apiserver connects the webhook on localhost (configured via URL). We should only use the SSL tunnel for the connections to nodes when the webhooks are running as services. /sig api-machinery /kind bug /assign
/close


Question: v1 "k8s.io/api/core/v1" policy "k8s.io/api/policy/v1beta1" <del> apierrors "k8s.io/apimachinery/pkg/api/errors" </del> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Conformance - GCE - master (ci-kubernetes-gce-conformance-latest) Which test(s) are failing: [sig-scheduling] SchedulerPreemption [Serial] PreemptionExecutionPath runs ReplicaSets to verify preemption running path [Conformance]Changes Since when has it been failing: 2020-06-08 22:27 PDT Testgrid link: Reason for failure: Regression by Anything else we need to know: /sig scheduling /priority critical-urgent
cc/ /assign


Question: "type": "string" }, "ports": { <del> "description": "List of ports to expose from the container. Exposing a port here gives the system additional information about the network connections a container uses, but is primarily informational. Not specifying a port here DOES NOT prevent that port from being exposed. Any port which is listening on the default "0.0.0.0" address inside a container will be accessible from the network. Cannot be updated.", </del> <ins> "description": "List of ports to expose from the container. Not specifying a port here DOES NOT prevent that port from being exposed. Any port which is listening on the default "0.0.0.0" address inside a container will be accessible from the network. Modifying this array with strategic merge patch may corrupt the data. For more information See https://github.com/kubernetes/kubernetes/issues/108255. Cannot be updated.", </ins> "items": { "$ref": "#/definitions/io.k8s.api.core.v1.ContainerPort" },

The comments in says that : However there are several cases where this is false : When using a Service with a targetPort that references a name in the ports arrays of the pod. Some software (ab)uses this API to get which ports are used (See Although unfixable, my goal is to fix the documentation so other people don't have the same problem). The Azure Load Balancer implementation does this for instance. Some other network/security software might use this information. Also, is it possible to document/reference the above issue in the comment ? Since it is described as informational I expected for the network to work without this information. But it's definitely not the case. Not needed since it's mostly a doc issue. I'm interested in contributing this patch but would like some feedbacks before doing it. :) <details</details<details</details<details</details<details</details<details</details<details</details
/sig network
/triage accepted -- thoughts? I believe the doc are due for a refresh.
I'd take this one. /assign
I didn't see that you wanted to do this patch, I am very sorry and now I unassign it. /unassign
No worries But I'm still not sure where / what to add. Should I only fix the part saying it is only informational ? Is there anything else ? (cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: AUTOSCALER_MIN_NODES="${KUBE_AUTOSCALER_MIN_NODES:-1}" AUTOSCALER_MAX_NODES="${KUBE_AUTOSCALER_MAX_NODES:-${NUM_MINIONS}}" TARGET_NODE_UTILIZATION="${KUBE_TARGET_NODE_UTILIZATION:-0.7}" <ins> ENABLE_CLUSTER_MONITORING=googleinfluxdb </ins> fi # Optional: Enable deployment experimental feature, not ready for production use.

Heapster pod: And logs: cc p0 for triage
Did the required node scope change?
Its either the API on the console or monitoring write scopes on the node. , Mike Danese wrote:
I think this affects only GCE
I will fix this issue since is on vacation.
Need to cherry pick so i'll leave open
is out for review.
has been closed and the cherrypick here () has been merged so I'm closing this issue. Thanks everyone for the debugging.


Question: ETCD_HOST=${ETCD_HOST:-127.0.0.1} ETCD_PORT=${ETCD_PORT:-4001} <ins> ETCD_PREFIX=${ETCD_PREFIX:-randomPrefix} </ins> API_PORT=${API_PORT:-8080} API_HOST=${API_HOST:-127.0.0.1} KUBE_API_VERSIONS=""

removed this test. We need to continue testing this. This wasn't about testing upgrades, the prefix has nothing to do with upgrades. It's just there so people can share etcd. We maybe don't need to double everything but we do need to make sure that keeps working.
To be honest - I think that what we was testing before didn't give us completely any information. We didn't testing what information we are storing in etcd and where. That said, if we had a hardcoded prefix somewhere in our code, we wouldn't discover it at all, and those tests would still be passing. We should have a test that is directly checking etcd contents. I think it should be possible to write a simple unit test for it. Will try to put something together next week.


Question: } // performPlatformSpecificContainerCleanup is responsible for doing any platform-specific cleanup <del> // after either: //   * the container creation has failed //   * the container has been successfully started //   * the container has been removed // whichever happens first. // Any errors it returns are simply logged, but do not prevent the container from being started or // removed. </del> <ins> // after either the container creation has failed or the container has been removed. </ins> func (ds *dockerService) performPlatformSpecificContainerCleanup(cleanupInfo *containerCleanupInfo) (errors []error) { if err := removeGMSARegistryValue(cleanupInfo); err != nil { errors = append(errors, err)

What would you like to be : Periodically remove cleanupInfos for old containers from dockershim's map. One way of doing that would be to just forget about s older than say an hour, as they should always be short-lived. Why is this needed: introduced a new in-memory map for the dockershim, that could potentially (in pathological cases) cause memory leaks - for containers that use GMSA cred specs, get created successfully, but then never get started nor removed. This might be okay for the alpha release of the GMSA feature, but should be addressed in the beta. See
/assign wk8
/sig windows
/milestone v1.15
got moved out. let's move this issue with it as well /milestone v1.16
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
: yup we'd love to have this in 1.16, still waiting for kind reviewers on :) ...


Question: DefaultDummyDevice = "kube-ipvs0" connReuseMinSupportedKernelVersion = "4.1" <ins> // https://github.com/torvalds/linux/commit/35dfb013149f74c2be1ff9c78f14e6a3cd1539d1 connReuseFixedKernelVersion = "5.9" </ins> ) // iptablesJumpChain is tables of iptables chains that ipvs proxier used to install iptables or cleanup iptables.

There were two patches merged recently in the kernel that will significantly improve the IPVS proxier: The first patch fixes an issue where a packet is dropped whenever a stale conntrack entry is reused. This caused a 1s delay whenever the stale conntrack entry was used. This was the original reason for setting . More details in The second patch fixes an issue when a source port is reused from a given client and the corresponding destination was deleted. This is easily reproducible when a single client sends a lot traffic to a small set of destinations and one of the destination servers are removed. More details in Based on these two patches, we should start setting (default) starting on Linux v5.9 when those two patches make it to the main line. The new sysctls for the IPVS proxier would be: We should probably wait for v5.9 before making any changes to kube-proxy, this is just a tracking issue so we don't forget.
/sig network /area ipvs
Should probably hold on the PR until v5.9 is released later this year. cc
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Yes it makes sense to wait until a kernel with the patch exists ;) I think we can also avoid touching it and follow the approach "only touch the sysctl we absolutely need, otherwise trust the admin to know what they need"
I install linux-5.9-rc2 in centos7. set /proc/sys/net/ipv4/vs/connreusemode = 1 i have three service: service-a ---service-b --service-c in k8s. service-c return "helloworld" I use ab to benchmark service-a. [root ~]# ab -n -c 1000 http://172.17.2.66:/ This is ApacheBench, Version 2.3 <Copyright 1996 Adam Twiss, Zeus Technology Ltd, Licensed to The Apache Software Foundation, Benchmarking 172.17.2.66 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed requests Finished requests Server Software: Server Hostname: 172.


Question: DefaultDummyDevice = "kube-ipvs0" connReuseMinSupportedKernelVersion = "4.1" <ins> // https://github.com/torvalds/linux/commit/35dfb013149f74c2be1ff9c78f14e6a3cd1539d1 connReuseFixedKernelVersion = "5.9" </ins> ) // iptablesJumpChain is tables of iptables chains that ipvs proxier used to install iptables or cleanup iptables.

17.2.66 Server Port: Document Path: / Document Length: 11 bytes Concurrency Level: 1000 Time taken for tests: 4.128 seconds Complete requests: Failed requests: 0 Write errors: 0 Total transferred: bytes HTML transferred: bytes Requests per second: 2422.76 [#/sec] (mean) Time per request: 412.753 [ms] (mean) Time per request: 0.413 [ms] (mean, across all concurrent requests) Transfer rate: 302.84 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 16 71.5 4 1004 Processing: 4 216 92.9 202 2340 Waiting: 4 215 92.7 202 2340 Total: 4 232 121.5 215 2341 Percentage of the requests served within a certain time (ms) 50% 215 66% 253 75% 280 80% 299 90% 348 95% 412 98% 490 99% 556 100% 2341 (longest request) [root ~]# ab -n -c 1000 http://172.17.2.66:/ This is ApacheBench, Version 2.3 <Copyright 1996 Adam Twiss, Zeus Technology Ltd, Licensed to The Apache Software Foundation, Benchmarking 172.17.2.66 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed requests Finished requests Server Software: Server Hostname: 172.17.2.66 Server Port: Document Path: / Document Length: 11 bytes Concurrency Level: 1000 Time taken for tests: 7.071 seconds Complete requests: Failed requests: 0 Write errors: 0 Total transferred: bytes HTML transferred: bytes Requests per second: 1414.21 [#/sec] (mean) Time per request: 707.110 [ms] (mean) Time per request: 0.707 [ms] (mean, across all concurrent requests) Transfer rate: 176.


Question: DefaultDummyDevice = "kube-ipvs0" connReuseMinSupportedKernelVersion = "4.1" <ins> // https://github.com/torvalds/linux/commit/35dfb013149f74c2be1ff9c78f14e6a3cd1539d1 connReuseFixedKernelVersion = "5.9" </ins> ) // iptablesJumpChain is tables of iptables chains that ipvs proxier used to install iptables or cleanup iptables.

78 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 9 66.2 2 1019 Processing: 10 661 575.2 372 3603 Waiting: 1 659 573.7 372 3603 Total: 10 670 584.6 380 3607 Percentage of the requests served within a certain time (ms) 50% 380 66% 742 75% 1008 80% 1183 90% 1535 95% 1784 98% 2376 99% 2479 100% 3607 (longest request) [root ~]# ab -n -c 1000 http://172.17.2.66:/ This is ApacheBench, Version 2.3 <Copyright 1996 Adam Twiss, Zeus Technology Ltd, Licensed to The Apache Software Foundation, Benchmarking 172.17.2.66 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed requests Finished requests Server Software: Server Hostname: 172.17.2.66 Server Port: Document Path: / Document Length: 11 bytes Concurrency Level: 1000 Time taken for tests: 13.580 seconds Complete requests: Failed requests: 0 Write errors: 0 Total transferred: bytes HTML transferred: bytes Requests per second: 736.40 [#/sec] (mean) Time per request: 1357.958 [ms] (mean) Time per request: 1.358 [ms] (mean, across all concurrent requests) Transfer rate: 92.05 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 4 6.6 1 49 Processing: 11 1333 491.9 1307 3185 Waiting: 1 1330 491.9 1305 3185 Total: 17 1337 491.


Question: DefaultDummyDevice = "kube-ipvs0" connReuseMinSupportedKernelVersion = "4.1" <ins> // https://github.com/torvalds/linux/commit/35dfb013149f74c2be1ff9c78f14e6a3cd1539d1 connReuseFixedKernelVersion = "5.9" </ins> ) // iptablesJumpChain is tables of iptables chains that ipvs proxier used to install iptables or cleanup iptables.

9 1311 3191 Percentage of the requests served within a certain time (ms) 50% 1311 66% 1565 75% 1679 80% 1748 90% 1911 95% 2129 98% 2384 99% 2650 100% 3191 (longest request) [root ~]# ab -n -c 1000 http://172.17.2.66:/ This is ApacheBench, Version 2.3 <Copyright 1996 Adam Twiss, Zeus Technology Ltd, Licensed to The Apache Software Foundation, Benchmarking 172.17.2.66 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed requests Finished requests Server Software: Server Hostname: 172.17.2.66 Server Port: Document Path: / Document Length: 11 bytes Concurrency Level: 1000 Time taken for tests: 23.720 seconds Complete requests: Failed requests: 936 (Connect: 0, Receive: 0, Length: 936, Exceptions: 0) Write errors: 0 Total transferred: bytes HTML transferred: bytes Requests per second: 421.58 [#/sec] (mean) Time per request: 2372.009 [ms] (mean) Time per request: 2.372 [ms] (mean, across all concurrent requests) Transfer rate: 47.77 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 5 7.5 1 49 Processing: 8 2345 3364.8 333 Waiting: 0 1744 3030.9 277 Total: 8 2350 3364.8 340 Percentage of the requests served within a certain time (ms) 50% 340 66% 1434 75% 3390 80% 5490 90% 8669 95% 9233 98% 99% 100% (longest request) the last time tcp connections of service-b :
/assign Sorry, but I do not understand what are you trying to say.


Question: DefaultDummyDevice = "kube-ipvs0" connReuseMinSupportedKernelVersion = "4.1" <ins> // https://github.com/torvalds/linux/commit/35dfb013149f74c2be1ff9c78f14e6a3cd1539d1 connReuseFixedKernelVersion = "5.9" </ins> ) // iptablesJumpChain is tables of iptables chains that ipvs proxier used to install iptables or cleanup iptables.

: (
/cc
Will this be implemented in 1.21?
/area kube-proxy
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale


Question: VolumeSubpath:                  {Default: true, PreRelease: featuregate.GA}, ConfigurableFSGroupPolicy:      {Default: false, PreRelease: featuregate.Alpha}, BalanceAttachedNodeVolumes:     {Default: false, PreRelease: featuregate.Alpha}, <del> VolumeSubpathEnvExpansion:      {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.19, </del> CSIBlockVolume:                 {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.20 CSIInlineVolume:                {Default: true, PreRelease: featuregate.Beta}, RuntimeClass:                   {Default: true, PreRelease: featuregate.Beta},

/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc


Question: tc.opts..., ) <ins> // Errors </ins> if len(tc.wantErr) != 0 { if err == nil || !strings.Contains(err.Error(), tc.wantErr) { t.Errorf("got error %q, want %q", err, tc.wantErr)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --here is supposed to be set through , but is not called from anywhere except tests, so always returns . should be set through here. Am I missing something? N/A None Kubernetes version (use ): master () Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: None
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
Thanks for pointing that out. The plumbing of extenders to the framework impact calling extenders on preemption. That seems to be a path not covered by tests end to end. Can you submit a PR to fix this? as you mentioned we need to pass the extenders on profile creation. We need to patch the fix back to v1.21 and v1.20 (the release where this was refactored). /cc
I have submitted a PR for a fix.


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package scheme import ( "testing" "k8s.io/apimachinery/pkg/api/testing/roundtrip" "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/fuzzer" ) func TestRoundTripTypes(t *testing.T) { scheme, _, err := NewSchemeAndCodecs() if err != nil { t.Fatalf("unexpected error: %v", err) } roundtrip.RoundTripTestForScheme(t, scheme, fuzzer.Funcs) } </ins>

Solution: I will add one.
/cc
xref
Thanks for sending a PR so quickly!
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code slush, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: If this issue is not resolved or labeled as by Wed, Nov 22 it will be moved out of the v1.9 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: type NodeServer interface { // NodePrepareResources prepares several ResourceClaims // for use on the node. If an error is returned, the <del> // response is ignored. Failures for individidual claims </del> <ins> // response is ignored. Failures for individual claims </ins> // can be reported inside NodePrepareResourcesResponse. NodePrepareResources(context.Context, *NodePrepareResourcesRequest) (*NodePrepareResourcesResponse, error) // NodeUnprepareResources is the opposite of NodePrepareResources.

When ResourceClaimParameters or ResourceClassParameters fail validation in resource driver during PodSchedulingContext sync loop, the error is visible only in the events of the PodSchedulingContext. The error should be visible on the object that fails validation as well, ResourceClaim or ResourceClass respectively. Creating ResourceClaim with unsupported parameters' values or unsupported No response <details</details<details</details<details</details<details</details<details</details<details</details
/assign
/sig node
/triage accepted
KEP:


Question: Interval_Sec 2 # Channels Setup,Windows PowerShell Channels     application,system,security <del> Tag          winevent.raw </del> <ins> Tag          winevt.raw </ins> DB           /var/run/google-fluentbit/pos-files/winlog.db # Json Log Example:

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --On Windows 20H2 we are observing crashloops with fluent-bit due to a malformed Windows Event Log entry in the Security log. Fluent-bit to not crash loop. Bring up a GCE Windows cluster on Windows 20H2. You may see a loss of logs. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted