Question: if err != nil { t.Fatalf("unexpected error %v", err) } err = c.Convert(&A{}, &C{}, 0, nil) if err == nil { t.Errorf("unexpected non-error")

As of now, I'm going to manually update this version marker to unblock 1.28.1 patch release. I'll also take a look into this. /assign
The PR that fixed this issue got reverted so this is still relevant: /reopen
Reopened this issue. <detailsIn response to : The PR that fixed this issue got reverted so this is still relevant: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're running into this issue again with stable-1.29 marker. It's currently pointing to which is It should instead point to I'm going to manually change it following the same procedure as the last time. Version marker updated to
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted
Same issue for 1.30, where I had to update the version marker manually again.


Question: <del> kind: ReplicationController apiVersion: v1 </del> <ins> apiVersion: extensions/v1beta1 kind: Deployment </ins> metadata: name: etcd-discovery <del> creationTimestamp: </del> spec: strategy: type: Recreate <del> resources: {} triggers: - type: ConfigChange </del> replicas: 1 selector: <del> name: etcd-discovery </del> <ins> matchLabels: name: etcd-discovery </ins> template: metadata: <del> creationTimestamp: </del> labels: name: etcd-discovery spec:

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Vagrant OS (e.g. from /etc/os-release): CoreOS Kernel (e.g. ):Install tools: Vagrant & Others: What happened: What you expected to happen: Success How to reproduce it (as minimally and precisely as possible): Followed instructions at Tried to run Received: Anything else do we need to know: From googling, it seems several of the fields in the config are for when using OpenShift origin. But creating the replication controller is a prerequisite to launching OpenShift Origin. Chicken & egg problem.


Question: bashExec, exitCode, err) } } <del> gomega.Expect(err).To(gomega.HaveOccurred(), "%q should fail with exit code %d, but exit without error", bashExec, exitCode) </del> <ins> framework.ExpectError(err, "%q should fail with exit code %d, but exit without error", bashExec, exitCode) </ins> } // KubeletCommand performs `start`, `restart`, or `stop` on the kubelet running on the node of the target pod and waits

What happened: After fixing golint failures under (now still in progress), we need to put for each and . Then common error checking code has become like: We can replace this code with which is more readable and easy to be understood what the code does. Now we have following e2e test directories which can be replaced: [x] test/e2e/apimachinery/ () [x] test/e2e/apps/ () [x] test/e2e/auth/ () [x] test/e2e/common/ () [x] test/e2e/node/ () [x] test/e2e/storage/ () [x] () [x] () [x] () [x] () [x] () [x] () [x] [x] test/e2e/storage/persistentvolumes- [x] () [x] () [x] [x] [x] [x] [x] ()
/sig testing /kind cleanup
/assign PR will be submitted soon.
Fix the following in
Fix in
Fix in
Hi, maybe I could take files beneath test/e2e/storage/vsphere/
oops, I'm finished and prepare for PR.
Thank you That's OK
Fix the following in
/assign
All related PRs were merged. I think this ticket can be closed. BTW, should we also replace to ?
Thanks for your work. thing is managed with We are almost finishing the work also :-)
/remove-kind bug
I found in these files. I'm working on :
I can work to the following item.


Question: e2eservice "k8s.io/kubernetes/test/e2e/framework/service" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" testutil "k8s.io/kubernetes/test/utils" <ins> imageutils "k8s.io/kubernetes/test/utils/image" </ins> utilpointer "k8s.io/utils/pointer" )

[x] Original approval issue: [x] Test pr: [x] Two weeks soak end date: 19 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. a Deployment with a static label the Deployment with a new Label and updated data the Deployment to ensure it's patched all Deployments in all Namespaces find the Deployment(1) ensure that the Deployment is found and is patched Namespaced Deployment(1) via a Collection with a LabelSelector Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 5 points* This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
BobyMcbobs commented in original Issue: Unsure if the endpoint replaceAppsV1NamespacedDeploymentStatus is able to be hit. When this endpoint is attempted to be updated the data after retrieving the update resource doesn't reflect the values in the update
/assign
When using the appv1.Deployment structure instead of maps of interfaces to build the structure in a patch of the Deployment, this error comes up. Still in progress yment-static":"true"}: does not match template , spec.selector: Invalid value: "null": field is immutable]
replaceAppsV1NamespacedDeploymentStatus to test. Increase cover by 1 Endpoint
/reopen PR reverted by due to flaky test
Reopened this issue. <detailsIn response to : /reopen PR reverted by due to flaky test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen Soaking, ready for promotion 19 November 2020
Reopened this issue. <detailsIn response to : /reopen Soaking, ready for promotion 19 November 2020 Instructions for interacting with me using PR comments are available .


Question: e2eservice "k8s.io/kubernetes/test/e2e/framework/service" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" testutil "k8s.io/kubernetes/test/utils" <ins> imageutils "k8s.io/kubernetes/test/utils/image" </ins> utilpointer "k8s.io/utils/pointer" )

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
/triage accepted
/merged


Question: // if you set fire time at X, you can get the bookmark within (X-1,X+1) period. // This is NOT thread-safe. type watcherBookmarkTimeBuckets struct { <ins> lock            sync.Mutex </ins> watchersBuckets map[int64][]*cacheWatcher startBucketID   int64 clock           clock.Clock

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I try to backport bookmark feature to our internal branch. When I enable bookmark feature, apiserver leaks memory. I spend some time to dig the root cause and find the bookmarkWatchers of service cacher is keep increasing. I am not sure if it exists at upstream code. Please let me know if it is ok at latest code. What you expected to happen: Avoid memory leaking How to reproduce it (as minimally and precisely as possible): a cluster. not update service. will find kube-apiserver's memory keep increasing. Anything else we need to know?: Kubelet will watch service with bookmark, and add a watcher to bookmarkWatchers. and service init resource version is 0 Without updating service, we cannot update "lastProcessedResourceVersion", it will keep 0, bookmarkTimer branch will always be continued. As a result, it cannot trigger popExpiredWatchers, so the service watcher will keep increasing every watching. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scalability PLAT :)
How about making this change ?
- can you also please take a look?
Agree with u. That's same way I fix in our branch. But I wanna hear the suggestion of and .
! With the same bugfix wrote above, we can see the memory not increasing any more.
Given that watch-bookmarks doesn't create any additional go-routines (well, creates 1, but cannot leak them), it has to leak the data. The only data that can be leaked is exactly bookmarkWatchers. So I looked into the code and got into exactly the same conclusion - that we must be leaking watchers for resources that aren't changing. The fix as above makes sense (though we also need a unit test for it).
Thanks a lot.
has the fix by
Thanks for reporting the bug, I will look at this day later. In addition, also need to think carefully about whether need to send a bookmark when there are no events at all.
cc


Question: return true, err } <ins> // scaleDownOldRCsForRecreate scales down old rcs when deployment strategy is "Recreate" func (dc *DeploymentController) scaleDownOldRCsForRecreate(oldRCs []*api.ReplicationController, deployment extensions.Deployment) (bool, error) { scaled := false for _, rc := range oldRCs { // Scaling not required. if rc.Spec.Replicas == 0 { continue } _, err := dc.scaleRCAndRecordEvent(rc, 0, deployment) if err != nil { return false, err } scaled = true } return scaled, nil } // scaleUpNewRCForRecreate scales up new rc when deployment strategy is "Recreate" func (dc *DeploymentController) scaleUpNewRCForRecreate(newRC *api.ReplicationController, deployment extensions.Deployment) (bool, error) { if newRC.Spec.Replicas == deployment.Spec.Replicas { // Scaling not required. return false, nil } _, err := dc.scaleRCAndRecordEvent(newRC, deployment.Spec.Replicas, deployment) return true, err } </ins> func (dc *DeploymentController) updateDeploymentStatus(allRCs []*api.ReplicationController, newRC *api.ReplicationController, deployment extensions.Deployment) error { totalReplicas := deploymentutil.GetReplicaCountForRCs(allRCs) updatedReplicas := deploymentutil.GetReplicaCountForRCs([]*api.ReplicationController{newRC})

There's some practical limit on the number of concurrent access tokens that can be used in a Kubernetes cluster before it begins to spend all of its time in the authn stack verifying them. Absent fixes to it was < 500; with the fixes linked there it's still not infinitely scalable-- I still hit (what I believe to be) with ~ 5k unique tokens @ ~700QPS. This request is to this dimension an official name & definition, and list it with others, e.g. here: it it have a useful value (I think it should be 10k) /kind bug /sig scalability
/cc
oh and 4. add tests which verify it is as high as believed.
cc since projected tokens will increase the number of unique tokens
/sig auth
LRUs are not good at scaling and sharding has some major pitfalls as pointed out. I would suggest that we try to drop the LRU cache for successful authentication results in the token cache. The main benefit of using an LRU here is bound cache size. Entries in the token cache have a fast expiration independent of the LRU policy. The records probably take up O(2kb) (guesstimate) of memory. Given that these are successful attempts with presumably valid tokens, I don't think we are getting any benefit from the LRU. If we don't have a constraint on fixed cache size, we can back the token cache with a sync.Map and run a single background goroutine to periodically walk the map and remove expired entries. This would fix the problematic locking and eviction due to nonuniform distribution of tokens over shards. The failure cache is less important and we don't cache failures by default. If we consider this the "slowpath", we can probably leave it as. If that's not good enough, we have a couple options (with various tradeoffs) to improve it.
If off by default is not enough, more reasoning why the failed cache is not super useful: it might help with an accidental DoS but it doesn't prevent anything actively malicious DoS. I think we can focus our improvements on the success cache.
/assign He is already working on adding an explicit test for it.
has taken it over and is actively working on this. Unfortunately, I cannot assign this issue to him as he's not a k8s member yet
It's not closed - all testes must be submitted and appropriate documentation must be
Thanks everyone!!


Question: Describe("guestbook", func() { var guestbookPath = filepath.Join(testContext.RepoRoot, "examples/guestbook") <del> if testContext.Provider != "gce" && testContext.Provider != "gke" { By(fmt.Sprintf("Skipping guestbook, uses createExternalLoadBalancer, a (gce|gke) feature")) } </del> It("should create and stop a working application", func() { <ins> if !providerIs("gce", "gke") { By(fmt.Sprintf("Skipping guestbook, uses createExternalLoadBalancer, a (gce|gke) feature")) return } </ins> defer cleanup(guestbookPath, frontendSelector, redisMasterSelector, redisSlaveSelector) By("creating all guestbook components")

I noticed while running e2e tests locally that some tests were apparently being incorrectly skipped: I checked, and this is also true for our Jenkins runs. Currently it looks like and have this bug, though only the former actually skips the test, I think. The issue appears to be that the code inside the top-level in these files is being run before the testContext struct is filled out (and before is even called), and so testContext.Provider is empty. I'm not immediately sure what the proper fix is; I'm also worried that there might be other things broken by the fact that these blocks are called before initialization is complete. We also probably investigate if there is a better way to mark these tests as skipped; right now, there's no indication in the test report (, Jenkins test report, etc) that these tests were skipped.
I merged and which tried to disable these tests for non-{gke,gce} but apparently didn't follow-up and verify that they were still running on our platforms. /cc
Seems easy enough to fix. I'll have a PR out to fix this before Monday.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Testgrid: https://k8s-
/assign
The test's assumption about test parameter is wrong. Previously it assumed the group name is the same across all zones, but they are different in regional clusters. Looking into using node taints as the permanent fix, as suggested by the TODO comment in the test.
Hm I think taint based evictions is still an alpha feature.
OK. I propose the following solution: Get the zone of Schedule the StatefulSet pod to that zone using node affinity; wait for the pod to run Remove the node affinity from StatefulSet Delete
Can it be simplified? Schedule the pod Figure out which zone that pod got scheduled in Delete the instance group of that zone
Unfortunately I don't know of a good way to look up the instance group given the zone, since there could be instance groups from multiple clusters in the same project
How about: Schedule the pod Figure out the zone Taint all the nodes in the zone Kill the pod
Taint with NoSchedule? yeah that sounds good


Question: return state } <ins> func updateRunningPodAndContainerMetrics(pods []*kubecontainer.Pod) { // Set the number of running pods in the parameter metrics.RunningPodCount.Set(float64(len(pods))) // intermediate map to store the count of each "container_state" containerStateCount := make(map[string]int) for _, pod := range pods { containers := pod.Containers for _, container := range containers { // update the corresponding "container_state" in map to set value for the gaugeVec metrics containerStateCount[string(container.State)]++ } } for key, value := range containerStateCount { metrics.RunningContainerCount.WithLabelValues(key).Set(float64(value)) } } </ins> func (pr podRecords) getOld(id types.UID) *kubecontainer.Pod { r, ok := pr[id] if !ok {

Some kubelet metrics use non-standard prometheus collectors unnecessarily. Specifically, we have which can be simple gauges. Since we cannot migrate non-standard prometheus collectors using the internal metrics stability framework, we will need to convert these metrics into standard prometheus variants. /sig instrumentation /assign
GitHub didn't allow me to assign the following users: solodov. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : Some kubelet metrics use non-standard prometheus collectors unnecessarily. Specifically, we have which can be simple gauges. Since we cannot migrate non-standard prometheus collectors using the internal metrics stability framework, we will need to convert these metrics into standard prometheus variants. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
recommends that we generate those metrics in the GenericPLEG during the relist. We can generate the data directly from the pods call there ().
I would be interested to work on this issue if no one else is working on it currently
Go for it!
/assign


Question: // Volume is added to asw. Because attach operation fails, volume should not be reported as attached to the node. waitForVolumeAddedToNode(t, generatedVolumeName, nodeName1, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> // When volume is added to the node, it is set to mounted by default. Then the status will be updated by checking node status VolumeInUse. // Without this, the delete operation will be delayed due to mounted status

TestRunOneVolumeDetachFailNodeWithReadWriteOnce 4 times since 03-19 23:00 PDT, intermittently ! https://k8s- Still investigating whether this is a tolerable flake or not with more history than testgrid view, or otherwise would appreciate input from anyone with expertise in this test, especially as regards to whether this should be a blocker for tomorrows beta cut. maybe: /sig storage /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is already open for this, looks like it is attempting to fix it
thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. /close
Closing this issue. <detailsIn response to : thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Volume is added to asw. Because attach operation fails, volume should not be reported as attached to the node. waitForVolumeAddedToNode(t, generatedVolumeName, nodeName1, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> // When volume is added to the node, it is set to mounted by default. Then the status will be updated by checking node status VolumeInUse. // Without this, the delete operation will be delayed due to mounted status

pull-kubernetes-unit TestRunOneVolumeDetachFailNodeWithReadWriteOnce Jan 7th, 2022 Check volume <fake-plugin/volume-nameis reported as attached to node <fail-detach-node, got false, expected true No response_ /sig storage
/cc
this is the top unit test flake according to
cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
I think this test is no longer flaky by checking the test grid, close it now.
Unfortunately this does still seem to be flaky: Here was the output from : /reopen
Reopened this issue. <detailsIn response to : Unfortunately this does still seem to be flaky: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
still the case
How about adding some logging to help troubleshoot this? I looked at it a little, but it is hard to tell which part of the test is failing. PR
/
This is still happening
/triage accepted
From the logs, in the OneVolumeDetachFailNodeWithReadWriteOnce test, succeeded, but the next verification failed. In verifyVolumeAttachedToNode, the attachState is from calling and within asw, it calls . In verifyVolumeReportedAsAttachedToNode, the volumes are obtained from calling , it seems that the node is not in asw.nodesToUpdateStatusFor at this step. Cannot reproduce locally. I tried to run the test 100 times locally with debug.Stack()... 3 * 100 times all passed.
Tried to run stress test on my linux machine as well but still cannot reproduce. Command: Result: Also tried running n process in parallel. Command: Result:
It seems like it happens more frequently.


Question: // Volume is added to asw. Because attach operation fails, volume should not be reported as attached to the node. waitForVolumeAddedToNode(t, generatedVolumeName, nodeName1, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> // When volume is added to the node, it is set to mounted by default. Then the status will be updated by checking node status VolumeInUse. // Without this, the delete operation will be delayed due to mounted status

But I may have no time to dig into the cause. Let's see.
cc as the test author.
The problem is that this test has a lot of dependency on the sleeps that are hardcoded with a very narrow scope, but our CI seems to be able to stress it much more. As soon as you slightly change one of the sleep duration it starts to flake locally The test can not make an assumption that timing is perfect, even more in the sub second range can someone in sig-storage review this test and make it less dependant on those tight sleeps?
According to most failures, it failed at Line 660. Let me run some stress locally. with 250ms sleep with 100ms sleep
I agree this test case is too brittle due to its specific timing dependencies. To fix this, I suggest reworking the test case to: 1) Add retry with timeout to wait for final conditions 2) Remove the intermediate timing-based state transition checks. Instead, we can keep track of state transitions in the Fakes and validate those stats. wdyt?
Friendly ping from 1.27 CI Signal that this test is still a noisy flake on the master-blocking dashboard as CI Signal has flagged it in and


Question: if err != nil { return currentRevision, updateRevision, currentStatus, err } <del> // update the set's status err = ssc.updateStatefulSetStatus(ctx, set, currentStatus) if err != nil { return currentRevision, updateRevision, currentStatus, err } klog.V(4).InfoS("StatefulSet pod status", "statefulSet", klog.KObj(set), "replicas", currentStatus.Replicas, "readyReplicas", currentStatus.ReadyReplicas, "currentReplicas", currentStatus.CurrentReplicas, "updatedReplicas", currentStatus.UpdatedReplicas) </del> klog.V(4).InfoS("StatefulSet revisions", "statefulSet", klog.KObj(set), "currentRevision", currentStatus.CurrentRevision,

I create a ResourceQuota with two pods limit in my namespace. Then, I create a statefuleSet with 3 replicas. As expected, two pods are created successfully, and the third pod failed to create. But, the ReadyReplicas is not 2 ,but 1. The ReadyReplicas is correct 1Create a ResourceQuota with two pods limit in a new namespace. 2Create a statefulSet with 3 replicas 3Check if the ReadyReplicas of statefulSet matchs the number of stateful running pods. 1According to the logic of the statefulSet controller, new pod created in current sync is not be immediately counted into the ReadyReplicas. 2Failure to create pod will cause to return error, then skip updating status. <details</details<details</details<detailskind v1.23.3 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign


Question: } } <ins> if oldCRD.UID != newCRD.UID { r.removeStorage_locked(oldCRD.UID) } </ins> storageMap := r.customStorage.Load().(crdStorageMap) oldInfo, found := storageMap[newCRD.UID] if !found {

Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestWebhookConverterWithDefaulting /assign /priority important-soon /area custom-resources /sig api-machinery /kind flake /milestone v1.16
/cc
several of the webhook converter tests have similar flakes. it looks like the wait for storage version to become effective times out occasionally Rare flakes seen in all of these tests: TestWebhookConverterWithWatchCache/nontrivial-converter TestWebhookConverterWithWatchCache/metadata-mutating-converter TestWebhookConverterWithDefaulting/metadata-mutating-converter TestWebhookConverterWithWatchCache/noop-converter TestWebhookConverterWithDefaulting/empty-response TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-2 TestWebhookConverterWithoutWatchCache/noop-converter/check-0/v1beta2 TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-0/v1beta2
cc
Yikes, I’ll sort it out.
iiuc sending an empty patch gets short-circuited and doesn't bump CR generation, even if the storage version gets changed, which means we do not re-write to etcd and go through the encoding path this would be racing and doing noop's if the first pass of didn't get the expected storage version I will send a fix to make the patch do actual mutation
It doesn't bump generation, but the bytes to persist in etcd would still be different if the storage version had changed, so they should persist. You can verify that by changing the storage version on a CRD yourself, do an empty patch, and see if the resourceVersion changes.
you're right. Generation is for non-metadata change only, and the RV changes. I queried etcd locally and the apiVersion did change for an empty patch (sidetracking: another observation is after changing the storage version (say v1 -v2; noop convertor), empty-patching v2 endpoint would bump generation, while empty-patching v1 endpoint wouldn't)
another theory: our cached storage strategy could be stale if the CRD informer merges multiple events together (e.g. the informer does a re-list).


Question: } } <ins> if oldCRD.UID != newCRD.UID { r.removeStorage_locked(oldCRD.UID) } </ins> storageMap := r.customStorage.Load().(crdStorageMap) oldInfo, found := storageMap[newCRD.UID] if !found {

Two scenarios: -create-update events merged into an update event: strategy with UID:a was created on demand for CRD Foo Foo was deleted and recreated strategy with UID:b was created on demand for the new Foo new Foo got updated (e.g. switching storage version) the strategy with UID:b is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the delete-create-update events into , the strategy with UID:a will be deleted, but the strategy with UID:b won't. -update events merged into a create event: strategy with UID:a was created on demand for CRD Foo Foo was updated (e.g. switching storage version) the strategy with UID:a is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the create-update events into , the strategy with UID:a won't be deleted (as we don't react on create events). another race found in , which could happen in a single update.
I think I :


Question: // this is ok because we know exactly how we want to be serialized TypeMeta: metav1.TypeMeta{APIVersion: batchv1.SchemeGroupVersion.String(), Kind: "Job"}, ObjectMeta: metav1.ObjectMeta{ <del> Name:        o.Name, Annotations: annotations, Labels:      cronJob.Spec.JobTemplate.Labels, OwnerReferences: []metav1.OwnerReference{ { APIVersion: batchv1.SchemeGroupVersion.String(), Kind:       "CronJob", Name:       cronJob.GetName(), UID:        cronJob.GetUID(), }, }, </del> <ins> Name:            o.Name, Annotations:     annotations, Labels:          cronJob.Spec.JobTemplate.Labels, OwnerReferences: []metav1.OwnerReference{*metav1.NewControllerRef(cronJob, batchv1.SchemeGroupVersion.WithKind("CronJob"))}, </ins> }, Spec: cronJob.Spec.JobTemplate.Spec, }

Given a cronjob created with a scedule of "0 6 *" that failed at last run (status.lastSuccessfulTime is now "before" status.lastScheduleTime). Then I trigger the cronjob by hand and this jobs ends successfully. However status.lastSuccessfulTime is not updated, thus making it difficult to monitor failed cronjobs. status.lastSuccessfulTime should be updated with a timestamp of the succeeded job endTime Install the attached files: Wait until cronjob has actually trigged, test with to see that lastScheduleTime and lastSuccessfulTime are populated ok (as expected). Then apply the fail config-map, now the job will not complete OK: Again wait until cronjob has trigged automatically, observe to see that lastScheduleTime is updated but lastSuccessfulTime is not. Then apply the "OK" configmap and then create the job by hand: when the job has finished and completed ok, observe that lastSuccessfulTime is still not updated. Wait until next schedule and observe that the field is now updated. No response <details</details<detailsTested on kind cluster on Mac </details<details</details<details</details<details</details<details</details
Had to upload the files as .txt files as github wouldn't accept .yaml or .yml files
(Double-check: I may read wrong messagess ) /sig apps
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : When I tried without the prefix in the job name, the status is updated. /sig apps /area job not sure if this is a bug or an expected behavior. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area workload-api/cronjob
I have just tested in my kind cluster and neither with the same naming pattern as the normal jobs (which apparently isn't epoch) or with epoch - this will not trigger a update of the field. I have also tested it on another cluster (rke2 based) and here I'm not able to get the field updated no matter which naming pattern I use.
Sorry for the mistake. I can repro the problem you have and not related to the prefix. /cc
/assign
/remove-sig apps /sig cli
/triage accepted


Question: v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/kubernetes/pkg/kubelet/cm" "k8s.io/kubernetes/pkg/kubelet/config" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container"

What happened: A deployment is created that uses a after the cluster master is created but before the cluster worker nodes are fully initialized. During the initialization of the worker nodes, the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector. Depending on how long the nodes are available for scheduling without the necessary node labels for the deployment, the scheduler will start to spam with cluster with pods in status. This spamming stops once the worker nodes are fully initialized and the pods are scheduled successful. The exact timing of all of this is TBD. What you expected to happen: Pods in status are annoying and require manual cleanup. It would be preferred that the scheduler avoid this an simply issue warning events. How to reproduce it (as minimally and precisely as possible): See issue description. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.18.3 Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 4.15.0-96-generic Install tools: Managed service install Network plugin and version (if this is a network-related bug): N/A Others: N/A
Related to and
/sig scheduling
by spamming are you referring to scheduler logs? also, why are the nodes not labelled from the get go?
Just to confirm, all the pods in question are still in Pending state, yes? If the pods are in pending state, all of them will get scheduled eventually once the nodes get their labels.
The pods are not in state. They are actually in state. The pods eventually get scheduled but we are left with anywhere from a few to dozens of these state pods. Also, we are looking at labeling the nodes via kubelet flags to see if that helps.
/remove-kind bug
After we add some retry logic and wait time in this problem can be solved: As you can see this function returns initial node as the first request failed.
I also get hit by this, when single controller node gets rebooted and scheduler starts as static container before kubelet becomes ready.


Question: v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/kubernetes/pkg/kubelet/cm" "k8s.io/kubernetes/pkg/kubelet/config" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container"

Can be cleaned up with:
/sig node
the spam cycle is this loop: "pod create, pod schedule, worker reject, pod failed" -- we don't have a lot of protection in higher level workload controllers to handle pods that fail at kubelet admission time versus pods that just dont create due to quota/admission. in general, is the spam causing any other stability issues? is pod gc pruning the pods?
I wonder if we can do a trick in node lister to mitigate any race condition here...
we could maybe do something similar to to ensure node lister has synced...
hacking on option to ensure if kubelet has a node lister with a valid kube client, we wait for it to sync at least once here which should mitigate this issue. see:
We noticed the problem on the OLM pod which is managed by the OLM operator. Right or wrong, the operator does not like seeing the pod go to status and no longer reconciles the pod. The end result is that manual intervention (i.e. pod deletion) is required to fix the problem.
sure... i would have thought the pod is failed in that instance and reached a terminal state. either way, i am looking at what i can do to mitigate this.
The problem persists in our environment as well, and the workaround cited above will not do anymore. Is there something in the works for this one?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: expected: &batchv1.Job{ TypeMeta: metav1.TypeMeta{APIVersion: batchv1.SchemeGroupVersion.String(), Kind: "Job"}, ObjectMeta: metav1.ObjectMeta{ <del> Name:        jobName, Annotations: map[string]string{"cronjob.kubernetes.io/instantiate": "manual"}, OwnerReferences: []metav1.OwnerReference{ { APIVersion: batchv1.SchemeGroupVersion.String(), Kind:       "CronJob", Name:       cronJob.GetName(), UID:        cronJob.GetUID(), }, }, </del> <ins> Name:            jobName, Annotations:     map[string]string{"cronjob.kubernetes.io/instantiate": "manual"}, OwnerReferences: []metav1.OwnerReference{*metav1.NewControllerRef(cronJob, batchv1.SchemeGroupVersion.WithKind("CronJob"))}, </ins> }, Spec: batchv1.JobSpec{ Template: corev1.PodTemplateSpec{


Question: s.GenericAPIServer.Handler.NonGoRestfulMux.Handle("/apis", apisHandler) s.GenericAPIServer.Handler.NonGoRestfulMux.UnlistedHandle("/apis/", apisHandler) <del> apiserviceRegistrationController := NewAPIServiceRegistrationController(informerFactory.Apiregistration().InternalVersion().APIServices(), c.GenericConfig.SharedInformerFactory.Core().V1().Services(), s) </del> <ins> apiserviceRegistrationController := NewAPIServiceRegistrationController(informerFactory.Apiregistration().InternalVersion().APIServices(), s) </ins> availableController := statuscontrollers.NewAvailableConditionController( informerFactory.Apiregistration().InternalVersion().APIServices(), c.GenericConfig.SharedInformerFactory.Core().V1().Services(),

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Currently APIServiceRegistrationController watch service change and find the related APIServices related if any. Then it AddAPIService/RemoveAPIService for APIAggregator. I dig into the kube-aggregator and find it no need totally, because update APIServices status when related service or endpoint change. So if does not watch services, it will not miss any event for APIServices. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/cc
/kind enhancement /sig api-machinery
/assign
that does not follow. AvailableConditionController modifies the APIService object. APIServiceRegistrationController must still watch the APIService objects to observe those changes.
oh, I misread the description. You're saying APIServiceRegistrationController doesn't need to watch v1.Service objects


Question:
PLEASE NOTE: This document applies to the HEAD of the source tree
 If you are using a released version of Kubernetes, you should refer to the docs that go with that version.  The latest 1.0.x release of this document can be found [here](http://releases.k8s.io/release-1.0/docs/devel/adding-an-APIGroup.md). Documentation for other releases can be found at [releases.k8s.io](http://releases.k8s.io).  --   Adding an API Group =============== This document includes the steps to add an API group. You may also want to take a look at PR [#16621](https://github.com/kubernetes/kubernetes/pull/16621) and PR [#13146](https://github.com/kubernetes/kubernetes/pull/13146), which add API groups. Please also read about [API conventions](api-conventions.md) and [API changes](api_changes.md) before adding an API group. ### Your core group package: We plan on improving the way the types are factored in the future; see [#16062](https://github.com/kubernetes/kubernetes/pull/16062) for the directions in which this might evolve. 1. Create a folder in pkg/apis to hold you group. Create types.go in pkg/apis/`<group>`/ and pkg/apis/`<group>`/`<version>`/ to define API objects in your group; 2. Create pkg/apis/`<group>`/{register.go, `<version>`/register.go} to register this group's API objects to the encoding/decoding scheme (e.g., [pkg/apis/extensions/register.go](../../pkg/apis/extensions/register.go) and [pkg/apis/extensions/v1beta1/register.go](../../pkg/apis/extensions/v1beta1/register.go); 3. Add a pkg/apis/`<group>`/install/install.go, which is responsible for adding the group to the `latest` package, so that other packages can access the group's meta through `latest.Group`. You probably only need to change the name of group and version in the [example](../../pkg/apis/extensions/install/install.go)). You need to import this `install` package in {pkg/master, pkg/client/unversioned, cmd/kube-version-change}/import_known_versions.go, if you want to make your group accessible to other packages in the kube-apiserver binary, binaries that uses the client package, or the kube-version-change tool. Step 2 and 3 are mechanical, we plan on autogenerate these using the cmd/libs/go2idl/ tool. ### Scripts changes and auto-generated code: 1. Generate conversions and deep-copies: 1. Add your "group/" or "group/version" into hack/after-build/{update-generated-conversions.sh, update-generated-deep-copies.sh, verify-generated-conversions.sh, verify-generated-deep-copies.sh}; 2. Run hack/update-generated-conversions.sh, hack/update-generated-deep-copies.sh. 2. Generate files for Ugorji codec: 1. Touch types.generated.go in pkg/apis/`<group>`{/, `<version>`}; 2. Run hack/update-codecgen.sh. ### Client (optional): We are overhauling pkg/client, so this section might be outdated; see [#15730](https://github.com/kubernetes/kubernetes/pull/15730) for how the client package might evolve. Currently, to add your group to the client package, you need to 1. Create pkg/client/unversioned/`<group>`.go, define a group client interface and implement the client. You can take pkg/client/unversioned/extensions.go as a reference. 2. Add the group client interface to the `Interface` in pkg/client/unversioned/client.go and add method to fetch the interface. Again, you can take how we add the Extensions group there as an example. 3. If you need to support the group in kubectl, you'll also need to modify pkg/kubectl/cmd/util/factory.go. ### Make the group/version selectable in unit tests (optional): 1. Add your group in pkg/api/testapi/testapi.go, then you can access the group in tests through testapi.`<group>`; 2. Add your "group/version" to `KUBE_API_VERSIONS` and `KUBE_TEST_API_VERSIONS` in hack/test-go.sh.  [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/devel/adding-an-APIGroup.md?pixel)]()

AFAIK, we don't have a documentation on how to add a new API group. Currently we have and that try to add a new API group. I will draft a guide to ease the future endeavor of adding groups. And probably by writing this guide, I will see how can we make the API group machinery easier to use. cc


Question: } // prepare kube clients. <del> client, leaderElectionClient, eventClient, err := createClients(c.ComponentConfig.ClientConnection, o.Master) </del> <ins> client, leaderElectionClient, eventClient, err := createClients(c.ComponentConfig.ClientConnection, o.Master, c.ComponentConfig.LeaderElection.RenewDeadline.Duration) </ins> if err != nil { return nil, err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kube-cm kube-scheduler does not exit when kube-apiserver suspended What you expected to happen: kube-cm kube-scheduler should exit after LeaderElector stops leading How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig api-machinery /kind bug
/assign
/cc


Question: v1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> unstructuredv1 "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" </ins> "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" </ins> "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch" <ins> "k8s.io/client-go/dynamic" </ins> clientset "k8s.io/client-go/kubernetes" appsclient "k8s.io/client-go/kubernetes/typed/apps/v1" watchtools "k8s.io/client-go/tools/watch"

[x] Original approval issue: [x] Test pr: [x] Two weeks soak end date: 19 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. a Deployment with a static label the Deployment with a new Label and updated data the Deployment to ensure it's patched all Deployments in all Namespaces find the Deployment(1) ensure that the Deployment is found and is patched Namespaced Deployment(1) via a Collection with a LabelSelector Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 5 points* This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
BobyMcbobs commented in original Issue: Unsure if the endpoint replaceAppsV1NamespacedDeploymentStatus is able to be hit. When this endpoint is attempted to be updated the data after retrieving the update resource doesn't reflect the values in the update
/assign
When using the appv1.Deployment structure instead of maps of interfaces to build the structure in a patch of the Deployment, this error comes up. Still in progress yment-static":"true"}: does not match template , spec.selector: Invalid value: "null": field is immutable]
replaceAppsV1NamespacedDeploymentStatus to test. Increase cover by 1 Endpoint
/reopen PR reverted by due to flaky test
Reopened this issue. <detailsIn response to : /reopen PR reverted by due to flaky test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen Soaking, ready for promotion 19 November 2020
Reopened this issue. <detailsIn response to : /reopen Soaking, ready for promotion 19 November 2020 Instructions for interacting with me using PR comments are available .


Question: v1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> unstructuredv1 "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" </ins> "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" </ins> "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch" <ins> "k8s.io/client-go/dynamic" </ins> clientset "k8s.io/client-go/kubernetes" appsclient "k8s.io/client-go/kubernetes/typed/apps/v1" watchtools "k8s.io/client-go/tools/watch"

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
/triage accepted
/merged


Question: deploymentReplicas := 4 deploymentImage := "gcr.io/google_samples/gb-redisslave:v1" deploymentMinReadySeconds := 5 <ins> deploymentStrategyType := extensions.RollingUpdateDeploymentStrategyType </ins> Logf("Creating deployment %s", deploymentName) <del> newDeployment := newDeployment(deploymentName, deploymentReplicas, deploymentPodLabels, deploymentImageName, deploymentImage) </del> <ins> newDeployment := newDeployment(deploymentName, deploymentReplicas, deploymentPodLabels, deploymentImageName, deploymentImage, deploymentStrategyType) </ins> newDeployment.Spec.Strategy.RollingUpdate = &extensions.RollingUpdateDeployment{ MaxUnavailable:  intstr.FromInt(1), MaxSurge:        intstr.FromInt(1),

There's some practical limit on the number of concurrent access tokens that can be used in a Kubernetes cluster before it begins to spend all of its time in the authn stack verifying them. Absent fixes to it was < 500; with the fixes linked there it's still not infinitely scalable-- I still hit (what I believe to be) with ~ 5k unique tokens @ ~700QPS. This request is to this dimension an official name & definition, and list it with others, e.g. here: it it have a useful value (I think it should be 10k) /kind bug /sig scalability
/cc
oh and 4. add tests which verify it is as high as believed.
cc since projected tokens will increase the number of unique tokens
/sig auth
LRUs are not good at scaling and sharding has some major pitfalls as pointed out. I would suggest that we try to drop the LRU cache for successful authentication results in the token cache. The main benefit of using an LRU here is bound cache size. Entries in the token cache have a fast expiration independent of the LRU policy. The records probably take up O(2kb) (guesstimate) of memory. Given that these are successful attempts with presumably valid tokens, I don't think we are getting any benefit from the LRU. If we don't have a constraint on fixed cache size, we can back the token cache with a sync.Map and run a single background goroutine to periodically walk the map and remove expired entries. This would fix the problematic locking and eviction due to nonuniform distribution of tokens over shards. The failure cache is less important and we don't cache failures by default. If we consider this the "slowpath", we can probably leave it as. If that's not good enough, we have a couple options (with various tradeoffs) to improve it.
If off by default is not enough, more reasoning why the failed cache is not super useful: it might help with an accidental DoS but it doesn't prevent anything actively malicious DoS. I think we can focus our improvements on the success cache.
/assign He is already working on adding an explicit test for it.
has taken it over and is actively working on this. Unfortunately, I cannot assign this issue to him as he's not a k8s member yet
It's not closed - all testes must be submitted and appropriate documentation must be
Thanks everyone!!


Question: vSphereInstance) } klog.V(4).Infof("DiskIsAttached result: %v and error: %q, for volume: %q", attached, err, volPath) <del> return attached, err </del> <ins> return attached, volPath, err </ins> } requestTime := time.Now() <del> isAttached, err := diskIsAttachedInternal(volPath, nodeName) </del> <ins> isAttached, newVolumePath, err := diskIsAttachedInternal(volPath, nodeName) </ins> if err != nil { if vclib.IsManagedObjectNotFoundError(err) { err = vs.nodeManager.RediscoverNode(nodeName) if err == vclib.ErrNoVMFound { isAttached, err = false, nil } else if err == nil { <del> isAttached, err = diskIsAttachedInternal(volPath, nodeName) </del> <ins> isAttached, newVolumePath, err = diskIsAttachedInternal(volPath, nodeName) </ins> } } } vclib.RecordvSphereMetric(vclib.OperationDiskIsAttached, requestTime, err) <del> return isAttached, err </del> <ins> return isAttached, newVolumePath, err </ins> } // DisksAreAttached returns if disks are attached to the VM using controllers supported by the plugin.

When user creates a static PV using folder name in path: on vSAN backed datastore then detach disk does not work because check fails - The code expects folder name to be UUID but it does not and hence it determines disk is not attached to the node and hence no detach is ever performed. /sig storage cc


Question: if m.dynamicTemplate { go wait.Forever(func() { // check if the current template matches what we last requested <del> if !reflect.DeepEqual(m.getLastRequest(), m.getTemplate()) { </del> <ins> if !m.certSatisfiesTemplate() && !reflect.DeepEqual(m.getLastRequest(), m.getTemplate()) { </ins> // if the template is different, queue up an interrupt of the rotation deadline loop. // if we've requested a CSR that matches the new template by the time the interrupt is handled, the interrupt is disregarded. templateChanged <- struct{}{}

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I'm testing the new option in kubernetes 1.12 and it seems that kubelet always requests the signing of a new CSR and tries to rotate the certificates when it is started. This is happening even though there's a valid (brand new) certificate on disk issued by kube-controller-manager. From what I can tell, this is happening because kubelet implements and is set after a successful certificate rotation, but is never persisted to disk. If kubelet is restarted, then the last requested CSR is nil. What you expected to happen: Ideally, kubelet should only request the signing of a new CSR if it doesn't already have a serving certificate or it's about to expire. How to reproduce it (as minimally and precisely as possible): Restart a kubelet instance that had it's serving certificate issued by kube-controller-manager Check and see that it requested the signing of a new CSR Anything else we need to know?: kubelet is being run with . , , and are all omitted to use the default values. The TLS bootstrapping for client certificates is currently not being used--just the bootstrapping for serving certificates. Environment: Kubernetes version: v1.12.0 - built from source at the tag () Cloud provider or hardware configuration: Bare metal OS: CentOS 7.5.1804 Kernel: Install tools: Custom RPMs built from this git repo with no additional patches Others:
/sig node
/sig auth
Not sure if this is the ideal fix, but I've patched it so that it compares the current CSR template to the last requested template only if it doesn't already match the current cert.
cc
thanks for the proposed fix. I think it looks good. Could you send a PR? We should also remove entirely and only look at current cert and template. does that sound reasonable?
Sure thing, I'd be happy to do that. EDIT: I will submit a PR after approval from my employer.
I think that's probably ok, as long as it checks everything we include in the template and we consider a certificate missing anything we request to be cause for re-requesting another cert (an issuer couldn't choose to issue a cert that drops some SANs and us not request another cert) left some comments in
This regression is a frequent flyer.


Question: if m.dynamicTemplate { go wait.Forever(func() { // check if the current template matches what we last requested <del> if !reflect.DeepEqual(m.getLastRequest(), m.getTemplate()) { </del> <ins> if !m.certSatisfiesTemplate() && !reflect.DeepEqual(m.getLastRequest(), m.getTemplate()) { </ins> // if the template is different, queue up an interrupt of the rotation deadline loop. // if we've requested a CSR that matches the new template by the time the interrupt is handled, the interrupt is disregarded. templateChanged <- struct{}{}

I'd love to see an integration test that would catch this (but that shouldn't block a fix).
Thanks for the comments! Now that I have the CLA signed, I'll address those and submit a PR.
Just submitted the PR:


Question: // The name of this port. All ports in an EndpointSlice must have a unique // name. If the EndpointSlice is dervied from a Kubernetes service, this // corresponds to the Service.ports[].name. <del> // Name must either be an empty string or pass IANA_SVC_NAME validation: // * must be no more than 15 characters long // * may contain only [-a-z0-9] // * must contain at least one letter [a-z] // * it must not start or end with a hyphen, nor contain adjacent hyphens </del> <ins> // Name must either be an empty string or pass DNS_LABEL validation: // * must be no more than 63 characters long. // * must consist of lower case alphanumeric characters or '-'. // * must start and end with an alphanumeric character. </ins> // Default is empty string. optional string name = 1;

What happened: No endpoints created in the EndpointSlice for my service due to the port on the service being too long. Logs show: EndpointSlice created has no endpoint: What you expected to happen: Service that works with Endpoints should work with EndpointSlice. If this is intentional, it should be clearly documented How to reproduce it (as minimally and precisely as possible): Create a service with a long port name Environment: Kubernetes version (use ): 1.16 Cloud provider or hardware configuration: kind
/sig networking
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /sig networking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
For a rough measure of impact, Istio had 5 ports that were over this limit (and many more that were right at 15 characters), so I suspect it may not be terribly rare.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This needs to be resolved before endpointslices graduate from alpha
/assign
/remove-triage unresolved


Question: nodeStatusUpdateRetry = 5 // Location of container logs. <del> containerLogsDir = "/var/log/containers" </del> <ins> ContainerLogsDir = "/var/log/containers" </ins> // max backoff period, exported for the e2e test MaxContainerBackOff = 300 * time.Second

Cluster logging assumes kubelet stores container log at a specific directory with specific name. Currently the path is: After the CRI work, the path will be: However, we don't have test coverage for this. We don't even know whether the log files are there or not. We should add a node e2e test for this, basically: 1) Create a container which prints logs 2) Check whether the expected log file is created. 3) Check whether the logs are collected in the file as expected. And it would be better to launch a pod to do the check instead of accessing local file system directly, which makes containerizing easier. /cc because you are collecting missing test. /cc
I will try to add a node-e2e test there
I think it's ok to add this for CRI. But there maybe problems on today's interface for rkt. If I'm not wrong, rkt does't store logs at same location with docker. cc/ Is this true?
Thanks! :)
After gets merged, we need to update the test to verify the cri log path.


Question: "//vendor/k8s.io/apiserver/pkg/server:go_default_library", "//vendor/k8s.io/apiserver/pkg/util/feature:go_default_library", "//vendor/k8s.io/apiserver/pkg/util/proxy:go_default_library", <del> "//vendor/k8s.io/client-go/informers/core/v1:go_default_library", </del> "//vendor/k8s.io/client-go/listers/core/v1:go_default_library", "//vendor/k8s.io/client-go/pkg/version:go_default_library", "//vendor/k8s.io/client-go/rest:go_default_library",


Question: "//pkg/util/metrics:go_default_library", "//staging/src/k8s.io/api/apps/v1:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/api/equality:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: a cluster, And node1 with resource 1 DaemonSet, request resource 4 run , There is no pod created. resource to the node: run , There is still no pod created. What you expected to happen: In step 5, There is one pod created. How to reproduce it (as minimally and precisely as possible): see above description Anything else we need to know?: Environment: Kubernetes version (use ):1.10.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release):CentOS7 Kernel (e.g. ): Install tools: Others: /sig app
xref
Thanks! is not work here! Because: DaemonSetsController didn't watch the changes.(we use to updateNode.) we restart kubelet, The DaemonSetsController watch the changes, but it seems that the node info is not updated, and still failed to create pod:
I found it is a bug here, ignore the changes. I made a fix , PTAL! Thanks!
Any workarounds for old cluster versions? Up: looks like in my case helped but after some delay.
, which version of k8s are you using? If it's sunset, you have to merge the pr yourself :). I think restart kubelet trigger delete/add to make sure it sync .
Sorry, I didn't say
I think we still support 1.12 for now :) let's cherry pick it if not and wait for next 1.12 release.


Question: }, } <ins> func addOrUpdateUnschedulablePod(p *PriorityQueue, pod *v1.Pod) { p.lock.Lock() defer p.lock.Unlock() p.unschedulableQ.addOrUpdate(pod) } func getUnschedulablePod(p *PriorityQueue, pod *v1.Pod) *v1.Pod { p.lock.Lock() defer p.lock.Unlock() return p.unschedulableQ.get(pod) } </ins> func TestPriorityQueue_Add(t *testing.T) { q := NewPriorityQueue(nil) if err := q.Add(&medPriorityPod); err != nil {

Which jobs are failing: pull-kubernetes-bazel-test Which test(s) are failing: bazel test //pkg/scheduler/internal/queue:godefaulttest Since when has it been failing: Testgrid link: Reason for failure: WARNING: DATA RACE Read at 0x00c000230db0 by goroutine 47: runtime.mapiterinit() +0x0 (PriorityQueue).flushUnschedulableQLeftover() +0x17a (PriorityQueue).flushUnschedulableQLeftover-fm() +0x41 () +0x61 () +0xce () +0x5a Previous write at 0x00c000230db0 by goroutine 45: runtime.mapassignfaststr() +0x0 (UnschedulablePodsMap).addOrUpdate() +0x30e () +0x601 testing.tRunner() +0x162 Goroutine 47 (running) created at: (PriorityQueue).run() +0x172 () +0xe0b () +0x50 () +0x5a5 testing.tRunner() +0x162 Goroutine 45 (running) created at: testing.(T).Run() +0x659 testing.runTests.func1() +0xa8 testing.tRunner() +0x162 testing.runTests() +0x4ee testing.(M).Run() +0x2ee () bazel-out/k8- +0x2fb Anything else we need to know: I'm very sorry about that. UnschedulablePodsMap is not thread-safe. In , lock will be acquired before operating UnschedulablePodsMap. But in , lock isn‘t be acquired. In the previous case, only a single thread runs so it's ok. But after add goroutine , there will be data race. So I need to acquire lock before operating UnschedulablePodsMap.
/sig testing /sig scheduling


Question: func NewManager(topology []cadvisorapi.Node, topologyPolicyName string, topologyScopeName string, topologyPolicyOptions map[string]string) (Manager, error) { klog.InfoS("Creating topology manager with policy per scope", "topologyPolicyName", topologyPolicyName, "topologyScopeName", topologyScopeName) <ins> // When policy is none, the scope is not relevant, so we can short circuit here. if topologyPolicyName == PolicyNone { return &manager{scope: NewNoneScope()}, nil } </ins> opts, err := NewPolicyOptions(topologyPolicyOptions) if err != nil { return nil, err

Follow up from: See details there. When is , there is no need to discover the NUMA topology. /sig node /priority backlog /kind feature /help-wanted
/cc
/triage accepted
I would like to take this up also can I get more brief information about it? code review comments from asking that inside the NewManager() implementation, we need to skip NewPolicyOptions and NewNUMAInfo but when the NumaTopology comes kubelet issue also comes. I need some more information to start with this!
/cc
When there is no topology manger policy specified, NewManager should be a no-op. Just skip all checks.
Exactly. Just to add: we may need few iterations over the change to get a clean flow in the call.
/assign


Question: kubectl set resources deployment nginx --limits=cpu=0,memory=0 --requests=cpu=0,memory=0 # Print the result (in yaml format) of updating nginx container limits from a local, without hitting the server <del> kubectl set resources -f path/to/file.yaml --limits=cpu=200m,memory=512Mi --local -o yaml`) </del> <ins> kubectl set resources -f path/to/file.yaml --limits=cpu=200m,memory=512Mi --dry-run -o yaml`) </ins> ) // ResourcesOptions is the start of the data required to perform the operation. As new fields are added, add them here instead of

from https://k8s-
Really suspect this is
This is putting us into test-go post-commit run is broken
cc:
marked this p0 - if a fix isn't easy let's try to identify what introduced this and revert.
This flake on: 2016-10-30 07:19 PDT First instance: : It is flaking frequently, blocking PRs from getting merged, and slowing the submit queue ( at 75%). As on-call, I'm going to try and identify the cluprit and revert. Can use all help in doing so.
can you investigate whether your additional tests could cause this (by breaking the deployment of those pods, for instance) , Saad Ali wrote:
PR looks most suspect to me. It mucked with , and the errors we are seeing in each of the failures is related to : That PR was merged at Oct 30, 2016, 6:18 AM PDT and the first flake in was at Oct 30, 2016, 7:19 AM PDT. Thoughts? CC
I'm going to initiate a revert of to see if that fixes the issue
IMO we can just change the flaky test to not get pods but some other resources, such as deployments. We can't really expect to clean up pods immediately.
I'm going to proceed with revert (). Please feel free to send a follow up PR to handle this more gracefully.
Going to leave this opened until we have a few hours of runs with the revert. Want to verify, with some degree of certainty, that the revert took care of this flake.
We had 15 green runs of in a row. Considering this flake fixed.


Question: Spec: v1.PodSpec{ Containers: []v1.Container{ { <del> Name:  name, Image: "gcr.io/google_containers/busybox:1.24", VolumeMounts: []v1.VolumeMount{ { Name:      "tmpfs", MountPath: "/tmp", }, }, Command: cmd, }, }, RestartPolicy: v1.RestartPolicyNever, Volumes: []v1.Volume{ { Name:         "tmpfs", VolumeSource: v1.VolumeSource{HostPath: &v1.HostPathVolumeSource{Path: "/tmp"}}, </del> <ins> Name:      name, Image:     image, Lifecycle: lifecycle, </ins> }, }, },

The prestop and poststart exec tests volume mount /tmp from the host and attempt to create a file there from inside the pod. This is blocked by selinux. Failure:
we need to have a mechanism to say that a test should not be run if selinux is enabled (if legitimately required) or fix each test (where appropriate)


Question: gvr("", "v1", "pods"):                                                          `{"metadata": {"deletionTimestamp": "2020-01-01T00:00:00Z", "ownerReferences":[]}, "spec": {"containers": [{"image": "` + image2 + `", "name": "container7"}]}}`, gvr("", "v1", "replicationcontrollers"):                                        `{"spec": {"selector": {"new": "stuff2"}}}`, gvr("", "v1", "resourcequotas"):                                                `{"spec": {"hard": {"cpu": "25M"}}}`, <del> gvr("", "v1", "services"):                                                      `{"spec": {"externalName": "service2name"}}`, </del> <ins> gvr("", "v1", "services"):                                                      `{"spec": {"type": "ClusterIP"}}`, </ins> gvr("apps", "v1", "daemonsets"):                                                `{"spec": {"template": {"spec": {"containers": [{"image": "` + image2 + `", "name": "container6"}]}}}}`, gvr("apps", "v1", "deployments"):                                               `{"metadata": {"labels": {"a":"c"}}, "spec": {"template": {"spec": {"containers": [{"image": "` + image2 + `", "name": "container6"}]}}}}`, gvr("apps", "v1", "replicasets"):                                               `{"spec": {"template": {"spec": {"containers": [{"image": "` + image2 + `", "name": "container4"}]}}}}`,

xref I think this may be one of the few cases where we can actually change the behavior in a "breaking" way. User creates service with , is empty Controller or user updates the status subresource, sets (not wiped or validated, should be) User updates in any way, is wiped It's clear that the is not meant to be set when is not "LoadBalancer". The question is: 1) Fail validation? 2) Silently discard the value? 3) Warn and then discard the value? 4) Just warn and leave it. Because we allowed it in the past, but it is so volatile, I lean towards 3. and - the people most familair with this type. Edit: PrepareForUpdate() runs before warnings. So: (1) is easy (2) is easy (4) is slightly less easy than (2) (3) is slightly less easy than (4)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
OMG change it to clusterIP as expected Status is wiped update status as ClusterIP ... and if I edit the service back to loadbalancer it keeps the status??
/kind bug
/retitle Service can be set when !=
We talked about this one at the SIG Network meeting today. suggested we use the feature gate for deprecation option to phase the current (problematic) functionality out.
/triage accept
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: genericfeatures.APIResponseCompression: {Default: true, PreRelease: featuregate.Beta}, <ins> genericfeatures.APIServerIdentity: {Default: true, PreRelease: featuregate.Beta}, genericfeatures.APIServerTracing: {Default: true, PreRelease: featuregate.Beta}, genericfeatures.ConsistentListFromCache: {Default: false, PreRelease: featuregate.Alpha}, genericfeatures.CustomResourceValidationExpressions: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.31 genericfeatures.EfficientWatchResumption: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, </ins> genericfeatures.KMSv1: {Default: false, PreRelease: featuregate.Deprecated}, genericfeatures.KMSv2: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.31 genericfeatures.KMSv2KDF: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.31 <del> genericfeatures.ValidatingAdmissionPolicy: {Default: false, PreRelease: featuregate.Beta}, genericfeatures.CustomResourceValidationExpressions: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.31 </del> genericfeatures.OpenAPIEnums: {Default: true, PreRelease: featuregate.Beta}, <ins> genericfeatures.RemainingItemCount: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, </ins> genericfeatures.ServerSideApply: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.29 genericfeatures.ServerSideFieldValidation: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.29 <ins> genericfeatures.StorageVersionAPI: {Default: false, PreRelease: featuregate.Alpha}, genericfeatures.StorageVersionHash: {Default: true, PreRelease: featuregate.Beta}, </ins> genericfeatures.StructuredAuthenticationConfiguration: {Default: false, PreRelease: featuregate.Alpha}, genericfeatures.StructuredAuthorizationConfiguration: {Default: false, PreRelease: featuregate.Alpha}, genericfeatures.UnauthenticatedHTTP2DOSMitigation: {Default: true, PreRelease: featuregate.Beta}, <ins> genericfeatures.ValidatingAdmissionPolicy: {Default: false, PreRelease: featuregate.Beta}, genericfeatures.WatchBookmark: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, genericfeatures.WatchList: {Default: false, PreRelease: featuregate.Alpha}, </ins> genericfeatures.ZeroLimitedNominalConcurrencyShares: {Default: false, PreRelease: featuregate.Beta}, // inherited features from apiextensions-apiserver, relisted here to get a conflict if it is changed

This avoids drift between Kubernetes components and allows for cluster management tools that pass uniform feature gates to all components. xref:
/assign
Hey I f you stuck anywhere. Please free to involve me in this.


Question: return NewDelayingQueueWithCustomClock(clock.RealClock{}, "") } <ins> // NewDelayingQueueWithCustomQueue constructs a new workqueue with ability to // inject custom queue Interface instead of the default one func NewDelayingQueueWithCustomQueue(q Interface, name string) DelayingInterface { return newDelayingQueue(clock.RealClock{}, q, name) } </ins> // NewNamedDelayingQueue constructs a new named workqueue with delayed queuing ability func NewNamedDelayingQueue(name string) DelayingInterface { return NewDelayingQueueWithCustomClock(clock.RealClock{}, name)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Current constructors: Could we export a new constructor like to inject custom queue. Why is this needed: In my project, i want to create a with a custom queue based on workqueue.Interface but have no way to do that.
/sig api-machinery
thanks for the issue! We've reviewed it in the bug triage, and the consensus was that to ask you to send a PR with this and discuss it there. Would you be able to do it? thanks!
Yes, i will do that.


Question: echo "COPY nsswitch.conf /etc/" >> "${docker_file_path}" fi <del> # provide `--pull` argument to `docker build` if `KUBE_BUILD_PULL_LATEST_IMAGES` # is set to y or Y; otherwise try to build the image without forcefully # pulling the latest base image. local -a docker_build_opts=() if [[ "${KUBE_BUILD_PULL_LATEST_IMAGES}" =~ [yY] ]]; then docker_build_opts+=("--pull") fi "${DOCKER[@]}" build "${docker_build_opts[@]}" -q -t "${docker_image_tag}" "${docker_build_path}" >/dev/null </del> <ins> "${DOCKER[@]}" build ${docker_build_opts} -q -t "${docker_image_tag}" "${docker_build_path}" >/dev/null </ins> # If we are building an official/alpha/beta release we want to keep # docker images and tag them appropriately. local -r release_docker_image_tag="${KUBE_DOCKER_REGISTRY-$docker_registry}/${binary_name}-${arch}:${KUBE_DOCKER_IMAGE_TAG-$docker_tag}"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: when I run , found bound variable err. What you expected to happen: Build success How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ):Linux paasn1 4.14.78-300.el7.Centos.x8664 SMP Sat Jan 5 15:49:43 CST 2019 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others: GNU bash version 4.2.46(2)-release (x8664-redhat-linux-gnu) Copyright (C) 2011 Free Software Foundation, Inc.
/sig release


Question: t.Errorf("unexpected object: %v", rc2) t.FailNow() } <del> if test.expectPullPolicy != rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy { t.Errorf("expected ImagePullPolicy: %s, got: %s", test.expectPullPolicy, rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy, ) </del> <ins> if err := assertInitContainers(rc2.Spec.Template.Spec.InitContainers, test.expected, test.validators); err != nil { t.Errorf("test %v failed: %v", test.name, err) </ins> } } }

What happened: kubelet doesn't seem to be able to manage containers started with CRI-O any more I'm testing kubelet + CRI-O on an Ubuntu 18.04 server installation, inside a qemu/kvm VM managed by Vagrant (see ), in order to be able to test k8s + KubeVirt + KataContainers, which are all compatible with CRI-O kubelet will obsess on starting new containers, which looks like and endless process, and the cluster never gets ready. For instance here are some syslogs : What you expected to happen: It used to work (see asciinema recording at ) a few days ago, but I fear some recent changes broke it. How to reproduce it (as minimally and precisely as possible): Testing with vagrant up --provider=libvirt on Linux with a clone of that commit: vagrant ssh into the VM and ps -edf f or tail -f /var/log/syslog would do... Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: qemu/kvm VM +OS (e.g: ): Ubuntu 18.04.2 LTS Kernel (e.g. ): Linux kubernetes-vagrant-01 4.15.0-48-generic -Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm as in Network plugin and version (if this is a network-related bug): Others:
/sig node
I can confirm this. If I start cri-o from a systemd service it fails with the above mentioned error message. Starting crio with without systemd works just fine. systemd file:
Thanks for your detailed report :) Just want to do some quick version confirming - you've been using k8s 1.14.1 both when it did and did not work? Any other version changes?
Not sure for before, when it used to work, but according to the asciinema recording mentioned above (), it seems the same. However, not sure for CRI-O, underlying OS stuff (systemd, kernel, etc.). Sorry.
I was looking for some information for days. I have exactly the same issue when I use podman + systemd on 18.04.2.


Question: t.Errorf("unexpected object: %v", rc2) t.FailNow() } <del> if test.expectPullPolicy != rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy { t.Errorf("expected ImagePullPolicy: %s, got: %s", test.expectPullPolicy, rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy, ) </del> <ins> if err := assertInitContainers(rc2.Spec.Template.Spec.InitContainers, test.expected, test.validators); err != nil { t.Errorf("test %v failed: %v", test.name, err) </ins> } } }

It is the same for both 14.15 and 14.18 kernels. Podman without systemd runs fine. Podman with systemd fails at runc with the same error podman[]: open /dev/stderr: no such device or address podman[]: NAME: runc - Open Container Initiative runtime podman[]: runc is a command line client for running applications packaged according to ... podman[]: time="2019-05-11T13:04:59-07:00" level=error msg="open /dev/stderr: no such device or addressn" podman[]: open /dev/stderr: no such device or address
I've tried to reproduce it on 18.04.2 using Podman from but I am not able to reproduce it. Do you have any reproducer? How did you install Podman?
I used ceph unit files to run ceph in podman rather than docker. Systemd unit successfully starts docker but fails with podman. I can successfully run the same podman container run command from the command line. How were you trying to replicate this? Can you post your unit file?
Hi, I'am having the same problem with Kubernetes with CRI-O. After joining a second worker node it fails to start flannel. gives the following error: What I've tried: Installing previous version of Kubernetes (from to ) Installing previous version of crio (from to ) Building runc myself from their github page and selecting this in the Running it on Ubunut 18.04.2 Issue: (the one giuseppe linked)
I wonder if the issue is with systemd and runc. In both cases, crio and podman, I see runc help output which tells me that something is not being passed to runc that it needs.
I've only used so it is different than your test and you confirmed it works from the CLI for you as well. Could you share your service file so I test it?
I've tried systemd and cgroupfs but no luck so far.
so it seems that runc fails to re-open . That seems like a runc issue, as it doesn't really need to reopen it. To fix the failure, you could use something like:
Just an example.


Question: t.Errorf("unexpected object: %v", rc2) t.FailNow() } <del> if test.expectPullPolicy != rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy { t.Errorf("expected ImagePullPolicy: %s, got: %s", test.expectPullPolicy, rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy, ) </del> <ins> if err := assertInitContainers(rc2.Spec.Template.Spec.InitContainers, test.expected, test.validators); err != nil { t.Errorf("test %v failed: %v", test.name, err) </ins> } } }

I use ceph-ansible to install cluster so it may need to install additional daemons like mon before running mgr but the error, in this case, should be different, [Unit] Description=Ceph Manager [Service] EnvironmentFile=-/etc/environment ExecStartPre=-/usr/bin/podman stop ceph-mgr-storage2n1-la ExecStartPre=-/usr/bin/podman rm ceph-mgr-storage2n1-la ExecStart=/usr/bin/podman run --rm --net=host  --memory=16G  --cpus=4  -v /var/lib/ceph:/var/lib/ceph:z  -v /etc/ceph:/etc/ceph:z  -v /var/run/ceph:/var/run/ceph:z  -v /etc/localtime:/etc/localtime:ro  -e CLUSTER=ceph  -e CEPHDAEMON=MGR  -e CONTAINERIMAGE=  {{ cephmgrdockerextraenv }}  --name=ceph-mgr-storage2n1-la  ExecStopPost=-/usr/bin/podman stop ceph-mgr-storage2n1-la Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi- delighted <:
BTW, I projectatomic ppa and used apt-get -y podman to install. My podman version is 1.3.0 Sasha Litvak — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
Corrected file.


Question: t.Errorf("unexpected object: %v", rc2) t.FailNow() } <del> if test.expectPullPolicy != rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy { t.Errorf("expected ImagePullPolicy: %s, got: %s", test.expectPullPolicy, rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy, ) </del> <ins> if err := assertInitContainers(rc2.Spec.Template.Spec.InitContainers, test.expected, test.validators); err != nil { t.Errorf("test %v failed: %v", test.name, err) </ins> } } }

[Unit] Description=Ceph Manager [Service] EnvironmentFile=-/etc/environment ExecStartPre=-/usr/bin/podman stop ceph-mgr-storage2n1-la ExecStartPre=-/usr/bin/podman rm ceph-mgr-storage2n1-la ExecStart=/usr/bin/podman run --rm --net=host  --memory=16G  --cpus=4  -v /var/lib/ceph:/var/lib/ceph:z  -v /etc/ceph:/etc/ceph:z  -v /var/run/ceph:/var/run/ceph:z  -v /etc/localtime:/etc/localtime:ro  -e CLUSTER=ceph  -e CEPHDAEMON=MGR  -e CONTAINERIMAGE=  --name=ceph-mgr-storage2n1-la  ExecStopPost=-/usr/bin/podman stop ceph-mgr-storage2n1-la Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi- Giuseppe Scrivano <:
Would it have an effect system wide? Can I add it to a unit file or unit override instead? Giuseppe Scrivano <:
yes, I think it is better to add it only to the unit file
Why would runc work outside of systemd? And if it needs stderr, giving it Null in settings will redirect stderr to Null I presume? On Tue, May 14, 2019, 9:46 AM Giuseppe Scrivano <:
Running into same issues on Ubuntu 18.04.2 using cri-o-1.13 via PPA.
PR here: opencontainers/runc
Until it is avaialble with project atomic and such, can I use a null workaround? Giuseppe Scrivano <:
yes, but you will lose anything written to stderr. You could probably also use a file, like
What about actually mounting dev and using /dev/stderr ? On Tue, May 14, 2019, 3:11 PM Giuseppe Scrivano <:
is a symlink to . It is present, the issue here is that is a socket. The fix for runc got merged, could you use an updated runc?
I tried apt update/upgrade and restarting crio, but I'm still getting the issue.


Question: t.Errorf("unexpected object: %v", rc2) t.FailNow() } <del> if test.expectPullPolicy != rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy { t.Errorf("expected ImagePullPolicy: %s, got: %s", test.expectPullPolicy, rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy, ) </del> <ins> if err := assertInitContainers(rc2.Spec.Template.Spec.InitContainers, test.expected, test.validators); err != nil { t.Errorf("test %v failed: %v", test.name, err) </ins> } } }

Could it be that crio's runc is not updated yet? Edit Build new runc from and it is working now. Think crio hasn't updated it yet with their repository.
I'm running this version from PPA repo for ubuntu and it fails: could you point to version/commit or whatever helps identify the fixed version/snapshot ? Thanks in advance.
at the moment it is only fixed upstream. You'd need to build runc manually
Is there a version of the package on ppa we can downgrade to? On Wed, May 15, 2019, 4:35 AM Giuseppe Scrivano <:
could we update the runc version in the PPA repo?
Thank you. I seem them built but nut pushed to ppa yet.
should be available now.. let me know..
cri-o-runc (1.0.0-rc6-1~dev~ubuntu18.04~ppa42) bionic; urgency=medium autobuilt -- Lokesh Mandvekar (Bot) <, 15 May 2019 16:54:40 +0000
Thank you, everyone, for a quick turn around. Quick testing showed that fix is working, but more testing is needed. Lokesh Mandvekar <:
I confirm my initial tests with seem to confirm the regression is fixed. Many thanks
I also confirm the fix working for podman. Olivier Berger <:


Question: t.Errorf("unexpected object: %v", rc2) t.FailNow() } <del> if test.expectPullPolicy != rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy { t.Errorf("expected ImagePullPolicy: %s, got: %s", test.expectPullPolicy, rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy, ) </del> <ins> if err := assertInitContainers(rc2.Spec.Template.Spec.InitContainers, test.expected, test.validators); err != nil { t.Errorf("test %v failed: %v", test.name, err) </ins> } } }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): initContainers imagePullPolicy initContainers required value Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: cluster/kube- Others: What happened: When trying to create either a ReplicaSet or Replication Controller whose Pod template contains an Init Container, the validation fails saying is a required value. What you expected to happen: The create should succeed and the Init Container should receive a default . How to reproduce it (as minimally and precisely as possible): Try to create either of the following: ReplicaSet Replication Controller Anything else do we need to know: If I add to the Init Container in the ReplicaSet or Replication Controller Pod template, the create succeeds. When I create an Init Container without in a StatefulSet Pod template, it succeeds. When I create an Init Container without in a bare Pod, it succeeds:
Looks like defaulting not being applied on server? That is probably due to removing defaulting from convert. Surprised we didn't catch this earlier
Some values still not being defaulted
, I'm going to create a PR if none works on it.
We decode from bytes, set defaults and then convert; but for init containers, we delay the decode from "decode phase" to "convert phase" at .


Question: t.Errorf("unexpected object: %v", rc2) t.FailNow() } <del> if test.expectPullPolicy != rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy { t.Errorf("expected ImagePullPolicy: %s, got: %s", test.expectPullPolicy, rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy, ) </del> <ins> if err := assertInitContainers(rc2.Spec.Template.Spec.InitContainers, test.expected, test.validators); err != nil { t.Errorf("test %v failed: %v", test.name, err) </ins> } } }

The defaults has been handled for initContainer at . /cc I'm working to find a solution for that; if any suggestion, please let me know.
Would this bug affect the deletion of jobs with an init-container? I ran into this bug and adjusted my manifest to add the pull policy to the init-container. However, I now cannot delete said job claiming a field is immutable. Example:
, not sure now. Let me check it,
Same error here with version 1.5.1, 1.5.2 The Job "grafana-import-dashboards" is invalid: [0].imagePullPolicy: Required value With this setup: kubectl apply -f monitoring Not working even if I specify a pullpolicy in the yaml. If I roll back to 1.4.7, it is working. Thank you. Greg.
any updates on a PR to address this issue? Is anyone from able to pick this issue up otherwise?
, here's the latest status: there's is a PR to generate for each Object (), it works for Object (Pod, Job) now but List; I'm working to make it generating for List.
Why would you need List to work? No client side logic should be decoding lists except as a generic submission.
The List tests are failed in . There's no for List, so no code to decode init-container annotation, e.g. of should call 's recursively.
Thanks for the update
+1
Any update on this? I still getting this issue with the last version of 1.5.3 on GKE?
ping for the comments? should we merge the PR for 1.5?
, there's a PR () that handled that issue; but we're discussing not to merge it because initContainers had been moved to fields, and fields is the recommended way to use since 1.6. But I'd like to get & 's comments on 1.5.
This is still an issue in 1.5.3 when using annotations: Without the deployment would not be accepted
ping & :).
i dont think this is fixed in 1.5 I just encountered this bug on 1.5.3
, yes, the PR is NOT merged to any branch, so the issue is still here. I'll create a PR for 1.5.
Hi Anyone help this issue.


Question: t.Errorf("unexpected object: %v", rc2) t.FailNow() } <del> if test.expectPullPolicy != rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy { t.Errorf("expected ImagePullPolicy: %s, got: %s", test.expectPullPolicy, rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy, ) </del> <ins> if err := assertInitContainers(rc2.Spec.Template.Spec.InitContainers, test.expected, test.validators); err != nil { t.Errorf("test %v failed: %v", test.name, err) </ins> } } }

#cat mongo- apiVersion: v1 kind: Pod metadata: labels: name: mongo-svc-a name: mongo-deployment spec: replicas: 1 template: metadata: labels: name: mongo spec: containers: image: mongo name: mongo ports: name: mongo containerPort: hostPort: volumeMounts: mountPath: "/data/db" name: mongopv volumes: name: mongopv persistentVolumeClaim: claimName: mongopvc When i want create mongo db container by using this file folowing error is occured. The Pod "mongo-deployment" is invalid:spec.containers: Required value
, it seems there're two here, please remove the first one and have a try.
Hi my kubectl binary is configured at ubuntu 16.04 machine.


Question: t.Errorf("unexpected object: %v", rc2) t.FailNow() } <del> if test.expectPullPolicy != rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy { t.Errorf("expected ImagePullPolicy: %s, got: %s", test.expectPullPolicy, rc2.Spec.Template.Spec.InitContainers[0].ImagePullPolicy, ) </del> <ins> if err := assertInitContainers(rc2.Spec.Template.Spec.InitContainers, test.expected, test.validators); err != nil { t.Errorf("test %v failed: %v", test.name, err) </ins> } } }

NAME LABELS STATUS 10.1 Ready,SchedulingDisabled 10.2 10.1.10.2 Ready 10.3 Ready apiVersion: v1 kind: Node metadata: creationTimestamp: 2017-03-03T14:41:52Z labels: 10.1.10.1 name: 10.1.10.1 resourceVersion: "" selfLink: /api/v1/nodes/10.1.10.1 uid: -001f-11e7-8e83- spec: externalID: 10.1.10.1 status: addresses: address: 10.1.10.1 type: LegacyHostIP address: 10.1.10.1 type: InternalIP capacity: cpu: "2" memory: 4041352Ki pods: "110" conditions: lastHeartbeatTime: 2017-04-05T18:58:42Z lastTransitionTime: 2017-03-03T14:41:52Z message: kubelet has sufficient disk space available reason: KubeletHasSufficientDisk status: "False" type: OutOfDisk lastHeartbeatTime: 2017-04-05T18:58:42Z lastTransitionTime: 2017-04-05T15:27:52Z message: kubelet is posting ready status reason: KubeletReady status: "True" type: Ready nodeInfo: bootID: -6124-4ac7-ab96- containerRuntimeVersion: docker://1.12.6 kernelVersion: 4.9.16-coreos-r1 kubeProxyVersion: v1.2.4+coreos.1 kubeletVersion: v1.2.4+coreos.1 machineID: osImage: Container Linux by CoreOS 1298.7.0 (Ladybug) systemUUID: C3FD9852-2DFB-21F4-CE07-2597A746D99D When i create pv for creation pod is shows pv is pending. Please give any suggestions.....
Is there solution of this problem with version 2.7. Still facing the same. what exactly expectation of helm upgrade --install relname chartsdirectory is is expected to work on first time installation in case with CI/CD Pipelines.


Question: package dockertools import ( <del> "bufio" </del> "bytes" "errors" "fmt"

As far as I can tell, container liveness probes of type exec flat out don't work against clusters built at head. I had the kubelet always log the output of the exec'ed command, and it's always empty no matter what I exec, causing the probe to fail since it requires the output to be equal to the string "ok". If you want to test it, try adding a liveness probe of this form to any container: I also tried these lovely forms, to more closely match the liveness example under examples/: On the plus side, this may be an opportunistic time to switch over to
The reason that the e2e test for exec liveness probes hasn't caught this is that it only tests that containers are restarted when they fail liveness probes. It doesn't test that containers aren't restarted when their liveness checks should be succeeding. The e2e test's exec'ed commands also only ever produced empty output, rather than the expected "fail" output.
Is it possible to include this in a new pre-release, since it fixes a bug in health checking? I see it merged in master and I'm wondering what the policy is of releasing new code in a separate (0.16.*) release.


Question: "github.com/golang/glog" "k8s.io/api/core/v1" <del> clientv1 "k8s.io/api/core/v1" </del> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" "k8s.io/client-go/tools/record" runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/v1alpha1/runtime" <del> "k8s.io/kubernetes/pkg/kubelet/events" </del> "k8s.io/kubernetes/pkg/kubelet/util/format" "k8s.io/kubernetes/pkg/kubelet/util/ioutils" hashutil "k8s.io/kubernetes/pkg/util/hash"

Is this a BUG REPORT or FEATURE REQUEST?: What happened: Now the and has been consistent in by PR . There is no need to convert v1.ObjectReference to clientv1.ObjectReference in . What you expected to happen: Remove this function and cleanup the codes which call this function. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Will submit the cleanup after PR merged.
/sig api-machinery
/sub


Question: // Score is a list of plugins that should be invoked when ranking nodes that have passed the filtering phase. Score *PluginSet `json:"score,omitempty"` <del> // NormalizeScore is a list of plugins that should be invoked after the scoring phase to normalize scores. NormalizeScore *PluginSet `json:"normalizeScore,omitempty"` </del> // Reserve is a list of plugins invoked when reserving a node to run the pod. Reserve *PluginSet `json:"reserve,omitempty"`

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Consolidate ) into Score plugin set in config API. Why is this needed: NormalizeScore extension point is implemented in . We decided to treat NormalizeScore as an "optional" step of Score plugin, see . However, we deferred the removal of due to API review delays. Now that PR was merged. We can start the actual removal from the API and start the API review process. /sig scheduling
/assign


Question: runtime.ReallyCrash = s.ReallyCrashForTesting rand.Seed(time.Now().UTC().UnixNano()) <del> credentialprovider.SetPreferredDockercfgPath(s.RootDirectory) glog.V(2).Infof("Using root directory: %v", s.RootDirectory) </del> // TODO(vmarmol): Do this through container config. oomAdjuster := kcfg.OOMAdjuster if err := oomAdjuster.ApplyOOMScoreAdj(0, int(s.OOMScoreAdj)); err != nil {

During the startup process of kubelet, we have Since we have following Code segment 1 and 2 seems dup to me. Am i getting this wrong?