Question: clientset "k8s.io/client-go/kubernetes" kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm" kubeadmconstants "k8s.io/kubernetes/cmd/kubeadm/app/constants" <ins> "k8s.io/kubernetes/cmd/kubeadm/app/features" </ins> kubeadmutil "k8s.io/kubernetes/cmd/kubeadm/app/util" "k8s.io/kubernetes/cmd/kubeadm/app/util/apiclient" "k8s.io/kubernetes/pkg/api/legacyscheme"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently, IPVS proxier does not run any e2e test cases. What you expected to happen: I think it's important to create test suite for IPVS in upstream CI which will help a lot in moving IPVS existing alpha. Anything else we need to know?: Let's work together. /sig network
Self /assign
There are to ways to do this work: , but doesn't support any kube-proxy config right now, so I had this pr , and also, need support it too, see this pr: kubernetes/kubernetes-anywhere build cluster. See this pr: kubernetes/test-infra The second way is much easier and faster, but I was told they gonna deprecated the , so in the long term, we still need and 's support. If anybody have better way or have some suggestion about this, please let me know, I'll be really appreciate, Thanks.


Question: } func (flunderStrategy) NamespaceScoped() bool { <del> return false </del> <ins> return true </ins> } func (flunderStrategy) PrepareForCreate(ctx genericapirequest.Context, obj runtime.Object) {

1 tests failed, 11 jobs failed, 241 builds failed. Failure stats cover 1 day time range '17 Aug 2017 22:57 UTC' to '18 Aug 2017 22:57 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Seeing this failure on the e2e release-master-blocking https://k8s- Example failure https://k8s- Looks like an RBAC issue cc
this is currently blocking the alpha.3 release
looks like this new test was by in which was merged about 6 days ago and is failing consistently
This has started failing again on our GKE test suite https://k8s- https://k8s- cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/api-machinery: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
Seems to be flaking now with the error: https://k8s-
- we need to determine (today if possible) if this is truly release blocking. If so, please add the release-blocker label. And, if not, how do we best continue work on this for 1.8.x/1.9.0?
this test has two passes and four failures on the same commit I'm seeing gke-specific authz grants in that test that are incorrect: That commit removed the wait for the authz grant to become effective (which can lead to flakes), and granted superuser permissions to all users, which is incorrect and invalidates any other authz-related tests run in parallel with this test
cc
I can add the wait back.


Question: } func (flunderStrategy) NamespaceScoped() bool { <del> return false </del> <ins> return true </ins> } func (flunderStrategy) PrepareForCreate(ctx genericapirequest.Context, obj runtime.Object) {

edit: and fix the test given admin to all authenticated users.
Actually after staring at this test for about half an hour I can't figure out what different users exist or what permissions they're being granted. ClientSet, InternalClientset, and AggregatorClient are all initialized from the same config so I don't see how one would be able to create an RBAC binding but another would fail later. any thoughts here?
I honestly think the gke specific BindClusterRole is a red-herring. It is needed so the client has permission to perform one of the setup steps. (I think it was either to create the wardler cluster role or to bind that role to the anonymous user) Once that setup step is complete we no longer need the that cluster role bound and so I don't think its related.
The gke authorizer allows the “bind” verb, so the client can create a binding to the cluster-admin. It cannot create a role directly unless it has permissions via RBAC. Since we don’t have a way to determine the username associated with , binding to all authenticated users is what was done as a workaround.
I agree that the point at which the tests are failing indicate that the previous authorization issues are not the cause.
/open
/reopen
So a lot more information to work with now but the error is still occurring. I am still looking into this.
any update on the investigation?
Friendly v1.8 release team ping. This failure still seems to be happening, is this actively being worked? Does this need to be in the v1.8 milestone?


Question: return rs.releaseClusterIPs(toRelease) } <ins> // tests if two preferred dual-stack service have matching ClusterIPFields // assumption: old service is a valid, default service (e.g., loaded from store) func isMatchingPreferDualStackClusterIPFields(oldService, service *api.Service) bool { if oldService == nil { return false } if service.Spec.IPFamilyPolicy == nil { return false } // if type mutated then it is an update // that needs to run through the entire process. if oldService.Spec.Type != service.Spec.Type { return false } // both must be type that gets an IP assigned if service.Spec.Type != api.ServiceTypeClusterIP && service.Spec.Type != api.ServiceTypeNodePort && service.Spec.Type != api.ServiceTypeLoadBalancer { return false } if oldService.Spec.Type != api.ServiceTypeClusterIP && oldService.Spec.Type != api.ServiceTypeNodePort && oldService.Spec.Type != api.ServiceTypeLoadBalancer { return false } // both must be of IPFamilyPolicy==PreferDualStack if service.Spec.IPFamilyPolicy != nil && *(service.Spec.IPFamilyPolicy) != api.IPFamilyPolicyPreferDualStack { return false } if oldService.Spec.IPFamilyPolicy != nil && *(oldService.Spec.IPFamilyPolicy) != api.IPFamilyPolicyPreferDualStack { return false } // compare ClusterIPs lengths. // due to validation. if len(service.Spec.ClusterIPs) != len(oldService.Spec.ClusterIPs) { return false } for i, ip := range service.Spec.ClusterIPs { if oldService.Spec.ClusterIPs[i] != ip { return false } } // compare IPFamilies if len(service.Spec.IPFamilies) != len(oldService.Spec.IPFamilies) { return false } for i, family := range service.Spec.IPFamilies { if oldService.Spec.IPFamilies[i] != family { return false } } // they match on // Policy: preferDualStack // ClusterIPs // IPFamilies return true } </ins> // attempts to default service ip families according to cluster configuration // while ensuring that provided families are configured on cluster. <del> func (rs *REST) tryDefaultValidateServiceClusterIPFields(service *api.Service) error { </del> <ins> func (rs *REST) tryDefaultValidateServiceClusterIPFields(oldService, service *api.Service) error { </ins> // can not do anything here if service.Spec.Type == api.ServiceTypeExternalName { return nil

1) start with a single stack cluster 2) create a PreferDualStack service 3) configure API server with a 2nd service CIDR 4) Confirm the above service is not dual-stack 5) Add a label to the above service 6) Observe that the service becomes dual-stack I think the correct answer is to have a "repair loop" that upgrades (and downgrades!) PreferDualStack Services to match the config. I think this is a GA blocker.
/triage accepted The current thought is: of type with are special new breed of api objects that I don't think we have any similar precedent in the apis, i could be wrong. Their behavior is tied to how the cluster is configured. Thus their behavior must change as the cluster behavior change. a way the user is stating . If it is cluster then i am good with two ClusterIPs. If it is single stack cluster then I am also good with a single ClusterIP. One can assume that if the user was ok was with getting a additional IP when possible then they must be ok with taking it away when it is no longer possible. Based on the above I believe best course of action is create a new repair loop or piggyback on any of the existing (or better: fix it in rest stack during list and get calls). The new logic takes care of matching to the .
reproducer and regression test if somebody takes this, feel free to add it
ack - Scheduled to tackle 3rd week of May. Will definitely include that patch.


Question: } } <del> func TestHistogramClear(t *testing.T) { samples := []float64{0.5, 0.5, 0.5, 0.5, 1.5, 1.5, 1.5, 1.5, 3, 3, 3, 3, 6, 6, 6, 6} bounds := []float64{1, 2, 4, 8} h := samples2Histogram(samples, bounds) if *h.SampleCount == 0 { t.Errorf("Expected histogram .SampleCount to be non-zero") } if *h.SampleSum == 0 { t.Errorf("Expected histogram .SampleSum to be non-zero") } for _, b := range h.Bucket { if b.CumulativeCount != nil { if *b.CumulativeCount == 0 { t.Errorf("Expected histogram bucket to have non-zero comulative count") } } } h.Clear() if *h.SampleCount != 0 { t.Errorf("Expected histogram .SampleCount to be zero, have %v instead", *h.SampleCount) } if *h.SampleSum != 0 { t.Errorf("Expected histogram .SampleSum to be zero, have %v instead", *h.SampleSum) } for _, b := range h.Bucket { if b.CumulativeCount != nil { if *b.CumulativeCount != 0 { t.Errorf("Expected histogram bucket to have zero comulative count, have %v instead", *b.CumulativeCount) } } if b.UpperBound != nil { *b.UpperBound = 0 } } } </del> func TestHistogramValidate(t *testing.T) { tests := []struct { name string

In the new scheduler performance testing framework, (bool) is provided to benchmark performance of pods created in an "operator": (L11 below) However, the initialized Pods created in L5~L7 seem to be also benchmarked, although it doesn't specify . (please see the reproduce steps below). I think it's b/c once scheduler starts, the associated Prometheus artifacts are also up, and it's a single in-memory datastore. So no matter we start to collect the metric in which step, the prometheus data is already up-to-date. In the above workload config, the final data have benchmarked 1500 pods instead of 1000 pods. I guess it won't work either if running in the pod initialization step or not. Unless we manage a breakpoint at the end of each step, and calculate delta histogram for the step that specifies . Pods created in step without are not benchmarked. Apply the following diff: And then run: You will get an output sth. like: /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Could you please help confirm if it's an issue?
cc
Unfortunately, does not apply for metrics gathered from . As you mentioned. The default gatherer collects all metrics produced by the scheduler. It's the scheduler itself who knows pod specs and would have to decide which pods will get ignored when updating metrics. The question is if computing metrics for a subset of pods processed by the scheduler is a valid use case.
This is probably what we want. Is something like that possible ?
I think being able to absolutely ignore certain pods for metrics is an overkill.
The use case is to ignore the pods created in steps that don't specify , so as to collect the Pod metrics create at particular step(s) (). Otherwise, the semantics of option is quite misleading. Nope, that's not the intention. We don't need to target certain pods, like using label selector or whatever. Just as I mentioned, measure pods created at particular steps.


Question: } } <del> func TestHistogramClear(t *testing.T) { samples := []float64{0.5, 0.5, 0.5, 0.5, 1.5, 1.5, 1.5, 1.5, 3, 3, 3, 3, 6, 6, 6, 6} bounds := []float64{1, 2, 4, 8} h := samples2Histogram(samples, bounds) if *h.SampleCount == 0 { t.Errorf("Expected histogram .SampleCount to be non-zero") } if *h.SampleSum == 0 { t.Errorf("Expected histogram .SampleSum to be non-zero") } for _, b := range h.Bucket { if b.CumulativeCount != nil { if *b.CumulativeCount == 0 { t.Errorf("Expected histogram bucket to have non-zero comulative count") } } } h.Clear() if *h.SampleCount != 0 { t.Errorf("Expected histogram .SampleCount to be zero, have %v instead", *h.SampleCount) } if *h.SampleSum != 0 { t.Errorf("Expected histogram .SampleSum to be zero, have %v instead", *h.SampleSum) } for _, b := range h.Bucket { if b.CumulativeCount != nil { if *b.CumulativeCount != 0 { t.Errorf("Expected histogram bucket to have zero comulative count, have %v instead", *b.CumulativeCount) } } if b.UpperBound != nil { *b.UpperBound = 0 } } } </del> func TestHistogramValidate(t *testing.T) { tests := []struct { name string

I would do something like the following: if a step has collectMetrics:false, we wait for them to finish scheduling, and then we take a snapshot of the histogram. Then for the next step that has collectMetrics:true, we take the diff with the latest snapshot to report the results. This might not be perfect, as we don't control if the metrics have finished syncing when we take the snapshot. But I think that's fine. Would doing this be possible? Now, if we have steps that run in parallel, one with collectMetrics:false and another with collectMetrics:true, then this doesn't work. Note that I don't know the details of how the gatherer works.
That adheres to what I have in my mind - we can only manage this on client side. I think it's fine. As we are already fetching the "snapshot" when pods get scheduled when equals true. True. An alternative is to design the case to be like: the step with to employ Pods that are all unschedulable, and then specify a custom metric "schedulere2eschedulingdurationseconds[result="unschedulable"]" to target a particular histogram instead of all histograms (default option) inside a histogramVec. Note this can only be possible when and some subsequent changes get in place.
We might just call Histogram.Clear() method which will clear what has been collected so far at the end of phase: Assuming individual phases have no pods in common and each phase can clear metrics without affecting other phases. Though, as said, it will work only for serial case.
WDYT
I'm afraid doesn't work. That was my initial thought as well, but no luck. I guess it's b/c, theoretically, we cannot reset a metric on the server side - Promoetheus' model doesn't support this. But let me know if I miss anything.
The performance tests assume all the metrics all gathered and accessed in the same binary. As long as that's the case, works.


Question: } } <del> func TestHistogramClear(t *testing.T) { samples := []float64{0.5, 0.5, 0.5, 0.5, 1.5, 1.5, 1.5, 1.5, 3, 3, 3, 3, 6, 6, 6, 6} bounds := []float64{1, 2, 4, 8} h := samples2Histogram(samples, bounds) if *h.SampleCount == 0 { t.Errorf("Expected histogram .SampleCount to be non-zero") } if *h.SampleSum == 0 { t.Errorf("Expected histogram .SampleSum to be non-zero") } for _, b := range h.Bucket { if b.CumulativeCount != nil { if *b.CumulativeCount == 0 { t.Errorf("Expected histogram bucket to have non-zero comulative count") } } } h.Clear() if *h.SampleCount != 0 { t.Errorf("Expected histogram .SampleCount to be zero, have %v instead", *h.SampleCount) } if *h.SampleSum != 0 { t.Errorf("Expected histogram .SampleSum to be zero, have %v instead", *h.SampleSum) } for _, b := range h.Bucket { if b.CumulativeCount != nil { if *b.CumulativeCount != 0 { t.Errorf("Expected histogram bucket to have zero comulative count, have %v instead", *b.CumulativeCount) } } if b.UpperBound != nil { *b.UpperBound = 0 } } } </del> func TestHistogramValidate(t *testing.T) { tests := []struct { name string

We already call in the code:
That was my understanding as well, but no luck :( Here is a diff to call in case when "concreteOp.CollectMetrics" is false: <details<summaryDiff:</summary<p</p</detailsAnd below is the output - it's still 1500 (500 init pods plus 1000 measured pods) instead of 1000 being benchmarked: <details<summaryOutput:</summary<p</p</details
Also, if you run the tests and together, the result proves doesn't work as expected:
But I do find a way to fix this: .


Question: "net/http" "path" "path/filepath" <ins> "runtime" </ins> "sync" "time"

What happened: I have a Kubernetes Cluster with 2 Windows Server 2019 Core VMs as worker nodes. I upgraded first the cluster, and then the first Windows node following the docs here: Every step ran without issue, but the kubelet service was not started. This was not immediately obvious, since "restart-service kubelet" does not propagate the error. I think "start-service kubelet" would be better advice. The logs said: After changing the value of cgroupDriver manually in from "systemd" to "cgroupfs", the kubelet started successfully. What you expected to happen: One of 2 things: Either the Windows upgrade docs telling me what I need to do to fix this, or the kubeadm not breaking the configuration for windows in the first place. How to reproduce it (as minimally and precisely as possible): a windows node to a Kubernetes v1.19 cluster via kubeadm the cluster via kubeadm the Windows node following the docs Anything else we need to know?: Environment: Kubernetes version (use ): 1.20.1 Cloud provider or hardware configuration: OS (e.g: ): Windows Server 2019 Core Kernel (e.g. ): Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others: /sig windows
this may cause by the default value of defaultCgroupDriver cgroupfs Would you provide the from Windows node?
/assign
kubeadm doesn't do anything about cgroup drivers on Windows because cgroups don't exist on Windows. ideally the kubelet checks about this should be skipped entirely in dockershim for non-Linux.
! However, windows docker return as cgroupfs.
probably 's output is the same structure on all OSes with being a default value. this field should not be checked on Windows at all.
/retitle kubelet: don't check for the Docker cgroup driver on Windows
adding since the PR is favorable. /triage accepted


Question: # Cluster Role # ################ <ins> kubectl "${kube_flags[@]}" api-resources </ins> if kube::test::if_supports_resource "${clusterroles}" ; then record_command run_clusterroles_tests fi

As the RBAC role need to be related to resources. I think we could use a command to get the supported resources. ref to
This would be cool.
I thought we a command that listed resources in the discover doc a long time ago.
is just the versions, but principle is the same. We want to list all the types, their resources, and tie that back to "explain".
What's the best way to determine this information currently? I'm having difficulty figuring out the exact set of permissions (and how they would be written in an RBAC role) for any given kubectl command. The documentation for this right now is split across multiple places and certain things are not explained at all. Specifically, it's difficult to determine which "API verb" is used for which operations. Take as an example. It's unclear if the way to allow this is to use the "proxy" subresource of a pod or service with "get"/"create"/etc. verbs, or to use the "proxy" verb with the resource (pod, service) directly. is a similar case. The almost works for this purpose, but it's still difficult. Operations for a given resource are grouped into categories like write, read, status, and proxy, but these things don't obviously correlate with API verbs. Again, proxy is especially confusing as there is no mention of a proxy "verb". The API operations are only shown using their HTTP verbs. One thing that could help on this document would be a section on the right column under "kubectl command" that shows exactly the rule needed in an RBAC role to allow this command via kubectl. And even then, that assumes an exact 1-to-1 mapping between API endpoints and kubectl commands, which I doubt is always the case. (For example, port fowarding to a pod seems to require both a GET on the pod resource and a CREATE on a port forward subresource.) There also needs to be better documentation about how this information could have been determined in the first place. I've resorted to manually looking through API requests made with but even then it's difficult to figure out. Amusingly, I was the one who wrote which is referenced in the original issue description (as well as the table explaining verbs in ).


Question: # Cluster Role # ################ <ins> kubectl "${kube_flags[@]}" api-resources </ins> if kube::test::if_supports_resource "${clusterroles}" ; then record_command run_clusterroles_tests fi

tl;dr: It's too difficult right now to go from "I want a user to be able to run " to "here is the RBAC rule to use for that" for any case where there isn't a specific example shown in the docs.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
It seem that relative PR stop updated. If this issue still valuable, i will help fix it
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: {Type: v1.NodeMemoryPressure, Status: v1.ConditionFalse}, {Type: v1.NodeDiskPressure, Status: v1.ConditionFalse}, {Type: v1.NodeNetworkUnavailable, Status: v1.ConditionFalse}, <del> {Type: v1.NodeInodePressure, Status: v1.ConditionFalse}, </del> } return node }(),

Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: It seems kubelet did not update inode pressure; and only test codes are using . So prefer to deprecate or remove it directly. (seems no node-misc group :). )


Question: return dswp, fakePodManager } <ins> func TestFindAndAddNewPods_WithRescontructedVolume(t *testing.T) { // create dswp dswp, fakePodManager := prepareDswpWithVolume(t) // create pod fakeOuterVolumeName := "dswp-test-volume-name" containers := []v1.Container{ { VolumeMounts: []v1.VolumeMount{ { Name:      fakeOuterVolumeName, MountPath: "/mnt", }, }, }, } pod := createPodWithVolume("dswp-test-pod", fakeOuterVolumeName, "file-bound", containers) fakePodManager.AddPod(pod) podName := util.GetUniquePodName(pod) mode := v1.PersistentVolumeFilesystem pv := &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: fakeOuterVolumeName, }, Spec: v1.PersistentVolumeSpec{ ClaimRef:   &v1.ObjectReference{Namespace: "ns", Name: "file-bound"}, VolumeMode: &mode, }, } generatedVolumeName := "fake-plugin/" + pod.Spec.Volumes[0].Name uniqueVolumeName := v1.UniqueVolumeName(generatedVolumeName) expectedOuterVolumeName := "dswp-test-volume-name" opts := operationexecutor.MarkVolumeOpts{ PodName:             podName, PodUID:              pod.UID, VolumeName:          uniqueVolumeName, OuterVolumeSpecName: generatedVolumeName, // fake reconstructed volume VolumeGidVolume:     "", VolumeSpec:          volume.NewSpecFromPersistentVolume(pv, false), VolumeMountState:    operationexecutor.VolumeMounted, } dswp.actualStateOfWorld.MarkVolumeAsAttached(opts.VolumeName, opts.VolumeSpec, "fake-node", "") dswp.actualStateOfWorld.MarkVolumeAsMounted(opts) dswp.findAndAddNewPods() mountedVolumes := dswp.actualStateOfWorld.GetMountedVolumesForPod(podName) found := false for _, volume := range mountedVolumes { if volume.OuterVolumeSpecName == expectedOuterVolumeName { found = true break } } if !found { t.Fatalf( "Could not found pod volume %v in the list of actual state of world volumes to mount.", expectedOuterVolumeName) } } </ins> func TestFindAndAddNewPods_WithReprocessPodAndVolumeRetrievalError(t *testing.T) { // create dswp dswp, fakePodManager := prepareDswpWithVolume(t)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --We have even encountered such case that pod continuously reports “Unable to attach or mount volumes for pod;...” when kubelet restarted. The reasons are listed in details as below: , as result of apiserver instantaneous unstablility, dswp get pvc‘s pv spec failed, then the pvc volume does not be to dsw. , syncStates() lists all the volume directories locally, and recontructs the volume from local volume, if the volume isn't in dsw, it will put the recontructed volume to asw, but now the volume.OuterVolumeSpecName isn't correct. chance, at this time, In , the fail to pvc volume is to dsw again normally in reconcile() -mountAttachVolumes(), it compares dsw and asw by volume's innerVolumename, it's all the same. but it could not sense the OuterVolumeSpecName dismatch. so it will not re-mount again to update asw cache , it checks pod's volume attach or mount success by comparing volume.OuterVolumeSpecName and , but this pv'c OuterVolumeSpecName isn't correct in asw. so pod continuously reports “Unable to attach or mount volumes for pod;...” reconcile() could sense the reconstruct volume's dismatch to the dsw, then re-mount again to make volume cache consistency Kubernetes version (use ): 1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
CC
I think this is a point that can be optimized. /cc
I will give a fix for this
Thanks for reporting the issue, this innerSpecName and outerSpecName are confusing and cause issues. Please ping me for review when you have a fix. Thanks!
Please don't triage your own issues immediately, give someone else a chance to review them :( /sig storage
okay, I know


Question: // +k8s:deepcopy-gen=package <del> package testing </del> <ins> package testing // import "k8s.io/apiserver/pkg/storage/testing" </ins>

Look at one of the directories under (Please don't pick more than one directory for one PR as it will cause issues for reviewers) Find all the files named (example find -name ) See the following link it has a this tells tools how this package needs to be imported. So we should ensure that each has an import Add a description appropriate to the package (similar to the one below) Look through sub directories and see if there are any other important directories that are likely to be used by outside users and add a Why are we doing this? We perfer to use vanity urls instead of directly using to refer to code, so this helps ensure that tools honor our wish. See It would be good to have some package level documentation since these directories under staging/ are meant to be for use by folks outside of kubernetes github org. /good-first-issue
/kind documentation /sig architecture
currently working my through Not going to use / assign since there could be a number of PRs for this issue.
/lifecycle active /good-first-issue edit: hmm..
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /lifecycle active /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
May I try on
IIUC this means that there needs to be a even at the root of the repo? Then how does work? :thinking:
Hi , can I make a PR for this directory ?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: // +k8s:deepcopy-gen=package <del> package testing </del> <ins> package testing // import "k8s.io/apiserver/pkg/storage/testing" </ins>

/lifecycle rotten
I guess this is still not done, was supposed to partially fix this issue. /reopen
Reopened this issue. <detailsIn response to : I guess this is still not done, was supposed to partially fix this issue. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /area code-organization


Question: <del> kind: ReplicationController apiVersion: v1 </del> <ins> apiVersion: extensions/v1beta1 kind: Deployment </ins> metadata: name: etcd <del> creationTimestamp: </del> spec: strategy: type: Recreate <del> resources: {} triggers: - type: ConfigChange </del> replicas: 3 selector: <del> name: etcd </del> <ins> matchLabels: name: etcd </ins> template: metadata: <del> creationTimestamp: </del> labels: name: etcd spec:

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Vagrant OS (e.g. from /etc/os-release): CoreOS Kernel (e.g. ):Install tools: Vagrant & Others: What happened: What you expected to happen: Success How to reproduce it (as minimally and precisely as possible): Followed instructions at Tried to run Received: Anything else do we need to know: From googling, it seems several of the fields in the config are for when using OpenShift origin. But creating the replication controller is a prerequisite to launching OpenShift Origin. Chicken & egg problem.


Question: // or the kubeadm preferred one for the desired Kubernetes version var desiredEtcdVersion *version.Version if cfg.Etcd.Local.ImageTag != "" { <del> desiredEtcdVersion, err = version.ParseSemantic(cfg.Etcd.Local.ImageTag) </del> <ins> desiredEtcdVersion, err = version.ParseSemantic( convertImageTagMetadataToSemver(cfg.Etcd.Local.ImageTag)) </ins> if err != nil { return true, errors.Wrapf(err, "failed to parse tag %q as a semantic version", cfg.Etcd.Local.ImageTag) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When attempting to upgrade kubernetes using kubeadm upgrade apply, kubeadm fails to parse an etcd imagetag containing build metadata in the form of as a valid semver. An example upgrade produces an error similar to: Note this will occur even if the etcd version is unchanged. The +` character. This pattern is used to support custom kubernetes images, for instance kube-apiserver:v1.20.4+patch.0 will be converted to imagetag v1.20.4_patch.0 when running make quick-release-images. Add the following to kubeadm- Install kubernetes. Then upgrade kubernetes with kubeadm upgrade apply. I have tested a fix for this issue and will PR shortly. Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: kubeadm Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig cluster-lifecycle
/area kubeadm
/triage accepted


Question: // Cannot be updated. // +optional WorkingDir string `json:"workingDir,omitempty" protobuf:"bytes,5,opt,name=workingDir"` <del> // List of ports to expose from the container. Exposing a port here gives // the system additional information about the network connections a // container uses, but is primarily informational. Not specifying a port here </del> <ins> // List of ports to expose from the container. Not specifying a port here </ins> // DOES NOT prevent that port from being exposed. Any port which is // listening on the default "0.0.0.0" address inside a container will be // accessible from the network. <ins> // Modifying this array with strategic merge patch may corrupt the data. // For more information See https://github.com/kubernetes/kubernetes/issues/108255. </ins> // Cannot be updated. // +optional // +patchMergeKey=containerPort

The comments in says that : However there are several cases where this is false : When using a Service with a targetPort that references a name in the ports arrays of the pod. Some software (ab)uses this API to get which ports are used (See Although unfixable, my goal is to fix the documentation so other people don't have the same problem). The Azure Load Balancer implementation does this for instance. Some other network/security software might use this information. Also, is it possible to document/reference the above issue in the comment ? Since it is described as informational I expected for the network to work without this information. But it's definitely not the case. Not needed since it's mostly a doc issue. I'm interested in contributing this patch but would like some feedbacks before doing it. :) <details</details<details</details<details</details<details</details<details</details<details</details
/sig network
/triage accepted -- thoughts? I believe the doc are due for a refresh.
I'd take this one. /assign
I didn't see that you wanted to do this patch, I am very sorry and now I unassign it. /unassign
No worries But I'm still not sure where / what to add. Should I only fix the part saying it is only informational ? Is there anything else ? (cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: const ( // StatusSuccess represents the successful completion of command. StatusSuccess = "Success" <del> // StatusFailed represents that the command failed. StatusFailure = "Failed" </del> // StatusNotSupported represents that the command is not supported. StatusNotSupported = "Not supported" )

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): statusFailure Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 4 Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: Others What happened: In FlexVolume code: The statusFailure constant should have string literal "Failure", not "Failed". Because in the test code or documentation everywhere, only "Failure" string is used, not "Failed" for the flexvolume cmd output status string. So this is a typo bug. Here is the test: here is the documentation: here is the example: What you expected to happen: The statusFailure constant should have string literal "Failure", not "Failed" at this line: How to reproduce it (as minimally and precisely as possible): code inspect: Anything else do we need to know: nothing else


Question: FlagTimeout      = "request-timeout" ) <ins> // RecommendedConfigOverrideFlags is a convenience method to return recommended flag names prefixed with a string of your choosing func RecommendedConfigOverrideFlags(prefix string) ConfigOverrideFlags { return ConfigOverrideFlags{ AuthOverrideFlags:    RecommendedAuthOverrideFlags(prefix), ClusterOverrideFlags: RecommendedClusterOverrideFlags(prefix), ContextOverrideFlags: RecommendedContextOverrideFlags(prefix), CurrentContext: FlagInfo{prefix + FlagContext, "", "", "The name of the kubeconfig context to use"}, Timeout:        FlagInfo{prefix + FlagTimeout, "", "0", "The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests."}, } } </ins> // RecommendedAuthOverrideFlags is a convenience method to return recommended flag names prefixed with a string of your choosing func RecommendedAuthOverrideFlags(prefix string) AuthOverrideFlags { return AuthOverrideFlags{

Found in openshift, but I'm pretty sure the codepaths are the same. That means the token is getting ignored and in-cluster config is being used in preference to flags.
Is a similar issue?
Yep, kind of regression.
Sigh.
Yes, most likely the same cause.
I'm on it.
Unit test :)
Probably will want to cherrypick this into 1.5.
WIP in


Question: t.Errorf("Timed out waiting for response.") } } <ins> func TestWarningWithRequestTimeout(t *testing.T) { type result struct { err        interface{} stackTrace string } clientDoneCh, resultCh := make(chan struct{}), make(chan result, 1) testHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // this will catch recoverable panic like 'Header called after Handler finished'. // go runtime crashes the program if it detects a program-ending // panic like 'concurrent map iteration and map write', so this // panic can not be caught. defer func() { result := result{} result.err = recover() if result.err != nil { // Same as stdlib http server code. Manually allocate stack // trace buffer size to prevent excessively large logs const size = 64 << 10 buf := make([]byte, size) buf = buf[:goruntime.Stack(buf, false)] result.stackTrace = string(buf) } resultCh <- result }() // add warnings while we're waiting for the request to timeout to catch read/write races loop: for { select { case <-r.Context().Done(): break loop default: warning.AddWarning(r.Context(), "a", "1") } } // the request has just timed out, write to catch read/write races warning.AddWarning(r.Context(), "agent", "text") // give time for the timeout response to be written, then try to // write a response header to catch the "Header after Handler finished" panic <-clientDoneCh warning.AddWarning(r.Context(), "agent", "text") }) handler := newGenericAPIServerHandlerChain(t, "/test", testHandler) server := httptest.NewUnstartedServer(handler) server.EnableHTTP2 = true server.StartTLS() defer server.Close() request, err := http.NewRequest("GET", server.URL+"/test?timeout=100ms", nil) if err != nil { t.Fatalf("unexpected error: %v", err) } client := server.Client() response, err := client.Do(request) close(clientDoneCh) if err != nil { t.Errorf("expected server to return an HTTP response: %v", err) } if want := http.StatusGatewayTimeout; response == nil || response.StatusCode != want { t.Errorf("expected server to return %d, but got: %v", want, response) } var resultGot result select { case resultGot = <-resultCh: case <-time.After(wait.ForeverTestTimeout): t.Errorf("the handler never returned a result") } if resultGot.err != nil { t.Errorf("Expected no panic, but got: %v", resultGot.err) t.Errorf("Stack Trace: %s", resultGot.stackTrace) } } // builds a handler chain with the given user handler as used by GenericAPIServer. func newGenericAPIServerHandlerChain(t *testing.T, path string, handler http.Handler) http.Handler { config, _ := setUp(t) s, err := config.Complete(nil).New("test", NewEmptyDelegate()) if err != nil { t.Fatalf("Error in bringing up the server: %v", err) } s.Handler.NonGoRestfulMux.Handle(path, handler) return s.Handler } </ins>

I am doing v1.28.3 kube-apiserver testing in the test environment, and I often encounter kube-apiserver panic, especially when kube-apiserver call webhook fails. The following are the captured panic logs: and: kube-apiserver does not panic even if calling webhook times out or fails Deploy a webhook, remove the service corresponding to the webhook, and then create pods in batches to trigger webhook calls No response <details</details<detailsprivate cloud </details<details</details<details</details<details</details<details</details
/sig api-machinery
somewhat related to
The scenarios that trigger panic are similar(the apiserver fails to call webhook), but the log shows that the location of panic is different.
/assign For context of  Thank you. /triage accepted
again, full stack logs:
I'll try to test the mr:
Let me try to explain the panic - both goroutines are trying to write to the of the first goroutine: This request has timed out, and consequently the filter is trying to send a response to the caller. This is the client facing goroutine from the timeout filter. Note that the call to is protected by a mutex, that means that if the other goroutine is using the delegated object via the adapter, then this panic should not happen. second goroutine: let's check the other goroutine, it is also trying to write to the of the object: Now uses an unprotected version of the ResponseWriter object: The filter precedes the filter in the execution order, so the in is not using the thread safe implementation. This is the order today: one quick resolution is to move the after the timeout filter: , but there could be code that precedes that needs to write warning? Also note that the panic mentioned in is not as severe, it does not cause the apiserver to crash. For a more proper solution, i think we would want any thoughts?


Question: t.Errorf("Timed out waiting for response.") } } <ins> func TestWarningWithRequestTimeout(t *testing.T) { type result struct { err        interface{} stackTrace string } clientDoneCh, resultCh := make(chan struct{}), make(chan result, 1) testHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // this will catch recoverable panic like 'Header called after Handler finished'. // go runtime crashes the program if it detects a program-ending // panic like 'concurrent map iteration and map write', so this // panic can not be caught. defer func() { result := result{} result.err = recover() if result.err != nil { // Same as stdlib http server code. Manually allocate stack // trace buffer size to prevent excessively large logs const size = 64 << 10 buf := make([]byte, size) buf = buf[:goruntime.Stack(buf, false)] result.stackTrace = string(buf) } resultCh <- result }() // add warnings while we're waiting for the request to timeout to catch read/write races loop: for { select { case <-r.Context().Done(): break loop default: warning.AddWarning(r.Context(), "a", "1") } } // the request has just timed out, write to catch read/write races warning.AddWarning(r.Context(), "agent", "text") // give time for the timeout response to be written, then try to // write a response header to catch the "Header after Handler finished" panic <-clientDoneCh warning.AddWarning(r.Context(), "agent", "text") }) handler := newGenericAPIServerHandlerChain(t, "/test", testHandler) server := httptest.NewUnstartedServer(handler) server.EnableHTTP2 = true server.StartTLS() defer server.Close() request, err := http.NewRequest("GET", server.URL+"/test?timeout=100ms", nil) if err != nil { t.Fatalf("unexpected error: %v", err) } client := server.Client() response, err := client.Do(request) close(clientDoneCh) if err != nil { t.Errorf("expected server to return an HTTP response: %v", err) } if want := http.StatusGatewayTimeout; response == nil || response.StatusCode != want { t.Errorf("expected server to return %d, but got: %v", want, response) } var resultGot result select { case resultGot = <-resultCh: case <-time.After(wait.ForeverTestTimeout): t.Errorf("the handler never returned a result") } if resultGot.err != nil { t.Errorf("Expected no panic, but got: %v", resultGot.err) t.Errorf("Stack Trace: %s", resultGot.stackTrace) } } // builds a handler chain with the given user handler as used by GenericAPIServer. func newGenericAPIServerHandlerChain(t *testing.T, path string, handler http.Handler) http.Handler { config, _ := setUp(t) s, err := config.Complete(nil).New("test", NewEmptyDelegate()) if err != nil { t.Fatalf("Error in bringing up the server: %v", err) } s.Handler.NonGoRestfulMux.Handle(path, handler) return s.Handler } </ins>

/cc
I tend to remove the timeout filter, the sooner the better, and I think the Apiserver panic is the P0-level bug, directly affecting the availability of the cluster I have cherry-picked and tested it in my cluster, and I still need to continue to observe for 1-2 days
I opened a minimal WIP fix for the warning writer in since the timeout filter removal is a much bigger change to reason through
thanks a lot, I will try to test and cherry-pick the above minimal WIP fix in
sorry, I completely overlooked , I only saw and that looked too big to take safely right now I prefer to
Can we backport the fix to 1.28?
Why 1.28 specifically? I'm not aware of anything that changed in 1.28 that made this more or less prevalent. If this meets backport criteria, going to all active branches could make sense (1.27+)


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/net" "k8s.io/apimachinery/pkg/util/runtime" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/wait" corev1client "k8s.io/client-go/kubernetes/typed/core/v1" "k8s.io/client-go/tools/record"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I created several Workloads which require up to ten ports for both UDP and TCP protocol. Now my logs get spammed with . What you expected to happen: should consider -combination instead of only . How to reproduce it (as minimally and precisely as possible): Deploy any service with a node-port with both protocols, e.g.: Anything else we need to know?: This was already acknowledged by here: However, as this thread addresses a different bug and seems kind of stuck, i'd like to suggest that this might be addressed in a new context. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Deployed via Rancher v2.4.3
/sig api-machinery
/remove-sig api-machinery /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
How did you get into this situation? The allocator should not have granted the same node port twice?
Oh right, that issue. The only real way out is to de-allocate the NodePorts and then re-allocate them, I think
As this happens for all of my ports in the said Production Application (the one where my log gets spammed) as well as complete fresh installation (where i verified the bug), i think there's no "way out" except fixing the bug. As said, for me it's necessary to assign both protocols to the same port. This might be a niche-use-case, but checking the it's actually very likely for ports to allow tcp AND udp. I'm not sure about the history of this, but it might be that issue was the origin, which was adressed in PR ("Allow services which use same port, different protocol to use the same nodePort for both"). But this PR doesn't contain the adjustment of the said , so still considers the behaviour as invalid, while it's actually allowed and supported.


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/net" "k8s.io/apimachinery/pkg/util/runtime" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/wait" corev1client "k8s.io/client-go/kubernetes/typed/core/v1" "k8s.io/client-go/tools/record"

/assign
Ahh, I thought you meant that you had accidentally been allocated the same nodePorts and I was very confused when you said "all of them". I have reproduced this issue, and I can see it by inspection. The obvious fix would be to collect ports split by protocol, but I think that's not right since we don't store it split. An even simpler fix would be for to build a . Then for each protocol, for each port, build that contains all the port numbers, but remove any duplicates ONLY IF they are duplicated across protocols. Duplicates in the same protocol should be retained (to report an error later). I don't have time to tackle this tonight (and write a test), but it seems like a good bug.
perhaps I can fix this problem /assign
things may be a little more complicated. Right now the is not aware of the protocol of the port so I guess simply rewriting function to return is not enough, the port allocator will still return on same node port even with different protocol. perhaps we need to refactor the to make it protocol-aware.
I don't think we need to do make the allocator be protocol aware. IIRC we allow you to use the same nodeport on different protocols ONLY within a single service. This means (I think) that we can do hyper-local deduplication as we collect the nodeports for each service. If we find the same nodeport number with different protocols, just remove the duplicates because it is allowed. I.e. a service that looks like: return - deduped. if ANY OTHER service tried to use it would be an error at the "rebuilt" allocator Likewise if a single Service claims to use the same nodeport number more than once on the same protocol, it should be an error: return - NOT deduped.
obviously, we want a great unit test here :P
ah got it! thanks for your detailed explanation!
Hi! I have filed a PR to resolve this issue, could you take a look?


Question: mkdir -p "${dst_dir}" local salt_dir="${KUBE_ROOT}/cluster/saltbase/salt" <ins> cp "${salt_dir}/cluster-autoscaler/cluster-autoscaler.manifest" "${dst_dir}/" </ins> cp "${salt_dir}/fluentd-es/fluentd-es.yaml" "${release_stage}/" cp "${salt_dir}/fluentd-gcp/fluentd-gcp.yaml" "${release_stage}/" cp "${salt_dir}/kube-registry-proxy/kube-registry-proxy.yaml" "${release_stage}/"

For some reason they stopped working and we even don't create manifest file for cluster autoscaler. I used the following command to start the cluster:
It's possible it's related to


Question: if pod == nil { return framework.NewStatus(framework.Error, "pod cannot be nil") } <del> state.RLock() defer state.RUnlock() </del> if v, e := state.Read(framework.StateKey(pod.Name)); e == nil { if value, ok := v.(*stateData); ok && value.data == "never bind" { return framework.NewStatus(framework.Unschedulable, "pod is not permitted")

What happened: kube-scheduler () crashes due to a panic while running : The race condition is triggered when an out-of-tree plugin, that writes to during its Filter phase, is registered through . Access to in general happens from a single goroutine and there are no practical concurrency issues, with the exception of the phase, that parallelizes calls for each node across plugins: A quick glance at the tip of the branch seems to indicate the bug is still present there. What you expected to happen: Ideally access to should be synchronized everywhere for correctness, including by all in-tree plugins, since there are no guarantees about how will call multiple plugins concurrently. It seems to me it is something that was considered at some point, given that has methods. Sadly, almost no in-tree plugins are synchronizing access. For now, I will try to work around the problem by doing a single write to during and storing a data bag structure where I will write state into during (with proper synchronization), instead of writing to directly. How to reproduce it (as minimally and precisely as possible): Install a scheduler plugin that writes to during its phase. In-tree (builtin) plugins are not synchronizing their access to . Environment: Kubernetes version (use ): v1.18.8 Cloud provider or hardware configuration: linux-amd64 (AWS) OS (e.g: ): Ubuntu 18.04.5 LTS (Bionic Beaver) Others: custom kube-scheduler binary built with an out-of-tree plugin using
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
AFAICT all in-tree plugins write to the cycle state in PreFilter and then only read from cycle state in Filter. So this shouldn't be a problem with in-tree plugins as concurrent read access should be fine during RunFilterPlugins. But since your out-of-tree plugin writes in Filter, I can see why this can be a problem.


Question: if pod == nil { return framework.NewStatus(framework.Error, "pod cannot be nil") } <del> state.RLock() defer state.RUnlock() </del> if v, e := state.Read(framework.StateKey(pod.Name)); e == nil { if value, ok := v.(*stateData); ok && value.data == "never bind" { return framework.NewStatus(framework.Unschedulable, "pod is not permitted")

Aside from adding Lock/Unlock in in-tree plugin, offering mutual exclusion on each key as a built-in feature within CycleState could be an alternative solution. This way, reads from in-tree plugins during Filter wouldn't interfere with writes during Filter from out-of-tree plugins.
cc
should we change the Read/Write CycleState methods to be thread safe?
Adding locks without a proper justification would just increase the latency. could you clarify your use case? Why do you need to write anything during Filter?
yeah, we should measure it, but I don't expect it to have significant effect...
Adding locks without a proper justification would just increase the latency. Without contention (which is the case with in-tree plugins), I don't think this is a concern.
Adapting source from : source: essentially, 11.3 ns/Lock-Unlock pair
I remember locks showing up in profiling for PreScore/Score for certain plugins. Although I have to admit that the accesses were not properly implemented. I'd like to hear an answer for before doing any experiments.
+1 to that I'd also like to understand the motivations / requirements of mutating CycleState in Filter().
My particular use-case is that we have resources being dynamically to/removed from Nodes, and those resources need to be allocated (and tracked) in the scheduling loop. Technically I could pre-compute all the available dynamic resources for all Nodes at the phase, and snapshot it, but that would require me to pre-compute everything for all Nodes on all scheduling cycles. Instead, what I'm currently doing is computing what dynamic resources can be utilized on-demand during the phase (kind of a lazy loading of dynamic resources for each pair), and saving state of what extra resources were considered during for each Node that was evaluated. Then, later in the scheduler plugin can lock-in the exact dynamic resources for the particular Node that was selected. I understand the concern with adding extra synchronization overhead without a real need. Even if my use-case is not good enough to justify it, there is another argument for it: technically, the lack of access synchronization on violates the . Even if in the end we decide synchronizing all access is not worth the overhead (for correctness), I would strongly encourage better documentation saying which plugin phases are allowed to modify , and perhaps even forcing a snapshot-read-only copy of to be used by plugins.
Using the scheduler to track resources is not a good idea.


Question: if pod == nil { return framework.NewStatus(framework.Error, "pod cannot be nil") } <del> state.RLock() defer state.RUnlock() </del> if v, e := state.Read(framework.StateKey(pod.Name)); e == nil { if value, ok := v.(*stateData); ok && value.data == "never bind" { return framework.NewStatus(framework.Unschedulable, "pod is not permitted")

The scheduler should be stateless. Can't you track the resources with an external controller or with a device plugin and making it part of the Node spec?
there may be some misunderstanding in my poor choice of words. What I mean by "tracking" is that the scheduler needs to track usage and availability of these extra per-Node resources, to be able to allocate them during placement/scheduling. Exactly the same it does with volumes, cpus, memory, ephemeral disk, etc Doing computations in a controller outside of the scheduling loop would be similar to "precomputing" these resources. In my particular use case I chose a different way because: introduces another point of failure, if the external controller stops pre-computing extra resources and publishing them somewhere (CRD, Node spec, etc) scheduling halts (at least scheduling on these extra resources) Node resources in my use case are volatile (they are coming and going frequently), and the extra latency by an extra controller (vs doing all computations right in the scheduling loop) may be problematic for my use case. Even if you consider my use case not enough to add proper synchronization to , I'd still encourage making it explicit what scheduling phases are allowed to modify it (and even passing in a snapshot read-only version of to phases that are not expected to update it), to avoid confusion and race conditions.
Ok, if you are not tracking the existence of resources in the scheduler, I wouldn't suggest the external controller. Still, have you considered the device plugin framework? That way, the default kube-scheduler could keep track of resources usage. Otherwise, I still don't see why you would need to update the CycleState during Filter. As its name indicates, CycleState only lives during a scheduling cycle.
Regardless of the correctness of usage of CycleState to do what wants, I still think we need to do at least one of the following: that writing to CycleState during Filter will lead to data races, a read-only CycleState to Filter plugins (whether we do this at the interface level or if we simply drop Write requests during Filter is an implementation detail), or make CycleState concurrency safe CycleState as it exists right now allows plugin authors to shoot themselves in the foot IMO.
I'm leaning towards 1, but it's fair to continue the discussion before deciding.
Can you run the "controller" as a goroutine along with your scheduler plugin?


Question: if pod == nil { return framework.NewStatus(framework.Error, "pod cannot be nil") } <del> state.RLock() defer state.RUnlock() </del> if v, e := state.Read(framework.StateKey(pod.Name)); e == nil { if value, ok := v.(*stateData); ok && value.data == "never bind" { return framework.NewStatus(framework.Unschedulable, "pod is not permitted")

So it's not an external controller binary, and hence don't introduce another point of failure. It's pretty like how the default scheduler manages its internal cache. But note that cache is only needed when you need a more efficient data structure. If a {key:value} map is good enough, spinning up a XYZInformer and obtain the latest XYZ resource on-demand is the most recommended way. I think it quite depends on how you design the internal XYZ cache. For example, it should only tweak the cache in an incremental manner - in other words, unchanged resource will be kept unmodified. You can check out the "doubly-pointer snapshot" of scheduler's cache. With above said, if I understand your case correctly, you may need to build a sophisticated XYZ cache, which runs in a separate goroutine and responds to your XYZ resource changes. And then upon each scheduling cycle, take a snapshot (may needs a specific snapshotting algorithm) in PreFilter() and WRITE to CycleState, and then in Filter() READ from CycleState. My personal option is to suggest all plugin developers to read from CycleState in Filter().
nailed it. What you describe is almost exactly what I’m doing today. There are some details particular to my use case (e.g. just tracking current state with an informer and an incremental cache is not enough, because there’s some logic involved in deciding if currently available resources are usable by each pair, and I currently do that computation in as pairs are evaluated). In any case, I just wanted to share my use case to provide some color, but I don’t want what I’m doing distracting us from the more important questions, mentioned by above. My personal preference would be as a very short term next step, then when someone has cycles to do it. I would be OK with punting on until a more strong need shows up.
Note that limiting to only reading from works for me, as long as I can still read my own data structure from it and do writes on it (i.e. don’t deepclone for the phase).
also adds overhead (from copying the state, which can be very expensive).
Could implement just by changing the interface passed to (e.g. ). That wouldn’t be backwards compatible, but very cheap.


Question: if pod == nil { return framework.NewStatus(framework.Error, "pod cannot be nil") } <del> state.RLock() defer state.RUnlock() </del> if v, e := state.Read(framework.StateKey(pod.Name)); e == nil { if value, ok := v.(*stateData); ok && value.data == "never bind" { return framework.NewStatus(framework.Unschedulable, "pod is not permitted")

I guess that would work in the sense that you cannot change the high level values of the CycleState map, but you could still override what's inside them, if they have pointers. But maybe that's ok.
but you could still override what's inside them, if they have pointers. But maybe that's ok. yeah, if it was pointers and if the underlying data needed to be modified, I'd imagine it would have its own Lock/Unlock semantics
for such implementation, the only remaining concern is backwards compatibility. So I would leave that decision to who leads the scheduling-plugins repo.
I rethink 's original problem a little bit: the original racing issue occurs when reading CycleState in in-tree plugins, this indicates one of the (out-of-tree) plugins is (over)writing the most high-level value of the CycleState map. But if we control the locking inside the out-of-tree plugin, and mutate that particular in-place, that would be good enough? Case 3 in the above program demonstrates the correct way to manipulate the specific out-of-tree state. Let me know your thoughts. If it resolves 's concern, we don't need to make any change on the Filter() interface.
Case 3 is how I am working around the problem today to avoid the race. I write a "data bag" once into top level in , then only modify my own inner structure (with proper locking) in , without doing any further modifications to directly (top level) in . That is acceptable, but I'd encourage you to consider making the interface (and documentation) more explicit for plugins, so they avoid the mistake of writing directly into . Without changing anything (no documentation, no readonly interface), it is pretty easy for a plugin to use in a way that you don't want to support.
Thanks for confirming that. Yes, documenting it, or(and) publishing examples would be quite helpful to clarify the semantics. As a wrap-up: a keyed value can be read from CycleState, literally , in any phase. It's always race-free.


Question: if pod == nil { return framework.NewStatus(framework.Error, "pod cannot be nil") } <del> state.RLock() defer state.RUnlock() </del> if v, e := state.Read(framework.StateKey(pod.Name)); e == nil { if value, ok := v.(*stateData); ok && value.data == "never bind" { return framework.NewStatus(framework.Unschedulable, "pod is not permitted")

if you want to mutate the top-level keyed value in CycleState, if it's a phase that CycleState is accessed consecutively, such as PreFilter, PreScore, it's safe to mutate CycleState in any manner if it's a phase that CycleState is accessed parallelly, to be race-free, (for now) you only have 1 option: keep the keyed value's pointer unchanged, and use an internal lock to mutate that pointer obj [TBD] we may come up with changes to add read locks to all in-tree plugins when reading values from CycleState, so that the out-of-tree plugin can surround its mutating logic between and - th mutating logic thereby can be any style Any thoughts on the TBD item? I would assume read locks wouldn't cost too much.
For now, I think the documentation is correct, it clearly says Read() and Write() are not thread-safe: The key thing here is whether in-tree plugin should use RLock/RUnlock or not. If yes, out-of-tree plugins can be freed to use Lock/Unlock. (see the TBD item in the previous comment)
I like the TBD, and optimizing for correctness in how in-tree plugins are accessing . I understand the concern with synchronization overhead, but I would be really surprised if it is the bottleneck in scheduling code.
It can be, when you have 5k-nodes clusters, or bigger.
I don't think it will be expensive, a rlock should be equivalent to an atomic add operation (basically it increments the number of readers). As a comparison, looking up the state from the map is likely to be more expensive.
Of course we are assuming no contention in the default plugins.
This is my impression as well. A vanilla version of scheduler doesn't impose write lock in Filter() and Score(), so should be good.
We can always see with the benchmarks and revert if its not negligible.
I will run the benchmarks to see if there is perf downgrade. /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale This is still relevant, yes?


Question: if pod == nil { return framework.NewStatus(framework.Error, "pod cannot be nil") } <del> state.RLock() defer state.RUnlock() </del> if v, e := state.Read(framework.StateKey(pod.Name)); e == nil { if value, ok := v.(*stateData); ok && value.data == "never bind" { return framework.NewStatus(framework.Unschedulable, "pod is not permitted")

IIUC the volumebinding plugin has been fixed, but external plugins can still shoot themselves in the foot.
/priority backlog
Yes. I'm working on this, late this week I will send out a PR along with perf test result.


Question: } } <ins> func TestValidateFinalizersUpdate(t *testing.T) { testcases := map[string]struct { Old         api.ObjectMeta New         api.ObjectMeta ExpectedErr string }{ "invalid adding finalizers": { Old:         api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &unversioned.Time{}, Finalizers: []string{"x/a"}}, New:         api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &unversioned.Time{}, Finalizers: []string{"x/a", "y/b"}}, ExpectedErr: "y/b", }, "invalid changing finalizers": { Old:         api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &unversioned.Time{}, Finalizers: []string{"x/a"}}, New:         api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &unversioned.Time{}, Finalizers: []string{"x/b"}}, ExpectedErr: "x/b", }, "valid removing finalizers": { Old:         api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &unversioned.Time{}, Finalizers: []string{"x/a", "y/b"}}, New:         api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &unversioned.Time{}, Finalizers: []string{"x/a"}}, ExpectedErr: "", }, "valid adding finalizers for objects not being deleted": { Old:         api.ObjectMeta{Name: "test", ResourceVersion: "1", Finalizers: []string{"x/a"}}, New:         api.ObjectMeta{Name: "test", ResourceVersion: "1", Finalizers: []string{"x/a", "y/b"}}, ExpectedErr: "", }, } for name, tc := range testcases { errs := ValidateObjectMetaUpdate(&tc.New, &tc.Old, field.NewPath("field")) if len(errs) == 0 { if len(tc.ExpectedErr) != 0 { t.Errorf("case: %q, expected error to contain %q", name, tc.ExpectedErr) } } else if e, a := tc.ExpectedErr, errs.ToAggregate().Error(); !strings.Contains(a, e) { t.Errorf("case: %q, expected error to contain %q, got error %q", name, e, a) } } } </ins> func TestValidateObjectMetaUpdatePreventsDeletionFieldMutation(t *testing.T) { now := unversioned.NewTime(time.Unix(1000, 0).UTC()) later := unversioned.NewTime(time.Unix(2000, 0).UTC())

says "If the deletionTimestamp of the object is non-nil, entries in this list can only be removed.". can you elucidate if I am assuming this does something different than reality
This is garbage collection finalizer field, not the namespace finalizer field that that new field was inspired from On Thursday, September 8, 2016, Tim Hockin wrote:
I don't parse that - is this not a general-purpose "refcount" thing? It seems to work that way, except not as documented
This is a general purpose field and it should work like Tim thinks it should work. can you take a look at this when you don't have anything pressing for 1.4?
The namespace finalizer should switch to using this field if it is currently doing something else.
I'm adding validation rule to prevent adding finalizers if deletionTimestamps is set. I'm still investigating why it can't be removed. This bug is strange.
Actually I guess it's a bug. It uses patch, and patch doesn't work on list of simple type, is a list of string.
Yep, it's a bug. I used to remove one finalizer and it worked as expected. I think this is a known bug of , we don't have a way to specify the mergekey of a list of simple type.
Is there a bug open for this? Frankly, if is going to fail for things like this, we should not be offering it as a feature, or it should use PUT rather than PATCH , Chao Xu wrote:
Created cannot use PUT, because it's doing a three way merge (current state, last configuration, current configuration).
These are intended for automated use, not human use, so kubectl edit not working is just unfortunate, not a blocker. Although it's really unfortunate that PATCH is so broken. , Chao Xu wrote:
Why does need to be 3-way? I don't think that is the semantic a user would expect from an edit operation AT ALL. It certainly surprised ME. WRT automated vs human - if PATCH doesn't work (worse - says it succeeded but didn't) for some class of fields, then PATCH doesn't work, and we should not support it. Full stop. It's going to end up causing someone an outage. These are not necessarily related to this bug, but I'd call them pretty serious flaws.


Question: } } <ins> func TestValidateFinalizersUpdate(t *testing.T) { testcases := map[string]struct { Old         api.ObjectMeta New         api.ObjectMeta ExpectedErr string }{ "invalid adding finalizers": { Old:         api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &unversioned.Time{}, Finalizers: []string{"x/a"}}, New:         api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &unversioned.Time{}, Finalizers: []string{"x/a", "y/b"}}, ExpectedErr: "y/b", }, "invalid changing finalizers": { Old:         api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &unversioned.Time{}, Finalizers: []string{"x/a"}}, New:         api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &unversioned.Time{}, Finalizers: []string{"x/b"}}, ExpectedErr: "x/b", }, "valid removing finalizers": { Old:         api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &unversioned.Time{}, Finalizers: []string{"x/a", "y/b"}}, New:         api.ObjectMeta{Name: "test", ResourceVersion: "1", DeletionTimestamp: &unversioned.Time{}, Finalizers: []string{"x/a"}}, ExpectedErr: "", }, "valid adding finalizers for objects not being deleted": { Old:         api.ObjectMeta{Name: "test", ResourceVersion: "1", Finalizers: []string{"x/a"}}, New:         api.ObjectMeta{Name: "test", ResourceVersion: "1", Finalizers: []string{"x/a", "y/b"}}, ExpectedErr: "", }, } for name, tc := range testcases { errs := ValidateObjectMetaUpdate(&tc.New, &tc.Old, field.NewPath("field")) if len(errs) == 0 { if len(tc.ExpectedErr) != 0 { t.Errorf("case: %q, expected error to contain %q", name, tc.ExpectedErr) } } else if e, a := tc.ExpectedErr, errs.ToAggregate().Error(); !strings.Contains(a, e) { t.Errorf("case: %q, expected error to contain %q, got error %q", name, e, a) } } } </ins> func TestValidateObjectMetaUpdatePreventsDeletionFieldMutation(t *testing.T) { now := unversioned.NewTime(time.Unix(1000, 0).UTC()) later := unversioned.NewTime(time.Unix(2000, 0).UTC())

, Daniel Smith wrote:
Discussed in real life. PATCH works, it's just that there are certain operations you cannot express using the strategicmerge patch syntax. I was wrong. is building . So perhaps it uses PATCH over PUT due to the efficiency.
FWIW, I printed out the patch generated by when I removed one finalizer. The patch is empty, i.e., "{}".
, Chao Xu wrote: Well, no, if there are operations that can not be represented, it really DOESN'T work. I still think it should not use PUT, but I bet Brian can prove me wrong.


Question: package clouddns import ( <ins> "google.golang.org/api/googleapi" </ins> "k8s.io/kubernetes/federation/pkg/dnsprovider" "k8s.io/kubernetes/federation/pkg/dnsprovider/providers/google/clouddns/internal/interfaces" )

/kind bug What happened: walteraa [6:55 AM] Hi, The controller manager of the federation using Google cloud DNS is crashing, the log showed is that: the hyperkube command is that: I have created this in the Google DNS Someone knows why the DNS zone query is getting the 404 error?? :disappointed: What you expected to happen: It should have created the missing domain. How to reproduce it (as minimally and precisely as possible): As per above. Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.4 Cloud provider or hardware configuration**: OpenStack (irrelevant) OS (e.g. from /etc/os-release): irrelevant Kernel (e.g. ): irrelevant Install tools: irrelevant Others: irrelevant
The bug is related to this: The above code assumes that the provider code returns no error but an empty list if the domain does not exist, but instead the provider returns a 404. Need to fix the provider and/or the above code.
/sig federation
I am unlikely to have time to work on this soon, so if anyone else wants to, go right ahead.
/assign
An additional information: I have created a which is based on the credential JSON file created through instruction. After that, I have configured the Federation controller manager to mount this secret as a volume in the container and set the environment variable to the JSON file inside this volume. Before doing that, I was getting an authentication error so the google DNS provider couldn't be instantiated, and after doing the instructions that I mentioned before, this authentication error was solved but now I'm getting 404 error when Google DNS provider tries access the specified managed zone. The file that I have generated follow this structure:


Question: // Package discovery provides ways to discover server-supported // API groups, versions and resources. <del> package discovery </del> <ins> package discovery // import "k8s.io/client-go/discovery" </ins>

Look at one of the directories under (Please don't pick more than one directory for one PR as it will cause issues for reviewers) Find all the files named (example find -name ) See the following link it has a this tells tools how this package needs to be imported. So we should ensure that each has an import Add a description appropriate to the package (similar to the one below) Look through sub directories and see if there are any other important directories that are likely to be used by outside users and add a Why are we doing this? We perfer to use vanity urls instead of directly using to refer to code, so this helps ensure that tools honor our wish. See It would be good to have some package level documentation since these directories under staging/ are meant to be for use by folks outside of kubernetes github org. /good-first-issue
/kind documentation /sig architecture
currently working my through Not going to use / assign since there could be a number of PRs for this issue.
/lifecycle active /good-first-issue edit: hmm..
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /lifecycle active /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
May I try on
IIUC this means that there needs to be a even at the root of the repo? Then how does work? :thinking:
Hi , can I make a PR for this directory ?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: // Package discovery provides ways to discover server-supported // API groups, versions and resources. <del> package discovery </del> <ins> package discovery // import "k8s.io/client-go/discovery" </ins>

/lifecycle rotten
I guess this is still not done, was supposed to partially fix this issue. /reopen
Reopened this issue. <detailsIn response to : I guess this is still not done, was supposed to partially fix this issue. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /area code-organization


Question: serviceHasSynced = func() bool { return true } } <del> nodeIndexer := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{}) </del> <ins> var nodeHasSynced cache.InformerSynced var nodeLister corelisters.NodeLister </ins> if kubeDeps.KubeClient != nil { <del> fieldSelector := fields.Set{api.ObjectNameField: string(nodeName)}.AsSelector() nodeLW := cache.NewListWatchFromClient(kubeDeps.KubeClient.CoreV1().RESTClient(), "nodes", metav1.NamespaceAll, fieldSelector) r := cache.NewReflector(nodeLW, &v1.Node{}, nodeIndexer, 0) go r.Run(wait.NeverStop) </del> <ins> kubeInformers := informers.NewSharedInformerFactoryWithOptions(kubeDeps.KubeClient, 0, informers.WithTweakListOptions(func(options *metav1.ListOptions) { options.FieldSelector = fields.Set{api.ObjectNameField: string(nodeName)}.String() })) nodeLister = kubeInformers.Core().V1().Nodes().Lister() nodeHasSynced = func() bool { if kubeInformers.Core().V1().Nodes().Informer().HasSynced() { klog.Infof("kubelet nodes sync") return true } klog.Infof("kubelet nodes not sync") return false } kubeInformers.Start(wait.NeverStop) klog.Info("Kubelet client is not nil") } else { // we dont have a client to sync! nodeIndexer := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{}) nodeLister = corelisters.NewNodeLister(nodeIndexer) nodeHasSynced = func() bool { return true } klog.Info("Kubelet client is nil") </ins> } <del> nodeLister := corelisters.NewNodeLister(nodeIndexer) </del> // TODO: get the real node object of ourself, // and use the real node name and UID.

What happened: A deployment is created that uses a after the cluster master is created but before the cluster worker nodes are fully initialized. During the initialization of the worker nodes, the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector. Depending on how long the nodes are available for scheduling without the necessary node labels for the deployment, the scheduler will start to spam with cluster with pods in status. This spamming stops once the worker nodes are fully initialized and the pods are scheduled successful. The exact timing of all of this is TBD. What you expected to happen: Pods in status are annoying and require manual cleanup. It would be preferred that the scheduler avoid this an simply issue warning events. How to reproduce it (as minimally and precisely as possible): See issue description. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.18.3 Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 4.15.0-96-generic Install tools: Managed service install Network plugin and version (if this is a network-related bug): N/A Others: N/A
Related to and
/sig scheduling
by spamming are you referring to scheduler logs? also, why are the nodes not labelled from the get go?
Just to confirm, all the pods in question are still in Pending state, yes? If the pods are in pending state, all of them will get scheduled eventually once the nodes get their labels.
The pods are not in state. They are actually in state. The pods eventually get scheduled but we are left with anywhere from a few to dozens of these state pods. Also, we are looking at labeling the nodes via kubelet flags to see if that helps.
/remove-kind bug
After we add some retry logic and wait time in this problem can be solved: As you can see this function returns initial node as the first request failed.
I also get hit by this, when single controller node gets rebooted and scheduler starts as static container before kubelet becomes ready.


Question: serviceHasSynced = func() bool { return true } } <del> nodeIndexer := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{}) </del> <ins> var nodeHasSynced cache.InformerSynced var nodeLister corelisters.NodeLister </ins> if kubeDeps.KubeClient != nil { <del> fieldSelector := fields.Set{api.ObjectNameField: string(nodeName)}.AsSelector() nodeLW := cache.NewListWatchFromClient(kubeDeps.KubeClient.CoreV1().RESTClient(), "nodes", metav1.NamespaceAll, fieldSelector) r := cache.NewReflector(nodeLW, &v1.Node{}, nodeIndexer, 0) go r.Run(wait.NeverStop) </del> <ins> kubeInformers := informers.NewSharedInformerFactoryWithOptions(kubeDeps.KubeClient, 0, informers.WithTweakListOptions(func(options *metav1.ListOptions) { options.FieldSelector = fields.Set{api.ObjectNameField: string(nodeName)}.String() })) nodeLister = kubeInformers.Core().V1().Nodes().Lister() nodeHasSynced = func() bool { if kubeInformers.Core().V1().Nodes().Informer().HasSynced() { klog.Infof("kubelet nodes sync") return true } klog.Infof("kubelet nodes not sync") return false } kubeInformers.Start(wait.NeverStop) klog.Info("Kubelet client is not nil") } else { // we dont have a client to sync! nodeIndexer := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{}) nodeLister = corelisters.NewNodeLister(nodeIndexer) nodeHasSynced = func() bool { return true } klog.Info("Kubelet client is nil") </ins> } <del> nodeLister := corelisters.NewNodeLister(nodeIndexer) </del> // TODO: get the real node object of ourself, // and use the real node name and UID.

Can be cleaned up with:
/sig node
the spam cycle is this loop: "pod create, pod schedule, worker reject, pod failed" -- we don't have a lot of protection in higher level workload controllers to handle pods that fail at kubelet admission time versus pods that just dont create due to quota/admission. in general, is the spam causing any other stability issues? is pod gc pruning the pods?
I wonder if we can do a trick in node lister to mitigate any race condition here...
we could maybe do something similar to to ensure node lister has synced...
hacking on option to ensure if kubelet has a node lister with a valid kube client, we wait for it to sync at least once here which should mitigate this issue. see:
We noticed the problem on the OLM pod which is managed by the OLM operator. Right or wrong, the operator does not like seeing the pod go to status and no longer reconciles the pod. The end result is that manual intervention (i.e. pod deletion) is required to fix the problem.
sure... i would have thought the pod is failed in that instance and reached a terminal state. either way, i am looking at what i can do to mitigate this.
The problem persists in our environment as well, and the workaround cited above will not do anymore. Is there something in the works for this one?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: b.Do().Visit(func(r *resource.Info) error { services := r.Object.(*api.ServiceList).Items for _, service := range services { <del> link := client.Host + "/api/v1beta3/proxy/namespaces/" + service.ObjectMeta.Namespace + "/services/" + service.ObjectMeta.Name </del> <ins> var link string if len(service.Spec.PublicIPs) > 0 { for _, port := range service.Spec.Ports { link += "http://" + service.Spec.PublicIPs[0] + ":" + strconv.Itoa(port.Port) + " " } } else { link = client.Host + "/api/v1beta3/proxy/namespaces/" + service.ObjectMeta.Namespace + "/services/" + service.ObjectMeta.Name } </ins> printService(out, service.ObjectMeta.Labels["name"], link) } return nil

I an ELB to the Kibana cluster log viewer but it is not reported by which, aside from having a URL, should check each cluster service and if it has an external IP it should report that instead of the master proxy address.
cc


Question: framework.ExpectNoError(err, "failed to delete Service %v in namespace %v", testService.ObjectMeta.Name, ns) framework.Logf("Service %s deleted", testSvcName) }) <ins> ginkgo.It("should delete a collection of services", func() { ns := f.Namespace.Name svcClient := f.ClientSet.CoreV1().Services(ns) svcResource := schema.GroupVersionResource{Group: "", Version: "v1", Resource: "services"} svcDynamicClient := f.DynamicClient.Resource(svcResource).Namespace(ns) svcLabel := map[string]string{"e2e-test-service": "delete"} deleteLabel := labels.SelectorFromSet(svcLabel).String() ginkgo.By("creating a collection of services") testServices := []struct { name  string label map[string]string port  int }{ { name:  "e2e-svc-a-" + utilrand.String(5), label: map[string]string{"e2e-test-service": "delete"}, port:  8001, }, { name:  "e2e-svc-b-" + utilrand.String(5), label: map[string]string{"e2e-test-service": "delete"}, port:  8002, }, { name:  "e2e-svc-c-" + utilrand.String(5), label: map[string]string{"e2e-test-service": "keep"}, port:  8003, }, } for _, testService := range testServices { func() { framework.Logf("Creating %s", testService.name) svc := v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:   testService.name, Labels: testService.label, }, Spec: v1.ServiceSpec{ Type: "ClusterIP", Ports: []v1.ServicePort{{ Name:       "http", Protocol:   v1.ProtocolTCP, Port:       int32(testService.port), TargetPort: intstr.FromInt(testService.port), }}, }, } _, err := svcClient.Create(context.TODO(), &svc, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create Service") }() } svcList, err := cs.CoreV1().Services(ns).List(context.TODO(), metav1.ListOptions{}) framework.ExpectNoError(err, "failed to list Services") framework.ExpectEqual(len(svcList.Items), 3, "Required count of services out of sync") ginkgo.By("deleting service collection") err = svcDynamicClient.DeleteCollection(context.TODO(), metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: deleteLabel}) framework.ExpectNoError(err, "failed to delete service collection. %v", err) svcList, err = cs.CoreV1().Services(ns).List(context.TODO(), metav1.ListOptions{}) framework.ExpectNoError(err, "failed to list Services") framework.ExpectEqual(len(svcList.Items), 1, "Required count of services out of sync") framework.Logf("Collection of services has been deleted") }) </ins> }) // execAffinityTestForSessionAffinityTimeout is a helper function that wrap the logic of

[X] Two weeks soak end date: Nov 18 According to this APIsnoop query, there are still a remaining Service endpoint which is untested. A service resource didn’t support endpoint until was merged. Create a watch to track service events Create a service with a static label. Confirm via the watch that the service was created. Delete the service via the service with the dynamic client with . Confirm via the watch that the service was deleted. The above steps have been implemented in the follwing . This query shows all the endpoints hit within a short period of running the above e2e test If a test with these calls gets merged, test coverage will go up by 1 point This test is also created with the goal of conformance promotion. /sig testing /sig network /area conformance
/triage accepted
Waiting on 2 week soak before promoting the test to conformance /reopen
Reopened this issue. <detailsIn response to : Waiting on 2 week soak before promoting the test to conformance /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if pod.Name != mirrorPod.Name || pod.Namespace != mirrorPod.Namespace { return false } <del> return api.Semantic.DeepEqual(&pod.Spec, &mirrorPod.Spec) </del> <ins> hash, ok := getHashFromMirrorPod(mirrorPod) if !ok { return false } return hash == getPodHash(pod) </ins> } func podsMapToPods(UIDMap map[types.UID]*api.Pod) []*api.Pod {

See for more context. Currently, kubelet compares the spec of a mirror pod (created via apiserver) with that of a static pod (config stored as a file) to decide whether the two are considered identical. If they differ, kubelet would delete the mirror pod and re-create it with the correct spec. Due to the fact that apiserver may set default values for some fields in the spec of the mirror pod, the specs could deviate undesirably. To avoid problems like this, we can store the config of a static pod as an annotation in the mirror pod, and compare the static pod with the annotation.
Would a hash of the config be sufficient if we're only going to compare? The whole spec would generally be more useful, but I wonder if that is too much data to put in an annotation.
This seems totally reasonable to me, but I'm still desperately curious why you can't just bypass defaulting when you generate mirror pod from static pod? (i.e. just use the static pod's configuration verbatim)
It's not kubelet who applies the default values here. The admission plugin was only run at the apiserver, and to bypass that, you have to ask apiserver to treat mirror pods differently and potentially at multiple places.
On May 22, 2015 10:38 AM, "Yu-Ju Hong" wrote: Understood, but is it really multiple places? If so then I agree it's a bad idea. Otherwise it might be worth considering as an alternative.
It's untenable for Kubelet to assume to know/circumvent all fields that will be set by other entities: admission control, defaulting, late initialization, auto-something, whatever. It needs to obey the same rules as other clients.
I don't understand why you would ever want to apply any of the things you mentioned to static pods. Static pods should be copied verbatim into the mirror pods, right? That's what I'm proposing.
What if the master is updated before the kubelet, and the master adds a new API field with a default value?
That case should be discussed in
I agree that mirror pod should reflect the true status of the static pod running on the node, and no default values should be set in this case.


Question: if pod.Name != mirrorPod.Name || pod.Namespace != mirrorPod.Namespace { return false } <del> return api.Semantic.DeepEqual(&pod.Spec, &mirrorPod.Spec) </del> <ins> hash, ok := getHashFromMirrorPod(mirrorPod) if !ok { return false } return hash == getPodHash(pod) </ins> } func podsMapToPods(UIDMap map[types.UID]*api.Pod) []*api.Pod {

However, I think it'd be harder to implement this way since, as stated above, that there are many things to circumvent in the apiserver itself. It could also be harder to keep this special case from being broken by a yet another new feature/plugin. Keeping the hash of the config in the annotation seems like a good compromise even though it may not reflect the true status 100% accurately. I think this is acceptable. What do you think?
Sure, if you think the annotation approach is easier, I would go with that. I guess in the case Brian mentoined (master adds a new API field with a default value), you would not update the mirror pod, because the static pod definition has not changed?
Yes, a hash would be sufficient.
It shouldn't be necessary to update the mirror pod in the case that a new default field was updated, because that field should be populated automatically by the apiserver on decode from etcd.
We should fix this so we don't have this problem again in 1.2.


Question: return p, err } <del> // waitForRCPodToDisappear returns nil if the pod from the given replication controller (described by rcName) no longer exists. // In case of failure or too long waiting time, an error is returned. func waitForRCPodToDisappear(c *client.Client, ns, rcName, podName string) error { label := labels.SelectorFromSet(labels.Set(map[string]string{"name": rcName})) return wait.Poll(20*time.Second, 5*time.Minute, func() (bool, error) { </del> <ins> func waitForPodToDisappear(c *client.Client, ns, podName string, label labels.Selector, interval, timeout time.Duration) error { return wait.Poll(interval, timeout, func() (bool, error) { </ins> Logf("Waiting for pod %s to disappear", podName) pods, err := c.Pods(ns).List(label, fields.Everything()) if err != nil {

More detail can be seen in e2e test
cc
, Marek Grabowski wrote:
you are running an older version of the test. can you get the latest and try again ? This test had not failed at all yesterday in our automated test runs yesterday. Closing issue for now.
This failure depends on if the pod is really deleted and kube-proxy (the test container on) catch that the endpoint is deleted in 5s . After sending "delete pod" request for 5s, if the pod was still there and kube-proxy still contains this endpoints(this may happen under pressure), hit test can still detect 5 endpoints. At least we should ensure the pod was deleted. Anyway, I think will be helpful to this test. What do you think?
It doesn't matter if the pod is actually deleted or not. We just care that kube-proxy thinks that it should not be part of the service endpoint. Having said that, it still doesn't hurt to do the check that you are doing. I will assign your PR to myself and review it. In the meantime, can you rebase your PR ?
Thanks for your review. The PR is up to date.
Failed twice today in kubernetes-e2e-gce-parallel
One more today. The error message is not very helpful.
18:28:37 STEP: dialing(http) endpointPodIP:endpointHttpPort from node1 18:28:37 STEP: Dialing from node. command:for i in ' wc -l This is the point where the test fails. Am looking into it. We need to have more tries, we just do seq (1 1), instead of seq (6 6). this is causing flakiness.
Fixed by PR