Question: # limitations under the License. ARG BASEIMAGE <ins> ARG RUNNERIMAGE </ins> <del> FROM ${BASEIMAGE} </del> <ins> FROM ${BASEIMAGE} as debbase FROM ${RUNNERIMAGE} # This is a dependency for `kubectl diff` tests COPY --from=debbase /usr/bin/diff /usr/local/bin/ COPY cluster /kubernetes/cluster </ins> COPY ginkgo /usr/local/bin/ COPY e2e.test /usr/local/bin/ COPY kubectl /usr/local/bin/ <del> COPY run_e2e.sh /run_e2e.sh </del> <ins> COPY gorunner /usr/local/bin/kubeconformance # Legacy executables -- deprecated COPY gorunner /run_e2e.sh </ins> COPY gorunner /gorunner <del> COPY cluster /kubernetes/cluster WORKDIR /usr/local/bin </del> ENV E2E_FOCUS="[Conformance]" ENV E2E_SKIP=""

<!-- Please only use this template for submitting enhancement requests --What would you like to be modified: image to be based on anything other than Docker Hub sourced Why is this needed: With Docker Hub rate-limit policy in-place, we probably should consider removing this dependency to stay in sync with other base images in Kubernetes project. Folks building their own binaries (i.e. with ) might run into this as well. Notes: Prior discussion happened on An idea from is to use the should probably be renamed to avoid confusion with k/release (h/t Currently, is set to , so migrating to may not be a drop-in move However, , so maybe we don't have to start from scratch /sig testing /area conformance /sig release (cc'ing because builds conformance image, though my quick search seem to show there isn't any build pipelines referencing )
/assign
do we still want to do this? I read somewhere that the images built in Google Cloud are being mirrored from Docker Hub, right? This would mean that the API rate limiting will not hit us any more.
Womp yeah I think DockerHub rate limit is not really the big motivator at this point. However, I think I still want to consider using distroless / go-runner to build and produce conformance image to reduce the noise from CVE scanning done in downstream places. What do you think?
Yeah I think this seems a valuable addition. Are you planning to work on this issue? :)
I will, though probably won't start until next week. If you have ideas and would like to start early feel free to go for it! Otherwise, I'd be happy to work on and deliver this.
Thank you Wilson! Feel free to ping again if you need any assistance. /assign
/triage accepted
one thing I need a little help with is to find a reference for using go-runner image that's used everywhere -- the one that I've found is inside conformance directory which seems to be different from what everyone else is using
We maintain a go-runner in k/release:
oops, I forgot to get back on this last week -- I meant to say that I need a reference for image which I can use as , so something like on Dockerfile. Is that a possible thing to do?


Question: # limitations under the License. ARG BASEIMAGE <ins> ARG RUNNERIMAGE </ins> <del> FROM ${BASEIMAGE} </del> <ins> FROM ${BASEIMAGE} as debbase FROM ${RUNNERIMAGE} # This is a dependency for `kubectl diff` tests COPY --from=debbase /usr/bin/diff /usr/local/bin/ COPY cluster /kubernetes/cluster </ins> COPY ginkgo /usr/local/bin/ COPY e2e.test /usr/local/bin/ COPY kubectl /usr/local/bin/ <del> COPY run_e2e.sh /run_e2e.sh </del> <ins> COPY gorunner /usr/local/bin/kubeconformance # Legacy executables -- deprecated COPY gorunner /run_e2e.sh </ins> COPY gorunner /gorunner <del> COPY cluster /kubernetes/cluster WORKDIR /usr/local/bin </del> ENV E2E_FOCUS="[Conformance]" ENV E2E_SKIP=""

after searching around, I found this -- going to try and use and see how it works :smile:
Started a PR -- plenty of things I'd like to be discussed but we should probably have the discussions there
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale there is work in progress


Question: return nil } <ins> // We don't want to upgrade (add an IP) or downgrade (remove an IP) // following a cluster downgrade/upgrade to/from dual-stackness // a PreferDualStack service following principle of least surprise // That means: PreferDualStack service will only be upgraded // if: // - changes type to RequireDualStack // - manually adding or removing ClusterIP (secondary) // - manually adding or removing IPFamily (secondary) if isMatchingPreferDualStackClusterIPFields(oldService, service) { return nil // nothing more to do. } </ins> // two families or two IPs with SingleStack if service.Spec.IPFamilyPolicy != nil { el := make(field.ErrorList, 0)

1) start with a single stack cluster 2) create a PreferDualStack service 3) configure API server with a 2nd service CIDR 4) Confirm the above service is not dual-stack 5) Add a label to the above service 6) Observe that the service becomes dual-stack I think the correct answer is to have a "repair loop" that upgrades (and downgrades!) PreferDualStack Services to match the config. I think this is a GA blocker.
/triage accepted The current thought is: of type with are special new breed of api objects that I don't think we have any similar precedent in the apis, i could be wrong. Their behavior is tied to how the cluster is configured. Thus their behavior must change as the cluster behavior change. a way the user is stating . If it is cluster then i am good with two ClusterIPs. If it is single stack cluster then I am also good with a single ClusterIP. One can assume that if the user was ok was with getting a additional IP when possible then they must be ok with taking it away when it is no longer possible. Based on the above I believe best course of action is create a new repair loop or piggyback on any of the existing (or better: fix it in rest stack during list and get calls). The new logic takes care of matching to the .
reproducer and regression test if somebody takes this, feel free to add it
ack - Scheduled to tackle 3rd week of May. Will definitely include that patch.


Question: pod.Spec.NodeName = nodeName pod.ObjectMeta.SelfLink = getSelfLink(pod.Name, pod.Namespace) <ins> if pod.Annotations == nil { pod.Annotations = make(map[string]string) } // The generated UID is the hash of the file. pod.Annotations[kubetypes.ConfigHashAnnotationKey] = string(pod.UID) </ins> return nil }

See for more context. Currently, kubelet compares the spec of a mirror pod (created via apiserver) with that of a static pod (config stored as a file) to decide whether the two are considered identical. If they differ, kubelet would delete the mirror pod and re-create it with the correct spec. Due to the fact that apiserver may set default values for some fields in the spec of the mirror pod, the specs could deviate undesirably. To avoid problems like this, we can store the config of a static pod as an annotation in the mirror pod, and compare the static pod with the annotation.
Would a hash of the config be sufficient if we're only going to compare? The whole spec would generally be more useful, but I wonder if that is too much data to put in an annotation.
This seems totally reasonable to me, but I'm still desperately curious why you can't just bypass defaulting when you generate mirror pod from static pod? (i.e. just use the static pod's configuration verbatim)
It's not kubelet who applies the default values here. The admission plugin was only run at the apiserver, and to bypass that, you have to ask apiserver to treat mirror pods differently and potentially at multiple places.
On May 22, 2015 10:38 AM, "Yu-Ju Hong" wrote: Understood, but is it really multiple places? If so then I agree it's a bad idea. Otherwise it might be worth considering as an alternative.
It's untenable for Kubelet to assume to know/circumvent all fields that will be set by other entities: admission control, defaulting, late initialization, auto-something, whatever. It needs to obey the same rules as other clients.
I don't understand why you would ever want to apply any of the things you mentioned to static pods. Static pods should be copied verbatim into the mirror pods, right? That's what I'm proposing.
What if the master is updated before the kubelet, and the master adds a new API field with a default value?
That case should be discussed in
I agree that mirror pod should reflect the true status of the static pod running on the node, and no default values should be set in this case.


Question: pod.Spec.NodeName = nodeName pod.ObjectMeta.SelfLink = getSelfLink(pod.Name, pod.Namespace) <ins> if pod.Annotations == nil { pod.Annotations = make(map[string]string) } // The generated UID is the hash of the file. pod.Annotations[kubetypes.ConfigHashAnnotationKey] = string(pod.UID) </ins> return nil }

However, I think it'd be harder to implement this way since, as stated above, that there are many things to circumvent in the apiserver itself. It could also be harder to keep this special case from being broken by a yet another new feature/plugin. Keeping the hash of the config in the annotation seems like a good compromise even though it may not reflect the true status 100% accurately. I think this is acceptable. What do you think?
Sure, if you think the annotation approach is easier, I would go with that. I guess in the case Brian mentoined (master adds a new API field with a default value), you would not update the mirror pod, because the static pod definition has not changed?
Yes, a hash would be sufficient.
It shouldn't be necessary to update the mirror pod in the case that a new default field was updated, because that field should be populated automatically by the apiserver on decode from etcd.
We should fix this so we don't have this problem again in 1.2.


Question: // More info: https://kubernetes.io/docs/concepts/containers/images#updating-images // +optional ImagePullPolicy PullPolicy `json:"imagePullPolicy,omitempty" protobuf:"bytes,14,opt,name=imagePullPolicy,casttype=PullPolicy"` <del> // Security options the pod should run with. </del> <ins> // SecurityContext defines the security options the container should be run with. // If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext. </ins> // More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ // +optional SecurityContext *SecurityContext `json:"securityContext,omitempty" protobuf:"bytes,15,opt,name=securityContext"`

What happened: states that Container.SecurityContext applies to a Pod, which can be misleading since we already have PodSecurityContext. What you expected to happen: Documentation states that it applies to container How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment:
/sig api-machinery /kind documentation
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Not stale.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle rotten
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err != nil { t.Fatalf("socks5Server: proxy_test: Listen: %v", err) } <del> defer func() { close(closed) l.Close() }() </del> <ins> defer l.Close() </ins> go func(shoulderror bool) { conn, err := l.Accept()

TestRoundTripSocks5AndNewConnection Disabled by No response No response No response /sig
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
there is a race shutting down the socks5 server


Question: if c.SecureServing != nil && !c.SecureServing.DisableHTTP2 && c.GoawayChance > 0 { handler = genericfilters.WithProbabilisticGoaway(handler, c.GoawayChance) } <del> handler = genericapifilters.WithWarningRecorder(handler) </del> handler = genericapifilters.WithCacheControl(handler) handler = genericfilters.WithHSTS(handler, c.HSTSDirectives) if c.ShutdownSendRetryAfter {

I am doing v1.28.3 kube-apiserver testing in the test environment, and I often encounter kube-apiserver panic, especially when kube-apiserver call webhook fails. The following are the captured panic logs: and: kube-apiserver does not panic even if calling webhook times out or fails Deploy a webhook, remove the service corresponding to the webhook, and then create pods in batches to trigger webhook calls No response <details</details<detailsprivate cloud </details<details</details<details</details<details</details<details</details
/sig api-machinery
somewhat related to
The scenarios that trigger panic are similar(the apiserver fails to call webhook), but the log shows that the location of panic is different.
/assign For context of  Thank you. /triage accepted
again, full stack logs:
I'll try to test the mr:
Let me try to explain the panic - both goroutines are trying to write to the of the first goroutine: This request has timed out, and consequently the filter is trying to send a response to the caller. This is the client facing goroutine from the timeout filter. Note that the call to is protected by a mutex, that means that if the other goroutine is using the delegated object via the adapter, then this panic should not happen. second goroutine: let's check the other goroutine, it is also trying to write to the of the object: Now uses an unprotected version of the ResponseWriter object: The filter precedes the filter in the execution order, so the in is not using the thread safe implementation. This is the order today: one quick resolution is to move the after the timeout filter: , but there could be code that precedes that needs to write warning? Also note that the panic mentioned in is not as severe, it does not cause the apiserver to crash. For a more proper solution, i think we would want any thoughts?


Question: if c.SecureServing != nil && !c.SecureServing.DisableHTTP2 && c.GoawayChance > 0 { handler = genericfilters.WithProbabilisticGoaway(handler, c.GoawayChance) } <del> handler = genericapifilters.WithWarningRecorder(handler) </del> handler = genericapifilters.WithCacheControl(handler) handler = genericfilters.WithHSTS(handler, c.HSTSDirectives) if c.ShutdownSendRetryAfter {

/cc
I tend to remove the timeout filter, the sooner the better, and I think the Apiserver panic is the P0-level bug, directly affecting the availability of the cluster I have cherry-picked and tested it in my cluster, and I still need to continue to observe for 1-2 days
I opened a minimal WIP fix for the warning writer in since the timeout filter removal is a much bigger change to reason through
thanks a lot, I will try to test and cherry-pick the above minimal WIP fix in
sorry, I completely overlooked , I only saw and that looked too big to take safely right now I prefer to
Can we backport the fix to 1.28?
Why 1.28 specifically? I'm not aware of anything that changed in 1.28 that made this more or less prevalent. If this meets backport criteria, going to all active branches could make sense (1.27+)


Question: start-kube-controller-manager start-kube-scheduler start-kube-addons <ins> start-cluster-autoscaler </ins> else start-kube-proxy # Kube-registry-proxy.

For some reason they stopped working and we even don't create manifest file for cluster autoscaler. I used the following command to start the cluster:
It's possible it's related to


Question: if "${need_download}"; then if [[ $(which gsutil) ]] && [[ "$kubernetes_tar_url" =~ ^https://storage.googleapis.com/.* ]]; then <del> gsutil cp "${kubernetes_tar_url//'https://storage.googleapis.com/'/'gs://'}" "${file}" </del> <ins> gsutil cp "${kubernetes_tar_url//'https://storage.googleapis.com/'/gs://}" "${file}" </ins> elif [[ $(which curl) ]]; then # if the url belongs to GCS API we should use oauth2_token in the headers curl_headers=""

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --command fails because of I think the link is from k/k's Am I right?? kubetest command succeeds With installed, OR Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release /sig testing
/assign
Need cherry-picks for 1.21, 1.20 to address this. is updated but downloaded downloads and executes its /reopen
Reopened this issue. <detailsIn response to : Need cherry-picks for 1.21, 1.20 to address this. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
this was cherry picked AFAICT, thank you!


Question: func (az *Cloud) EnsureLoadBalancerDeleted(ctx context.Context, clusterName string, service *v1.Service) error { isInternal := requiresInternalLoadBalancer(service) serviceName := getServiceName(service) <del> klog.V(5).Infof("delete(%s): START clusterName=%q", serviceName, clusterName) </del> <ins> klog.V(5).Infof("Delete service (%s): START clusterName=%q", serviceName, clusterName) ignoreErrors := func(err error) error { if ignoreStatusNotFoundFromError(err) == nil { klog.V(5).Infof("EnsureLoadBalancerDeleted: ignoring StatusNotFound error because the resource doesn't exist (%v)", err) return nil } if ignoreStatusForbiddenFromError(err) == nil { klog.V(5).Infof("EnsureLoadBalancerDeleted: ignoring StatusForbidden error (%v). This may be caused by wrong configuration via service annotations", err) return nil } return err } </ins> serviceIPToCleanup, err := az.findServiceIPAddress(ctx, clusterName, service, isInternal) <del> if err != nil { </del> <ins> if ignoreErrors(err) != nil { </ins> return err } klog.V(2).Infof("EnsureLoadBalancerDeleted: reconciling security group for service %q with IP %q, wantLb = false", serviceName, serviceIPToCleanup) if _, err := az.reconcileSecurityGroup(clusterName, service, &serviceIPToCleanup, false /* wantLb */); err != nil { <del> return err </del> <ins> if ignoreErrors(err) != nil { return err } </ins> } if _, err := az.reconcileLoadBalancer(clusterName, service, nil, false /* wantLb */); err != nil { <del> return err </del> <ins> if ignoreErrors(err) != nil { return err } </ins> } if _, err := az.reconcilePublicIP(clusterName, service, nil, false /* wantLb */); err != nil { <del> return err </del> <ins> if ignoreErrors(err) != nil { return err } </ins> } <del> klog.V(2).Infof("delete(%s): FINISH", serviceName) </del> <ins> klog.V(2).Infof("Delete service (%s): FINISH", serviceName) </ins> return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: controller-manager is stuck in deleting an incorrectly configured LB type service. it cannot be recovered until restarted. What you expected to happen: controller-manager should recover. At least it should allow us to create the service without restarting it. How to reproduce it (as minimally and precisely as possible): put a random IP in above yaml and make sure the resource group doen't exist. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.4 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig azure
I also want to add that the above repro is stuck in 403. It's possible for it to be stuck in 404 as well when the SP has the right permission.
Can you share the log for 404?
Thanks for reporting the issue. /assign Do you mean other services couldn't be configured when this happens?
correct.
FYI, we had the same issue when the Idle Timeout annotation was set to "3" which isn't a supported case. Will this PR also cover this edge case ?
Unfortunately , this is still not fixed. The idle timeout is not 403 or 404. We need another fix for this (and possibly more for other annotations)


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package fuzzer import ( "time" "github.com/google/gofuzz" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" runtimeserializer "k8s.io/apimachinery/pkg/runtime/serializer" "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig" "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/v1alpha1" "k8s.io/kubernetes/pkg/kubelet/qos" kubetypes "k8s.io/kubernetes/pkg/kubelet/types" "k8s.io/kubernetes/pkg/master/ports" ) // Funcs returns the fuzzer functions for the kubeletconfig apis. func Funcs(codecs runtimeserializer.CodecFactory) []interface{} { return []interface{}{ // provide non-empty values for fields with defaults, so the defaulter doesn't change values during round-trip func(obj *kubeletconfig.KubeletConfiguration, c fuzz.Continue) { c.FuzzNoCustom(obj) obj.ConfigTrialDuration = &metav1.Duration{Duration: 10 * time.Minute} obj.Authentication.Anonymous.Enabled = true obj.Authentication.Webhook.Enabled = false obj.Authentication.Webhook.CacheTTL = metav1.Duration{Duration: 2 * time.Minute} obj.Authorization.Mode = kubeletconfig.KubeletAuthorizationModeAlwaysAllow obj.Authorization.Webhook.CacheAuthorizedTTL = metav1.Duration{Duration: 5 * time.Minute} obj.Authorization.Webhook.CacheUnauthorizedTTL = metav1.Duration{Duration: 30 * time.Second} obj.Address = "0.0.0.0" obj.CAdvisorPort = 4194 obj.VolumeStatsAggPeriod = metav1.Duration{Duration: time.Minute} obj.RuntimeRequestTimeout = metav1.Duration{Duration: 2 * time.Minute} obj.CPUCFSQuota = true obj.EventBurst = 10 obj.EventRecordQPS = 5 obj.EnableControllerAttachDetach = true obj.EnableDebuggingHandlers = true obj.EnableServer = true obj.FileCheckFrequency = metav1.Duration{Duration: 20 * time.Second} obj.HealthzBindAddress = "127.0.0.1" obj.HealthzPort = 10248 obj.HostNetworkSources = []string{kubetypes.AllSource} obj.HostPIDSources = []string{kubetypes.AllSource} obj.HostIPCSources = []string{kubetypes.AllSource} obj.HTTPCheckFrequency = metav1.Duration{Duration: 20 * time.Second} obj.ImageMinimumGCAge = metav1.Duration{Duration: 2 * time.Minute} obj.ImageGCHighThresholdPercent = 85 obj.ImageGCLowThresholdPercent = 80 obj.MaxOpenFiles = 1000000 obj.MaxPods = 110 obj.NodeStatusUpdateFrequency = metav1.Duration{Duration: 10 * time.Second} obj.CPUManagerPolicy = "none" obj.CPUManagerReconcilePeriod = obj.NodeStatusUpdateFrequency obj.OOMScoreAdj = int32(qos.KubeletOOMScoreAdj) obj.Port = ports.KubeletPort obj.ReadOnlyPort = ports.KubeletReadOnlyPort obj.RegistryBurst = 10 obj.RegistryPullQPS = 5 obj.ResolverConfig = kubetypes.ResolvConfDefault obj.SerializeImagePulls = true obj.StreamingConnectionIdleTimeout = metav1.Duration{Duration: 4 * time.Hour} obj.SyncFrequency = metav1.Duration{Duration: 1 * time.Minute} obj.ContentType = "application/vnd.kubernetes.protobuf" obj.KubeAPIQPS = 5 obj.KubeAPIBurst = 10 obj.HairpinMode = v1alpha1.PromiscuousBridge obj.EvictionHard = map[string]string{ "memory.available":  "100Mi", "nodefs.available":  "10%", "nodefs.inodesFree": "5%", "imagefs.available": "15%", } obj.EvictionPressureTransitionPeriod = metav1.Duration{Duration: 5 * time.Minute} obj.MakeIPTablesUtilChains = true obj.IPTablesMasqueradeBit = v1alpha1.DefaultIPTablesMasqueradeBit obj.IPTablesDropBit = v1alpha1.DefaultIPTablesDropBit obj.CgroupsPerQOS = true obj.CgroupDriver = "cgroupfs" obj.EnforceNodeAllocatable = v1alpha1.DefaultNodeAllocatableEnforcement obj.ManifestURLHeader = make(map[string][]string) }, } } </ins>

Solution: I will add one.
/cc
xref
Thanks for sending a PR so quickly!
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code slush, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: If this issue is not resolved or labeled as by Wed, Nov 22 it will be moved out of the v1.9 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: // We check for double because it needs to support at least the cri-o minimum // plus whatever delta between node usages (which could be up to or at least crioMinMemLimit) func nodesAreTooUtilized(cs clientset.Interface, nodeList *v1.NodeList) bool { <ins> nodeNameToPodList := podListForEachNode(cs) </ins> for _, node := range nodeList.Items { <del> _, memFraction, _, memAllocatable := computeCPUMemFraction(cs, node, podRequestedResource) </del> <ins> _, memFraction, _, memAllocatable := computeCPUMemFraction(node, podRequestedResource, nodeNameToPodList[node.Name]) </ins> if float64(memAllocatable)-(memFraction*float64(memAllocatable)) < float64(2*crioMinMemLimit) { return true }

From After , Multi-AZ Clusters tests in 1.19 and 1.20 started timing out as createBalancedPodForNodes is extremely inefficient in scale: for each node (out of 5k), it lists all pods in a cluster serially (each list takes ~5s), which translates to ~7 hours. Optimizing this function by listing pods once and then in memory by Spec.NodeName /kind bug /priority critical-urgent /sig scheduling
/triage accepted
/assign
/reopen Looks that ComputeCPUMemFraction took only 12 minutes, but it stuck later, on It is scheduling one pod per each node: Which is waiting 5s to be ready: 5k nodes * 5s = ~7h
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Can you do the pod creations in parallel?
Yes, good idea. I'll update to do that


Question: } func TestCreate(t *testing.T) { <del> simpleStorage := &SimpleRESTStorage{} </del> <ins> wait := sync.WaitGroup{} wait.Add(1) simpleStorage := &SimpleRESTStorage{ injectedFunction: func(obj runtime.Object) (returnObj runtime.Object, err error) { wait.Wait() return &Simple{}, nil }, } </ins> handler := Handle(map[string]RESTStorage{ "foo": simpleStorage, }, codec, "/prefix/version", selfLinker)

Intermittent failures on Travis.
Hrm, this passes 100/100 locally for me. I'll look at the code and see if I see anything. --brendan


Question: // records an event using ref, event msg.  log to glog using prefix, msg, logFn func (m *imageManager) logIt(ref *v1.ObjectReference, eventtype, event, prefix, msg string, logFn func(args ...interface{})) { if ref != nil { <del> m.recorder.Event(events.ToObjectReference(ref), eventtype, event, msg) </del> <ins> m.recorder.Event(ref, eventtype, event, msg) </ins> } else { logFn(fmt.Sprint(prefix, " ", msg)) }

Is this a BUG REPORT or FEATURE REQUEST?: What happened: Now the and has been consistent in by PR . There is no need to convert v1.ObjectReference to clientv1.ObjectReference in . What you expected to happen: Remove this function and cleanup the codes which call this function. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Will submit the cleanup after PR merged.
/sig api-machinery
/sub


Question: if _, ok = t.objects[gvr][namespacedName]; ok { if replaceExisting { for _, w := range t.getWatches(gvr, ns) { <del> w.Modify(obj) </del> <ins> // To avoid the object from being accidentally modified by watcher w.Modify(obj.DeepCopyObject()) </ins> } t.objects[gvr][namespacedName] = obj return nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Hello, Found a potential data race in testing. There is a test If we update it like this (we actually can do it ): If watcher updates received object we get a data race. No data race My suggestion: send copy of object here:
/sig api-machinery
/cc /triage accepted Do you remember if we fixed similar issues before?
Hi ,The code avoiding data race seems to already exist If you commented the code on above, the following test case will pass (because of data race), so I think there is no data race in the code : )
Hi Did you run my test with flag? Of course if I comment there will be data race. But let's take a look at: We store in and do with the same . So, watcher receives that is stored in map. we call we take the same from : Therefore, if watcher modifies received in another goroutine we have data race. Output:
Hi You're right ,thank you for solving my confusion. One solution could be to copy of , I will try to fix it.


Question: serviceAccountName: fluentd-gcp-scaler containers: - name: fluentd-gcp-scaler <del> image: gcr.io/google-containers/fluentd-gcp-scaler:0.1 </del> <ins> image: gcr.io/google-containers/fluentd-gcp-scaler:0.2 </ins> command: - /scaler.sh - --ds-name=fluentd-gcp-v3.0.0

Forking from To summarize, here's what we observed: daemonset calls coming every minute from both fluentd-scaler and addon-manager (verified this by individually turning them on/off). Things we need to understand here: Is the daemonset object continuously toggling b/w 2 states? (we know here that it's RV is increasing continuously) If yes, what field(s) in the object are changing? IIRC value of some label/annotation (I think 'UpdatedPodsScheduled') is changing (probably related to 2 below) Also, why/should the fluentd-scaler send any api request if the resources are already set to the right value? pods are getting deleted and recreated by daemonset-controller when the scaler is enabled (as was also seen in ). Why this is happening? One thing to note here - all those delete calls are preceded by calls from respective kubelets (but maybe that's expected). cc /priority critical-urgent /assign
Marked this as release-blocker as at least some part of it seems like a correctness issue (i.e fluentd pods dying and getting recreated). Note also that this is causing increased mem-usage with daemonset-controller (depicted here - )
Thanks for filing this! The only thing that was changing (apart from ds/fluentd-gcp status) was the resource version.
I modified fluentd scaler to use --dry-run in to check whether there would be a change, but this still keeps on updating, even though there is no change. Apparently, the logic inside kubectl is broken...
Btw, addon manager sends tons of PATCH requests for all resources every minute anyway, so even though I will fix fluent-scaler not to send them, it won't solve everything.
Should this issue actually be closed? It was auto-closed by the bot, but nobody has confirmed that the actual issue is fixed.
I'll let to confirm (looks like he'll be back on Monday)
/status in-progress
Yes, we should keep this issue open until we fully understand what's going wrong and fix the root-cause. I believe much of the discussion wrt this is happening under related issue
ACK. In progress ETA: 20/03/2018 Risks: May delay the release if not satisfactorily resolved
Need a number of successive green runs, then close.


Question: serviceAccountName: fluentd-gcp-scaler containers: - name: fluentd-gcp-scaler <del> image: gcr.io/google-containers/fluentd-gcp-scaler:0.1 </del> <ins> image: gcr.io/google-containers/fluentd-gcp-scaler:0.2 </ins> command: - /scaler.sh - --ds-name=fluentd-gcp-v3.0.0

So I took a look and seems like the fluentd-scaler version bump (which was supposed to fix) was merged b/w runs and of our . However, I don't see any visible improvement in controller-manager memory usage around that time: !
Also, I'm noticing such lines occasionally (once every 5-10 secs in the start and once every ~1min later on) from of latest run: And back when the fluentd-scaler wasn't doing anything useful (i.e before ), the had such lines only sparingly.
Now it isn't even sending requests (other than GET) to the apiserver. How do you generate that graph? Is it 99th percentile of memory usage over time? Or what exactly? One difference after introducing fluentd-gcp-scaler is that now there is a change occurring on the DaemonSet (once, but still). Also, what test is failing because of this? I'd like to take a look into the logs.
Answering your questions: I fetched the ResourceUsageSummaries for density test from historical runs (e.g ) and then obtained the 99% mem-usage for controller-manager for each run. Yes. More precisely, we're collecting mem-usage from controller-manager once every 1 min (or sth like that) and then taking 99th %ile of the resulting sample. Density and load tests. Where we're checking that mem usage of controller-manager is less than the given threshold (i.e 500MB). Ref: The tests are occasionally failing due to just crossing the threshold. Overall, the mem-usage had seen an increase back during the scaler change (see graph above). The job is this - https://k8s- Inside , you'll find all necessary logs (also a file named which has the resource-usage data).
The fluentd-scaler update eliminated fluentd restarts problem. Verified from controller-manager logs that we no longer see such logs that we were : This is great, and I believe that moves this out of release-blocker as we no longer face problems I mentioned in
That said, and me will continue looking into increase in mem-usage (which is not that serious). We'll start by verifying that turning the scaler off improves mem-usage.
Feel free to override me if you have a good reason :)
I agree that this doesn't sounds to be release-blocker.


Question: serviceAccountName: fluentd-gcp-scaler containers: - name: fluentd-gcp-scaler <del> image: gcr.io/google-containers/fluentd-gcp-scaler:0.1 </del> <ins> image: gcr.io/google-containers/fluentd-gcp-scaler:0.2 </ins> command: - /scaler.sh - --ds-name=fluentd-gcp-v3.0.0

But I would like to see follow-ups at least for issues like: does this memory increase come from? what was claimed to be "no-op" changes for daemon set were causing problems [is that issues in kubectl, apiserver, somewhere else?] Something like that may hit us again in the (near?) future if we won't understand it now. The current solution is really volatile [if there is a bug somewhere we shouldn't workaround it on our side, but instead have this fixed] at the cluster start (maybe it's fine to leave as is, but I would like to see a discussion of that and a conscious decision being made here)
Definitely agree those are questions we need to answer.
Yup. 1) is tested in . 2) I believe apiserver sends watch events even when only resource version changes, which causes the daemonset controller to recreate the pods. So either: resource version shouldn't change when there is no change anywhere else in the resource apiserver shouldn't send watch events when only resource version is changing daemonset should check whether watch event it got is actually changing anything 3) Isn't that an inherent part of the way addon manager works? It creates all sorts of resources, some dependent on the other (e.g. cluster roles and cluster role bindings), all at once. If some dependency is not satisfied, it will get fixed during the next run.
This is the code that is supposed to protect us from situation that something must change when we write object to etcd (resource version is not stored in etcd): So there should never be a situation that only resource version is changing. If it is, there is bug and need to be fixed. - FYI Reconciliation is a different thing that creating something in one state only to change that to a different state by a different component. I'm not saying this is for sure wrong approach. But I would like to see a conscious decision process discussing pros-and-cons of that.
That's what I'd expect as well. This I'm not sure if should be correct behaviour as IIUC watchers need to receive every RV update for correctness reasons ( ) Maybe. But are we sure if RV is just increasing without any object change? For creation of addons, yes (that race will likely last only for starting few seconds).


Question: serviceAccountName: fluentd-gcp-scaler containers: - name: fluentd-gcp-scaler <del> image: gcr.io/google-containers/fluentd-gcp-scaler:0.1 </del> <ins> image: gcr.io/google-containers/fluentd-gcp-scaler:0.2 </ins> command: - /scaler.sh - --ds-name=fluentd-gcp-v3.0.0

But this is wrt addon changing cluster resources, which can potentially create race at arbitrary time after cluster start (though your fix may have gotten rid of this).
Ok.. Seems like I missed Wojtek's comment above.
So I just checked the ds-controller code and it seems like we're updating the ds status only if this condition is being violated: Which means one of those fields is probably mismatching. I'll try to add some logs and run the test at --v=4 to get more info.
anything is possible, but we do have tests ensuring this works properly (no-op updates do not actually result in an update+resourceVersion bump in etcd):
So I ran some 100-node experiments against with more debug logs and I think I now understand the reason for this. The controller-manager logs and mainly show the following difference: This logs show-up when the scaler is active. And these pod deletions are happening as part of rolling-upgrade of daemonset to newer version (the one having resources set). You can see they're getting deleted one-by-one, which also explains why it takes so long (and so are happening even 30 mins after the cluster creation). And the reason why they're deleted one-by-one is because ds-controller has a rolling-upgrade strategy where by default. Proof: And the increased mem-usage FMU is because of the rolling-upgrade performed by ds-controller (over the prolonged period) - as it's the only difference I observe from logs Further, the reason why the RV was increasing was because of few fields 'NumUnavailable', 'NumAvailable', etc fluctuating while the rolling-upgrade
I spoke offline with and suggested that we should increase maxUnavailable for the fluentd daemonset to a large enough value so that we're not bottlenecked by it. My reasoning is: fluentd is not some user workload which will have some service downtime in case there are many unavailable at once fluentds on nodes are independent of each other (i.e one fluentd shouldn't need to wait for another node's fluentd to come up) we should try to make the scaling happen as fast as possible - otherwise it may be too late before they cope-up with increased log-load in the cluster I'm going to make that change and test it against my PR (thanks for pointing that we can change maxUnavailable directly in ds config).


Question: serviceAccountName: fluentd-gcp-scaler containers: - name: fluentd-gcp-scaler <del> image: gcr.io/google-containers/fluentd-gcp-scaler:0.1 </del> <ins> image: gcr.io/google-containers/fluentd-gcp-scaler:0.2 </ins> command: - /scaler.sh - --ds-name=fluentd-gcp-v3.0.0

So increasing maxUnavailable indeed seems to help (ref: ): All the 101 replicas were deleted at once instead of one-by-one and happened pretty much within 2 mins of the original fluentd-ds seen by the controller: We should get that PR into 1.10.
Thanks for confirming this! Did increasing maxUnavailable also reduce c-m memory usage?
So the run was still over the threshold, but I was expecting it as I ran c-m with increased verbosity (which ). That said, I'll observe how the usages are once that PR gets in - and FMU it's still possible to have higher values due to the rolling-upgrade potentially happening during the test.
Is there anything else to do here or should this be closed now?
Let's close it once we're sure there are no more such failures. We may also need to bump the threshold. On Mon, Mar 26, 2018, 9:46 AM Kubernetes Submit Queue < :
So bumping maxUnavailable for fluentd rolling-upgrade in is causing couple of issues: Our latest 5k-node performance failed as part of BeforeSuite where we're checking that ~all kube-system pods are running or ready. The reason is because quite some fluentd pods were killed as part of the rolling-upgrade and we timed out before all those pods could come up Our 100-node performance runs due to high CPU/mem-usage of c-m observed during density test, in cases when the fluentd upgrade happens during the test - which is kind of expected So at this point, we might want to choose among one of the 2 options: Rollback , which comes at the cost of tradeoffs I explained in Increase our CPU/mem thresholds to eliminate those flakes in 100-node job and increase the "max-allowed-not-ready-pods" in kube-system ns to a higher value, so that our 5k-node test doesn't fail in BeforeSuite Thoughts?
TBH, I don't know why it was merged without waiting for opinions... I really vote for reverting that. We don't want this - this may mean making cluster "not fully usable" from user perspective.Which is not really what we want to test.
SGTM. Let's revert it and increase our thresholds for the time-being. The discussion about changing fluentd rolling-upgrades can continue to happen in the background.


Question: serviceAccountName: fluentd-gcp-scaler containers: - name: fluentd-gcp-scaler <del> image: gcr.io/google-containers/fluentd-gcp-scaler:0.1 </del> <ins> image: gcr.io/google-containers/fluentd-gcp-scaler:0.2 </ins> command: - /scaler.sh - --ds-name=fluentd-gcp-v3.0.0

[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 5 days, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
This issue is now fixed after my above PRs. We're having continuous for the job


Question: func RunEdit(f *cmdutil.Factory, out, errOut io.Writer, cmd *cobra.Command, args []string, options *EditOptions) error { var printer kubectl.ResourcePrinter var ext string <ins> var addHeader bool </ins> switch format := cmdutil.GetFlagString(cmd, "output"); format { case "json": printer = &kubectl.JSONPrinter{} ext = ".json" <ins> addHeader = false </ins> case "yaml": printer = &kubectl.YAMLPrinter{} ext = ".yaml" <ins> addHeader = true </ins> default: return cmdutil.UsageError(cmd, "The flag 'output' must be one of yaml|json") }

Recently I was experimenting with changing for a Golang pre-purposed faux editor when editing resources when I noticed that it couldn't be marshaled. Upon inspecting the output JSON I noticed it had the same comments that the default yaml version has. Valid JSON cannot have comments. Since the comment also gives warning / error information upon saving it must be decided where to put that in the JSON output. Perhaps a new JSON property?
Maybe just not save the comments as part of the json to get rid of the warning, but still should them?
Is this a 1.2 thing? I do not see any errors when running the two commands. I made an edit in one run, and did not make an edit in a separate run. Maybe I don't understand the problem that is seen.
there isn't a kubernetes error that is shown, the issue is that the output JSON that kubernetes generates is actually invalid JSON due to the comments at the top. This caused errors when I was trying to unmarshal that JSON in another project.
Okay. Got it. The editor is being very strict about json having no comments.
It's not only about editor, it's simply part of language specification. Explanation for this: IMO if we want to use format, we should comply with it's standard. Also currently used sign for comments is unusual in javascript world, from which json evolved. In this world comments are marked in "C++ like" style (what in early days of JSON was also supported), i mean using and . So IMO in time of converting into JSON - we should strip such comments (I would prefer that), or they should be in some documented and specific to kubernetes way into JSON object.


Question: dns_replicas: '$(echo "$DNS_REPLICAS" | sed -e "s/'/''/g")' dns_server: '$(echo "$DNS_SERVER_IP" | sed -e "s/'/''/g")' dns_domain: '$(echo "$DNS_DOMAIN" | sed -e "s/'/''/g")' <ins> instance_prefix: '$(echo "$INSTANCE_PREFIX" | sed -e "s/'/''/g")' </ins> EOF # Configure the salt-master

reported a variety of failure modes in .
+1
I figured I'd add to this issue rather than create a new one. I attempted to create a vagrant cluster today (dev tree is up to date) with 1 Master and 1 Minion and the cluster creation is stuck in the 'Validating master' loop ( for over 10 minutes in the validating loop ). salt-master and salt-minion logs seem to be empty ( they don't appear to be running, in fact ).
I still have the cluster running in case anybody wants additional information.
/cc please let me know if there is any other information about my vagrant setup that I need to post here.
I have been working to get more stability in this environment over last few days. Can you try again from HEAD after this merges:
Can you also verify that you built a release prior to running cluster/Kube-up?
yes, I started from scratch and built a release before running cluster/kube- - cluster/kube- seems to fail fairly early in case there is no release build available.
It looks like got merged. I'll try again and get back to you.
The vagrant cluster seems to come up ( completes master and minion validation ). However, there is new credentials related error that is showing up at the end of running cluster/kube- :
I'll add one more problem I've encountered. Runing from head:
New to me. Will investigate more tomorrow. Sent from my iPhone
Also new to me :-) - will see if I can reproduce either. Sent from my iPhone
Your credendials and certificates are not working with new environment. Just delete your .kubernetesvagrant* files and recreate the cluster. It would be nice to automate this task, if you tear down the cluster.
I am able to reproduce the 502 Bad Gateway error and am working on determining what changed in order to provide a resolution over the last couple days. Please assign this issue to me.
The issue appears to me that the kube-apiserver is not getting provisioned by salt correctly. Parsing Salt logs to determine why.
Looks like the file is not getting laid down by salt for some reason ;-)
The Jinja template was changed as part of identifying the cluster. Fixing up now, and will send a PR to close this issue.
that was it, thanks!


Question: "k8s.io/kubernetes/pkg/controlplane" "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/test/integration/framework" <ins> "k8s.io/utils/ptr" </ins> ) const (

My application is clustered and I have 2 pods running. I am using a rolling upgrade strategy and my partitioning number is 1. First I need to upgrade the pod with the ordinal number 1 so I triggered that upgrade with partitioning number 1 and my upgrade gets completed. After this, my pod having the ordinal number 0 gets restarted and it's automatically upgraded to a new image. This is not an expected behavior because still, my partition number is 1 and pod-0 should come up with an older image only. I am facing this issue for the v1.27 cluster. Prior to this, it was working fine. pod-0 should not upgrade to new image until I upgrade the partition number to 0.a Do helm install and upgrade pod-1 only keeping partition number to 1. No response <detailsconsole $ cat /etc/os-release $ uname -a C:wmic os get Caption, Version, BuildNumber, OSArchitecture </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig/api-machinery
/sig apps /sig storage
Hi, Can someone suggest here? As this is introduced in my setup after the v1.27 upgrade? Is there any change in partition logic?
This can be reproduced by using the said STS. 1 - Deploy STS using attached that has a replica count is 2 and updatingstrategy.rolling.partition as 1. 2 - Do edit STS and update the image to 1.15. This will upgrade, pod-1 with image 1.15. 3 - Next delete the pod pod-0 and check that POD-0 came up with a new image which is not the expected behavior. POD 0 should not be updated to a new image until the partition is changed to 0. We have noticed this behavior change in v1.27.


Question: "k8s.io/kubernetes/pkg/controlplane" "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/test/integration/framework" <ins> "k8s.io/utils/ptr" </ins> ) const (

Pla suggest : apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: ports: port: 80 name: web clusterIP: None selector: app: nginx apiVersion: apps/v1 kind: StatefulSet metadata: name: web-am spec: updateStrategy: type: RollingUpdate rollingUpdate: partition: 1 serviceName: "nginx" replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: name: nginx args: /bin/sh -c sleep 30;touch /tmp/healthy; sleep 3600 image: nginx:1.14.2 ports: containerPort: 80 name: web


Question: // If "only_cpu_and_memory" GET param is true then only cpu and memory is returned in response. func (h *handler) handleSummary(request *restful.Request, response *restful.Response) { onlyCPUAndMemory := false <del> request.Request.ParseForm() </del> <ins> err := request.Request.ParseForm() if err != nil { handleError(response, "/stats/summary", errors.Wrapf(err, "parse form failed")) return } </ins> if onlyCluAndMemoryParam, found := request.Request.Form["only_cpu_and_memory"]; found && len(onlyCluAndMemoryParam) == 1 && onlyCluAndMemoryParam[0] == "true" { onlyCPUAndMemory = true } var summary *statsapi.Summary <del> var err error </del> if onlyCPUAndMemory { summary, err = h.summaryProvider.GetCPUAndMemoryStats() } else {

What happened: In handler#handleSummary, return value from request.Request.ParseForm() is ignored. What you expected to happen: The error return should be checked.
/sig node
Hi this seems to be easy to fix. Could I solve this?
Please go ahead.


Question: <del> apiVersion: extensions/v1beta1 </del> <ins> apiVersion: networking.k8s.io/v1beta1 </ins> kind: Ingress metadata: name: echomap

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority important-soon /sig network
Ref:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
my fu points me to this commit [] Fix multiple conversion tests that belongs to this PR , that seems related can you confirm if that PR is causing this test to fail? (should we gate on this test?)
Yes, that PR is related to the failure. The test utility is trying to convert extensions/v1beta1 data read from various files directly to a go type. This was never a valid thing to do, and happened to silently succeed because the two structs had an identical structure. This is exactly the type of bug was designed to flush out
the correct fix is to update the test/e2e/testing- files to
/assign
It was my PR that broke this - I can also fix that.
Thanks, o it was the commit that was good but the test that was wrong :upsidedownface: , At least I got right the commit :sweat_smile:
is out for review (talked with offline, he was aware that I'll send that out)


Question: import ( "context" <ins> "errors" </ins> "fmt" "io/ioutil" "net/http"

In we discovered a bug where kube-apiserver metrics were being incorrectly reported (GET vs LIST problem). In this was addressed together with test exercising that proper metrics are exposed. However, this test isn't ideal because if params wouldn't be swapped, it wouldn't catch them. We should ideally develop a lower-level unit test for mapping url paths to exact values of labels/parameters. /sig apimachinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : In we discovered a bug where kube-apiserver metrics were being incorrectly reported (GET vs LIST problem). In this was addressed together with test exercising that proper metrics are exposed. We should ideally develop a lower-level unit test for mapping url paths to exact values of labels/parameters. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
are you planning on sending a fix for this one too? Thanks!
Not very soon - if someone else could help with it, it would be great.
/assign
/triage accepted
cc


Question: return true } } <ins> for _, c := range pod.Spec.InitContainers { if c.Name == containerName { return true } } </ins> return false }

Need to finish so the client allows you to select an init container, and verify the kubelet works correctly.
(this is a nice intersection of new in 1.3, node and usability, if anyone wants to give it a shot. Pretty sure Clayton is busier than I am and I don't think I'll get to it)
Will be happy to review any changes - should be fairly straightforward.
Yes, init containers are "run to completion" always. , harryz wrote:
IIUC when the init containers of pod have not complete, the pod should be in status, which means will fail, if init containers run to completion, they will exit, still fail. I don't know what you mean exec should work on init containers?
the init container is still up and running even when the pod is in pending, we should be able to exec and debug why it's hung, for example.
add a straightforward exec implementation ptal.


Question: "github.com/Azure/go-autorest/autorest/to" "github.com/stretchr/testify/assert" "k8s.io/api/core/v1" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> ) func TestFindProbe(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When service is set with ILB annotation and empty string subnet, e.g. The service would be pending forever with following error: What you expected to happen: ILB should be provisioned successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /kind bug /assign
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
Thanks


Question: }, nil } <del> func (c *configFactory) getNextPod() *v1.Pod { pod, err := c.podQueue.Pop() if err == nil { klog.V(4).Infof("About to try and schedule pod %v/%v", pod.Namespace, pod.Name) return pod } klog.Errorf("Error while retrieving next pod from scheduling queue: %v", err) return nil } </del> // assignedPod selects pods that are assigned (scheduled and running). func assignedPod(pod *v1.Pod) bool { return len(pod.Spec.NodeName) != 0

PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. Main tracking issue: The method of is a simple wrapper around . It should be converted into a simple closure instead of being a method. i.e. we should have, (code is in ) /sig scheduling /kind cleanup /help
I'd like to take this task. /assign /remove-help
Thanks, go for it! :smile:


Question: CGROUP_ROOT=${CGROUP_ROOT:-""} # name of the cgroup driver, i.e. cgroupfs or systemd CGROUP_DRIVER=${CGROUP_DRIVER:-""} <ins> # owner of client certs, default to current user if not specified USER=${USER:-$(whoami)} </ins> # enables testing eviction scenarios locally. EVICTION_HARD=${EVICTION_HARD:-"memory.available<100Mi"}

Encountered this error while running on master: Should we set to by default? cc
Why is it empty? Would be better?
seems to expect $(whoami) based on the value of that variable for my machine. I can make that change if you would like.
Can you keep as an option if it is specified? returns a 6 digit numeric value on my machine (corporate provisioned) and that is incompatible with GCI images, as well as error-ing out during e2e provisioning.
that is a great idea, so should be defaulting to whoami in the case where it has not been defined.
OK, will file an PR to fix this.
Filed PR at . PTAL


Question: return nil, err } <del> return newRuntimeVersion(typedVersion.Version) </del> <ins> return newRuntimeVersion(typedVersion.RuntimeVersion) </ins> } // APIVersion returns the cached API version information of the container

Is this a BUG REPORT or FEATURE REQUEST? : BUG: With CRI enabled by default, kubelet reports the version of CRI instead of container runtime version: Kubernetes version (use ): Master and 1.6. What you expected to happen: Kubelet should report the version of container runtime. Will file a PR to fix this. cc


Question: // use kubeletServer to construct the default KubeletDeps kubeletDeps, err := app.UnsecuredDependencies(kubeletServer) <ins> if err != nil { die(err) } </ins> // add the kubelet config controller to kubeletDeps kubeletDeps.KubeletConfigController = kubeletConfigController

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Testing bringing up k8s cluster using kubeadm from latest k8s code from master branch. kubelet is not coming up and below is the errors from syslog: upon looking at the code from where the error is happening, it is evident that the err checking is missing. What you expected to happen: deployment to succeed. How to reproduce it (as minimally and precisely as possible): Use the latest code from k8s and build a k8s release and bring up cluster using kubeadm Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration**: GCE OS (e.g. from /etc/os-release): 16.04 LTS (Xenial Xerus) Kernel (e.g. ): Linux e2e--master 4.4.0-21-generic -Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Others: /sig node /cc
PTAL


Question: obj.CgroupDriver = "cgroupfs" } if obj.EnforceNodeAllocatable == nil { <del> obj.EnforceNodeAllocatable = defaultNodeAllocatableEnforcement </del> <ins> obj.EnforceNodeAllocatable = DefaultNodeAllocatableEnforcement </ins> } }

Solution: I will add one.
/cc
xref
Thanks for sending a PR so quickly!
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code slush, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: If this issue is not resolved or labeled as by Wed, Nov 22 it will be moved out of the v1.9 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: package testutil import ( <ins> "fmt" </ins> "io" "github.com/prometheus/client_golang/prometheus/testutil" <ins> apimachineryversion "k8s.io/apimachinery/pkg/version" </ins> "k8s.io/component-base/metrics" )

What would you like to be : Once a metric be deprecated, we should update the unit test accordingly if there is any. We need to work out a way for a unit test freely select codebase version, and also should make sure the unit test won't fail in the subsequent release. Why is this needed: Deprecated also needs to be protected. /sig instrumentation /assign


Question: // alpha: v1.4 ExternalTrafficLocalOnly utilfeature.Feature = "AllowExtTrafficLocalEndpoints" <del> // owner: @saad-ali // alpha: v1.3 DynamicVolumeProvisioning utilfeature.Feature = "DynamicVolumeProvisioning" </del> // owner: @mtaufen // alpha: v1.4 DynamicKubeletConfig utilfeature.Feature = "DynamicKubeletConfig"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kube-apiserver --help still shows this: It looks like the related feature can be turned off. However, dynamic volume provisioning Alpha has retried in 1.6. What you expected to happen: Remove from feature gate to avoid confusion. How to reproduce it (as minimally and precisely as possible): Run Anything else we need to know?: No Environment: Kubernetes version (use ): 1.7+ Cloud provider or hardware configuration**: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
/sig api-machinery
SIG API Machinery doesn't own the contents of the API. /sig storage


Question: // For each groupversion served by our resourcemanager, create an APIService // object connected to our fake APIServer <ins> var groupVersions []metav1.GroupVersion </ins> for _, versionInfo := range basicTestGroup.Versions { groupVersion := metav1.GroupVersion{ Group:   basicTestGroup.Name,

reported that conformance testing with the latest version of sonobuoy against a version 1.29.0-alpha.3 cluster fails on the support the 1.17 Sample API Server using the current Aggregator test: , related to I found the reason why the test is working fine in Kubernetes CI jobs but fails with sonobuoy is because the service account used by sonobuoy "sonobuoy-serviceaccount" doesn't have read permission of the non-resource url "/", which is used by the test to confirm the removal of group path: The permission of "sonobuoy-serviceaccount" is as below: There may be two ways to fix it: the RBAC rule of sonobuoy to include "get" of "/". However, could other test runners encounter this issue? Is there a doc listing the required permissions for running e2e tests? the e2e to skip checking the removal of the group path if the service account doesn't have permission to access the root path. Which way do we usually prefer? Running e2e with sonobuoy should succeed. a v1.29.0-alpha.3 cluster. No response <details</details<details</details<details</details<details</details<details</details<details</details
implemented this solution.
/sig api-machinery
thank you for opening this issue and working on a solution. I'm probably not the right person to decide on which solution is preferred. From my perspective, either solution is fine.
Note: Sonobuoy is not owned by any Kubernetes SIG. At most it is between SIG Architecture and the CNCF to choose whether to recommend it for running tests. SIG Testing owns the e2e image that sonobuoy can be used to schedule but not sonobuoy. IMHO this is a bug in running the tests via a pod, which is not how we develop the tests (they're run outside the cluster, by a cluster admin account typically). I would say 1) is the correct answer. xref: /sig architecture /sig testing /triage accepted cc
thanks for your comment. I guess 1) means non of the existing sonobuoy versions can pass K8s conformance test against a cluster 1.29.0-alpha3, as the RBAC rule is hardcoded.


Question: // For each groupversion served by our resourcemanager, create an APIService // object connected to our fake APIServer <ins> var groupVersions []metav1.GroupVersion </ins> for _, versionInfo := range basicTestGroup.Versions { groupVersion := metav1.GroupVersion{ Group:   basicTestGroup.Name,

I will file an issue to sonobuoy first.
question here is what are the set of client permissions required to run the conformance suite? perhaps a level of access or a more granular recommendation should be documented. what is your take on the root path requirement as approver for: EDIT: slack thread:
I agree, we already require e.g. privileged pods for a long time in conformance tests and we should make it clear that we expect a highly privileged client.
+1 for what is saying, that we should expect that the user running the tests has required permissions. These tests are not built for being run unprivileged.
i wanted to PR the admin note in our conformance docs under sig-arch, but then i saw this: so broke this guarantee. sounds like SIG Arch needs to comment.
This statement surprises me. Root discovery is one of the least-privileged permissions you can have on a cluster. For years this was even open for unauthorized users on the internet (if your api endpoint has been exposed). In other words, just add the permission. While thinking about it, it is surprising that the user in question does not have this access already. Don't we allow that for every authorized user?
In contrast, I hope that lots of privileged operations are tested by conformance today. If we don't test things like admission webhooks for example, we have much bigger problems. And admission webhooks are basically equivalent to cluster-admin.
that is outdated, we use hostNetwork pods one some of the tests IIRC
looks like we don't. before this ticket, i was under the impression that the conformance suite already requires admin and that all testers (like Sonobuoy) use admin level credentials. it is non-privileged (e.g., does not require root on nodes, access to raw network interfaces, or cluster admin permissions) looks like we may have to change that sentence completely.
Would have expected it here: The PR in question explicitly fixes the result returned by . If that is part of conformance (IMO it should), every conformance test needs access it obviously. Whether we require admin, or we add it manually, either would work.


Question: // For each groupversion served by our resourcemanager, create an APIService // object connected to our fake APIServer <ins> var groupVersions []metav1.GroupVersion </ins> for _, versionInfo := range basicTestGroup.Versions { groupVersion := metav1.GroupVersion{ Group:   basicTestGroup.Name,

adding to milestone and making release blocking ... we should not release 1.29 in a state that the most commonly used conformance tool will fail
IMO we should clarify the state and go fix the tool, but we should keep the test and we should NOT add skip. Conformance docs are not a release artifact, and sonobuoy release lifecycle is not a conformance artifact. If we can't get sonobuoy working in a reasonable time, we already know the tests can be run by scheduling the image (which is ours) directly with kubectl instead and we can document that in the upstream conformance docs. We need to fix the for this image: ,%20master%20(dev) And it should probably be to release-blocking in the future.
it looks like the e2e change made in was not consciously made part of conformance, it was just trying to functionally test the bugfix being made. For 1.29, I think we should revert the change to and make an integration test that validates the fix instead. We can revisit whether content of is part of conformance or not for 1.30
I opened to revert the e2e change and move the functional test to an integration test
Filed to try to get the image job passing again.
We probably need another issue to track 1.30 follow-up?
If no one has done it, I could create another issue and follow up with the items that have been discussed here: outdated privilege requirement in commonly used conformance tools like sonobuoy aware of the required priviledge to avoid similar situations in the future. permission of GET "/" to authorized users via the role for root discovery.


Question: fi done <ins> errors_expect_error=() for file in "${all_e2e_files[@]}" do if grep "Expect(.*).To(.*HaveOccurred()" "${file}" > /dev/null then errors_expect_error+=( "${file}" ) fi done </ins> if [ ${#errors_expect_no_error[@]} -ne 0 ]; then { echo "Errors:"

What happened: After fixing golint failures under (now still in progress), we need to put for each and . Then common error checking code has become like: We can replace this code with which is more readable and easy to be understood what the code does. Now we have following e2e test directories which can be replaced: [x] test/e2e/apimachinery/ () [x] test/e2e/apps/ () [x] test/e2e/auth/ () [x] test/e2e/common/ () [x] test/e2e/node/ () [x] test/e2e/storage/ () [x] () [x] () [x] () [x] () [x] () [x] () [x] [x] test/e2e/storage/persistentvolumes- [x] () [x] () [x] [x] [x] [x] [x] ()
/sig testing /kind cleanup
/assign PR will be submitted soon.
Fix the following in
Fix in
Fix in
Hi, maybe I could take files beneath test/e2e/storage/vsphere/
oops, I'm finished and prepare for PR.
Thank you That's OK
Fix the following in
/assign
All related PRs were merged. I think this ticket can be closed. BTW, should we also replace to ?
Thanks for your work. thing is managed with We are almost finishing the work also :-)
/remove-kind bug
I found in these files. I'm working on :
I can work to the following item.


Question: func (p *PriorityQueue) MoveAllToActiveQueue() { p.lock.Lock() defer p.lock.Unlock() <ins> // There is a chance of errors when adding pods to other queues, // we make a temporary slice to store the pods, // since the probability is low, we set its len to 0 addErrorPods := make([]*framework.PodInfo, 0) </ins> for _, pInfo := range p.unschedulableQ.podInfoMap { pod := pInfo.Pod if p.isPodBackingOff(pod) { if err := p.podBackoffQ.Add(pInfo); err != nil { klog.Errorf("Error adding pod %v to the backoff queue: %v", pod.Name, err) <ins> addErrorPods = append(addErrorPods, pInfo) </ins> } } else { if err := p.activeQ.Add(pInfo); err != nil { klog.Errorf("Error adding pod %v to the scheduling queue: %v", pod.Name, err) <ins> addErrorPods = append(addErrorPods, pInfo) </ins> } } } p.unschedulableQ.clear() <ins> // Adding pods that we could not move to Active queue or Backoff queue back to the Unschedulable queue for _, podInfo := range addErrorPods { p.unschedulableQ.addOrUpdate(podInfo) } </ins> p.moveRequestCycle = p.schedulingCycle p.cond.Broadcast() }

What happened: We will now always delete the pods from unschedulable queue no matter the Add operations succeed or not in function. What you expected to happen: Delete pods from unschedulable queue only when add operations succeeds
/sig scheduling
/assign
Will the pods be to the queues again somewhere if we delete them by mistake here ? It seems that can not make sure to add them again.
I am not sure if we will add them back again somewhere. If not, not only but also need to be updated. I have already send a PR here:


Question: if len(q.nominatedPods.nominatedPods) != 1 { t.Errorf("Expected nomindatePods to have one element: %v", q.nominatedPods) } <del> if q.unschedulableQ.get(&unschedulablePod) != &unschedulablePod { </del> <ins> if getUnschedulablePod(q, &unschedulablePod) != &unschedulablePod { </ins> t.Errorf("Pod %v was not found in the unschedulableQ.", unschedulablePod.Name) } }

Which jobs are failing: pull-kubernetes-bazel-test Which test(s) are failing: bazel test //pkg/scheduler/internal/queue:godefaulttest Since when has it been failing: Testgrid link: Reason for failure: WARNING: DATA RACE Read at 0x00c000230db0 by goroutine 47: runtime.mapiterinit() +0x0 (PriorityQueue).flushUnschedulableQLeftover() +0x17a (PriorityQueue).flushUnschedulableQLeftover-fm() +0x41 () +0x61 () +0xce () +0x5a Previous write at 0x00c000230db0 by goroutine 45: runtime.mapassignfaststr() +0x0 (UnschedulablePodsMap).addOrUpdate() +0x30e () +0x601 testing.tRunner() +0x162 Goroutine 47 (running) created at: (PriorityQueue).run() +0x172 () +0xe0b () +0x50 () +0x5a5 testing.tRunner() +0x162 Goroutine 45 (running) created at: testing.(T).Run() +0x659 testing.runTests.func1() +0xa8 testing.tRunner() +0x162 testing.runTests() +0x4ee testing.(M).Run() +0x2ee () bazel-out/k8- +0x2fb Anything else we need to know: I'm very sorry about that. UnschedulablePodsMap is not thread-safe. In , lock will be acquired before operating UnschedulablePodsMap. But in , lock isn‘t be acquired. In the previous case, only a single thread runs so it's ok. But after add goroutine , there will be data race. So I need to acquire lock before operating UnschedulablePodsMap.
/sig testing /sig scheduling


Question: // GetCustomResourceListerCollectionDeleter returns the ListerCollectionDeleter of // the given crd. func (r *crdHandler) GetCustomResourceListerCollectionDeleter(crd *apiextensions.CustomResourceDefinition) (finalizer.ListerCollectionDeleter, error) { <del> info, err := r.getOrCreateServingInfoFor(crd) </del> <ins> info, err := r.getOrCreateServingInfoFor(crd.UID, crd.Name) </ins> if err != nil { return nil, err } return info.storages[info.storageVersion].CustomResource, nil } <del> func (r *crdHandler) getOrCreateServingInfoFor(crd *apiextensions.CustomResourceDefinition) (*crdInfo, error) { </del> <ins> // getOrCreateServingInfoFor gets the CRD serving info for the given CRD UID if the key exists in the storage map. // Otherwise the function fetches the up-to-date CRD using the given CRD name and creates CRD serving info. func (r *crdHandler) getOrCreateServingInfoFor(uid types.UID, name string) (*crdInfo, error) { </ins> storageMap := r.customStorage.Load().(crdStorageMap) <del> if ret, ok := storageMap[crd.UID]; ok { </del> <ins> if ret, ok := storageMap[uid]; ok { </ins> return ret, nil } r.customStorageLock.Lock() defer r.customStorageLock.Unlock() <ins> // Get the up-to-date CRD when we have the lock, to avoid racing with updateCustomResourceDefinition. // If updateCustomResourceDefinition sees an update and happens later, the storage will be deleted and // we will re-create the updated storage on demand. If updateCustomResourceDefinition happens before, // we make sure that we observe the same up-to-date CRD. crd, err := r.crdLister.Get(name) if err != nil { return nil, err } </ins> storageMap = r.customStorage.Load().(crdStorageMap) if ret, ok := storageMap[crd.UID]; ok { return ret, nil

Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestWebhookConverterWithDefaulting /assign /priority important-soon /area custom-resources /sig api-machinery /kind flake /milestone v1.16
/cc
several of the webhook converter tests have similar flakes. it looks like the wait for storage version to become effective times out occasionally Rare flakes seen in all of these tests: TestWebhookConverterWithWatchCache/nontrivial-converter TestWebhookConverterWithWatchCache/metadata-mutating-converter TestWebhookConverterWithDefaulting/metadata-mutating-converter TestWebhookConverterWithWatchCache/noop-converter TestWebhookConverterWithDefaulting/empty-response TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-2 TestWebhookConverterWithoutWatchCache/noop-converter/check-0/v1beta2 TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-0/v1beta2
cc
Yikes, I’ll sort it out.
iiuc sending an empty patch gets short-circuited and doesn't bump CR generation, even if the storage version gets changed, which means we do not re-write to etcd and go through the encoding path this would be racing and doing noop's if the first pass of didn't get the expected storage version I will send a fix to make the patch do actual mutation
It doesn't bump generation, but the bytes to persist in etcd would still be different if the storage version had changed, so they should persist. You can verify that by changing the storage version on a CRD yourself, do an empty patch, and see if the resourceVersion changes.
you're right. Generation is for non-metadata change only, and the RV changes. I queried etcd locally and the apiVersion did change for an empty patch (sidetracking: another observation is after changing the storage version (say v1 -v2; noop convertor), empty-patching v2 endpoint would bump generation, while empty-patching v1 endpoint wouldn't)
another theory: our cached storage strategy could be stale if the CRD informer merges multiple events together (e.g. the informer does a re-list).


Question: // GetCustomResourceListerCollectionDeleter returns the ListerCollectionDeleter of // the given crd. func (r *crdHandler) GetCustomResourceListerCollectionDeleter(crd *apiextensions.CustomResourceDefinition) (finalizer.ListerCollectionDeleter, error) { <del> info, err := r.getOrCreateServingInfoFor(crd) </del> <ins> info, err := r.getOrCreateServingInfoFor(crd.UID, crd.Name) </ins> if err != nil { return nil, err } return info.storages[info.storageVersion].CustomResource, nil } <del> func (r *crdHandler) getOrCreateServingInfoFor(crd *apiextensions.CustomResourceDefinition) (*crdInfo, error) { </del> <ins> // getOrCreateServingInfoFor gets the CRD serving info for the given CRD UID if the key exists in the storage map. // Otherwise the function fetches the up-to-date CRD using the given CRD name and creates CRD serving info. func (r *crdHandler) getOrCreateServingInfoFor(uid types.UID, name string) (*crdInfo, error) { </ins> storageMap := r.customStorage.Load().(crdStorageMap) <del> if ret, ok := storageMap[crd.UID]; ok { </del> <ins> if ret, ok := storageMap[uid]; ok { </ins> return ret, nil } r.customStorageLock.Lock() defer r.customStorageLock.Unlock() <ins> // Get the up-to-date CRD when we have the lock, to avoid racing with updateCustomResourceDefinition. // If updateCustomResourceDefinition sees an update and happens later, the storage will be deleted and // we will re-create the updated storage on demand. If updateCustomResourceDefinition happens before, // we make sure that we observe the same up-to-date CRD. crd, err := r.crdLister.Get(name) if err != nil { return nil, err } </ins> storageMap = r.customStorage.Load().(crdStorageMap) if ret, ok := storageMap[crd.UID]; ok { return ret, nil

Two scenarios: -create-update events merged into an update event: strategy with UID:a was created on demand for CRD Foo Foo was deleted and recreated strategy with UID:b was created on demand for the new Foo new Foo got updated (e.g. switching storage version) the strategy with UID:b is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the delete-create-update events into , the strategy with UID:a will be deleted, but the strategy with UID:b won't. -update events merged into a create event: strategy with UID:a was created on demand for CRD Foo Foo was updated (e.g. switching storage version) the strategy with UID:a is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the create-update events into , the strategy with UID:a won't be deleted (as we don't react on create events). another race found in , which could happen in a single update.
I think I :


Question: else echo "Detecting nodes in the cluster" detect-node-names &> /dev/null <del> node_names=( "${NODE_NAMES[@]}" ) </del> <ins> if [[ -n "${NODE_NAMES:-}" ]]; then node_names=( "${NODE_NAMES[@]}" ) fi </ins> fi if [[ "${#node_names[@]}" == 0 ]]; then

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In https://k8s- failed due to . Afterwards crashed as follows: What you expected to happen: Graceful exit.
/assign


Question: // 3. While a volume is being unmounted, add it back to the desired state of world t.Logf("UnmountDevice called") generatedVolumeName, err = dsw.AddPodToVolume( <del> podName, pod, volumeSpec, volumeSpec.Name(), "" /* volumeGidValue */) </del> <ins> podName, pod, volumeSpecCopy, volumeSpec.Name(), "" /* volumeGidValue */) </ins> dsw.MarkVolumesReportedInUse([]v1.UniqueVolumeName{generatedVolumeName}) return nil }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-unit : (PR with no go code changes, should not have failed any unit tests) links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assigne
/assign
/assign tried to fix it in


Question: klog.V(4).Infof("GetHostsInZone %v returning: %v", zoneFailureDomain, hosts) return hosts, nil } <ins> func (nm *NodeManager) SetNodeLister(nodeLister corelisters.NodeLister) { nm.nodeLister = nodeLister } func (nm *NodeManager) SetNodeGetter(nodeGetter coreclients.NodesGetter) { nm.nodeGetter = nodeGetter } </ins>

I am enabling CSI migration on a vSphere cluster. A node was fully drained, all its Pods are deleted, but just before KCM detaches in-tree vSphere volume volumes from it, KCM restarts. A new KCM starts, sees attached volumes in that are not needed by any Pod on the node and calls vSphere in-tree volume plugin . The in-tree vSphere volume plugin + cloud provider marks the volume as detached right away, without reaching to vCenter API. The volume remains attached forever. I don't think it's related to CSI migration, IMO a regular node drain + KCM restart at the same time can cause the same issue. KCM logs: The in-tree cloud provider has a and from it. It apparently has not processed all nodes at the time A/D controller tries to detach the node. The in-tree vSphere volume plugin + cloud provider detaches the volume in vCenter. That's really tricky, it's very hard / impossible to reproduce on a real cluster. I had to inject errors to Detach() in-tree volume call, so they fail until KCM is restarted. Newly started KCM would detach the volume just fine (no error would be injected there), but marks the volume as detached right away. For the record (and search-ability), the CSI driver that tries to attach the volume to another node complains either: Or: cc It could be somewhat related to , however, not all cases of can be related to KCM restart. As I mentioned above, it's very hard/impossible to synchronize KCM restart with draining a node in a real cluster. <details</details<detailsvSphere 7.0.3 </details<details</details<detailsOpenShift </details<detailsCRI-O </details<details</details
/sig storage
Tried just moving a Pod between nodes without CSI migration involved + restarting KCM when the volume is still attached to the old node. In-tree volume plugin on the new node says:
/assign
What CSI driver version and K8s version are you testing with?
Kubernetes is 1.25 and CSI driver is 2.7, still, I think it's not releated to the CSI driver, it's KCM who marks the in-tree volume as detached due to not populated node cache.


Question: klog.V(4).Infof("GetHostsInZone %v returning: %v", zoneFailureDomain, hosts) return hosts, nil } <ins> func (nm *NodeManager) SetNodeLister(nodeLister corelisters.NodeLister) { nm.nodeLister = nodeLister } func (nm *NodeManager) SetNodeGetter(nodeGetter coreclients.NodesGetter) { nm.nodeGetter = nodeGetter } </ins>

/assign
/triage accepted
I found an easier way how to reproduce this without extra code in KCM: a StatefulSet with 1 Pod that uses 1 PVC. the node where the Pod runs. down kube-controller-manager to zero (not really possible for me) or kill all kube-controller-manager Pods. This will cause new KCM processes to wait for leader election. In my env. this gives me ~30 seconds for the next step. Delete the Pod. A new one will be scheduled on a different node (because the current node is cordoned) + there is no KCM running at this time that would detach the volume from the old node. KCM wins its leader election, sees a volume attached to a node where it is not needed -it calls of the volume plugin. VSphere cloud provider does not have its caches populated yet (= this bug) -the volume is not detached, but returns success.
dangling attach does not help here. The volume is still attached to the old node even after ~10 minutes. fails with and not with . I can see there is a lot of dangling attach code in the cloud provider, but I did not debug further.


Question: } if !proxier.lbWhiteListCIDRSet.isEmpty() || !proxier.lbWhiteListIPSet.isEmpty() { // link kube-services chain -> kube-fire-wall chain <del> args := []string{"-m", "set", "--match-set", proxier.lbIngressSet.Name, "dst,dst", "-j", string(KubeFireWallChain)} if _, err := proxier.iptables.EnsureRule(utiliptables.Append, utiliptables.TableNAT, kubeServicesChain, args...); err != nil { glog.Errorf("Failed to ensure that ipset %s chain %s jumps to %s: %v", proxier.lbIngressSet.Name, kubeServicesChain, KubeFireWallChain, err) </del> <ins> args := []string{ "-A", string(kubeServicesChain), "-m", "set", "--match-set", proxier.lbIngressSet.Name, "dst,dst", "-j", string(KubeFireWallChain), </ins> } <ins> writeLine(proxier.natRules, args...) </ins> if !proxier.lbWhiteListCIDRSet.isEmpty() { args = append(args[:0], "-A", string(KubeFireWallChain),

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Service loadbalancer source range don't work in ipvs proxy mode. Which result in gci-gce-ipvs CI failed. http://k8s- How to reproduce it (as minimally and precisely as possible): In gce environment. a service with backend. service type to service util service external IP is specified. the service from pod inside and pod out of the . Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): Install tools: cluster/kube- Others:
/kind bug /sig network /label ipvs
/area ipvs
The test seems to be flaky now. Can you take a look please?
I don't know why, but I think this flaky have the similiar reason with other normal service-test cases, like Those flaky are pretty annoying, problem is, I have run those cases in my own environment many times, never fail... : ( But still, I'll do my best to find the reason.


Question: command := app.NewProxyCommand() <del> utilflag.InitFlags() </del> <ins> // TODO: once we switch everything over to Cobra commands, we can go back to calling // utilflag.InitFlags() (by removing its pflag.Parse() call). For now, we have to set the // normalize func and add the go flag set by hand. pflag.CommandLine.SetNormalizeFunc(utilflag.WordSepNormalizeFunc) pflag.CommandLine.AddGoFlagSet(goflag.CommandLine) // utilflag.InitFlags() </ins> logs.InitLogs() defer logs.FlushLogs()

There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
probably caused by
Could be a clash between these two?
cc
I still see the issue with local-cluster on the latest master: F0225 17:06:49. ] error: v1alpha1.KubeProxyConfiguration: FeatureGates: ReadMapCB: expect { or n, but found ", error found in byte of ......
looks like that's another issue, can you please file a new bug? with more logs? this one was about none of the hyperkube based processes starting up.
will do as it is also broke container kube-proxy as well. Same type of error.


Question: public List<InetAddress> getSeeds() { List<InetAddress> list = new ArrayList<InetAddress>(); <del> String host = getEnvOrDefault("KUBERNETES_API_HOST","https://kubernetes.default.svc.cluster.local"); </del> <ins> //String host = "https://kubernetes.default.svc.cluster.local"; String proto = "https://"; String host = getEnvOrDefault("KUBERNETES_PORT_443_TCP_ADDR", "kubernetes.default.svc.cluster.local"); String port = getEnvOrDefault("KUBERNETES_PORT_443_TCP_PORT", "443"); </ins> String serviceName = getEnvOrDefault("CASSANDRA_SERVICE", "cassandra"); String podNamespace = getEnvOrDefault("POD_NAMESPACE", "default"); String path = String.format("/api/v1/namespaces/%s/endpoints/", podNamespace);

What would you like to be : There is a tab in the k8s perf dashaboard for the scheduler perf benchmarks, but it is showing nothing, we should fix that: http://perf- What this dashboard should be showing is multiple runs per day of We can run it every hour, and summarize the 24 runs of each day to show 50%, 90% and 99% percentile. Why is this needed: Having daily runs of the benchmarks is important to track performance regressions. /sig scheduling /assign /priority important-soon
Ref
/cc
The ci job's config is at And one of failed logs could be found at I tried to reproduce it on my local machine(because I do not have Gloud credentials, I run the task directly in the container): However it works well on my local machine. It is so weird. It will be very appreciated if anyone could help reproduce it with Gloud credentials.
can you try and reach out to the test-infra people? ping them on the slack channel perhaps.
yes, I have tried to get help from , he is busy now. I'll try more.
While working on reproducing this in order to file a bug against test-infra folks I think I found a solution. Will send PRs shortly. /assign
/retitle sig-scalability-benchmarks#scheduler fails consistently due to build issues
/sig scalability Testgrid link: https://k8s- Error logs:
A word about the solution, the job uses the deprecated script which is not supported anymore by test-infra oncall / folks. I've rewritten the job definition to use pod utilities and it started working (will require to get merged first).
Thanks
Thanks !


/kind bug What happened: walteraa [6:55 AM] Hi, The controller manager of the federation using Google cloud DNS is crashing, the log showed is that: the hyperkube command is that: I have created this in the Google DNS Someone knows why the DNS zone query is getting the 404 error?? :disappointed: What you expected to happen: It should have created the missing domain. How to reproduce it (as minimally and precisely as possible): As per above. Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.4 Cloud provider or hardware configuration**: OpenStack (irrelevant) OS (e.g. from /etc/os-release): irrelevant Kernel (e.g. ): irrelevant Install tools: irrelevant Others: irrelevant
The bug is related to this: The above code assumes that the provider code returns no error but an empty list if the domain does not exist, but instead the provider returns a 404. Need to fix the provider and/or the above code.
/sig federation
I am unlikely to have time to work on this soon, so if anyone else wants to, go right ahead.
/assign
An additional information: I have created a which is based on the credential JSON file created through instruction. After that, I have configured the Federation controller manager to mount this secret as a volume in the container and set the environment variable to the JSON file inside this volume. Before doing that, I was getting an authentication error so the google DNS provider couldn't be instantiated, and after doing the instructions that I mentioned before, this authentication error was solved but now I'm getting 404 error when Google DNS provider tries access the specified managed zone. The file that I have generated follow this structure:


Question: Name: stats.Attributes.Metadata.Name, // The StartTime in the summary API is the container creation time. StartTime: metav1.NewTime(time.Unix(0, container.CreatedAt)), <del> // Work around heapster bug. https://github.com/kubernetes/kubernetes/issues/54962 // TODO(random-liu): Remove this after heapster is updated to newer than 1.5.0-beta.0. CPU: &statsapi.CPUStats{ UsageNanoCores: proto.Uint64(0), }, Memory: &statsapi.MemoryStats{ RSSBytes: proto.Uint64(0), }, Rootfs: &statsapi.FsStats{}, </del> <ins> CPU:       &statsapi.CPUStats{}, Memory:    &statsapi.MemoryStats{}, Rootfs:    &statsapi.FsStats{}, </ins> // UserDefinedMetrics is not supported by CRI. } if stats.Cpu != nil {

In , we merged a work around for heapster panic We should get rid of the work around after heapster version is updated. /cc /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close