Question: readonly KUBE_APISERVER_BASE_IMAGE="${KUBE_APISERVER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" readonly KUBE_CONTROLLER_MANAGER_BASE_IMAGE="${KUBE_CONTROLLER_MANAGER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" readonly KUBE_SCHEDULER_BASE_IMAGE="${KUBE_SCHEDULER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" <del> readonly KUBE_PROXY_BASE_IMAGE="${KUBE_PROXY_BASE_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/debian-iptables:$__default_debian_iptables_version}" </del> <ins> readonly KUBE_PROXY_BASE_IMAGE="${KUBE_PROXY_BASE_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/distroless-iptables:$__default_distroless_iptables_version}" </ins> # This is the image used in a multi-stage build to apply capabilities to Docker-wrapped binaries. readonly KUBE_BUILD_SETCAP_IMAGE="${KUBE_BUILD_SETCAP_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/setcap:$__default_setcap_version}"

Let's introduce it gradually, I suggest to wait until the image is promoted for kube-proxy
I've lost track of this, what do we need to do to promote it for kube-proxy?
I think Ricardo was on it But if someone is busy and others have time, let's sync next monday on slack and organize the work, so someone can get this to the finish line, there is no need to overload one person if there are other wanting to help
I didn’t saw there was a comment there. I can fix that, and wanted to plan with you folks next steps. I agree with Antonio and don’t want to block this on my availability :) I will check suggestion tomorrow and as we get this merged I’m more than happy to handle to someone else! On Sat, 16 Jul 2022 at 07:21 Antonio Ojea
I've opened a new PR to do the proper promotion. As soon as this is promoted I think we can start testing and migrating to distroless :) From here I think I can handle to someone else as I wont have much time and attention to this specific effort and I know this is important :) I'm still willing to migrate iptables-wrapper to go code (we started it in kubernetes-sigs/iptables-wrapper) so I will probably finish fixing Dan's reviews and we can discuss the distroless v0.2.0 being based entirely in Go :)
we should probably nudge a few of the subprojects to migrate, we just migrated KIND but I'm sure we have at least a few more out there like ip-masq-agent
I'm migrating nodelocaldns. I've got a couple of questions: nodelocaldns uses registry . Should I switch it to ? Is considered sort of a prod image registry? a multi-arch image? Because currently we use is it better to keep up with the new tags for ? So that we could quickly upgrade base image version as well? Thanks for the work!
YES! Make happy! This registry is actually a "proxy" that sig-k8s-infra is maintaining so people get directed to the closest Image registry, based on their cloud provider. As an example, with this users in AWS wont need to go to GCR to pull their image Yes, it is. It follows the same build pattern of debian-iptables. See Well, this is a good question.


Question: readonly KUBE_APISERVER_BASE_IMAGE="${KUBE_APISERVER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" readonly KUBE_CONTROLLER_MANAGER_BASE_IMAGE="${KUBE_CONTROLLER_MANAGER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" readonly KUBE_SCHEDULER_BASE_IMAGE="${KUBE_SCHEDULER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" <del> readonly KUBE_PROXY_BASE_IMAGE="${KUBE_PROXY_BASE_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/debian-iptables:$__default_debian_iptables_version}" </del> <ins> readonly KUBE_PROXY_BASE_IMAGE="${KUBE_PROXY_BASE_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/distroless-iptables:$__default_distroless_iptables_version}" </ins> # This is the image used in a multi-stage build to apply capabilities to Docker-wrapped binaries. readonly KUBE_BUILD_SETCAP_IMAGE="${KUBE_BUILD_SETCAP_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/setcap:$__default_setcap_version}"

AFAIK we don't have an event / notification system that sends a message (in Slack, as an example) whenever a base image gets promoted. It would be cool to have at least for debian-base, debian-iptables and distroless-iptables, probably it's worth to question sig-release on this :) ( maybe know)
Thanks for the answer! Regarding the - we use it as a compile image for dnsmasq. Will this image still be supported or should I find something other to replace this compile image?
Short answer: I imagine/suppose there is no deprecation plan for this image right now, as it is used as the compilation base for a bunch of other things (and it is base of etcd image) Long answer: In case of distroless-iptables the rationale was: we need to reduce costs (and by reducing costs, reducing image size) we need to use something with a smaller footprint and reduce the amount of packages, and as a result the amount of CVEs in image package. kube-proxy exists in all nodes (unless you use a CNI that implements its features) so it is a good candidate I guess right now debian-base will still be used (I don't remember on the top of my head if other kubernetes components are already running using distroless or compiled based on debian-base, as far as I can see at least etcd image depends on debian-base () so probably any effort on this image should be announced.


Question: }, } } else if metricSpec.External.TargetValue != nil { <del> replicaCountProposal, utilizationProposal, timestampProposal, err = a.replicaCalc.GetExternalMetricReplicas(currentReplicas, metricSpec.External.TargetValue.MilliValue(), metricSpec.External.MetricName, hpa.Namespace, metricSpec.External.MetricSelector) </del> <ins> replicaCountProposal, utilizationProposal, timestampProposal, err = a.replicaCalc.GetExternalMetricReplicas(currentReplicas, metricSpec.External.TargetValue.MilliValue(), metricSpec.External.MetricName, hpa.Namespace, metricSpec.External.MetricSelector, selector) </ins> if err != nil { a.eventRecorder.Event(hpa, v1.EventTypeWarning, "FailedGetExternalMetric", err.Error()) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "FailedGetExternalMetric", "the HPA was unable to compute the replica count: %v", err)

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: We are starting to use the HPA (horizontal pod autoscaling) for one of our components. What is special about this particular component is that the warm up period is very long (sometimes even more than 30 minutes). During that time, the pod is in Running state but its containers are in Not Ready state. This pod is of course not able to receive any traffic so the metric we use in the HPA is ignoring those pods that are not Ready. The equation used by HPA to calculate the new number of replicas is as follows Our problem is that the equation used by HPA to calculate the new number of replicas is including in the "current replica count" the pods that are "Not Ready" so the value for "new replica count" is not correct. Since this warm up is so long we end up by having a lot of more instances of our services of what it should be necessary. They eventually get deleted once gradually the metrics start going down but that will take a lot of time and a big waste of resources without mentioning that with a changing traffic this HPA will never stabilize What you expected to happen: I would like to have a flag somewhere where I tell HPA to don't include in the "current replica count" the pods that are in "Not Ready" state How to reproduce it (as minimally and precisely as possible): Have a component that takes a lot of time to start and being able to receive traffic. Then define a metric that will ignore the pods that are Not Ready. Finally create a metric that will scale that component based on the mentioned metric Anything else we need to know?: Environment: Kubernetes version 1.7 Cloud provider or hardware configuration: Azure
/sig autoscaling
Hi - thanks for reaching out about this! It looks like this section of the , handles when pods aren't ready. I think you this behavior in , so I'm interested in your thoughts on whats best here. If we do end up adding a flag for this, I'd be happy to write the code :) Also, purely for my own curiosity, I'm curious what type of pod is taking 30 minutes to become ready. If your traffic pattern varies at a rate faster than 30 minutes, it may be difficult for auto-scaling to be helpful.
can you also share a bit more about your HPA?


Question: }, } } else if metricSpec.External.TargetValue != nil { <del> replicaCountProposal, utilizationProposal, timestampProposal, err = a.replicaCalc.GetExternalMetricReplicas(currentReplicas, metricSpec.External.TargetValue.MilliValue(), metricSpec.External.MetricName, hpa.Namespace, metricSpec.External.MetricSelector) </del> <ins> replicaCountProposal, utilizationProposal, timestampProposal, err = a.replicaCalc.GetExternalMetricReplicas(currentReplicas, metricSpec.External.TargetValue.MilliValue(), metricSpec.External.MetricName, hpa.Namespace, metricSpec.External.MetricSelector, selector) </ins> if err != nil { a.eventRecorder.Event(hpa, v1.EventTypeWarning, "FailedGetExternalMetric", err.Error()) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "FailedGetExternalMetric", "the HPA was unable to compute the replica count: %v", err)

What kind of metric type is it? Based on the description I'm suspecting that it is an Object metric, am I right?
So answer about why our process takes that much time to start. Basically this is a server that in order to do what he needs to do, he needs to load a lot of data in memory (several GBs) and then perform some warm up that are pretty intensive. The idea is that he doesn't need to grab (almost) anything from outside to attend a request so it's latency is extremely low (in the order of millisecond) By the way. Because of this we have another problem which is that there is not an specific flag per HPA object that we could tell k8s to don't try to scale up again passed some time so the new pods have enough time to get ready before the next scale happens. I know that there is a cluster wide flag but we will need to have a different one by resource that we scale. Now to answer You pointed another interesting topic. The metric we use is % Utilization. We are able to calculate the level of utilization of our pods and we want to keep this metric in an specific value. About the flag. I suggested the flag so it doesn't change the existing behavior by default but I honestly think that the default behavior should be to don't include the NonReady pods as part of the calculation
do you think we can add this change to 1.10? it's critical bug/feature for us and we are happy to help with code/testing.
It's really late for 1.10, especially since we don't have any code here, and I'd like to know more about what's going on here, and especially see if it reproduces on Kubernetes 1.9 (we've had HPA controller bugs in the past). FWIW, currently unready pods are treated as consuming 0% of their utilization on scale up, and it was intended that this should prevent such issues. That is, the formula is actually: If the pods are marked as not ready, I find it strange that they would be affecting the scaling that much, unless your other pods are under real heavy load (i.e. I would assume that the unready pods would skew the scaling back down so that the number of pods just stayed constant).
I believe there is different code branch here.


Question: }, } } else if metricSpec.External.TargetValue != nil { <del> replicaCountProposal, utilizationProposal, timestampProposal, err = a.replicaCalc.GetExternalMetricReplicas(currentReplicas, metricSpec.External.TargetValue.MilliValue(), metricSpec.External.MetricName, hpa.Namespace, metricSpec.External.MetricSelector) </del> <ins> replicaCountProposal, utilizationProposal, timestampProposal, err = a.replicaCalc.GetExternalMetricReplicas(currentReplicas, metricSpec.External.TargetValue.MilliValue(), metricSpec.External.MetricName, hpa.Namespace, metricSpec.External.MetricSelector, selector) </ins> if err != nil { a.eventRecorder.Event(hpa, v1.EventTypeWarning, "FailedGetExternalMetric", err.Error()) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "FailedGetExternalMetric", "the HPA was unable to compute the replica count: %v", err)

metric definition { "type": "Object", "object": { "target": { "kind": "Service", "name": "prometheus-exporter" }, "metricName": "PercentUtilization", "targetValue": "20" } } which should handle as and doesn't check Pod state.
I think proper handling of unready pods is a difficult problem that will require significant work. I suspect we may find out we need to start distinguishing pods that are booting up from pods that are 'broken' and it took us quite a while to solve analogous problem with nodes in Cluster Autoscaler (to be clear: I don't know if we will have to do that, that's just my intuition for now). That being said I agree the formula in GetObjectMetricReplicas will generally lead to scaling-up too much if the pod takes longer to initialize then backoff period in HPA (3 minutes). Assume it takes pod 4 minutes to boot up and we're currently running 2 pods. Metric value is 20 and target is 10. In this case: scales-up to 4 pods (20/10 utilization 2 pods). are starting up, they're not doing anything yet. 3 minutes later HPA scales-up to 8 pods (20/10 utilization 4 pods). If we only counted ready pods the scale-up in 3 would not occur, because the desired replicas would still be 4 (20/10 * 2). My intuition would be that this is the expected behavior of HPA. Also if there is no space in the cluster we will just keep scaling-up until we hit max replicas. The only case where current behavior would be more correct than filtering out unready pods in this formula is if someone uses a metric that is calculated based on total number of replicas. Do we advertise that as the right way to use HPA? If we add to Object metrics in 1.11 this usage pattern will stop making sense. To sum up - I think we will need to change the formula to only count ready pods at some point. I can see how current behavior could be a problem in any use-case except the one described in paragraph above. I'm inclined to agree with and provide some way to change the behavior in 1.10 (a flag or an annotation?), treating it as a bug fix.


Question: }, } } else if metricSpec.External.TargetValue != nil { <del> replicaCountProposal, utilizationProposal, timestampProposal, err = a.replicaCalc.GetExternalMetricReplicas(currentReplicas, metricSpec.External.TargetValue.MilliValue(), metricSpec.External.MetricName, hpa.Namespace, metricSpec.External.MetricSelector) </del> <ins> replicaCountProposal, utilizationProposal, timestampProposal, err = a.replicaCalc.GetExternalMetricReplicas(currentReplicas, metricSpec.External.TargetValue.MilliValue(), metricSpec.External.MetricName, hpa.Namespace, metricSpec.External.MetricSelector, selector) </ins> if err != nil { a.eventRecorder.Event(hpa, v1.EventTypeWarning, "FailedGetExternalMetric", err.Error()) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "FailedGetExternalMetric", "the HPA was unable to compute the replica count: %v", err)

WDYT
Ah, yeah, I missed that it was object metrics before. I'm not entirely convinced we need a flag here -- it might just make sense to bring Object metrics in line with all the other metrics types. On the other hand, there's a subtlety here -- the other metric types don't quite ignore unready pods -- they just ignore the metrics from unready pods, which causes a heavy bias in the direction of not scaling, but an overwhelming spike can still cause scaling. That behavior is not as achievable using object metrics. I'm inclined to just ignore that subtlety, though, and say that object metrics should ignore unready pods entirely.
I think this is what we should do long term. I was worried about breaking someone who relies on targetValue + averaged metric to get effect similar to targetAverageValue and though about providing something for transitional period, but if you're ok with it I'd rather just change it as you say.
I think it's safe to change as-is -- that feature's still in beta, and it's not an API change, so it shouldn't just hard-break anyone. Plus, we've change stuff like this before (when we did the initial resource metrics change).
As said, we are more than ready to help with the fix/test if required. This since this is a major blocking for us in the implementation of the HPA in our side
is anyone actively working on implementing this? If not, I'm happy to. I think I have a fairly clear picture of the change we need to make and will likely have time early this week to get it done.
(Also, from the conversation above, to we want to tag this as a release blocker for 1.10)?
I probably won't be able to work on this before the end of this week. Happy to do a review if you jump in and implement it.
I don't think this qualifies as a release blocking bug (but I'm willing to have someone try and convince me otherwise ;-)). There doesn't seem to be any behavior here that's explicitly broken -- it's just sub-optimal in some use cases. I'd definitely like to address this in 1.11, though.
it's not release blocker, but to share our experience ;).


Question: }, } } else if metricSpec.External.TargetValue != nil { <del> replicaCountProposal, utilizationProposal, timestampProposal, err = a.replicaCalc.GetExternalMetricReplicas(currentReplicas, metricSpec.External.TargetValue.MilliValue(), metricSpec.External.MetricName, hpa.Namespace, metricSpec.External.MetricSelector) </del> <ins> replicaCountProposal, utilizationProposal, timestampProposal, err = a.replicaCalc.GetExternalMetricReplicas(currentReplicas, metricSpec.External.TargetValue.MilliValue(), metricSpec.External.MetricName, hpa.Namespace, metricSpec.External.MetricSelector, selector) </ins> if err != nil { a.eventRecorder.Event(hpa, v1.EventTypeWarning, "FailedGetExternalMetric", err.Error()) setCondition(hpa, autoscalingv2.ScalingActive, v1.ConditionFalse, "FailedGetExternalMetric", "the HPA was unable to compute the replica count: %v", err)

without this feature we cannot use HPA as is and we need to figure out how to build/run fork (which can be challenging because everything inside controller-manager). also we are planning to upgrade to 1.10 quickly, but not sure for 1.11 (e.g. we are two versions behind now).
Sounds like a plan :) Will try and get this done tomorrow morning!
- I submitted a pr for this - would love your eyes on it so you can see if you think it meets your use case :)
/assign
GitHub didn't allow me to assign the following users: mattjmcnaughton. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I checked the code and it seems good to me. About the comment about scaling down if the readyPods<CurrentPods, we should be fine since our metric is already not including the pods that are not ready.
Is there any advance in this ticket? I'm wondering if this made it to k8s 1.10 or if we need to wait for k8s 1.11?
The PR is lgtm-ed and it will merge once the code freeze is lifted. Meaning it will go to 1.11. To get it into 1.10 you would need to get approval for milestone from (sig-autoscaling lead).


Question: var _ volume.VolumePlugin = &localVolumePlugin{} var _ volume.PersistentVolumePlugin = &localVolumePlugin{} var _ volume.BlockVolumePlugin = &localVolumePlugin{} <ins> var _ volume.NodeExpandableVolumePlugin = &localVolumePlugin{} </ins> const ( localVolumePluginName = "kubernetes.io/local-volume"

There are use cases where users want to expand the local volume (may be managed by LVM or something similar), but even though they can edit underlying PV (and expand the underlying device) - the reported size in pvc.Status keeps reporting older value because in-tree local volumes are non-expandable. We could make this easier for users by - implementing interface for local volumes. This would allow following steps for local volume expansion: /admin edits the pvc - requesting new size. expands the underlying device. edits the PV and sets new size. will expand the filesystem and update This would at least get the user from impossible territory to possible. cc /sig storage
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hi & any news on this issue ? The related PR seems stalled.
merged. So this is fixed now.
Oh right ! I didn't looked close enough and thought it was only for directory based volumes. Thank you.


Question:
  <img src="http://kubernetes.io/img/warning.png" alt="WARNING"   <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"  width="25" height="25">  <img src="http://kubernetes.io/img/warning.png" alt="WARNING"   <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"  width="25" height="25">  <img src="http://kubernetes.io/img/warning.png" alt="WARNING"   <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"  width="25" height="25">  <img src="http://kubernetes.io/img/warning.png" alt="WARNING"   <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"  width="25" height="25">  <img src="http://kubernetes.io/img/warning.png" alt="WARNING"   <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"  width="25" height="25">
PLEASE NOTE: This document applies to the HEAD of the source tree

This includes but effects multiple docs.
This issue was also raised on the repository, see:
What's needed to fix this? Are the broken headers being generated automatically somewhere?
- The warning section is pre-pended to the md docs from the following script The fixes this issue, updating this script and all the docs... However, there is an open question as to whether "old" docs should be removed from this repo, which has prevented this PR from being merged.


Question: Verb:     "get", Resource: schema.GroupVersionResource{Resource: "group"}, } <del> c.Invokes(action, nil) </del> <ins> if _, err := c.Invokes(action, nil); err != nil { return nil, err } </ins> groups := map[string]*metav1.APIGroup{}

I'm writing a UT for the following function: The code branch where the server returns an error other than could not be tested using 's . This is because does not handle the error returned by and therefore could not return errors other than : Ref: Add a to so simulate errors other than . Run the following UT (should fail):
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
Same issue for some other methods of : -
I'll submit a PR shortly to address this issue. /assign


Question: visitedUids.Insert(string(uid)) } <del> if string(patchBytes) == "{}" { </del> <ins> if string(patchBytes) == "{}" && !printObject { </ins> count++ cmdutil.PrintSuccess(shortOutput, out, info.Object, false, "unchanged") return nil } } count++ <del> if len(output) > 0 && !shortOutput { </del> <ins> if printObject { </ins> return cmdutil.PrintObject(cmd, info.Object, out) } cmdutil.PrintSuccess(shortOutput, out, info.Object, dryRun, "configured")

Is this a BUG REPORT or FEATURE REQUEST?: Bug Report /kind bug What happened: Applying a file that has no changes compared to current will output something like , which muddles the output when using What you expected to happen: Using the -o flag should suppress the unchanged messages in stdout to prevent output from being muddled How to reproduce it (as minimally and precisely as possible): Given the file Will result in proper yaml being generated as expected Run again, and is outputted to stdout, and is invalid yaml Anything else we need to know?: I do need to check whether this still affects the most recent version kubectl Environment: Kubernetes version (use ): 1.8.6 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Darwin Install tools: Others:
confirmed on latest stable version (1.9.2)
Reiterating the mentions to trigger a notification: <detailsIn response to : confirmed on latest stable version (1.9.2) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have a solution to this, still waiting on my company to look over the agreement


Question: kube::log::error "${gcs_destination} not empty." [[ ${KUBE_GCS_DELETE_EXISTING} =~ ^[yY]$ ]] || { read -p "Delete everything under ${gcs_destination}? [y/n] " -r || { <del> echo "EOF on prompt.  Skipping upload" </del> <ins> kube::log::status "EOF on prompt.  Skipping upload" </ins> return } [[ $REPLY =~ ^[yY]$ ]] || { <del> echo "Skipping upload" </del> <ins> kube::log::status "Skipping upload" </ins> return } } <ins> kube::log::status "Deleting everything under ${gcs_destination}" </ins> gsutil -q -m rm -f -R "${gcs_destination}" || return 1 fi

(cc Despite doing very, very similar things, doesn't rely on . It should, or the logic should otherwise be consolidated. I'm going to maybe work on this to make easier.


Question: HardPodAffinitySymmetricWeight: v1.DefaultHardPodAffinitySymmetricWeight, Client:          clientSet, InformerFactory: informerFactory, <del> PodInformer:     factory.NewPodInformer(clientSet, 0, v1.DefaultSchedulerName), </del> <ins> PodInformer:     factory.NewPodInformer(clientSet, 0), </ins> EventClient:     clientSet.CoreV1(), Recorder:        eventBroadcaster.NewRecorder(legacyscheme.Scheme, v1.EventSource{Component: v1.DefaultSchedulerName}), Broadcaster:     eventBroadcaster,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Scheduler's pod informer has a selector that . This is incorrect. Scheduler must watch all the pods of the cluster, including those scheduled by other schedulers in order to update its cache of the state of the cluster. Otherwise, it will miss important events and the amount of available resources, pods on nodes, ... which are needed for scheduler to work correctly, will not be updated properly. /sig scheduling /assign
cc
For updating it cache, shouldn't the default scheduler rely on node info/status? Also its not very clear, what information the default scheduler needs from other pods (not to be scheduled by it) that it makes decision on? Just trying to understand the issue more clearly.
Scheduler watches informer events to update its cache. Node status is not enough to update all the information that scheduler caches. Scheduler must know what pods are placed on what nodes. For example, in order to check inter-pod affinity/anti-affinity it needs to know the placement of all the pods, including those scheduled by other schedulers.
But this information is present here: So it seems that your PR will now make sure to get the scheduled pods' information directly from the pod informer.
So when I check the history, I see it was here:
have we seen any specific issues due to this change (like between default scheduler and DS controller)?
Yes, but scheduler relies on its pod informers to keep this location up to date. That's exactly why it is important for the scheduler to receive updates for all the pods in the cluster. Yes. We missed the problem at the time. This is problematic only in clusters that use multiple schedulers. I initially thought that DS controller could cause issues as well, but DS controller does not set , so it must not cause an issue. We have seen instances of scheduler's missing certain events in cluster and as a result scheduler's cache getting stale, but I doubt if it is related to this particular issue. This only affects clusters with more than one scheduler. If you run the test that I have written in with the head, it will fail. That tests simulate a multi-scheduler scenario.


Question: fi # Create coverage report directories. <del> KUBE_TEST_API_HASH="$(echo -n "${KUBE_TEST_API////-}"| ${SHA1SUM} |awk '{print $1}')" </del> if [[ -z "${KUBE_COVER_REPORT_DIR}" ]]; then <del> cover_report_dir="/tmp/k8s_coverage/${KUBE_TEST_API_HASH}/$(kube::util::sortable_date)" </del> <ins> cover_report_dir="/tmp/k8s_coverage/$(kube::util::sortable_date)" </ins> else cover_report_dir="${KUBE_COVER_REPORT_DIR}" fi

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: unit tests with coverage succeed How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looking at : It is expecting group version string.
fixed in , clean up from


Question: wantToRun, shouldSchedule, shouldContinueRunning, err := manager.nodeShouldRunDaemonPod(node, c.ds) if wantToRun != c.wantToRun { <del> t.Errorf("[%v] expected wantToRun: %v, got: %v", i, c.wantToRun, wantToRun) </del> <ins> t.Errorf("[%v] strategy: %v, predicateName: %v expected wantToRun: %v, got: %v", i, c.ds.Spec.UpdateStrategy.Type, c.predicateName, c.wantToRun, wantToRun) </ins> } if shouldSchedule != c.shouldSchedule { <del> t.Errorf("[%v] expected shouldSchedule: %v, got: %v", i, c.shouldSchedule, shouldSchedule) </del> <ins> t.Errorf("[%v] strategy: %v, predicateName: %v expected shouldSchedule: %v, got: %v", i, c.ds.Spec.UpdateStrategy.Type, c.predicateName, c.shouldSchedule, shouldSchedule) </ins> } if shouldContinueRunning != c.shouldContinueRunning { <del> t.Errorf("[%v] expected shouldContinueRunning: %v, got: %v", i, c.shouldContinueRunning, shouldContinueRunning) </del> <ins> t.Errorf("[%v] strategy: %v, predicateName: %v expected shouldContinueRunning: %v, got: %v", i, c.ds.Spec.UpdateStrategy.Type, c.predicateName, c.shouldContinueRunning, shouldContinueRunning) </ins> } if err != c.err { <del> t.Errorf("[%v] expected err: %v, got: %v", i, c.err, err) </del> <ins> t.Errorf("[%v] strategy: %v, predicateName: %v expected err: %v, got: %v", i, c.predicateName, c.ds.Spec.UpdateStrategy.Type, c.err, err) </ins> } } }

Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement /sig apps Description: is a key function of DaemonSet; it's better to have more unit test case for it, e.g. node status, pod on the node. There's several test cases to the DaemonSet's finial decision, which are kind of integration test to me; prefer to provide more unit test case for this key function.
GitHub didn't allow me to assign the following users: mathspanda. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
do you have any specific cases or shall I figure out something? I can take a look at this issue if you don't mind.
, , thanks very much for our effort on this enhancement; as you known, DaemonSet will use scheduler's predicates to decide which node can place pods; there's several predicates errors , the purpose of this issue is to trigger every error in unit test for code coverage.
Sure, thanks. I'll cover these predicates.
Also I've found interesting issue when I wanted to add test for predicate. It looks like is not handled or maybe I did something wrong, I couldn't find check. According following comments it should fail because AWS EBS forbids any two pods mounting the same volume ID : Test case: Any thoughts? is it correct behavior?


Question: // ListenAndServePodResources runs the kubelet podresources grpc service func (kl *Kubelet) ListenAndServePodResources() { <del> server.ListenAndServePodResources(util.LocalEndpoint(kl.getPodResourcesDir(), podresources.Socket), kl.podManager, kl.containerManager) </del> <ins> socket, err := util.LocalEndpoint(kl.getPodResourcesDir(), podresources.Socket) if err != nil { klog.V(2).Infof("Failed to get local endpoint for PodResources endpoint: %v", err) return } server.ListenAndServePodResources(socket, kl.podManager, kl.containerManager) </ins> } // Delete the eligible dead container instances in a pod. Depending on the configuration, the latest dead containers may be kept around.

What happened: I replaced the kubelet on one of my 1.14 nodes with a build from master, and now I'm getting this error starting up. Test passes since 5/31/2019 commit have been failing on testgrid. If you look at the logs, the Windows nodes are not coming up. What you expected to happen: No errors starting up kubelet How to reproduce it (as minimally and precisely as possible): Build master, then try to run the kubelet on Windows Anything else we need to know?: /sig windows Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Azure OS (e.g: ): Windows Server 2019 Kernel (e.g. ): 10.0..529 Install tools: aks-engine
cc /priority critical-urgent
/milestone v1.15 Release team - it looks like a recent PR broke windows nodes. We will need a fix here for 1.15
Ok, so the problem here is a little more complicated and we're actually seeing a bug that is actually in the code for quite some time, just that we never used that particular code path. But first: TL;DR: Quick fix is to start the kubelet on Windows with: --feature-gates="KubeletPodResources=false" Now, as to why this is happening: PR [1] was merged 3 days ago and it graduates KubeletPodResources feature as Beta. That means kubelet will enable it by default.With the introduction of the above feature, this branch of code is entered when starting kubelet [2]. We can see a call to the function: defined here [3].While innocent enough, gets the necessary npipe by calling: . is defined here [4] for Windows . The problem is actually that url.URL can't handle Windows style paths [5] and will replace "" with "%5C" as it's to be expected. The paths are transformed along the way from the default "/var/lib/kubelet" to "varlibkubelet" by defined here [6]. More specifically in the use of to create a path. It's inputs are unix style but returns the path Windows style as it's to be expected.The solution is to replace the usage of url.URL in with some alternative package or just implement it.


Question: // ListenAndServePodResources runs the kubelet podresources grpc service func (kl *Kubelet) ListenAndServePodResources() { <del> server.ListenAndServePodResources(util.LocalEndpoint(kl.getPodResourcesDir(), podresources.Socket), kl.podManager, kl.containerManager) </del> <ins> socket, err := util.LocalEndpoint(kl.getPodResourcesDir(), podresources.Socket) if err != nil { klog.V(2).Infof("Failed to get local endpoint for PodResources endpoint: %v", err) return } server.ListenAndServePodResources(socket, kl.podManager, kl.containerManager) </ins> } // Delete the eligible dead container instances in a pod. Depending on the configuration, the latest dead containers may be kept around.

[1] [2] [3] [4] [5] [6]
Thanks for the analysis! I was a bit concerned around the fact that this started showing up around my commit: but the root cause here is indeed, as you said, the graduation of feature to Beta and the way the invalid escape sequences are being introduced.
GCE tests failing too /cc
I'm trying a quick to see if anything else is broken. There are other functions in that same file doing that substitution already so it may make sense to use that as a workaround until a better solution is available.
Still working on it - looks like the path to create the directories is also busted. With still getting


Question: github.com/lpabon/godbc v0.1.1 // indirect github.com/magiconair/properties v1.8.1 // indirect github.com/miekg/dns v1.1.4 <del> github.com/moby/ipvs v1.0.0 </del> <ins> github.com/moby/ipvs v1.0.1 </ins> github.com/mohae/deepcopy v0.0.0-20170603005431-491d3605edfb // indirect github.com/mrunalp/fileutils v0.0.0-20171103030105-7d4729fb3618 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822

Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error.


Question: github.com/lpabon/godbc v0.1.1 // indirect github.com/magiconair/properties v1.8.1 // indirect github.com/miekg/dns v1.1.4 <del> github.com/moby/ipvs v1.0.0 </del> <ins> github.com/moby/ipvs v1.0.1 </ins> github.com/mohae/deepcopy v0.0.0-20170603005431-491d3605edfb // indirect github.com/mrunalp/fileutils v0.0.0-20171103030105-7d4729fb3618 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822

I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO.


Question: github.com/lpabon/godbc v0.1.1 // indirect github.com/magiconair/properties v1.8.1 // indirect github.com/miekg/dns v1.1.4 <del> github.com/moby/ipvs v1.0.0 </del> <ins> github.com/moby/ipvs v1.0.1 </ins> github.com/mohae/deepcopy v0.0.0-20170603005431-491d3605edfb // indirect github.com/mrunalp/fileutils v0.0.0-20171103030105-7d4729fb3618 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822

However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).


Question: github.com/lpabon/godbc v0.1.1 // indirect github.com/magiconair/properties v1.8.1 // indirect github.com/miekg/dns v1.1.4 <del> github.com/moby/ipvs v1.0.0 </del> <ins> github.com/moby/ipvs v1.0.1 </ins> github.com/mohae/deepcopy v0.0.0-20170603005431-491d3605edfb // indirect github.com/mrunalp/fileutils v0.0.0-20171103030105-7d4729fb3618 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822

FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks


Question: }, func(client kubeclientset.Interface, obj pkgruntime.Object) error { svc := obj.(*v1.Service) <del> err := client.Core().Services(svc.Namespace).Delete(svc.Name, &metav1.DeleteOptions{}) </del> <ins> orphanDependents := false err := client.Core().Services(svc.Namespace).Delete(svc.Name, &metav1.DeleteOptions{OrphanDependents: &orphanDependents}) </ins> return err })

Problem: When users sends a cascading deletion request to federation i.e they set orphanDependents=false, then federation should also set orphanDependents=false, when it sends the request to underlying clusters.
Fixed by


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/tools" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> "github.com/coreos/go-etcd/etcd" "github.com/golang/glog" )

kube-proxy regularly stops running with the below panic / stack-trace. This is when configured with vagrant. (It is also running on a laptop, so my unsupported-by-any-evidence theory is that it happens when the laptop resumes after sleep.) Restarting the service causes the proxy to continue working happily. (Should systemd be restarting the service?) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: panic: runtime error: invalid memory address or nil pointer dereference Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: [signal 0xb code=0x1 addr=0x0 pc=0x431b51] Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 22 [running]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0x6b7c00, 0xa51de8) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210066600, 0xc210037660, 0xc210037720, 0x77359400, 0x0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x841 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210066600, 0xc210037660, 0xc210037720,


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/tools" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> "github.com/coreos/go-etcd/etcd" "github.com/golang/glog" )

0x77359400) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x165 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x1b3 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 1 [select (no cases)]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: () Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x47a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 3 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (loggingT).flushDaemon(0xa57340) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x50 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by ·1 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x276 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 4 [syscall]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: () Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /usr/local/go/src/pkg/runtime/proc.c:1394 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 6 [sleep]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: time.Sleep(0x12a05f200) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: +0x31 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0x7aca18,


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/tools" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> "github.com/coreos/go-etcd/etcd" "github.com/golang/glog" )

0x12a05f200) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x57 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x99 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 7 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210077120, 0x7ff2602627f0, 0xc210077060, 0xc210037540) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x196 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 8 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210077210, 0x7ff260262840, 0xc210077150, 0xc2100375a0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x196 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 9 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (Mux).listen(0xc2100770c0, 0x718260, 0x4,


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/tools" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> "github.com/coreos/go-etcd/etcd" "github.com/golang/glog" )

0xc210037600) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·001() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x43 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·001() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x40 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc21000a920, 0x0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x49 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by (*Mux).Channel Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x20a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 10 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·004() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x4d Aug 13 12:04:06 kubernetes-minion-1 systemd[1]: kube-proxy.service: main process exited, code=exited, status=2/INVALIDARGUMENT Aug 13 12:04:06 kubernetes-minion-1 systemd[1]: Unit kube-proxy.service entered failed state.
I looked at this one a little bit and I think it's because of This line isn't present at , but I'm not sure which is newer. Presumably results in the channel being closed (and receiving a zero value that it then tries to dereference) on error.
Diagnostic: This line doesn't check for a closed channel, and it needs to. Yeah, how are we supposed to know if our client is going to close the channel or not?


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/tools" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> "github.com/coreos/go-etcd/etcd" "github.com/golang/glog" )

I think the behavior has changed at least once already. This is part of why I'm trying to (for our code, anyway) standardize all of our watches with a watch.Interface. I took the liberty of assigning this to you since you're more familiar with this code, but feel free to unassign if you can't take it right now.
Got it
can you give a try?
after i do the minion watch for kubelet config i'll come back and gut this and replace it.
Thanks - with merged I haven't seen this happening. Thanks for the super-quick fix!


Question: import ( "bytes" "encoding/binary" <ins> "errors" </ins> "fmt" "net" "os/exec"


Question: import ( "bytes" "encoding/binary" <ins> "errors" </ins> "fmt" "net" "os/exec"


Question: import ( "bytes" "encoding/binary" <ins> "errors" </ins> "fmt" "net" "os/exec"


Question: import ( "bytes" "encoding/binary" <ins> "errors" </ins> "fmt" "net" "os/exec"


Question: }, } if utilfeature.DefaultFeatureGate.Enabled(features.PodDisruptionConditions) { <ins> role.Rules = append(role.Rules, rbacv1helpers.NewRule("get").Groups(legacyGroup).Resources("pods").RuleOrDie()) </ins> role.Rules = append(role.Rules, rbacv1helpers.NewRule("patch").Groups(legacyGroup).Resources("pods/status").RuleOrDie()) } return role

For 1.25 clusters with PodDIsruptionConditions enabled pods continue to run if the node is tainted with untolerated NoExecute taint. Also, the error is not logged. The pod running on the tainted node should be terminated (prep) Create a 1.25 cluster with the enabled (tested on kind) a job with a single Pod running long enough for the test (say 3min), the node with a custom taint (so that it is not tolerated by the Pod) with the NoExecute effect (nodeName No response <details</details<detailsreproducible on kind </details<details</details<details</details<details</details<details</details
/sig apps /triage accepted /priority important-soon
/assign


Question: "//pkg/controller/nodeipam/ipam/cidrset:go_default_library", "//pkg/controller/nodeipam/ipam/sync:go_default_library", "//pkg/controller/util/node:go_default_library", <ins> "//pkg/scheduler/algorithm:go_default_library", </ins> "//pkg/util/node:go_default_library", <ins> "//pkg/util/taints:go_default_library", </ins> "//vendor/github.com/golang/glog:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/errors:go_default_library",

forked from As from the discussion with and we have to start checking taints wherever we check for node.Spec.Unschedulable. Also overlaps on some parts with /sig node /assign
cc
No sure the discussion history :); personally, prefer to deprecate and replace it with taint.
- same here, the goal would be to deprecate its usage, start checking taints everywhere unschedulable is used. Also, as long as we as we have v1, we might have to keep checking for both.
great, there's same subtask in my mind: deprecated description to the taint for command with taint others ??
- yup, but for 4 it is more something like check taints wherever we checn for unschedulable, since we cannot just replace until v1 is gone.
, my major point is to handle it task by task to make sure we did not miss anything :).
yes we definitely should
- should we define a specific taint for unschedulable (key and effect) ? (i.e what happens when a user uses cordon/uncordon)
I'm not sure about having specific Taint for Unschedulable is a good idea. Pretty much any Taint with NoSchedule effect will do (and this is what we should be checking).
- I agree, but in case of we can add a taint with key or something like that with a effect. This might help us especially if the node has no taints.
, not sure whether we should handle specially; AFAIK, set this flag for node maintenance, this taint should not be tolerated.
The whole point of moving from to Taints is to make things more configurable. as they're decision makers here.
If we are discussing about deprecate/re-implement by Unschedulable taint, +1 as it make sense. Current cordon is causing confusion in community since you can always taint it instead, and tainting is what most Kubernetes setup tool is doing today.
I agree with that we should use NoSchedule taint. I also agree that generally no pod should tolerate NoSchedule taints, but they may want to do so if the desire, knowing that anything could happen to the node they are scheduled on. For example, today DS pods tolerate "Unschedulable" bit.


Question: "//pkg/controller/nodeipam/ipam/cidrset:go_default_library", "//pkg/controller/nodeipam/ipam/sync:go_default_library", "//pkg/controller/util/node:go_default_library", <ins> "//pkg/scheduler/algorithm:go_default_library", </ins> "//pkg/util/node:go_default_library", <ins> "//pkg/util/taints:go_default_library", </ins> "//vendor/github.com/golang/glog:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/errors:go_default_library",

They may as well want to tolerate "NoSchedule" taints.
/sig scheduling
Agree; it's up to 'user' to decide whether tolerates the taint, we just need to taint it and doc what's happened in node for taints :)
We should have a mechanism that allows a cluster admin to limit types of tolerations that users can set. This is needed to prevent abuse in multi-tenant clusters. For example, cluster admin may want to taint nodes with rare resources to prevent pods from certain group of users to land on them. Users shouldn't be able to tolerate such taints. That said, this is a separate topic of its own which is related to policy and should be discussed separately.
- we’re already facing this in production for our spare nodes. I’ll bring this up in our next sig-meeting, I’ll also file an issue for this


Question: }) } } <ins> func TestHTTPProbeChecker_HostHeaderPreservedAfterRedirect(t *testing.T) { successHostHeader := "www.success.com" failHostHeader := "www.fail.com" handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { switch r.URL.Path { case "/redirect": http.Redirect(w, r, "/success", http.StatusFound) case "/success": if r.Host == successHostHeader { w.WriteHeader(http.StatusOK) } else { http.Error(w, "", http.StatusBadRequest) } default: http.Error(w, "", http.StatusInternalServerError) } }) server := httptest.NewServer(handler) defer server.Close() testCases := map[string]struct { hostHeader     string expectedResult probe.Result }{ "success": {successHostHeader, probe.Success}, "fail":    {failHostHeader, probe.Failure}, } for desc, test := range testCases { headers := http.Header{} headers.Add("Host", test.hostHeader) t.Run(desc+"local", func(t *testing.T) { followNonLocalRedirects := false prober := New(followNonLocalRedirects) target, err := url.Parse(server.URL + "/redirect") require.NoError(t, err) result, _, _ := prober.Probe(target, headers, wait.ForeverTestTimeout) assert.Equal(t, test.expectedResult, result) }) t.Run(desc+"nonlocal", func(t *testing.T) { followNonLocalRedirects := true prober := New(followNonLocalRedirects) target, err := url.Parse(server.URL + "/redirect") require.NoError(t, err) result, _, _ := prober.Probe(target, headers, wait.ForeverTestTimeout) assert.Equal(t, test.expectedResult, result) }) } } </ins>

Hello there, I did search around and didn't find a report for this already, I hope it's not a dupe. What happened: In an HTTP probe definition (readyness or liveness), the custom host header is not set properly after a redirect occurs. What you expected to happen: Seeing that states that redirects are weird but allowed, I would expect the custom headers to be maintained after the redirect occurs. How to reproduce it (as minimally and precisely as possible): Setup an HTTP service that expects a specific Host header to be used (using Django here) and would return a 400 if the Host doesn't match (for a more detail view at how Django deals with this ). Then setup a probe like: You should now see in your logs that the first call to the service succeeded with a 301 (or 3XX) which means the Host header was set properly, but that the call to the redirected url will fail with a 400 which means the Host header was not set anymore and makes the probe raise an error. Anything else we need to know?: In the above test, setting the "path" to the redirected url works and the probe is green. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: intel NUC OS (e.g. from /etc/os-release): Ubuntu 16.04.5 LTS Kernel (e.g. ): Linux nuc 4.4.0-138-generic -Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Minikube Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig node /sig network
We are calling Go's HTTP client here - is it possible that this is "normal" behavior for Go? I don't know...
Seems like an old bug indeed: - not sure where to go from there...
want to try attempting a patch using ? code that mentioned is in
I can definitely try to submit a PR with that change! Thanks for pointing in the right direction
I just tested this on GKE and also have a unit test confirming that this is not an issue anymore, I guess HTTP go client has been fixed.
Awesome! looks like losing track of my commitment here was a good investment thx for the feedback !
Awesome. Looking forward to more tests :)


Question: # Use host IP instead of localhost in control plane kubeconfig files. export KUBECONFIG_USE_HOST_IP="${KUBECONFIG_USE_HOST_IP:-false}" <ins> # Optional: Install Pigz on Windows. # Pigz is a multi-core optimized version of unzip.exe. # It improves container image pull performance since most time is spent # unzipping the image layers to disk. export WINDOWS_ENABLE_PIGZ="${WINDOWS_ENABLE_PIGZ:-true}" </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : pigz for GCE Windows Why is this needed: Pigz improves the image layer extraction performance for multi-core systems. Pigz is usually preinstalled on Linux. This enhancement is scoped specifically for the GKE Windows node scripts. For Context:
/assign me
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
GitHub didn't allow me to assign the following users: me. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign me Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows /assign jeremyje
s/GKE/GCE/? as in kube-up?
Replaced but I don't understand the kube-up part.
is how one creates a GCE cluster from this repo. Otherwise we'd be talking about something that isn't part of kubernetes.


Question: // Creating pods and services in namespace-b nsBpodServerA, nsBserviceA = createServerPodAndService(f, nsB, "ns-b-server-a", []int{80}) <ins> defer cleanupServerPodAndService(f, nsBpodServerA, nsBserviceA) </ins> nsBpodServerB, nsBserviceB = createServerPodAndService(f, nsB, "ns-b-server-b", []int{80}) <ins> defer cleanupServerPodAndService(f, nsBpodServerB, nsBserviceB) </ins> // Wait for Server with Service in NS-A to be ready <del> framework.Logf("Waiting for servers to come up.") err = e2epod.WaitForPodRunningInNamespace(f.ClientSet, podServer) framework.ExpectNoError(err, "Error occurred while waiting for pod status in namespace: Running.") </del> <ins> framework.Logf("Waiting for servers to be ready.") err = e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, podServer.Name, podServer.Namespace, framework.PodStartTimeout) framework.ExpectNoError(err, "Error occurred while waiting for pod status in namespace: Ready.") </ins> // Wait for Servers with Services in NS-B to be ready <del> err = e2epod.WaitForPodRunningInNamespace(f.ClientSet, nsBpodServerA) framework.ExpectNoError(err, "Error occurred while waiting for pod status in namespace: Running.") </del> <ins> err = e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, nsBpodServerA.Name, nsBpodServerA.Namespace, framework.PodStartTimeout) framework.ExpectNoError(err, "Error occurred while waiting for pod status in namespace: Ready.") </ins> <del> err = e2epod.WaitForPodRunningInNamespace(f.ClientSet, nsBpodServerB) framework.ExpectNoError(err, "Error occurred while waiting for pod status in namespace: Running.") defer cleanupServerPodAndService(f, nsBpodServerA, nsBserviceA) defer cleanupServerPodAndService(f, nsBpodServerB, nsBserviceB) </del> <ins> err = e2epod.WaitTimeoutForPodReadyInNamespace(f.ClientSet, nsBpodServerB.Name, nsBpodServerB.Namespace, framework.PodStartTimeout) framework.ExpectNoError(err, "Error occurred while waiting for pod status in namespace: Ready.") </ins> ginkgo.By("Creating a network policy for the server which allows traffic only to a server in different namespace.") protocolUDP := v1.ProtocolUDP

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: A NetworkPolicy test in , not run in Kubernetes CI. Which test(s) are flaking: Testgrid link: Reason for failure: Its failure rate is 6.66% according to 30 latest tests in (a Kubernetes CNI) CI. The reason is, the test doesn't wait for the server Pod to be ready before testing traffic via its service, then even the NetworkPolicy allows it, the SYN packets will be rejected by iptables because the service has no endpoints at that moment. e2e log: server pod status: kube-proxy log: In short: 14:35:58.874 server Pod was created 14:36:00.916 server Pod became running 14:36:00.916 The traffic test against Service started, there was no endpoints for it at this moment so it was always rejected 14:36:10.921 The traffic test failed and terminated 14:36:10 server Pod became ready, Endpoints is updated 14:36:10.547 kube-proxy received Endpoints update and configured nat table The other tests don't have this problem because they all wait for server Pods to be ready, this test should do it too to avoid flakes.
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: experimentalHostUserNamespaceDefaulting: utilfeature.DefaultFeatureGate.Enabled(features.ExperimentalHostUserNamespaceDefaultingGate), keepTerminatedPodVolumes:                keepTerminatedPodVolumes, nodeStatusMaxImages:                     nodeStatusMaxImages, <ins> lastContainerStartedTime:                newTimeCache(), </ins> } if klet.cloud != nil {

These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions


Question: if mapping.Key != strings.ToLower(mapping.Key) { allErrs = append(allErrs, field.Invalid(fldPath.Child("key"), mapping.Key, "key must be lowercase")) } <ins> if isKubernetesDomainPrefix(mapping.Key) { allErrs = append(allErrs, field.Invalid(fldPath.Child("key"), mapping.Key, "k8s.io, kubernetes.io and their subdomains are reserved for Kubernetes use")) } </ins> if seenExtraKeys.Has(mapping.Key) { allErrs = append(allErrs, field.Duplicate(fldPath.Child("key"), mapping.Key)) continue

Disallow setting all and namespaced extra info in custom ways for now. /assign aramase enj /sig auth
/triage accepted


Question: "description": "Endpoint represents a single logical "backend" implementing a service.", "properties": { "addresses": { <del> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "items": { "type": "string" },

An can contain up to 1000 , which can each contain up to 100 . Or at least, that's what the docs say, but actually, kube-proxy (or more specifically, ) only looks at the first Address in each Endpoint: (This is compatible with how works, since it would only generate multi-Address Endpoints if a pod had multiple of the same address family, which is not allowed. But it's not compatible with manually-generated EndpointSlices.) Either all are used, or else the docs state that only is used and the field is an array solely for historical reasons.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /priority important-longterm /cc
All addresses in the slice must be of the same type, do you mean creating an Slice with different address types in the Addresses field?
I'm not sure if/how kube-proxy should handle an endpoint with more than 1 address. I think the goal of kube-proxy is to give each endpoint an equivalent probability of being selected. To extend that logic here, I think we'd need to take the probability assigned to an endpoint and divide it by the number of addresses attached to that endpoint. So if an endpoint has a 50% probability of being selected, and it provides 50 addresses, each should end up with a 1% probability of being selected. Of course with the way we need to do descending probabilities, the logic may get a bit complicated. This isn't as clear as it should be, but I don't think kube-proxy or any consumer of the EndpointSlice API needs to commit to supporting every value specified by that API. For example, many may filter by address type, Service, or some other means. I think it's also valid for a consumer to say they only support a single address per family per endpoint. So all of that to say, I'd support either documenting that kube-proxy only selects the first address in an endpoint, or updating kube-proxy to support more than one address per endpoint.


Question: "description": "Endpoint represents a single logical "backend" implementing a service.", "properties": { "addresses": { <del> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "items": { "type": "string" },

Given the number of changes currently in flight for kube-proxy, documentation seems like a safer initial approach to me.
/assign
can I have this editing only the first address(Address[0]) in Endpoint.Address to be used. The use of an array in this field is based on historical reasons. Isn't that right
thanks for volunteering to help with this one! I know has some big kube-proxy changes/fixes in flight right now, so it would be good to be sure you don't conflict with him. Otherwise, as long as isn't already working on this, I think you're free to have it.
I don't think we agreed that the array is for historical reasons only. We aren't currently using it, but we aren't ready to say that we won't use it in the future at this point. So, the docs should warn that kube-proxy currently ignores additional addresses, but it shouldn't imply that it's actually wrong to have additional addresses I'm not
thanks i will work on this
It was for a reason, but I think that reason was mixed-family slices (before we decided to break it up into single-family slices). What would it mean to have multiple addresses here? Are they assumed to be equivalent? Or do they have different context (like a split-horizon)? If the latter, there isn't any metadata here (i.e. it's not a struct like to allow new fields). I am inclined to believe it was meant as equivalent, and as such it's probably OK to document that assumption clearly. That said, I don't see how anyone could use multiple addresses in a meaningful way - they are either assumed equivalent or they lack enough metadata to distinguish their differences. ?? If we do document the equivalence, it should be generically worded. E.g. "These addresses are assumed to be equivalent and some consumers may choose to ignore all but the first value."
if it was meant as equivalent, why put multiple same addresses into Addresses? I don't understand what the purpose of this is.
The first version of the API had both IP families in a single slice, so it made sense to have multiple addresses. We changed that, though we didn't immediately remove the old support. So my best guess is that we should have converted this to a singular value when we did that, but we missed it.


Question: "description": "Endpoint represents a single logical "backend" implementing a service.", "properties": { "addresses": { <del> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "items": { "type": "string" },

On Wed, Nov 24, 2021, 5:45 PM Cyclinder
so we should change the API doc? Or It's just that documenting it will only use the first address in the , like you said above
I'd like some of the others to comment, but it's this a US holiday, so it might be a few days..
So maybe we should just declare that the array is essentially a bug, and only the first address is used? Maybe something should warn if you ever set additional addresses? (FTR, the context that I noticed this in was that I was adding unit tests, and was creating EndpointSlices with a single Endpoint and multiple Addresses, which doesn't work; you need multiple Endpoints with a single Address each.)
I think the simplest and most compat approach is to say "these are all assumed to be fungible and clients may choose to only use the first element". Thoughts?
I think the simplest and most compat approach is to say "these are all assumed to be fungible and clients may choose to only use the first element". Big +1 to that approach. I don't think it's necessarily invalid to set multiple addresses here, there may actually be some use cases for it, but adding this kind of statement should help clear things up significantly.
hmm as we push for multi homed pod this will make a lot of sense. There might also be a chance for community to provide a drop-in replacement for proxy that does much more than we do now. I think we should update the docs and don't modify anything for now.
Note that this API doesn't include per-IP metadata, so a multi-homed pod would be challenging with this mechanic. I think it's maybe time to reconsider multi-net more holistically. On Mon, Dec 13, 2021, 10:24 AM Khaled Henidak (Kal) <
:thinking:
If they're fungible, then "clients may choose to use only one of the elements" (or "only a subset"?). No reason to call out the first one specifically.
That's fair On Tue, Dec 14, 2021, 6:09 AM Dan Winship
I would like to add to this issue that it would makes sense to have kube-proxy reduce the subset of endpoints in the endpointslice based on the label of the node and the & attributes of the endpoint.


Question: "description": "Endpoint represents a single logical "backend" implementing a service.", "properties": { "addresses": { <del> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "items": { "type": "string" },

Taking only into consideration the first endpoint which is in the same zone of the node or has no zone dependency and is ready for serving. & Would it make sense to reopen this issue?
I don't follow entirely what you are suggesting, can you elaborate?
kube-proxy should understand which zone a node is in and prefer endpoints in the same zone.
but that is unrelated to this issue, topology is considered See KEPs related, topology-aware-routing was replaced by topology-aware-hints
thank you for clarifying. It seems ambiguous to me to have a zone attribute and a hint attribute. I fail to implement the hinting on a service in v1.23.8, but this is urelated to this issue.
heh, don't worry, this topic is complex and we went through different solutions, you have more historical context on the KEPs I linked above and this thread in the mailing list


Question: # Post-condition: valid-pod doesn't exist kube::test::get_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' <ins> ### Verify flag all-namespaces is ignored for rootScoped resources # Pre-condition: node exists kube::test::get_object_assert nodes "{{range.items}}{{$id_field}}:{{end}}" '127.0.0.1:' # Command output_message=$(kubectl get nodes --all-namespaces 2>&1) # Post-condition: output with no NAMESPACE field kube::test::if_has_not_string "${output_message}" "NAMESPACE" </ins> set +o nounset set +o errexit }

This is wrong, cluster scoped resources should never have a NAMESPACE column. We need to fix and add a test to ensure this doesn't regress.
The presence of the flag is what triggers this.
I would help fix it.
I mean to add a test after modification, but found that current test framework could not support the case: To cover the scene, we need a real NewHumanReadablePrinter in test case, to ensure the output format, and modify the related funcs in FakeFactory to support pass in param withNamespace. I feel this not like behavior of unit test to some extent. The loss may outweighs the gain to have such a big impact on the whole. What's your opinion
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing the label for more than 7 days. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: // +groupName=wardle.k8s.io // Package v1beta1 is the v1beta1 version of the API. <del> package v1beta1 </del> <ins> package v1beta1 // import "k8s.io/sample-apiserver/pkg/apis/wardle/v1beta1" </ins>

Look at one of the directories under (Please don't pick more than one directory for one PR as it will cause issues for reviewers) Find all the files named (example find -name ) See the following link it has a this tells tools how this package needs to be imported. So we should ensure that each has an import Add a description appropriate to the package (similar to the one below) Look through sub directories and see if there are any other important directories that are likely to be used by outside users and add a Why are we doing this? We perfer to use vanity urls instead of directly using to refer to code, so this helps ensure that tools honor our wish. See It would be good to have some package level documentation since these directories under staging/ are meant to be for use by folks outside of kubernetes github org. /good-first-issue
/kind documentation /sig architecture
currently working my through Not going to use / assign since there could be a number of PRs for this issue.
/lifecycle active /good-first-issue edit: hmm..
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /lifecycle active /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
May I try on
IIUC this means that there needs to be a even at the root of the repo? Then how does work? :thinking:
Hi , can I make a PR for this directory ?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: // +groupName=wardle.k8s.io // Package v1beta1 is the v1beta1 version of the API. <del> package v1beta1 </del> <ins> package v1beta1 // import "k8s.io/sample-apiserver/pkg/apis/wardle/v1beta1" </ins>

/lifecycle rotten
I guess this is still not done, was supposed to partially fix this issue. /reopen
Reopened this issue. <detailsIn response to : I guess this is still not done, was supposed to partially fix this issue. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /area code-organization


Question: return nil, err } } <del> sgList := []string{securityGroupID} </del> <ins> sgList := []string{} for _, extraSG := range strings.Split(annotations[ServiceAnnotationLoadBalancerSecurityGroups], ",") { extraSG = strings.TrimSpace(extraSG) if len(extraSG) > 0 { sgList = append(sgList, extraSG) } } // If no Security Groups have been specified with the ServiceAnnotationLoadBalancerSecurityGroups annotation, we add the default one. if len(sgList) == 0 { sgList = append(sgList, securityGroupID) } </ins> for _, extraSG := range strings.Split(annotations[ServiceAnnotationLoadBalancerExtraSecurityGroups], ",") { extraSG = strings.TrimSpace(extraSG)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: We run Kuberentes on AWS. We are trying out service type. We use Terraform heavily for managing infrastructure on AWS. As a result we have several shared security groups with "perfect" rules. One of them is supposed to be attached to ELBs so our own IPs, partners, etc are whitelisted. Right now, it seems there isn't a way for Kubernetes to use the security group as is. There are several alternatives: Pass my existing security group as annotation. However, Kubernetes still tries to create one first, or use a global security group. If I don't provide any IP range, Kubernetes will whitelist . This seems insecure. Provide my own IP ranges as either annotation or . Then Kubernetes will create a security group for this ELB with the given IPs. This means I need to duplicate my IPs in all microservices. Provide my own IP ranges as either annotation or , and use a global security group. Then Kubernetes will modify the global SG with the given IPs. This seems problematic when lots of microservices share one SG but some want different IPs whitelisted. I want to entertain the idea below: Allow users to sepcify another security group to or object, which Kubernetes will take it and attach it to ELBs as is. If such security group is provided, skip other IP whitelisting steps. What you expected to happen: I can give Kubernetes existing security group(s). Kuberentes just attaches the security group(s) to the managed ELBs without modifying the rules. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: It may already be possible to achieve this, and I'm not aware of. Or there are other concerns so that Kubernetes didn't choose to implement this feature. Environment: Kubernetes version (use ): v1.7.1 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig aws
PR may help with this. . .
This problem could be really critical if one has the Kubernetes dashboard installed.


Question: return nil, err } } <del> sgList := []string{securityGroupID} </del> <ins> sgList := []string{} for _, extraSG := range strings.Split(annotations[ServiceAnnotationLoadBalancerSecurityGroups], ",") { extraSG = strings.TrimSpace(extraSG) if len(extraSG) > 0 { sgList = append(sgList, extraSG) } } // If no Security Groups have been specified with the ServiceAnnotationLoadBalancerSecurityGroups annotation, we add the default one. if len(sgList) == 0 { sgList = append(sgList, securityGroupID) } </ins> for _, extraSG := range strings.Split(annotations[ServiceAnnotationLoadBalancerExtraSecurityGroups], ",") { extraSG = strings.TrimSpace(extraSG)

If for some reasons the service recreates AWS ELB, the related security group will be recreated as well with the "0.0.0.0/0" input/output masks and then botnets could find the ELB and . JFYI, in our case in the one of clusters with K8s 1.5, the deployment and replication controller were installed from the dashboard using this vector of attack. Then special pods were created and root volume mounted inside. The system crontab was modified by processes in these pods and the crypto miner appeared in all K8s nodes on the system level. After this another pods were installed and the miner was executed inside the Kubernetes pods in parallel with the system level.
just had some downtime on a production cluster because of this. i really need it too ;)
+1
As for me, it isn't a feature, it's a kind of bug.
Some comments on which may be relevant - please comment if an alternative annotation that did replace the default security-groups (as opposed to adding to them) would/wouldn't meet your use-case:
For me, I don't add any annotation, I just want to change its security group rule. After it will create a aws elb, and a specificed secruity group with 0.0.0.0, I change 0.0.0.0 to However, the security group rule will change back to 0.0.0.0 after some minutes. Is this by design? Is there any way for me keep my changes, thanks.
I guess this is part of how the controller works. If you look at , the PR fixes this problem by allowing you to specify the behaviour you want with an annotation. For the way Kubernetes works, the ELB is owned by Kubernetes and you should never be forced to modify the resource manually, thing that ultimately will not work.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale This is still relevant cause is not merged yet.
This is a good change.


Question: return nil, err } } <del> sgList := []string{securityGroupID} </del> <ins> sgList := []string{} for _, extraSG := range strings.Split(annotations[ServiceAnnotationLoadBalancerSecurityGroups], ",") { extraSG = strings.TrimSpace(extraSG) if len(extraSG) > 0 { sgList = append(sgList, extraSG) } } // If no Security Groups have been specified with the ServiceAnnotationLoadBalancerSecurityGroups annotation, we add the default one. if len(sgList) == 0 { sgList = append(sgList, securityGroupID) } </ins> for _, extraSG := range strings.Split(annotations[ServiceAnnotationLoadBalancerExtraSecurityGroups], ",") { extraSG = strings.TrimSpace(extraSG)

The situation pre-1.13.0 is somewhat surprising because explicitly setting to an empty array like so: has no effect, and the Security Group Ingress is still .


Question: "encoding/json" "fmt" "io" <ins> "net/url" </ins> "strings" "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/meta" "k8s.io/kubernetes/pkg/api/unversioned" apiutil "k8s.io/kubernetes/pkg/api/util" <ins> "k8s.io/kubernetes/pkg/api/v1" </ins> "k8s.io/kubernetes/pkg/apimachinery/registered" "k8s.io/kubernetes/pkg/apis/extensions" "k8s.io/kubernetes/pkg/runtime"

A user report that it's broken in the slack channel:
Also and -- this probably broke with one of our recent refactors. The server should be accepting list options from any version, I thought?
FWIW, I think it's this line barf at the ListOptions: The parameter codec first decodes the query parameter to a versioned object and then convert it to the internal object. The code fails because it cannot create a "" ListOptions.
The server should accept list options in the API version it is sent. ListOptions should be registered in all third party groups.
The thirdPartyResourceDataCreator should handle "ListOptions", which surprises me that it doesn't then respect the group.
I think we should add a wrapper around the api.ParameterCodec here: Do something similar as what does.
Putting together a PR now and will let you guys review.
I guess my point is, the list options are just there from the URL query parameters. The user didn't send a group. Why do we assume that value for the group rather than "/v1" or something?
My understanding is we want to allow options to evolve differently in different groups. We do have v1.ListOptions and v1beta1.ListOptions. And if a client sends requeset to extensions/v1beta1 endpoint, then it's expected to encode the ListOptions to v1beta1 and then convert it to query parameters.
Mostly because the parameter codec is supposed to abstract that. "virtual" groups need to be implemented by the virtual "parameter codec". The statement on floating server objects (List, Delete etc) was that they have a locked API schema, and we have a mechanism to do that via the scheme which is registering them in a group. , Daniel Smith wrote:


Question: preferredPathLock sync.Mutex preferredPath     = "" workingDirPath    = "" <del> homeDirPath       = os.Getenv("HOME") </del> <ins> homeDirPath, _    = os.UserHomeDir() </ins> rootDirPath       = "/" homeJsonDirPath   = filepath.Join(homeDirPath, ".docker") rootJsonDirPath   = filepath.Join(rootDirPath, ".docker")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is securit pty related, please disclose irivately via --What happened: There is a Windows minion node in my k8s cluster, I have specified "--pod-infra-container-image" for kubelet which is pointing to a private docker repository. The kubelet running as a Windows service, service account is LocalSystem: I have login to the repository with Administrator, and then copied from Administrator's profile folder to "C:windowssystem32configsystemprofile.docker". When I schedule pod on Windows node, kubelet cannot pull infra container image from the private repository with error: I can see there is a PR solved this issue, but it seems not work on Windows. What you expected to happen: Able to pull infra container image from a private repository on Windows node. How to reproduce it (as minimally and precisely as possible): Run kubelet as a Windows service and specify --pod-infra-container-image to point to a private repository with basic auth. Anything else we need to know?: Environment: Kubernetes version (use ): Linux node: Windows node: Cloud provider or hardware configuration: OS (e.g: ): Windows Server core 1709 Kernel (e.g. ): 10.0..125 Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Your private repo only provides http service? does it need authentication?
the private repo built by Nexus, it requires basic authentication
/assign
Hello, First of all, the PR you mentioned was closed, but not merged. Secondly, why do you need to use a private repo for the [2]. Even if you opt to use a private repo, if you pull it after you've authenticated, it would fix your problem. Finally, I can confirm that kubelet is not able to pull the configured private image case). [1] [2] [3]
Ok, so, I have a couple of updates regarding this.


Question: preferredPathLock sync.Mutex preferredPath     = "" workingDirPath    = "" <del> homeDirPath       = os.Getenv("HOME") </del> <ins> homeDirPath, _    = os.UserHomeDir() </ins> rootDirPath       = "/" homeJsonDirPath   = filepath.Join(homeDirPath, ".docker") rootJsonDirPath   = filepath.Join(rootDirPath, ".docker")

First of all, indeed, the PR mentioned in the issue was closed without merging, but another one was merged in its stead: [1] Secondly, it seems that kubelet is trying multiple credential providers, including for azure, aws, gcp, kubernetes secrets, and local configurations, as it can be seen here: [2]. I actually ended up getting it to work by setting up the local configrations properly. From what I can see in the code, it could still be a Windows related issue [3] (the env var name should be ), but it can be avoided, since it will check multiple locations for the configurations: I0513 16:22:05. ] looking for at C:varlibkubelet I0513 16:22:05. ] looking for at C:k I0513 16:22:05. ] looking for at C:k.docker I0513 16:22:05. ] looking for at C:.docker I0513 16:22:05. ] looking for .dockercfg at C:varlibkubelet.dockercfg I0513 16:22:05. ] looking for .dockercfg at C:k.dockercfg I0513 16:22:05. ] looking for .dockercfg at C:k.dockercfg I0513 16:22:05. ] looking for .dockercfg at C:.dockercfg I0513 16:22:05. ] Unable to parse Docker config file: couldn't find valid .dockercfg after checking in [/var/lib/kubelet /] So, basically, I just had to copy the file to one of those locations. [1] [2] [3]
Yes, %HOMEDRIVE%+%HOMEPATH% is the right profile path: HOMEDRIVE=C: HOMEPATH=UsersAdministrator
Hm, keep in mind that you're trying to run the Kubelet as a service, under the ` (probably).
sure, just to show the name of environment variables
this has a PR open, so adding /milestone v1.15


Question: github.com/mitchellh/go-wordwrap # github.com/mitchellh/mapstructure v1.1.2 => github.com/mitchellh/mapstructure v1.1.2 github.com/mitchellh/mapstructure <del> # github.com/moby/ipvs v1.0.0 => github.com/moby/ipvs v1.0.0 </del> <ins> # github.com/moby/ipvs v1.0.1 => github.com/moby/ipvs v1.0.1 </ins> github.com/moby/ipvs # github.com/moby/term v0.0.0-20200312100748-672ec06f55cd => github.com/moby/term v0.0.0-20200312100748-672ec06f55cd github.com/moby/term

Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error.