Question: func (c *csiMountMgr) SetUpAt(dir string, mounterArgs volume.MounterArgs) error { klog.V(4).Infof(log("Mounter.SetUpAt(%s)", dir)) <del> corruptedDir := false mounted, err := isDirMounted(c.plugin, dir) if err != nil { if isCorruptedDir(dir) { corruptedDir = true // leave to CSI driver to handle corrupted mount klog.Warning(log("mounter.SetUpAt detected corrupted mount for dir [%s]", dir)) } else { return errors.New(log("mounter.SetUpAt failed while checking mount status for dir [%s]: %v", dir, err)) } } if mounted && !corruptedDir { klog.V(4).Info(log("mounter.SetUpAt skipping mount, dir already mounted [%s]", dir)) return nil } </del> csi, err := c.csiClientGetter.Get() if err != nil { return volumetypes.NewTransientOperationFailure(log("mounter.SetUpAt failed to get CSI client: %v", err))

For the mount point checking here before calling NodePublishVolume, I am wondering it is just for performance optimization, or it has other purposes. Have concern about some drivers might depend on this check if they are not implemented with idenpotency.
Those drivers would be operating in violation of the specification. I'd prefer that we don't penalize already conforming and yet-to-be written drivers in order to accommodate these existing drivers—if they even exist.
More fallout in last night's testing: I introduced a bug into my driver that caused it to panic during ("Writing that data to a set of files in the mounted filesystem"). My handler function is set up to unmount a successfully mounted volume via if it doesn't complete steps 2 and 3 above, in order to deal with this issue's principal complaint. My driver container would crash due to this panic. The kubelet would call on my driver twice, it would crash twice, but the pod still started running. I'm not yet sure whether or not the filesystem got unmounted successfully via the on the first call, but somehow the pod started despite my handler function never completing successfully. I'm going to try to get some more debugging time on this later today to see if I can figure out what the kubelet was doing there.
Studying my code more, I see that my function only unmounted the mounted directory when the containing handler function was returning an error. When unwinding due to a panic, the named return value of type I was inspecting was still nil. Revising the criteria in that function covers both failure reasons (returning an and unwinding due to a panic). Regardless, this further underscores the problem: If a driver makes it as far as mounting a filesystem at the target directory, but fails afterward—including the process crashing and never responding positively to the RPC—the mounting pod will start. That's likely to then cause the pod to fail, for lack of the files it expected to find mounted.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.


Question: func (c *csiMountMgr) SetUpAt(dir string, mounterArgs volume.MounterArgs) error { klog.V(4).Infof(log("Mounter.SetUpAt(%s)", dir)) <del> corruptedDir := false mounted, err := isDirMounted(c.plugin, dir) if err != nil { if isCorruptedDir(dir) { corruptedDir = true // leave to CSI driver to handle corrupted mount klog.Warning(log("mounter.SetUpAt detected corrupted mount for dir [%s]", dir)) } else { return errors.New(log("mounter.SetUpAt failed while checking mount status for dir [%s]: %v", dir, err)) } } if mounted && !corruptedDir { klog.V(4).Info(log("mounter.SetUpAt skipping mount, dir already mounted [%s]", dir)) return nil } </del> csi, err := c.csiClientGetter.Get() if err != nil { return volumetypes.NewTransientOperationFailure(log("mounter.SetUpAt failed to get CSI client: %v", err))

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: Status: v1.ConditionTrue, }) <del> hostIP, err := kl.getHostIPAnyWay() if err != nil { glog.V(4).Infof("Cannot get host IP: %v", err) return *s } s.HostIP = hostIP.String() if kubecontainer.IsHostNetworkPod(pod) && s.PodIP == "" { s.PodIP = hostIP.String() </del> <ins> if kl.kubeClient != nil { hostIP, err := kl.getHostIPAnyWay() if err != nil { glog.V(4).Infof("Cannot get host IP: %v", err) } else { s.HostIP = hostIP.String() if kubecontainer.IsHostNetworkPod(pod) && s.PodIP == "" { s.PodIP = hostIP.String() } } </ins> } return *s }

http://k8s- /priority failing-test /kind bug /status approved-for-milestone /sig cluster-lifecycle the job is on the master-blocking dashboard and the e2e test has been failing, please take a look cc
testgrid change suggests
(author of the change in that range) -- any idea whether might have broken reboot tests?
I reverted the change as it may lead to this regression. The other change is really small () and unlikely related to this. Still the continues to say otherwise, I think we can reopen this issue.
The reboot test is still failing.
Just opened to achieve the revert.
Is the test passing now? Can this issue be closed?
looks green now


Question: # These are "local" but are visible to and relied on by functions this # function calls.  They are effectively part of the calling API to # build_binaries_for_platform. <del> local goflags goldflags goasmflags gogcflags gotags </del> <ins> local goflags goldflags gogcflags gotags </ins> <del> # This is $(pwd) because we use run-in-gopath to build.  Once that is # excised, this can become ${KUBE_ROOT}. local trimroot # two lines to appease shellcheck SC2155 trimroot=$(pwd) goasmflags="all=-trimpath=${trimroot}" </del> <ins> goflags=() gogcflags="${GOGCFLAGS:-}" goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" </ins> <del> gogcflags="all=-trimpath=${trimroot} ${GOGCFLAGS:-}" </del> if [[ "${DBG:-}" == 1 ]]; then # Debugging - disable optimizations and inlining and trimPath <del> gogcflags="${GOGCFLAGS:-} all=-N -l" goasmflags="" fi goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" if [[ "${DBG:-}" != 1 ]]; then # Not debugging - disable symbols and DWARF. </del> <ins> gogcflags="${gogcflags} all=-N -l" else # Not debugging - disable symbols and DWARF, trim embedded paths </ins> goldflags="${goldflags} -s -w" <ins> goflags+=("-trimpath") </ins> fi # Extract tags if any specified in GOFLAGS

While working on for , I found that our kubernetes1.24 package stopped to produce bit-identical results on 2 fresh builds. The first chunk is likely just the hash of the remainder, so can be ignored. The other chunks indicate that a random build temporary path is used for Sources should be able to reproduce bit-identical binaries e.g. for SLSA level4 On Debian or openSUSE do No response <detailsWARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-05-05T00:00:00Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 console $ cat /etc/os-release $ uname -a Linux bmwiedemann-factory 5.18.6-1-default SMP PREEMPTDYNAMIC Thu Jun 23 05:46:18 UTC 2022 () x8664 x8664 x86_64 GNU/Linux </details<detailsgo1.18 </details<details</details<details</details
/sig release /area build-release
can you please post a link to the full logs from the build?
any news on this? Haven't heard from (context: we worked to fix this a couple of times in )
Unfortunately not, is this something we can fix on the OBS side? If the go build timestamp () is the main issue here then we can approach a fix in our build pipeline.
It would be good if would not be embedded in the first place. It might be possible to replace it somehow in our builds, but all other people wanting reproducible builds, would need to replicate that.
full logs from the build: from kubernetes-1.24.3 built 2022-08-18
can you please try setting GOCACHE=off when you run the build? I am suspecting that the temp directory is being generated here: (or add an export in for that env var)
I to but still got these diffs.


Question: # These are "local" but are visible to and relied on by functions this # function calls.  They are effectively part of the calling API to # build_binaries_for_platform. <del> local goflags goldflags goasmflags gogcflags gotags </del> <ins> local goflags goldflags gogcflags gotags </ins> <del> # This is $(pwd) because we use run-in-gopath to build.  Once that is # excised, this can become ${KUBE_ROOT}. local trimroot # two lines to appease shellcheck SC2155 trimroot=$(pwd) goasmflags="all=-trimpath=${trimroot}" </del> <ins> goflags=() gogcflags="${GOGCFLAGS:-}" goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" </ins> <del> gogcflags="all=-trimpath=${trimroot} ${GOGCFLAGS:-}" </del> if [[ "${DBG:-}" == 1 ]]; then # Debugging - disable optimizations and inlining and trimPath <del> gogcflags="${GOGCFLAGS:-} all=-N -l" goasmflags="" fi goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" if [[ "${DBG:-}" != 1 ]]; then # Not debugging - disable symbols and DWARF. </del> <ins> gogcflags="${gogcflags} all=-N -l" else # Not debugging - disable symbols and DWARF, trim embedded paths </ins> goldflags="${goldflags} -s -w" <ins> goflags+=("-trimpath") </ins> fi # Extract tags if any specified in GOFLAGS

So i built golang from source, peppered it with changes to track down where this is getting created and found that it is coming from: I don't know yet how to fix it. will dig more when i can (travelling for the next few days)
Would it work to override the , I see that they use the : Ref: https://reproducible- Edit: Probably not, we may make it configurable in golang :thinking:
this is so tricky :( will dig more when i get back from my event this week
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned".


Question: # These are "local" but are visible to and relied on by functions this # function calls.  They are effectively part of the calling API to # build_binaries_for_platform. <del> local goflags goldflags goasmflags gogcflags gotags </del> <ins> local goflags goldflags gogcflags gotags </ins> <del> # This is $(pwd) because we use run-in-gopath to build.  Once that is # excised, this can become ${KUBE_ROOT}. local trimroot # two lines to appease shellcheck SC2155 trimroot=$(pwd) goasmflags="all=-trimpath=${trimroot}" </del> <ins> goflags=() gogcflags="${GOGCFLAGS:-}" goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" </ins> <del> gogcflags="all=-trimpath=${trimroot} ${GOGCFLAGS:-}" </del> if [[ "${DBG:-}" == 1 ]]; then # Debugging - disable optimizations and inlining and trimPath <del> gogcflags="${GOGCFLAGS:-} all=-N -l" goasmflags="" fi goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" if [[ "${DBG:-}" != 1 ]]; then # Not debugging - disable symbols and DWARF. </del> <ins> gogcflags="${gogcflags} all=-N -l" else # Not debugging - disable symbols and DWARF, trim embedded paths </ins> goldflags="${goldflags} -s -w" <ins> goflags+=("-trimpath") </ins> fi # Extract tags if any specified in GOFLAGS

<detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is still relevant even with 1.27. It might be worth revisiting this issue again. /reopen /triage accepted
Reopened this issue. <detailsIn response to : This is still relevant even with 1.27. It might be worth revisiting this issue again. /reopen /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
latest run -
I found that our package has the same issue, so it probably it is a toolchain issue in golang - and much faster to debug outside of the slow kubernetes build. =filed
AIUI, the conclusion by the golang people seems to be that commit is not the right approach and one should use with the go call instead. Not sure how that interacts with other parts of your build.
cc (please see referenced commit above!)
I see. I inherited the gcflags before so I plead innocence. It seems plausible to change it from to - I don't know when I will have time to look at that, though.


Question: # These are "local" but are visible to and relied on by functions this # function calls.  They are effectively part of the calling API to # build_binaries_for_platform. <del> local goflags goldflags goasmflags gogcflags gotags </del> <ins> local goflags goldflags gogcflags gotags </ins> <del> # This is $(pwd) because we use run-in-gopath to build.  Once that is # excised, this can become ${KUBE_ROOT}. local trimroot # two lines to appease shellcheck SC2155 trimroot=$(pwd) goasmflags="all=-trimpath=${trimroot}" </del> <ins> goflags=() gogcflags="${GOGCFLAGS:-}" goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" </ins> <del> gogcflags="all=-trimpath=${trimroot} ${GOGCFLAGS:-}" </del> if [[ "${DBG:-}" == 1 ]]; then # Debugging - disable optimizations and inlining and trimPath <del> gogcflags="${GOGCFLAGS:-} all=-N -l" goasmflags="" fi goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" if [[ "${DBG:-}" != 1 ]]; then # Not debugging - disable symbols and DWARF. </del> <ins> gogcflags="${gogcflags} all=-N -l" else # Not debugging - disable symbols and DWARF, trim embedded paths </ins> goldflags="${goldflags} -s -w" <ins> goflags+=("-trimpath") </ins> fi # Extract tags if any specified in GOFLAGS

My patch doesn't seem to fix the whole problem: thockin-glaptop4 kubernetes buildtrimpath /$ make kubelet +++ [1114 19:10:12] Building go targets for linux/amd64 (non-static) thockin-glaptop4 kubernetes buildtrimpath /$ md5sum output/bin/kubelet output/bin/kubelet thockin-glaptop4 kubernetes buildtrimpath /$ make kubelet +++ [1114 19:12:40] Building go targets for linux/amd64 (non-static) thockin-glaptop4 kubernetes buildtrimpath /$ md5sum output/bin/kubelet output/bin/kubelet thockin-glaptop4 kubernetes build_trimpath /$
diff --- a 2023-11-14 19:13:45. -0800 +++ b 2023-11-14 19:13:56. -0800 -1,4 +1,4 -38rZHdQckYF9EEzmndR/BAXOwzCcq0Mk1UHT62ha/QLiRgXP4oojWh3oWP9/TWG7KW90IL8vKt1UOEPR +LeLArvl7sgWAd55ROKN2/BAXOwzCcq0Mk1UHT62ha/QLiRgXP4oojWh3oWP9/cYb5ZGQwa6js077YiAvK /lib64/ld-linux-x86- AUATUSH []AA] -,8 +,8 go1.21.3 runtime/cgo: call frame too large -2023-11-15T03:12:40Z -2023-11-15T03:12:40Z +2023-11-15T03:13:47Z +2023-11-15T03:13:47Z pthreadcreate failed: %s bad stack bounds: lo=%p hi=%p expand 32-byte kexpand 32-byte k
That seems unrelated. I tested with and it made our openSUSE package fully reproducible. Your diff contains a timestamp - maybe would get rid of that. The top part of the diff is just the go build-id hash of the remainder and can be ignored.


Question: # These are "local" but are visible to and relied on by functions this # function calls.  They are effectively part of the calling API to # build_binaries_for_platform. <del> local goflags goldflags goasmflags gogcflags gotags </del> <ins> local goflags goldflags gogcflags gotags </ins> <del> # This is $(pwd) because we use run-in-gopath to build.  Once that is # excised, this can become ${KUBE_ROOT}. local trimroot # two lines to appease shellcheck SC2155 trimroot=$(pwd) goasmflags="all=-trimpath=${trimroot}" </del> <ins> goflags=() gogcflags="${GOGCFLAGS:-}" goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" </ins> <del> gogcflags="all=-trimpath=${trimroot} ${GOGCFLAGS:-}" </del> if [[ "${DBG:-}" == 1 ]]; then # Debugging - disable optimizations and inlining and trimPath <del> gogcflags="${GOGCFLAGS:-} all=-N -l" goasmflags="" fi goldflags="all=$(kube::version::ldflags) ${GOLDFLAGS:-}" if [[ "${DBG:-}" != 1 ]]; then # Not debugging - disable symbols and DWARF. </del> <ins> gogcflags="${gogcflags} all=-N -l" else # Not debugging - disable symbols and DWARF, trim embedded paths </ins> goldflags="${goldflags} -s -w" <ins> goflags+=("-trimpath") </ins> fi # Extract tags if any specified in GOFLAGS

Thanks for confirming
Will there be backports to 1.24++ ? I found it applies cleanly to 1.26++ and works with some small tweaks in 1.24 and 1.25. If not, I'd submit the patch for our openSUSE packages.
1.24 and 1.25 reached EOL and we don't support these branches any longer. We can cherry-pick this to 1.26+ if folks ( agree
Cherry-picks: 1.29 1.28 1.27 1.26


Question: Containers: []v1.Container{ { Name:    contName, <del> Image:   BusyBoxImage, </del> <ins> Image:   AgnHostImage, </ins> Command: command, }, },

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: Since when has it been failing: Ever Reason for failure: Vm on Azure with no public IP assigned aren't able to perform a command to the internet. Anything else we need to know: The test (see ) tries to test the internet connection for a pod by pinging 8.8.8.8. But azure does not provide ICMP support for VM without a public IP therefore this test cannot succeed.(see ) A fix would be to change the test method to a tcp request instead of a ping request (the test already performs a http get to in its BeforeSuite (see ))
/sig network /sig testing /sig azure
you may refer to (Pods don't have connectivity to external networks )
cc WDYT about the proposal to change this from an ICMP call? We probably need input from sig-architecture as well since this as well /sig architecture
/triage unresolved
An alternative is to issue a TCP/UDP request to 8.8.8.8 on port 53.
ICMP is dropped by Azure for security reasons. This is intended. Can use curl as a replacement. I will fix the corresponding test.
SGTM
Can you please review the related PR? I'll in the PR again. Thanks!


Question: } } <ins> func nodeSchedulingPropertiesChanged(newNode *v1.Node, oldNode *v1.Node) bool { if nodeSpecUnschedulableChanged(newNode, oldNode) { return true } if nodeAllocatableChanged(newNode, oldNode) { return true } if nodeLabelsChanged(newNode, oldNode) { return true } if nodeTaintsChanged(newNode, oldNode) { return true } if nodeConditionsChanged(newNode, oldNode) { return true } return false } func nodeAllocatableChanged(newNode *v1.Node, oldNode *v1.Node) bool { return !reflect.DeepEqual(oldNode.Status.Allocatable, newNode.Status.Allocatable) } func nodeLabelsChanged(newNode *v1.Node, oldNode *v1.Node) bool { return !reflect.DeepEqual(oldNode.GetLabels(), newNode.GetLabels()) } func nodeTaintsChanged(newNode *v1.Node, oldNode *v1.Node) bool { return !reflect.DeepEqual(newNode.Spec.Taints, oldNode.Spec.Taints) } func nodeConditionsChanged(newNode *v1.Node, oldNode *v1.Node) bool { strip := func(conditions []v1.NodeCondition) map[v1.NodeConditionType]v1.ConditionStatus { conditionStatuses := make(map[v1.NodeConditionType]v1.ConditionStatus, len(conditions)) for i := range conditions { conditionStatuses[conditions[i].Type] = conditions[i].Status } return conditionStatuses } return !reflect.DeepEqual(strip(oldNode.Status.Conditions), strip(newNode.Status.Conditions)) } func nodeSpecUnschedulableChanged(newNode *v1.Node, oldNode *v1.Node) bool { return newNode.Spec.Unschedulable != oldNode.Spec.Unschedulable && newNode.Spec.Unschedulable == false } </ins> func (c *configFactory) deleteNodeFromCache(obj interface{}) { var node *v1.Node switch t := obj.(type) {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Today, the scheduler re-processes unschedulable nodes . Node updates are sent by Kubelet frequently (every 10 seconds) and often times they don't change any of the node properties that could potentially make them feasible. We should be smarter about these updates and move pods to the schedulable queue only when certain properties of nodes change that could make them feasible. These properties include node allocatable, node conditions, taints, labels, .... Why is this needed: Improves performance of the scheduler and avoids doing unnecessary attempts to schedule pods. This is particularly helpful when a cluster has a lot of unschedulable pods. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature This is not a feature really. It is performance optimization, but in the absence of such a label, I need to use "feature" here. /sig scheduling
I can help to fix it. /assign mlmhl
The main reason for this problem is / update their status too frequently and most of these updates are heartbeats without any meaningful change. Recently sig-node introduces a new to alleviate this problem. With this mechanism, kubelet will split heartbeat and node status updates into two independent operations, and reduce the frequency of status updates to once per 1 minute(or more). In the long run this mechanism can indeed alleviate our problems, but at present it does not help us much as this is still an alpha feature and the node status updates frequency is still a bit high(per 1 minute right now).
Maybe we can try comparing old and new nodes after stripping their heartbeat info, like this:
I think that had sent earlier is more efficient. However, that PR does not filter out . In Wei's code, the heartbeat time is removed correctly. I think we should fix this issue and resubmit that PR.
could you resubmit a PR to: disregard in conditions comparison: and (optionally) disregard timestamp in taints comparison:
Thanks for all your advice, a new PR is submitted: This PR does not include your second suggestion(disregard timestamp in taints comparison) as I'm not sure if can be changed when node updating.


Question: } } <ins> func nodeSchedulingPropertiesChanged(newNode *v1.Node, oldNode *v1.Node) bool { if nodeSpecUnschedulableChanged(newNode, oldNode) { return true } if nodeAllocatableChanged(newNode, oldNode) { return true } if nodeLabelsChanged(newNode, oldNode) { return true } if nodeTaintsChanged(newNode, oldNode) { return true } if nodeConditionsChanged(newNode, oldNode) { return true } return false } func nodeAllocatableChanged(newNode *v1.Node, oldNode *v1.Node) bool { return !reflect.DeepEqual(oldNode.Status.Allocatable, newNode.Status.Allocatable) } func nodeLabelsChanged(newNode *v1.Node, oldNode *v1.Node) bool { return !reflect.DeepEqual(oldNode.GetLabels(), newNode.GetLabels()) } func nodeTaintsChanged(newNode *v1.Node, oldNode *v1.Node) bool { return !reflect.DeepEqual(newNode.Spec.Taints, oldNode.Spec.Taints) } func nodeConditionsChanged(newNode *v1.Node, oldNode *v1.Node) bool { strip := func(conditions []v1.NodeCondition) map[v1.NodeConditionType]v1.ConditionStatus { conditionStatuses := make(map[v1.NodeConditionType]v1.ConditionStatus, len(conditions)) for i := range conditions { conditionStatuses[conditions[i].Type] = conditions[i].Status } return conditionStatuses } return !reflect.DeepEqual(strip(oldNode.Status.Conditions), strip(newNode.Status.Conditions)) } func nodeSpecUnschedulableChanged(newNode *v1.Node, oldNode *v1.Node) bool { return newNode.Spec.Unschedulable != oldNode.Spec.Unschedulable && newNode.Spec.Unschedulable == false } </ins> func (c *configFactory) deleteNodeFromCache(obj interface{}) { var node *v1.Node switch t := obj.(type) {

If it can be changed, I will add this filter later, otherwise I think we can keep the current implementation to make the compare operation simpler and faster. What's more, just as I said in the PR's description, I'm not sure how much side effects this change has on scheduler's resource consumption, as it always copy node's conditions before comparison.
I am thinking whether we could move this logic(whether node became more schedulable) to kubelet, kubelet reports it through heartbeats.


Question: return o, nil } <ins> // Complete completes the remaining instantiation of the options obj. // In particular, it injects the latest internal versioned ComponentConfig. func (o *Options) Complete(nfs *cliflag.NamedFlagSets) error { cfg, err := latest.Default() if err != nil { return err } hhost, hport, err := splitHostIntPort(cfg.HealthzBindAddress) if err != nil { return err } // Obtain CLI args related with insecure serving. // If not specified in command line, derive the default settings from cfg. insecureServing := nfs.FlagSet("insecure serving") if !insecureServing.Changed("address") { o.CombinedInsecureServing.BindAddress = hhost } if !insecureServing.Changed("port") { o.CombinedInsecureServing.BindPort = hport } // Obtain deprecated CLI args. Set them to cfg if specified in command line. deprecated := nfs.FlagSet("deprecated") if deprecated.Changed("profiling") { cfg.EnableProfiling = o.ComponentConfig.EnableProfiling } if deprecated.Changed("contention-profiling") { cfg.EnableContentionProfiling = o.ComponentConfig.EnableContentionProfiling } if deprecated.Changed("kubeconfig") { cfg.ClientConnection.Kubeconfig = o.ComponentConfig.ClientConnection.Kubeconfig } if deprecated.Changed("kube-api-content-type") { cfg.ClientConnection.ContentType = o.ComponentConfig.ClientConnection.ContentType } if deprecated.Changed("kube-api-qps") { cfg.ClientConnection.QPS = o.ComponentConfig.ClientConnection.QPS } if deprecated.Changed("kube-api-burst") { cfg.ClientConnection.Burst = o.ComponentConfig.ClientConnection.Burst } if deprecated.Changed("lock-object-namespace") { cfg.LeaderElection.ResourceNamespace = o.ComponentConfig.LeaderElection.ResourceNamespace } if deprecated.Changed("lock-object-name") { cfg.LeaderElection.ResourceName = o.ComponentConfig.LeaderElection.ResourceName } // Obtain CLI args related with leaderelection. Set them to cfg if specified in command line. leaderelection := nfs.FlagSet("leader election") if leaderelection.Changed("leader-elect") { cfg.LeaderElection.LeaderElect = o.ComponentConfig.LeaderElection.LeaderElect } if leaderelection.Changed("leader-elect-lease-duration") { cfg.LeaderElection.LeaseDuration = o.ComponentConfig.LeaderElection.LeaseDuration } if leaderelection.Changed("leader-elect-renew-deadline") { cfg.LeaderElection.RenewDeadline = o.ComponentConfig.LeaderElection.RenewDeadline } if leaderelection.Changed("leader-elect-retry-period") { cfg.LeaderElection.RetryPeriod = o.ComponentConfig.LeaderElection.RetryPeriod } if leaderelection.Changed("leader-elect-resource-lock") { cfg.LeaderElection.ResourceLock = o.ComponentConfig.LeaderElection.ResourceLock } if leaderelection.Changed("leader-elect-resource-name") { cfg.LeaderElection.ResourceName = o.ComponentConfig.LeaderElection.ResourceName } if leaderelection.Changed("leader-elect-resource-namespace") { cfg.LeaderElection.ResourceNamespace = o.ComponentConfig.LeaderElection.ResourceNamespace } o.ComponentConfig = *cfg return nil } </ins> func splitHostIntPort(s string) (string, int, error) { host, port, err := net.SplitHostPort(s) if err != nil {

ci-kubernetes-e2e-gci-gce-alpha-features kubetest.Up Jul 02 2021, 01:26:42 UTC+0200 /sig release
based on testgrid probably cc <img width="1203" alt="Screen Shot 2021-07-02 at 4 24 25 PM" src="https://user-"
/cc working on it
/assign
the scheduler fails to start with this error: it seems the feature gates from the command line does not take effect in the defaulting phrase, do you know how to fix it?
ah, found there is a cycle here: The scheduler options , but the default config's default values depend on the feature gates options. I think this disables other feature gate controlled logic in the defaulting too:
I tried to fix it by updating the component config after the CLI flags (including the feature gates) are parsed: PTAL
fixed the flake. is back green now.
thanks!


Question: } } <del> // TestClientReceivedGOAWAY tests the in-flight watch requests will not be affected and new requests use a // connection after client received GOAWAY, and server response watch request with GOAWAY will not break client // watching body read. func TestClientReceivedGOAWAY(t *testing.T) { const ( urlNormal          = "/normal" urlWatch           = "/watch" urlGoaway          = "/goaway" urlWatchWithGoaway = "/watch-with-goaway" ) </del> <ins> const ( urlGet             = "/get" urlPost            = "/post" urlWatch           = "/watch" urlGetWithGoaway   = "/get-with-goaway" urlPostWithGoaway  = "/post-with-goaway" urlWatchWithGoaway = "/watch-with-goaway" ) </ins> <del> const ( // indicate the bytes watch request will be sent // used to check if watch request was broke by GOAWAY watchExpectSendBytes = 5 ) </del> <ins> var ( // responseBody is the response body which test GOAWAY server sent for each request, // for watch request, test GOAWAY server push 1 byte in every second. responseBody = []byte("hello") // responseBodySize is the size of response body which test GOAWAY server sent for watch request, // used to check if watch request was broken by GOAWAY frame. responseBodySize = len(responseBody) // requestPostBody is the request body which client must send to test GOAWAY server for POST method, // otherwise, test GOAWAY server will respond 400 HTTP status code. requestPostBody = responseBody ) // newTestGOAWAYServer return a test GOAWAY server instance. func newTestGOAWAYServer() (*httptest.Server, error) { </ins> watchHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { timer := time.NewTicker(time.Second) <ins> defer timer.Stop() </ins> w.Header().Set("Transfer-Encoding", "chunked") w.WriteHeader(200)

What happened: We enabled feature in our large-scale cluster, and we found client(Kubelet) may got error(500 http code response) like after received GOAWAY frame. I had already re-produce this bug and open an in golang repo. So far I think this bug only occur with a small probability when client request server in high concurrency use a same httpclient instance, and it's a golang sdk bug in http client. Before I solved this issue, I suggest do NOT enable GOAWAY() feature. As it's a golang sdk bug(currently I known and doubt) in http client, revert GOAWAY feature(codes) is not necessary.
cc
/sig api-machinery
Thanks for the report.
Based on not setting the field, I think it defaults that way, so that's good. We will leave the option and filter so that when golang updates its client, we will have a simple way to turn it on and execute it.
Sure. I am trying to fix the golang http client issue and will update progress at . After golang http client issue fixed, I will add more concurrent test cases to see if there are any other bugs may occur by GOAWAY feature. BTW, we still enable GOAWAY feature in our production cluster, because there is no any evidence told us this error will influence SLO, as all most components(kubelet, controller-manager, scheduler) is controller work mode and will re-try after got error. /assign
Golang version or required, which will release at 07/01/2020.
Kubernetes v1.19(or master branch) now is using golang 1.15, and golang 1.15 has fixed the issue. And is on going. Let me add more test cases to ensure the error will not happen again.
I think we should re-open this issue because of . /re-open


Question: o.Resources = args o.restClientGetter = f <del> fullCmdName := "" </del> cmdParent := cmd.Parent() if cmdParent != nil { <del> fullCmdName = cmdParent.CommandPath() </del> <ins> o.ParentCommandName = cmdParent.CommandPath() </ins> } <del> if len(fullCmdName) > 0 && cmdutil.IsSiblingCommandExists(cmd, "describe") { o.SuggestedCmdUsage = fmt.Sprintf("Use '%s describe pod/%s -n %s' to see all of the containers in this pod.", fullCmdName, o.PodName, o.Namespace) </del> <ins> if len(o.ParentCommandName) > 0 && cmdutil.IsSiblingCommandExists(cmd, "describe") { o.EnableSuggestedCmdUsage = true </ins> } config, err := f.ToRESTConfig()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The pod name is missing in suggested cmd usage when executing : What you expected to happen: The suggested cmd usage should be a complete cmd like the one got from How to reproduce it (as minimally and precisely as possible): Use to attach any pod/deployment/service, the PodName will be missing in the suggested cmd usage like above. Anything else we need to know?: It works fine for kubectl < v1.12 Environment: Kubernetes version (use ): v1.13.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli /assign


Question: reportedVerb := cleanVerb(CanonicalVerb(strings.ToUpper(req.Method), scope), req) if requestInfo.IsResourceRequest { <ins> e = longRunningRequestsGauge.WithContext(req.Context()).WithLabelValues(reportedVerb, requestInfo.APIGroup, requestInfo.APIVersion, requestInfo.Resource, requestInfo.Subresource, scope, component) </ins> g = longRunningRequestGauge.WithContext(req.Context()).WithLabelValues(reportedVerb, requestInfo.APIGroup, requestInfo.APIVersion, requestInfo.Resource, requestInfo.Subresource, scope, component) } else { <ins> e = longRunningRequestsGauge.WithContext(req.Context()).WithLabelValues(reportedVerb, "", "", "", requestInfo.Path, scope, component) </ins> g = longRunningRequestGauge.WithContext(req.Context()).WithLabelValues(reportedVerb, "", "", "", requestInfo.Path, scope, component) } <ins> e.Inc() </ins> g.Inc() <del> defer g.Dec() </del> <ins> defer func() { e.Dec() g.Dec() }() </ins> fn() }

apiserverlongrunninggauge and apiserverregisterwatchers are kinda redundant . We discussed this in our SIG instrumentation meeting and while it's clear that apiserverlongrunninggauge, the name does not conform to instrumentation guidelines (it was created before those guidelines existed). What we want to do is deprecate both metrics and create a new one which actually does conform to our naming conventions probably something like . has volunteered to undertake this. /cc /sig instrumentation
/assign
GitHub didn't allow me to assign the following users: yan-lgtm. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted
Hello! Have you still working on this issue? I am new to the community, and can I try it?
Hey I the deprecate in If you want to add the new metric, feel free to do so.
Can I add the new metric? If I can, can you give me some advice about the new metric? Thank you~
/assign


Question: }, }, }, <del> AccessModes: []v1.PersistentVolumeAccessMode{v1.ReadWriteOnce}, </del> <ins> AccessModes: []v1.PersistentVolumeAccessMode{am}, </ins> }, <del> } return pv, nil </del> <ins> }, nil </ins> } // TranslateInTreePVToCSI takes a PV with GCEPersistentDisk set from in-tree

When running CSI migration tests focussing on multi-attach scearnios with GCE PD inline volumes that are read-only, the following error pops up in the logs and the second attachment takes 6 mins: For read-only inline volumes, the above warning/error around is unexpected with CSI migration enabled and needs to be root-caused/resolved. What happened: CSI migration tests for read-only inline GCE PD volumes occasionally fails with the above messages in the logs when executing tests like What you expected to happen: The tests and multi-attach scenarios to execute smoothly with no warnings in the logs. How to reproduce it (as minimally and precisely as possible): Execute with focus set to and run a few times. Anything else we need to know?: This has been observed with CSI migration enabled and the tests exercising read-only inline volume scenarios. Environment: Kubernetes version (use ): v1.16 Cloud provider or hardware configuration: GCE OS (e.g: ): Kernel (e.g. ): Install tools: kube- Network plugin and version (if this is a network-related bug): Others:
/cc
/sig storage
Looks like hard coding access modes to ReadWriteOnce during translation for GCE PD at is causing isMultiAttachForbidden to return leading down to We need to set the access mode to ReadOnlyMany for in-tree inline volumes like in GCE PD (as well as others like EBS that support multi-attach) depending on instead of always setting it to ReadWriteOnce.
I can pick up this fix /assign


Question: <ins> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This command resizes a replication controller to 0. KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../.. KUBECTL="${KUBE_ROOT}/cluster/kubectl.sh" RESIZE="${KUBE_ROOT}/contrib/rctools/resize.sh" if [[ $# != 1 ]] ; then echo "usage: $0 <replication controller name>" >&2 exit 1 fi rc="$1" "${RESIZE}" "$rc" 0 # kubectl describe output includes a line like: # Replicas:       2 current / 2 desired # Wait until it shows 0 pods while true; do pods=$(${KUBECTL} describe rc "$rc" | awk '/^Replicas:/{print $2}') if [[ "$pods" -eq 0 ]] ; then exit 0 else echo "$pods remaining..." fi sleep 1 done exit 1 </ins>

kubectl describe emits 1024 NULL characters prior to actual text. This prevents all text-based processing tools from working on its output. /cc
This has got to be due to:
(IRL discussion: there's no reason to pass 1024 0 bytes to NewBuffer.)
Fixing in PR .


Question: <del> # Kubernetes Conformance Test Suite - v1.9 </del> <ins> # Kubernetes Conformance Test Suite -  {{.Version}} </ins> ## **Summary** This document provides a summary of the tests included in the Kubernetes conformance test suite.

What happened: The Conformance Document currently has a static header that is displayed from a file. The release information in the header should reflect proper release number so that the document reflect the current state. What you expected to happen: Should show proper current release information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
in reaction to comments is still hardcoded to mention v1.9 at the top, could we parameterize this?
/sig testing /area conformance /release-note-none
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
/priority backlog
This seems to have been addressed at Should we just close this?
No, specifically did not address this issue, this should stay open. See


Question: } func (plugin *azureDataDiskPlugin) IsMigratedToCSI() bool { <del> return false </del> <ins> return utilfeature.DefaultFeatureGate.Enabled(features.CSIMigration) && utilfeature.DefaultFeatureGate.Enabled(features.CSIMigrationAzureDisk) </ins> } func (plugin *azureDataDiskPlugin) RequiresRemount() bool {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Azure is not listed as "migratable" in could refer to Why is this needed:
/sig azure /assign


Question: container.Add(ws) } <ins> // stripVersionEncoder strips APIVersion field from the encoding output. It's // used to keep the responses at the discovery endpoints backward compatible // with release-1.1, when the responses have empty APIVersion. type stripVersionEncoder struct { encoder    runtime.Encoder serializer runtime.Serializer } func (c stripVersionEncoder) EncodeToStream(obj runtime.Object, w io.Writer, overrides ...unversioned.GroupVersion) error { buf := bytes.NewBuffer([]byte{}) err := c.encoder.EncodeToStream(obj, buf, overrides...) if err != nil { return err } roundTrippedObj, gvk, err := c.serializer.Decode(buf.Bytes(), nil, nil) if err != nil { return err } gvk.Group = "" gvk.Version = "" roundTrippedObj.GetObjectKind().SetGroupVersionKind(gvk) return c.serializer.EncodeToStream(roundTrippedObj, w) } // StripVersionNegotiatedSerializer will return stripVersionEncoder when // EncoderForVersion is called. See comments for stripVersionEncoder. type StripVersionNegotiatedSerializer struct { runtime.NegotiatedSerializer } func (n StripVersionNegotiatedSerializer) EncoderForVersion(serializer runtime.Serializer, gv unversioned.GroupVersion) runtime.Encoder { encoder := n.NegotiatedSerializer.EncoderForVersion(serializer, gv) return stripVersionEncoder{encoder, serializer} } func keepUnversioned(group string) bool { return group == "" || group == "extensions" } </ins> // Adds a service to return the supported api versions at /apis. func AddApisWebService(s runtime.NegotiatedSerializer, container *restful.Container, apiPrefix string, f func(req *restful.Request) []unversioned.APIGroup) { <del> rootAPIHandler := RootAPIHandler(s, f) </del> <ins> // Because in release 1.1, /apis returns response with empty APIVersion, we // use StripVersionNegotiatedSerializer to keep the response backwards // compatible. ss := StripVersionNegotiatedSerializer{s} rootAPIHandler := RootAPIHandler(ss, f) </ins> ws := new(restful.WebService) ws.Path(apiPrefix) ws.Doc("get available API versions")

Kubectl client Kubectl api-versions should check if v1 is in available api versions [Conformance] Expected error: <_errors.errorString | 0xc208641770: { s: "Error running &{/jenkins-master-data/jobs/kubernetes-upgrade-gke-1.1-master-step1-deploy/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://130.211.136.232 --kubeconfig=/var/lib/jenkins/jobs/kubernetes-upgrade-gke-1.1-master-step1- api-versions] [] error: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1"n [] 0xc208382fa0 exit status 1 true [0xc208627148 0xc208627168 0xc208627188] [0xc208627148 0xc208627168 0xc208627188] [0xc208627160 0xc208627180] [0x6d81a0 0x6d81a0] 0xc2082fb2c0}:nCommand stdout:nnstderr:nerror: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1"nn", } Error running &{/jenkins-master-data/jobs/kubernetes-upgrade-gke-1.1-master-step1-deploy/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://130.211.136.232 --kubeconfig=/var/lib/jenkins/jobs/kubernetes-upgrade-gke-1.


Question: container.Add(ws) } <ins> // stripVersionEncoder strips APIVersion field from the encoding output. It's // used to keep the responses at the discovery endpoints backward compatible // with release-1.1, when the responses have empty APIVersion. type stripVersionEncoder struct { encoder    runtime.Encoder serializer runtime.Serializer } func (c stripVersionEncoder) EncodeToStream(obj runtime.Object, w io.Writer, overrides ...unversioned.GroupVersion) error { buf := bytes.NewBuffer([]byte{}) err := c.encoder.EncodeToStream(obj, buf, overrides...) if err != nil { return err } roundTrippedObj, gvk, err := c.serializer.Decode(buf.Bytes(), nil, nil) if err != nil { return err } gvk.Group = "" gvk.Version = "" roundTrippedObj.GetObjectKind().SetGroupVersionKind(gvk) return c.serializer.EncodeToStream(roundTrippedObj, w) } // StripVersionNegotiatedSerializer will return stripVersionEncoder when // EncoderForVersion is called. See comments for stripVersionEncoder. type StripVersionNegotiatedSerializer struct { runtime.NegotiatedSerializer } func (n StripVersionNegotiatedSerializer) EncoderForVersion(serializer runtime.Serializer, gv unversioned.GroupVersion) runtime.Encoder { encoder := n.NegotiatedSerializer.EncoderForVersion(serializer, gv) return stripVersionEncoder{encoder, serializer} } func keepUnversioned(group string) bool { return group == "" || group == "extensions" } </ins> // Adds a service to return the supported api versions at /apis. func AddApisWebService(s runtime.NegotiatedSerializer, container *restful.Container, apiPrefix string, f func(req *restful.Request) []unversioned.APIGroup) { <del> rootAPIHandler := RootAPIHandler(s, f) </del> <ins> // Because in release 1.1, /apis returns response with empty APIVersion, we // use StripVersionNegotiatedSerializer to keep the response backwards // compatible. ss := StripVersionNegotiatedSerializer{s} rootAPIHandler := RootAPIHandler(ss, f) </ins> ws := new(restful.WebService) ws.Path(apiPrefix) ws.Doc("get available API versions")

1-master-step1- api-versions] [] error: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1" [] 0xc208382fa0 exit status 1 true [0xc208627148 0xc208627168 0xc208627188] [0xc208627148 0xc208627168 0xc208627188] [0xc208627160 0xc208627180] [0x6d81a0 0x6d81a0] 0xc2082fb2c0}: Command stdout: stderr: error: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1" not to have occurred* See and Investigating. cc
Okiedokie. Looks like the v1.1 client is getting a response from the v1.2 master, but the v1.1 client doesn't know how to handle the v1 APIVersions object, because in v1.1, that object was never expected to have an . From a 1.1 server, we get which doesn't include a . We should expect to have plenty of pre-v1.1.7 clients floating around, and releasing a fixed v1.1.8 probably isn't going to solve this; if we want to be backward compatible, we need to not return an string with unversioned kinds like . Looking into more details about how we got here... cc cc
I have a maybe feeling that this is caused by ? In particular, we went from: to
Probably it's caused by this:
Let me double check the code. cc
What about that line? I've never dealt with the version negotiation; should the server and client be negotiating in a way that the server can return an object without ?
More precisely it's this line: This line sets the version to "v1". In , during encoding, we set unversioned object's APIVersion field to "v1".
I think in the long run, because we expect to move many unversioned types to group "common", so they should be versioned, so making APIVersion=v1 is the right move. Back to the backwards compatibility issue, how's the situation at other url, like http://127.0.0.1:8080/apis/extensions?


Question: container.Add(ws) } <ins> // stripVersionEncoder strips APIVersion field from the encoding output. It's // used to keep the responses at the discovery endpoints backward compatible // with release-1.1, when the responses have empty APIVersion. type stripVersionEncoder struct { encoder    runtime.Encoder serializer runtime.Serializer } func (c stripVersionEncoder) EncodeToStream(obj runtime.Object, w io.Writer, overrides ...unversioned.GroupVersion) error { buf := bytes.NewBuffer([]byte{}) err := c.encoder.EncodeToStream(obj, buf, overrides...) if err != nil { return err } roundTrippedObj, gvk, err := c.serializer.Decode(buf.Bytes(), nil, nil) if err != nil { return err } gvk.Group = "" gvk.Version = "" roundTrippedObj.GetObjectKind().SetGroupVersionKind(gvk) return c.serializer.EncodeToStream(roundTrippedObj, w) } // StripVersionNegotiatedSerializer will return stripVersionEncoder when // EncoderForVersion is called. See comments for stripVersionEncoder. type StripVersionNegotiatedSerializer struct { runtime.NegotiatedSerializer } func (n StripVersionNegotiatedSerializer) EncoderForVersion(serializer runtime.Serializer, gv unversioned.GroupVersion) runtime.Encoder { encoder := n.NegotiatedSerializer.EncoderForVersion(serializer, gv) return stripVersionEncoder{encoder, serializer} } func keepUnversioned(group string) bool { return group == "" || group == "extensions" } </ins> // Adds a service to return the supported api versions at /apis. func AddApisWebService(s runtime.NegotiatedSerializer, container *restful.Container, apiPrefix string, f func(req *restful.Request) []unversioned.APIGroup) { <del> rootAPIHandler := RootAPIHandler(s, f) </del> <ins> // Because in release 1.1, /apis returns response with empty APIVersion, we // use StripVersionNegotiatedSerializer to keep the response backwards // compatible. ss := StripVersionNegotiatedSerializer{s} rootAPIHandler := RootAPIHandler(ss, f) </ins> ws := new(restful.WebService) ws.Path(apiPrefix) ws.Doc("get available API versions")

That address also returns a versioned object, which at release 1.1 is unversioned: Do we need to keep all unversioned objects unversioned, or do we only need to ensure the server returns unversioned object at api/?
cc
Per our versioning rules, we should expect a v1.2 server to remain backward compatible with v1.1 clients. I don't have much experience with the api machinery, but if clients weren't expecting an unversioned object, it seems like we can't up and switch to versioned objects.
I think we have to maintain compatibility somehow. any ideas for a fix other than the obvious one?
Yes, we have to maintain backward compatibility with 1.1 (and 1.0) clients. New paths that old clients didn't know about could behave differently than paths that existed in previous releases.
Nope, let's make sure we leave a test and comment.
We can't make the change here though, this would break "versioned unversioned" objects (like )
The other implication of course is now we "versioned unversioned" objects (v1 status) and "unversioned unversioned", which means we shouldn't ever add new objects that don't return a version. I don't think anything in 1.2 qualifies as a new object without a version, just 1.1 objects that didn't have correct versions. Having objects from extensions group be returned without versions terrifies me - I don't see any way of describing that except as a bug.
We may need per API group version rules registered on scheme about whether "unversioned" objects are returned in their scheme version or as empty. That's probably the backward compatible fix that gives us a hope of reasoning about behavior correctly (status in 1.0 is v1, status in extensions might be "", apiversions in any new group is "v1"). The unversioned marker on the objects should ensure 1.2 clients can receive objects in both "" and "v1" form, we should add a test to verify that. I'm against letting empty API versions spread, but agree backcompat for existing groups versions should match previous behavior.
Can we make (or use) a different serializer/codec just for the endpoints that 1.1 clients used?


Question: container.Add(ws) } <ins> // stripVersionEncoder strips APIVersion field from the encoding output. It's // used to keep the responses at the discovery endpoints backward compatible // with release-1.1, when the responses have empty APIVersion. type stripVersionEncoder struct { encoder    runtime.Encoder serializer runtime.Serializer } func (c stripVersionEncoder) EncodeToStream(obj runtime.Object, w io.Writer, overrides ...unversioned.GroupVersion) error { buf := bytes.NewBuffer([]byte{}) err := c.encoder.EncodeToStream(obj, buf, overrides...) if err != nil { return err } roundTrippedObj, gvk, err := c.serializer.Decode(buf.Bytes(), nil, nil) if err != nil { return err } gvk.Group = "" gvk.Version = "" roundTrippedObj.GetObjectKind().SetGroupVersionKind(gvk) return c.serializer.EncodeToStream(roundTrippedObj, w) } // StripVersionNegotiatedSerializer will return stripVersionEncoder when // EncoderForVersion is called. See comments for stripVersionEncoder. type StripVersionNegotiatedSerializer struct { runtime.NegotiatedSerializer } func (n StripVersionNegotiatedSerializer) EncoderForVersion(serializer runtime.Serializer, gv unversioned.GroupVersion) runtime.Encoder { encoder := n.NegotiatedSerializer.EncoderForVersion(serializer, gv) return stripVersionEncoder{encoder, serializer} } func keepUnversioned(group string) bool { return group == "" || group == "extensions" } </ins> // Adds a service to return the supported api versions at /apis. func AddApisWebService(s runtime.NegotiatedSerializer, container *restful.Container, apiPrefix string, f func(req *restful.Request) []unversioned.APIGroup) { <del> rootAPIHandler := RootAPIHandler(s, f) </del> <ins> // Because in release 1.1, /apis returns response with empty APIVersion, we // use StripVersionNegotiatedSerializer to keep the response backwards // compatible. ss := StripVersionNegotiatedSerializer{s} rootAPIHandler := RootAPIHandler(ss, f) </ins> ws := new(restful.WebService) ws.Path(apiPrefix) ws.Doc("get available API versions")

Probably we just need to change the version registered here:
That won't work like you expect - the type registry then has no knowledge in v1 of what schema the object is. That's not what we want - we want to not serialize the version of the object for specific API groups, which is a much more targeted change. This is a special, exceptional case (v1 group should return v1 Status. v1beta1 should not). , Chao Xu wrote:
Changing v1 breaks 1.0 backwards compat (we always returned apiVersion=v1 for 1.0) , Clayton Coleman wrote:
I mean this: Will this work?
I'm not sure without looking at the outcome (which is why unversioned is a bit of a disaster in our codebase). I would much prefer to explicitly identify which objects should have an unset apiversion via a method on the scheme, and do the explicit check at the time we do convert to version. That's an extremely controlled change, is obvious, and doesn't make the unversioned problem worse.
I remember somewhere in asserts version cannot be empty, so I guess my proposed solution won't work without extra work. I'll take the approach suggested by


Question: watchChannel := make(chan *etcd.Response) go s.client.Watch("/registry/services/", 0, true, watchChannel, nil) for { <del> watchResponse := <-watchChannel </del> <ins> watchResponse, ok := <-watchChannel if !ok { break } </ins> s.ProcessChange(watchResponse) } }

kube-proxy regularly stops running with the below panic / stack-trace. This is when configured with vagrant. (It is also running on a laptop, so my unsupported-by-any-evidence theory is that it happens when the laptop resumes after sleep.) Restarting the service causes the proxy to continue working happily. (Should systemd be restarting the service?) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: panic: runtime error: invalid memory address or nil pointer dereference Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: [signal 0xb code=0x1 addr=0x0 pc=0x431b51] Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 22 [running]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0x6b7c00, 0xa51de8) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210066600, 0xc210037660, 0xc210037720, 0x77359400, 0x0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x841 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210066600, 0xc210037660, 0xc210037720,


Question: watchChannel := make(chan *etcd.Response) go s.client.Watch("/registry/services/", 0, true, watchChannel, nil) for { <del> watchResponse := <-watchChannel </del> <ins> watchResponse, ok := <-watchChannel if !ok { break } </ins> s.ProcessChange(watchResponse) } }

0x77359400) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x165 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x1b3 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 1 [select (no cases)]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: () Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x47a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 3 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (loggingT).flushDaemon(0xa57340) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x50 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by ·1 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x276 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 4 [syscall]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: () Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /usr/local/go/src/pkg/runtime/proc.c:1394 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 6 [sleep]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: time.Sleep(0x12a05f200) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: +0x31 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0x7aca18,


Question: watchChannel := make(chan *etcd.Response) go s.client.Watch("/registry/services/", 0, true, watchChannel, nil) for { <del> watchResponse := <-watchChannel </del> <ins> watchResponse, ok := <-watchChannel if !ok { break } </ins> s.ProcessChange(watchResponse) } }

0x12a05f200) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x57 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x99 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 7 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210077120, 0x7ff2602627f0, 0xc210077060, 0xc210037540) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x196 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 8 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210077210, 0x7ff260262840, 0xc210077150, 0xc2100375a0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x196 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 9 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (Mux).listen(0xc2100770c0, 0x718260, 0x4,


Question: watchChannel := make(chan *etcd.Response) go s.client.Watch("/registry/services/", 0, true, watchChannel, nil) for { <del> watchResponse := <-watchChannel </del> <ins> watchResponse, ok := <-watchChannel if !ok { break } </ins> s.ProcessChange(watchResponse) } }

0xc210037600) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·001() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x43 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·001() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x40 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc21000a920, 0x0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x49 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by (*Mux).Channel Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x20a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 10 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·004() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x4d Aug 13 12:04:06 kubernetes-minion-1 systemd[1]: kube-proxy.service: main process exited, code=exited, status=2/INVALIDARGUMENT Aug 13 12:04:06 kubernetes-minion-1 systemd[1]: Unit kube-proxy.service entered failed state.
I looked at this one a little bit and I think it's because of This line isn't present at , but I'm not sure which is newer. Presumably results in the channel being closed (and receiving a zero value that it then tries to dereference) on error.
Diagnostic: This line doesn't check for a closed channel, and it needs to. Yeah, how are we supposed to know if our client is going to close the channel or not?


Question: watchChannel := make(chan *etcd.Response) go s.client.Watch("/registry/services/", 0, true, watchChannel, nil) for { <del> watchResponse := <-watchChannel </del> <ins> watchResponse, ok := <-watchChannel if !ok { break } </ins> s.ProcessChange(watchResponse) } }

I think the behavior has changed at least once already. This is part of why I'm trying to (for our code, anyway) standardize all of our watches with a watch.Interface. I took the liberty of assigning this to you since you're more familiar with this code, but feel free to unassign if you can't take it right now.
Got it
can you give a try?
after i do the minion watch for kubelet config i'll come back and gut this and replace it.
Thanks - with merged I haven't seen this happening. Thanks for the super-quick fix!


Question: } mc := newDiskMetricContext("create", zone) <del> createOp, err := gce.service.Disks.Insert(gce.projectID, zone, diskToCreate).Do() </del> <ins> createOp, err := gce.manager.CreateDisk(gce.projectID, zone, diskToCreate) </ins> if isGCEError(err, "alreadyExists") { glog.Warningf("GCE PD %q already exists, reusing", name) return nil

Unit tests are needed for the GCE cloud provider disk interface.
Assigned to
/assign
GitHub didn't allow me to assign the following users: verult. Note that only can be assigned.. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details


Question: "description": "EndpointPort is a tuple that describes a single port.", "properties": { "name": { <del> "description": "The name of this port (corresponds to ServicePort.Name). Must be a DNS_LABEL. Optional only if one port is defined.", </del> <ins> "description": "The name of this port.  This must match the 'name' field in the corresponding ServicePort. Must be a DNS_LABEL. Optional only if one port is defined.", </ins> "type": "string" }, "port": {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When running a cluster service and manually created endpoints, the port name set on the cluster definition acts as an implicit selector - suitable end-points do not get mapped to the cluster service. What you expected to happen: Port name shouldn't be an implicit selector. How to reproduce it (as minimally and precisely as possible): Using the following spec file "": kubectl create -f , then describe the service. Output shows the endpoint as being correctly mapped Now run kubectl delete -f the and uncomment the name: foobar Re-run the kubectl create -f command. Now, checking the service gives the output below (end point isn't mapped): Anything else we need to know?: Adding the name: foobar to the endpoint definition makes the end point to be mapped again into the cluster. I.e. there is an implicit matching of the end-point and cluster port definitions. This is either a bug or a very surprising and undocumented feature Environment: kubectl version Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-03-25T02:52:13Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-01-16T18:14:49Z", GoVersion:"go1.10.7", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): root cat /etc/os-release NAME="Ubuntu" VERSION="18.04.1 LTS (Bionic Beaver)" Kernel (e.g.


Question: "description": "EndpointPort is a tuple that describes a single port.", "properties": { "name": { <del> "description": "The name of this port (corresponds to ServicePort.Name). Must be a DNS_LABEL. Optional only if one port is defined.", </del> <ins> "description": "The name of this port.  This must match the 'name' field in the corresponding ServicePort. Must be a DNS_LABEL. Optional only if one port is defined.", </ins> "type": "string" }, "port": {

): uname -a Linux 4.15.0-51-generic -Ubuntu SMP Wed May 15 14:27:21 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/remove-sig api-machinery /sig network The name mapping to the endpoints is intentional, but needs documentation
/triage unresolved
/assign
It may be under-documented, but it is not UNdocmented.
Also
working on a PR to fix doc


Question: CacheCapacity:  10, Storage:        s, Versioner:      etcdstorage.APIObjectVersioner{}, <del> ListFromCache:  true, </del> Type:           &api.Pod{}, ResourcePrefix: prefix, KeyFunc:        func(obj runtime.Object) (string, error) { return storage.NamespaceKeyFunc(prefix, obj) },

The label(s) cannot be applied. These labels are supported: <detailsIn response to : This one is trying to use with a generator to create an rc. This was deprecated and now is removed, so the test is failing now. /label sig/undefined Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/label failing-test
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label failing-test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test
At first I thought this was because generators were removed from . That still might be the cause of this, but I'm not 100% sure off-hand.
/sig cli
/assign
Funny thing about this is if you look in the build logs it is not immediately obvious that it is caused by generators being removed from because actually has sort of an unexpected behavior, where it tells you it is deprecated and gives you a pod instead... For example: Output:
/reopen should fix this is one of the affected jobs
Reopened this issue. <detailsIn response to : /reopen should fix this Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Since it's been a few weeks, I wanted to provide an update on this. I'm still waiting to confirm this is resolved by It looks like it is still running the old tests not marked as . Since 1.17.9 was released 14 days ago, and k8s-ci-robot only merged the PR 13 days ago, I'm guessing we need to wait until there is a 1.17.10 to see it skip the deprecated tests.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity.


Question: CacheCapacity:  10, Storage:        s, Versioner:      etcdstorage.APIObjectVersioner{}, <del> ListFromCache:  true, </del> Type:           &api.Pod{}, ResourcePrefix: prefix, KeyFunc:        func(obj runtime.Object) (string, error) { return storage.NamespaceKeyFunc(prefix, obj) },

Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: # install.  See https://github.com/saltstack/salt-bootstrap/issues/270 # # -M installs the master <del> # FIXME: The following line should be replaced with: # curl -L http://bootstrap.saltstack.com | sh -s -- -M # when the merged salt-api service is included in the fedora salt-master rpm # Merge is here: https://github.com/saltstack/salt/pull/13554 # Fedora git repository is here: http://pkgs.fedoraproject.org/cgit/salt.git/ # (a new service file needs to be added for salt-api) curl -sS -L https://raw.githubusercontent.com/saltstack/salt-bootstrap/v2014.06.30/bootstrap-salt.sh | sh -s -- -M </del> <ins> curl -sS -L --connect-timeout 20 --retry 6 --retry-delay 10 https://bootstrap.saltstack.com | sh -s -- -M </ins> fi # Build release

Salt was failing to install on the master vagrant instance. The change noted by the FIXME solved it.
It's possible the required merge in the upstream repo occurred, will take a look.
I just attempted to install a new cluster from upstream/master and was not able to reproduce the issue described. Was there an issue with curl-ing that specific version of salt-bootstrap? I guess more details on what the failure you encountered appeared as would help know how to proceed.
I will submit a PR to revert back to using the default non-versioned bootstrap URL. Salt-stack PR: Fixed the errors in install that were previously encountered around salt-api not being found.
See , thanks!


Question: } } <del> if plugins.PreFilter != nil { for _, pf := range plugins.PreFilter.Enabled { if pg, ok := pluginsMap[pf.Name]; ok { p, ok := pg.(PreFilterPlugin) if !ok { return nil, fmt.Errorf("plugin %q does not extend prefilter plugin", pf.Name) } f.preFilterPlugins = append(f.preFilterPlugins, p) } else { return nil, fmt.Errorf("prefilter plugin %q does not exist", pf.Name) } } </del> <ins> if err := updatePluginList(reflect.ValueOf(&f.preFilterPlugins), plugins.PreFilter, reflect.TypeOf((*PreFilterPlugin)(nil)), pluginsMap); err != nil { return nil, err </ins> } <del> if plugins.Filter != nil { for _, r := range plugins.Filter.Enabled { if pg, ok := pluginsMap[r.Name]; ok { p, ok := pg.(FilterPlugin) if !ok { return nil, fmt.Errorf("plugin %q does not extend filter plugin", r.Name) } f.filterPlugins = append(f.filterPlugins, p) } else { return nil, fmt.Errorf("filter plugin %q does not exist", r.Name) } } </del> <ins> if err := updatePluginList(reflect.ValueOf(&f.filterPlugins), plugins.Filter, reflect.TypeOf((*FilterPlugin)(nil)), pluginsMap); err != nil { return nil, err </ins> } <del> if plugins.Score != nil { for _, sc := range plugins.Score.Enabled { if pg, ok := pluginsMap[sc.Name]; ok { p, ok := pg.(ScorePlugin) if !ok { return nil, fmt.Errorf("plugin %q does not extend score plugin", sc.Name) } if f.pluginNameToWeightMap[p.Name()] == 0 { return nil, fmt.Errorf("score plugin %q is not configured with weight", p.Name()) } f.scorePlugins = append(f.scorePlugins, p) } else { return nil, fmt.Errorf("score plugin %q does not exist", sc.Name) } } </del> <ins> if err := updatePluginList(reflect.ValueOf(&f.reservePlugins), plugins.Reserve, reflect.TypeOf((*ReservePlugin)(nil)), pluginsMap); err != nil { return nil, err </ins> } <del> if plugins.Reserve != nil { for _, r := range plugins.Reserve.Enabled { if pg, ok := pluginsMap[r.Name]; ok { p, ok := pg.(ReservePlugin) if !ok { return nil, fmt.Errorf("plugin %q does not extend reserve plugin", r.Name) } f.reservePlugins = append(f.reservePlugins, p) } else { return nil, fmt.Errorf("reserve plugin %q does not exist", r.Name) } } </del> <ins> if err := updatePluginList(reflect.ValueOf(&f.postFilterPlugins), plugins.PostFilter, reflect.TypeOf((*PostFilterPlugin)(nil)), pluginsMap); err != nil { return nil, err </ins> } <del> if plugins.PostFilter != nil { for _, r := range plugins.PostFilter.Enabled { if pg, ok := pluginsMap[r.Name]; ok { p, ok := pg.(PostFilterPlugin) if !ok { return nil, fmt.Errorf("plugin %q does not extend post-filter plugin", r.Name) } f.postFilterPlugins = append(f.postFilterPlugins, p) } else { return nil, fmt.Errorf("post-filter plugin %q does not exist", r.Name) } } </del> <ins> if err := updatePluginList(reflect.ValueOf(&f.scorePlugins), plugins.Score, reflect.TypeOf((*ScorePlugin)(nil)), pluginsMap); err != nil { return nil, err </ins> } <del> if plugins.PreBind != nil { for _, pb := range plugins.PreBind.Enabled { if pg, ok := pluginsMap[pb.Name]; ok { p, ok := pg.(PreBindPlugin) if !ok { return nil, fmt.Errorf("plugin %q does not extend prebind plugin", pb.Name) } f.preBindPlugins = append(f.preBindPlugins, p) } else { return nil, fmt.Errorf("prebind plugin %q does not exist", pb.Name) } } </del> <ins> if err := updatePluginList(reflect.ValueOf(&f.preBindPlugins), plugins.PreBind, reflect.TypeOf((*PreBindPlugin)(nil)), pluginsMap); err != nil { return nil, err </ins> } <del> if plugins.Bind != nil { for _, pb := range plugins.Bind.Enabled { if pg, ok := pluginsMap[pb.Name]; ok { p, ok := pg.(BindPlugin) if !ok { return nil, fmt.Errorf("plugin %q does not extend bind plugin", pb.Name) } f.bindPlugins = append(f.bindPlugins, p) } else { return nil, fmt.Errorf("bind plugin %q does not exist", pb.Name) } } </del> <ins> if err := updatePluginList(reflect.ValueOf(&f.bindPlugins), plugins.Bind, reflect.TypeOf((*BindPlugin)(nil)), pluginsMap); err != nil { return nil, err </ins> } <del> if plugins.PostBind != nil { for _, pb := range plugins.PostBind.Enabled { if pg, ok := pluginsMap[pb.Name]; ok { p, ok := pg.(PostBindPlugin) if !ok { return nil, fmt.Errorf("plugin %q does not extend postbind plugin", pb.Name) } f.postBindPlugins = append(f.postBindPlugins, p) } else { return nil, fmt.Errorf("postbind plugin %q does not exist", pb.Name) } } </del> <ins> if err := updatePluginList(reflect.ValueOf(&f.postBindPlugins), plugins.PostBind, reflect.TypeOf((*PostBindPlugin)(nil)), pluginsMap); err != nil { return nil, err </ins> } <del> if plugins.Unreserve != nil { for _, ur := range plugins.Unreserve.Enabled { if pg, ok := pluginsMap[ur.Name]; ok { p, ok := pg.(UnreservePlugin) if !ok { return nil, fmt.Errorf("plugin %q does not extend unreserve plugin", ur.Name) } f.unreservePlugins = append(f.unreservePlugins, p) } else { return nil, fmt.Errorf("unreserve plugin %q does not exist", ur.Name) } } </del> <ins> if err := updatePluginList(reflect.ValueOf(&f.unreservePlugins), plugins.Unreserve, reflect.TypeOf((*UnreservePlugin)(nil)), pluginsMap); err != nil { return nil, err </ins> } <del> if plugins.Permit != nil { for _, pr := range plugins.Permit.Enabled { if pg, ok := pluginsMap[pr.Name]; ok { p, ok := pg.(PermitPlugin) if !ok { return nil, fmt.Errorf("plugin %q does not extend permit plugin", pr.Name) } f.permitPlugins = append(f.permitPlugins, p) } else { return nil, fmt.Errorf("permit plugin %q does not exist", pr.Name) } } </del> <ins> if err := updatePluginList(reflect.ValueOf(&f.permitPlugins), plugins.Permit, reflect.TypeOf((*PermitPlugin)(nil)), pluginsMap); err != nil { return nil, err </ins> } <del> if plugins.QueueSort != nil { for _, qs := range plugins.QueueSort.Enabled { if pg, ok := pluginsMap[qs.Name]; ok { p, ok := pg.(QueueSortPlugin) if !ok { return nil, fmt.Errorf("plugin %q does not extend queue sort plugin", qs.Name) } f.queueSortPlugins = append(f.queueSortPlugins, p) if len(f.queueSortPlugins) > 1 { return nil, fmt.Errorf("only one queue sort plugin can be enabled") } } else { return nil, fmt.Errorf("queue sort plugin %q does not exist", qs.Name) } </del> <ins> if err := updatePluginList(reflect.ValueOf(&f.queueSortPlugins), plugins.QueueSort, reflect.TypeOf((*QueueSortPlugin)(nil)), pluginsMap); err != nil { return nil, err } for _, scorePlugin := range f.scorePlugins { if f.pluginNameToWeightMap[scorePlugin.Name()] == 0 { return nil, fmt.Errorf("score plugin %q is not configured with weight", scorePlugin.Name()) </ins> } } <ins> if len(f.queueSortPlugins) > 1 { return nil, fmt.Errorf("only one queue sort plugin can be enabled") } </ins> return f, nil } <ins> func updatePluginList(pluginList reflect.Value, pluginSet *config.PluginSet, pluginType reflect.Type, pluginsMap map[string]Plugin) error { if pluginSet == nil { return nil } plugins := pluginList.Elem() pluginType = pluginType.Elem() set := sets.NewString() for _, ep := range pluginSet.Enabled { pg, ok := pluginsMap[ep.Name] if !ok { return fmt.Errorf("%s %q does not exist", pluginType.String(), ep.Name) } if !reflect.TypeOf(pg).Implements(pluginType) { return fmt.Errorf("plugin %q does not extend %s plugin", ep.Name, pluginType.String()) } if set.Has(ep.Name) { return fmt.Errorf("plugin %q already registered as %q", ep.Name, pluginType.String()) } set.Insert(ep.Name) newPlugins := reflect.Append(plugins, reflect.ValueOf(pg)) plugins.Set(newPlugins) } return nil } </ins> // QueueSortFunc returns the function to sort pods in scheduling queue func (f *framework) QueueSortFunc() LessFunc { if len(f.queueSortPlugins) == 0 {

There is no reason this should be allowed to happen, especially that we can't configure duplicate plugins differently since plugin configs is a separate list . /sig scheduling
/label help-wanted
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help-wanted
/help /good-first-issue
/assign
/remove-good-first-issue /remove-help


Question: cfg.APIServer.ExtraArgs = []kubeadmapi.Arg{} } authzVal, _ := kubeadmapi.GetArgValue(cfg.APIServer.ExtraArgs, "authorization-mode", -1) <del> defaultArguments = kubeadmapi.SetArgValues(defaultArguments, "authorization-mode", getAuthzModes(authzVal), 1) </del> <ins> _, hasStructuredAuthzVal := kubeadmapi.GetArgValue(cfg.APIServer.ExtraArgs, "authorization-config", -1) if hasStructuredAuthzVal == -1 { defaultArguments = kubeadmapi.SetArgValues(defaultArguments, "authorization-mode", getAuthzModes(authzVal), 1) } </ins> command = append(command, kubeadmutil.ArgumentsToCommand(defaultArguments, cfg.APIServer.ExtraArgs)...) return command

TL;DR prevents us from using structured authz (setting flag) as those flags are mutually exclusive /cc (as the author of ) /cc FYI To use structured authz you need to define the authorization configuration using the flag. However this flag is mutually exclusive with and flags. (refer to & ) Kubeadam adds by default ( & ) which prevents us from using I should be able to define the flag and have structured authz working. Spin up a kind cluster using this kind config (make sure the images are 1.29 for structured authz to be avaliable) (For this bug, the content of doesn't matter.) However, the cluster become unhealthy. docker exec to the kind cluster container, where we can see the inner container of the apiserver has exited. Running we can see this error <img width="1720" alt="image" src=""No response_ <details</details<detailsN/A -I am using kind </details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
FYI, in case someone opens similar issue in Kind. I might be wrong but I think this effectively means that kind users cant use structured authz (unless it supports a mode to spin up a cluster without kubeadam).


Question: // Adds a service to return the supported resources, E.g., a such web service // will be registered at /apis/extensions/v1. func AddSupportedResourcesWebService(s runtime.NegotiatedSerializer, ws *restful.WebService, groupVersion unversioned.GroupVersion, apiResources []unversioned.APIResource) { <del> resourceHandler := SupportedResourcesHandler(s, groupVersion, apiResources) </del> <ins> ss := s if keepUnversioned(groupVersion.Group) { // Because in release 1.1, /apis/extensions/v1beta1 returns response // with empty APIVersion, we use StripVersionNegotiatedSerializer to // keep the response backwards compatible. ss = StripVersionNegotiatedSerializer{s} } resourceHandler := SupportedResourcesHandler(ss, groupVersion, apiResources) </ins> ws.Route(ws.GET("/").To(resourceHandler). Doc("get available resources"). Operation("getAPIResources").

Kubectl client Kubectl api-versions should check if v1 is in available api versions [Conformance] Expected error: <_errors.errorString | 0xc208641770: { s: "Error running &{/jenkins-master-data/jobs/kubernetes-upgrade-gke-1.1-master-step1-deploy/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://130.211.136.232 --kubeconfig=/var/lib/jenkins/jobs/kubernetes-upgrade-gke-1.1-master-step1- api-versions] [] error: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1"n [] 0xc208382fa0 exit status 1 true [0xc208627148 0xc208627168 0xc208627188] [0xc208627148 0xc208627168 0xc208627188] [0xc208627160 0xc208627180] [0x6d81a0 0x6d81a0] 0xc2082fb2c0}:nCommand stdout:nnstderr:nerror: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1"nn", } Error running &{/jenkins-master-data/jobs/kubernetes-upgrade-gke-1.1-master-step1-deploy/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://130.211.136.232 --kubeconfig=/var/lib/jenkins/jobs/kubernetes-upgrade-gke-1.


Question: // Adds a service to return the supported resources, E.g., a such web service // will be registered at /apis/extensions/v1. func AddSupportedResourcesWebService(s runtime.NegotiatedSerializer, ws *restful.WebService, groupVersion unversioned.GroupVersion, apiResources []unversioned.APIResource) { <del> resourceHandler := SupportedResourcesHandler(s, groupVersion, apiResources) </del> <ins> ss := s if keepUnversioned(groupVersion.Group) { // Because in release 1.1, /apis/extensions/v1beta1 returns response // with empty APIVersion, we use StripVersionNegotiatedSerializer to // keep the response backwards compatible. ss = StripVersionNegotiatedSerializer{s} } resourceHandler := SupportedResourcesHandler(ss, groupVersion, apiResources) </ins> ws.Route(ws.GET("/").To(resourceHandler). Doc("get available resources"). Operation("getAPIResources").

1-master-step1- api-versions] [] error: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1" [] 0xc208382fa0 exit status 1 true [0xc208627148 0xc208627168 0xc208627188] [0xc208627148 0xc208627168 0xc208627188] [0xc208627160 0xc208627180] [0x6d81a0 0x6d81a0] 0xc2082fb2c0}: Command stdout: stderr: error: Couldn't get available api versions from server: no kind "APIVersions" is registered for version "v1" not to have occurred* See and Investigating. cc
Okiedokie. Looks like the v1.1 client is getting a response from the v1.2 master, but the v1.1 client doesn't know how to handle the v1 APIVersions object, because in v1.1, that object was never expected to have an . From a 1.1 server, we get which doesn't include a . We should expect to have plenty of pre-v1.1.7 clients floating around, and releasing a fixed v1.1.8 probably isn't going to solve this; if we want to be backward compatible, we need to not return an string with unversioned kinds like . Looking into more details about how we got here... cc cc
I have a maybe feeling that this is caused by ? In particular, we went from: to
Probably it's caused by this:
Let me double check the code. cc
What about that line? I've never dealt with the version negotiation; should the server and client be negotiating in a way that the server can return an object without ?
More precisely it's this line: This line sets the version to "v1". In , during encoding, we set unversioned object's APIVersion field to "v1".
I think in the long run, because we expect to move many unversioned types to group "common", so they should be versioned, so making APIVersion=v1 is the right move. Back to the backwards compatibility issue, how's the situation at other url, like http://127.0.0.1:8080/apis/extensions?


Question: // Adds a service to return the supported resources, E.g., a such web service // will be registered at /apis/extensions/v1. func AddSupportedResourcesWebService(s runtime.NegotiatedSerializer, ws *restful.WebService, groupVersion unversioned.GroupVersion, apiResources []unversioned.APIResource) { <del> resourceHandler := SupportedResourcesHandler(s, groupVersion, apiResources) </del> <ins> ss := s if keepUnversioned(groupVersion.Group) { // Because in release 1.1, /apis/extensions/v1beta1 returns response // with empty APIVersion, we use StripVersionNegotiatedSerializer to // keep the response backwards compatible. ss = StripVersionNegotiatedSerializer{s} } resourceHandler := SupportedResourcesHandler(ss, groupVersion, apiResources) </ins> ws.Route(ws.GET("/").To(resourceHandler). Doc("get available resources"). Operation("getAPIResources").

That address also returns a versioned object, which at release 1.1 is unversioned: Do we need to keep all unversioned objects unversioned, or do we only need to ensure the server returns unversioned object at api/?
cc
Per our versioning rules, we should expect a v1.2 server to remain backward compatible with v1.1 clients. I don't have much experience with the api machinery, but if clients weren't expecting an unversioned object, it seems like we can't up and switch to versioned objects.
I think we have to maintain compatibility somehow. any ideas for a fix other than the obvious one?
Yes, we have to maintain backward compatibility with 1.1 (and 1.0) clients. New paths that old clients didn't know about could behave differently than paths that existed in previous releases.
Nope, let's make sure we leave a test and comment.
We can't make the change here though, this would break "versioned unversioned" objects (like )
The other implication of course is now we "versioned unversioned" objects (v1 status) and "unversioned unversioned", which means we shouldn't ever add new objects that don't return a version. I don't think anything in 1.2 qualifies as a new object without a version, just 1.1 objects that didn't have correct versions. Having objects from extensions group be returned without versions terrifies me - I don't see any way of describing that except as a bug.
We may need per API group version rules registered on scheme about whether "unversioned" objects are returned in their scheme version or as empty. That's probably the backward compatible fix that gives us a hope of reasoning about behavior correctly (status in 1.0 is v1, status in extensions might be "", apiversions in any new group is "v1"). The unversioned marker on the objects should ensure 1.2 clients can receive objects in both "" and "v1" form, we should add a test to verify that. I'm against letting empty API versions spread, but agree backcompat for existing groups versions should match previous behavior.
Can we make (or use) a different serializer/codec just for the endpoints that 1.1 clients used?


Question: // Adds a service to return the supported resources, E.g., a such web service // will be registered at /apis/extensions/v1. func AddSupportedResourcesWebService(s runtime.NegotiatedSerializer, ws *restful.WebService, groupVersion unversioned.GroupVersion, apiResources []unversioned.APIResource) { <del> resourceHandler := SupportedResourcesHandler(s, groupVersion, apiResources) </del> <ins> ss := s if keepUnversioned(groupVersion.Group) { // Because in release 1.1, /apis/extensions/v1beta1 returns response // with empty APIVersion, we use StripVersionNegotiatedSerializer to // keep the response backwards compatible. ss = StripVersionNegotiatedSerializer{s} } resourceHandler := SupportedResourcesHandler(ss, groupVersion, apiResources) </ins> ws.Route(ws.GET("/").To(resourceHandler). Doc("get available resources"). Operation("getAPIResources").

Probably we just need to change the version registered here:
That won't work like you expect - the type registry then has no knowledge in v1 of what schema the object is. That's not what we want - we want to not serialize the version of the object for specific API groups, which is a much more targeted change. This is a special, exceptional case (v1 group should return v1 Status. v1beta1 should not). , Chao Xu wrote:
Changing v1 breaks 1.0 backwards compat (we always returned apiVersion=v1 for 1.0) , Clayton Coleman wrote:
I mean this: Will this work?
I'm not sure without looking at the outcome (which is why unversioned is a bit of a disaster in our codebase). I would much prefer to explicitly identify which objects should have an unset apiversion via a method on the scheme, and do the explicit check at the time we do convert to version. That's an extremely controlled change, is obvious, and doesn't make the unversioned problem worse.
I remember somewhere in asserts version cannot be empty, so I guess my proposed solution won't work without extra work. I'll take the approach suggested by


Question: ${KUBE_ROOT}/hack/update-swagger-spec.sh git commit -am "Versioning docs and examples for ${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}" <ins> dochash=$(git log -n1 --format=%H) </ins> VERSION_FILE="${KUBE_ROOT}/pkg/version/base.go" GIT_MINOR="${VERSION_MINOR}.${VERSION_PATCH}"

doesn't work with our existing release flow. I thought it did, but, from cc closing out .
Actually, I just realized my last suggestion in isn't the worst. The pull requests to master are essentially only there to get the tags backmerged, so we can construct a release branch like so: and then a release branch backmerge branch like so: and it'll still work.