Question: serviceConfig.RegisterEventHandler(s.Proxier) go serviceConfig.Run(wait.NeverStop) <del> if utilfeature.DefaultFeatureGate.Enabled(features.EndpointSlice) { </del> <ins> if s.UseEndpointSlices { </ins> endpointSliceConfig := config.NewEndpointSliceConfig(informerFactory.Discovery().V1beta1().EndpointSlices(), s.ConfigSyncPeriod) endpointSliceConfig.RegisterEventHandler(s.Proxier) go endpointSliceConfig.Run(wait.NeverStop)

After this () has been merged and EnpointSlice feature is now enabled by default, we noticed Kube-proxy will not work on Windows. Only message I can see in the logs is: /sig windows
/cc /cc /cc
How about separating the above check into two if statements so that the log can pinpoint which field is false ?
It seems long term, EndpointSlice support should be so that is kept in sync with other implementations.
cc
Thanks for reporting this! Windows proxiers were still intended to be reading from Endpoints, but apparently that's not working as intended. EndpointSlices are moving back to disabled by default for 1.17, so that should fix the failing tests for now. I'll work on an actual fix for this as well, but it may not make it in until after KubeCon. /assign /sig network /priority important-soon
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
/assign cc
GitHub didn't allow me to assign the following users: jocelynberrendonner. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
: Please let me know if you need any help on the fix! I'll be more than happy to review/help test the fix on Windows.


Question: annotations: storageclass.beta.kubernetes.io/is-default-class: "true" labels: <del> addonmanager.kubernetes.io/mode: Reconcile </del> <ins> addonmanager.kubernetes.io/mode: EnsureExists </ins> provisioner: kubernetes.io/host-path

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Not able to change the default storage class. What you expected to happen: Default storage class to change. How to reproduce it (as minimally and precisely as possible): But after few mins. Anything else we need to know?: Tried to remove the annotation but no luck. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: minikube OS (e.g. from /etc/os-release): Buildroot 2018.05 Kernel (e.g. ): 4.16.14 Install tools: minikube Others:
/assign
Those labels are not set on the issue: <detailsIn response to : /remove-sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
The cause is that kube-addon-manager is reconciling the standard storage class.
Those labels are not set on the issue: <detailsIn response to : /remove-sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for looking into it. Sorry for native question, I want to keep only ceph as a default storage class how can I do that?
hi I'm not really sure about the actual command to fix this. Basically, you could log into the VM and do the same modification as the PR above does, then restart minikube.
Has the fix made it into minikube yet? I'm still seeing the issue in 0.31.0
Looking at PR , the fix was made in kubernetes core, not minikube. The error still exists in minikube:


Question: get(t, "valid TLS cert again", false) } <ins> func TestConcurrentUpdateTransportConfig(t *testing.T) { n := time.Now() now := func() time.Time { return n } env := []string{""} environ := func() []string { s := make([]string, len(env)) copy(s, env) return s } c := api.ExecConfig{ Command:    "./testdata/test-plugin.sh", APIVersion: "client.authentication.k8s.io/v1alpha1", } a, err := newAuthenticator(newCache(), &c) if err != nil { t.Fatal(err) } a.environ = environ a.now = now a.stderr = ioutil.Discard stopCh := make(chan struct{}) defer close(stopCh) numConcurrent := 2 for i := 0; i < numConcurrent; i++ { go func() { for { tc := &transport.Config{} a.UpdateTransportConfig(tc) select { case <-stopCh: return default: continue } } }() } time.Sleep(2 * time.Second) } </ins> // genClientCert generates an x509 certificate for testing. Certificate and key // are returned in PEM encoding. func genClientCert(t *testing.T) ([]byte, []byte) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Data race in client-go with below sample code: Data race: What you expected to happen: No data race How to reproduce it (as minimally and precisely as possible): Using above given sample code Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ): OSX
/sig api-machinery
I'd like to take it. /assign
/remove-sig api-machinery /sig auth cli
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then?
Another reminder here that code freeze is in less than two weeks (Nov. 14). We will need to move this issue out of the milestone if isn't going to be resolved before then.


Question: <del> apiVersion: extensions/v1beta1 </del> <ins> apiVersion: apps/v1 </ins> kind: DaemonSet metadata: name: nvidia-gpu-device-plugin

This issue is here to satisfy PR requirements of issue association. A lot of times we need to do version bumps for minor bug fixes that don't deserve a separate issue in the kubernetes/kubernetes repo. /kind bug /sig node
/lifecycle frozen
all the linked PRs are emrged and no activity since more than 3 years /close
Closing this issue. <detailsIn response to : all the linked PRs are emrged and no activity since more than 3 years /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: fmt.Fprint(out, "n") } <ins> func printMissingMetricsNodeLine(out io.Writer, nodeName string) { printValue(out, nodeName) unknownMetricsStatus := "<unknown>" for i := 0; i < len(MeasuredResources); i++ { printValue(out, unknownMetricsStatus) printValue(out, "t") printValue(out, unknownMetricsStatus) printValue(out, "t") } fmt.Fprint(out, "n") } </ins> func printValue(out io.Writer, value interface{}) { fmt.Fprintf(out, "%vt", value) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: kubectl top nodes didn't report one of my nodes. I have a cluster of 8 and only 7 printed.


Question: fmt.Fprint(out, "n") } <ins> func printMissingMetricsNodeLine(out io.Writer, nodeName string) { printValue(out, nodeName) unknownMetricsStatus := "<unknown>" for i := 0; i < len(MeasuredResources); i++ { printValue(out, unknownMetricsStatus) printValue(out, "t") printValue(out, unknownMetricsStatus) printValue(out, "t") } fmt.Fprint(out, "n") } </ins> func printValue(out io.Writer, value interface{}) { fmt.Fprintf(out, "%vt", value) }

<pre:::: fang 10:04:02 (cgthayer-pchat) 0 platform; kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-charles-test-cluster-default-pool--jkrj 311m 33% 2470Mi 93% gke-charles-test-cluster-default-pool--krsh 154m 16% 1763Mi 66% gke-charles-test-cluster-default-pool--vhgq 265m 28% 1954Mi 73% gke-charles-test-cluster-default-pool--x3bs 186m 19% 1990Mi 75% gke-charles-test-cluster-default-pool--073f 270m 28% 1729Mi 65% gke-charles-test-cluster-default-pool--09zf 403m 42% 2698Mi 102% gke-charles-test-cluster-default-pool--p7h1 191m 20% 1582Mi 59% :::: fang 10:04:07 (cgthayer-pchat) 0 platform; kubectl get nodes NAME STATUS ROLES AGE VERSION gke-charles-test-cluster-default-pool--073f Ready <none7d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--09zf Ready <none7d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--cl65 Ready <none6d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--jkrj Ready <none8d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--krsh Ready <none8d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--p7h1 Ready <none7d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--vhgq Ready <none8d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--x3bs Ready <none8d v1.7.12-gke.1 <preWhat you expected to happen: I expected all nodes to be printed.


Question: fmt.Fprint(out, "n") } <ins> func printMissingMetricsNodeLine(out io.Writer, nodeName string) { printValue(out, nodeName) unknownMetricsStatus := "<unknown>" for i := 0; i < len(MeasuredResources); i++ { printValue(out, unknownMetricsStatus) printValue(out, "t") printValue(out, unknownMetricsStatus) printValue(out, "t") } fmt.Fprint(out, "n") } </ins> func printValue(out io.Writer, value interface{}) { fmt.Fprintf(out, "%vt", value) }

Since the missing node might have bad data I might expect it to have the same number of columns with "-" values and a note in a new 6th column saying something like "unavailabe" "unresponsive" "stale-data" etc. e.g. <pregke-charles-test-cluster-default-pool--p7h1 - - - - unreachable </preHow to reproduce it (as minimally and precisely as possible): Not sure. Perhaps run a pod where you can intentionally trigger it to use too much cpu or memory. Anything else we need to know?: Environment: Kubernetes version (use ): kubectl version Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-12-21T06:34:11Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.8-gke.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-16T18:26:58Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind feature
I raise a PR to print when the metrics is missing. As for the 6th column you mentioned, I think it could be debatable for its necessity. :)


Question: import ( "context" <del> "fmt" </del> "math/rand" <del> "strings" </del> "time" v1 "k8s.io/api/core/v1"

The metric used to report acutal pod in the label , but now I see the podname prefix only. The full pod name in the label, so I can trace the metric to the failing pod. Check raw api and verify the actual pod name with uids. <details</detailsAzure AKS <details</detailsNo response No response No response_
/sig node
/triage accepted /priority important-soon /cc As discussed this is a change in functionaloty that wasn't highlighted in release notes. Need to decide on what do do with it. I understand it's a change of functionality. what is the side effect of this for you? Does the label without last part not informational enough?
I've been using the metric in my custom grafana dashboard that shows CPU, RAM, IO usage of pods with graph of increases in probes. If the apps are stable, the probe graph looks steadily. I can see if some irregularities in system metrics are influencing probes of respective pod. Also, if readiness probe is failing, the pod is removed from service, but is not restarted so I must have knowledge of pods with failing readiness probes. I attach the screenshot of my graph. ! So I'd welcome the possibility to pair the probes with exact pod and thank you for looking into it.
are we thinking that this resulted in an undesirable change? Spologies for being a bit late to reply, I'm returning from a brief paternity leave staycation.
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
Well this is awkward... I still believe the actual behavior does not respect the intended idea...
This issue makes it hard to link probe metrics with right pods. The value of pod label in and are also inconsistent with how it is used in other kubelet metrics, see: vs Definitely a bug, regression since 1.25.
I can't get anyone to look at the fix/revert: you may have better luck
/milestone v1.28
discussion in
You've pushed it through, congrats!


Question: // Ensure all of the pods that we found on this node before the reboot are // running / healthy. if !checkPodsRunningReady(c, ns, podNames, rebootPodReadyAgainTimeout) { <ins> newPods := ps.List() printStatusAndLogsForNotReadyPods(c, pods, newPods) </ins> return false }

We are running heapster, dns server, kube-proxy etc. in kube-system namespace. Some e2e tests try to make sure all those kube-system pods are at ready state before firing the real tests, others just simply ingore them. But both cases might cause e2e flakiness when some of those kube-system containers are in crashloop, for example, for heapster crashloop, for influxdb crashloop, etc. When such cases happen, the developer doesn't have much information on why since the nodes might be teared down. We should watch those kube-system pods, and onContainerFailure, we should collect the related logs. One simple way is using kubectl logs <Pod<Container--previous cc/
is this related enough to and that you can bundle it in?
The logs from the pods should be retrieved via the api server. This is different from the kubelet logs. I'll upload a PR for this.
Great job It will be very helpful for debugging.


Question: "reason":                   "FailedScheduling", }.AsSelector()) expectNoError(err) <ins> // If we failed to find event with a capitalized first letter of reason // try looking for one starting with a small one for backward compatibility. // If we don't do it we end up in #15806. // TODO: remove this block when we don't care about supporting v1.0 too much. if len(schedEvents.Items) == 0 { schedEvents, err = c.Events(ns).List( labels.Everything(), fields.Set{ "involvedObject.kind":      "Pod", "involvedObject.name":      podName, "involvedObject.namespace": ns, "source":                   "scheduler", "reason":                   "failedScheduling", }.AsSelector()) expectNoError(err) } </ins> printed := false printOnce := func(msg string) string {

See Jenkins job kubernetes-upgrade-1.0-1.1-gke-step3-e2e-old's last few runs. The relevant line is . Note that this is running e2e tests off of the branch. It's possible that this is also caused by , but it's not clear to me.
This is also failing on kubernetes-upgrade-1.0-1.1-gke-step5-e2e-old, where we're running tests against a cluster that's been fully upgraded, (master and nodes,) from 1.0 to 1.1.
Status update: this may be caused by , so that issue should be prioritized, but it's not totally clear if that will fix this problem.
said he would take a look and update the issue with any findings. It may be a bit premature to look into it before is resolved, but at least we can try to figure out how to manually repro this now, so that we will know whether fixes this issue.
I'm unable to find logs from failed runs, which makes everything slightly harder. I'll try to reproduce the problem on GCE cluster, but it may not work. FYI: most of was caused by waiting for skydns to come back up (which doesn't happen: ). I see that there's a difference in number of system pods between version 1.1 and 1.0 - there's only 5 of them in 1.1 and 8 of them in 1.0. I'm wondering if this may cause problems in 'old' version of test which we're running there. I'm going to try reproduce it on GCE (as this is the setup I understand better) and if I'll fail I'll try again on GKE.
AFAICT there's a problem with the test, not the upgrade itself. The test from 1.1 branch fails on 1.0 cluster.
It's caused by . Can we please stop renaming things? AFAICT the problem is with test only, everything should be working 'on production'. In v1.0 scheduler Events reasons start with a small letter (e.g. failedScheduling) and v1.0 test is looking for those kind of events to occur. In v1.1 we changed Event reasons to start with a capital letter (e.g.


Question: "reason":                   "FailedScheduling", }.AsSelector()) expectNoError(err) <ins> // If we failed to find event with a capitalized first letter of reason // try looking for one starting with a small one for backward compatibility. // If we don't do it we end up in #15806. // TODO: remove this block when we don't care about supporting v1.0 too much. if len(schedEvents.Items) == 0 { schedEvents, err = c.Events(ns).List( labels.Everything(), fields.Set{ "involvedObject.kind":      "Pod", "involvedObject.name":      podName, "involvedObject.namespace": ns, "source":                   "scheduler", "reason":                   "failedScheduling", }.AsSelector()) expectNoError(err) } </ins> printed := false printOnce := func(msg string) string {

FailedScheduling) and modified test accordingly. Effects are obvious. We can: revert patch it to v1.0 merge and patch a test fix which adds handling both versions () I'm for the last option, but whatever we do (except reverting) will break backward compatibility for everyone depending/counting/filtering/whatnot on scheduler Event reasons
thanks for your work on this. From on : Furthermore, in my (rookie) opinion, seems like it should be considered a bug fix rather than an API change. Thus I think makes good sense.
Thanks a lot for debugging this, and I agree we should go with your PR. I talked to Brian and he said we don't consider changes to values of fields in events to be breaking API changes. Unfortunately people do seem to depend on these in various places. One solution is what I proposed in so the compiler could prevent these incompatibilities (at least for components and tests that are compiled together).
+1 to creating a file containing constants for the event reasons that are currently in use.
I'll prepare a cherry pick.
Reopening until cherry pick is merged.
Cherry pick is .
The test mentioned in the first entry in this thread is still failing but not due to the error message related to this issue. Since the cherrypick has been merged, can we close this issue, and followup with additional debugging in ?
Works for me. Thanks to again for your work on this.
I actually still see failures of the same kind in (master 1.1, nodes 1.0) and (master 1.1, nodes 1.1). I assume this is because we haven't cherrypicked test fix into . So this is now a duplicate of .


Question: import ( "fmt" "net" <ins> "os" </ins> "strconv" "strings" "testing"

When a --resolv-conf (DNS resolver config) is passed to kubelet on a Windows node, the config content is used as part of sandbox initialization (through getHostDNSConfig()). Now with the recent changes (), it seems now to fail with an error like: E0315 10:22:16. 4764 ] "Error syncing pod, skipping" err="failed to "CreatePodSandbox" for "img-pull-(-b40a-459d-9033-)" with CreatePodSandboxError: "Failed to generate sandbox config for pod "img-pull-(-b40a-459d-9033-)": Unexpected resolver config value: "C:etckubernetescni". Expected "" or "Host"."" pod="img-puller-3842/img-pull-" podUID=-b40a-459d-9033- This is showing on all the runs of GCE testgrid (latest): Reading/using the passed resolv config on Windows Run a Windows pool with DNS resolv config being set as part of kubelet params. No response_ 27 alpha.3 1.27 beta.0 master <detailsGCE </details<details</details<details</details<details</details<details</details
/triage accepted /milestone v1.27 /sig node /sig network
/sig windows /cc
/priority criticial-urgent as this is likely a release blocking
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority criticial-urgent Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
Hmm, so you're setting the [1]. This PR introduced the capability to read the DNS config options from the Windows host registries while removing the capability to read from a file, introducing a regression. I've sent a PR addressing this issue: [1]
Is the file you are pointing to a UNIX-style or something else?
Was the file changing the behavior of the node?


Question: import ( "fmt" "net" <ins> "os" </ins> "strconv" "strings" "testing"

Yeah, what used on GCE/GKE, seems to be a Linux-like conf, eg: nameserver 10.128.0.1 search c.<project_name.internal google.internal ` We haven't tried to remove from kubelet to see the behavior.
This Issue/PR has been marked as release blocker. Please have the fix merged by eod today otherwise the feature has to be reverted. For the Issue/PR owner: please have a revert/feature disable PR open today in case the fix failed to get in. Thank you! See the -- Kubernetes 1.27 release manager
I opened up a revert PR in case this PR does not merge by code freeze.
Thanks for the resolution! both 2019 and 2022 grids seems to be back to green now!
Thanks for confirming! I opened to ensure gets updated too


Question: serviceHealthServer := healthcheck.NewServiceHealthServer(hostname, recorder) isIPv6 := ipt.IsIPv6() <ins> var incorrectAddresses []string nodePortAddresses, incorrectAddresses = utilproxy.FilterIncorrectCIDRVersion(nodePortAddresses, isIPv6) if len(incorrectAddresses) > 0 { klog.Warning("NodePortAddresses of wrong family; ", incorrectAddresses) } </ins> proxier := &Proxier{ portsMap:                 make(map[utilproxy.LocalPort]utilproxy.Closeable), serviceMap:               make(proxy.ServiceMap),

What happened: In a dual-stack cluster when specifying both ipv6 and ipv4 addresses in the entries are inserted in ipvs for the ipv6 nodeports but with an invalid ipv4 address; The example shows the ipv4 loopback address but any ipv4 address in will cause a faulty entry. What you expected to happen: No ipv4 entry shall be inserted for the ipv6 nodeport. How to reproduce it (as minimally and precisely as possible): Configure with both ipv4 and ipv6 addresses Bring up a dual-stack cluster with proxy-mode Create a service with ipFamily:IPv6 Do Anything else we need to know?: Proxy-mode=iptables seems to work. With no ipv4 entries are inserted for ipv6 nodeports. In it is recommended to use but doing so makes kube-proxy enter rules for address (in addition to "127.0.0.1"). While these entries cause no harm they can't be used and should not be there. These entries are created for both proxy-mode "iptables" and "ipvs". This should not be handled in this issue but in another (new) issue if needed. Example; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Kvm's on Ubuntu OS (e.g: ): xcluster Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): calico/cni:v3.12.0 Others: /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/area ipvs /area ipv6
/remove-triage unresolved /assign I take this myself if no one objects.
The ipv4 address is obviously a ipv4-converted ipv6 from .
According to the code here (), we may need to filter the before passing it to .
Looks that that's it. Please go ahead and make a PR, or I can do it if you prefer. I will help test the PR of course. Here are some review comments right away; Use v1.IPFamily as param to the filter function rather than The filter function seem usable and generic and may be placed in some "util" package, but leave it as-is and let the maintainers decide that.
Hi feel free to send a PR please, as you have took this already :) I'd like to help review it!
The "addresses" are really CIDRs which make the fix even simpler.


Question: go_library( name = "go_default_library", srcs = [ <ins> "bindata.go", </ins> "main.go", <del> ":bindata", ], ) genrule( name = "bindata", srcs = [ "//examples:sources", "//test/images:sources", "//test/fixtures:sources", "//test/e2e/testing-manifests:sources", ], outs = ["bindata.go"], cmd = """ $(location //vendor:github.com/jteeuwen/go-bindata/go-bindata)  -nometadata -o "$(OUTS)" -pkg generated  -prefix $$(pwd)  -ignore .jpg -ignore .png -ignore .md  $(SRCS) """, tools = [ "//vendor:github.com/jteeuwen/go-bindata/go-bindata", </del> ], <ins> tags = ["automanaged"], </ins> )

Cluster logging assumes kubelet stores container log at a specific directory with specific name. Currently the path is: After the CRI work, the path will be: However, we don't have test coverage for this. We don't even know whether the log files are there or not. We should add a node e2e test for this, basically: 1) Create a container which prints logs 2) Check whether the expected log file is created. 3) Check whether the logs are collected in the file as expected. And it would be better to launch a pod to do the check instead of accessing local file system directly, which makes containerizing easier. /cc because you are collecting missing test. /cc
I will try to add a node-e2e test there
I think it's ok to add this for CRI. But there maybe problems on today's interface for rkt. If I'm not wrong, rkt does't store logs at same location with docker. cc/ Is this true?
Thanks! :)
After gets merged, we need to update the test to verify the cri log path.


Question: ctx, cancel := context.WithCancel(context.Background()) defer cancel() // Start to follow the container's log. <ins> fileName := file.Name() </ins> go func(ctx context.Context) { podLogOptions := v1.PodLogOptions{ Follow: true, } opts := NewLogOptions(&podLogOptions, time.Now()) <del> ReadLogs(ctx, file.Name(), containerID, opts, fakeRuntimeService, stdoutBuf, stderrBuf) </del> <ins> _ = ReadLogs(ctx, fileName, containerID, opts, fakeRuntimeService, stdoutBuf, stderrBuf) </ins> }(ctx) // log in stdout

Runs k8s with master golang: test Failure was observed on Aug 25th, it is passing after that. However it can be reproduced with stress tool. No response No response It is failing on both ppc and x86 architectures. Steps to reproduce: up golang tip version. clone kubernetes master /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
it is a sig node test, sig testing charter is not about doing or maintain tests, we need to remove that old mindset of having a QA department, that is not how software development works or should work :) Besides, thanks for spotting it and submittn a patch /sig node /remove-sig testing
Flaky in
merging this PR will fix the issue


Question: return nc.evictionLimiterQPS } <del> // ReducedQPSFunc returns the QPS for when a the cluster is large make </del> <ins> // ReducedQPSFunc returns the QPS for when the cluster is large make </ins> // evictions slower, if they're small stop evictions altogether. func (nc *Controller) ReducedQPSFunc(nodeNum int) float32 { if int32(nodeNum) > nc.largeClusterThreshold {

The arg large-cluster-size-threshold for kube-controller-manager works as zone size threshold when determining node eviction rate. If there are multiple zones, the description for arg stating "cluster" threshold would be missleading. ARG: Matched arg description Multiple zones cluster No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig docs


Question: name = "go_default_test", srcs = ["healthz_test.go"], embed = [":go_default_library"], <ins> deps = [ "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", ], </ins> ) go_library(

What would you like to be : Add support to the kube-apiserver so that we can disable health-checks via query params. Why is this needed: Currently, the healthz endpoint is inflexible in that any and all health checks are currently executed when the endpoint is hit. Since the kube-apiserver only currently supports a liveness check, this means all persistent health check failures will eventually result in a kubelet driven restart. However, this may not be desirable behavior if you have additional monitoring services layered on top of your control plane. For instance, take the etcd failure from the perspective of the kube-apiserver. Currently, persistent failures during the etcd health check from the api-server healthz endpoint will cause the kube-apiserver to be restarted. This sometimes helps (in the case of a faulty etcd-client connection) but also can be detrimental (the kube-apiserver restarts until the etcd cluster is functioning, then the kube-apiserver bombards the etcd cluster with requests to update local caches, saturating the etcd cluster with requests, causing it to timeout on its own liveness probes and trigger a kubelet-driven restart). Let's say that we could now disable etcd from our liveness health check. Then if we had an external monitoring service, we could ask independently ask etcd if it was healthy and we could also ask the kube-apiserver if it thought etcd was unhealthy. Based on the combination of those answers, we could enable more intelligent responses to certain situation, like restarting the kube-apiserver when it thought etcd was unhealthy but etcd was actually reporting as healthy but not doing so when etcd was actually unhealthy. This feature is backwards-compatible and opt-in. There would be no impact to existing behavior unless these query params are actually used. /sig api-machinery /kind feature
/cc
SGTM
/assign


Question: }, } <del> var betaTest *testsuites.StorageClassTest </del> for i, t := range tests { // Beware of closure, use local variables instead of those from // outer scope

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-slow [sig-storage] Dynamic Provisioning DynamicProvisioner [Slow] should provision storage with different parameters 5/13 https://k8s- v1beta1.StorageClass got removed in We should remove the v1beta1 test.
/assign can you help take a look?
/triage accepted
/sig storage
I run the test with the latest from master without removing the beta test and it failed with a different error before getting to the beta test part: Do you have any idea what would've happened in my setup? I redeployed k/k using master HEAD
For this testcase: I saw that the provisioned PD has a size of 1.5Gi according to kubernetes: And in gcloud It's really interesting that the test was expecting 2Gi to be created, is this something that was changed recently?
The logs you posted are showing pd csi as the provisioner. Were you testing with the intree driver or csi driver? The CSI driver should to the nearest Gb. I believe intree does the .
I think I did intree because I didn't send as an argument in my command: It's interesting that the PVC has this annotation too:
mmm something interesting about my setup is that I created the cluster with CSIMigration turned on Do you think that might be forcing the usage of ?
Yes likely. Try with CSIMigration off and see if that helps. If so, then this potentially seems like a rounding bug with the csi driver
Thanks I tried with and it did indeed round up to 2Gi.


Question: t.Errorf("No client request should have been made") } } <ins> // TestAdmissionNamespaceExistsUnknownToHandler func TestAdmissionNamespaceExistsUnknownToHandler(t *testing.T) { namespace := "test" mockClient := &client.Fake{ Err: errors.NewAlreadyExists("namespaces", namespace), } store := cache.NewStore(cache.MetaNamespaceKeyFunc) handler := &provision{ client: mockClient, store:  store, } pod := api.Pod{ ObjectMeta: api.ObjectMeta{Name: "123", Namespace: namespace}, Spec: api.PodSpec{ Volumes:    []api.Volume{{Name: "vol"}}, Containers: []api.Container{{Name: "ctr", Image: "image"}}, }, } err := handler.Admit(admission.NewAttributesRecord(&pod, namespace, "pods", "CREATE")) if err != nil { t.Errorf("Unexpected error returned from admission handler") } } </ins>

While running one of my soak tests I noticed the error message: I don't see why I should get this error message. Why should it matter if the namespace already exists? It was created as part of the service creation call just beforehand. And the second attempt to create this pod in exactly the same namespace works. Is this a bug?
Running a second time did not produce this error.
I agree it should not matter. The error is produced there, and I suspect there were multiple concurrent calls being made that try to create a resource in a namespace that had not yet existed. We need to update the code in question to swallow the error if the namespace already exists.
FYI - longer term, I would like to eliminate the use of autoprovision of namespace and require you to first create a namespace manually. what is there now is more to preserve existing behavior prior to namespace as object was introduced.


Question: // will not need to be updated until new changes are applied (detach is triggered again) time.Sleep(100 * time.Millisecond) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) // After the first detach fails, reconciler will wait for a period of time before retrying to detach. // The wait time is increasing exponentially from initial value of 0.5s (0.5, 1, 2, 4, ...). // The test here waits for 100 Millisecond to make sure it is in exponential backoff period after // the first detach operation. At this point, volumes status should not be updated time.Sleep(100 * time.Millisecond) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) verifyVolumeNoStatusUpdateNeeded(t, logger, generatedVolumeName, nodeName1, asw) // Wait for 600ms to make sure second detach operation triggered. Again, The volume will be // removed from being reported as attached on node status and then added back as attached. // The volume will be in the list of attached volumes that need to be updated to node status. time.Sleep(600 * time.Millisecond) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> // Add a second pod which tries to attach the volume to the same node. // After adding pod to the same node, detach will not be triggered any more.

TestRunOneVolumeDetachFailNodeWithReadWriteOnce 4 times since 03-19 23:00 PDT, intermittently ! https://k8s- Still investigating whether this is a tolerable flake or not with more history than testgrid view, or otherwise would appreciate input from anyone with expertise in this test, especially as regards to whether this should be a blocker for tomorrows beta cut. maybe: /sig storage /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is already open for this, looks like it is attempting to fix it
thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. /close
Closing this issue. <detailsIn response to : thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // will not need to be updated until new changes are applied (detach is triggered again) time.Sleep(100 * time.Millisecond) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) // After the first detach fails, reconciler will wait for a period of time before retrying to detach. // The wait time is increasing exponentially from initial value of 0.5s (0.5, 1, 2, 4, ...). // The test here waits for 100 Millisecond to make sure it is in exponential backoff period after // the first detach operation. At this point, volumes status should not be updated time.Sleep(100 * time.Millisecond) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) verifyVolumeNoStatusUpdateNeeded(t, logger, generatedVolumeName, nodeName1, asw) // Wait for 600ms to make sure second detach operation triggered. Again, The volume will be // removed from being reported as attached on node status and then added back as attached. // The volume will be in the list of attached volumes that need to be updated to node status. time.Sleep(600 * time.Millisecond) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> // Add a second pod which tries to attach the volume to the same node. // After adding pod to the same node, detach will not be triggered any more.

pull-kubernetes-unit TestRunOneVolumeDetachFailNodeWithReadWriteOnce Jan 7th, 2022 Check volume <fake-plugin/volume-nameis reported as attached to node <fail-detach-node, got false, expected true No response_ /sig storage
/cc
this is the top unit test flake according to
cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
I think this test is no longer flaky by checking the test grid, close it now.
Unfortunately this does still seem to be flaky: Here was the output from : /reopen
Reopened this issue. <detailsIn response to : Unfortunately this does still seem to be flaky: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
still the case
How about adding some logging to help troubleshoot this? I looked at it a little, but it is hard to tell which part of the test is failing. PR
/
This is still happening
/triage accepted
From the logs, in the OneVolumeDetachFailNodeWithReadWriteOnce test, succeeded, but the next verification failed. In verifyVolumeAttachedToNode, the attachState is from calling and within asw, it calls . In verifyVolumeReportedAsAttachedToNode, the volumes are obtained from calling , it seems that the node is not in asw.nodesToUpdateStatusFor at this step. Cannot reproduce locally. I tried to run the test 100 times locally with debug.Stack()... 3 * 100 times all passed.
Tried to run stress test on my linux machine as well but still cannot reproduce. Command: Result: Also tried running n process in parallel. Command: Result:
It seems like it happens more frequently.


Question: // will not need to be updated until new changes are applied (detach is triggered again) time.Sleep(100 * time.Millisecond) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) // After the first detach fails, reconciler will wait for a period of time before retrying to detach. // The wait time is increasing exponentially from initial value of 0.5s (0.5, 1, 2, 4, ...). // The test here waits for 100 Millisecond to make sure it is in exponential backoff period after // the first detach operation. At this point, volumes status should not be updated time.Sleep(100 * time.Millisecond) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) verifyVolumeNoStatusUpdateNeeded(t, logger, generatedVolumeName, nodeName1, asw) // Wait for 600ms to make sure second detach operation triggered. Again, The volume will be // removed from being reported as attached on node status and then added back as attached. // The volume will be in the list of attached volumes that need to be updated to node status. time.Sleep(600 * time.Millisecond) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> // Add a second pod which tries to attach the volume to the same node. // After adding pod to the same node, detach will not be triggered any more.

But I may have no time to dig into the cause. Let's see.
cc as the test author.
The problem is that this test has a lot of dependency on the sleeps that are hardcoded with a very narrow scope, but our CI seems to be able to stress it much more. As soon as you slightly change one of the sleep duration it starts to flake locally The test can not make an assumption that timing is perfect, even more in the sub second range can someone in sig-storage review this test and make it less dependant on those tight sleeps?
According to most failures, it failed at Line 660. Let me run some stress locally. with 250ms sleep with 100ms sleep
I agree this test case is too brittle due to its specific timing dependencies. To fix this, I suggest reworking the test case to: 1) Add retry with timeout to wait for final conditions 2) Remove the intermediate timing-based state transition checks. Instead, we can keep track of state transitions in the Fakes and validate those stats. wdyt?
Friendly ping from 1.27 CI Signal that this test is still a noisy flake on the master-blocking dashboard as CI Signal has flagged it in and


Question: <del> apiVersion: admissionregistration.k8s.io/v1alpha1 kind: ValidatingAdmissionPolicy metadata: annotations: annotationsKey: annotationsValue creationTimestamp: "2008-01-01T01:01:01Z" deletionGracePeriodSeconds: 10 deletionTimestamp: "2009-01-01T01:01:01Z" finalizers: - finalizersValue generateName: generateNameValue generation: 7 labels: labelsKey: labelsValue managedFields: - apiVersion: apiVersionValue fieldsType: fieldsTypeValue fieldsV1: {} manager: managerValue operation: operationValue subresource: subresourceValue time: "2004-01-01T01:01:01Z" name: nameValue namespace: namespaceValue ownerReferences: - apiVersion: apiVersionValue blockOwnerDeletion: true controller: true kind: kindValue name: nameValue uid: uidValue resourceVersion: resourceVersionValue selfLink: selfLinkValue uid: uidValue spec: auditAnnotations: - key: keyValue valueExpression: valueExpressionValue failurePolicy: failurePolicyValue matchConditions: - expression: expressionValue name: nameValue matchConstraints: excludeResourceRules: - apiGroups: - apiGroupsValue apiVersions: - apiVersionsValue operations: - operationsValue resourceNames: - resourceNamesValue resources: - resourcesValue scope: scopeValue matchPolicy: matchPolicyValue namespaceSelector: matchExpressions: - key: keyValue operator: operatorValue values: - valuesValue matchLabels: matchLabelsKey: matchLabelsValue objectSelector: matchExpressions: - key: keyValue operator: operatorValue values: - valuesValue matchLabels: matchLabelsKey: matchLabelsValue resourceRules: - apiGroups: - apiGroupsValue apiVersions: - apiVersionsValue operations: - operationsValue resourceNames: - resourceNamesValue resources: - resourcesValue scope: scopeValue paramKind: apiVersion: apiVersionValue kind: kindValue validations: - expression: expressionValue message: messageValue messageExpression: messageExpressionValue reason: reasonValue variables: null status: conditions: - lastTransitionTime: "2004-01-01T01:01:01Z" message: messageValue observedGeneration: 3 reason: reasonValue status: statusValue type: typeValue observedGeneration: 1 typeChecking: expressionWarnings: - fieldRef: fieldRefValue warning: warningValue </del>

spec.variables should be marked omitempty. I'm trying to create a validating admission policy that uses in v1.27 as follows: I'm setting : to "Strict" in CreateOptions so that the server will reject requests with unknown fields but it complains that the don't exist in v1.27 and it does make sense because this field was introduced in v1.28: I'm using a 1.28 client.
/sig api-machinery
/assign
/assign /triage accepted
Thanks for raising this! /cc


Question: return } <ins> func (az *azPublicIPAddressesClient) GetVirtualMachineScaleSetPublicIPAddress(ctx context.Context, resourceGroupName string, virtualMachineScaleSetName string, virtualmachineIndex string, networkInterfaceName string, IPConfigurationName string, publicIPAddressName string, expand string) (result network.PublicIPAddress, err error) { if !az.rateLimiterReader.TryAccept() { err = createRateLimitErr(false, "VMSSPublicIPGet") return } klog.V(10).Infof("azPublicIPAddressesClient.GetVirtualMachineScaleSetPublicIPAddress(%q,%q): start", resourceGroupName, publicIPAddressName) defer func() { klog.V(10).Infof("azPublicIPAddressesClient.GetVirtualMachineScaleSetPublicIPAddress(%q,%q): end", resourceGroupName, publicIPAddressName) }() mc := newMetricContext("vmss_public_ip_addresses", "get", resourceGroupName, az.client.SubscriptionID, "") result, err = az.client.GetVirtualMachineScaleSetPublicIPAddress(ctx, resourceGroupName, virtualMachineScaleSetName, virtualmachineIndex, networkInterfaceName, IPConfigurationName, publicIPAddressName, expand) mc.Observe(err) return } </ins> func (az *azPublicIPAddressesClient) List(ctx context.Context, resourceGroupName string) ([]network.PublicIPAddress, error) { if !az.rateLimiterReader.TryAccept() { return nil, createRateLimitErr(false, "PublicIPList")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Node external IP of azure VMSS cannot be obtained when cloud-controller-manager is enabled. Corresponding logs are shown as follows:
/sig azure
/milestone v1.16


Question: return nil, err } <del> res := metricObj.(*v1beta2.MetricValueList) </del> <ins> var res *v1beta2.MetricValueList var ok bool if res, ok = metricObj.(*v1beta2.MetricValueList); !ok { return nil, fmt.Errorf("the custom metrics API server didn't return MetricValueList, the type is %v", reflect.TypeOf(metricObj)) } </ins> return res, nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: kube-controller-manager should not panic. How to reproduce it (as minimally and precisely as possible): Kubernetes 1.12.6, , and Anything else we need to know?: It's running and , and logs says: Don't see any notable errors in . Is that because it's running the old ? Basically, is expecting , but somehow it's getting which I assume Environment: Kubernetes version (use ): v1.12.6 Cloud provider or hardware configuration: AWS EC2 OS (e.g: ): Amazon Linux 2 Kernel (e.g. ): 4.14.109-99.92.amzn2.x86_64
Could be related to , but this happens with all the patches from the issue /sig scalability /cc
/sig autoscaling /assign can you triage please?
/sig-remove scalabilty
/sig remove scalability
/sig scalability remove
/remove-sig scalability
API returned v1.Status instead of v1beta2.MetricValueList. We should check what is the type of the object
/sig instrumention
/assign
/sig instrumentation
seems promising. Is it missing anything?
I've root-caused the issue. Explained here -
Yep, should address the issue.


Question: if protocol == "udp" { // TODO: It would be enough to pass 1s+epsilon to timeout, but unfortunately // busybox timeout doesn't support non-integer values. <del> cmd = fmt.Sprintf("echo 'hostName' | nc -w 1 -u %s %d", targetIP, targetPort) </del> <ins> cmd = fmt.Sprintf("echo hostName | nc -w 1 -u %s %d", targetIP, targetPort) </ins> } else { ipPort := net.JoinHostPort(targetIP, strconv.Itoa(targetPort)) // The current versions of curl included in CentOS and RHEL distros

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: should function for node-pod communication: udp Since when has it been failing: Unknown. Testgrid link: The test is currently excluded. Reason for failure: By default, cmd is used to execute commands in Windows containers, but it has some issues executing the command (no output): echo 'hostName' | nc -w 1 -u <targetIP<targetPortBut it has no issues running (output is correct): echo hostName | nc -w 1 -u <targetIP<targetPortThis causes the test "should function for node-pod communication: udp" to fail when using Windows Containers. Anything else we need to know: /sig testing /sig windows


Question: } func mountPointsEqual(a, b *MountPoint) bool { <del> if a.Device != b.Device || a.Path != b.Path || a.Type != b.Type || !slicesEqual(a.Opts, b.Opts) || a.Pass != b.Pass || a.Freq != b.Freq { </del> <ins> if a.Device != b.Device || a.Path != b.Path || a.Type != b.Type || !reflect.DeepEqual(a.Opts, b.Opts) || a.Pass != b.Pass || a.Freq != b.Freq { </ins> return false } return true } <del> func slicesEqual(a, b []string) bool { if len(a) != len(b) { return false } for i := range a { if a[i] != b[i] { return false } } return true } </del> func TestGetMountRefs(t *testing.T) { fm := &FakeMounter{ MountPoints: []MountPoint{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: In some files, i found that there are many self-contained function, whose feature is as same as some util function. Obviously, they are redundant. What you expected to happen: Use util function to replace self-contained function How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
similar comment to Can you edit the description in to say instead of ? Thanks!
Thanks. Done.


Question: } // Verify the proxy server logs saw the connection <del> expectedProxyLog := fmt.Sprintf("Accepting CONNECT to %s", strings.TrimRight(strings.TrimLeft(framework.TestContext.Host, "https://"), "/api")) </del> <ins> expectedProxyLog := fmt.Sprintf("Accepting CONNECT to %s", strings.TrimSuffix(strings.TrimPrefix(framework.TestContext.Host, "https://"), "/api")) </ins> proxyLog := proxyLogs.String() if !strings.Contains(proxyLog, expectedProxyLog) {

TrimLeft returns a slice of the string s with all leading Unicode code points contained in cutset removed. To remove a prefix, use TrimPrefix instead. /sig release /kind bug
/assign


Question: framework.ExpectNoError(err, "error deleting Service") return svc } <ins> // endpointSlicesEqual compare if the Endpoint and the EndpointSliceList contains the same endpoints values // as in addresses and ports, considering Ready and Unready addresses func endpointSlicesEqual(endpoints *v1.Endpoints, endpointSliceList *discoveryv1.EndpointSliceList) bool { // get the apiserver endpoint addresses epAddresses := sets.NewString() epPorts := sets.NewInt32() for _, subset := range endpoints.Subsets { for _, addr := range subset.Addresses { epAddresses.Insert(addr.IP) } for _, addr := range subset.NotReadyAddresses { epAddresses.Insert(addr.IP) } for _, port := range subset.Ports { epPorts.Insert(port.Port) } } framework.Logf("Endpoints addresses: %v , ports: %v", epAddresses.List(), epPorts.List()) // Endpoints are single stack, and must match the primary IP family of the Service kubernetes.default // However, EndpointSlices can be IPv4 or IPv6, we can only compare the Slices that match the same IP family // framework.TestContext.ClusterIsIPv6() reports the IP family of the kubernetes.default service var addrType discoveryv1.AddressType if framework.TestContext.ClusterIsIPv6() { addrType = discoveryv1.AddressTypeIPv6 } else { addrType = discoveryv1.AddressTypeIPv4 } // get the apiserver addresses from the endpoint slice list sliceAddresses := sets.NewString() slicePorts := sets.NewInt32() for _, slice := range endpointSliceList.Items { if slice.AddressType != addrType { framework.Logf("Skipping slice %s: wanted %s family, got %s", slice.Name, addrType, slice.AddressType) continue } for _, s := range slice.Endpoints { sliceAddresses.Insert(s.Addresses...) } for _, ports := range slice.Ports { if ports.Port != nil { slicePorts.Insert(*ports.Port) } } } framework.Logf("EndpointSlices addresses: %v , ports: %v", sliceAddresses.List(), slicePorts.List()) if sliceAddresses.Equal(epAddresses) && slicePorts.Equal(epPorts) { return true } return false } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --E2E test fails. The test is implemented at: I would expect the test pass with these objects in place: AFAIK this fulfills the general requirement of the test that there's a corresponding endpointslice object with same addressing information as is on the endpoint object. IMO the test should rather look for epslice which is owned by the endpoint rather than relying on a hardcoded name. Add flag to kube-apiserver Manually create the endpoint with proper addressing for the api-server epslice "mirroring" create new endpointslice with name like --The test fails as it's hardcoded to look for endpointslice named I'm working on a PR to provide a fix proposal Kubernetes version (use ): 1.22.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: k0s Network plugin and version (if this is a network-related bug): kube-router 1.3.1 Others:
/sig testing
/remove-sig testing /sig network EndpointSlices are owned by sig network.
/kind failing-test
/remove kind-bug /remove kind-failing-test The test is correct, apiserver must generates both endpoint and enpointslices with the special name "kubernetes" , that are the backends for the "kubernetes.default" service with the IP addresses of the apiservers. This special service "kubernetes.default" and its endpoints are used by clients with In-Cluster auto discovery. I think that whatever cluster that is not generating the endpoint and endpointslices for the "kubernetes.default" service should not meet Conformance, at least I can't see how automatic in-cluster configuration will work without that ... I only see option used for webhook tests, the other 2 reconcilers always create the endpoints and slices for "kubernetes.default" should we clarify the goal of this "none" option?
Yes, I do understand this. And in our case this is true, there is a svc called as there is also endpoints object called . This makes the in-cluster discovery work.


Question: framework.ExpectNoError(err, "error deleting Service") return svc } <ins> // endpointSlicesEqual compare if the Endpoint and the EndpointSliceList contains the same endpoints values // as in addresses and ports, considering Ready and Unready addresses func endpointSlicesEqual(endpoints *v1.Endpoints, endpointSliceList *discoveryv1.EndpointSliceList) bool { // get the apiserver endpoint addresses epAddresses := sets.NewString() epPorts := sets.NewInt32() for _, subset := range endpoints.Subsets { for _, addr := range subset.Addresses { epAddresses.Insert(addr.IP) } for _, addr := range subset.NotReadyAddresses { epAddresses.Insert(addr.IP) } for _, port := range subset.Ports { epPorts.Insert(port.Port) } } framework.Logf("Endpoints addresses: %v , ports: %v", epAddresses.List(), epPorts.List()) // Endpoints are single stack, and must match the primary IP family of the Service kubernetes.default // However, EndpointSlices can be IPv4 or IPv6, we can only compare the Slices that match the same IP family // framework.TestContext.ClusterIsIPv6() reports the IP family of the kubernetes.default service var addrType discoveryv1.AddressType if framework.TestContext.ClusterIsIPv6() { addrType = discoveryv1.AddressTypeIPv6 } else { addrType = discoveryv1.AddressTypeIPv4 } // get the apiserver addresses from the endpoint slice list sliceAddresses := sets.NewString() slicePorts := sets.NewInt32() for _, slice := range endpointSliceList.Items { if slice.AddressType != addrType { framework.Logf("Skipping slice %s: wanted %s family, got %s", slice.Name, addrType, slice.AddressType) continue } for _, s := range slice.Endpoints { sliceAddresses.Insert(s.Addresses...) } for _, ports := range slice.Ports { if ports.Port != nil { slicePorts.Insert(*ports.Port) } } } framework.Logf("EndpointSlices addresses: %v , ports: %v", sliceAddresses.List(), slicePorts.List()) if sliceAddresses.Equal(epAddresses) && slicePorts.Equal(epPorts) { return true } return false } </ins>

Otherwise we'd get much more errors in the conformance instead of this only one... :) In our case we're relying on the endpointslice mirroring to take care of creating the endpointslice object, which makes this test fail as it uses generated "suffix" on the object name. I have not dived too deep (yet) into the internals of endpointslices but I fail to see how the endpointslice object name really matters when both svc and endpoint object does have proper name.
oh, this seems interesting then ... who is creating the corresponding endpoint? can you paste a kubectl get endpoints -o yaml?
this is how it looks like in my cluster
In our case (k0s distro) it's our custom controller that is triggered when the controlplane is configured in HA mode. So this basically sets the api to mode . When our controller runs it has a configured LB address which it resolves to IP address(es) and creates the corresponding EP object. We do NOT create the endpointslice as we rely on the mirroring functionality to let it create and sync it. Here's the object details: Reading the code for API built-in endpoint reconcilers I see they set and create the endpointslice object by themselves. Yes, we could do that too in k0s but I fail to see why the name of the endpointslice object really matters at all.
I agree the conformance test should not be looking up endpointslice by name. It should be using a label selector of to locate one or more endpointslices and verifying the ports and endpoints across them.
Fair enough, on the good side this also will simplify the KEP from to dual-stack the apiserver you are the expert in this area, can you PTAL at this discussion? /kind failing-test
hmm the test description explicitly says the name of the object has to be "kubernetes", if the name is mandatory the test is correct, otherwise we need to rewrite the test description and talk with conformance people /area conformance
the description should be rewritten... the test is enforcing stricter requirements on endpointslices than required/expected
/triage accepted


Question: docs/user-guide/kubectl/kubectl_alpha.md docs/user-guide/kubectl/kubectl_alpha_diff.md docs/user-guide/kubectl/kubectl_annotate.md <ins> docs/user-guide/kubectl/kubectl_api-resources.md </ins> docs/user-guide/kubectl/kubectl_api-versions.md docs/user-guide/kubectl/kubectl_apply.md docs/user-guide/kubectl/kubectl_apply_edit-last-applied.md

As the RBAC role need to be related to resources. I think we could use a command to get the supported resources. ref to
This would be cool.
I thought we a command that listed resources in the discover doc a long time ago.
is just the versions, but principle is the same. We want to list all the types, their resources, and tie that back to "explain".
What's the best way to determine this information currently? I'm having difficulty figuring out the exact set of permissions (and how they would be written in an RBAC role) for any given kubectl command. The documentation for this right now is split across multiple places and certain things are not explained at all. Specifically, it's difficult to determine which "API verb" is used for which operations. Take as an example. It's unclear if the way to allow this is to use the "proxy" subresource of a pod or service with "get"/"create"/etc. verbs, or to use the "proxy" verb with the resource (pod, service) directly. is a similar case. The almost works for this purpose, but it's still difficult. Operations for a given resource are grouped into categories like write, read, status, and proxy, but these things don't obviously correlate with API verbs. Again, proxy is especially confusing as there is no mention of a proxy "verb". The API operations are only shown using their HTTP verbs. One thing that could help on this document would be a section on the right column under "kubectl command" that shows exactly the rule needed in an RBAC role to allow this command via kubectl. And even then, that assumes an exact 1-to-1 mapping between API endpoints and kubectl commands, which I doubt is always the case. (For example, port fowarding to a pod seems to require both a GET on the pod resource and a CREATE on a port forward subresource.) There also needs to be better documentation about how this information could have been determined in the first place. I've resorted to manually looking through API requests made with but even then it's difficult to figure out. Amusingly, I was the one who wrote which is referenced in the original issue description (as well as the table explaining verbs in ).


Question: docs/user-guide/kubectl/kubectl_alpha.md docs/user-guide/kubectl/kubectl_alpha_diff.md docs/user-guide/kubectl/kubectl_annotate.md <ins> docs/user-guide/kubectl/kubectl_api-resources.md </ins> docs/user-guide/kubectl/kubectl_api-versions.md docs/user-guide/kubectl/kubectl_apply.md docs/user-guide/kubectl/kubectl_apply_edit-last-applied.md

tl;dr: It's too difficult right now to go from "I want a user to be able to run " to "here is the RBAC rule to use for that" for any case where there isn't a specific example shown in the docs.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
It seem that relative PR stop updated. If this issue still valuable, i will help fix it
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: fs.MarkDeprecated("non-masquerade-cidr", "will be removed in a future version") fs.BoolVar(&f.KeepTerminatedPodVolumes, "keep-terminated-pod-volumes", f.KeepTerminatedPodVolumes, "Keep terminated pod volumes mounted to the node after the pod terminates.  Can be useful for debugging volume related issues.") fs.MarkDeprecated("keep-terminated-pod-volumes", "will be removed in a future version") <del> fs.BoolVar(&f.EnableCAdvisorJSONEndpoints, "enable-cadvisor-json-endpoints", f.EnableCAdvisorJSONEndpoints, "Enable cAdvisor json /spec and /stats/* endpoints.  [default=false]") </del> <ins> fs.BoolVar(&f.EnableCAdvisorJSONEndpoints, "enable-cadvisor-json-endpoints", f.EnableCAdvisorJSONEndpoints, "Enable cAdvisor json /spec and /stats/* endpoints. This flag has no effect on the /stats/summary endpoint.  [default=false]") </ins> // TODO: Remove this flag in 1.20+.  https://github.com/kubernetes/kubernetes/issues/68522 fs.MarkDeprecated("enable-cadvisor-json-endpoints", "will be removed in a future version") fs.BoolVar(&f.ReallyCrashForTesting, "really-crash-for-testing", f.ReallyCrashForTesting, "If true, when panics occur crash. Intended for testing.")

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Update the help text for kubelet's flag to explicitly mention that it has no effect over the endpoint. Why is this needed: The help text for kubelet's flag describes it as "Enable cAdvisor json and endpoints." which would seem to include , however this is not the case as that endpoint is required by kube-metrics-server and exposes stats from cadvisor or CRI (depending on the stat provider selected internally). Related Issues: /sig node /sig instrumentation /kind documentation
/kind documentation /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /kind documentation /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign This should be updated in as well as (unless there is some automated process that updates the website from the options in code that I'm not aware of).
To summarize a conversation in slack, it appears there is a documented procedure for updating the website found at The procedure is long, so it is tempting to just copy in the small amount of text that was changed from into a website PR rather than pulling down so many moving parts. In any case, we likely should keep this issue open until both the change to and an associated website change are merged.
can you link to the Slack convo?
Can do - The conversation ended with a question whether there is a bug that would prevent an automated update of the documentation or if we need to submit a PR now that the fix has merged. I searched for an issue but didn't find a match. So we can either try to find someone on sig-docs who might know if there is an issue if it should be updated automagically, or just fix the doc if the website didn't get updated (and I just checked the master branch and it has not been updated yet).
Just in case that is the right answer, I created the PR.
I think the website gets autogenerated upon point releases but that'd be a good thing to clarify with the website team. I'll leave a comment asking as much.


Question: <del> ## JSON-Patch </del> <ins> # JSON-Patch `jsonpatch` is a library which provides functionallity for both applying [RFC6902 JSON patches](http://tools.ietf.org/html/rfc6902) against documents, as well as for calculating & applying [RFC7396 JSON merge patches](https://tools.ietf.org/html/rfc7396). </ins> <del> Provides the ability to modify and test a JSON according to a [RFC6902 JSON patch](http://tools.ietf.org/html/rfc6902) and [RFC7396 JSON Merge Patch](https://tools.ietf.org/html/rfc7396). </del> <ins> [![GoDoc](https://godoc.org/github.com/evanphx/json-patch?status.svg)](http://godoc.org/github.com/evanphx/json-patch) [![Build Status](https://travis-ci.org/evanphx/json-patch.svg?branch=master)](https://travis-ci.org/evanphx/json-patch) [![Report Card](https://goreportcard.com/badge/github.com/evanphx/json-patch)](https://goreportcard.com/report/github.com/evanphx/json-patch) </ins> <del> *Version*: **1.0** </del> <ins> # Get It! </ins> <del> [![GoDoc](https://godoc.org/github.com/evanphx/json-patch?status.svg)](http://godoc.org/github.com/evanphx/json-patch) </del> <ins> **Latest and greatest**: ```bash go get -u github.com/evanphx/json-patch ``` </ins> <del> [![Build Status](https://travis-ci.org/evanphx/json-patch.svg?branch=master)](https://travis-ci.org/evanphx/json-patch) </del> <ins> **Stable Versions**: * Version 3: `go get -u gopkg.in/evanphx/json-patch.v3` (previous versions below `v3` are unavailable) # Use It! * [Create and apply a merge patch](#create-and-apply-a-merge-patch) * [Create and apply a JSON Patch](#create-and-apply-a-json-patch) * [Comparing JSON documents](#comparing-json-documents) * [Combine merge patches](#combine-merge-patches) ## Create and apply a merge patch Given both an original JSON document and a modified JSON document, you can create a [Merge Patch](https://tools.ietf.org/html/rfc7396) document. It can describe the changes needed to convert from the original to the modified JSON document. Once you have a merge patch, you can apply it to other JSON documents using the `jsonpatch.MergePatch(document, patch)` function. ```go package main import ( "fmt" jsonpatch "github.com/evanphx/json-patch" ) func main() { // Let's create a merge patch from these two documents... original := []byte(`{"name": "John", "age": 24, "height": 3.21}`) target := []byte(`{"name": "Jane", "age": 24}`) patch, err := jsonpatch.CreateMergePatch(original, target) if err != nil { panic(err) } // Now lets apply the patch against a different JSON document... alternative := []byte(`{"name": "Tina", "age": 28, "height": 3.75}`) modifiedAlternative, err := jsonpatch.MergePatch(alternative, patch) fmt.Printf("patch document:   %sn", patch) fmt.Printf("updated alternative doc: %sn", modifiedAlternative) } ``` When ran, you get the following output: ```bash $ go run main.go patch document:   {"height":null,"name":"Jane"} updated tina doc: {"age":28,"name":"Jane"} ``` ## Create and apply a JSON Patch You can create patch objects using `DecodePatch([]byte)`, which can then be applied against JSON documents. The following is an example of creating a patch from two operations, and applying it against a JSON document. ```go package main import ( "fmt" jsonpatch "github.com/evanphx/json-patch" ) func main() { original := []byte(`{"name": "John", "age": 24, "height": 3.21}`) patchJSON := []byte(`[ {"op": "replace", "path": "/name", "value": "Jane"}, {"op": "remove", "path": "/height"} ]`) patch, err := jsonpatch.DecodePatch(patchJSON) if err != nil { panic(err) } modified, err := patch.Apply(original) if err != nil { panic(err) } fmt.Printf("Original document: %sn", original) fmt.Printf("Modified document: %sn", modified) } ``` When ran, you get the following output: ```bash $ go run main.go Original document: {"name": "John", "age": 24, "height": 3.21} Modified document: {"age":24,"name":"Jane"} ``` ## Comparing JSON documents Due to potential whitespace and ordering differences, one cannot simply compare JSON strings or byte-arrays directly. As such, you can instead use `jsonpatch.Equal(document1, document2)` to determine if two JSON documents are _structurally_ equal. This ignores whitespace differences, and key-value ordering. ```go package main import ( "fmt" jsonpatch "github.com/evanphx/json-patch" ) func main() { original := []byte(`{"name": "John", "age": 24, "height": 3.21}`) similar := []byte(` { "age": 24, "height": 3.21, "name": "John" } `) different := []byte(`{"name": "Jane", "age": 20, "height": 3.37}`) if jsonpatch.Equal(original, similar) { fmt.Println(`"original" is structurally equal to "similar"`) } if !jsonpatch.Equal(original, different) { fmt.Println(`"original" is _not_ structurally equal to "similar"`) } } ``` When ran, you get the following output: ```bash $ go run main.go "original" is structurally equal to "similar" "original" is _not_ structurally equal to "similar" ``` ## Combine merge patches Given two JSON merge patch documents, it is possible to combine them into a single merge patch which can describe both set of changes. The resulting merge patch can be used such that applying it results in a document structurally similar as merging each merge patch to the document in succession. ```go package main import ( "fmt" jsonpatch "github.com/evanphx/json-patch" ) func main() { original := []byte(`{"name": "John", "age": 24, "height": 3.21}`) nameAndHeight := []byte(`{"height":null,"name":"Jane"}`) ageAndEyes := []byte(`{"age":4.23,"eyes":"blue"}`) // Let's combine these merge patch documents... combinedPatch, err := jsonpatch.MergeMergePatches(nameAndHeight, ageAndEyes) if err != nil { panic(err) } // Apply each patch individual against the original document withoutCombinedPatch, err := jsonpatch.MergePatch(original, nameAndHeight) if err != nil { panic(err) } withoutCombinedPatch, err = jsonpatch.MergePatch(withoutCombinedPatch, ageAndEyes) if err != nil { panic(err) } // Apply the combined patch against the original document withCombinedPatch, err := jsonpatch.MergePatch(original, combinedPatch) if err != nil { panic(err) } // Do both result in the same thing? They should! if jsonpatch.Equal(withCombinedPatch, withoutCombinedPatch) { fmt.Println("Both JSON documents are structurally the same!") } fmt.Printf("combined merge patch: %s", combinedPatch) } ``` When ran, you get the following output: ```bash $ go run main.go Both JSON documents are structurally the same! combined merge patch: {"age":4.23,"eyes":"blue","height":null,"name":"Jane"} ``` # CLI for comparing JSON documents You can install the commandline program `json-patch`. This program can take multiple JSON patch documents as arguments, and fed a JSON document from `stdin`. It will apply the patch(es) against the document and output the modified doc. **patch.1.json** ```json [ {"op": "replace", "path": "/name", "value": "Jane"}, {"op": "remove", "path": "/height"} ] ``` **patch.2.json** ```json [ {"op": "add", "path": "/address", "value": "123 Main St"}, {"op": "replace", "path": "/age", "value": "21"} ] ``` </ins> <del> ### API Usage </del> <ins> **document.json** ```json { "name": "John", "age": 24, "height": 3.21 } ``` </ins> <del> * Given a `[]byte`, obtain a Patch object </del> <ins> You can then run: </ins> <del> `obj, err := jsonpatch.DecodePatch(patch)` </del> <ins> ```bash $ go install github.com/evanphx/json-patch/cmd/json-patch $ cat document.json | json-patch -p patch.1.json -p patch.2.json {"address":"123 Main St","age":"21","name":"Jane"} ``` </ins> <del> * Apply the patch and get a new document back </del> <ins> # Help It! Contributions are welcomed! Leave [an issue](https://github.com/evanphx/json-patch/issues) or [create a PR](https://github.com/evanphx/json-patch/compare). </ins> <del> `out, err := obj.Apply(doc)` * Create a JSON Merge Patch document based on two json documents (a to b): </del> <ins> Before creating a pull request, we'd ask that you make sure tests are passing and that you have added new tests when applicable. </ins> <del> `mergeDoc, err := jsonpatch.CreateMergePatch(a, b)` * Bonus API: compare documents for structural equality </del> <ins> Contributors can run tests using: </ins> <del> `jsonpatch.Equal(doca, docb)` </del> <ins> ```bash go test -cover ./... ``` Builds for pull requests are tested automatically using [TravisCI](https://travis-ci.org/evanphx/json-patch). </ins>

/kind bug What happened: Wrote a that returned a faulty json patch which specified an our of range index (e.G. when there is only one container). This resulted in the apiserver crashing. Stacktrace: What you expected to happen: No crash /sig-api-machinery
/sig api-machinery
This maybe a bug in . I will follow this question.
This bug had been fixed in PR: you can checkout it.
Thanks for your comment but I don't think the linked PR fixed the issue, as it got merged almost a year ago and I am running Kube 1.10.1 which does contain that patch but still panics. Just from quickly looking at the code I the linked PR I believe there are two things wrong: The check if the index is valid only happens when a negative index was given should be
Once the PR of is merged I will update kubernetes.
/kind bug


Question: glog.Fatalf("unable to create flag defaults: %v", err) } <del> opts.AddFlags(pflag.CommandLine) </del> <ins> opts.AddFlags(cmd.Flags()) </ins> cmd.MarkFlagFilename("config", "yaml", "yml", "json")

There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
probably caused by
Could be a clash between these two?
cc
I still see the issue with local-cluster on the latest master: F0225 17:06:49. ] error: v1alpha1.KubeProxyConfiguration: FeatureGates: ReadMapCB: expect { or n, but found ", error found in byte of ......
looks like that's another issue, can you please file a new bug? with more logs? this one was about none of the hyperkube based processes starting up.
will do as it is also broke container kube-proxy as well. Same type of error.


Question: if [ ! -w $(dirname `which gcloud`) ]; then sudo_prefix="sudo" fi <del> ${sudo_prefix} gcloud ${gcloud_prompt:-} components update alpha || true ${sudo_prefix} gcloud ${gcloud_prompt:-} components update beta || true </del> <ins> ${sudo_prefix} gcloud ${gcloud_prompt:-} components install alpha || true ${sudo_prefix} gcloud ${gcloud_prompt:-} components install beta || true </ins> ${sudo_prefix} gcloud ${gcloud_prompt:-} components update || true }

This is really annoying because it requires me to interact with my shell while running up / down whereas before the commands just ran to completion on their own.


Question: package testing import ( <del> "encoding/json" </del> "fmt" "math/rand" <del> "os" "path/filepath" </del> <ins> "sigs.k8s.io/structured-merge-diff/v4/typed" </ins> "strconv" <del> "strings" </del> "sync" "testing"

Smoke tests run by publishing-bot on client-go master are failing Ref for failure logs: Tests defined for client-go, after publishing, Since commit No response As part of , when module is used from within the staging directory, can be used by traversing to the parent directory. But once the module is made as a separate repository as part of publishing, then traversing to repository using path will not work. No response /sig-api-machinery
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looking
I'll remove the swagger file dependency, I don't think we strictly need it.
/triage accepted


Question: <ins> apiVersion: v1beta1 desiredState: manifest: containers: - image: kubernetes/pause name: testpd volumeMounts: - mountPath: "/testpd" name: "testpd" id: testpd version: v1beta1 volumes: - name: testpd source: persistentDisk: # This GCE PD must already exist and be formatted ext4 pdName: "%insert_pd_name_here%" fsType: ext4 id: testpd kind: Pod labels: test: testpd </ins>

07-02 and 13-02 No response No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
I might be related to the npd.
The same test failures happening again on 25.02:
/sig node
/sig node /cc The restart of caused the flake. I also opened to track it in NPD repo.
I missed this log. It is caused by oom. Adding some memory may help. I submitted to fix it. Does mean killing metrics-server?
/remove-sig node Sorry, I misread the log. The NPD found the metric server is OOM. It is about the metrics server, not the NPD. Sorry for disturb. I may add some memory for metric-server and check why metric server memory changed. I heard that metrics server recently released a new version. I will check if it is related.
/sig node
OOM metrics shows the metric server is OOMed. This is odd, that only metrics-server-nanny is with a memory limit. It uses whose command is . metrics-server container has no memory limit, but it got killed by OOM. Does this indicate that the node is OOM? (edited)
The last run failed in node-problem-detector ContainerStateTerminated. I0227 05:03:31. ] PodStartupLatency: perc50: 14m29.018886827s, perc90: 28m44.625518807s, perc99: 32m2.19303496s; threshold 1h0m0s I0227 05:03:32. ]


Question: <ins> apiVersion: v1beta1 desiredState: manifest: containers: - image: kubernetes/pause name: testpd volumeMounts: - mountPath: "/testpd" name: "testpd" id: testpd version: v1beta1 volumes: - name: testpd source: persistentDisk: # This GCE PD must already exist and be formatted ext4 pdName: "%insert_pd_name_here%" fsType: ext4 id: testpd kind: Pod labels: test: testpd </ins>

Loaded restart count threshold overrides: map[coredns:2 fluentd-gcp:999 konnectivity-agent:2 kube-controller-manager:1 kube-proxy:2 kube-scheduler:1 l7-lb-controller:1 metadata-proxy:2 prometheus-to-sd-exporter:2] I0227 05:03:32. ] ClusterOOMsTracker: gathering cluster OOMs tracking measurement E0227 05:03:32. ] TestMetrics: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0] I0227 05:03:32. ] Step "[step: 29] gathering measurements" ended W0227 05:03:32. ] Got errors during step execution: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] I0227 05:03:32. ] Waiting for the chaos monkey subroutine to end... I0227 05:03:32. ] Chaos monkey ended. I0227 05:03:32. ] I0227 05:03:32. ] Stopping DnsLookupLatency probe... I0227 05:03:47. ] Stopping InClusterNetworkLatency probe... I0227 05:04:22. ] Resources cleanup time: 50.539700215s E0227 05:04:22. ]
E0227 05:04:22. ] Test Finished E0227 05:04:22. ] Test: E0227 05:04:22. ] Status: Fail E0227 05:04:22. ]


Question: <ins> apiVersion: v1beta1 desiredState: manifest: containers: - image: kubernetes/pause name: testpd volumeMounts: - mountPath: "/testpd" name: "testpd" id: testpd version: v1beta1 volumes: - name: testpd source: persistentDisk: # This GCE PD must already exist and be formatted ext4 pdName: "%insert_pd_name_here%" fsType: ext4 id: testpd kind: Pod labels: test: testpd </ins>

Errors: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] E0227 05:04:22. ]
I0227 05:04:22. ]
I0227 05:04:22. ] Running testing/huge- I0227 05:04:22. ]


Question: - subresource - verb - version <ins> - name: apiserver_storage_object_counts help: Number of stored objects at the time of last check split by kind. type: Gauge stabilityLevel: STABLE labels: - resource </ins>

What would you like to be : We want to start migrating metrics which people depend on for operations to stable status. etcdobjectcounts was mentioned in a recent SIG meeting as a viable candidate. However, the 'etcd' prefix is not in line with stated metric conventions. SIG consensus determined that creating a new version of the metric with the metric name 'apiserverstorageobjectcounts' would be sufficient allow this metric to be promoted. We'd want to simultaneously deprecate 'etcdobjectcounts' (since it would be a duplicate) and remove it after the deprecation grace period window. /assign /cc
GitHub didn't allow me to assign the following users: erain. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : What would you like to be : /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig instrumentation api-machinery /cc
Ack. I am going to work on this.
/assign /triage accepted
In the SIG api-machinery meeting today, we agreed that the stable version of this metric should be named 'apiserverstorageobject_count'
Thanks and will use in the PR. (And sorry for missing today's meeting - due to a conflict)


Question: return nil } <ins> //waitForServiceEndpointsNum waits until the amount of endpoints that implement service to expectNum. func waitForServiceEndpointsNum(c *client.Client, namespace, serviceName string, expectNum int, interval, timeout time.Duration) error { return wait.Poll(interval, timeout, func() (bool, error) { Logf("Waiting for amount of service:%s endpoints to %d", serviceName, expectNum) list, err := c.Endpoints(namespace).List(labels.Everything()) if err != nil { return false, err } for _, e := range list.Items { if e.Name == serviceName && countEndpointsNum(&e) == expectNum { return true, nil } } return false, nil }) } func countEndpointsNum(e *api.Endpoints) int { num := 0 for _, sub := range e.Subsets { num += len(sub.Addresses) } return num } </ins> // waitForReplicationController waits until the RC appears (exist == true), or disappears (exist == false) func waitForReplicationController(c *client.Client, namespace, name string, exist bool, interval, timeout time.Duration) error { err := wait.Poll(interval, timeout, func() (bool, error) {

More detail can be seen in e2e test
cc
, Marek Grabowski wrote:
you are running an older version of the test. can you get the latest and try again ? This test had not failed at all yesterday in our automated test runs yesterday. Closing issue for now.
This failure depends on if the pod is really deleted and kube-proxy (the test container on) catch that the endpoint is deleted in 5s . After sending "delete pod" request for 5s, if the pod was still there and kube-proxy still contains this endpoints(this may happen under pressure), hit test can still detect 5 endpoints. At least we should ensure the pod was deleted. Anyway, I think will be helpful to this test. What do you think?
It doesn't matter if the pod is actually deleted or not. We just care that kube-proxy thinks that it should not be part of the service endpoint. Having said that, it still doesn't hurt to do the check that you are doing. I will assign your PR to myself and review it. In the meantime, can you rebase your PR ?
Thanks for your review. The PR is up to date.
Failed twice today in kubernetes-e2e-gce-parallel
One more today. The error message is not very helpful.
18:28:37 STEP: dialing(http) endpointPodIP:endpointHttpPort from node1 18:28:37 STEP: Dialing from node. command:for i in ' wc -l This is the point where the test fails. Am looking into it. We need to have more tries, we just do seq (1 1), instead of seq (6 6). this is causing flakiness.
Fixed by PR


Question: import ( "fmt" "path" <ins> "runtime" </ins> "strings" "sync"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I have tried FlexVolume on a Windows node, it does not work. I am now working on this feature now. Below is the FlexVolume initialization error What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.7.7 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig windows /assign
Detailed steps about how to use flexvolume on Windows with this PR:
cc:


Question: //   3.  CurrentDirectoryPath //   4.  HomeDirectoryPath // Empty filenames are ignored.  Files with non-deserializable content produced errors. <del> // The first file to set a particular value or map key wins and the value or map key is never changed. // This means that the first file to set CurrentContext will have its context preserved.  It also means // that if two files specify a "red-user", only values from the first file's red-user are used.  Even </del> <ins> // The first file to set a particular map key wins and map key's value is never changed. // BUT, if you set a struct value that is NOT contained inside of map, the value WILL be changed. // This results in some odd looking logic to merge in one direction, merge in the other, and then merge the two. // It also means that if two files specify a "red-user", only values from the first file's red-user are used.  Even </ins> // non-conflicting entries from the second file's "red-user" are discarded. // Relative paths inside of the .kubeconfig files are resolved against the .kubeconfig file's parent folder // and only absolute file paths are returned. func (rules *ClientConfigLoadingRules) Load() (*clientcmdapi.Config, error) { <del> config := clientcmdapi.NewConfig() mergeConfigWithFile(config, rules.CommandLinePath) resolveLocalPaths(rules.CommandLinePath, config) </del> <ins> kubeConfigFiles := []string{rules.CommandLinePath, rules.EnvVarPath, rules.CurrentDirectoryPath, rules.HomeDirectoryPath} </ins> <del> mergeConfigWithFile(config, rules.EnvVarPath) resolveLocalPaths(rules.EnvVarPath, config) </del> <ins> // first merge all of our maps mapConfig := clientcmdapi.NewConfig() for _, file := range kubeConfigFiles { mergeConfigWithFile(mapConfig, file) resolveLocalPaths(file, mapConfig) } </ins> <del> mergeConfigWithFile(config, rules.CurrentDirectoryPath) resolveLocalPaths(rules.CurrentDirectoryPath, config) </del> <ins> // merge all of the struct values in the reverse order so that priority is given correctly nonMapConfig := clientcmdapi.NewConfig() for i := len(kubeConfigFiles) - 1; i >= 0; i-- { file := kubeConfigFiles[i] mergeConfigWithFile(nonMapConfig, file) resolveLocalPaths(file, nonMapConfig) } </ins> <del> mergeConfigWithFile(config, rules.HomeDirectoryPath) resolveLocalPaths(rules.HomeDirectoryPath, config) </del> <ins> // since values are overwritten, but maps values are not, we can merge the non-map config on top of the map config and // get the values we expect. config := clientcmdapi.NewConfig() mergo.Merge(config, mapConfig) mergo.Merge(config, nonMapConfig) </ins> return config, nil }

I've been pulling my hair out trying get kubectl talk to my local cluster. It seems that kubectl merges the local file with values pulled down from the cloud. I've done the following: The I just set is silently overridden. Questions: is this documented anywhere? should kubectl merge in my GCE cluster when I set my cloud-provider to something else? should kubectl override my previously set current-context?
workaround is to use flag to override the config file value.
but not a good UX long term
Someone else hit this as well: . Better fix is to have the local-up script set a context, which I will try to add today. As for the overwriting behavior, it's documented in , but based on what you describe, the intended load order is not working; local .kubeconfig should overwrite global. cc
That sounds infuriating. I'll take a look this morning.


Question: apiGroup = apiVersionsToAPIGroup(&v) } apiGroupList.Groups = []metav1.APIGroup{apiGroup} <del> case AcceptV2Beta1: var aggregatedDiscovery apidiscovery.APIGroupDiscoveryList err = json.Unmarshal(body, &aggregatedDiscovery) if err != nil { return nil, nil, nil, err } apiGroupList, resourcesByGV, failedGVs = SplitGroupsAndResources(aggregatedDiscovery) default: return nil, nil, nil, fmt.Errorf("Unknown discovery response content-type: %s", responseContentType) </del> } return apiGroupList, resourcesByGV, failedGVs, nil

We are using (and ) to interact with Kubernetes clusters behind a couple of proxy layers. One of these proxies completes the headers to include character encoding details. This was working fine until recently, when client-go/kubectl have started to fail when interacting with the discovery API. It seems that a recent change has made kubectl more restrictive of the content-type that it expects. I suspect that it is related to this commit: For example, this is an error from a kubectl command: We can of course try to fix it on our end, removing that charset information. However, it is a valid content-type and the way kubectl is handling the content-type there seems like it is relying on implementation details of kube-apiserver (that it does not add metadata to the content-type). I propose to fix this issue in , by properly parsing and handling the content-type.
cc /kind regression /kind bug /sig api-machinery /priority important-soon looks like the switch statement in downloadLegacy() needs to be more tolerant of the variety of content-types it can get from a server (including text/plain or no content-type if old servers could have returned that...)
/transfer kubernetes/kubernetes
Something went wrong or the destination repo kubernetes/kubernetes/kubernetes does not exist. <detailsIn response to : /transfer kubernetes/kubernetes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubernetes
Yep, the string comparison for the content-type is flimsy, let's address that. Thanks for the report!
/assign
/triage accepted
/cc
The fix is being cherry-picked back to: : :


Question: fmt.Fprintf(&verboseOut, "[+]%v okn", check.Name()) } } <ins> if excluded.Len() > 0 { fmt.Fprintf(&verboseOut, "warn: some health checks cannot be excluded: no matches for %vn", formatQuoted(excluded.List()...)) klog.Warningf("cannot exclude some health checks, no health checks are installed matching %v", formatQuoted(excluded.List()...)) } </ins> // always be verbose on failure if failed { http.Error(w, fmt.Sprintf("%vhealthz check failed", verboseOut.String()), http.StatusInternalServerError)

What would you like to be : Add support to the kube-apiserver so that we can disable health-checks via query params. Why is this needed: Currently, the healthz endpoint is inflexible in that any and all health checks are currently executed when the endpoint is hit. Since the kube-apiserver only currently supports a liveness check, this means all persistent health check failures will eventually result in a kubelet driven restart. However, this may not be desirable behavior if you have additional monitoring services layered on top of your control plane. For instance, take the etcd failure from the perspective of the kube-apiserver. Currently, persistent failures during the etcd health check from the api-server healthz endpoint will cause the kube-apiserver to be restarted. This sometimes helps (in the case of a faulty etcd-client connection) but also can be detrimental (the kube-apiserver restarts until the etcd cluster is functioning, then the kube-apiserver bombards the etcd cluster with requests to update local caches, saturating the etcd cluster with requests, causing it to timeout on its own liveness probes and trigger a kubelet-driven restart). Let's say that we could now disable etcd from our liveness health check. Then if we had an external monitoring service, we could ask independently ask etcd if it was healthy and we could also ask the kube-apiserver if it thought etcd was unhealthy. Based on the combination of those answers, we could enable more intelligent responses to certain situation, like restarting the kube-apiserver when it thought etcd was unhealthy but etcd was actually reporting as healthy but not doing so when etcd was actually unhealthy. This feature is backwards-compatible and opt-in. There would be no impact to existing behavior unless these query params are actually used. /sig api-machinery /kind feature
/cc
SGTM
/assign