Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/apimachinery/pkg/util/validation" </ins> utilfeature "k8s.io/apiserver/pkg/util/feature" cliflag "k8s.io/component-base/cli/flag" "k8s.io/component-base/logs"

kubelet won't return error when invalid node labels are passed to flag, and node can't take part in a cluster. kubelet should return errors when invalid node labels are passed. You pass flag to kubelet (value is invalid), node doesn't take part in a cluster. ref. No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig node
/triage accepted
this should not be a breaking change as invalid labels make the kubelet unusable anyway. So better not even start the kubelet. /assign since there is a PR already
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: //   3 Standalone 'kubernetes' binary // Eventually, #2 will be replaced with instances of #3 func RunKubelet(kcfg *KubeletConfig) { <ins> kcfg.Hostname = util.GetHostname(kcfg.HostnameOverride) </ins> if kcfg.KubeClient != nil { <del> kubelet.SetupEventSending(kcfg.KubeClient) </del> <ins> kubelet.SetupEventSending(kcfg.KubeClient, kcfg.Hostname) </ins> } else { glog.Infof("No api server defined - no events will be sent.") } kubelet.SetupLogging() kubelet.SetupCapabilities(kcfg.AllowPrivileged) <del> kcfg.Hostname = util.GetHostname(kcfg.HostnameOverride) </del> cfg := makePodSourceConfig(kcfg) k, err := createAndInitKubelet(kcfg, cfg) if err != nil {

Running the pre-built 0.8.0 binaries this flag does not work. Looks like the
I think that code happens only when setting up events, right? The hostnameOverride flag seems to still be there: what do you think about making this function take hostnameOverride as a parameter?
Yeah, it should fix the issue. Will fix it tonight.


Question: detect-project local kubectl="${KUBE_ROOT}/cluster/kubectl.sh" <del> "${kubectl}" delete pods heapster &> /dev/null || true "${kubectl}" delete pods influx-grafana &> /dev/null || true "${kubectl}" delete services influx-master &> /dev/null || true </del> <ins> local kubecfg="${KUBE_ROOT}/cluster/kubecfg.sh" "${kubecfg}" resize monitoring-influxGrafanaController 0 &> /dev/null || true "${kubecfg}" resize monitoring-heapsterController 0 &> /dev/null || true "${kubectl}" delete -f "${KUBE_ROOT}/cluster/addons/cluster-monitoring/" &> /dev/null || true </ins> if gcloud compute firewall-rules describe --project "${PROJECT}" monitoring-heapster &> /dev/null; then gcloud compute firewall-rules delete  --project "${PROJECT}"

examples/monitoring/influx-grafana- and friends NEED to be replaced by a replication controller ASAP. As it stands they are bad examples. Also, they should probably move to contrib/monitoring to match contrib/logging and contrib/dns


Question: result := c.client.client. Put(). <del> AbsPath(append(c.makeURLSegments(accessor.GetName()), subresources...)...). </del> <ins> AbsPath(append(c.makeURLSegments(name), subresources...)...). </ins> Body(outBytes). SpecificallyVersionedParams(&opts, dynamicParameterCodec, versionV1). Do()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Providing an empty to the dynamic client's method results in a panic. What you expected to happen: An error should be returned on empty . How to reproduce it (as minimally and precisely as possible): Call with Anything else we need to know?: The expects a nonempty to be provided, but does not validate this. If an empty name is provided, this method will panic because it expects an , but the response will actually be an . Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: client-go 10.0.0
/sig api-machinery


Question: // TODO(resouer): think about lazily initialize this set invalidPredicates := sets.NewString() <del> oldTaints, oldErr := helper.GetTaintsFromNodeAnnotations(oldNode.GetAnnotations()) if oldErr != nil { glog.Errorf("Failed to get taints from old node annotation for equivalence cache") } newTaints, newErr := helper.GetTaintsFromNodeAnnotations(newNode.GetAnnotations()) if newErr != nil { glog.Errorf("Failed to get taints from new node annotation for equivalence cache") } </del> if !reflect.DeepEqual(oldNode.Status.Allocatable, newNode.Status.Allocatable) { invalidPredicates.Insert("GeneralPredicates") // "PodFitsResources" }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: alpha suite has been failing in the BeforeSuite since 9/22 Did we recently enable some alpha taints feature?
It may be related to cc
/assign
, do you know how to trigger this test in CI, e.g. ?
I don't think there's a way to launch a ci job from here.
This seems to be the same failure as . You can trigger the GKE GPU tests on PRs using . GKE GPU tests use alpha features.
sure, let me add some instrument code and trigger test by ci :).


Question: return &Kubelet{ hostname:              hn, dockerClient:          dc, <del> cadvisorClient:        cc, </del> etcdClient:            ec, rootDirectory:         rd, resyncInterval:        ri,

So today, cadvisor became wedged and just stopped paying attention to its socket. Not kubelet's fault. But when I would launch all I would get would be: It took me a bit of trolling through netstat and lsof to figure out that the problem was cadvisor being completely locked up. kill -9 cadvisor and everything started working well. I wonder if we should emit more messages during startup about what operation we are about to attempt, not just the results of operations, as I was blaming docker for a while....
kubelet shouldn't block on this. I'll take a look.
Any ideas what was locking cAdvisor? and was it just not responding to requests?
I don't have any idea why it was locked. If I find another one locked I'll hang onto it to debug. Journal had: obviously the 'kill' came from me. It would not respond to systemctl cadvisor stop. netstat looking at port 4194 showed me At which point I realized cadvisor wasn't handling its sockets right since so many long dead kubelet sockets hadn't been cleaned up, so I kill -9'd it.
I cut some interesting info from the cadvisor journal log (trying to cut hostnames) You can see the last message about failed to update stats for container ... happened Oct 08 01:06:01. It was then silent until I killed it Oct 08 19:20:13, even though I'm pretty sure dnf was still being launched every hour on the hour. So it was wedged for about 18 hours.
What kernel is this on? We've seen there is a kernel bug fixed in 3.12+ that sometimes hangs us forever on exit. We're not quite sure how it gets triggered...
16.3


Question: } }) <del> It("should invoke init containers on a RestartAlways pod", func() { </del> <ins> /* Release: v1.12 Testname: init-container-starts-app-restartalways-pod Description: Ensure that all InitContainers are started and all containers in pod started and at least one container is still running or is in the process of being restarted when Pod has restart policy as RestartAlways. */ framework.ConformanceIt("should invoke init containers on a RestartAlways pod", func() { </ins> By("creating the pod") name := "pod-init-" + string(uuid.NewUUID()) value := strconv.Itoa(time.Now().Nanosecond())

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Evaluating e2e tests based on available options which are covering important aspects with user scenarios and eligible to promote for conformance. What you expected to happen: These 4 e2e tests evaluate different and based on . [NodeConformance] should invoke init containers on a RestartNever pod [NodeConformance] should invoke init containers on a RestartAlways pod [NodeConformance] should not start app containers if init containers fail on a RestartAlways pod [NodeConformance] should not start app containers and fail the pod if init containers fail on a RestartNever pod Anything else we need to know?: comment, Are these tests good to promote for conformance or do we need to look on other aspects as well ? are observed api endpoint hits during tests execution. (API coverage improvement) - InitContainer [NodeConformance] should invoke init containers on a RestartNever pod InitContainer [NodeConformance] should invoke init containers on a RestartAlways pod InitContainer [NodeConformance] should not start app containers if init containers fail on a RestartAlways pod InitContainer [NodeConformance] should not start app containers and fail the pod if init containers fail on a RestartNever pod /area conformance cc
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Go ahead and open a PR to ask this question. Yes == PR is merged, no == PR is closed (or amended to doc the reason these tests can't be promoted)


Question: return true }, 1*time.Minute, 1*time.Second).Should(gomega.BeTrue()) }) <ins> ginkgo.It("can read and write file to remote SMB folder", func() { defer ginkgo.GinkgoRecover() ginkgo.By("finding the worker node that fulfills this test's assumptions") nodes := findPreconfiguredGmsaNodes(f.ClientSet) if len(nodes) != 1 { e2eskipper.Skipf("Expected to find exactly one node with the %q label, found %d", gmsaFullNodeLabel, len(nodes)) } node := nodes[0] ginkgo.By("retrieving the contents of the GMSACredentialSpec custom resource manifest from the node") crdManifestContents := retrieveCRDManifestFileContents(f, node) ginkgo.By("downloading the GMSA webhook deploy script") deployScriptPath, err := downloadFile(gmsaWebhookDeployScriptURL) defer func() { os.Remove(deployScriptPath) }() if err != nil { framework.Failf(err.Error()) } ginkgo.By("deploying the GMSA webhook") webhookCleanUp, err := deployGmsaWebhook(f, deployScriptPath) defer webhookCleanUp() if err != nil { framework.Failf(err.Error()) } ginkgo.By("creating the GMSA custom resource") customResourceCleanup, err := createGmsaCustomResource(f.Namespace.Name, crdManifestContents) defer customResourceCleanup() if err != nil { framework.Failf(err.Error()) } ginkgo.By("creating an RBAC role to grant use access to that GMSA resource") rbacRoleName, rbacRoleCleanup, err := createRBACRoleForGmsa(f) defer rbacRoleCleanup() if err != nil { framework.Failf(err.Error()) } ginkgo.By("creating a service account") serviceAccountName := createServiceAccount(f) ginkgo.By("binding the RBAC role to the service account") bindRBACRoleToServiceAccount(f, serviceAccountName, rbacRoleName) ginkgo.By("creating a pod using the GMSA cred spec") podName := createPodWithGmsa(f, serviceAccountName) ginkgo.By("getting the ip of GMSA domain") gmsaDomainIP := getGmsaDomainIP(f, podName) ginkgo.By("checking that file can be read and write from the remote folder successfully") filePath := fmt.Sprintf("%s%swrite-test-%s.txt", gmsaDomainIP, gmsaSharedFolder, string(uuid.NewUUID())[0:4]) gomega.Eventually(func() bool { // The filePath is a remote folder, do not change the format of it _, _ = runKubectlExecInNamespace(f.Namespace.Name, podName, "--", "powershell.exe", "-Command", "echo 'This is a test file.' > "+filePath) output, err := runKubectlExecInNamespace(f.Namespace.Name, podName, "powershell.exe", "--", "cat", filePath) if err != nil { framework.Logf("unable to get file from AD server: %s", err) return false } return strings.Contains(output, "This is a test file.") }, 1*time.Minute, 1*time.Second).Should(gomega.BeTrue()) }) </ins> }) })

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --There was a regression in containerd that required a fix for gmsa working with UNC paths. See the fix at We should have a test that validates this functionality. See for set up and place to add this test. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted /priority important-long-term /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --#### What happened: There was a regression in containerd that required a fix for gmsa working with UNC paths. See the fix at We should have a test that validates this functionality. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted /priority important-long-term /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : There was a regression in containerd that required a fix for gmsa working with UNC paths. See the fix at We should have a test that validates this functionality. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g.


Question: return true }, 1*time.Minute, 1*time.Second).Should(gomega.BeTrue()) }) <ins> ginkgo.It("can read and write file to remote SMB folder", func() { defer ginkgo.GinkgoRecover() ginkgo.By("finding the worker node that fulfills this test's assumptions") nodes := findPreconfiguredGmsaNodes(f.ClientSet) if len(nodes) != 1 { e2eskipper.Skipf("Expected to find exactly one node with the %q label, found %d", gmsaFullNodeLabel, len(nodes)) } node := nodes[0] ginkgo.By("retrieving the contents of the GMSACredentialSpec custom resource manifest from the node") crdManifestContents := retrieveCRDManifestFileContents(f, node) ginkgo.By("downloading the GMSA webhook deploy script") deployScriptPath, err := downloadFile(gmsaWebhookDeployScriptURL) defer func() { os.Remove(deployScriptPath) }() if err != nil { framework.Failf(err.Error()) } ginkgo.By("deploying the GMSA webhook") webhookCleanUp, err := deployGmsaWebhook(f, deployScriptPath) defer webhookCleanUp() if err != nil { framework.Failf(err.Error()) } ginkgo.By("creating the GMSA custom resource") customResourceCleanup, err := createGmsaCustomResource(f.Namespace.Name, crdManifestContents) defer customResourceCleanup() if err != nil { framework.Failf(err.Error()) } ginkgo.By("creating an RBAC role to grant use access to that GMSA resource") rbacRoleName, rbacRoleCleanup, err := createRBACRoleForGmsa(f) defer rbacRoleCleanup() if err != nil { framework.Failf(err.Error()) } ginkgo.By("creating a service account") serviceAccountName := createServiceAccount(f) ginkgo.By("binding the RBAC role to the service account") bindRBACRoleToServiceAccount(f, serviceAccountName, rbacRoleName) ginkgo.By("creating a pod using the GMSA cred spec") podName := createPodWithGmsa(f, serviceAccountName) ginkgo.By("getting the ip of GMSA domain") gmsaDomainIP := getGmsaDomainIP(f, podName) ginkgo.By("checking that file can be read and write from the remote folder successfully") filePath := fmt.Sprintf("%s%swrite-test-%s.txt", gmsaDomainIP, gmsaSharedFolder, string(uuid.NewUUID())[0:4]) gomega.Eventually(func() bool { // The filePath is a remote folder, do not change the format of it _, _ = runKubectlExecInNamespace(f.Namespace.Name, podName, "--", "powershell.exe", "-Command", "echo 'This is a test file.' > "+filePath) output, err := runKubectlExecInNamespace(f.Namespace.Name, podName, "powershell.exe", "--", "cat", filePath) if err != nil { framework.Logf("unable to get file from AD server: %s", err) return false } return strings.Contains(output, "This is a test file.") }, 1*time.Minute, 1*time.Second).Should(gomega.BeTrue()) }) </ins> }) })

): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted /priority important-long-term /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-longterm
/assign
if you have questions feel free to reachout on slack in sig-windows
Will do, thanks!
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


Question: return true }, 1*time.Minute, 1*time.Second).Should(gomega.BeTrue()) }) <ins> ginkgo.It("can read and write file to remote SMB folder", func() { defer ginkgo.GinkgoRecover() ginkgo.By("finding the worker node that fulfills this test's assumptions") nodes := findPreconfiguredGmsaNodes(f.ClientSet) if len(nodes) != 1 { e2eskipper.Skipf("Expected to find exactly one node with the %q label, found %d", gmsaFullNodeLabel, len(nodes)) } node := nodes[0] ginkgo.By("retrieving the contents of the GMSACredentialSpec custom resource manifest from the node") crdManifestContents := retrieveCRDManifestFileContents(f, node) ginkgo.By("downloading the GMSA webhook deploy script") deployScriptPath, err := downloadFile(gmsaWebhookDeployScriptURL) defer func() { os.Remove(deployScriptPath) }() if err != nil { framework.Failf(err.Error()) } ginkgo.By("deploying the GMSA webhook") webhookCleanUp, err := deployGmsaWebhook(f, deployScriptPath) defer webhookCleanUp() if err != nil { framework.Failf(err.Error()) } ginkgo.By("creating the GMSA custom resource") customResourceCleanup, err := createGmsaCustomResource(f.Namespace.Name, crdManifestContents) defer customResourceCleanup() if err != nil { framework.Failf(err.Error()) } ginkgo.By("creating an RBAC role to grant use access to that GMSA resource") rbacRoleName, rbacRoleCleanup, err := createRBACRoleForGmsa(f) defer rbacRoleCleanup() if err != nil { framework.Failf(err.Error()) } ginkgo.By("creating a service account") serviceAccountName := createServiceAccount(f) ginkgo.By("binding the RBAC role to the service account") bindRBACRoleToServiceAccount(f, serviceAccountName, rbacRoleName) ginkgo.By("creating a pod using the GMSA cred spec") podName := createPodWithGmsa(f, serviceAccountName) ginkgo.By("getting the ip of GMSA domain") gmsaDomainIP := getGmsaDomainIP(f, podName) ginkgo.By("checking that file can be read and write from the remote folder successfully") filePath := fmt.Sprintf("%s%swrite-test-%s.txt", gmsaDomainIP, gmsaSharedFolder, string(uuid.NewUUID())[0:4]) gomega.Eventually(func() bool { // The filePath is a remote folder, do not change the format of it _, _ = runKubectlExecInNamespace(f.Namespace.Name, podName, "--", "powershell.exe", "-Command", "echo 'This is a test file.' > "+filePath) output, err := runKubectlExecInNamespace(f.Namespace.Name, podName, "powershell.exe", "--", "cat", filePath) if err != nil { framework.Logf("unable to get file from AD server: %s", err) return false } return strings.Contains(output, "This is a test file.") }, 1*time.Minute, 1*time.Second).Should(gomega.BeTrue()) }) </ins> }) })

/lifecycle rotten [1]:
/remove-lifecycle rotten
/remove-lifecycle rotten
/assign
I have thought about this a bit and think it might be easiest to just expose a share on the domain controller. This does push more requirements into the configuration that needs to happen outside of the tests but it will make the tests a lot cleaner IMO.
/assign
GitHub didn't allow me to assign the following users: iXinqi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "//pkg/runtime:go_default_library", "//pkg/types:go_default_library", "//pkg/util/errors:go_default_library", <del> "//pkg/util/hash:go_default_library", </del> "//pkg/util/metrics:go_default_library", "//pkg/util/runtime:go_default_library", "//pkg/util/wait:go_default_library",

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux What happened: Deployed a scheduled job from this manifest: After some time the jobs are no longer executed: What you expected to happen: Jobs are executed as defined in the manifest. How to reproduce it (as minimally and precisely as possible): Deploy a scheduled job and wait for some time.
Some logs from the controller manager:
Ping. More than 2 weeks and no action at all? Anything else i can do?
I'm also encountering this using both and as concurrency policies, fwiw.
I'm hitting this as well. Kube
Do the jobs start to reschedule if you delete old one? Try running . cc
From kube-controller-manager logs: The job is scheduled for 1:00. I think the "Multiple unmet start times" message is a clue of where things start going downhill. Also, notice how at 1:00 the controller wants to create job (for a whole day, it had failed on ).
Deleting (all) old jobs / pods fixes the problem temporarily. After some time the scheduled jobs face the same issue again.
There's a fix () that I've submitted against master to fix the Replace strategy, I'll mark it for inclusion in 1.4.
The other problem that I see here is this: It is possible that our is creating duplicates. I'll try to investigate that a bit more.
sorry for the delay in response time, I must have missed that while going through issues. Next time you're experiencing problems/have a question wrt Jobs/ScheduledJobs, just ping me directly in the issue.
No problem and I will. Thanks!
I'm still hitting this on
Fix in flight , I'll make sure to cherry-pick it back to 1.4
does this ship in to latest kubernetes release?
yes, this should be present in 1.5, 1.4 cherry-pick hasn't been merged yet.
so this issue is still present in v1.4.6?


Question: "//pkg/runtime:go_default_library", "//pkg/types:go_default_library", "//pkg/util/errors:go_default_library", <del> "//pkg/util/hash:go_default_library", </del> "//pkg/util/metrics:go_default_library", "//pkg/util/runtime:go_default_library", "//pkg/util/wait:go_default_library",

yes - it is still present there. My fix was merged to 1.4 yesterday. You should wait for the next 1.4.x release.
thanks!
which according to should happen Dec 8th.
any news on the release?
See !msg/kubernetes-dev/o0At2-4KKQ8/_kPVM1XyDAAJ
so v1.4.7 will be released on that same day?
Look like should be available later today: !msg/kubernetes-dev/00lJpT9pxDc/YZfDzCxMDQAJ


Question: // Assume info is ready to process podStatus.Phase = getPhase(spec, info) <ins> podStatus.Info = api.PodInfo{} </ins> for _, c := range spec.Containers { containerStatus := info[c.Name] containerStatus.Ready = kl.readiness.IsReady(containerStatus) <del> info[c.Name] = containerStatus </del> <ins> podStatus.Info[c.Name] = containerStatus </ins> } <del> podStatus.Conditions = append(podStatus.Conditions, getPodReadyCondition(spec, info)...) </del> <ins> podStatus.Conditions = append(podStatus.Conditions, getPodReadyCondition(spec, podStatus.Info)...) </ins> netContainerInfo, found := info[dockertools.PodInfraContainerName] if found { podStatus.PodIP = netContainerInfo.PodIP } <del> // TODO(dchen1107): Change Info to list from map podStatus.Info = info </del> return podStatus, nil }

Kubelet now only reports ContainerStatus. PodStatus / PodCondition is observed at master side based on a set of ContainerStatus. There is no other way to get PodIP except net container for master side. This is very confusion to the end user, , , ... Now we are finalizing on v1beta3 API, including PodStatus & ContainerStatus. We can make net container completely invisible to the end users by finishing the following steps: 1) Expose network related errors at API level 2) Generate PodStaus directly from Kubelet
3) once PodStatus generated by kubelet, PodIP can be removed from ContainerStatus.
I think reporting PodStatus rather than just container status is most important. How hard is that?
Used to be hard since there are many dependencies when I first report Status back. Now it is totally feasible. I already started to write PRs for this. :-)
One more PR required to close this one.


Question: // Test that a handler associated with an APIService gets pinged after the // APIService has been marked as dirty func TestDirty(t *testing.T) { <del> pinged := false </del> <ins> var pinged atomic.Bool service := discoveryendpoint.NewResourceManager() aggregatedResourceManager := discoveryendpoint.NewResourceManager() aggregatedManager := newDiscoveryManager(aggregatedResourceManager) aggregatedManager.AddAPIService(&apiregistrationv1.APIService{ ObjectMeta: metav1.ObjectMeta{ Name: "v1.stable.example.com", }, Spec: apiregistrationv1.APIServiceSpec{ Group:   "stable.example.com", Version: "v1", Service: &apiregistrationv1.ServiceReference{ Name: "test-service", }, }, }, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { pinged.Store(true) service.ServeHTTP(w, r) })) testCtx, cancel := context.WithCancel(context.Background()) defer cancel() go aggregatedManager.Run(testCtx.Done()) require.True(t, waitForQueueComplete(testCtx.Done(), aggregatedManager)) // immediately check for ping, since Run() should block for local services if !pinged.Load() { t.Errorf("service handler never pinged") } } // Shows that waitForQueueComplete also waits for syncing to // complete by artificially making the sync handler take a long time func TestWaitForSync(t *testing.T) { pinged := atomic.Bool{} </ins> service := discoveryendpoint.NewResourceManager() aggregatedResourceManager := discoveryendpoint.NewResourceManager()

pull-kubernetes-unit sometimes fails with data race. TestDirty I think the test was 3 months ago with the flaking state. There are two issues. First, updating and reading a bool-type variable is not thread-safe. and there is also another problem checking the state before completely syncing it. No response /sig api-machinery
/assign
/triage accepted


Question: <ins> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package openapi import ( "context" "encoding/json" "strings" "testing" "time" apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset" "k8s.io/apiextensions-apiserver/test/integration/fixtures" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/dynamic" kubernetes "k8s.io/client-go/kubernetes" "k8s.io/kube-openapi/pkg/validation/spec" apiservertesting "k8s.io/kubernetes/cmd/kube-apiserver/app/testing" "k8s.io/kubernetes/test/integration/framework" ) func TestOpenAPIV2CRDMergeNoDuplicateTypes(t *testing.T) { server, err := apiservertesting.StartTestServer(t, apiservertesting.NewDefaultTestServerOptions(), nil, framework.SharedEtcd()) if err != nil { t.Fatal(err) } defer server.TearDownFn() config := server.ClientConfig apiExtensionClient, err := clientset.NewForConfig(config) if err != nil { t.Fatal(err) } dynamicClient, err := dynamic.NewForConfig(config) if err != nil { t.Fatal(err) } clientset, err := kubernetes.NewForConfig(config) if err != nil { t.Fatal(err) } foo := &apiextensionsv1.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{ Name: "foosubs.cr.bar.com", }, Spec: apiextensionsv1.CustomResourceDefinitionSpec{ Group: "cr.bar.com", Scope: apiextensionsv1.NamespaceScoped, Names: apiextensionsv1.CustomResourceDefinitionNames{ Plural: "foosubs", Kind:   "FooSub", }, Versions: []apiextensionsv1.CustomResourceDefinitionVersion{ { Name:    "v1", Served:  true, Storage: true, Schema: &apiextensionsv1.CustomResourceValidation{ OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensionsv1.JSONSchemaProps{ "spec": { Type: "object", Properties: map[string]apiextensionsv1.JSONSchemaProps{ "replicas": { Type: "integer", }, }, }, "status": { Type: "object", Properties: map[string]apiextensionsv1.JSONSchemaProps{ "replicas": { Type: "integer", }, }, }, }, }, }, Subresources: &apiextensionsv1.CustomResourceSubresources{ Status: &apiextensionsv1.CustomResourceSubresourceStatus{}, Scale: &apiextensionsv1.CustomResourceSubresourceScale{ SpecReplicasPath:   ".spec.replicas", StatusReplicasPath: ".status.replicas", }, }, }, }, }, } baz := &apiextensionsv1.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{ Name: "bazsubs.cr.bar.com", }, Spec: apiextensionsv1.CustomResourceDefinitionSpec{ Group: "cr.bar.com", Scope: apiextensionsv1.NamespaceScoped, Names: apiextensionsv1.CustomResourceDefinitionNames{ Plural: "bazsubs", Kind:   "BazSub", }, Versions: []apiextensionsv1.CustomResourceDefinitionVersion{ { Name:    "v1", Served:  true, Storage: true, Schema: &apiextensionsv1.CustomResourceValidation{ OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensionsv1.JSONSchemaProps{ "spec": { Type: "object", Properties: map[string]apiextensionsv1.JSONSchemaProps{ "replicas": { Type: "integer", }, }, }, "status": { Type: "object", Properties: map[string]apiextensionsv1.JSONSchemaProps{ "replicas": { Type: "integer", }, }, }, }, }, }, Subresources: &apiextensionsv1.CustomResourceSubresources{ Status: &apiextensionsv1.CustomResourceSubresourceStatus{}, Scale: &apiextensionsv1.CustomResourceSubresourceScale{ SpecReplicasPath:   ".spec.replicas", StatusReplicasPath: ".status.replicas", }, }, }, }, }, } _, err = fixtures.CreateNewV1CustomResourceDefinition(foo, apiExtensionClient, dynamicClient) if err != nil { t.Fatal(err) } _, err = fixtures.CreateNewV1CustomResourceDefinition(baz, apiExtensionClient, dynamicClient) if err != nil { t.Fatal(err) } var openAPISpec spec.Swagger // Poll for the OpenAPI to be updated with the new CRD wait.Poll(time.Second*1, wait.ForeverTestTimeout, func() (bool, error) { jsonData, err := clientset.RESTClient().Get().AbsPath("/openapi/v2").Do(context.TODO()).Raw() if err != nil { t.Fatal(err) } openAPISpec = spec.Swagger{} err = json.Unmarshal(jsonData, &openAPISpec) if err != nil { t.Fatal(err) } for schemaName := range openAPISpec.Definitions { if strings.HasPrefix(schemaName, "com.bar.cr.v1.BazSub") { return true, nil } } return false, nil }) for schemaName := range openAPISpec.Definitions { if strings.HasSuffix(schemaName, "_v2") { t.Errorf("Error: Expected no _v2 types, got %s", schemaName) } } } </ins>

Noticed while reproducing with client-side validation: Steps to reproduce: published openapi only contains expected metadata definitions: a CRD, e.g. simple- at published openapi: where the _v2 definition is referenced: For some reason, the same server publishing openapi for built-in types and custom resource definitions thinks the openapi definition differs. Reproduced against at least the following branches: master release-1.23 release-1.22 release-1.21 In addition to fixing this to make coherently refer to ObjectMeta definitions from built-in and custom types, we need a test for this scenario to ensure it doesn't creep back in.
/sig api-machinery /priority important-longterm
cc - this is the sort of thing I was remembering which prompted the question about merging openapi internally () This needs to be resolved and test coverage improved here ahead of making v2-facing openapi changes
is there a rule that we can validate and enforce here? i.e. are there cases where it's normal to have two metadata? Probably in the case where an extension apiserver has a different version of kubernetes that has a slightly different metadata. I wish we could catch somehow that this doesn't happen.
In a server with no aggregated API servers, we should never have multiple definitions for ObjectMeta (CRDs and built-in types should reference the same ObjectMeta)
/triage accepted
/cc
This is fixed by on master. For 1.24, 1.23, 1.22 and 1.21, cherry picks have been created: 1.24: 1.23: 1.22: 1.21:
also need it on 1.24?
Updated the list above.


Question: ### Setup <del> By default, the Vagrant setup will create a single kubernetes-master and 3 kubernetes-minions.  You can control the number of minions that are instantiated via an environment variable on your host machine.  If you plan to work with replicas, we strongly encourage you to work with enough minions to satisfy your largest intended replica size.  If you do not plan to work with replicas, you can save some system resources by running with a single minion. ``` export KUBERNETES_NUM_MINIONS=3 ``` To start your local cluster, open a terminal window and run: </del> <ins> By default, the Vagrant setup will create a single kubernetes-master and 3 kubernetes-minions. Each VM will take 512 MB, so make sure you have at least 2 GB of free memory. To start your local cluster, open a shell and run: </ins> ``` cd kubernetes

Which jobs are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /priority important-soon /cc /cc /milestone v1.18 /sig release
/assign
This is failing because a 1.18 release for CRI tools doesn't exist yet. I can tweak the test job arguments to skip this later.
(As a note, this job is important, but not ) /remove-priority critical-urgent /priority important-soon
Do we need to remove the job from dashboard?
-- No. I'll work on fixing the job later today.
I've started to investigate, but I haven't yet reproduced the new error locally: This is likely related to a recent refactor of the package that handles interpreting deb/rpm package versions. As is not currently in use, I'm moving this to the milestone. /milestone v1.19
the version issue is fixed via The test is now failing because it tries to download cri-tools v1.18.0: which I think is related to this code part:
Updating testgrid link here: Any updates here /retitle [Flaky Test] build-packages-debs (ci-release-build-packages-debs) /kind flaky-test
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Updating testgrid link here: /kind flaky-test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake
I was assuming that is working on this issue. I can jump in if you want and do not find the time to work on it. :)
/assign /unassign Sascha -- Can you take this? I thought I was going to have time to look at this, but it's been a while.
Sure, the reason for the failing test seems to have changed, I'm on it.
Fix is incoming in


Question: } return buf.String(), nil } <ins> type conditionalTransport struct { oauthTransport *oauth2.Transport } func (t *conditionalTransport) RoundTrip(req *http.Request) (*http.Response, error) { if len(req.Header.Get("Authorization")) != 0 { return t.oauthTransport.Base.RoundTrip(req) } return t.oauthTransport.RoundTrip(req) } </ins>

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubectl token BUG REPORT Kubernetes version (use ): kubectl version Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): macOS Sierra 10.12.4 Kernel (e.g. ): uname -a Darwin jacobsimpson- 16.5.0 Darwin Kernel Version 16.5.0: Fri Mar 3 16:52:33 PST 2017; root:xnu-3789.51.2~3/RELEASEX8664 x8664 Install tools: Others: What happened: I set up a second user, then I use to authenticate as the second user. The parameter is ignored and the authentication token from the file is used instead, so I'm authenticated as the user configured in the config file, not as the user set on the command line parameters. I further verified this by commenting out the section in the config file with the authentication token and then the parameter starts to work as expected. What you expected to happen: I expect the command line parameters, when present, to override the config file settings. How to reproduce it (as minimally and precisely as possible): At this point, the error message should be indicating kubectl tried to authenticate with the account. Something like this: Now edit , search for the section for your cluster and comment out the whole section for the test cluster. Now it should be possible to run these two commands. The one using the user_token will fail with the error message indicating the user name.


Question: } return buf.String(), nil } <ins> type conditionalTransport struct { oauthTransport *oauth2.Transport } func (t *conditionalTransport) RoundTrip(req *http.Request) (*http.Response, error) { if len(req.Header.Get("Authorization")) != 0 { return t.oauthTransport.Base.RoundTrip(req) } return t.oauthTransport.RoundTrip(req) } </ins>

Anything else we need to know:
The token arg on the command line overrides the bearer token field in the config file, but I'm not sure the precedence of the token arg and the auth provider in the config file. cc
All auth providers currently override the "Authorization" header even if it's set. Should be an easy-ish check to add. Long term we probably want to have the auth provider stop wrapping the transport and just act like a token source , that way some central code would handle the HTTP headers and we wouldn't have to enforce this on a per plugin basis.
So the tokens from command line should be used, right? if so I can send a pr to fix this.
the client auth plugins shouldn't override the "Authorization" headers. Yeah feel free to send a PR.
Is this a regression or why 1.7?
bug is pretty self contained in the GCP client auth plugin and the fix is short Not strictly a regression, but something that popped up over this cycle as a bug impacting users. Is that reason enough to get it in during the 1.7 freeze?


Question: <del> #!/usr/bin/env bash # Copyright 2018 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. KUBE_ROOT=$(dirname "${BASH_SOURCE[0]}")/../../.. # Creates a new kube-spawn cluster function create-clusters { # shellcheck disable=SC2154	# Color defined in sourced script echo -e "${color_yellow}CHECKING CLUSTERS${color_norm}" if ibmcloud ks clusters | grep -Fq 'deleting'; then echo -n "Deleting old clusters" fi while ibmcloud ks clusters | grep -Fq 'deleting' do echo -n "." sleep 10 done echo "" ibmcloud ks region-set us-east >/dev/null ibmcloud ks vlans wdc06 >/dev/null PRIVLAN=$(ibmcloud ks vlans wdc06 --json | jq '. | .[] | select(.type == "private") | .id' | sed -e "s/"//g") PUBVLAN=$(ibmcloud ks vlans wdc06 --json | jq '. | .[] | select(.type == "public") | .id' | sed -e "s/"//g") if ! ibmcloud ks clusters | grep -Fq 'kubeSpawnTester'; then echo "Creating spawning cluster" # make number and spec of node workers configurable # otherwise it can't afford tests like kubemark-5000 # TODO: dynamically adjust the number and spec ibmcloud ks cluster-create --location "${CLUSTER_LOCATION}" --public-vlan "${PUBVLAN}" --private-vlan "${PRIVLAN}" --workers "${NUM_NODES:-2}" --machine-type "${NODE_SIZE}" --name kubeSpawnTester fi if ! ibmcloud ks clusters | grep -Fq 'kubeMasterTester'; then echo "Creating master cluster" # if we can't make it a bare master (workers = 0) # then make workers = 1 with the smallest machine spec ibmcloud ks cluster-create --location "${CLUSTER_LOCATION}" --public-vlan "${PUBVLAN}" --private-vlan "${PRIVLAN}" --workers 1 --machine-type u2c.2x4 --name kubeMasterTester fi push-image if ! ibmcloud ks clusters | grep 'kubeSpawnTester' | grep -Fq 'normal'; then # shellcheck disable=SC2154 # Color defined in sourced script echo -e "${color_cyan}Warning: new clusters may take up to 60 minutes to be ready${color_norm}" echo -n "Clusters loading" fi while ! ibmcloud ks clusters | grep 'kubeSpawnTester' | grep -Fq 'normal' do echo -n "." sleep 5 done while ! ibmcloud ks clusters | grep 'kubeMasterTester' | grep -Fq 'normal' do echo -n "." sleep 5 done echo -e "${color_yellow}CLUSTER CREATION COMPLETE${color_norm}" } # Builds and pushes image to registry function push-image { if [[ "${ISBUILD}" = "y" ]]; then if ! ibmcloud cr namespaces | grep -Fq "${KUBE_NAMESPACE}"; then echo "Creating registry namespace" ibmcloud cr namespace-add "${KUBE_NAMESPACE}" echo "ibmcloud cr namespace-rm ${KUBE_NAMESPACE}" >> "${RESOURCE_DIRECTORY}/iks-namespacelist.sh" fi docker build -t "${KUBEMARK_INIT_TAG}" "${KUBEMARK_IMAGE_LOCATION}" docker tag "${KUBEMARK_INIT_TAG}" "${KUBEMARK_IMAGE_REGISTRY}${KUBE_NAMESPACE}/${PROJECT}:${KUBEMARK_IMAGE_TAG}" docker push "${KUBEMARK_IMAGE_REGISTRY}${KUBE_NAMESPACE}/${PROJECT}:${KUBEMARK_IMAGE_TAG}" echo "Image pushed" else KUBEMARK_IMAGE_REGISTRY="${KUBEMARK_IMAGE_REGISTRY:-brandondr96}" KUBE_NAMESPACE="" fi } # Allow user to use existing clusters if desired function choose-clusters { echo -n -e "Do you want to use custom clusters? [y/N]${color_cyan}>${color_norm} " read -r USE_EXISTING if [[ "${USE_EXISTING}" = "y" ]]; then echo -e "${color_yellow}Enter path for desired hollow-node spawning cluster kubeconfig file:${color_norm}" read -r CUSTOM_SPAWN_CONFIG echo -e "${color_yellow}Enter path for desired hollow-node hosting cluster kubeconfig file:${color_norm}" read -r CUSTOM_MASTER_CONFIG push-image elif [[ "${USE_EXISTING}" = "N" ]]; then create-clusters else # shellcheck disable=SC2154 # Color defined in sourced script echo -e "${color_red}Invalid response, please try again:${color_norm}" choose-clusters fi } # Ensure secrets are correctly set function set-registry-secrets { spawn-config kubectl get secret bluemix-default-secret-regional -o yaml | sed 's/default/kubemark/g' | kubectl -n kubemark create -f - kubectl patch serviceaccount -n kubemark default -p '{"imagePullSecrets": [{"name": "bluemix-kubemark-secret"}]}' kubectl -n kubemark get serviceaccounts default -o json | jq 'del(.metadata.resourceVersion)' | jq 'setpath(["imagePullSecrets"];[{"name":"bluemix-kubemark-secret-regional"}])' | kubectl -n kubemark replace serviceaccount default -f - } # Sets the hollow-node master # Exported variables: #   MASTER_IP - IP Address of the Kubemark master function set-hollow-master { echo -e "${color_yellow}CONFIGURING MASTER${color_norm}" master-config MASTER_IP=$(grep server "$KUBECONFIG" | awk -F "/" '{print $3}') export MASTER_IP } # Set up master cluster environment # Exported variables: #   KUBECONFIG - Overrides default kube config for the purpose of setting up the Kubemark master components. function master-config { if [[ "${USE_EXISTING}" = "y" ]]; then export KUBECONFIG=${CUSTOM_MASTER_CONFIG} else eval "$(ibmcloud ks cluster-config kubeMasterTester --admin | grep export)" fi } # Set up spawn cluster environment # Exported variables: #    KUBECONFIG - Overrides default kube config for the purpose of setting up the hollow-node cluster. function spawn-config { if [[ "${USE_EXISTING}" = "y" ]]; then export KUBECONFIG=${CUSTOM_SPAWN_CONFIG} else eval "$(ibmcloud ks cluster-config kubeSpawnTester --admin | grep export)" fi } # Deletes existing clusters function delete-clusters { echo "DELETING CLUSTERS" ibmcloud ks cluster-rm kubeSpawnTester ibmcloud ks cluster-rm kubeMasterTester while ! ibmcloud ks clusters | grep 'kubeSpawnTester' | grep -Fq 'deleting' do sleep 5 done while ! ibmcloud ks clusters | grep 'kubeMasterTester' | grep -Fq 'deleting' do sleep 5 done kubectl delete ns kubemark } # Login to cloud services function complete-login { echo -e "${color_yellow}LOGGING INTO CLOUD SERVICES${color_norm}" echo -n -e "Do you have a federated IBM cloud login? [y/N]${color_cyan}>${color_norm} " read -r ISFED if [[ "${ISFED}" = "y" ]]; then ibmcloud login --sso -a "${REGISTRY_LOGIN_URL}" elif [[ "${ISFED}" = "N" ]]; then ibmcloud login -a "${REGISTRY_LOGIN_URL}" else echo -e "${color_red}Invalid response, please try again:${color_norm}" complete-login fi ibmcloud cr login } # Generate values to fill the hollow-node configuration templates. # Exported variables: #   KUBECTL - The name or path to the kubernetes client binary. function generate-values { echo "Generating values" master-config KUBECTL=kubectl export KUBECTL KUBEMARK_DIRECTORY="${KUBE_ROOT}/test/kubemark" RESOURCE_DIRECTORY="${KUBEMARK_DIRECTORY}/resources" CONFIGPATH=${KUBECONFIG%/*} KUBELET_CERT_BASE64="${KUBELET_CERT_BASE64:-$(base64 "${CONFIGPATH}/admin.pem" | tr -d 'rn')}" KUBELET_KEY_BASE64="${KUBELET_KEY_BASE64:-$(base64 "${CONFIGPATH}/admin-key.pem" | tr -d 'rn')}" CA_CERT_BASE64="${CA_CERT_BASE64:-$( base64 "$(find "${CONFIGPATH}" -name "*ca*" | head -n 1)" | tr -d 'rn')}" } # Build image for kubemark function build-kubemark-image { echo -n -e "Do you want to build the kubemark image? [y/N]${color_cyan}>${color_norm} " read -r ISBUILD if [[ "${ISBUILD}" = "y" ]]; then echo -e "${color_yellow}BUILDING IMAGE${color_norm}" "${KUBE_ROOT}/build/run.sh" make kubemark cp "${KUBE_ROOT}/_output/dockerized/bin/linux/amd64/kubemark" "${KUBEMARK_IMAGE_LOCATION}" elif [[ "${ISBUILD}" = "N" ]]; then echo -n "" else echo -e "${color_red}Invalid response, please try again:${color_norm}" build-kubemark-image fi } # Clean up repository function clean-repo { echo -n -e "Do you want to remove build output and binary? [y/N]${color_cyan}>${color_norm} " read -r ISCLEAN if [[ "${ISCLEAN}" = "y" ]]; then echo -e "${color_yellow}CLEANING REPO${color_norm}" rm -rf "${KUBE_ROOT}/_output" rm -f "${KUBEMARK_IMAGE_LOCATION}/kubemark" elif [[ "${ISCLEAN}" = "N" ]]; then echo -n "" else echo -e "${color_red}Invalid response, please try again:${color_norm}" clean-repo fi } </del>

While trying to run with a valid ibmcloud account, the following error happened I also tried to run and got the same error. Looks like a compatibility issue. ibmcloud CLI shouldn't return error Run with a valid IBM cloud account. No response I am trying to build kubemark from source <details</details<detailsIBM cloud IKS </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:


Question: } func (cm *containerManagerImpl) GetAllocateResourcesPodAdmitHandler() lifecycle.PodAdmitHandler { <del> return nil </del> <ins> return &noopWindowsResourceAllocator{} </ins> } func (cm *containerManagerImpl) UpdateAllocatedDevices() {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Pods will not start on Windows nodes, even though nodes are ready and pods are scheduled to nodes. Kubelet logs are filled with this: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Build K8s from master, create any pod on Windows node. Anything else we need to know?: I strongly believe this commit is responsible: What happened is that now, with the commit mentioned above, a nil podAdmitHandler is registered, that when called will result in a panic. More specifically here: For Windows, returns : Environment: All K8s built from master. This affects all sig-windows master tests.
/sig windows
/sig node
/reopen Is this resolved? The jobs still failing because of this. ref Can you check?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The issue is fixed, you can look here: . I don't know what's happening with gce tests, but all other windows tests ( based on aks ) are working. PS: Maybe I should add aks job to master release informing as well :)
Since this affected the Kubelet starting, I think we can safely close this issue based on just aks test results.
Thanks, /close
Closing this issue. <detailsIn response to : Thanks, /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks
can you take a look at the Windows GCE test passes? It looks like they are still unhealthy and there are a few issues being tracked by the release team and


Question: }, } <del> // addOrUpdateAvoidPodOnNode adds avoidPods annotations to node, will override if it exists func addOrUpdateAvoidPodOnNode(c clientset.Interface, nodeName string, avoidPods v1.AvoidPods) { err := wait.PollImmediate(framework.Poll, framework.SingleCallTimeout, func() (bool, error) { node, err := c.CoreV1().Nodes().Get(context.TODO(), nodeName, metav1.GetOptions{}) if err != nil { return false, err } taintsData, err := json.Marshal(avoidPods) framework.ExpectNoError(err) if node.Annotations == nil { node.Annotations = make(map[string]string) } node.Annotations[v1.PreferAvoidPodsAnnotationKey] = string(taintsData) _, err = c.CoreV1().Nodes().Update(context.TODO(), node, metav1.UpdateOptions{}) if err != nil { if !apierrors.IsConflict(err) { framework.ExpectNoError(err) } else { framework.Logf("Conflict when trying to add/update avoidPods %v to %v with error %v", avoidPods, nodeName, err) return false, nil } } return true, nil }) framework.ExpectNoError(err) } // removeAvoidPodsOffNode removes AvoidPods annotations from the node. It does not fail if no such annotation exists. func removeAvoidPodsOffNode(c clientset.Interface, nodeName string) { err := wait.PollImmediate(framework.Poll, framework.SingleCallTimeout, func() (bool, error) { node, err := c.CoreV1().Nodes().Get(context.TODO(), nodeName, metav1.GetOptions{}) if err != nil { return false, err } if node.Annotations == nil { return true, nil } delete(node.Annotations, v1.PreferAvoidPodsAnnotationKey) _, err = c.CoreV1().Nodes().Update(context.TODO(), node, metav1.UpdateOptions{}) if err != nil { if !apierrors.IsConflict(err) { framework.ExpectNoError(err) } else { framework.Logf("Conflict when trying to remove avoidPods to %v", nodeName) return false, nil } } return true, nil }) framework.ExpectNoError(err) } </del> // nodesAreTooUtilized ensures that each node can support 2*crioMinMemLimit // We check for double because it needs to support at least the cri-o minimum // plus whatever delta between node usages (which could be up to or at least crioMinMemLimit)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Kubernetes e2e suite.[sig-scheduling] SchedulerPriorities [Serial] Pod should avoid nodes that have avoidPod annotation June 19th 2021 () Testgrid: https://k8s- Prow history: Looks like pods are scheduled where they shouldn't be? : The tests passes in maybe 20% of the builds, but it looks random, so I don't treat it as a flake. At the moment there are also sig-storage tests consistently failing in this job: , They don't look related though. /sig scheduling /kind failing-test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
try to have a look.
try to have a look.
/cc
This is expected. The feature (which was alpha) is now disabled by default. Users can still enable it by switching to a v1beta1 scheduler component config. Let's remove the test.


Question: glog.V(4).Infof("EnsureLoadBalancer(%v, %v, %v, %v, %v, %v, %v)", clusterName, apiService.Namespace, apiService.Name, apiService.Spec.LoadBalancerIP, apiService.Spec.Ports, hosts, apiService.Annotations) ports := apiService.Spec.Ports <del> if len(ports) > 1 { return nil, fmt.Errorf("multiple ports are not yet supported in openstack load balancers") } else if len(ports) == 0 { </del> <ins> if len(ports) == 0 { </ins> return nil, fmt.Errorf("no ports provided to openstack load balancer") } <del> // The service controller verified all the protocols match on the ports, just check and use the first one </del> <ins> // Check for TCP protocol on each port </ins> // TODO: Convert all error messages to use an event recorder <del> if ports[0].Protocol != api.ProtocolTCP { return nil, fmt.Errorf("Only TCP LoadBalancer is supported for openstack load balancers") </del> <ins> for _, port := range ports { if port.Protocol != api.ProtocolTCP { return nil, fmt.Errorf("Only TCP LoadBalancer is supported for openstack load balancers") } </ins> } affinity := api.ServiceAffinityNone //apiService.Spec.SessionAffinity

LBaaS implementation (v1 or v2) will not support services that require to load balancer more than one port.
Fixed for LBaaS v2 by I suspect LBaaS v1 can't support multiple ports (not sure if reusing the VIP across multiple loadbalancers is a universally supported thing). I also believe LBaaS v1 is also mostly gone in favour of v2 in deployments so trying to address this in v1 is extremely low priority unless someone steps forward to ask for it. So: I think we can close this bug now, but I lack the powers to do that directly.
thank you for the update.


Question: } pod.Spec.Priority = &priority <del> if utilfeature.DefaultFeatureGate.Enabled(features.NonPreemptingPriority) { </del> <ins> if p.nonPreemptingPriority { </ins> var corePolicy core.PreemptionPolicy if preemptionPolicy != nil { corePolicy = core.PreemptionPolicy(*preemptionPolicy)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the documentation at the , I tried enabling quota for a namespace other than but I was still not able to create a pod with priorityClass in my namespace with following error: Error from server (Forbidden): error when creating "": pods "memory-demo" is forbidden: pods with system-cluster-critical priorityClass is not permitted in my-namespace namespace What you expected to happen: Expected pod to be created with namespace where I allowed quota. I can see that both ResourceQuota and Priority Plugins are enabled from logs. I0408 17:15:08. ] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook. I0408 17:15:08. ] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. How to reproduce it (as minimally and precisely as possible): Follow the steps at Anything else we need to know?: I think, the issue is with priorityPlugin where we are rejecting instead of letting resourceQuota plugin to handle the admission when is enabled. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig scheduling /cc
Same happens to me. Correction, it does not allow to use the default permissions outside of kube-system
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
See also:
/remove-lifecycle stale
/assign Per offline discussion with
Same issue here.


Question: } pod.Spec.Priority = &priority <del> if utilfeature.DefaultFeatureGate.Enabled(features.NonPreemptingPriority) { </del> <ins> if p.nonPreemptingPriority { </ins> var corePolicy core.PreemptionPolicy if preemptionPolicy != nil { corePolicy = core.PreemptionPolicy(*preemptionPolicy)

We are using namespaces other than kube-system for system pods as it is getting too crowded, therefore we would like to run cluster-node-critical pods in other namespaces.
Waiting for this to be available myself. No reason my Node Exporter dataset has to run in the namespace instead of my namespace.


Question: master_ip | (Optional) The IP address that the apiserver will bind against node_ip | (Optional) The IP address to use to address this node minion_ip | (Optional) Mapped to the kubelet hostname_override, K8S TODO - change this name <ins> network_mode | (Optional) Networking model to use among nodes: *openvswitch* </ins> roles | (Required) 1. **kubernetes-master** means this machine is the master in the kubernetes cluster.  2. **kubernetes-pool** means this machine is a kubernetes-minion.  Depending on the role, the Salt scripts will provision different resources on the machine. These keys may be leveraged by the Salt sls files to branch behavior.

The salt configs assume that anything under "os_family = RedHat" will use the ovs mechanisms that are only set up for vagrant. This means that we can't use Fedora on GCE.
I can clean this up today.


Question: "k8s.io/kubernetes/pkg/kubelet/nodeshutdown/systemd" ) <ins> // lock is to prevent systemDbus from being modified in the case of concurrency. var lock sync.Mutex </ins> type fakeDbus struct { currentInhibitDelay        time.Duration overrideSystemInhibitDelay time.Duration

/kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
/sig node
I find the reason for it, but I don't know the solution is suitable, correct me if not.
/triage accepted /assign


Question: t.Fatal("ServerVersion should return expected error, returned different error instead") } } <ins> func TestFakingServerResourcesForGroupVersionWithError(t *testing.T) { expectedError := errors.New("an error occurred") fakeClient := fakeclientset.NewClientset() fakeClient.Discovery().(*fakediscovery.FakeDiscovery).PrependReactor("*", "*", func(action kubetesting.Action) (handled bool, ret runtime.Object, err error) { return true, nil, expectedError }) result, err := fakeClient.Discovery().ServerResourcesForGroupVersion("dummy.group.io/v1beta2") if result != nil { t.Errorf(`expect result to be nil but got "%v" instead`, result) } if !errors.Is(err, expectedError) { t.Errorf(`expect error to be "%v" but got "%v" instead`, expectedError, err) } } func TestFakingServerGroupsWithError(t *testing.T) { expectedError := errors.New("an error occurred") fakeClient := fakeclientset.NewClientset() fakeClient.Discovery().(*fakediscovery.FakeDiscovery).PrependReactor("*", "*", func(action kubetesting.Action) (handled bool, ret runtime.Object, err error) { return true, nil, expectedError }) result, err := fakeClient.Discovery().ServerGroups() if result != nil { t.Errorf(`expect result to be nil but got "%v" instead`, result) } if !errors.Is(err, expectedError) { t.Errorf(`expect error to be "%v" but got "%v" instead`, expectedError, err) } } func TestFakingServerGroupsAndResourcesWithError(t *testing.T) { expectedError := errors.New("an error occurred") fakeClient := fakeclientset.NewClientset() fakeClient.Discovery().(*fakediscovery.FakeDiscovery).PrependReactor("get", "resource", func(action kubetesting.Action) (handled bool, ret runtime.Object, err error) { return true, nil, expectedError }) _, _, err := fakeClient.Discovery().ServerGroupsAndResources() if !errors.Is(err, expectedError) { t.Errorf(`expect error to be "%v" but got "%v" instead`, expectedError, err) } } </ins>

I'm writing a UT for the following function: The code branch where the server returns an error other than could not be tested using 's . This is because does not handle the error returned by and therefore could not return errors other than : Ref: Add a to so simulate errors other than . Run the following UT (should fail):
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
Same issue for some other methods of : -
I'll submit a PR shortly to address this issue. /assign


Question: }) } <ins> testDone := make(chan struct{}) defer close(testDone) </ins> ts := httptest.NewServer( WithHTTPLogging( withDeadline(

Similar to and easily reproducible. The handler occasionally completes and sends a response before timing out. /kind failing-test /kind flake /sig api-machinery
/assign
/assign Thank you very much for fixing these tests.
/triage accepted


Question: } return nil, framework.AsStatus(err) } <ins> // Check if the returned victims are valid. for nodeName, victims := range nodeNameToVictims { if victims == nil || len(victims.Pods) == 0 { if extender.IsIgnorable() { delete(nodeNameToVictims, nodeName) klog.InfoS("Ignoring node without victims", "node", nodeName) continue } return nil, framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeName)) } } </ins> // Replace victimsMap with new result after preemption. So the // rest of extenders can continue use it as parameter. victimsMap = nodeNameToVictims

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Kubernetes 1.19 (confirmed with 1.19.7 and 1.19.10) kube-scheduler crashes and restarts with the following errors in . panic: runtime error: index out of range [0] with length 0 The problem code is line 389. When is nil or is empty, the error happens. If we skip when it's nil or empty, the error is gone. panic: runtime error: invalid memory address or nil poster dereference The problem is does not exist and returns nil sometimes. Simply skipping it won't solve the problem. The scheduler will reach either line 456 or 341. The scheduler works without failures. When there are preemptions in a cluster. Kubernetes version (use ): v1.19.10 or v1.19.7 Cloud provider or hardware configuration: bare metal OS (e.g: ): centos 7 Kernel (e.g. ): Linux 5.4.77-7.el7pie SMP Sat Nov 21 01:16:27 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc
/assign
Are you running the vanilla scheduler? or a customized one that leverages the utilities in
It's a custom scheduler with some internal out-of-tree plugins (just like ), but it doesn't have any custom postFilter plugins. Which utility functions are you referring to? Thanks.
are those custom plugins filter plugins? do they maintain state?


Question: } return nil, framework.AsStatus(err) } <ins> // Check if the returned victims are valid. for nodeName, victims := range nodeNameToVictims { if victims == nil || len(victims.Pods) == 0 { if extender.IsIgnorable() { delete(nodeNameToVictims, nodeName) klog.InfoS("Ignoring node without victims", "node", nodeName) continue } return nil, framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeName)) } } </ins> // Replace victimsMap with new result after preemption. So the // rest of extenders can continue use it as parameter. victimsMap = nodeNameToVictims

If yes, then one hypothesis is that those custom filters filter the node in the filter phase, but not when executed in the preemption phase, and so this will result in a candidate node with no victims?
Yes, it includes filter plugins and uses cycle state.
you need to make sure that those filter plugins play nicely when executed again in the preemption phase in the same cycle: i.e., produce the same result.
Thanks What do you mean by performing filter in the preemption phase? would you mind elaborating it a little?
What additional logic is needed to handle the preemption case?
in the preemption phase we run the filters again to check if removing lower priority pods make the node schedulable: we later add them one by one to reduce the set of victim pods to the absolute minimum: My hypothesis is that somehow your filter returns success when adding all the pods back, and so you end up with zero victims and a candidate node. This shouldn't happen because if the pod fits the node without removing any victims, then we shouldn't be running preemption in the first place. So the theory is that the custom filter is returning false when run in the filter phase, but returns true when executed in the preemption phase perhaps because it makes some assumptions about cyclestate that makes it behave this way (e.g. that a the filter will be executed once per node in a scheduling cycle).
Disabled the filter plugin, but still see the same issues.
are you able to change the scheduler code and run the test again? I can send a patch tomorrow to add some debugging messages to help us root cause the issue.
yes, thanks a lot! really appreciate it!
You are right. We still use a deprecated Predicate extender, which causes the problem. After disabling it, it's working fine. It used to work with 1.18 though. We are retiring it. Thank you so much!!!
If the test cases can help debug scheduler extender Webhook (Predicate) too, I'd like to try it. Thanks again!
How can we improve the preemption logic to prevent bugs and issues in custom Filters or Predicates crashing the scheduler?
for starter, we should check the list len, the scheduler should not panic in all cases.


Question: } return nil, framework.AsStatus(err) } <ins> // Check if the returned victims are valid. for nodeName, victims := range nodeNameToVictims { if victims == nil || len(victims.Pods) == 0 { if extender.IsIgnorable() { delete(nodeNameToVictims, nodeName) klog.InfoS("Ignoring node without victims", "node", nodeName) continue } return nil, framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeName)) } } </ins> // Replace victimsMap with new result after preemption. So the // rest of extenders can continue use it as parameter. victimsMap = nodeNameToVictims

Other than that, I think we need to clearly document what the preemption logic does (that it executes the prefilter extensions points and the filter plugins multiple times) and the expectations from filter plugins (that they may get executed more than once in the same cycle)
Is it a predicate extender or preemption extender? If it's a predicate extender, I don't quite think we found the root cause - b/c predicate extenders are not invoked during preemption (). It makes more sense if it's a preemption extender - if the preemption extender mutates the to make it invalid. If it's the case, could you verify it by adding debug info before and after L157 to check the validness of ?
That's always true :)
filed a PR to mitigate it.
It's a predicate extender. I'll debug it more.
Added some debug info. after and . Here's the result. returned 4 candidates, but one of candidates () has zero pod. The candidate happened to be the best candidate and hence caused the "" error. It should have nothing to do with since our extender doesn't support Preemption. Disabling it may change pod placements/candidates selection and bypassed the error. It's like a coincidence. To remove the panic, a simple fix is to check if is empty. If it is, do not add the node to candidates. Alternatively, a change to may work too. Here is the log with debug info.
This looks more promising. Back to digging into the root cause, it seems returns an empty but with . So that implies firstly we have a non-nil , and then in , the is separated into two list: violatingVictims and nonViolatingVictims. Finally the two victims list are tried one by one to run . It looks like returns {fits=true, nil error} for every victim, which is quite abnormal. I think it's problematic that every pod is retrievable and in which we can debug into.
Yes, should not return fit=true with empty pods in the first place, but it's hard to locate the problematic code in this function.
The following code in looks questionable to me. If is empty, should the function return nil, nil? Also, if is true for all extenders, the loop won't do anything and candidates will remain the same.


Question: } return nil, framework.AsStatus(err) } <ins> // Check if the returned victims are valid. for nodeName, victims := range nodeNameToVictims { if victims == nil || len(victims.Pods) == 0 { if extender.IsIgnorable() { delete(nodeNameToVictims, nodeName) klog.InfoS("Ignoring node without victims", "node", nodeName) continue } return nil, framework.AsStatus(fmt.Errorf("expected at least one victim pod on node %q", nodeName)) } } </ins> // Replace victimsMap with new result after preemption. So the // rest of extenders can continue use it as parameter. victimsMap = nodeNameToVictims

Why do we bother creating a new candidates later? Can it just return candidates?
It can prevent from returning candidates with empty pods, but what about ? It's possible to add nodes without victim pods to candidates if returned by contain such nodes. How about adding the following check to CallExtender?
Updated the PR based on the finding and discussions.
Yes, we can use a flag to mark if all extenders don't support preemption. If yes, simply return the candidates.
Technically this can exist due to faulty extender implementation. If we really want to guard it, instead of logging the error and continue, I'm more inclined to return the error immediately as this is a fatal error - the victimsMap cannot be used either for latter extender or further preemptor nominating.
Is my understanding correct? In , if victimsMap is empty, should it return instead of ?
The only case victimsMap can be empty is empty candidates, and so return either doesn't quite matter, right?
You are right.
As long as one extender returns an invalid victimsMap (with empty pods), s returns an error immediately? What about ? Should it return an error immediately if it returnswith empty pods?
I think so. Because the result it returned is faulty, and we don't want to continue based on it The same. We may don't want to continue the scheduling cycle based on faulty result.


Question: framework.ExpectNotEqual(labelPod.Spec.NodeName, nodeName) }) <del> ginkgo.It("Pod should avoid nodes that have avoidPod annotation", func() { nodeName := nodeList.Items[0].Name // make the nodes have balanced cpu,mem usage cleanUp, err := createBalancedPodForNodes(f, cs, ns, nodeList.Items, podRequestedResource, 0.5) defer cleanUp() framework.ExpectNoError(err) ginkgo.By("Create a RC, with 0 replicas") rc := createRC(ns, "scheduler-priority-avoid-pod", int32(0), map[string]string{"name": "scheduler-priority-avoid-pod"}, f, podRequestedResource) // Cleanup the replication controller when we are done. defer func() { // Resize the replication controller to zero to get rid of pods. if err := e2erc.DeleteRCAndWaitForGC(f.ClientSet, f.Namespace.Name, rc.Name); err != nil { framework.Logf("Failed to cleanup replication controller %v: %v.", rc.Name, err) } }() ginkgo.By("Trying to apply avoidPod annotations on the first node.") avoidPod := v1.AvoidPods{ PreferAvoidPods: []v1.PreferAvoidPodsEntry{ { PodSignature: v1.PodSignature{ PodController: &metav1.OwnerReference{ APIVersion: "v1", Kind:       "ReplicationController", Name:       rc.Name, UID:        rc.UID, Controller: func() *bool { b := true; return &b }(), }, }, Reason:  "some reson", Message: "some message", }, }, } action := func() error { addOrUpdateAvoidPodOnNode(cs, nodeName, avoidPod) return nil } predicate := func(node *v1.Node) bool { val, err := json.Marshal(avoidPod) if err != nil { return false } return node.Annotations[v1.PreferAvoidPodsAnnotationKey] == string(val) } success, err := observeNodeUpdateAfterAction(f.ClientSet, nodeName, predicate, action) framework.ExpectNoError(err) framework.ExpectEqual(success, true) defer removeAvoidPodsOffNode(cs, nodeName) ginkgo.By(fmt.Sprintf("Scale the RC: %s to len(nodeList.Item)-1 : %v.", rc.Name, len(nodeList.Items)-1)) e2erc.ScaleRC(f.ClientSet, f.ScalesGetter, ns, rc.Name, uint(len(nodeList.Items)-1), true) testPods, err := cs.CoreV1().Pods(ns).List(context.TODO(), metav1.ListOptions{ LabelSelector: "name=scheduler-priority-avoid-pod", }) framework.ExpectNoError(err) ginkgo.By(fmt.Sprintf("Verify the pods should not scheduled to the node: %s", nodeName)) for _, pod := range testPods.Items { framework.ExpectNotEqual(pod.Spec.NodeName, nodeName) } }) </del> ginkgo.It("Pod should be preferably scheduled to nodes pod can tolerate", func() { // make the nodes have balanced cpu,mem usage ratio cleanUp, err := createBalancedPodForNodes(f, cs, ns, nodeList.Items, podRequestedResource, 0.5)


Question: }, }, }, <del> "spec.timeZone: Invalid value: "AMERICA/new_york": unknown time zone AMERICA/new_york": { ObjectMeta: metav1.ObjectMeta{ Name:      "mycronjob", Namespace: metav1.NamespaceDefault, UID:       types.UID("1a2b3c"), }, Spec: batch.CronJobSpec{ Schedule:          "0 * * * *", TimeZone:          &timeZoneBadCasing, ConcurrencyPolicy: batch.AllowConcurrent, JobTemplate: batch.JobTemplateSpec{ Spec: batch.JobSpec{ Template: validPodTemplateSpec, }, }, }, }, </del> "spec.timeZone: Invalid value: " America/New_York": unknown time zone  America/New_York": { ObjectMeta: metav1.ObjectMeta{ Name:      "mycronjob",

test withnot FAIL In a macos. do or golang upstrem disscusion. We use time.LoadLocation to validate timezone setting in object spec like cronjob.(It is ok) We also add a test try to validate is not a crroct value. This test is not true sometimes, for example, in default macos this is a true time_zone value.(maybe also true in other case insensitive os, but I have no device to try) At least we should get the test to pass on macos, I hope. <details</details<details</details<details</details<details</details<details</details<details</details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig test (I guess) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing (I guess)
see
/remove-sig testing /sig apps
I was very surprised to find that this error only occurs on darwin, and does not occur on windows, although windows is also case-sensitive by default. When using , windows finally use (see ), but darwin and linux have more choices (see ) they will first try , using the method, which will call the system's open method to open , because darwin is case-insensitive by default, and this operation will succeed.
/triage accepted


Question: }() } <del> // set up stdout stream </del> if e.stdout != nil { <del> headers.Set(api.StreamType, api.StreamTypeStdout) remoteStdout, err := conn.CreateStream(headers) if err != nil { return err } </del> wg.Add(1) go func() { defer wg.Done()

We create the stdin stream and immediately start copying local stdin to the container. We do the same for stdout and stderr (create stream, start copying immediately). In the kubelet, we wait to receive all the streams before we start the exec and copy between the streams and the container. The spdy connection frame handling code has 5 frame workers, each with a queue of up to 50 frames. In the event that we try to push more than 50 frames to a single queue before all streams are ready, the 51st attempt to push a frame blocks until there is space available in the queue. But there won't be any space until the server starts reading from the streams, which won't happen until all the streams have been received, so we're stuck at this point. The fix is to create all the streams first, and only start copying after all the streams have been created and the server is ready to copy between the streams and the container. Originally reported here:
FYI
This seems pretty high priority.
It's an easy fix. has already tested the patch for OpenShift and it works. Are we willing to pull it in before is resolved, as that will take longer to put in place?
Here's the patch, for reference:
5 * 50 = 250 frames before you get stuck? Or did I misunderstand?
Oh, I guess all the frames go to the same worker for some reason?
anyway, I'm fine with taking this and cherry picking it. Order doesn't matter; someone has to rebase either way, right?
frames are assigned to each queue based on stream id, so if you sent 50 frames to the stdin stream, that queue fills up and the loop that processes incoming frames is blocked until something starts reading from the stream. Working on a PR now.
Gotcha, thanks. , Andy Goldstein wrote:
an update: I spent some time trying to create an e2e that would fail more often than not without this fix, but I haven't had any luck so far (I have it stream a copy of the kubectl binary from the local system to the pod). The issue that originally uncovered the problem was from running exec on a Windows system, where it reproduces consistently, but I haven't seen it fail much on Linux. I will keep trying...


Question: } } <ins> func TestModifyExistingAuthInfo(t *testing.T) { tests := []struct { name            string flags           []string wantParseErr    bool wantCompleteErr bool wantValidateErr bool existingAuthInfo clientcmdapi.AuthInfo wantAuthInfo     clientcmdapi.AuthInfo }{ { name: "1. create new exec config", flags: []string{ "--exec-command=example-client-go-exec-plugin", "--exec-api-version=client.authentication.k8s.io/v1", "me", }, existingAuthInfo: clientcmdapi.AuthInfo{}, wantAuthInfo: clientcmdapi.AuthInfo{ Exec: &clientcmdapi.ExecConfig{ Command:    "example-client-go-exec-plugin", APIVersion: "client.authentication.k8s.io/v1", }, }, }, { name: "2. redefine exec args", flags: []string{ "--exec-arg=new-arg1", "--exec-arg=new-arg2", "me", }, existingAuthInfo: clientcmdapi.AuthInfo{ Exec: &clientcmdapi.ExecConfig{ Command:    "example-client-go-exec-plugin", APIVersion: "client.authentication.k8s.io/v1beta1", Args:       []string{"existing-arg1", "existing-arg2"}, }, }, wantAuthInfo: clientcmdapi.AuthInfo{ Exec: &clientcmdapi.ExecConfig{ Command:    "example-client-go-exec-plugin", APIVersion: "client.authentication.k8s.io/v1beta1", Args:       []string{"new-arg1", "new-arg2"}, }, }, }, { name: "3. reset exec args", flags: []string{ "--exec-command=example-client-go-exec-plugin", "me", }, existingAuthInfo: clientcmdapi.AuthInfo{ Exec: &clientcmdapi.ExecConfig{ Command:    "example-client-go-exec-plugin", APIVersion: "client.authentication.k8s.io/v1beta1", Args:       []string{"existing-arg1", "existing-arg2"}, }, }, wantAuthInfo: clientcmdapi.AuthInfo{ Exec: &clientcmdapi.ExecConfig{ Command:    "example-client-go-exec-plugin", APIVersion: "client.authentication.k8s.io/v1beta1", }, }, }, { name: "4. modify exec env variables", flags: []string{ "--exec-command=example-client-go-exec-plugin", "--exec-env=name1=value1000", "--exec-env=name3=value3", "--exec-env=name2-", "--exec-env=non-existing-", "me", }, existingAuthInfo: clientcmdapi.AuthInfo{ Exec: &clientcmdapi.ExecConfig{ Command:    "existing-command", APIVersion: "client.authentication.k8s.io/v1beta1", Env: []clientcmdapi.ExecEnvVar{ {Name: "name1", Value: "value1"}, {Name: "name2", Value: "value2"}, }, }, }, wantAuthInfo: clientcmdapi.AuthInfo{ Exec: &clientcmdapi.ExecConfig{ Command:    "example-client-go-exec-plugin", APIVersion: "client.authentication.k8s.io/v1beta1", Env: []clientcmdapi.ExecEnvVar{ {Name: "name1", Value: "value1000"}, {Name: "name3", Value: "value3"}, }, }, }, }, { name: "5. modify auth provider arguments", flags: []string{ "--auth-provider=new-auth-provider", "--auth-provider-arg=key1=val1000", "--auth-provider-arg=key3=val3", "--auth-provider-arg=key2-", "--auth-provider-arg=non-existing-", "me", }, existingAuthInfo: clientcmdapi.AuthInfo{ AuthProvider: &clientcmdapi.AuthProviderConfig{ Name: "auth-provider", Config: map[string]string{ "key1": "val1", "key2": "val2", }, }, }, wantAuthInfo: clientcmdapi.AuthInfo{ AuthProvider: &clientcmdapi.AuthProviderConfig{ Name: "new-auth-provider", Config: map[string]string{ "key1": "val1000", "key3": "val3", }, }, }, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { buff := new(bytes.Buffer) opts := new(createAuthInfoOptions) cmd := newCmdConfigSetAuthInfo(buff, opts) if err := cmd.ParseFlags(tt.flags); err != nil { if !tt.wantParseErr { t.Errorf("case %s: parsing error for flags %q: %v: %s", tt.name, tt.flags, err, buff) } return } if tt.wantParseErr { t.Errorf("case %s: expected parsing error for flags %q: %s", tt.name, tt.flags, buff) return } if err := opts.complete(cmd, buff); err != nil { if !tt.wantCompleteErr { t.Errorf("case %s: complete() error for flags %q: %s", tt.name, tt.flags, buff) } return } if tt.wantCompleteErr { t.Errorf("case %s: complete() expected errors for flags %q: %s", tt.name, tt.flags, buff) return } if err := opts.validate(); err != nil { if !tt.wantValidateErr { t.Errorf("case %s: flags %q: validate failed: %v", tt.name, tt.flags, err) } return } if tt.wantValidateErr { t.Errorf("case %s: flags %q: expected validate to fail", tt.name, tt.flags) return } modifiedAuthInfo := opts.modifyAuthInfo(tt.existingAuthInfo) if !reflect.DeepEqual(modifiedAuthInfo, tt.wantAuthInfo) { t.Errorf("case %s: flags %q: mis-matched auth info,nwanted=%#vngot=   %#v", tt.name, tt.flags, tt.wantAuthInfo, modifiedAuthInfo) } }) } } </ins> type createAuthInfoTest struct { description    string config         clientcmdapi.Config

Now it's possible to configure third-party auth providers using command with option. It's nice to have the same possibility for the recently . Example: To specify the executable path we can run This will generate a new config for the user: Various auxiliary configuration options are also could be . For instance: to define an argument for the executable; to set an environment variable; to specify a desired API version. /kind feature
I'm going to start working on this when the API is promoted to beta.
/sig auth
/assign
Can someone share why you can't do : Is there a way to set map values (e.g. something that is -name: value ; versus something that is -context: )
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I would like to work on it, if this is still persist.
cc
This still seems useful and is outstanding. any objections to taking it over?
No, it's okay, I'm working on it. I'll post the code this week, sorry for the delay
I think this would help out with generating valid kubectl configurations for our EKS developers. I'm using Pulumi to build EKS environments and if all the flags are available, I think I can generate the necessary entries to assume a role. Thanks! Ex: name: k8s.developer user: exec: apiVersion: args: token -i cluster-eksCluster- -r arn:aws:iam:::role/k8sDevAdminRole- command: aws-iam-authenticator env: null
done :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/priority important-longterm


Question: <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="975.431,762.352 965.093,758.095 970.506,756.696 972.634,751.527 "/> </g> <g> <del> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 478.975 491 C 552.046,491 637.129,312.5 711.464,312.5"/> <polygon style="fill: #000000" points="478.975,486 468.975,491 478.975,496 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="478.975,486 468.975,491 478.975,496 "/> </del> <ins> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 478.974 491 C 552.046,491 637.13,312.5 711.464,312.5"/> <polygon style="fill: #000000" points="478.974,486 468.974,491 478.974,496 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="478.974,486 468.974,491 478.974,496 "/> </ins> <polygon style="fill: #000000" points="718.964,312.5 708.964,317.5 711.464,312.5 708.964,307.5 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="718.964,312.5 708.964,317.5 711.464,312.5 708.964,307.5 "/> </g> <g> <del> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 478.933 505.457 C 677,522.75 650,676.75 712.305,704.479"/> <polygon style="fill: #000000" points="479.368,500.476 468.971,504.587 478.498,510.438 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="479.368,500.476 468.971,504.587 478.498,510.438 "/> </del> <ins> <path style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" d="M 478.932 505.457 C 677,522.75 650,676.75 712.305,704.479"/> <polygon style="fill: #000000" points="479.367,500.476 468.97,504.587 478.497,510.438 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="479.367,500.476 468.97,504.587 478.497,510.438 "/> </ins> <polygon style="fill: #000000" points="719.157,707.529 707.988,708.031 712.305,704.479 712.054,698.895 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="719.157,707.529 707.988,708.031 712.305,704.479 712.054,698.895 "/> </g>

Architecture picture: says kubecfg is deprecated. The architecture picture should be updated to show kubectl instead of kubecfg.
ref /cc


Question: } // NewPodInformer creates a shared index informer that returns only non-terminal pods. <del> func NewPodInformer(client clientset.Interface, resyncPeriod time.Duration, schedulerName string) coreinformers.PodInformer { </del> <ins> func NewPodInformer(client clientset.Interface, resyncPeriod time.Duration) coreinformers.PodInformer { </ins> selector := fields.ParseSelectorOrDie( <del> "spec.schedulerName=" + schedulerName + ",status.phase!=" + string(v1.PodSucceeded) + </del> <ins> "status.phase!=" + string(v1.PodSucceeded) + </ins> ",status.phase!=" + string(v1.PodFailed)) lw := cache.NewListWatchFromClient(client.CoreV1().RESTClient(), string(v1.ResourcePods), metav1.NamespaceAll, selector) return &podInformer{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Scheduler's pod informer has a selector that . This is incorrect. Scheduler must watch all the pods of the cluster, including those scheduled by other schedulers in order to update its cache of the state of the cluster. Otherwise, it will miss important events and the amount of available resources, pods on nodes, ... which are needed for scheduler to work correctly, will not be updated properly. /sig scheduling /assign
cc
For updating it cache, shouldn't the default scheduler rely on node info/status? Also its not very clear, what information the default scheduler needs from other pods (not to be scheduled by it) that it makes decision on? Just trying to understand the issue more clearly.
Scheduler watches informer events to update its cache. Node status is not enough to update all the information that scheduler caches. Scheduler must know what pods are placed on what nodes. For example, in order to check inter-pod affinity/anti-affinity it needs to know the placement of all the pods, including those scheduled by other schedulers.
But this information is present here: So it seems that your PR will now make sure to get the scheduled pods' information directly from the pod informer.
So when I check the history, I see it was here:
have we seen any specific issues due to this change (like between default scheduler and DS controller)?
Yes, but scheduler relies on its pod informers to keep this location up to date. That's exactly why it is important for the scheduler to receive updates for all the pods in the cluster. Yes. We missed the problem at the time. This is problematic only in clusters that use multiple schedulers. I initially thought that DS controller could cause issues as well, but DS controller does not set , so it must not cause an issue. We have seen instances of scheduler's missing certain events in cluster and as a result scheduler's cache getting stale, but I doubt if it is related to this particular issue. This only affects clusters with more than one scheduler. If you run the test that I have written in with the head, it will fail. That tests simulate a multi-scheduler scenario.


Question: return $KubeEnv.Contains('EXEC_AUTH_PLUGIN_URL') } <ins> # Permanently adds a directory to the $env:PATH environment variable. function Add-MachineEnvironmentPath { param ( [parameter(Mandatory=$true)] [string]$Path ) # Verify that the $Path is not already in the $env:Path variable. $pathForCompare = $Path.TrimEnd('').ToLower() foreach ($p in $env:Path.Split(";")) { if ($p.TrimEnd('').ToLower() -eq $pathForCompare) { return } } $newMachinePath = $Path + ";" + ` [System.Environment]::GetEnvironmentVariable("Path","Machine") [Environment]::SetEnvironmentVariable("Path", $newMachinePath, ` [System.EnvironmentVariableTarget]::Machine) $env:Path = $Path + ";" + $env:Path } </ins> # Export all public functions: Export-ModuleMember -Function *-*

<!-- Please only use this template for submitting enhancement requests --What would you like to be : pigz for GCE Windows Why is this needed: Pigz improves the image layer extraction performance for multi-core systems. Pigz is usually preinstalled on Linux. This enhancement is scoped specifically for the GKE Windows node scripts. For Context:
/assign me
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
GitHub didn't allow me to assign the following users: me. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign me Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows /assign jeremyje
s/GKE/GCE/? as in kube-up?
Replaced but I don't understand the kube-up part.
is how one creates a GCE cluster from this repo. Otherwise we'd be talking about something that isn't part of kubernetes.


Question: etcd_servers: $MASTER_IP roles: - kubernetes-pool <ins> - kubernetes-pool-vagrant </ins> cbr-cidr: $MINION_IP_RANGE minion_ip: $MINION_IP EOF

As a sysadmin When using the Vagrant cluster provided And logged in as the vagrant user to the master or minions I would like to be able to call docker commands wihout requiring the 'sudo' prefix. For Fedora =19 and RHEL=7 this can be enabled by adding the vagrant user to the docker group in /etc/groups
This should be trivial, but its tricker to do because of the whole salt configuration process as the docker group is only created on each kubernetes-minion once docker is installed by the salt-minion. To do this properly, we would need a salt state change for this, and new flag in to let you specify additional users that should be in the docker group.