Question: **All done !** <del> You can also use kubectl command to see if the newly created k8s is working correctly. </del> <ins> You can also use `kubectl` command to see if the newly created k8s is working correctly. The `kubectl` binary is under the `cluster/ubuntu/binaries` directory. You can move it into your PATH. Then you can use the below command smoothly. </ins> For example, use `$ kubectl get minions` to see if you get all your minion nodes comming up and ready. It may take some times for the minions be ready to use like below.

Though I have successfully installed kubernetes via :smiley: , still want to point out this doc missing the step to install . Thus, if user just follow the guide in the doc without having much knowledge about kubernetes, he/she would get error "It looks as if you don't have a compiled kubectl binary" during the step of . Manually install kubectl on at least master node should be necessary (suggest to let user also install on minion node for convenience), any way, the installation step is .
I forget to mention that.I will update the doc this tuned
I update the scripts in pr , you can try it now. I copied the kubectl binary from the kubernetes release and put it in , so there is no extra step for user to manually download . Actually, in the first version of ubuntu k8s deployment, I copied the to every node, but I think it is an overhead, so I removed it.I think this is just enough.
Doc update : Move the kubectl into the PATH, it may need sudo privilege so I decide not do for the user.
Have the same issue, please update the doc ASAP.
Great job :+1:
Also ran into this bug after recklessly running ( instead of ) :)


Question: var enqueued bool cases := []struct { <del> test          string newNode       *v1.Node oldNode       *v1.Node ds            *apps.DaemonSet shouldEnqueue bool </del> <ins> test               string newNode            *v1.Node oldNode            *v1.Node ds                 *apps.DaemonSet expectedEventsFunc func(strategyType apps.DaemonSetUpdateStrategyType) int shouldEnqueue      bool </ins> }{ { test:    "Nothing changed, should not enqueue",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: a cluster, And node1 with resource 1 DaemonSet, request resource 4 run , There is no pod created. resource to the node: run , There is still no pod created. What you expected to happen: In step 5, There is one pod created. How to reproduce it (as minimally and precisely as possible): see above description Anything else we need to know?: Environment: Kubernetes version (use ):1.10.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release):CentOS7 Kernel (e.g. ): Install tools: Others: /sig app
xref
Thanks! is not work here! Because: DaemonSetsController didn't watch the changes.(we use to updateNode.) we restart kubelet, The DaemonSetsController watch the changes, but it seems that the node info is not updated, and still failed to create pod:
I found it is a bug here, ignore the changes. I made a fix , PTAL! Thanks!
Any workarounds for old cluster versions? Up: looks like in my case helped but after some delay.
, which version of k8s are you using? If it's sunset, you have to merge the pr yourself :). I think restart kubelet trigger delete/add to make sure it sync .
Sorry, I didn't say
I think we still support 1.12 for now :) let's cherry pick it if not and wait for next 1.12 release.


Question: } func (f *fakeAttachDetachVolumeHost) CSINodeLister() storagelistersv1.CSINodeLister { <del> // not needed for testing return nil </del> <ins> csiNode := &storagev1.CSINode{ ObjectMeta: metav1.ObjectMeta{Name: f.nodeName}, Spec: storagev1.CSINodeSpec{ Drivers: []storagev1.CSINodeDriver{}, }, } enableMigrationOnNode(csiNode, csilibplugins.GCEPDInTreePluginName) return getFakeCSINodeLister(csiNode) } func enableMigrationOnNode(csiNode *storagev1.CSINode, pluginName string) { nodeInfoAnnotations := csiNode.GetAnnotations() if nodeInfoAnnotations == nil { nodeInfoAnnotations = map[string]string{} } newAnnotationSet := sets.NewString() newAnnotationSet.Insert(pluginName) nas := strings.Join(newAnnotationSet.List(), ",") nodeInfoAnnotations[v1.MigratedPluginsAnnotationKey] = nas csiNode.Annotations = nodeInfoAnnotations </ins> } func (f *fakeAttachDetachVolumeHost) CSIDriverLister() storagelistersv1.CSIDriverLister {

After the cluster was updated to enable PD CSI Driver, there was a race condition where AD Controller (kube-controller-manager) started to BEFORE the PD CSI Driver was up, thus AD Controller gave us the error Then AD Controller triggered the disk detach because the disk/volume was not populated in the desired state of world. Unfortunately the disk was in use during this detach and got corrupted. Since cluster upgrades can cause the controller manager and CSI drivers to start at different times, the AD controller needs to be more conservative when detaching We have not been able to reproduce this, it came up from a customer. /sig storage /assign <detailsk8s 1.23, CSI migration enabled </details<detailsGKE </details<details</details<details</details<details</details<details</details
/triage accepted
I think there's potentially two changes we should do: 1) for the initial DSW population, we should skip the check that if the csi driver is not installed. 2) in general, we should not skip volumes during dsw initial population if an error is returned. If we had some temporary api server error then that would cause volumes to not get to dsw and similar behavior happen. cc
for a repro, maybe you can try something like: a cluster with csi migration enabled (control plane + nodes) a pod with intree-volume and get it running the csi driver node daemonset. verify the driver is removed from CSINode kube-controller-manager
cc
/cc was trying to repro this, but on GKE with the managed driver which made it complicated.
Yeah maybe gce kube-up would be easier for a repro
If CSINode says that it has migrated GCP PD volume plugin to CSI, then IMO A/D controller should always attach GCP PD volumes to it using CSI, regardless if the CSI driver is present or not. In-tree volume plugin in the kubelet is disabled and can't mount any in-tree volumes anyway. To flip the migration feature gate / upgrade / downgrade kubelet, the cluster admin must drain the node first to make sure everything is detached. A/D controller should not see change of CSINode migration annotations with any volume attached to it.
/priority important-soon
Another option could be to process these volumes anyways but add them to DSOW with migration status as "uncertain", so as they can be re-processed later.


Question: } func (f *fakeAttachDetachVolumeHost) CSINodeLister() storagelistersv1.CSINodeLister { <del> // not needed for testing return nil </del> <ins> csiNode := &storagev1.CSINode{ ObjectMeta: metav1.ObjectMeta{Name: f.nodeName}, Spec: storagev1.CSINodeSpec{ Drivers: []storagev1.CSINodeDriver{}, }, } enableMigrationOnNode(csiNode, csilibplugins.GCEPDInTreePluginName) return getFakeCSINodeLister(csiNode) } func enableMigrationOnNode(csiNode *storagev1.CSINode, pluginName string) { nodeInfoAnnotations := csiNode.GetAnnotations() if nodeInfoAnnotations == nil { nodeInfoAnnotations = map[string]string{} } newAnnotationSet := sets.NewString() newAnnotationSet.Insert(pluginName) nas := strings.Join(newAnnotationSet.List(), ",") nodeInfoAnnotations[v1.MigratedPluginsAnnotationKey] = nas csiNode.Annotations = nodeInfoAnnotations </ins> } func (f *fakeAttachDetachVolumeHost) CSIDriverLister() storagelistersv1.CSIDriverLister {

But surely it looks like might be easiest unless it has any downsides.
There wasn't a node update in this case---there was a control plane update which restarted the PD CSI driver, and it looks like the CSIDriver resource was updated or changed in some way which raced with the kube-controller-manager restart.
Is there anyway we can know if the csi driver is up and running(like a health check ping) before populating DSW?
We use the CSINode registration to determine that. The discussion here I think is concluding that we should not use that as a criteria for adding the volume to DSW. We only need to check that CSI migration is enabled on the node.
+1 we need to be very resilient against races, I think we have to assume a driver is in process of being installed at any point in the AD controller reconcilliation.
/assign sunnylovestiramisu
Reproduced with the steps: In /var/log/kube-controller-
Tested with the and doing the reproduce steps above. The error of does not show up anymore. The error message still get logged in the kube-controller- for debugging purpose: Ran Ran On the GCP console, the disk is used by node: Ran I redeploy the pd-csi-driver and then make sure it is up and running. Ran


Question: <ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package topologymanager import ( "k8s.io/api/core/v1" "k8s.io/kubernetes/pkg/kubelet/cm/containermap" "k8s.io/kubernetes/pkg/kubelet/lifecycle" ) type noneScope struct { scope } // Ensure noneScope implements Scope interface var _ Scope = &noneScope{} // NewNoneScope returns a none scope. func NewNoneScope() Scope { return &noneScope{ scope{ name:             noneTopologyScope, podTopologyHints: podTopologyHints{}, policy:           NewNonePolicy(), podMap:           containermap.NewContainerMap(), }, } } func (s *noneScope) Admit(pod *v1.Pod) lifecycle.PodAdmitResult { return s.admitPolicyNone(pod) } </ins>

Follow up from: See details there. When is , there is no need to discover the NUMA topology. /sig node /priority backlog /kind feature /help-wanted
/cc
/triage accepted
I would like to take this up also can I get more brief information about it? code review comments from asking that inside the NewManager() implementation, we need to skip NewPolicyOptions and NewNUMAInfo but when the NumaTopology comes kubelet issue also comes. I need some more information to start with this!
/cc
When there is no topology manger policy specified, NewManager should be a no-op. Just skip all checks.
Exactly. Just to add: we may need few iterations over the change to get a clean flow in the call.
/assign


Question: assert.Equal(t, test.output, output) } } <del> func TestPodResourcesAreReclaimed(t *testing.T) { type args struct { pod           *v1.Pod status        v1.PodStatus runtimeStatus kubecontainer.PodStatus } tests := []struct { name string args args want bool }{ { "pod with running containers", args{ pod: &v1.Pod{}, status: v1.PodStatus{ ContainerStatuses: []v1.ContainerStatus{ runningState("containerA"), runningState("containerB"), }, }, }, false, }, { "pod with containers in runtime cache", args{ pod:    &v1.Pod{}, status: v1.PodStatus{}, runtimeStatus: kubecontainer.PodStatus{ ContainerStatuses: []*kubecontainer.ContainerStatus{ {}, }, }, }, false, }, { "pod with sandbox present", args{ pod:    &v1.Pod{}, status: v1.PodStatus{}, runtimeStatus: kubecontainer.PodStatus{ SandboxStatuses: []*runtimeapi.PodSandboxStatus{ {}, }, }, }, false, }, } testKubelet := newTestKubelet(t, false) defer testKubelet.Cleanup() kl := testKubelet.kubelet for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { testKubelet.fakeRuntime.PodStatus = tt.args.runtimeStatus if got := kl.PodResourcesAreReclaimed(tt.args.pod, tt.args.status); got != tt.want { t.Errorf("PodResourcesAreReclaimed() = %v, want %v", got, tt.want) } }) } } </del>

What happened: It's most visible in the test <img width="323" alt="Screen Shot 2020-05-20 at 6 09 10 PM" src="https://user-"It shows up in kind, kube- and kops jobs The regression was introduced somewhere within , possible culprits are: (FYI (FYI (FYI There are more tests affected, just not as prominently (visually inspecting ) [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data I wasn't sure whether to tag this as kind/regression or not, I'll leave that up to or someone /sig node /sig api-machimery /sig scheduling
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: It's most visible in the test (FYI (FYI (FYI StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data /sig api-machimery /sig scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.


Question: assert.Equal(t, test.output, output) } } <del> func TestPodResourcesAreReclaimed(t *testing.T) { type args struct { pod           *v1.Pod status        v1.PodStatus runtimeStatus kubecontainer.PodStatus } tests := []struct { name string args args want bool }{ { "pod with running containers", args{ pod: &v1.Pod{}, status: v1.PodStatus{ ContainerStatuses: []v1.ContainerStatus{ runningState("containerA"), runningState("containerB"), }, }, }, false, }, { "pod with containers in runtime cache", args{ pod:    &v1.Pod{}, status: v1.PodStatus{}, runtimeStatus: kubecontainer.PodStatus{ ContainerStatuses: []*kubecontainer.ContainerStatus{ {}, }, }, }, false, }, { "pod with sandbox present", args{ pod:    &v1.Pod{}, status: v1.PodStatus{}, runtimeStatus: kubecontainer.PodStatus{ SandboxStatuses: []*runtimeapi.PodSandboxStatus{ {}, }, }, }, false, }, } testKubelet := newTestKubelet(t, false) defer testKubelet.Cleanup() kl := testKubelet.kubelet for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { testKubelet.fakeRuntime.PodStatus = tt.args.runtimeStatus if got := kl.PodResourcesAreReclaimed(tt.args.pod, tt.args.status); got != tt.want { t.Errorf("PodResourcesAreReclaimed() = %v, want %v", got, tt.want) } }) } } </del>

</details
/cc
will increase the time for deletion of pod and it should be expected because currently pod is deleted from apiserver before removing the sandbox container but after this fix pod will be deleted from the apiserver only after deleting the sandbox container.
/remove-sig scheduling doesn't affect scheduling running time at all.
Yeah, is almost definitely the culprit. Last time we changed that code path, it also caused a regression. The degradation in performance is expected, but a 5x slowdown is definitely larger than it should be.
do we know if that's because the status used by the path is stale, or if removing sandboxes is actually that slow?
if there's not a quick fix, should we revert and bring it back in once the root issue is understood?
I think a revert is appropriate in this case.
ok, I'll open it
open at
I agree with that 5x is something which shouldn't happen and i can take a look on why thats happening and that might require changing the way how the sandbox is currently deleted which is only during GC.


Question: echo -n Attempt "$(($attempt+1))" to check for SSH to master local output local ok=1 <del> output=$(ssh -oStrictHostKeyChecking=no -i "${AWS_SSH_KEY}" ubuntu@${KUBE_MASTER_IP} uptime 2> $LOG) || ok=0 </del> <ins> output=$(ssh -oStrictHostKeyChecking=no -i "${AWS_SSH_KEY}" ${SSH_USER}@${KUBE_MASTER_IP} uptime 2> $LOG) || ok=0 </ins> if [[ ${ok} == 0 ]]; then if (( attempt > 30 )); then echo

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I visited the Kubernetes 1.11 release blog post and clicked on the link to the article on resizing persistent volumes. The server returned a 404. What you expected to happen: I expected to navigate to the article on resizing persistent volumes in Kubernetes. How to reproduce it (as minimally and precisely as possible): Visit the release announcement blog post (), scroll down to the "4 Day Features Blog Series" section, and click on the link to the article on resizing persistent volumes. Anything else we need to know?: The link currently points to: The correct link is: Environment: Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A
Reiterating the mentions to trigger a notification: <detailsIn response to : <!-- This form is for bug reports and feature requests ONLY! --/kind bug /kind feature What happened: I visited the Kubernetes 1.11 release blog post and clicked on the link to the article on resizing persistent volumes. The server returned a 404. I expected to navigate to the article on resizing persistent volumes in Kubernetes. Visit the release announcement blog post (), scroll down to the "4 Day Features Blog Series" section, and click on the link to the article on resizing persistent volumes. The link currently points to: The correct link is: Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It should be reported to kubernetes/website, but doesn't matter.


Question: // Enable swagger api on master. components.KubeMaster.InstallSwaggerAPI() cluster := clientcmdapi.NewCluster() cluster.Server = components.ApiServer.URL cluster.InsecureSkipTLSVerify = true <ins> cfg.Contexts = map[string]*clientcmdapi.Context{"test": ctx} cfg.CurrentContext = "test" </ins> overrides := clientcmd.ConfigOverrides{ <del> ClusterInfo:    *cluster, Context:        *ctx, CurrentContext: "test", </del> <ins> ClusterInfo: *cluster, </ins> } cmdConfig := clientcmd.NewNonInteractiveClientConfig(*cfg, "test", &overrides, nil) factory := util.NewFactory(cmdConfig)

In 1.1, if you specify "current-context: i-do-not-exist" in a kubeconfig, and there's no matching context by that name in the contexts, kubelet silently uses what I assume is either an empty or a default configuration. This can result in hilarity like wondering why your kubelet isn't using the TLS settings you provided, and spending several hours debugging the whole stack only to discover that you typo'd current-context :). It would be better if kubelet and other users of errored out on such a misconfiguration, to make the failure obvious and trivial.
cc
I think kubelet silently uses an empty configuration. Guess we should give a waring here.
Did you start to work on this?
Below is related code: can you confirm this? Besides if we want to log a waring in kubelet, we may need to get in kubelet.
cc
cc
I believe kubelet already uses . Adding a check in that current context exists if specified sounds like the right approach.
Are there any backwards compatibility implications here? would this break users who have existing configs (that fallback to default namespace)? , Jeff Lowdermilk wrote:
I think kubelet directly use not . Also already checks whether the user-specified current context exists, the problem is that if not exist, we just skip. We need to do something if the current context not exist, like log a warning.
Erroring on a specified and bad seems reasonable. If it is not specified and there's only one option, I don't think we should fail.
yeap, the case is that user specify the but not exist, in such case we should just fail.
If no one is working on this I can take it.
I think this is the root cause. Basically we should error on this not exist context. If you have extra bandwidth, please go ahead :)
This is related to - the defaulting is what allows us to continue. When that merges, only kubectl will have the "use localhost:8080" as defaulting.


Question: klog.Errorf("Unsupported protocol type: %s", protocol) } if nodePortLocalSet != nil { <del> if valid := nodePortLocalSet.validateEntry(entry); !valid { klog.Errorf("%s", fmt.Sprintf(EntryInvalidErr, entry, nodePortLocalSet.Name)) </del> <ins> entryInvalidErr := false for _, entry := range entries { if valid := nodePortLocalSet.validateEntry(entry); !valid { klog.Errorf("%s", fmt.Sprintf(EntryInvalidErr, entry, nodePortLocalSet.Name)) entryInvalidErr = true break } nodePortLocalSet.activeEntries.Insert(entry.String()) } if entryInvalidErr { </ins> continue } <del> nodePortLocalSet.activeEntries.Insert(entry.String()) </del> } }

What happened: SCTP nodeport service is not working for all IPs present in the node when ipvs is enabled. It is working only for node's InternalIP. What you expected to happen: SCTP nodeport service must work for all ips present in the node when ipvs is enabled. How to reproduce it (as minimally and precisely as possible): Create a kubernetes cluster with at least one node having multiple interfaces with different IPs. Start an SCTP server deployment. Then create a nodeport service for the same. Now try to reach the service using all the IPs present in that node. The service will be accessible only via node's InternalIP not any other IPs. Environment: Kubernetes version: v1.15.2 OS: ubuntu 18.04 Kernel: 4.15.0-54-generic Network plugin and version (if this is a network-related bug): ipset version v6.34
/sig network /area kube-proxy /area ipvs
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: return b } <del> // ResourceTypeOrNameArgs indicates that the builder should accept one or two arguments // of the form `(<type1>[,<type2>,...]|<type> <name>)`. When one argument is received, the types // provided will be retrieved from the server (and be comma delimited).  When two arguments are // received, they must be a single type and name. If more than two arguments are provided an // error is set. The allowEmptySelector permits to select all the resources (via Everything func). </del> <ins> // ResourceTypeOrNameArgs indicates that the builder should accept arguments // of the form `(<type1>[,<type2>,...]|<type> <name1>[,<name2>,...])`. When one argument is // received, the types provided will be retrieved from the server (and be comma delimited). // When two or more arguments are received, they must be a single type and resource name(s). // The allowEmptySelector permits to select all the resources (via Everything func). </ins> func (b *Builder) ResourceTypeOrNameArgs(allowEmptySelector bool, args ...string) *Builder { <del> switch len(args) { case 2: b.name = args[1] </del> <ins> switch { case len(args) > 2: b.names = append(b.names, args[1:]...) b.ResourceTypes(SplitResourceArgument(args[0])...) case len(args) == 2: b.names = append(b.names, args[1]) </ins> b.ResourceTypes(SplitResourceArgument(args[0])...) <del> case 1: </del> <ins> case len(args) == 1: </ins> b.ResourceTypes(SplitResourceArgument(args[0])...) if b.selector == nil && allowEmptySelector { b.selector = labels.Everything() } <del> case 0: </del> <ins> case len(args) == 0: </ins> default: b.errs = append(b.errs, fmt.Errorf("when passing arguments, must be resource or resource and name")) }

Currently, it could not delete multiple pods at once, need to delete them one by one, it's very inconvenient if I have tons of pods to delete. Unlike docker, docker can multiple stop/rm containers/images at once.
This is possible using label selectors as of , which is better if you have tons of pods.
I agree that using label selector is better to delete all of pods under the label. But to delete multile pods at once is still usable: I just want to delete 1/2 pods under the label, label selector will not work selector could not span over other selectors or pods Another reason is docker rm/stop x y z works, I will wonder why kubectl cannot. This is not urgent, but I hope it can be implemented.
I agree. Thanks for the suggestion.
Would this be a suitable task for someone new to k8s development? If so, I'd be willing to take a look at this, if someone could provide a couple of hints where to start.
Further to my last message, I have a functional work in progress I would like to share, if anyone is interested?
PRs are welcome. See
Yes, this is a reasonable place to start. Since its in the client, you won't get much orientation to the core server code, but it will be a good place to orient to the client side API. There us also prior art that you can model off of. The kubectl create ... command already supports multiple resources. A good command to adapter would be the 'label' command. You can find it in Many thanks in advance! Brendan On Feb 19, 2015 8:21 AM, "Eric Tune" wrote:


Question: Expect(err).NotTo(HaveOccurred(), "error failing a daemon pod") err = wait.PollImmediate(dsRetryPeriod, dsRetryTimeout, checkRunningOnAllNodes(f, ds)) Expect(err).NotTo(HaveOccurred(), "error waiting for daemon pod to revive") <ins> By("Wait for the failed daemon pod to be completely deleted.") err = wait.PollImmediate(dsRetryPeriod, dsRetryTimeout, waitFailedDaemonPodDeleted(c, &pod)) Expect(err).NotTo(HaveOccurred(), "error waiting for the failed daemon pod to be completely deleted") </ins> }) // This test should not be added to conformance. We will consider deprecating OnDelete when the

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: DaemonSet has an e2e test running that has dubious purpose. What you expected to happen: The e2e test should be refactored and commented clearer as it will be part of the conformance.
To elaborate, we're concerned that the following may not actually ensure that DaemonSet recreates Failed Pods: For example, it seems possible that the Node might simply overwrite the Pod status with its own information if it hasn't yet seen the change to the Phase. Ideally, we should verify that the Pod UID we force to Failed no longer shows up in the DaemonSet's list of Pods after the DaemonSet becomes happy again.


Question: }, expectErr: false, }, <ins> // IPVS Timeout can be 0 { config: kubeproxyconfig.KubeProxyIPVSConfiguration{ SyncPeriod:    metav1.Duration{Duration: 5 * time.Second}, TCPTimeout:    metav1.Duration{Duration: 0 * time.Second}, TCPFinTimeout: metav1.Duration{Duration: 0 * time.Second}, UDPTimeout:    metav1.Duration{Duration: 0 * time.Second}, }, expectErr: false, }, // IPVS Timeout > 0 { config: kubeproxyconfig.KubeProxyIPVSConfiguration{ SyncPeriod:    metav1.Duration{Duration: 5 * time.Second}, TCPTimeout:    metav1.Duration{Duration: 1 * time.Second}, TCPFinTimeout: metav1.Duration{Duration: 2 * time.Second}, UDPTimeout:    metav1.Duration{Duration: 3 * time.Second}, }, expectErr: false, }, // TCPTimeout Timeout < 0 { config: kubeproxyconfig.KubeProxyIPVSConfiguration{ SyncPeriod: metav1.Duration{Duration: 5 * time.Second}, TCPTimeout: metav1.Duration{Duration: -1 * time.Second}, }, expectErr: true, reason:    "TCPTimeout must be greater than or equal to 0", }, // TCPFinTimeout Timeout < 0 { config: kubeproxyconfig.KubeProxyIPVSConfiguration{ SyncPeriod:    metav1.Duration{Duration: 5 * time.Second}, TCPFinTimeout: metav1.Duration{Duration: -1 * time.Second}, }, expectErr: true, reason:    "TCPFinTimeout must be greater than or equal to 0", }, // UDPTimeout Timeout < 0 { config: kubeproxyconfig.KubeProxyIPVSConfiguration{ SyncPeriod: metav1.Duration{Duration: 5 * time.Second}, UDPTimeout: metav1.Duration{Duration: -1 * time.Second}, }, expectErr: true, reason:    "UDPTimeout must be greater than or equal to 0", }, </ins> } for _, test := range testCases {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kube-proxy doesn't validate the configurable IPVS tcp, tcpfin and udp timeouts. For example, when kube-proxy starts with , it will always log a warning: . What you expected to happen: kube-proxy should validate --ipvs-tcp-timeout, --ipvs-tcpfin-timeout, --ipvs-udp-timeout flag values at start. How to reproduce it (as minimally and precisely as possible): add flags to kube-proxy Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network /area ipvs
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: }, }, }, <del> map[string]interface{}{"field": "foo"}, }, true}, </del> <ins> failingObjects: []interface{}{map[string]interface{}{"field": "foo"}}, }, </ins> } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { <del> validator, _, err := NewSchemaValidator(&apiextensions.CustomResourceValidation{OpenAPIV3Schema: &tt.args.schema}) </del> <ins> validator, _, err := NewSchemaValidator(&apiextensions.CustomResourceValidation{OpenAPIV3Schema: &tt.schema}) </ins> if err != nil { t.Fatal(err) } <del> if err := ValidateCustomResource(tt.args.object, validator); (err != nil) != tt.wantErr { if err == nil { t.Error("expected error, but didn't get one") } else { t.Errorf("unexpected validation error: %v", err) </del> <ins> for _, obj := range tt.objects { if err := ValidateCustomResource(obj, validator); err != nil { t.Errorf("unexpected validation error for %v: %v", obj, err) } } for _, obj := range tt.failingObjects { if err := ValidateCustomResource(obj, validator); err == nil { t.Errorf("missing error for %v", obj) </ins> } } })

The server-side validation does not check the type of values with . Create a CRD like: Then create a CR: It is not rejected, but should because must not be a boolean.


Question: CSI: &v1.CSIPersistentVolumeSource{ Driver:           AzureDiskDriverName, VolumeHandle:     azureSource.DataDiskURI, <del> ReadOnly:         *azureSource.ReadOnly, FSType:           *azureSource.FSType, </del> VolumeAttributes: map[string]string{azureDiskKind: "Managed"}, }, }, AccessModes: []v1.PersistentVolumeAccessMode{v1.ReadWriteOnce}, }, } <ins> if azureSource.ReadOnly != nil { pv.Spec.PersistentVolumeSource.CSI.ReadOnly = *azureSource.ReadOnly } </ins> <del> if *azureSource.CachingMode != "" { </del> <ins> if azureSource.CachingMode != nil && *azureSource.CachingMode != "" { </ins> pv.Spec.PersistentVolumeSource.CSI.VolumeAttributes[azureDiskCachingMode] = string(*azureSource.CachingMode) } <del> if *azureSource.FSType != "" { </del> <ins> if azureSource.FSType != nil { pv.Spec.PersistentVolumeSource.CSI.FSType = *azureSource.FSType </ins> pv.Spec.PersistentVolumeSource.CSI.VolumeAttributes[azureDiskFSType] = *azureSource.FSType } if azureSource.Kind != nil {

What happened: Some of the Azure CSI translation checks in and needs to perform NPE checks for pointer based fields as pointed out in and What you expected to happen: NPE checks should be there How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.15 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
/priority critical-urgent


Question: // Volume is added to asw. Because attach operation fails, volume should not be reported as attached to the node. waitForVolumeAddedToNode(t, generatedVolumeName, nodeName1, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateUncertain, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, false, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, false, asw, volumeAttachedCheckTimeout) </ins> // When volume is added to the node, it is set to mounted by default. Then the status will be updated by checking node status VolumeInUse. // Without this, the delete operation will be delayed due to mounted status

TestRunOneVolumeDetachFailNodeWithReadWriteOnce 4 times since 03-19 23:00 PDT, intermittently ! https://k8s- Still investigating whether this is a tolerable flake or not with more history than testgrid view, or otherwise would appreciate input from anyone with expertise in this test, especially as regards to whether this should be a blocker for tomorrows beta cut. maybe: /sig storage /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is already open for this, looks like it is attempting to fix it
thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. /close
Closing this issue. <detailsIn response to : thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Volume is added to asw. Because attach operation fails, volume should not be reported as attached to the node. waitForVolumeAddedToNode(t, generatedVolumeName, nodeName1, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateUncertain, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, false, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, false, asw, volumeAttachedCheckTimeout) </ins> // When volume is added to the node, it is set to mounted by default. Then the status will be updated by checking node status VolumeInUse. // Without this, the delete operation will be delayed due to mounted status

pull-kubernetes-unit TestRunOneVolumeDetachFailNodeWithReadWriteOnce Jan 7th, 2022 Check volume <fake-plugin/volume-nameis reported as attached to node <fail-detach-node, got false, expected true No response_ /sig storage
/cc
this is the top unit test flake according to
cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
I think this test is no longer flaky by checking the test grid, close it now.
Unfortunately this does still seem to be flaky: Here was the output from : /reopen
Reopened this issue. <detailsIn response to : Unfortunately this does still seem to be flaky: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
still the case
How about adding some logging to help troubleshoot this? I looked at it a little, but it is hard to tell which part of the test is failing. PR
/
This is still happening
/triage accepted
From the logs, in the OneVolumeDetachFailNodeWithReadWriteOnce test, succeeded, but the next verification failed. In verifyVolumeAttachedToNode, the attachState is from calling and within asw, it calls . In verifyVolumeReportedAsAttachedToNode, the volumes are obtained from calling , it seems that the node is not in asw.nodesToUpdateStatusFor at this step. Cannot reproduce locally. I tried to run the test 100 times locally with debug.Stack()... 3 * 100 times all passed.
Tried to run stress test on my linux machine as well but still cannot reproduce. Command: Result: Also tried running n process in parallel. Command: Result:
It seems like it happens more frequently.


Question: // Volume is added to asw. Because attach operation fails, volume should not be reported as attached to the node. waitForVolumeAddedToNode(t, generatedVolumeName, nodeName1, asw) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateUncertain, asw) <del> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, false, asw) </del> <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, false, asw, volumeAttachedCheckTimeout) </ins> // When volume is added to the node, it is set to mounted by default. Then the status will be updated by checking node status VolumeInUse. // Without this, the delete operation will be delayed due to mounted status

But I may have no time to dig into the cause. Let's see.
cc as the test author.
The problem is that this test has a lot of dependency on the sleeps that are hardcoded with a very narrow scope, but our CI seems to be able to stress it much more. As soon as you slightly change one of the sleep duration it starts to flake locally The test can not make an assumption that timing is perfect, even more in the sub second range can someone in sig-storage review this test and make it less dependant on those tight sleeps?
According to most failures, it failed at Line 660. Let me run some stress locally. with 250ms sleep with 100ms sleep
I agree this test case is too brittle due to its specific timing dependencies. To fix this, I suggest reworking the test case to: 1) Add retry with timeout to wait for final conditions 2) Remove the intermediate timing-based state transition checks. Instead, we can keep track of state transitions in the Fakes and validate those stats. wdyt?
Friendly ping from 1.27 CI Signal that this test is still a noisy flake on the master-blocking dashboard as CI Signal has flagged it in and


Question: func FilterActivePods(pods []api.Pod) []*api.Pod { var result []*api.Pod for i := range pods { <del> if api.PodSucceeded != pods[i].Status.Phase && api.PodFailed != pods[i].Status.Phase && pods[i].DeletionTimestamp == nil { result = append(result, &pods[i]) </del> <ins> p := pods[i] if api.PodSucceeded != p.Status.Phase && api.PodFailed != p.Status.Phase && p.DeletionTimestamp == nil { result = append(result, &p) } else { glog.V(4).Infof("Ignoring inactive pod %v/%v in state %v, deletion time %v", p.Namespace, p.Name, p.Status.Phase, p.DeletionTimestamp) </ins> } } return result } <ins> // SyncAllPodsWithStore lists all pods and inserts them into the given store. // Though this function is written in a generic manner, it is only used by the // controllers for a specific purpose, to synchronously populate the store // with the first batch of pods that would otherwise be sent by the Informer. // Doing this avoids more complicated forms of synchronization with the // Informer, though it also means that the controller calling this function // will receive "OnUpdate" events for all the pods in the store, instead of // "OnAdd". This should be ok, since most controllers are level triggered // and make decisions based on the contents of the store. // // TODO: Extend this logic to load arbitrary local state for the controllers // instead of just pods. func SyncAllPodsWithStore(kubeClient client.Interface, store cache.Store) { var allPods *api.PodList var err error listOptions := api.ListOptions{LabelSelector: labels.Everything(), FieldSelector: fields.Everything()} for { if allPods, err = kubeClient.Pods(api.NamespaceAll).List(listOptions); err != nil { glog.Warningf("Retrying pod list: %v", err) continue } break } pods := []interface{}{} for i := range allPods.Items { p := allPods.Items[i] glog.V(4).Infof("Initializing store with pod %v/%v", p.Namespace, p.Name) pods = append(pods, &p) } store.Replace(pods, allPods.ResourceVersion) return } </ins>

The test is in flaky suite and no one seems to care. We need an owner for this test. cc Last failure log:
do you think you can take a look at this?
This is happening because of a bad assumption. The assumption is: if the reflector's store is consistent with etcd, the store given back to clients of the framework.Controller is too. I'd really like a general fix applicable to all controllers for this problem, but fixing it just for the RCs is pretty easy. In more detail: This function reports when the reflector has listed from apiserver. But the store given to the reflector is the fifo of the framework.Controller. The store given back to the caller is still not consistent, and will not be till the framework.Controller puts all the objects from the fifo into the client state store (). The rc specific fix: Before starting the rc manager, seed the expectations store with an expectation like {rc:foo, adds: foo.Status.Replicas, deletes:0}. This will wait for status.Replicas number of ADD events till a timeout. The general fix: Somehow expose this through framework.Controller.HasSynced (clients don't care about the fifo anyway, shouldn't it just be an implementation detail?) ideas? cc'ing people who have written controllers that might be interested I don't think this (and generally anything flaky for 1 release) is a p0, I can fix it if you don't mind dealing with the flakes till I've cleared things off my plate. I'd be more than happy to review a fix.
Well, it's a P0 because test is flaky. E.g last run:
- did you have chance to look into it?
The "DaemonRestart" tests are still labeled at . I will send a PR to promote them.
are you still planning to send aPR to promote the DaemonRestart tests out of flaky? I think this is the right thing to do.
Ack; sorry, I lost this in the muck.


Question: <del> /* Copyright 2015 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ // Tests for liveness probes, both with http and with docker exec. // These tests use the descriptions in examples/liveness to create test pods. package e2e import ( "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" "github.com/golang/glog" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) func runLivenessTest(c *client.Client, podDescr *api.Pod) bool { ns := "e2e-test-" + string(util.NewUUID()) glog.Infof("Creating pod %s in namespace %s", podDescr.Name, ns) _, err := c.Pods(ns).Create(podDescr) if err != nil { glog.Infof("Failed to create pod %s: %v", podDescr.Name, err) return false } // At the end of the test, clean up by removing the pod. defer c.Pods(ns).Delete(podDescr.Name) // Wait until the pod is not pending. (Here we need to check for something other than // 'Pending' other than checking for 'Running', since when failures occur, we go to // 'Terminated' which can cause indefinite blocking.) err = waitForPodNotPending(c, ns, podDescr.Name, 60*time.Second) if err != nil { glog.Infof("Failed to start pod %s in namespace %s: %v", podDescr.Name, ns, err) return false } glog.Infof("Started pod %s in namespace %s", podDescr.Name, ns) // Check the pod's current state and verify that restartCount is present. pod, err := c.Pods(ns).Get(podDescr.Name) if err != nil { glog.Errorf("Get pod %s in namespace %s failed: %v", podDescr.Name, ns, err) return false } initialRestartCount := pod.Status.Info["liveness"].RestartCount glog.Infof("Initial restart count of pod %s is %d", podDescr.Name, initialRestartCount) // Wait for at most 48 * 5 = 240s = 4 minutes until restartCount is incremented for i := 0; i < 48; i++ { // Wait until restartCount is incremented. time.Sleep(5 * time.Second) pod, err = c.Pods(ns).Get(podDescr.Name) if err != nil { glog.Errorf("Get pod %s failed: %v", podDescr.Name, err) return false } restartCount := pod.Status.Info["liveness"].RestartCount glog.Infof("Restart count of pod %s in namespace %s is now %d", podDescr.Name, ns, restartCount) if restartCount > initialRestartCount { glog.Infof("Restart count of pod %s in namespace %s increased from %d to %d during the test", podDescr.Name, ns, initialRestartCount, restartCount) return true } } glog.Errorf("Did not see the restart count of pod %s in namespace %s increase from %d during the test", podDescr.Name, ns, initialRestartCount) return false } // TestLivenessHttp tests restarts with a /healthz http liveness probe. func TestLivenessHttp(c *client.Client) bool { return runLivenessTest(c, &api.Pod{ ObjectMeta: api.ObjectMeta{ Name:   "liveness-http", Labels: map[string]string{"test": "liveness"}, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:    "liveness", Image:   "kubernetes/liveness", Command: []string{"/server"}, LivenessProbe: &api.Probe{ Handler: api.Handler{ HTTPGet: &api.HTTPGetAction{ Path: "/healthz", Port: util.NewIntOrStringFromInt(8080), }, }, InitialDelaySeconds: 15, }, }, }, }, }) } // TestLivenessExec tests restarts with a docker exec "cat /tmp/health" liveness probe. func TestLivenessExec(c *client.Client) bool { return runLivenessTest(c, &api.Pod{ ObjectMeta: api.ObjectMeta{ Name:   "liveness-exec", Labels: map[string]string{"test": "liveness"}, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:    "liveness", Image:   "busybox", Command: []string{"/bin/sh", "-c", "echo ok >/tmp/health; sleep 10; echo fail >/tmp/health; sleep 600"}, LivenessProbe: &api.Probe{ Handler: api.Handler{ Exec: &api.ExecAction{ Command: []string{"cat", "/tmp/health"}, }, }, InitialDelaySeconds: 15, }, }, }, }, }) } var _ = Describe("TestLivenessHttp", func() { It("should pass", func() { c, err := loadClient() Expect(err).NotTo(HaveOccurred()) Expect(TestLivenessHttp(c)).To(BeTrue()) }) }) var _ = Describe("TestLivenessExec", func() { It("should pass", func() { c, err := loadClient() Expect(err).NotTo(HaveOccurred()) Expect(TestLivenessExec(c)).To(BeTrue()) }) }) </del>

Park of ginkgo-ization
Per IRC, is working on this
Pushed an update yesterday to address comments.
what's the status of this? There's an issue open to de-flake this test (), but it's tough to do that if the code is in the middle of being moved to ginkgo.
This has been merged.


Question: } // Write stores the given "val" in CycleState with the given "key". <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Write(key StateKey, val StateData) { <ins> c.mx.Lock() </ins> c.storage[key] = val <ins> c.mx.Unlock() </ins> } // Delete deletes data with the given key from CycleState. <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Delete(key StateKey) { <del> delete(c.storage, key) } // Lock acquires CycleState lock. func (c *CycleState) Lock() { </del> c.mx.Lock() <del> } // Unlock releases CycleState lock. func (c *CycleState) Unlock() { </del> <ins> delete(c.storage, key) </ins> c.mx.Unlock() } <del> // RLock acquires CycleState read lock. func (c *CycleState) RLock() { c.mx.RLock() } // RUnlock releases CycleState read lock. func (c *CycleState) RUnlock() { c.mx.RUnlock() } </del>

What happened: kube-scheduler () crashes due to a panic while running : The race condition is triggered when an out-of-tree plugin, that writes to during its Filter phase, is registered through . Access to in general happens from a single goroutine and there are no practical concurrency issues, with the exception of the phase, that parallelizes calls for each node across plugins: A quick glance at the tip of the branch seems to indicate the bug is still present there. What you expected to happen: Ideally access to should be synchronized everywhere for correctness, including by all in-tree plugins, since there are no guarantees about how will call multiple plugins concurrently. It seems to me it is something that was considered at some point, given that has methods. Sadly, almost no in-tree plugins are synchronizing access. For now, I will try to work around the problem by doing a single write to during and storing a data bag structure where I will write state into during (with proper synchronization), instead of writing to directly. How to reproduce it (as minimally and precisely as possible): Install a scheduler plugin that writes to during its phase. In-tree (builtin) plugins are not synchronizing their access to . Environment: Kubernetes version (use ): v1.18.8 Cloud provider or hardware configuration: linux-amd64 (AWS) OS (e.g: ): Ubuntu 18.04.5 LTS (Bionic Beaver) Others: custom kube-scheduler binary built with an out-of-tree plugin using
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
AFAICT all in-tree plugins write to the cycle state in PreFilter and then only read from cycle state in Filter. So this shouldn't be a problem with in-tree plugins as concurrent read access should be fine during RunFilterPlugins. But since your out-of-tree plugin writes in Filter, I can see why this can be a problem.


Question: } // Write stores the given "val" in CycleState with the given "key". <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Write(key StateKey, val StateData) { <ins> c.mx.Lock() </ins> c.storage[key] = val <ins> c.mx.Unlock() </ins> } // Delete deletes data with the given key from CycleState. <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Delete(key StateKey) { <del> delete(c.storage, key) } // Lock acquires CycleState lock. func (c *CycleState) Lock() { </del> c.mx.Lock() <del> } // Unlock releases CycleState lock. func (c *CycleState) Unlock() { </del> <ins> delete(c.storage, key) </ins> c.mx.Unlock() } <del> // RLock acquires CycleState read lock. func (c *CycleState) RLock() { c.mx.RLock() } // RUnlock releases CycleState read lock. func (c *CycleState) RUnlock() { c.mx.RUnlock() } </del>

Aside from adding Lock/Unlock in in-tree plugin, offering mutual exclusion on each key as a built-in feature within CycleState could be an alternative solution. This way, reads from in-tree plugins during Filter wouldn't interfere with writes during Filter from out-of-tree plugins.
cc
should we change the Read/Write CycleState methods to be thread safe?
Adding locks without a proper justification would just increase the latency. could you clarify your use case? Why do you need to write anything during Filter?
yeah, we should measure it, but I don't expect it to have significant effect...
Adding locks without a proper justification would just increase the latency. Without contention (which is the case with in-tree plugins), I don't think this is a concern.
Adapting source from : source: essentially, 11.3 ns/Lock-Unlock pair
I remember locks showing up in profiling for PreScore/Score for certain plugins. Although I have to admit that the accesses were not properly implemented. I'd like to hear an answer for before doing any experiments.
+1 to that I'd also like to understand the motivations / requirements of mutating CycleState in Filter().
My particular use-case is that we have resources being dynamically to/removed from Nodes, and those resources need to be allocated (and tracked) in the scheduling loop. Technically I could pre-compute all the available dynamic resources for all Nodes at the phase, and snapshot it, but that would require me to pre-compute everything for all Nodes on all scheduling cycles. Instead, what I'm currently doing is computing what dynamic resources can be utilized on-demand during the phase (kind of a lazy loading of dynamic resources for each pair), and saving state of what extra resources were considered during for each Node that was evaluated. Then, later in the scheduler plugin can lock-in the exact dynamic resources for the particular Node that was selected. I understand the concern with adding extra synchronization overhead without a real need. Even if my use-case is not good enough to justify it, there is another argument for it: technically, the lack of access synchronization on violates the . Even if in the end we decide synchronizing all access is not worth the overhead (for correctness), I would strongly encourage better documentation saying which plugin phases are allowed to modify , and perhaps even forcing a snapshot-read-only copy of to be used by plugins.
Using the scheduler to track resources is not a good idea.


Question: } // Write stores the given "val" in CycleState with the given "key". <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Write(key StateKey, val StateData) { <ins> c.mx.Lock() </ins> c.storage[key] = val <ins> c.mx.Unlock() </ins> } // Delete deletes data with the given key from CycleState. <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Delete(key StateKey) { <del> delete(c.storage, key) } // Lock acquires CycleState lock. func (c *CycleState) Lock() { </del> c.mx.Lock() <del> } // Unlock releases CycleState lock. func (c *CycleState) Unlock() { </del> <ins> delete(c.storage, key) </ins> c.mx.Unlock() } <del> // RLock acquires CycleState read lock. func (c *CycleState) RLock() { c.mx.RLock() } // RUnlock releases CycleState read lock. func (c *CycleState) RUnlock() { c.mx.RUnlock() } </del>

The scheduler should be stateless. Can't you track the resources with an external controller or with a device plugin and making it part of the Node spec?
there may be some misunderstanding in my poor choice of words. What I mean by "tracking" is that the scheduler needs to track usage and availability of these extra per-Node resources, to be able to allocate them during placement/scheduling. Exactly the same it does with volumes, cpus, memory, ephemeral disk, etc Doing computations in a controller outside of the scheduling loop would be similar to "precomputing" these resources. In my particular use case I chose a different way because: introduces another point of failure, if the external controller stops pre-computing extra resources and publishing them somewhere (CRD, Node spec, etc) scheduling halts (at least scheduling on these extra resources) Node resources in my use case are volatile (they are coming and going frequently), and the extra latency by an extra controller (vs doing all computations right in the scheduling loop) may be problematic for my use case. Even if you consider my use case not enough to add proper synchronization to , I'd still encourage making it explicit what scheduling phases are allowed to modify it (and even passing in a snapshot read-only version of to phases that are not expected to update it), to avoid confusion and race conditions.
Ok, if you are not tracking the existence of resources in the scheduler, I wouldn't suggest the external controller. Still, have you considered the device plugin framework? That way, the default kube-scheduler could keep track of resources usage. Otherwise, I still don't see why you would need to update the CycleState during Filter. As its name indicates, CycleState only lives during a scheduling cycle.
Regardless of the correctness of usage of CycleState to do what wants, I still think we need to do at least one of the following: that writing to CycleState during Filter will lead to data races, a read-only CycleState to Filter plugins (whether we do this at the interface level or if we simply drop Write requests during Filter is an implementation detail), or make CycleState concurrency safe CycleState as it exists right now allows plugin authors to shoot themselves in the foot IMO.
I'm leaning towards 1, but it's fair to continue the discussion before deciding.
Can you run the "controller" as a goroutine along with your scheduler plugin?


Question: } // Write stores the given "val" in CycleState with the given "key". <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Write(key StateKey, val StateData) { <ins> c.mx.Lock() </ins> c.storage[key] = val <ins> c.mx.Unlock() </ins> } // Delete deletes data with the given key from CycleState. <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Delete(key StateKey) { <del> delete(c.storage, key) } // Lock acquires CycleState lock. func (c *CycleState) Lock() { </del> c.mx.Lock() <del> } // Unlock releases CycleState lock. func (c *CycleState) Unlock() { </del> <ins> delete(c.storage, key) </ins> c.mx.Unlock() } <del> // RLock acquires CycleState read lock. func (c *CycleState) RLock() { c.mx.RLock() } // RUnlock releases CycleState read lock. func (c *CycleState) RUnlock() { c.mx.RUnlock() } </del>

So it's not an external controller binary, and hence don't introduce another point of failure. It's pretty like how the default scheduler manages its internal cache. But note that cache is only needed when you need a more efficient data structure. If a {key:value} map is good enough, spinning up a XYZInformer and obtain the latest XYZ resource on-demand is the most recommended way. I think it quite depends on how you design the internal XYZ cache. For example, it should only tweak the cache in an incremental manner - in other words, unchanged resource will be kept unmodified. You can check out the "doubly-pointer snapshot" of scheduler's cache. With above said, if I understand your case correctly, you may need to build a sophisticated XYZ cache, which runs in a separate goroutine and responds to your XYZ resource changes. And then upon each scheduling cycle, take a snapshot (may needs a specific snapshotting algorithm) in PreFilter() and WRITE to CycleState, and then in Filter() READ from CycleState. My personal option is to suggest all plugin developers to read from CycleState in Filter().
nailed it. What you describe is almost exactly what I’m doing today. There are some details particular to my use case (e.g. just tracking current state with an informer and an incremental cache is not enough, because there’s some logic involved in deciding if currently available resources are usable by each pair, and I currently do that computation in as pairs are evaluated). In any case, I just wanted to share my use case to provide some color, but I don’t want what I’m doing distracting us from the more important questions, mentioned by above. My personal preference would be as a very short term next step, then when someone has cycles to do it. I would be OK with punting on until a more strong need shows up.
Note that limiting to only reading from works for me, as long as I can still read my own data structure from it and do writes on it (i.e. don’t deepclone for the phase).
also adds overhead (from copying the state, which can be very expensive).
Could implement just by changing the interface passed to (e.g. ). That wouldn’t be backwards compatible, but very cheap.


Question: } // Write stores the given "val" in CycleState with the given "key". <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Write(key StateKey, val StateData) { <ins> c.mx.Lock() </ins> c.storage[key] = val <ins> c.mx.Unlock() </ins> } // Delete deletes data with the given key from CycleState. <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Delete(key StateKey) { <del> delete(c.storage, key) } // Lock acquires CycleState lock. func (c *CycleState) Lock() { </del> c.mx.Lock() <del> } // Unlock releases CycleState lock. func (c *CycleState) Unlock() { </del> <ins> delete(c.storage, key) </ins> c.mx.Unlock() } <del> // RLock acquires CycleState read lock. func (c *CycleState) RLock() { c.mx.RLock() } // RUnlock releases CycleState read lock. func (c *CycleState) RUnlock() { c.mx.RUnlock() } </del>

I guess that would work in the sense that you cannot change the high level values of the CycleState map, but you could still override what's inside them, if they have pointers. But maybe that's ok.
but you could still override what's inside them, if they have pointers. But maybe that's ok. yeah, if it was pointers and if the underlying data needed to be modified, I'd imagine it would have its own Lock/Unlock semantics
for such implementation, the only remaining concern is backwards compatibility. So I would leave that decision to who leads the scheduling-plugins repo.
I rethink 's original problem a little bit: the original racing issue occurs when reading CycleState in in-tree plugins, this indicates one of the (out-of-tree) plugins is (over)writing the most high-level value of the CycleState map. But if we control the locking inside the out-of-tree plugin, and mutate that particular in-place, that would be good enough? Case 3 in the above program demonstrates the correct way to manipulate the specific out-of-tree state. Let me know your thoughts. If it resolves 's concern, we don't need to make any change on the Filter() interface.
Case 3 is how I am working around the problem today to avoid the race. I write a "data bag" once into top level in , then only modify my own inner structure (with proper locking) in , without doing any further modifications to directly (top level) in . That is acceptable, but I'd encourage you to consider making the interface (and documentation) more explicit for plugins, so they avoid the mistake of writing directly into . Without changing anything (no documentation, no readonly interface), it is pretty easy for a plugin to use in a way that you don't want to support.
Thanks for confirming that. Yes, documenting it, or(and) publishing examples would be quite helpful to clarify the semantics. As a wrap-up: a keyed value can be read from CycleState, literally , in any phase. It's always race-free.


Question: } // Write stores the given "val" in CycleState with the given "key". <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Write(key StateKey, val StateData) { <ins> c.mx.Lock() </ins> c.storage[key] = val <ins> c.mx.Unlock() </ins> } // Delete deletes data with the given key from CycleState. <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Delete(key StateKey) { <del> delete(c.storage, key) } // Lock acquires CycleState lock. func (c *CycleState) Lock() { </del> c.mx.Lock() <del> } // Unlock releases CycleState lock. func (c *CycleState) Unlock() { </del> <ins> delete(c.storage, key) </ins> c.mx.Unlock() } <del> // RLock acquires CycleState read lock. func (c *CycleState) RLock() { c.mx.RLock() } // RUnlock releases CycleState read lock. func (c *CycleState) RUnlock() { c.mx.RUnlock() } </del>

if you want to mutate the top-level keyed value in CycleState, if it's a phase that CycleState is accessed consecutively, such as PreFilter, PreScore, it's safe to mutate CycleState in any manner if it's a phase that CycleState is accessed parallelly, to be race-free, (for now) you only have 1 option: keep the keyed value's pointer unchanged, and use an internal lock to mutate that pointer obj [TBD] we may come up with changes to add read locks to all in-tree plugins when reading values from CycleState, so that the out-of-tree plugin can surround its mutating logic between and - th mutating logic thereby can be any style Any thoughts on the TBD item? I would assume read locks wouldn't cost too much.
For now, I think the documentation is correct, it clearly says Read() and Write() are not thread-safe: The key thing here is whether in-tree plugin should use RLock/RUnlock or not. If yes, out-of-tree plugins can be freed to use Lock/Unlock. (see the TBD item in the previous comment)
I like the TBD, and optimizing for correctness in how in-tree plugins are accessing . I understand the concern with synchronization overhead, but I would be really surprised if it is the bottleneck in scheduling code.
It can be, when you have 5k-nodes clusters, or bigger.
I don't think it will be expensive, a rlock should be equivalent to an atomic add operation (basically it increments the number of readers). As a comparison, looking up the state from the map is likely to be more expensive.
Of course we are assuming no contention in the default plugins.
This is my impression as well. A vanilla version of scheduler doesn't impose write lock in Filter() and Score(), so should be good.
We can always see with the benchmarks and revert if its not negligible.
I will run the benchmarks to see if there is perf downgrade. /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale This is still relevant, yes?


Question: } // Write stores the given "val" in CycleState with the given "key". <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Write(key StateKey, val StateData) { <ins> c.mx.Lock() </ins> c.storage[key] = val <ins> c.mx.Unlock() </ins> } // Delete deletes data with the given key from CycleState. <del> // This function is not thread safe. In multi-threaded code, lock should be // acquired first. </del> <ins> // This function is thread safe by acquiring an internal lock first. </ins> func (c *CycleState) Delete(key StateKey) { <del> delete(c.storage, key) } // Lock acquires CycleState lock. func (c *CycleState) Lock() { </del> c.mx.Lock() <del> } // Unlock releases CycleState lock. func (c *CycleState) Unlock() { </del> <ins> delete(c.storage, key) </ins> c.mx.Unlock() } <del> // RLock acquires CycleState read lock. func (c *CycleState) RLock() { c.mx.RLock() } // RUnlock releases CycleState read lock. func (c *CycleState) RUnlock() { c.mx.RUnlock() } </del>

IIUC the volumebinding plugin has been fixed, but external plugins can still shoot themselves in the foot.
/priority backlog
Yes. I'm working on this, late this week I will send out a PR along with perf test result.


Question: creationTimestamp: null labels: kubernetes.io/bootstrapping: rbac-defaults <del> name: system:csi-external-attacher rules: - apiGroups: - "" resources: - persistentvolumes verbs: - get - list - patch - update - watch - apiGroups: - "" resources: - nodes verbs: - get - list - watch - apiGroups: - storage.k8s.io resources: - volumeattachments verbs: - get - list - patch - update - watch - apiGroups: - "" resources: - events verbs: - create - get - list - patch - update - watch - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" creationTimestamp: null labels: kubernetes.io/bootstrapping: rbac-defaults name: system:csi-external-provisioner rules: - apiGroups: - "" resources: - persistentvolumes verbs: - create - delete - get - list - watch - apiGroups: - "" resources: - persistentvolumeclaims verbs: - get - list - patch - update - watch - apiGroups: - storage.k8s.io resources: - storageclasses verbs: - list - watch - apiGroups: - "" resources: - events verbs: - create - get - list - patch - update - watch - apiGroups: - "" resources: - nodes verbs: - get - list - watch - apiGroups: - storage.k8s.io resources: - csinodes verbs: - get - list - watch - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: "true" creationTimestamp: null labels: kubernetes.io/bootstrapping: rbac-defaults </del> name: system:discovery rules: - nonResourceURLs:

<!-- Please only use this template for submitting enhancement requests --The built-in and cluster roles here are deprecated in 1.13 when this issue was resolved: /sig storage /milestone v1.17 /priorit important-longterm
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please only use this template for submitting enhancement requests -- /milestone v1.17 /priorit important-longterm Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /priority important-longterm
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /assign
Bug triage for 1.17 here with a gentle reminder that code freeze for this release is on November 14. Is this issue still intended for 1.17?


Question: "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/watch:go_default_library", "//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library", <ins> "//staging/src/k8s.io/client-go/dynamic:go_default_library", </ins> "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/tools/cache:go_default_library", "//staging/src/k8s.io/client-go/tools/watch:go_default_library",

[x] Original approval issue: - Reverted due to flakes [x] Two weeks soak end date: 20 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Pod endpoints which are untested. a Pod with a static label the Pod with a new Label and updated data the Pod to ensure it's patched the Pod's status the Pod's status Ready condition to False the Pod's status to check if Ready is False all Pods in all Namespaces find the Pod(1) ensure that the Pod is found and is patched Namespaced Pod(1) via a Collection with a LabelSelector the Pod to check that it's deleted Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/reopen Test PR Merge Promotion PR to be created next
Reopened this issue. <detailsIn response to : /reopen Test PR Merge Promotion PR to be created next Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
reverted by due to flaky test
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/reopen PR on test grid for 2 weeks before promotion to conformance
Reopened this issue. <detailsIn response to : /reopen PR on test grid for 2 weeks before promotion to conformance Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: func (s *sourceFile) processEvent(e *inotify.Event) error { // Ignore file start with dots <del> if strings.HasPrefix(e.Name, ".") { </del> <ins> if strings.HasPrefix(filepath.Base(e.Name), ".") { </ins> glog.V(4).Infof("Ignored pod manifest: %s, because it starts with dots", e.Name) return nil }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? static manifest Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Bare metal and reproduced in Vagrant OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: custom scratch install and reproduced with kubeadm Others: Calico for networking. Used (Also able to reproduce without calico) What happened: Modified one of the static manifest files in . The kubelet killed the running pod and did NOT create a new pod. (waited 30 minutes, still no pod) What you expected to happen: Kubelet should have killed the pod and started a new one. How to reproduce it (as minimally and precisely as possible): Started with single node, vanilla Ubuntu Xenial, installed kubernetes via kubeadm, nothing special. Modify , for example change the logging level or even just whitespace at the end of the file. Observe the kubelet kill the controller manager pod, and does not restart it. Anything else we need to know: If I remove the file from the directory, and then replace the file, kubelet will start it up. Kubelet logs from the period where I updated the file:
Did you modify the yaml file using an editor (e.g., vim)? If the editor creates a temporary file in the same directory, kubelet can get confused and thinking that's a new pod (with the same name & namespace). When the temporary file gets removed, kubelet'd inadvertently delete the pod from its cache. Since kubelet relies on inotify for changes, if it doesn't get notified for any new changes from the original yaml file, it'd forever ignore the pod. There is an issue open to add periodic listing in addition to inotify (introduced in 1.5). This would make sure kubelet eventually recover from the situation. I can't find the issue, but there is another one for inotify: . Another fix is to make sure kubelet ignore such known temporary files (it already ignores files starting with ). In general, don't edit the file directly.


Question: func (s *sourceFile) processEvent(e *inotify.Event) error { // Ignore file start with dots <del> if strings.HasPrefix(e.Name, ".") { </del> <ins> if strings.HasPrefix(filepath.Base(e.Name), ".") { </ins> glog.V(4).Infof("Ignored pod manifest: %s, because it starts with dots", e.Name) return nil }

Copy the file to another place, edit, and copy it back to the manifests directory.
I'm seeing a similar problem except I never edited the file, might be fit for a separate issue
Yes I was using vim, and this explanation makes sense. Would be nice if it was smart about it. Would a PR to ignore the vim temp file be acceptable
I think it's reasonable. I doubt if anyone uses those file names on purpose.
The dot ignore should have covered the vim temp file too, but it doesn't work because the filename passed in is the absolute filename including the path. So the first character is not a dot. I'll try to submit a PR fix.
PR submitted
fixed?
How to work around? I coudn't start api server when using kubeadm to deploy k8s. Thanks!


Question: return nil, err } defer os.RemoveAll(tmpDir) <del> out, err := exec.Command("nice", "-n", "19", "du", "-s", "-B", "1", tmpDir).CombinedOutput() </del> <ins> out, err := exec.Command("nice", "-n", "19", "du", "-x", "-s", "-B", "1", tmpDir).CombinedOutput() </ins> if err != nil { return nil, fmt.Errorf("failed command 'du' on %s with error %v", tmpDir, err) }

I'm using K8S 1.10 and have a specific use case where I'm trying to use an emptyDir to share mounts between two containers in a pod. I'm using the MountPropagation feature successfully, but while debugging another issue I noticed that the node was running a very long running du command. Looking into the source I noted: Which runs du on any emptyDir, do measure disk usage. I recommend that in this case we should add the -x flag that will skip any mounted directories as they do not count against the file usage and would make these scans a lot faster. I could of course add a simple pull request for this but first wanted to see if this was known and on purpose. To reproduce bind mount/loop mount/device mount any large/slow directory into an emptyDir volume and watch as the du process slogs away Environment: Kubernetes 1.10 GKE <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think it makes sense to ignore submounts.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
do you still want to submit a PR to fix this? Thanks!
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: return nil, err } defer os.RemoveAll(tmpDir) <del> out, err := exec.Command("nice", "-n", "19", "du", "-s", "-B", "1", tmpDir).CombinedOutput() </del> <ins> out, err := exec.Command("nice", "-n", "19", "du", "-x", "-s", "-B", "1", tmpDir).CombinedOutput() </ins> if err != nil { return nil, fmt.Errorf("failed command 'du' on %s with error %v", tmpDir, err) }

/lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: restful "github.com/emicklei/go-restful" cadvisorapi "github.com/google/cadvisor/info/v1" <ins> "github.com/pkg/errors" </ins> "k8s.io/klog" "k8s.io/api/core/v1"

What happened: In handler#handleSummary, return value from request.Request.ParseForm() is ignored. What you expected to happen: The error return should be checked.
/sig node
Hi this seems to be easy to fix. Could I solve this?
Please go ahead.


Question: "//federation/apis/federation:go_default_library", "//pkg/api:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", <ins> "//vendor/k8s.io/apimachinery/pkg/util/validation/field:go_default_library", </ins> ], )

Right now it is possible to add a cluster for which it won't be possible to create a client. For example client-related code does: while validation checks only: cc:
switching this to 1.6 as it's too late for 1.5. ok? (please switch it right back if you disagree)
Not a release blocker for 1.7.
are you interested in fixing this?
of course, will do that.
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.8 milestone for more than 7 days: kind: Must specify exactly one of [, , ]. priority: Must specify exactly one of [, , ]. Removing it from the milestone. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: }, }, }, <del> }) </del> <ins> }, true) }) It("should *not* be restarted with a docker exec "cat /tmp/health" liveness probe", func() { runLivenessTest(c, &api.Pod{ ObjectMeta: api.ObjectMeta{ Name:   "liveness-exec", Labels: map[string]string{"test": "liveness"}, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:    "liveness", Image:   "gcr.io/google_containers/busybox", Command: []string{"/bin/sh", "-c", "echo ok >/tmp/health; sleep 600"}, LivenessProbe: &api.Probe{ Handler: api.Handler{ Exec: &api.ExecAction{ Command: []string{"cat", "/tmp/health"}, }, }, InitialDelaySeconds: 15, }, }, }, }, }, false) </ins> }) It("should be restarted with a /healthz http liveness probe", func() {

As far as I can tell, container liveness probes of type exec flat out don't work against clusters built at head. I had the kubelet always log the output of the exec'ed command, and it's always empty no matter what I exec, causing the probe to fail since it requires the output to be equal to the string "ok". If you want to test it, try adding a liveness probe of this form to any container: I also tried these lovely forms, to more closely match the liveness example under examples/: On the plus side, this may be an opportunistic time to switch over to
The reason that the e2e test for exec liveness probes hasn't caught this is that it only tests that containers are restarted when they fail liveness probes. It doesn't test that containers aren't restarted when their liveness checks should be succeeding. The e2e test's exec'ed commands also only ever produced empty output, rather than the expected "fail" output.
Is it possible to include this in a new pre-release, since it fixes a bug in health checking? I see it merged in master and I'm wondering what the policy is of releasing new code in a separate (0.16.*) release.


Question: `service`, a client can simply connect to $MYAPP_SERVICE_HOST on port $MYAPP_SERVICE_PORT. <ins> ## Service without selector Services, in addition to providing clean abstraction to access pods, can also abstract any kind of backend: - you want to have an external database cluster in production, but in test you use your own databases. - you want to point your service to a service in another [`namespace`](namespaces.md) or on another cluster. - you are migrating your workload to Kubernetes and some of your backends run outside of Kubernetes. In any of these scenarios you can define a service without a selector: ```json "kind": "Service", "apiVersion": "v1beta1", "id": "myapp", "port": 8765 ``` then you can explicitly map the service to a specific endpoint(s): ```json "kind": "Endpoints", "apiVersion": "v1beta1", "id": "myapp", "endpoints": ["173.194.112.206:80"] ``` Access to the service without a selector works the same as if it had selector. The traffic will be routed to endpoints defined by the user (`173.194.112.206:80` in case of this example). </ins> ## How do they work? Each node in a Kubernetes cluster runs a `service proxy`.  This application

For example, kubernetes-ro service appears with empty selector (this service was created by the hack/local cluster) - is that ok? What meaning such service has if it has no entity to fwd traffic to?
kubernetes and kubernetes-ro are currently "virtual" - they are not backed by running pods, but by the master itself. , abonas wrote:
if that's internal impl, why is it exposed to user? let's say a client pulls all entities and builds a UI around it. then a service with no selector arrives, and it's internal to kube - how the client should know not to present it/touch it? (similar to the "net" container in pods)
It's exposed exactly FOR users - so they can find the apiserver. It has a corresponding Endpoints object, it gets a portal IP and a kube-proxy rule, and it gets service env vars and a DNS name. The fact that it is virtual is an implementation detail. That it exists is all users should need to know. , abonas wrote:
And FWIW, the master components may one day actually be pods... , Tim Hockin wrote:
thanks for the explanation. so any general rule how to distinguish kube pods/services from user defined?
Do we need to distinguish them? We could add labels, I guess, if it is really important to distinguish them.
sometimes users would want to see only the things they defined and not implementation details of the system.
One could argue this is why the users should be using labels. I'm not against defining labels for "special" stuff, but it's new territory, so we should enter with caution. , abonas wrote:
I think the right answer might be to put them in a special "cluster" or "system" namespace. , Tim Hockin wrote:
See If we put them in a "kubernetes" ns, the DNS name would be :) I'm not against segmenting them into a different namespace. We just need to decide on what that namespace is called, plumb through that configuration, decide on compatibility for a short while or not, and decide if that namespace is "special" or just "different" - i.e. should the skydns, heapster, etc pods go into this namespace? , Brendan Burns wrote:
As said, the services are there for applications that need to contact the apiserver.


Question: `service`, a client can simply connect to $MYAPP_SERVICE_HOST on port $MYAPP_SERVICE_PORT. <ins> ## Service without selector Services, in addition to providing clean abstraction to access pods, can also abstract any kind of backend: - you want to have an external database cluster in production, but in test you use your own databases. - you want to point your service to a service in another [`namespace`](namespaces.md) or on another cluster. - you are migrating your workload to Kubernetes and some of your backends run outside of Kubernetes. In any of these scenarios you can define a service without a selector: ```json "kind": "Service", "apiVersion": "v1beta1", "id": "myapp", "port": 8765 ``` then you can explicitly map the service to a specific endpoint(s): ```json "kind": "Endpoints", "apiVersion": "v1beta1", "id": "myapp", "endpoints": ["173.194.112.206:80"] ``` Access to the service without a selector works the same as if it had selector. The traffic will be routed to endpoints defined by the user (`173.194.112.206:80` in case of this example). </ins> ## How do they work? Each node in a Kubernetes cluster runs a `service proxy`.  This application

Additionally, nil selectors can be used by users to import external services. Adding labels to the kubernetes services won't help. Users need to manage their objects using labels. /cc
Yeah, lots of consumers will expose "special" services. IN fact, in many clusters the "special" services will outweigh the unspecial ones. At this point, the namespace they are in is completely arbitrary, and I think because it is a name, it should be free for the end user to choose where they are surfaced.
Per the proposal here: , I think the namespace should continue to hold these services, and they require no special labels. I sense comment is made without thinking through what Kubernetes operators do to bootstrap a cluster. I think it's reasonable that namespace holds everything you need to run, and operators may or may not choose to segment their data in other namespaces. If they want to not see this information, its an exact reason for why they should use their own namespace. I don't think we need anything new here.
- I'm not clear on your meaning. Did you mean that we should make the "cluster admin" namespace a config param that everyone knows about and can be different cluster-to-cluster? Or did you mean that default is as arbitrary a name as any other? I'd be fine moving things out of default into "kube" or "kubernetes" or "cluster" or something, but I am less convinced that a flag is a good thing. , Clayton Coleman wrote:
- wherever we stick this stuff, it should be the namespace that is initially bootstrapped, which today is . I continue to believe that is as good a name as any other. I do not see a need to do more here.
I think as soon as you bring naming into the picture, making decisions based on names gets very tricky (what if I want my logging service to be everywhere?). If there is a magic namespace that exposes things to the rest of the cluster, we should ask whether there's only one magic namespace, or multiple? Are services supposed to cross namespaces? It's great that pods can find the master... but that's not the only thing they'll need to find in each namespace. And baking in special names (or customizable names) into your app code is fragile, because then your code only works inside Kubernetes.


Question: `service`, a client can simply connect to $MYAPP_SERVICE_HOST on port $MYAPP_SERVICE_PORT. <ins> ## Service without selector Services, in addition to providing clean abstraction to access pods, can also abstract any kind of backend: - you want to have an external database cluster in production, but in test you use your own databases. - you want to point your service to a service in another [`namespace`](namespaces.md) or on another cluster. - you are migrating your workload to Kubernetes and some of your backends run outside of Kubernetes. In any of these scenarios you can define a service without a selector: ```json "kind": "Service", "apiVersion": "v1beta1", "id": "myapp", "port": 8765 ``` then you can explicitly map the service to a specific endpoint(s): ```json "kind": "Endpoints", "apiVersion": "v1beta1", "id": "myapp", "endpoints": ["173.194.112.206:80"] ``` Access to the service without a selector works the same as if it had selector. The traffic will be routed to endpoints defined by the user (`173.194.112.206:80` in case of this example). </ins> ## How do they work? Each node in a Kubernetes cluster runs a `service proxy`.  This application

I'm just trying to pump the brakes on special services prior to us designing them... :) ----- Original Message -----
oh, to be clear, I don't want "special" services - I am merely asking if moving non-user-defined objects (where that is defined very loosely) out of "default" is worthwhile. I tend to agree with Derek that it's not worth energy. , Clayton Coleman wrote:
agree with you that lots of consumers will expose "special" services We should not use the (namespace + k8s-service + skydns) mechanism for publishing the location of those special services. We should decouple the choice of namespace to run the pods of the service in from the well-known dns address of special services. Otherwise, it will be hard to move the pods of the service to another namespace, which people will definitely want to do as they grow. Instead, we need a place to publish well-known services which is not tied to the namespace of the pods of that service.
you open a whole new topic. I've noodled with the idea of a Name resource type. We would generate a Name for each Service as "service.namespace", but users could also add their own Name resources. , Eric Tune wrote:
I'd like to understand better your comment regarding nil selectors used to export external services - can you elaborate? give an example? my understanding is that service redirects traffic to pods based on labels. with nil selector service does nothing, so why is it a service in the first place? I'm pro decoupling special services/internal implementation services from user defined entities. whether by separate namespace, or by saying that the default namespace is reserved, or by any other solution. because otherwise internal implementation and user defined content are mixed and that's no good for a proper management. just tagging you so you could see this issue and the related discussions.
Services abstract backends - they don't have to abstract pods. Some examples: You want to have an external database cluster in production, but in test you use your own databases. You would have a service for both and use the environment variables, but the production service has a nil selector with its endpoints set to the database cluster. You want to have an empty service because you're reserving a name (that isn't accessible yet) but will be in the future You want to point your service to a service in another namespace or on another cluster.


Question: `service`, a client can simply connect to $MYAPP_SERVICE_HOST on port $MYAPP_SERVICE_PORT. <ins> ## Service without selector Services, in addition to providing clean abstraction to access pods, can also abstract any kind of backend: - you want to have an external database cluster in production, but in test you use your own databases. - you want to point your service to a service in another [`namespace`](namespaces.md) or on another cluster. - you are migrating your workload to Kubernetes and some of your backends run outside of Kubernetes. In any of these scenarios you can define a service without a selector: ```json "kind": "Service", "apiVersion": "v1beta1", "id": "myapp", "port": 8765 ``` then you can explicitly map the service to a specific endpoint(s): ```json "kind": "Endpoints", "apiVersion": "v1beta1", "id": "myapp", "endpoints": ["173.194.112.206:80"] ``` Access to the service without a selector works the same as if it had selector. The traffic will be routed to endpoints defined by the user (`173.194.112.206:80` in case of this example). </ins> ## How do they work? Each node in a Kubernetes cluster runs a `service proxy`.  This application

You set the endpoints for your service to the publicIPs of the other service. ----- Original Message -----
, thanks. regarding the last point - I was under impression (based on some other discussion) that there would not be cross namespace interaction? that entities can can access only entities in the same namespace?
Cross-namespace is currently allowed and I think has to be allowed, though perhaps (in the fullness of time) that can be policy based rather than wide open. , abonas wrote:
please see this correspondence. my understanding is that there is isolation
Responded there. I think there's some less-than-crystal-clear words in play. , abonas wrote:
+1 to Eric's comment about ensuring there is a consistent manner for system services like kubernetes-ro, if that's not fixes for all clusters it's going to be a nightmare to write reusable containers for multiple different clusters. I don't care if we do that by using a constant namespace, or by giving system services two different names, but there needs to be at least one name that is constant. Regarding default vs. cluster namespace, I think the issue is one of clutter as a user is starting to understand k8s, you turn up a new cluster, launch a pod, or service, and it's hard to find your one object on the sea of system created pods and services. It's visual clutter that confuses people as they onboard. On Jan 23, 2015 3:01 PM, "Tim Hockin" wrote:
- can you drop me a line (my github name at google) - I want to ask you something, and you have no contact info :) , abonas wrote:
sorry for naive question but why would I care as an user that wants to run its existing workload on Kubernetes about kubernetes-ro ? If I write an application that uses kubernetes-ro then I'm locked into Kubernetes. I understand the need for kubernetes-ro being accessible by cluster level services such as monitoring, logging. I also can imagine that a power user might want to start its own cluster services that use kubernetes-ro. I don't see clearly why would majority of users care about kubernetes-ro being visible from their container.