Question: serviceAccountName: csi-mock containers: - name: csi-provisioner <del> image: registry.k8s.io/sig-storage/csi-provisioner:v3.3.0 </del> <ins> image: registry.k8s.io/sig-storage/csi-provisioner:v3.4.0 </ins> args: - "--csi-address=$(ADDRESS)" # Topology support is needed for the pod rescheduling test

Seen in pull-kubernetes-e2e-gce-cos-alpha-features Specific job failure
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from the error message I don't know if it refers to the sig-storage error or to the sig-auth one
/sig storage
/kind failing-test
/assign From : ... I am not sure if it's related:
is the culprit. cannot be used inside a . This only gets caught when the test actually runs, otherwise Ginkgo never sees the incorrect .
I don’t understand why sometimes it can get the pv by namebut sometimes it failed
but then is a sig-auth test , not storage , sorry /sig auth this seems that was recently in
/remove sig-storage
/assign /priority important-soon
/unassign
is still failed. seems not fixing the whole problem. /reopen feel free to close if the ci is green.
Reopened this issue. <detailsIn response to : is still failed. /reopen feel free to close if the ci is green. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the failure now looks like it is just a sig storage issue:
cc
/assign looks like the new test is still an issue
I think this line is the problem: When the test creates the object, it may take time for the PV to be provisioned and to be bound to the PVC. So you should do another before this call so you can get the latest PVC object with the bound PV.
I opened to get PVC again before using .
Only do PersistentVolumeClaims.Get() is not enough, whether we need to wait for sometimes to wait it bound? because sometimes the testcase is successful and sometime is failed.
If the Pod was able to run, then the PVC has to be bound.
so I can't understand why sometimes the testcase is successful and sometimes is failed.


Question: serviceAccountName: csi-mock containers: - name: csi-provisioner <del> image: registry.k8s.io/sig-storage/csi-provisioner:v3.3.0 </del> <ins> image: registry.k8s.io/sig-storage/csi-provisioner:v3.4.0 </ins> args: - "--csi-address=$(ADDRESS)" # Topology support is needed for the pod rescheduling test

By default, the DelayBinding is false, it will wait for PVC bound during creation. This is weird.
It waits for the claim to be bound, but the test case still needs to refresh its pvc object.
Because of the refresh is async, so after the pvc was bound, we need to get the pvc again, isn't it?


Question: w.Flush() if trackingWriter.Written == 0 && !o.IgnoreNotFound && len(allErrs) == 0 { // if we wrote no output, and had no errors, and are not ignoring NotFound, be sure we output something <del> fmt.Fprintln(o.ErrOut, "No resources found.") </del> <ins> noResourcesFoundFormat := fmt.Sprintf("No resources found in %s namespace.", o.Namespace) fmt.Fprintln(o.ErrOut, noResourcesFoundFormat) </ins> } return utilerrors.NewAggregate(allErrs) }

So I started using namespaces for the first time, and I ran: nothing was showing up...I was really confused, debugging stuff for 30+ minutes.. finally I was like, even tho this is really dumb, maybe they are behind a namespace, so I did: and finally my pods showed up. so yes there is: but this should really be the default, shouldn't it? You will save people a lot of time if you make this fix. tbh, this is a really crappy cli design in this case.
While this might sound like a good idea if you're running a single deployment with a few pods, once you start running multiple dozens of deployments and objects it starts to become less and less useful to have this become the default. I also suspect exposing all objects without specifying namespace is probably a non-goal. Namespaces were created for a reason which is to logically segment resources together. In my opinion, what you are suggesting just nullifies the entire purpose of namespaces.
I think this is more of a usability bug. kubectl get pods gives the list of all pods in default namespace.By default no pods are present in default namespace. So it does give a message saying No Resources found. Improving this message a little bit will be helpful like No Resources found in default namespace
/sig cli
I could implement this if that's alright with you !
sure, how about
We should wait for kubernetes members to comment if this change should be taken up
The title of this issue could use an update, if the priority has been shifted to improving the output of an empty namespace. Maybe it's early to claim the consensus seems to be that --all-namespaces should not be the default, but I personally agree that it shouldn't be. Take note of the shorthand , to make typing it a bit easier, or the option to to something other than to get around typing your namespace altogether.
:) IMHO, is innocent. But we could make the messages more clear. For example, will show also. And is a bit long, but a is too simple to describe our intent, "all namespaces of cluster or all some resource of some namespace", so puzzled. My conclusion: is competent, namespace is ok, making more clear message is enough.
"~" is already implemented as in , in .


Question: w.Flush() if trackingWriter.Written == 0 && !o.IgnoreNotFound && len(allErrs) == 0 { // if we wrote no output, and had no errors, and are not ignoring NotFound, be sure we output something <del> fmt.Fprintln(o.ErrOut, "No resources found.") </del> <ins> noResourcesFoundFormat := fmt.Sprintf("No resources found in %s namespace.", o.Namespace) fmt.Fprintln(o.ErrOut, noResourcesFoundFormat) </ins> } return utilerrors.NewAggregate(allErrs) }

Not sure if I came across as suggesting a new shorthand or if I misread your comment, but for clarity's sake there it is.
the problem is that the nixos-rebuild command is not available on non-NixOS linux machines. However, the underlying issue is that if we run we get this error:


Question: import ( "fmt" <ins> "io/ioutil" </ins> "os" "runtime"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: azure file plugin on Windows does not work after node restart, this is due to powershell cmdlet has lost account name/key after reboot, we should remove the invalid link and do the mount again after kubelet restart. What you expected to happen: azure file plugin on Windows should work after node restart How to reproduce it (as minimally and precisely as possible): has created a pod with azure file mount on Windows agent node could get following error: Anything else we need to know?: Environment: Kubernetes version (use ): v1.9, v1.10 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign /sig windows /sig azure
Added to v1.10 milestone since it is a critical bug fix. cc


Question: return volumesToReportAttached } <ins> func (asw *actualStateOfWorld) GetNodesToUpdateStatusFor() map[types.NodeName]nodeToUpdateStatusFor { return asw.nodesToUpdateStatusFor } </ins> func getAttachedVolume( attachedVolume *attachedVolume, nodeAttachedTo *nodeAttachedTo) AttachedVolume {

Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: $ kubectl describe pods test-ipj93 Name: test-ipj93 Namespace: default Node: kubernetes-minion-group-yxra/10.128.0.3 Start Time: Mon, 28 Nov 2016 14:39:28 -0800 Labels: <noneStatus: Pending IP: Controllers: <noneContainers: busybox: Container ID: Image: busybox Image ID: Port: Command: sh -c while true; do echo (hostname)-(cat /proc/stat cut -f 4 -d" "); sleep 30; done QoS Tier: cpu: Burstable memory: BestEffort Requests: cpu: 100m State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test ReadOnly: false default-token-kjoqg: Type: Secret (a volume populated by a Secret) SecretName: default-token-kjoqg Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 57m 57m 1 {default-scheduler } Normal Scheduled Successfully assigned test-ipj93 to kubernetes-minion-group-yxra 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulling pulling image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulled Successfully pulled image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Created Created container with docker id ; Security:[seccomp=unconfined] 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Started Started container with docker id 47m 47m 6 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.


Question: return volumesToReportAttached } <ins> func (asw *actualStateOfWorld) GetNodesToUpdateStatusFor() map[types.NodeName]nodeToUpdateStatusFor { return asw.nodesToUpdateStatusFor } </ins> func getAttachedVolume( attachedVolume *attachedVolume, nodeAttachedTo *nodeAttachedTo) AttachedVolume {

SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig.


Question: return volumesToReportAttached } <ins> func (asw *actualStateOfWorld) GetNodesToUpdateStatusFor() map[types.NodeName]nodeToUpdateStatusFor { return asw.nodesToUpdateStatusFor } </ins> func getAttachedVolume( attachedVolume *attachedVolume, nodeAttachedTo *nodeAttachedTo) AttachedVolume {

This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes


Question: } } <ins> func proxyRedirectsforRootPath(path string, w http.ResponseWriter, req *http.Request) bool { redirect := false method := req.Method // From pkg/genericapiserver/endpoints/handlers/proxy.go#ServeHTTP: // Redirect requests with an empty path to a location that ends with a '/' // This is essentially a hack for http://issue.k8s.io/4958. // Note: Keep this code after tryUpgrade to not break that flow. if len(path) == 0 && (method == http.MethodGet || method == http.MethodHead) { var queryPart string if len(req.URL.RawQuery) > 0 { queryPart = "?" + req.URL.RawQuery } w.Header().Set("Location", req.URL.Path+"/"+queryPart) w.WriteHeader(http.StatusMovedPermanently) redirect = true } return redirect } </ins> // ServeHTTP handles the proxy request func (h *UpgradeAwareHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) { if h.tryUpgrade(w, req) {

What happened: As part of creating an e2e test for , it was found that the apiserver proxy server is processing a blank redirect for any for 21 endpoints across pods, services and nodes. For a pod, redirecting for any method other than GET or HEAD is a bug, you expected to happen: This is the current behaviour for the apiserver but with a minimal code change the redirect can apply to only the following endpoints; - - - How to reproduce it (as minimally and precisely as possible): Create any standard pod and service in the default namespace run in another terminal the response from running the following commands produces a can be replaced with other http verbs , , , , and which also produce 301 responses. Anything else we need to know?: networking /sig testing /sig architecture
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: - - - run in another terminal the response from running the following commands produces a testing /sig architecture Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Does this issue and PR address your concerns about the apiserver proxy server doing a blanket direct of any http call to the root proxy endpoints?
commented on the associated PR, thanks
/remove-sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Will be addressed late in 1.21
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: } } <ins> func proxyRedirectsforRootPath(path string, w http.ResponseWriter, req *http.Request) bool { redirect := false method := req.Method // From pkg/genericapiserver/endpoints/handlers/proxy.go#ServeHTTP: // Redirect requests with an empty path to a location that ends with a '/' // This is essentially a hack for http://issue.k8s.io/4958. // Note: Keep this code after tryUpgrade to not break that flow. if len(path) == 0 && (method == http.MethodGet || method == http.MethodHead) { var queryPart string if len(req.URL.RawQuery) > 0 { queryPart = "?" + req.URL.RawQuery } w.Header().Set("Location", req.URL.Path+"/"+queryPart) w.WriteHeader(http.StatusMovedPermanently) redirect = true } return redirect } </ins> // ServeHTTP handles the proxy request func (h *UpgradeAwareHandler) ServeHTTP(w http.ResponseWriter, req *http.Request) { if h.tryUpgrade(w, req) {

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle stale
/remove-lifecycle stale


Question: By("Removing the test directory") removeCmd := fmt.Sprintf("rm -r %s", path.Join(hostBase, discoveryDir)) <del> _, err := framework.NodeExec(config.node0.Name, removeCmd) </del> <ins> err := framework.IssueSSHCommand(removeCmd, framework.TestContext.Provider, config.node0) </ins> Expect(err).NotTo(HaveOccurred()) By("Cleaning up persistent volume")

8 tests failed, 2 jobs failed, 88 builds failed. Failure stats cover 1 day time range '3 Aug 2017 03:31 UTC' to '4 Aug 2017 03:31 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
This should have been fixed by /cc
The test panic is fixed, but now it's failing because it can't SSH into the node.
/assign


Question: Limiter flowcontrol.RateLimiter }{ { <ins> Name:    "with no QPS", </ins> Config:  Config{}, Limiter: flowcontrol.NewTokenBucketRateLimiter(5, 10), }, { <ins> Name:    "with QPS:10", </ins> Config:  Config{QPS: 10}, Limiter: flowcontrol.NewTokenBucketRateLimiter(10, 10), }, { <ins> Name:    "with QPS:-1", </ins> Config:  Config{QPS: -1}, Limiter: nil, }, { <ins> Name: "with RateLimiter", </ins> Config: Config{ RateLimiter: flowcontrol.NewTokenBucketRateLimiter(11, 12), },

Reading TestRESTClientLimiter in I noticed a few things. test case struct is defined to have a name field, and its value is used in naming the subtests, but the actual test cases leave the name blank. error statements do not identify which case is failing (do not identify the expected value). Should be clear from the above. Read the code. No response_ <detailsbranch </details<detailsN/A </details<detailsN/A </details<details</details<details</details<details</details
/sig api-machinery
/assign
/triage accepted


Question: if svcInfo.preserveDIP || svcInfo.localTrafficDSR { lbIngressEndpoints = hnsLocalEndpoints } <del> hnsLoadBalancer, err := hns.getLoadBalancer( lbIngressEndpoints, loadBalancerFlags{isDSR: svcInfo.preserveDIP || svcInfo.localTrafficDSR, useMUX: svcInfo.preserveDIP, preserveDIP: svcInfo.preserveDIP, sessionAffinity: sessionAffinityClientIP, isIPv6: proxier.isIPv6Mode}, sourceVip, lbIngressIP.ip, Enum(svcInfo.Protocol()), uint16(svcInfo.targetPort), uint16(svcInfo.Port()), ) if err != nil { klog.ErrorS(err, "Policy creation failed") continue </del> <ins> if len(lbIngressEndpoints) > 0 { hnsLoadBalancer, err := hns.getLoadBalancer( lbIngressEndpoints, loadBalancerFlags{isDSR: svcInfo.preserveDIP || svcInfo.localTrafficDSR, useMUX: svcInfo.preserveDIP, preserveDIP: svcInfo.preserveDIP, sessionAffinity: sessionAffinityClientIP, isIPv6: proxier.isIPv6Mode}, sourceVip, lbIngressIP.ip, Enum(svcInfo.Protocol()), uint16(svcInfo.targetPort), uint16(svcInfo.Port()), ) if err != nil { klog.ErrorS(err, "Policy creation failed") continue } lbIngressIP.hnsID = hnsLoadBalancer.hnsID klog.V(3).InfoS("Hns LoadBalancer resource created for loadBalancer Ingress resources", "lbIngressIP", lbIngressIP) } else { klog.V(3).InfoS("Skipped creating Hns LoadBalancer for loadBalancer Ingress resources", "lbIngressIP", lbIngressIP) </ins> } <del> lbIngressIP.hnsID = hnsLoadBalancer.hnsID klog.V(3).InfoS("Hns LoadBalancer resource created for loadBalancer Ingress resources", "lbIngressIP", lbIngressIP) </del> } svcInfo.policyApplied = true klog.V(2).InfoS("Policy successfully applied for service", "serviceInfo", svcInfo)

During a Kubernetes deployment the connectivity to web services from Windows nodes is flapping. There are intermittent traffic delays and some errors observed on web services during the deployment. Post deployment the flapping is no longer observed. No flapping in connectivity during the deployment. Create a deployment with multiple services and monitor connectivity. No response <details20 </details<detailsAKS </details<details</details<details</details<details</details<details</details
/sig windows
/triage accepted /priority important-soon


Question: klog.V(3).Infof("No %s PriorityLevelConfiguration found, imagining one", proto.Name) labelValues := []string{proto.Name} reqsObsPair := meal.cfgCtlr.reqsObsPairGenerator.Generate(1, 1, labelValues) <del> execSeatsObs := meal.cfgCtlr.execSeatsObsGenerator.Generate(1, 1, labelValues) </del> <ins> execSeatsObs := meal.cfgCtlr.execSeatsObsGenerator.Generate(0, 1, labelValues) </ins> qsCompleter, err := queueSetCompleterForPL(meal.cfgCtlr.queueSetFactory, nil, proto, requestWaitLimit, reqsObsPair, execSeatsObs) if err != nil { // This can not happen because proto is one of the mandatory

While investigating another issue, I noticed that the histograms never get an Observation of zero. This makes no sense. See and related following remarks for some details. I expected the count of seats occupied by executing requests to be zero some times. Start an apiserver and scrape it; look at and see that the value is zero. No response <details</details<detailsnone </details<details</details<details</details<details</details<details</details


Question: {decQuantity(1024, 0, BinarySI), 1024}, {decQuantity(8*1024, 0, BinarySI), 8 * 1024}, {decQuantity(7*1024*1024, 0, BinarySI), 7 * 1024 * 1024}, <del> {decQuantity(7*1024*1024, 1, BinarySI), (7 * 1024 * 1024) * 1024}, {decQuantity(7*1024*1024, 4, BinarySI), (7 * 1024 * 1024) * (1024 * 1024 * 1024 * 1024)}, {decQuantity(7*1024*1024, 8, BinarySI), (7 * 1024 * 1024) * (1024 * 1024 * 1024 * 1024 * 1024 * 1024 * 1024 * 1024)}, {decQuantity(7*1024*1024, -1, BinarySI), (7 * 1024 * 1024) / float64(1024)}, {decQuantity(7*1024*1024, -8, BinarySI), (7 * 1024 * 1024) / float64(1024*1024*1024*1024*1024*1024*1024*1024)}, </del> <ins> {decQuantity(7*1024*1024, 1, BinarySI), (7 * 1024 * 1024) * 10}, {decQuantity(7*1024*1024, 4, BinarySI), (7 * 1024 * 1024) * 10000}, {decQuantity(7*1024*1024, 8, BinarySI), (7 * 1024 * 1024) * 100000000}, {decQuantity(7*1024*1024, -1, BinarySI), (7 * 1024 * 1024) * math.Pow10(-1)}, // '* Pow10' and '/ float(10)' do not round the same way {decQuantity(7*1024*1024, -8, BinarySI), (7 * 1024 * 1024) / float64(100000000)}, </ins> {decQuantity(1024, 0, DecimalSI), 1024}, {decQuantity(8*1024, 0, DecimalSI), 8 * 1024},

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I found that for with returns a very small value. The value should be almost the same as the one of . My minimal : Result: While the top three lines () are good, the float values in the bottom three lines () might be broken. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/assign
/cc /triage accepted


Question: diff=$(diff -Naupr "${KUBE_ROOT}" "${_tmp_kuberoot}" || true) if [[ -n "${diff}" ]]; then <del> echo "${diff}" echo echo "Run ./hack/update-bazel.sh" </del> <ins> echo "${diff}" >&2 echo >&2 echo "Run ./hack/update-bazel.sh" >&2 </ins> exit 1 fi

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: When running on my commits locally I got the following error: Affected PRs:
/sig testing
/cc
There's likely something up with your local environment. These jobs are running correctly in CI, although you need to look at the build log for output


Question: mockLoCNI.On("AddNetworkList", cniPlugin.loNetwork.NetworkConfig, mock.AnythingOfType("*libcni.RuntimeConf")).Return(&types020.Result{IP4: &types020.IPConfig{IP: net.IPNet{IP: []byte{127, 0, 0, 1}}}}, nil) <ins> // Check that status returns an error if err := cniPlugin.Status(); err == nil { t.Fatalf("cniPlugin returned non-err with no podCidr") } cniPlugin.Event(network.NET_PLUGIN_EVENT_POD_CIDR_CHANGE, map[string]interface{}{ network.NET_PLUGIN_EVENT_POD_CIDR_CHANGE_DETAIL_CIDR: "10.0.2.0/24", }) if err := cniPlugin.Status(); err != nil { t.Fatalf("unexpected status err: %v", err) } </ins> ports := map[string][]*hostport.PortMapping{ containerID.ID: { {

/kind feature The CNI "capability arguments" parameter - currently used to pass port mappings - has a few new capabilities: bandwidth limiting and network address management. The dockershim's networking driver should use them. The network address is just passing the PodCIDR dynamically to the plugin, so it no longer has to retrieve it from the APIServer. Bandwidth limiting is already implemented in Kubenet. It is configured via annotations. There is no upgrade risk in providing these values; non-compliant plugins will simply ignore them. Administrators need to explicitly enable each capability individually in their CNI configuration file. To avoid skew between runtimes, the other runtimes should also be upgraded to pass these. We may also wish to look in to a way to add this more directly to the CRI, so we don't have this skew problem, but that's a different issue.
/sig network
/assign
fyi.
Looks like this was never done for CRI?
cc


Question: assert.Equal(t, test.expected, cached, test.desc) } } <ins> func TestExtractIDFromCgroupPath(t *testing.T) { tests := []struct { cgroupPath string expected   string }{ { cgroupPath: "/kubepods/burstable/pod2fc932ce-fdcc-454b-97bd-aadfdeb4c340/9be25294016e2dc0340dd605ce1f57b492039b267a6a618a7ad2a7a58a740f32", expected:   "9be25294016e2dc0340dd605ce1f57b492039b267a6a618a7ad2a7a58a740f32", }, { cgroupPath: "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc932ce_fdcc_454b_97bd_aadfdeb4c340.slice/cri-containerd-aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9.scope", expected:   "aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9", }, { cgroupPath: "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc932ce_fdcc_454b_97bd_aadfdeb4c340.slice/cri-o-aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9.scope", expected:   "aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9", }, } for _, test := range tests { id := extractIDFromCgroupPath(test.cgroupPath) assert.Equal(t, test.expected, id) } } </ins>

Since upgrading from Kubernetes to we no longer get memory metrics from the endpoint on . Example: In our previous version, we would get a response like the following that included utilisation metrics: Hit the endpoint of on a cluster running version . Kubernetes version (use ): Cloud provider or hardware configuration: Provisioned on AWS with OS (e.g: ): Flatcar Container Linux by Kinvolk 2765.2.6 (Oklo) Kernel (e.g. ): 5.10.43-flatcar
/sig node
Hi AlexLastwhich container runtimes are you using? containerd?
Hi AlexLast, I have test on v1.22 with containerd, I did not reproduce this issue. If memory limit is set in the container spec of the pod, /stats/summary will report availableBytes, if memory limit is not set, /stats/summary will not be reported. just like this: ! ! code is here:
/cc We have reports of missing memory metrics on 1.20.8 and 1.21.2 now... was there a cadvisor or similar change that happened?
also /cc as an FYI
If you are using cri-o, it looks like a similar issue to and (possibly an issue with cadvisor/the way cri-o integrates with cadvisor). This would also apply to the dockershim
I'm not able to repro on GKE v1.20.8-gke.700 on containerd
Can you please provide more info about your environment, container runtime etc and possible repro steps? Thanks!
/triage needs-information
Hi, First of all, I am sorry to cut in on you. I have the same issue with Kubelet, in my case with Kubeadm. Here you are the issue: Here you are my installation: Thanks in advance
^ quick summary of the above, appears to be single-node ubuntu (not sure what kernel?) containerd 1.4.6 k8s 1.21.2 kubeadm installer
It is one node. It is only for making easier where is the problem .


Question: assert.Equal(t, test.expected, cached, test.desc) } } <ins> func TestExtractIDFromCgroupPath(t *testing.T) { tests := []struct { cgroupPath string expected   string }{ { cgroupPath: "/kubepods/burstable/pod2fc932ce-fdcc-454b-97bd-aadfdeb4c340/9be25294016e2dc0340dd605ce1f57b492039b267a6a618a7ad2a7a58a740f32", expected:   "9be25294016e2dc0340dd605ce1f57b492039b267a6a618a7ad2a7a58a740f32", }, { cgroupPath: "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc932ce_fdcc_454b_97bd_aadfdeb4c340.slice/cri-containerd-aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9.scope", expected:   "aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9", }, { cgroupPath: "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc932ce_fdcc_454b_97bd_aadfdeb4c340.slice/cri-o-aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9.scope", expected:   "aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9", }, } for _, test := range tests { id := extractIDFromCgroupPath(test.cgroupPath) assert.Equal(t, test.expected, id) } } </ins>

Kubernetes version: 1.21.2 Cloud provider or hardware configuration: Bare metal (VMware) OS: Ubuntu 18.04.5 LTS Kernel: 4.15.0-147-generic Install tools: Kubeadm Network plugin and version (if this is a network-related bug): Weave 2.8.1 CRI: Containerd 1.4.6
Sure, we're setting the following flags for : Version: We deploy our clusters using , these missing metrics are for all pods in the cluster, including pods with memory requests and limits set.
Hitting the same issue on such kind of environments Then I tested and found no issue on environment
Besides memory metrics, we noticed a loss of network in pods. I commented at the block of container and the block of pod The loss of pods occurs at the same time as loss of pods
I created cluster as follows on 1.21.2 with kubetest containerd 1.4.6 on ubuntu. This is on older version of ubuntu 18.04 LTS on kernel I don't see memory metrics being lost here.
Can we have more details on exact kernel/OS image being used here? I can't seem to repro this on 1.21.2 with same containerd/ ubuntu version as mentioned in
Hi, may I know your output of network metrics in pods?
Hi, I have tested with two kernel versions: P.S: I have just updated my in order to make easier the test. You can use it (it is necessary to install and ). Thanks in advance
We tested with two kernel versions too.
I was able to repro finally -- it looks like missing key here is that you must be using the systemd cgroup driver. With this patch and starting cluster as follows:
Hi do you mean that if using systemd cgroup driver, this issue would happen. In my environment, both containerd and kubelet are using cgroupfs driver, but the issue still exists containerd config dump
Hi, Here you are my Containerd & Kubelet config: ` As far as I know, systemd cgroup driver should be configured in order to have a proper perfomance in Kubernetes. Is it a bug with Kubelet?


Question: assert.Equal(t, test.expected, cached, test.desc) } } <ins> func TestExtractIDFromCgroupPath(t *testing.T) { tests := []struct { cgroupPath string expected   string }{ { cgroupPath: "/kubepods/burstable/pod2fc932ce-fdcc-454b-97bd-aadfdeb4c340/9be25294016e2dc0340dd605ce1f57b492039b267a6a618a7ad2a7a58a740f32", expected:   "9be25294016e2dc0340dd605ce1f57b492039b267a6a618a7ad2a7a58a740f32", }, { cgroupPath: "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc932ce_fdcc_454b_97bd_aadfdeb4c340.slice/cri-containerd-aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9.scope", expected:   "aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9", }, { cgroupPath: "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc932ce_fdcc_454b_97bd_aadfdeb4c340.slice/cri-o-aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9.scope", expected:   "aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9", }, } for _, test := range tests { id := extractIDFromCgroupPath(test.cgroupPath) assert.Equal(t, test.expected, id) } } </ins>

Thanks in advance
Hi thanks for the response, I've found the following in the , so will have a go at setting that parameter
Thanks for all the info! I was only able to repro this on systemd cgroup driver as mentioned in It did not repro for me when using cgroupfs. This will need some additional investigation to figure out the root cause and why memory metrics are not being reported correctly and see if this is a kubelet or cAdvisor issue. /cc
Hi, Thanks . By the way, there is a new issue reporting the same behaviour:
After setting kubelet's log verbosity level to 10, I found logs like Is this related? And I checked that according the log, all the pod sandboxID are missing
Thanks I was able to confirm that switching to fixed the missing metrics. From a bit of research, it's my understanding that using the driver on a node with is not recommended - can you confirm?
By stepping into the code, I found that the container info returned by this func is nil, so that the will always be false. Thus will always be . That explains the last log has expected fields. But because of 1), this func cannot be called. There's really a label-related issue on cadvisor: (Thanks With the workaround, I can recover the missed metrics
/remove-triage needs-information /triage accepted
Thanks for digging deeper into this I'm still a bit confused on the relationship between this and systemd cgroup driver. Do the labels being missing have something to do with systemd cgroup driver being used? Also are you using a different socket path for containerd i.e. not the default as mentioned in ?
We didn't use the default but used and not specified
Interesting! In the test I made in the container-runtime-endpoint is set to default (i.e. ).
With debugging, I found that on kubernetes 1.21.2 when using different cgroup driver, the cgroupPath format of containers are different. : systemd + containerd:cgroupfs : systemd + containerd: systemd : cgroupfs + containerd: cgroupfs The would return a map using the path.Base of cgroupPath Later, we use containerID to check to add metrics. So only in the third case, the key of are . That's how the cgroup driver impacts the metrics.


Question: assert.Equal(t, test.expected, cached, test.desc) } } <ins> func TestExtractIDFromCgroupPath(t *testing.T) { tests := []struct { cgroupPath string expected   string }{ { cgroupPath: "/kubepods/burstable/pod2fc932ce-fdcc-454b-97bd-aadfdeb4c340/9be25294016e2dc0340dd605ce1f57b492039b267a6a618a7ad2a7a58a740f32", expected:   "9be25294016e2dc0340dd605ce1f57b492039b267a6a618a7ad2a7a58a740f32", }, { cgroupPath: "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc932ce_fdcc_454b_97bd_aadfdeb4c340.slice/cri-containerd-aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9.scope", expected:   "aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9", }, { cgroupPath: "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc932ce_fdcc_454b_97bd_aadfdeb4c340.slice/cri-o-aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9.scope", expected:   "aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9", }, } for _, test := range tests { id := extractIDFromCgroupPath(test.cgroupPath) assert.Equal(t, test.expected, id) } } </ins>

To sum up, and are two different cases that caused this issue. For the case, I think it's a bug we need fix: we should make sure the key of always be BTW do you know why the cgroupPath format are different when using different cgroup driver?
I believe it's expected that the cgroup naming will differ when using systemd cgroup driver vs cgroupfs. Kubelet internally has some logic for parsing and converting between the two different formats, e.g. and I left some comments on your proposed PR: Thanks for helping look into this issue!
/priority important-soon


Question: } } <del> func getCRICadvisorStats(infos map[string]cadvisorapiv2.ContainerInfo) map[string]cadvisorapiv2.ContainerInfo { </del> <ins> func getCRICadvisorStats(infos map[string]cadvisorapiv2.ContainerInfo) (map[string]cadvisorapiv2.ContainerInfo, map[string]cadvisorapiv2.ContainerInfo) { </ins> stats := make(map[string]cadvisorapiv2.ContainerInfo) <del> infos = removeTerminatedContainerInfo(infos) for key, info := range infos { </del> <ins> filteredInfos, cinfosByPodCgroupKey := filterTerminatedContainerInfoAndAssembleByPodCgroupKey(infos) for key, info := range filteredInfos { </ins> // On systemd using devicemapper each mount into the container has an // associated cgroup. We ignore them to ensure we do not get duplicate // entries in our summary. For details on .mount units:

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --For single node with hundreds and thousands of pods and containers, the cpu usage of kubelet becomes high, one bottleneck is the cadvisor stats provider as below: the method iterate all containerInfos for every podStats cause high cpu the method should not iterate allcontainerInfos for every podStats, use map instead Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
I’ve done an analysis on the code and this is what I have found, I believe we have two issues to address. There are multiple range loops being performed over v2.ContainerInfo / PodStat maps within the ListPodStatus method which can be contributing to the high cpu utilisation. The allInfos argument which is supplied to the getCadvisorContainerInfo function is a map which uses a cgroup path as the key, in order to find matching container info to podUID, processing of the keys in the map needs to be performed until a match is found. Examples of keys are as follows: /kubepods/burstable/podf9e3a173-1470-451a-9ca2-/d8426b82582fbcfae16085bc794c49769d3a9648bd5463976aca4071f9342754 /docker/2dfa17fc81e781212bcc26e189ebb724721fe097394b4de3cbeb7e069bf23290 My recommendation would be to apply the following updates: Refactor the getCadvisorContainerInfo function to allow for a filter and key type parameter to be supplied by the caller, this enables reducing the number of range loops being performed over the container info map by removing elements that are not required and the ability to set the key being used in the returned container infos map to enable fast lookup of container info.


Question: } } <del> func getCRICadvisorStats(infos map[string]cadvisorapiv2.ContainerInfo) map[string]cadvisorapiv2.ContainerInfo { </del> <ins> func getCRICadvisorStats(infos map[string]cadvisorapiv2.ContainerInfo) (map[string]cadvisorapiv2.ContainerInfo, map[string]cadvisorapiv2.ContainerInfo) { </ins> stats := make(map[string]cadvisorapiv2.ContainerInfo) <del> infos = removeTerminatedContainerInfo(infos) for key, info := range infos { </del> <ins> filteredInfos, cinfosByPodCgroupKey := filterTerminatedContainerInfoAndAssembleByPodCgroupKey(infos) for key, info := range filteredInfos { </ins> // On systemd using devicemapper each mount into the container has an // associated cgroup. We ignore them to ensure we do not get duplicate // entries in our summary. For details on .mount units:

My initial thoughts are to use an enumeration which has two values KEYCGROUPPATH to provide backwards compatibility and KEYPODUID to enable fast lookups via podUID Refactor ListPodStats method to be more efficient in constructing the PodStats slice and taking into account the fast lookup via podUID I think this is the best approach instead of pushing the data structure implementation for the container info upstream to the cadviser clients, this will at least initially provide a better optimisation scoped within the kubelet/stats package and reduce other possible impacts to changing the cadvisor clients Please advise.
Just realized this looks like a duplicate issue and looking as fixed by PR Both this issue and were raised by Zac
/unassign
/close
You can't close an active issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Didn't think I could close this, leaving this with
this will be automatically closed once the PR merges


Question: defer c.watchCache.RUnlock() initEvents, err := c.watchCache.GetAllEventsSinceThreadUnsafe(watchRV) if err != nil { <del> return nil, err </del> <ins> // To match the uncached watch implementation, once we have passed authn/authz/admission, // and successfully parsed a resource version, other errors must fail with a watch event of type ERROR, // rather than a directly returned error. return newErrWatcher(err), nil </ins> } c.Lock()

When attempting to fetch a resource version that is too old, the behavior of watch (for both HTTP and web sockets) changed when watch cache was enabled. The watch cache returns a 410 - without the watch cache, we return 200 and then write an "error" entry to the watch stream. This broke a client that dependent on the watch behavior, and moreover is inconsistent with other errors we set in the registry. Also, for watch over web sockets, most javascript clients will be unable to get at a 410 error from the connection, because browsers don't get access to the data. We should restore the watch cache behavior to be consistent with the previous behavior as this is a regression in API semantics. Resource without watch cache, watching too old: Resource with watch cache, watching too old:
I think this is candidate for back port to 1.2 to restore the previous behavior.
since this broke your javascript client, FYI because of watch cache
+1 for backport
hmm - we enabled watch cache in 1.1 - so this should also be broken in 1.1
It's possible that it was always nondeterministic and just depends on how fast the call to the storage layer executes. I think all of our in-tree clients will handle it either way.
I'm fairly certain this never used to behave this way, because OpenShift hasn't enabled the watch catch and we have never had a report of this until we enabled the watch cache for Kube resources just recently. , Daniel Smith wrote:
I see what you mean about ordering - etcd remote call would almost never be fast enough to beat the watch event. In practice, I suspect no one ever saw this without the watch error. Either way, for web sockets if we don't fix this clients can't get this error (the browser eats the 410 during the websocket negotiation and client code can't get access). So it's at least broken for web sockets. , Clayton Coleman wrote:
Well, it should be easy to just not wait at all, maybe? I'm OK with changing this behavior.
Although, are web clients XSS hardened? I'm not sure that we're optimizing for them currently.
If you mean CSRF, it isn't an issue with bearer token auth.


Question: defer c.watchCache.RUnlock() initEvents, err := c.watchCache.GetAllEventsSinceThreadUnsafe(watchRV) if err != nil { <del> return nil, err </del> <ins> // To match the uncached watch implementation, once we have passed authn/authz/admission, // and successfully parsed a resource version, other errors must fail with a watch event of type ERROR, // rather than a directly returned error. return newErrWatcher(err), nil </ins> } c.Lock()

Basic auth should already have a big warning around it saying "not safe with browsers"
Maybe I'm missing something, but it seems deterministic to me. This is the uncached watch impl: once the resourceVersion parses, there is no path that returns a direct error, so all errors would have to be returned as an event via the ResultChan(). I don't see us reading from that until after we've already written a 200 and flushed:
Having read the docs, I was very surprised that "type": "ERROR" notices are sent as 200 (even over plain http), with the semantic code e.g. 410 only represented under inside. I see there was good motivation here to stick to 200, but it should be called out in the documentation. doesn't show at all how watch returns errors, and it even says "clients must handle the case by recognizing the status code 410 Gone" which really makes one expect an actual HTTP 410.
I agree this should be called out in the docs. Can you file a bug there and/or send a PR? I don't think this ~4 year old issue is the best place to track that. (Good detective work finding this, though!)


Question: // LocalTestResourceManager represents interface to create/destroy local test resources on node type LocalTestResourceManager interface { Create(node *v1.Node, volumeType LocalVolumeType, parameters map[string]string) *LocalTestResource <ins> ExpandBlockDevice(ltr *LocalTestResource, mbToAdd int) error </ins> Remove(ltr *LocalTestResource) }

There are use cases where users want to expand the local volume (may be managed by LVM or something similar), but even though they can edit underlying PV (and expand the underlying device) - the reported size in pvc.Status keeps reporting older value because in-tree local volumes are non-expandable. We could make this easier for users by - implementing interface for local volumes. This would allow following steps for local volume expansion: /admin edits the pvc - requesting new size. expands the underlying device. edits the PV and sets new size. will expand the filesystem and update This would at least get the user from impossible territory to possible. cc /sig storage
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hi & any news on this issue ? The related PR seems stalled.
merged. So this is fixed now.
Oh right ! I didn't looked close enough and thought it was only for directory based volumes. Thank you.


Question: return func() (bool, error) { endpoints, err := c.Endpoints(serviceNamespace).Get(serviceID) if err != nil { <ins> glog.Infof("Error on creating endpoints: %v", err) </ins> return false, nil } <ins> glog.Infof("endpoints: %v", endpoints.Endpoints) </ins> return len(endpoints.Endpoints) == endpointCount, nil } }

Kubelet now only reports ContainerStatus. PodStatus / PodCondition is observed at master side based on a set of ContainerStatus. There is no other way to get PodIP except net container for master side. This is very confusion to the end user, , , ... Now we are finalizing on v1beta3 API, including PodStatus & ContainerStatus. We can make net container completely invisible to the end users by finishing the following steps: 1) Expose network related errors at API level 2) Generate PodStaus directly from Kubelet
3) once PodStatus generated by kubelet, PodIP can be removed from ContainerStatus.
I think reporting PodStatus rather than just container status is most important. How hard is that?
Used to be hard since there are many dependencies when I first report Status back. Now it is totally feasible. I already started to write PRs for this. :-)
One more PR required to close this one.


Question: "kind": "Deployment", "metadata": { "name": "my-dep", <del> "namespace": "my-ns", </del> "unknownMeta": "foo", "labels": {"app": "nginx"} },

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: "kind": "Deployment", "metadata": { "name": "my-dep", <del> "namespace": "my-ns", </del> "unknownMeta": "foo", "labels": {"app": "nginx"} },


Question: const ServiceAnnotationLoadBalancerCrossZoneLoadBalancingEnabled = "service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled" // ServiceAnnotationLoadBalancerExtraSecurityGroups is the annotation used <del> // one the service to specify additional security groups to be added to ELB created </del> <ins> // on the service to specify additional security groups to be added to ELB created </ins> const ServiceAnnotationLoadBalancerExtraSecurityGroups = "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups" <ins> // ServiceAnnotationLoadBalancerSecurityGroups is the annotation used // on the service to specify the security groups to be added to ELB created. Differently from the annotation // "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups", this replaces all other security groups previously assigned to the ELB. const ServiceAnnotationLoadBalancerSecurityGroups = "service.beta.kubernetes.io/aws-load-balancer-security-groups" </ins> // ServiceAnnotationLoadBalancerCertificate is the annotation used on the // service to request a secure listener. Value is a valid certificate ARN. // For more, see http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-listener-config.html

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: We run Kuberentes on AWS. We are trying out service type. We use Terraform heavily for managing infrastructure on AWS. As a result we have several shared security groups with "perfect" rules. One of them is supposed to be attached to ELBs so our own IPs, partners, etc are whitelisted. Right now, it seems there isn't a way for Kubernetes to use the security group as is. There are several alternatives: Pass my existing security group as annotation. However, Kubernetes still tries to create one first, or use a global security group. If I don't provide any IP range, Kubernetes will whitelist . This seems insecure. Provide my own IP ranges as either annotation or . Then Kubernetes will create a security group for this ELB with the given IPs. This means I need to duplicate my IPs in all microservices. Provide my own IP ranges as either annotation or , and use a global security group. Then Kubernetes will modify the global SG with the given IPs. This seems problematic when lots of microservices share one SG but some want different IPs whitelisted. I want to entertain the idea below: Allow users to sepcify another security group to or object, which Kubernetes will take it and attach it to ELBs as is. If such security group is provided, skip other IP whitelisting steps. What you expected to happen: I can give Kubernetes existing security group(s). Kuberentes just attaches the security group(s) to the managed ELBs without modifying the rules. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: It may already be possible to achieve this, and I'm not aware of. Or there are other concerns so that Kubernetes didn't choose to implement this feature. Environment: Kubernetes version (use ): v1.7.1 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig aws
PR may help with this. . .
This problem could be really critical if one has the Kubernetes dashboard installed.


Question: const ServiceAnnotationLoadBalancerCrossZoneLoadBalancingEnabled = "service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled" // ServiceAnnotationLoadBalancerExtraSecurityGroups is the annotation used <del> // one the service to specify additional security groups to be added to ELB created </del> <ins> // on the service to specify additional security groups to be added to ELB created </ins> const ServiceAnnotationLoadBalancerExtraSecurityGroups = "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups" <ins> // ServiceAnnotationLoadBalancerSecurityGroups is the annotation used // on the service to specify the security groups to be added to ELB created. Differently from the annotation // "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups", this replaces all other security groups previously assigned to the ELB. const ServiceAnnotationLoadBalancerSecurityGroups = "service.beta.kubernetes.io/aws-load-balancer-security-groups" </ins> // ServiceAnnotationLoadBalancerCertificate is the annotation used on the // service to request a secure listener. Value is a valid certificate ARN. // For more, see http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-listener-config.html

If for some reasons the service recreates AWS ELB, the related security group will be recreated as well with the "0.0.0.0/0" input/output masks and then botnets could find the ELB and . JFYI, in our case in the one of clusters with K8s 1.5, the deployment and replication controller were installed from the dashboard using this vector of attack. Then special pods were created and root volume mounted inside. The system crontab was modified by processes in these pods and the crypto miner appeared in all K8s nodes on the system level. After this another pods were installed and the miner was executed inside the Kubernetes pods in parallel with the system level.
just had some downtime on a production cluster because of this. i really need it too ;)
+1
As for me, it isn't a feature, it's a kind of bug.
Some comments on which may be relevant - please comment if an alternative annotation that did replace the default security-groups (as opposed to adding to them) would/wouldn't meet your use-case:
For me, I don't add any annotation, I just want to change its security group rule. After it will create a aws elb, and a specificed secruity group with 0.0.0.0, I change 0.0.0.0 to However, the security group rule will change back to 0.0.0.0 after some minutes. Is this by design? Is there any way for me keep my changes, thanks.
I guess this is part of how the controller works. If you look at , the PR fixes this problem by allowing you to specify the behaviour you want with an annotation. For the way Kubernetes works, the ELB is owned by Kubernetes and you should never be forced to modify the resource manually, thing that ultimately will not work.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale This is still relevant cause is not merged yet.
This is a good change.


Question: const ServiceAnnotationLoadBalancerCrossZoneLoadBalancingEnabled = "service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled" // ServiceAnnotationLoadBalancerExtraSecurityGroups is the annotation used <del> // one the service to specify additional security groups to be added to ELB created </del> <ins> // on the service to specify additional security groups to be added to ELB created </ins> const ServiceAnnotationLoadBalancerExtraSecurityGroups = "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups" <ins> // ServiceAnnotationLoadBalancerSecurityGroups is the annotation used // on the service to specify the security groups to be added to ELB created. Differently from the annotation // "service.beta.kubernetes.io/aws-load-balancer-extra-security-groups", this replaces all other security groups previously assigned to the ELB. const ServiceAnnotationLoadBalancerSecurityGroups = "service.beta.kubernetes.io/aws-load-balancer-security-groups" </ins> // ServiceAnnotationLoadBalancerCertificate is the annotation used on the // service to request a secure listener. Value is a valid certificate ARN. // For more, see http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-listener-config.html

The situation pre-1.13.0 is somewhat surprising because explicitly setting to an empty array like so: has no effect, and the Security Group Ingress is still .


Question: # journal message level, 0 is debug KUBE_LOG_LEVEL=0 <ins> KUBE_ALLOW_PRIV="true" </ins> ``` * Start the appropriate services on fed2.

https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
There are 3 different issues in this bug: high resource consumption (or too strict limits on our side) - should fix it to remove namespace: [I looked into it, and it was because events etcd didn't even come up in this test. The problem here is that we are not collecting logs from etcd in kubemark - I will send a PR fixing it later today hopefully. (that I didn't have time to look into yet).
So actually, the non-started events etcd was the pulling image problem: So we may want to retry in case of failures also.
The first one should be fixed by The second one will be fixed by The third problem is actually related to pod-nanny, which is frequently sending list requests to apiserver. It should be rewritten to use watch instead.
Since the main issue is fixed, I'm decreasing the priority.
For the third problem, I've just sent out: for review. Once that one is merged, we should build the new image and bump tags in our config files.


Question: # journal message level, 0 is debug KUBE_LOG_LEVEL=0 <ins> KUBE_ALLOW_PRIV="true" </ins> ``` * Start the appropriate services on fed2.

https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
So the remaining things here to do are: kubernetes/contrib + update version of pod-nanny in heapster manifests tun kubemark-5 resource constraints
What we should also do, is to add "kubemark-scale" to list of non-blocking suites of merge-queue, so that bugs will be automatically filed for that suite.
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
There are three remaining things to do here: tweak limits for kubemark-5 merge add kubemark-scale to non-blocking suites of merge-queue can you please take care of the first one? I will take care of the other two.
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
It seems that this PR broke scalability: I'm reverting it and will try to debug it offline.
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
All the scalability failures from last day are because of - that PR was reverted, and I fixed the problem in and built a new image. (I did some 100-node cluster tests locally and it worked fine after the change).


Question: q.queue.Clear() } <del> // SetRemove remove value from the set of the queue func (q *RateLimitedTimedQueue) SetRemove(value string) { q.queue.SetRemove(value) } </del> // SwapLimiter safely swaps current limiter for this queue with the // passed one if capacities or qps's differ. func (q *RateLimitedTimedQueue) SwapLimiter(newQPS float32) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Nodelifecycle controller does not set NoExecute taint to NotReady node when kubelet has been stopped What you expected to happen: effect: NoExecute key: Nodelifecycle controller sets NoExecute taint to NotReady node when kubelet has been stopped How to reproduce it (as minimally and precisely as possible): systemctl stop kubelet Anything else we need to know?: I use default args to start kube-controller-manager. I see log from kube-controller-manager: Environment: Kubernetes version (use ): [root ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): [root ~]# cat /etc/os-release NAME="CentOS Linux" VERSION="8 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="8" PLATFORMID="platform:el8" PRETTYNAME="CentOS Linux 8 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:8" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-8" CENTOSMANTISBTPROJECTVERSION="8" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="8" Kernel (e.g.


Question: q.queue.Clear() } <del> // SetRemove remove value from the set of the queue func (q *RateLimitedTimedQueue) SetRemove(value string) { q.queue.SetRemove(value) } </del> // SwapLimiter safely swaps current limiter for this queue with the // passed one if capacities or qps's differ. func (q *RateLimitedTimedQueue) SwapLimiter(newQPS float32) {

): [root ~]# uname -a Linux liabio 4.18.0-147.5.1.el81.x8664 SMP Wed Feb 5 02:00:39 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle /wg cluster-lifecycle
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-cluster-lifecycle /wg sig-cluster-lifecycle /committee sig-cluster-lifecycle Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig cluster-lifecycle /sig node
Those labels are not set on the issue: <detailsIn response to : /remove-sig cluster-lifecycle /sig node </details
is NoExecute the expected taint for this condition?
- yes it is when running the taint-manager: evictions are usually safe eviction as we don't force delete pods
ref
I see the same, a node which has node condition set to Unknown, does not get an Unreachable taint with NoExecute effect. However I only see this on 1.19, works fine till 1.18.10 Till 1.18.10, such a node gets two Unreachable taints, one with effect NoExecute and one with effect NoSchedule. But on 1.19.3 it only gets the NoSchedule taint. I see that this is the only commit made in that area between 1.18 and 1.19: , could be related?
It totally breaks production readiness of all recent releases, yet doesn't seem to be regarded as a top priority issue. WTF?


Question: spec: containers: - name: fluentd-elasticsearch <del> image: gcr.io/google_containers/fluentd-elasticsearch:1.17 </del> <ins> image: gcr.io/google_containers/fluentd-elasticsearch:1.19 </ins> resources: limits: memory: 200Mi

It's 400+mb and we need to pull it on every node. This leads to a lot of image pulling flake (), and also gets very expensive in some countries where bandwidth is sold at a premium. We took a stab at halving the size with: , but then realized that we were using an unsupported base image We tried to optimize to ~300Mb by building a smaller ubuntu:15 base, but that's still huge. Every other mandatory system daemon is built on busybox and so takes ~10Mb.
I can try again now that the agent supports
Newer Docker images using the spec from the official repository are based on AlpineLinux and are around ~38MB:


Question: gomega.Expect(e2epod.WaitForPodToDisappear(c, ns, name, labels.Everything(), pollInterval, pollTimeout)).To(gomega.Succeed()) ginkgo.By("Delete the config map") c.CoreV1().ConfigMaps(ns).Delete(context.TODO(), configName, metav1.DeleteOptions{}) <ins> ginkgo.By("Delete the service account") c.CoreV1().ServiceAccounts(ns).Delete(context.TODO(), name, metav1.DeleteOptions{}) ginkgo.By("Delete the clusterRoleBinding") c.RbacV1().ClusterRoleBindings().Delete(context.TODO(), name, metav1.DeleteOptions{}) </ins> ginkgo.By("Clean up the events") gomega.Expect(c.CoreV1().Events(eventNamespace).DeleteCollection(context.TODO(), *metav1.NewDeleteOptions(0), eventListOptions)).To(gomega.Succeed()) ginkgo.By("Clean up the node condition")

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Last successful run (23 Oct 2020, 10:07 PDT): First unsuccessful run (23 Oct 2020, 12:08 PDT): Testgrid link: Reason for failure: Anything else we need to know: presubmit job in repository also started to fail in the same fashion. Attempted to debug it myself, found out the following stuff: last successful run was accessing the apiserver via HTTP: all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail: perused the history of to find out that the PR is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests): first, it coincides with the start of the failures of , second, we changed the default hostname of the apiserver in node tests in that PR. Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a flag to the of these jobs. /assign /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Opened for a mitigation that would make those jobs stop failing.
This won't work, unfortunately. Copying response:
I have filed to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
Unfortunately I don't :/. My PR was also a wild guess. could you take a look at the proposed fix? Thanks once again for your help - really appreciated :).
the submission of hasn't fixed the problems, unfortunately :/. Sample failing run: We still have some peculiar problem related to health checking: and now NPD panics, too:
Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. /reopen
Reopened this issue.


Question: gomega.Expect(e2epod.WaitForPodToDisappear(c, ns, name, labels.Everything(), pollInterval, pollTimeout)).To(gomega.Succeed()) ginkgo.By("Delete the config map") c.CoreV1().ConfigMaps(ns).Delete(context.TODO(), configName, metav1.DeleteOptions{}) <ins> ginkgo.By("Delete the service account") c.CoreV1().ServiceAccounts(ns).Delete(context.TODO(), name, metav1.DeleteOptions{}) ginkgo.By("Delete the clusterRoleBinding") c.RbacV1().ClusterRoleBindings().Delete(context.TODO(), name, metav1.DeleteOptions{}) </ins> ginkgo.By("Clean up the events") gomega.Expect(c.CoreV1().Events(eventNamespace).DeleteCollection(context.TODO(), *metav1.NewDeleteOptions(0), eventListOptions)).To(gomega.Succeed()) ginkgo.By("Clean up the node condition")

<detailsIn response to : Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
filed . I hope we could completely address this issue this time.
/milestone v1.20


Question: _ "github.com/stretchr/testify/assert" v1 "k8s.io/api/core/v1" <del> apierrors "k8s.io/apimachinery/pkg/api/errors" </del> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Kubernetes e2e suite.[sig-scheduling] SchedulerPriorities [Serial] Pod should avoid nodes that have avoidPod annotation June 19th 2021 () Testgrid: https://k8s- Prow history: Looks like pods are scheduled where they shouldn't be? : The tests passes in maybe 20% of the builds, but it looks random, so I don't treat it as a flake. At the moment there are also sig-storage tests consistently failing in this job: , They don't look related though. /sig scheduling /kind failing-test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
try to have a look.
try to have a look.
/cc
This is expected. The feature (which was alpha) is now disabled by default. Users can still enable it by switching to a v1beta1 scheduler component config. Let's remove the test.


Question: discoveryendpoint "k8s.io/apiserver/pkg/endpoints/discovery/aggregated" scheme "k8s.io/client-go/kubernetes/scheme" "k8s.io/client-go/tools/cache" <ins> "k8s.io/client-go/util/workqueue" </ins> apiregistrationv1 "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1" ) func newDiscoveryManager(rm discoveryendpoint.ResourceManager) *discoveryManager { <del> return NewDiscoveryManager(rm).(*discoveryManager) </del> <ins> dm := NewDiscoveryManager(rm).(*discoveryManager) dm.dirtyAPIServiceQueue = newCompleterWorkqueue(dm.dirtyAPIServiceQueue) return dm </ins> } <del> // Returns true if the queue of services to sync empty this means everything has // been reconciled and placed into merged document func waitForEmptyQueue(stopCh <-chan struct{}, dm *discoveryManager) bool { </del> <ins> // Returns true if the queue of services to sync is complete which means // everything has been reconciled and placed into merged document func waitForQueueComplete(stopCh <-chan struct{}, dm *discoveryManager) bool { </ins> return cache.WaitForCacheSync(stopCh, func() bool { <del> // Once items have successfully synced they are removed from queue. return dm.dirtyAPIServiceQueue.Len() == 0 </del> <ins> return dm.dirtyAPIServiceQueue.(*completerWorkqueue).isComplete() </ins> }) }

pull-kubernetes-unit sometimes fails with data race. TestDirty I think the test was 3 months ago with the flaking state. There are two issues. First, updating and reading a bool-type variable is not thread-safe. and there is also another problem checking the state before completely syncing it. No response /sig api-machinery
/assign
/triage accepted


Question: _, err = members.Create(lb.network, members.CreateOpts{ PoolID:       pool.ID, <del> ProtocolPort: int(ports[0].NodePort), //TODO: need to handle multi-port </del> <ins> ProtocolPort: int(ports[0].NodePort), //Note: only handles single port </ins> Address:      addr, }).Extract() if err != nil {

LBaaS implementation (v1 or v2) will not support services that require to load balancer more than one port.
Fixed for LBaaS v2 by I suspect LBaaS v1 can't support multiple ports (not sure if reusing the VIP across multiple loadbalancers is a universally supported thing). I also believe LBaaS v1 is also mostly gone in favour of v2 in deployments so trying to address this in v1 is extremely low priority unless someone steps forward to ask for it. So: I think we can close this bug now, but I lack the powers to do that directly.
thank you for the update.


Question: var _ volume.ExpandableVolumePlugin = &azureDataDiskPlugin{} var _ volume.DeviceMountableVolumePlugin = &azureDataDiskPlugin{} <del> // store vm size list in current region var vmSizeList *[]compute.VirtualMachineSize </del> const ( azureDataDiskPluginName = "kubernetes.io/azure-disk" defaultAzureVolumeLimit = 16

<!-- Please only use this template for submitting enhancement requests -- What would you like to be : related to Why is this needed: /assign /sig azure
here is the way about how to get MaxDataDiskCount: and then build kubelet, copy kubelet to agent node and then get MaxDataDiskCount data: There are 207 VM sizes totally.
Update: this command could get VM size list easily:


Question: "azure_common_windows.go", "azure_dd.go", "azure_dd_block.go", <ins> "azure_dd_max_disk_count.go", </ins> "azure_mounter.go", "azure_provision.go", ],


Question: "github.com/stretchr/testify/assert" ) <del> func TestEnsureNodeExistsByProviderIDOrNodeName(t *testing.T) { </del> <ins> func TestEnsureNodeExistsByProviderID(t *testing.T) { </ins> testCases := []struct { testName           string node               *v1.Node expectedCalls      []string <del> existsByNodeName   bool </del> <ins> expectedNodeExists bool hasInstanceID      bool </ins> existsByProviderID bool nodeNameErr        error providerIDErr      error

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The change here removes exiternal id, but the call to instance id was not made used at all, it will always return false. At this time no node was able to register What you expected to happen: ProviderId was used in check How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment:
similar to are you using --cloud-provider in your controller manager? are you also setting it on your kubelet?
I'm using acs-engine to deploy the cluster. Config is as following: I've proposed a fix. PTAL
/sig cluster-lifecycle


Question: } func (c *dynamicResourceClient) Delete(name string, opts *metav1.DeleteOptions, subresources ...string) error { <ins> if len(name) == 0 { return fmt.Errorf("name is required") } </ins> if opts == nil { opts = &metav1.DeleteOptions{} }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Providing an empty to the dynamic client's method results in a panic. What you expected to happen: An error should be returned on empty . How to reproduce it (as minimally and precisely as possible): Call with Anything else we need to know?: The expects a nonempty to be provided, but does not validate this. If an empty name is provided, this method will panic because it expects an , but the response will actually be an . Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: client-go 10.0.0
/sig api-machinery


Question: // Enable pods to set sysctls on a pod Sysctls featuregate.Feature = "Sysctls" <del> // owner @smarterclayton // alpha: v1.16 // beta:  v1.19 // ga:  v1.21 // // Enable legacy behavior to vary cluster functionality on the node-role.kubernetes.io labels. On by default (legacy), will be turned off in 1.18. // Lock to false in v1.21 and remove in v1.22. LegacyNodeRoleBehavior featuregate.Feature = "LegacyNodeRoleBehavior" // owner @brendandburns // kep: http://kep.k8s.io/1143 // alpha: v1.9 // beta:  v1.19 // ga:  v1.21 // // Enable nodes to exclude themselves from service load balancers ServiceNodeExclusion featuregate.Feature = "ServiceNodeExclusion" // owner @smarterclayton // alpha: v1.16 // beta:  v1.19 // ga:  v1.21 // // Enable nodes to exclude themselves from network disruption checks NodeDisruptionExclusion featuregate.Feature = "NodeDisruptionExclusion" </del> // owner: @pospispa // GA: v1.11 //

What is missing? Based on SIG call it's probably "tests" but I am not sure. Could use some investigation and followup - how do we get it to GA? /sig network /kind cleanup
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
This one seems not a good first one, I'll take this to myself and start bothering people in Slack to check what's missing, mostly as I've seen this is also related to cloud providers :)
I think this one originally. According to the KEP in This should have went GA in v.19 and be removed in v1.20. Given it's still marked beta, at the very least we should mark it GA in v1.20 and remove it in v1.21. anything I'm missing here?
yeap. Honest question, changing a Feature Gate to GA process is only changing the related line in and also putting a comment that this goes to GA? If so, probably this one can also be a good first issue and we can open a future issue for the removal in v1.21 once v1.20 ships. I think this will also need to be changed in the in v1.21 right? If is ok with that, I would like then to mark this as a good first issue to turn the feature GA in v1.20 :)
Well, taking a look into how secure is to move to GA: In the KEP there are some considerations about removing the IsMasterNode function, which seems to be already dealt in commit E2E tests needs to be also checked about usage of IsMasterNode(), which I could not find anymore and also seems to be removed from the same commit above. About the same Kep, it's directly related to other Feature Gate: ` So what I think that needs to be done: Promote LegacyNodeRoleBehavior to GA and Default it to false as proposed in the Kep -I'll open another issue just for this According to the LegacyNodeRoleBehavior should be removed in v1.19 but according to the Kep in v1.21, so let's stick with v1.21? ServiceNodeExclusion - Should be moved to GA, already defaulted to true. This issue is going to deal ONLY with this.


Question: // Enable pods to set sysctls on a pod Sysctls featuregate.Feature = "Sysctls" <del> // owner @smarterclayton // alpha: v1.16 // beta:  v1.19 // ga:  v1.21 // // Enable legacy behavior to vary cluster functionality on the node-role.kubernetes.io labels. On by default (legacy), will be turned off in 1.18. // Lock to false in v1.21 and remove in v1.22. LegacyNodeRoleBehavior featuregate.Feature = "LegacyNodeRoleBehavior" // owner @brendandburns // kep: http://kep.k8s.io/1143 // alpha: v1.9 // beta:  v1.19 // ga:  v1.21 // // Enable nodes to exclude themselves from service load balancers ServiceNodeExclusion featuregate.Feature = "ServiceNodeExclusion" // owner @smarterclayton // alpha: v1.16 // beta:  v1.19 // ga:  v1.21 // // Enable nodes to exclude themselves from network disruption checks NodeDisruptionExclusion featuregate.Feature = "NodeDisruptionExclusion" </del> // owner: @pospispa // GA: v1.11 //

NodeDisruptionExclusion - Should be moved to GA, already defaulted to true. I'll open another issue for this I'm afraid of this having some sort of direct impact in cloud providers, do you want me to take this to the sig-cloudproviders meet? Otherwise can we move the above Feature Gates to GA, and schedule their removal for v1.21, wdyt?
Per deprecation lifecycle it seems that a feature needs 6 months or 2 releases (the longer one) in Beta before going to GA. So this actually seems to be the missing part before moving to GA. As the 3 FeatureGates are directed related, probably this issue should be: Move all of them to GA in v1.21 Default LegacyNodeRoleBehavior to false Sounds a good approach? I can keep this opened until we reach v1.21
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
/retitle Move Feature gate ServiceNodeExclusion to GA and LockToDefault
/triage accepted as this FG is scheduled to be moved to GA in v1.21, do you want to tackle on this?
fixed in /close
Closing this issue. <detailsIn response to : fixed in /close </details


Question: } // LocalEndpoint returns the full path to a unix socket at the given endpoint <del> func LocalEndpoint(path, file string) string { </del> <ins> func LocalEndpoint(path, file string) (string, error) { </ins> u := url.URL{ Scheme: unixProtocol, Path:   path, } <del> return filepath.Join(u.String(), file+".sock") </del> <ins> return filepath.Join(u.String(), file+".sock"), nil </ins> }

What happened: I replaced the kubelet on one of my 1.14 nodes with a build from master, and now I'm getting this error starting up. Test passes since 5/31/2019 commit have been failing on testgrid. If you look at the logs, the Windows nodes are not coming up. What you expected to happen: No errors starting up kubelet How to reproduce it (as minimally and precisely as possible): Build master, then try to run the kubelet on Windows Anything else we need to know?: /sig windows Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Azure OS (e.g: ): Windows Server 2019 Kernel (e.g. ): 10.0..529 Install tools: aks-engine
cc /priority critical-urgent
/milestone v1.15 Release team - it looks like a recent PR broke windows nodes. We will need a fix here for 1.15
Ok, so the problem here is a little more complicated and we're actually seeing a bug that is actually in the code for quite some time, just that we never used that particular code path. But first: TL;DR: Quick fix is to start the kubelet on Windows with: --feature-gates="KubeletPodResources=false" Now, as to why this is happening: PR [1] was merged 3 days ago and it graduates KubeletPodResources feature as Beta. That means kubelet will enable it by default.With the introduction of the above feature, this branch of code is entered when starting kubelet [2]. We can see a call to the function: defined here [3].While innocent enough, gets the necessary npipe by calling: . is defined here [4] for Windows . The problem is actually that url.URL can't handle Windows style paths [5] and will replace "" with "%5C" as it's to be expected. The paths are transformed along the way from the default "/var/lib/kubelet" to "varlibkubelet" by defined here [6]. More specifically in the use of to create a path. It's inputs are unix style but returns the path Windows style as it's to be expected.The solution is to replace the usage of url.URL in with some alternative package or just implement it.


Question: } // LocalEndpoint returns the full path to a unix socket at the given endpoint <del> func LocalEndpoint(path, file string) string { </del> <ins> func LocalEndpoint(path, file string) (string, error) { </ins> u := url.URL{ Scheme: unixProtocol, Path:   path, } <del> return filepath.Join(u.String(), file+".sock") </del> <ins> return filepath.Join(u.String(), file+".sock"), nil </ins> }

[1] [2] [3] [4] [5] [6]
Thanks for the analysis! I was a bit concerned around the fact that this started showing up around my commit: but the root cause here is indeed, as you said, the graduation of feature to Beta and the way the invalid escape sequences are being introduced.
GCE tests failing too /cc
I'm trying a quick to see if anything else is broken. There are other functions in that same file doing that substitution already so it may make sense to use that as a workaround until a better solution is available.
Still working on it - looks like the path to create the directories is also busted. With still getting


Question: //UploadCerts save certs needs to join a new control-plane on kubeadm-certs sercret. func UploadCerts(client clientset.Interface, cfg *kubeadmapi.InitConfiguration, key string) error { <del> fmt.Printf("[upload-certs] Storing the certificates in ConfigMap %q in the %q Namespacen", kubeadmconstants.KubeadmCertsSecret, metav1.NamespaceSystem) </del> <ins> fmt.Printf("[upload-certs] Storing the certificates in Secret %q in the %q Namespacen", kubeadmconstants.KubeadmCertsSecret, metav1.NamespaceSystem) </ins> decodedKey, err := hex.DecodeString(key) if err != nil { return err

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: (as documented ) reports this: But certificates are actually stored in a Secret (not ConfigMap) by that name and in that namespace. The message printed by kubeadm should be changed accordingly. /sig cluster-lifecycle Environment: Kubernetes version (use ): 1.14.1 Cloud provider or hardware configuration: n/a OS (e.g: ): Debian Install tools: kubeadm


Question: return rt.Response, rt.Err } <ins> func TestMaskValue(t *testing.T) { tcs := []struct { key      string value    string expected string }{ { key:      "Authorization", value:    "Basic YWxhZGRpbjpvcGVuc2VzYW1l", expected: "Basic <masked>", }, { key:      "Authorization", value:    "basic", expected: "basic", }, { key:      "Authorization", value:    "Basic", expected: "Basic", }, { key:      "Authorization", value:    "Bearer cn389ncoiwuencr", expected: "Bearer <masked>", }, { key:      "Authorization", value:    "Bearer", expected: "Bearer", }, { key:      "Authorization", value:    "bearer", expected: "bearer", }, { key:      "Authorization", value:    "bearer ", expected: "bearer", }, { key:      "Authorization", value:    "Negotiate cn389ncoiwuencr", expected: "Negotiate <masked>", }, { key:      "ABC", value:    "Negotiate cn389ncoiwuencr", expected: "Negotiate cn389ncoiwuencr", }, { key:      "Authorization", value:    "Negotiate", expected: "Negotiate", }, { key:      "Authorization", value:    "Negotiate ", expected: "Negotiate", }, { key:      "Authorization", value:    "negotiate", expected: "negotiate", }, { key:      "Authorization", value:    "abc cn389ncoiwuencr", expected: "<masked>", }, { key:      "Authorization", value:    "", expected: "", }, } for _, tc := range tcs { maskedValue := maskValue(tc.key, tc.value) if tc.expected != maskedValue { t.Errorf("unexpected value %s, given %s.", maskedValue, tc.value) } } } </ins> func TestBearerAuthRoundTripper(t *testing.T) { rt := &testRoundTripper{} req := &http.Request{}

This issue was reported in the Description Kubernetes requires an authentication mechanism to enforce users’ privileges. One method of authentication, bearer tokens, are opaque strings used to associate a user with their having successfully authenticated previously. Any user with possession of this token may masquerade as the original user (the “bearer”) without further authentication. Within Kubernetes, the bearer token is captured within the hyperkube kube-apiserver system logs at high verbosity levels (--v 10). A malicious user with access to the system logs on such a system could masquerade as any user who has previously logged into the system. Exploit Scenario Alice logs into a Kubernetes cluster and is issued a Bearer token. The system logs her token. Eve, who has access to the logs but not the production Kubernetes cluster, replays Alice’s Bearer token, and can masquerade as Alice to the cluster. Recommendation Short term, remove the Bearer token from the log. Do not log any authentication credentials within the system, including tokens, private keys, or passwords that may be used to authenticate to the production Kubernetes cluster, regardless of the logging level. Long term, either implement policies that enforce code review to ensure that sensitive data is not exposed in logs, or implement logging filters that check for sensitive data and remove it prior to outputting the log. In either case, ensure that sensitive data cannot be trivially stored in logs. Anything else we need to know?: See for current status of all issues created from these findings. The vendor gave this issue an ID of TOB-K8S-001 and it was finding 6 of the report. The vendor considers this issue Medium Severity. To view the original finding, begin on page 31 of the Environment: Kubernetes version: 1.13.4
/sig auth /sig api-machinery
/remove-kind feature /kind bug
Should remote the "Authorization" header here?
for authorization headers, everything after the type should be masked (and potentially, only recognized auth types should be logged). Something like:
I think people would be unpleasantly surprised to see credentials in a logfile after turning up logging to debug an issue. /priority important-soon
Does either of you intend to createe PR to remove the "Authorization" header ?


Question: return rt.Response, rt.Err } <ins> func TestMaskValue(t *testing.T) { tcs := []struct { key      string value    string expected string }{ { key:      "Authorization", value:    "Basic YWxhZGRpbjpvcGVuc2VzYW1l", expected: "Basic <masked>", }, { key:      "Authorization", value:    "basic", expected: "basic", }, { key:      "Authorization", value:    "Basic", expected: "Basic", }, { key:      "Authorization", value:    "Bearer cn389ncoiwuencr", expected: "Bearer <masked>", }, { key:      "Authorization", value:    "Bearer", expected: "Bearer", }, { key:      "Authorization", value:    "bearer", expected: "bearer", }, { key:      "Authorization", value:    "bearer ", expected: "bearer", }, { key:      "Authorization", value:    "Negotiate cn389ncoiwuencr", expected: "Negotiate <masked>", }, { key:      "ABC", value:    "Negotiate cn389ncoiwuencr", expected: "Negotiate cn389ncoiwuencr", }, { key:      "Authorization", value:    "Negotiate", expected: "Negotiate", }, { key:      "Authorization", value:    "Negotiate ", expected: "Negotiate", }, { key:      "Authorization", value:    "negotiate", expected: "negotiate", }, { key:      "Authorization", value:    "abc cn389ncoiwuencr", expected: "<masked>", }, { key:      "Authorization", value:    "", expected: "", }, } for _, tc := range tcs { maskedValue := maskValue(tc.key, tc.value) if tc.expected != maskedValue { t.Errorf("unexpected value %s, given %s.", maskedValue, tc.value) } } } </ins> func TestBearerAuthRoundTripper(t *testing.T) { rt := &testRoundTripper{} req := &http.Request{}

This issue seems CVE-worthy to me: loss of confidentiality (bearer tokens) + cross of privilege boundary (debug log level could be set by admin, then logs read later by low privileged user). My draft cvss would be: CVSS:3.0/AV:N/AC:H/PR:L/UI:N/S:U/C:H/I:N/A:N (5.3)
Two potential vectors: local user with access to system logs (local access, low privilege) (4.7) remote user with access to system logs (network access, high privilege): (4.4) CVE-2019- has been assigned to this issue.
If you want to, go ahead
- should this have the official CVE feed label ( retrospectively, I mean)?
probably? I have no idea how that label is supposed to get applied, but there appear to be several CVE issues that don't have it -
I think you're likely to know who to highlight this to. Sorry to mention you individually; please feel free to point out docs if there are any.
/label official-cve-feed thanks for the tag, the label now!
thanks for the useful filter. Currently in-scope issues are issues that were closed as fixed by code changes to k/k. So with that in mind, I have the labels to some of the closed issues too that conform to the scope defined here:
I would like to reply nearly four years later: There are valid cases to want to see the token, for example my current case, trying to debug token validation between the API server and other services.


Question: Expect(err).ToNot(HaveOccurred()) }) }) <ins> Context("Pods sharing a single local PV [Serial]", func() { var ( pv *v1.PersistentVolume ) BeforeEach(func() { localVolume := &localTestVolume{ node:            config.node0, hostDir:         "/tmp", localVolumeType: DirectoryLocalVolumeType, } pvConfig := makeLocalPVConfig(config, localVolume) var err error pv, err = framework.CreatePV(config.client, framework.MakePersistentVolume(pvConfig)) framework.ExpectNoError(err) }) AfterEach(func() { if pv == nil { return } By(fmt.Sprintf("Clean PV %s", pv.Name)) err := config.client.CoreV1().PersistentVolumes().Delete(pv.Name, &metav1.DeleteOptions{}) framework.ExpectNoError(err) }) It("all pods should be running", func() { var ( pvc   *v1.PersistentVolumeClaim pods  = map[string]*v1.Pod{} count = 50 err   error ) pvc = framework.MakePersistentVolumeClaim(makeLocalPVCConfig(config, DirectoryLocalVolumeType), config.ns) By(fmt.Sprintf("Create a PVC %s", pvc.Name)) pvc, err = framework.CreatePVC(config.client, config.ns, pvc) framework.ExpectNoError(err) By(fmt.Sprintf("Create %d pods to use this PVC", count)) for i := 0; i < count; i++ { pod := framework.MakeSecPod(config.ns, []*v1.PersistentVolumeClaim{pvc}, false, "", false, false, selinuxLabel, nil) pod, err := config.client.CoreV1().Pods(config.ns).Create(pod) Expect(err).NotTo(HaveOccurred()) pods[pod.Name] = pod } By("Wait for all pods are running") err = wait.PollImmediate(time.Second, 5*time.Minute, func() (done bool, err error) { podsList, err := config.client.CoreV1().Pods(config.ns).List(metav1.ListOptions{}) if err != nil { return false, err } runningPods := 0 for _, pod := range podsList.Items { switch pod.Status.Phase { case v1.PodRunning: runningPods++ } } return runningPods == count, nil }) Expect(err).ToNot(HaveOccurred()) }) }) </ins> }) func deletePodAndPVCs(config *localTestConfig, pod *v1.Pod) error {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : New test case to cover scenario in Why is this needed: Prevent this issue happening again.
/assign /sig storage