Question: } type nodeHolder struct { <ins> lock sync.Mutex </ins> node *v1.Node } <ins> func (n *nodeHolder) setNode(node *v1.Node) { n.lock.Lock() defer n.lock.Unlock() n.node = node } </ins> func (n *nodeHolder) getNode(name string) (*v1.Node, error) { <ins> n.lock.Lock() defer n.lock.Unlock() </ins> return n.node, nil }

i was more inquiring if there was a test that could automate the reproduction, so even just a linked commit may be helpful for someone that wants to pick this up. i am happy to help if you want to reach out over slack later this week.
looks like a unit test could reproduce the issue, opened a WIP fix in
our code is not publicly accessible, I'm sorry. PR linked by seems to cover both the fix and the test that proves it.
/remove-lifecycle frozen
frozen keeps the bot from auto-closing
Ah, sorry then.


Question: ### **Output:** ## [Kubelet-OutputToLogs](https://github.com/kubernetes/kubernetes/blob/release-1.9/test/e2e_node/kubelet_test.go#L42) <ins> ### Release v1.9 </ins> By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs. Note this test needs to be fixed to also test for stderr

What happened: The Conformance Document currently has a static header that is displayed from a file. The release information in the header should reflect proper release number so that the document reflect the current state. What you expected to happen: Should show proper current release information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
in reaction to comments is still hardcoded to mention v1.9 at the top, could we parameterize this?
/sig testing /area conformance /release-note-none
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
/priority backlog
This seems to have been addressed at Should we just close this?
No, specifically did not address this issue, this should stay open. See


Question: OptionsExternalVersion: &optionsExternalVersion, Serializer:     thirdpartyresourcedata.NewNegotiatedSerializer(api.Codecs, kind, externalVersion, internalVersion), <del> ParameterCodec: api.ParameterCodec, </del> <ins> ParameterCodec: thirdpartyresourcedata.NewThirdPartyParameterCodec(api.ParameterCodec), </ins> Context: m.RequestContextMapper,

A user report that it's broken in the slack channel:
Also and -- this probably broke with one of our recent refactors. The server should be accepting list options from any version, I thought?
FWIW, I think it's this line barf at the ListOptions: The parameter codec first decodes the query parameter to a versioned object and then convert it to the internal object. The code fails because it cannot create a "" ListOptions.
The server should accept list options in the API version it is sent. ListOptions should be registered in all third party groups.
The thirdPartyResourceDataCreator should handle "ListOptions", which surprises me that it doesn't then respect the group.
I think we should add a wrapper around the api.ParameterCodec here: Do something similar as what does.
Putting together a PR now and will let you guys review.
I guess my point is, the list options are just there from the URL query parameters. The user didn't send a group. Why do we assume that value for the group rather than "/v1" or something?
My understanding is we want to allow options to evolve differently in different groups. We do have v1.ListOptions and v1beta1.ListOptions. And if a client sends requeset to extensions/v1beta1 endpoint, then it's expected to encode the ListOptions to v1beta1 and then convert it to query parameters.
Mostly because the parameter codec is supposed to abstract that. "virtual" groups need to be implemented by the virtual "parameter codec". The statement on floating server objects (List, Delete etc) was that they have a locked API schema, and we have a mechanism to do that via the scheme which is registering them in a group. , Daniel Smith wrote:


Question: pod        *v1.Pod nodeInfo   *framework.NodeInfo name       string <ins> features   map[featuregate.Feature]bool </ins> wantStatus *framework.Status }{ {

What happened: In a v1.18 cluster with the LocalStorageCapacityIsolation feature gate disabled setting an ephemeral-storage request or limit causes the pod to be unschedulable: If the ephemeral-storage request is removed the pod is scheduled. This behavior exists in v1.19 and a recent v1.20 build. What you expected to happen: In previous releases such pods are scheduled. I expect that any ephemeral-storage limit is not enforced. How to reproduce it (as minimally and precisely as possible): Disable the LocalStorageCapacityIsolation feature guide by adding to the kube-scheduler. Create a pod like this: Anything else we need to know?: The problem was originally seen after moving from IBM Cloud Red Hat OpenShift v4.4 to v4.5. Pods that worked in 4.4 (Kubernetes v1.17) remained in a Pending state in OpenShift v4.5 (Kubernetes 1.18). We discovered that the IBM Cloud OpenShift service has the LocalStorageCapacityIsolation feature-gate disabled while Red Hat has enabled the feature. We then replicated the behavior with a plain Kubernetes cluster. Environment: Kubernetes version (use ): v1.18.10, v1.19.3, v1.20.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-scheduling
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
I can confirm it's a bug. Will work on it.


Question: pod        *v1.Pod nodeInfo   *framework.NodeInfo name       string <ins> features   map[featuregate.Feature]bool </ins> wantStatus *framework.Status }{ {

/assign
Clarifying the recreate steps: I disabled the LocalStorageCapacityIsolation feature guide by adding to the kube-scheduler since this behavior seems to be within the scheduler.
Thanks for confirming!
This bug has been fixed in master branch, and will be backported to [1.18] and [1.19] soon. [1.18]: [1.19]:


Question: `, }, { <ins> name: "arbitrary container metrics for negative StartTime", summary: &statsapi.Summary{ Pods: []statsapi.PodStats{ { PodRef: statsapi.PodReference{ Name:      "pod_a", Namespace: "namespace_a", }, Containers: []statsapi.ContainerStats{ { Name:      "container_a", StartTime: metav1.NewTime(time.Unix(0, -1624396278302091597)), CPU: &statsapi.CPUStats{ Time:                 testTime, UsageCoreNanoSeconds: uint64Ptr(10000000000), }, Memory: &statsapi.MemoryStats{ Time:            testTime, WorkingSetBytes: uint64Ptr(1000), }, }, }, }, }, }, summaryErr: nil, expectedMetrics: ` # HELP scrape_error [ALPHA] 1 if there was an error while getting container metrics, 0 otherwise # TYPE scrape_error gauge scrape_error 0 # HELP container_cpu_usage_seconds_total [ALPHA] Cumulative cpu time consumed by the container in core-seconds # TYPE container_cpu_usage_seconds_total counter container_cpu_usage_seconds_total{container="container_a",namespace="namespace_a",pod="pod_a"} 10 1624396278302 # HELP container_memory_working_set_bytes [ALPHA] Current working set of the container in bytes # TYPE container_memory_working_set_bytes gauge container_memory_working_set_bytes{container="container_a",namespace="namespace_a",pod="pod_a"} 1000 1624396278302 `, }, { </ins> name: "nil container metrics", summary: &statsapi.Summary{ Pods: []statsapi.PodStats{

As described in , When the pod start or stop, negative metrics generated. generate the correct metrics pod restart No response <details</details<detailsnone </details<details</details<details</details<details</details<details</details
/sig node /sig instrumentation
can you please give a little bit more information on steps to repro. Is it happens all the time? Is the root cause known (I see a linked PR). /triage needs-information
This is a feedback from contributors in kubernetes-sigs/metrics-server. It is riggered on specific conditions. From the description of several issues, the way to reproduce should be pod start or stop. such as:
Do you know if this is specific to ?
I'm not sure. However, obtains data from the kubelet endpoint. If there is a negative value, it will cause statistical problems.
/triage accepted /assign
FYI I encounter this with . seems to be an uninitialized time type. The in is create from the response of cri RPC, this maybe cause either by cri-o or the metrics collect part of Kubernetes. I have not dig into it yet.


Question: // Allow traffic bound for external IPs that happen to be recognized as local IPs to stay local. // This covers cases like GCE load-balancers which get added to the local routing table. writeLine(natRules, append(dstLocalOnlyArgs, "-j", string(svcChain))...) <ins> // If the service has no endpoints then reject packets coming via externalIP // Install ICMP Reject rule in filter table for destination=externalIP and dport=svcport if len(proxier.endpointsMap[svcName]) == 0 { writeLine(filterRules, "-A", string(kubeServicesChain), "-m", "comment", "--comment", fmt.Sprintf(`"%s has no endpoints"`, svcNameString), "-m", protocol, "-p", protocol, "-d", fmt.Sprintf("%s/32", externalIP), "--dport", fmt.Sprintf("%d", svcInfo.port), "-j", "REJECT", ) } </ins> } // Capture load-balancer ingress.

Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? externalIPs kube-proxy connection accepted no pods running. icmp reject rules. Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): kubectl version Environment: Cloud provider or hardware configuration: Bare Metal. OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others: What happened: is exposed using externalIP construct. pods are backing the service (using scale deployment 0). The pod is expected to run a TCP based server (non-http) client connects to this service using externalIP as dest-ip; connection gets accepted by kube-proxy. What you expected to happen: When no pods are backing service; the connection should have been rejected and definitely not accepted by kube-proxy. How to reproduce it (as minimally and precisely as possible): a service and expose it using externalIP construct pods are present; bring them down using"kubectl scale deployment my_svc --replicas 0 --namespace xyz initiates the connection to this service using externalIP and connection goes fine but to kube-proxy. Anything else we need to know: Info on iptable rules - When service and pods are present the rules like these - pods are not present; iptables rules look like this - that the reject rule exist only for the destination=ServiceIP=10.100.200.6/32 no pods are present; client gets connected to kube-proxy On VM - Based on and ; I manually installed reject rule on INPUT chain. I do not see the issue happening again - meaning client then cannot connect to the server when no pods are present.
I have a suggestion for installing icmp reject rules. Why cant icmp reject rule always present in INPUT chain? kube-proxy do not really need to check if pods are present or not for installing these rules. For the svcip/externaip to pod-ip translation; kube-proxy anyways install the rule to do DNAT in which case the icmp reject rule from INPUT wont be hit. In case when DNAT fails to happen - e.g. in cases like pods not present - icmp reject rule will anyways kick in to reject such packet.


Question: // Allow traffic bound for external IPs that happen to be recognized as local IPs to stay local. // This covers cases like GCE load-balancers which get added to the local routing table. writeLine(natRules, append(dstLocalOnlyArgs, "-j", string(svcChain))...) <ins> // If the service has no endpoints then reject packets coming via externalIP // Install ICMP Reject rule in filter table for destination=externalIP and dport=svcport if len(proxier.endpointsMap[svcName]) == 0 { writeLine(filterRules, "-A", string(kubeServicesChain), "-m", "comment", "--comment", fmt.Sprintf(`"%s has no endpoints"`, svcNameString), "-m", protocol, "-p", protocol, "-d", fmt.Sprintf("%s/32", externalIP), "--dport", fmt.Sprintf("%d", svcInfo.port), "-j", "REJECT", ) } </ins> } // Capture load-balancer ingress.

This would be better than accidentally giving the packet to kube-proxy which is always listening on the port and accepting this accidental connection even when it does not intend to. Actually, for now that's how I am putting workaround for this problem in our setup.
That might be a clever answer - I hadn't thought of it that way. Would need good tests and comments.
I can give a shot at testing it. I didnt find a good reference document / todo for testing such changes. If you can help me locate it; I can give it a try.


Question: "//pkg/util/async:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/util/intstr:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//staging/src/k8s.io/apiserver/pkg/features:go_default_library",

There is no value by that conversion, even worse it can be misused.
should we use in such case?
I do not think this is a bug. 25% == .25 and int(.25) == 0. ;) Even in the case where the user puts in something like 100% I do not think it's a bug. Users should not put strings like "25%" into intOrString and expect to get the integer 25 out. Likely a user doing that is confused about what format the field takes in. I am not sure why the function GetValueFromIntOrPercent exists, and I bet I am not going to like the answer :)
No. First, this depends on how you do the rounding. If you are rounding up (MaxSurge in Deployments), this is 1. Then, the replica size of the deployment also plays a part. 25% of a controller with 1 replica is different from a controller with 4 replicas, etc.
Also, the fact that IntValue returns 0 for every percentage has nothing to do with any logic. It's purely because the conversion errors out:
the first line there was supposed to be a joke, sorry. :) I know that the actual reason for returning 0 is the % sign means it is not an int. More seriously: I actually agree with the parsing function that percentages are not ints. Users who want to do this should not use IntOrString. Is there a place in the API that wants to do this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen /remove-lifecycle stale
Opened to ensure all in-tree callers only call this when the type is Int (or verify it is a parseable number first), and to update the method documentation


Question: // ensureSandboxImageExists pulls the sandbox image when it's not present. func ensureSandboxImageExists(client libdocker.Interface, image string) error { <del> dockerCfgSearchPath := []string{"/.docker", filepath.Join(os.Getenv("HOME"), ".docker")} return ensureSandboxImageExistsDockerCfg(client, image, dockerCfgSearchPath) } func ensureSandboxImageExistsDockerCfg(client libdocker.Interface, image string, dockerCfgSearchPath []string) error { </del> _, err := client.InspectImageByRef(image) if err == nil { return nil

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: We have a private registry configured (where all images including the sandbox one), are hosted. The registry credentials are being pulled by one of k8s credentials providers () With Kubernetes 1.7 sandbox image failed to be pulled with the following error message: Looks like with this only creds from docker config are being evaluated on the sandbox image pull What you expected to happen: Sandbox image pull should be attempted using credentials returned by not just the basic docker credential provider, but other credentials providers registered in the system. How to reproduce it (as minimally and precisely as possible): 1) Register private repo credentials with one of the k8s credential providers (we've tried with rancher). Configure the to use the image from the the private repo. 2) Make sure you are not logged in to this private repo on the kubelet host 3) Try to create a pod. Sandbox image pull will fail with an error mentioned above. Anything else we need to know?: Non-sandbox images from the same private repo were able to download correctly using private repo credentials provider. Environment: Kubernetes version (use ): 1.7.2 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node Just tested the same use case against k8s 1.5.x, it worked fine there and sandbox image was pulled using the credentials passed by the credential provider, not from docker config file. So it might have regressed in 1.7.x (haven't tested it on 1.6, so can't tell for sure)
Confirmed a bug introduced in dockershim. Will file a PR to fix this.


Question: // ensureSandboxImageExists pulls the sandbox image when it's not present. func ensureSandboxImageExists(client libdocker.Interface, image string) error { <del> dockerCfgSearchPath := []string{"/.docker", filepath.Join(os.Getenv("HOME"), ".docker")} return ensureSandboxImageExistsDockerCfg(client, image, dockerCfgSearchPath) } func ensureSandboxImageExistsDockerCfg(client libdocker.Interface, image string, dockerCfgSearchPath []string) error { </del> _, err := client.InspectImageByRef(image) if err == nil { return nil

Could you also verify whether fixed your problem?
tested against your commits cherry-picked on top of 1.8 beta, the fix seems to work
thanks.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: echo "Setting up cluster monitoring using Heapster." detect-project <del> if ! gcloud compute firewall-rules --project "{$PROJECT}" describe monitoring-heapster &>/dev/null; then </del> <ins> if ! gcloud compute firewall-rules --project "${PROJECT}" describe monitoring-heapster &> /dev/null; then </ins> if ! gcloud compute firewall-rules create monitoring-heapster  --project "${PROJECT}"  --target-tags="${MINION_TAG}"  --network="${NETWORK}"  <del> --allow tcp:80 tcp:8083 tcp:8086 tcp:9200; then echo "Failed to set up firewall for monitoring" && false </del> <ins> --allow tcp:80 tcp:8083 tcp:8086; then echo -e "${color_red}Failed to set up firewall for monitoring ${color_norm}" && false </ins> fi fi <del> # Re-use master auth for Grafana get-password ensure-temp-dir sed -e "s/{KUBE_USER}/$KUBE_USER/g"  -e "s/{KUBE_PASSWORD}/$KUBE_PASSWORD/g"  "${KUBE_ROOT}/cluster/addons/cluster-monitoring/influx-grafana-pod.json"  > "${KUBE_TEMP}/influx-grafana-pod.json" </del> local kubectl="${KUBE_ROOT}/cluster/kubectl.sh" <del> if "${kubectl}" create -f "${KUBE_TEMP}/influx-grafana-pod.json" &> /dev/null  && "${kubectl}" create -f "${KUBE_ROOT}/cluster/addons/cluster-monitoring/influx-grafana-service.json" &> /dev/null  && "${kubectl}" create -f "${KUBE_ROOT}/cluster/addons/cluster-monitoring/heapster-pod.json" &> /dev/null; then local dashboard_url="http://$(${kubectl} get -o json pod influx-grafana | grep hostIP | awk '{print $2}' | sed 's/[,|"]//g')" echo echo "Grafana dashboard will be available at $dashboard_url. Wait for the monitoring dashboard to be online." echo "Use the master user name and password for the dashboard." echo </del> <ins> local grafana_host="" if "${kubectl}" create -f "${KUBE_ROOT}/cluster/addons/cluster-monitoring/" &> /dev/null; then # wait for pods to be scheduled on a node. echo "waiting for monitoring pods to be scheduled." for i in `seq 1 10`; do grafana_host=$("${kubectl}" get pods -l name=influxGrafana -o template -t {{range.items}}{{.currentState.hostIP}}:{{end}}) if [[ $grafana_host != *"<"* ]]; then break fi sleep 10 done if [[ $grafana_host != *"<"* ]]; then echo echo -e "${color_green}Grafana dashboard will be available at ${color_yellow}http://$grafana_host${color_green}. Wait for the monitoring dashboard to be online.${color_norm}" echo else echo -e "${color_red}monitoring pods failed to be scheduled.${color_norm}" fi </ins> else <del> echo "Failed to Setup Monitoring" </del> <ins> echo -e "${color_red}Failed to Setup Monitoring ${color_norm}" </ins> teardown-monitoring fi fi

examples/monitoring/influx-grafana- and friends NEED to be replaced by a replication controller ASAP. As it stands they are bad examples. Also, they should probably move to contrib/monitoring to match contrib/logging and contrib/dns


Question: fi; # BEGIN_PROMETHEUS_TO_SD - name: prometheus-to-sd-exporter <del> image: k8s.gcr.io/prometheus-to-sd:v0.5.0 </del> <ins> image: k8s.gcr.io/prometheus-to-sd:v0.11.1 </ins> command: - /monitor - --stackdriver-prefix={{ prometheus_to_sd_prefix }}/addons

the following images are using old client-go that would not work correctly starting 1.22+ if those images use in-cluster config(mounted service account token) to talk to kube-apiserver: if the above images use in-cluster config, they should use client-go 1.14+. xref: /assign /cc /sig instrumentation /priority important-soon
/triage accepted


Question: if svcInfo.localTrafficDSR { externalIPEndpoints = hnsLocalEndpoints } <del> // Try loading existing policies, if already available hnsLoadBalancer, err = hns.getLoadBalancer( externalIPEndpoints, loadBalancerFlags{isDSR: svcInfo.localTrafficDSR, sessionAffinity: sessionAffinityClientIP, isIPv6: proxier.isIPv6Mode}, sourceVip, externalIP.ip, Enum(svcInfo.Protocol()), uint16(svcInfo.targetPort), uint16(svcInfo.Port()), ) if err != nil { klog.ErrorS(err, "Policy creation failed") continue </del> <ins> if len(externalIPEndpoints) > 0 { // Try loading existing policies, if already available hnsLoadBalancer, err = hns.getLoadBalancer( externalIPEndpoints, loadBalancerFlags{isDSR: svcInfo.localTrafficDSR, sessionAffinity: sessionAffinityClientIP, isIPv6: proxier.isIPv6Mode}, sourceVip, externalIP.ip, Enum(svcInfo.Protocol()), uint16(svcInfo.targetPort), uint16(svcInfo.Port()), ) if err != nil { klog.ErrorS(err, "Policy creation failed") continue } externalIP.hnsID = hnsLoadBalancer.hnsID klog.V(3).InfoS("Hns LoadBalancer resource created for externalIP resources", "externalIP", externalIP, "hnsID", hnsLoadBalancer.hnsID) } else { klog.V(3).InfoS("Skipped creating Hns LoadBalancer for externalIP resources", "externalIP", externalIP, "hnsID", hnsLoadBalancer.hnsID) </ins> } <del> externalIP.hnsID = hnsLoadBalancer.hnsID klog.V(3).InfoS("Hns LoadBalancer resource created for externalIP resources", "externalIP", externalIP, "hnsID", hnsLoadBalancer.hnsID) </del> } // Create a Load Balancer Policy for each loadbalancer ingress for _, lbIngressIP := range svcInfo.loadBalancerIngressIPs {

During a Kubernetes deployment the connectivity to web services from Windows nodes is flapping. There are intermittent traffic delays and some errors observed on web services during the deployment. Post deployment the flapping is no longer observed. No flapping in connectivity during the deployment. Create a deployment with multiple services and monitor connectivity. No response <details20 </details<detailsAKS </details<details</details<details</details<details</details<details</details
/sig windows
/triage accepted /priority important-soon


Question: type JSONPath struct { name       string parser     *Parser <del> stack      [][]reflect.Value // push and pop values in different scopes cur        []reflect.Value   // current scope values </del> beginRange int inRange    int endRange   int <ins> lastEndNode *Node </ins> allowMissingKeys bool }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: (or in my instance) with does not correctly handle nested range statements. For example, one would expect that the following: would result in a list being output similar to this: but instead, results in a list being output like this: What you expected to happen: Nested range statements complete in the order in which they are specified in jsonpath query. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.11.0 Cloud provider or hardware configuration: OpenShift 3.11.141 on vSphere 6.5 OS (e.g: ): RHEL 7.7 Kernel (e.g. ): 3.10.0 Install tools:
/sig cli
/area kubectl
/assign
/sig api-machinery
/remove-sig cli
/remove-area cli /area api
Those labels are not set on the issue: <detailsIn response to : /remove-area cli /area api Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-area kubectl


Question: } func copyToPod(f cmdutil.Factory, cmd *cobra.Command, stdout, stderr io.Writer, src, dest fileSpec) error { <del> if len(src.File) == 0 { </del> <ins> if len(src.File) == 0 || len(dest.File) == 0 { </ins> return errFileCannotBeEmpty } reader, writer := io.Pipe() // strip trailing slash (if any) <del> if strings.HasSuffix(string(dest.File[len(dest.File)-1]), "/") { </del> <ins> if dest.File != "/" && strings.HasSuffix(string(dest.File[len(dest.File)-1]), "/") { </ins> dest.File = dest.File[:len(dest.File)-1] }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: panics What you expected to happen: Syntax error, as it is not legit cp syntax. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Occurs in every version 1.9.x and 1.10.x, but 1.8.x is safe. is similar, but it has a fix candidate already going in, that is not fixing this, so opening a separate issue. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: VM OS (e.g. from /etc/os-release): Ubuntu 18.04 Kernel (e.g. ): /sig cli
It seems to be fixed in 1.11 by
It was merged three months ago, any idea why it isn't in 1.10 too?
It was merged on March 31 and 1.10 was released on March 27: Can confirm it is not present in 1.10: Maybe we can cherry-pick it?
I've opened
Fixed by /close
Closing this issue. <detailsIn response to : Fixed by Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return &container, nil } } <ins> for _, container := range pod.Spec.InitContainers { if container.Name == containerName { return &container, nil } } </ins> return nil, fmt.Errorf("container %s not found", containerName) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When referencing init container's env value from its fields, kubelet complains What you expected to happen: Init container should be supported to get env values from its own fields, as to How to reproduce it (as minimally and precisely as possible): a pod with init container refering its fields value as env value kubectl create -f test-pod-1.yaml cannot be created pod events Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The err happens at this line , it seems that we should search for init containers as well


Question: }, }, { <del> testName:           "node exists by node name", existsByProviderID: false, providerIDErr:      errors.New("unimplemented"), existsByNodeName:   true, </del> <ins> testName:           "exists by instance id", existsByProviderID: true, providerIDErr:      nil, hasInstanceID:      true, </ins> nodeNameErr:        nil, <del> expectedCalls:      []string{"instance-exists-by-provider-id", "instance-id"}, </del> <ins> expectedCalls:      []string{"instance-id", "instance-exists-by-provider-id"}, expectedNodeExists: true, </ins> node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name: "node0", }, <del> Spec: v1.NodeSpec{ ProviderID: "node0", </del> <ins> }, }, { testName:           "does not exist by no instance id", existsByProviderID: true, providerIDErr:      nil, hasInstanceID:      false, nodeNameErr:        cloudprovider.InstanceNotFound, expectedCalls:      []string{"instance-id"}, expectedNodeExists: false, node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name: "node0", </ins> }, }, }, { <del> testName:           "does not exist by node name", </del> <ins> testName:           "provider id returns error", </ins> existsByProviderID: false, providerIDErr:      errors.New("unimplemented"), <del> existsByNodeName:   false, </del> <ins> hasInstanceID:      true, </ins> nodeNameErr:        cloudprovider.InstanceNotFound, <del> expectedCalls:      []string{"instance-exists-by-provider-id", "instance-id"}, </del> <ins> expectedCalls:      []string{"instance-exists-by-provider-id"}, expectedNodeExists: false, </ins> node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name: "node0",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The change here removes exiternal id, but the call to instance id was not made used at all, it will always return false. At this time no node was able to register What you expected to happen: ProviderId was used in check How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment:
similar to are you using --cloud-provider in your controller manager? are you also setting it on your kubelet?
I'm using acs-engine to deploy the cluster. Config is as following: I've proposed a fix. PTAL
/sig cluster-lifecycle


Question: framework.ExpectNoError(err) } <ins> func (l *ltrMgr) expandLocalVolumeBlockFS(ltr *LocalTestResource, mbToAdd int) error { ddCmd := fmt.Sprintf("dd if=/dev/zero of=%s/file conv=notrunc oflag=append bs=1M count=%d", ltr.loopDir, mbToAdd) loopDev := l.findLoopDevice(ltr.loopDir, ltr.Node) losetupCmd := fmt.Sprintf("losetup -c %s", loopDev) return l.hostExec.IssueCommand(fmt.Sprintf("%s && %s", ddCmd, losetupCmd), ltr.Node) } func (l *ltrMgr) ExpandBlockDevice(ltr *LocalTestResource, mbtoAdd int) error { switch ltr.VolumeType { case LocalVolumeBlockFS: return l.expandLocalVolumeBlockFS(ltr, mbtoAdd) } return fmt.Errorf("Failed to expand local test resource, unsupported volume type: %s", ltr.VolumeType) } </ins> func (l *ltrMgr) Create(node *v1.Node, volumeType LocalVolumeType, parameters map[string]string) *LocalTestResource { var ltr *LocalTestResource switch volumeType {

There are use cases where users want to expand the local volume (may be managed by LVM or something similar), but even though they can edit underlying PV (and expand the underlying device) - the reported size in pvc.Status keeps reporting older value because in-tree local volumes are non-expandable. We could make this easier for users by - implementing interface for local volumes. This would allow following steps for local volume expansion: /admin edits the pvc - requesting new size. expands the underlying device. edits the PV and sets new size. will expand the filesystem and update This would at least get the user from impossible territory to possible. cc /sig storage
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hi & any news on this issue ? The related PR seems stalled.
merged. So this is fixed now.
Oh right ! I didn't looked close enough and thought it was only for directory based volumes. Thank you.


Question: source "${KUBE_ROOT}/hack/lib/init.sh" source "${KUBE_ROOT}/hack/lib/util.sh" # This sets up the environment, like GOCACHE, which keeps the worktree cleaner. kube::golang::setup_env kube::util::ensure-temp-dir # Creating a new repository tree # Deleting vendor directory to make go-licenses fetch license URLs from go-packages source repository git worktree add -f "${KUBE_TEMP}"/tmp_test_licenses/kubernetes HEAD >/dev/null 2>&1 || true cd "${KUBE_TEMP}"/tmp_test_licenses/kubernetes && rm -rf vendor # Ensure that we find the binaries we build before anything else. export GOBIN="${KUBE_OUTPUT_BINPATH}" PATH="${GOBIN}:${PATH}" # Explicitly opt into go modules, even though we're inside a GOPATH directory export GO111MODULE=on <ins> function http_code() { curl -I -s -o /dev/null -w "%{http_code}" "$1" } </ins> allowed_licenses=() packages_flagged=()

When I run it manually, I get: Clearly that is wrong, it's just looking in the wrong place. I also see a reference to which is still open 2 years later. Does this script run? Is it useful?
/sig testing
This job runs the script and fails as well:
If it ran fast, would we run it on every PR?
Another note: It runs around 5-6 minutes right now:
It has a very poorly placed n^2 loop. That can be fixed. Unfortunately the go-licenses tool itself is really slow, too.
It looks like that go-licenses cannot handle multi module repos if the license is only available in the repo root: The question is: do we want to contribute such a feature or don't we need the license test at all?
We should vet licenses in presubmit. Though I think dep approvers are pretty good at checking this, one less thing to manually vet would be ideal. If we can't speed it up, we can add a job that only runs on vendor/ changes.
/triage accepted


Question: func TestEmptyCPURequest(t *testing.T) { tc := testCase{ <del> minReplicas:     1, maxReplicas:     5, initialReplicas: 1, desiredReplicas: 1, CPUTarget:       100, reportedLevels:  []uint64{200}, useMetricsAPI:   true, </del> <ins> minReplicas:         1, maxReplicas:         5, initialReplicas:     1, desiredReplicas:     1, CPUTarget:           100, reportedLevels:      []uint64{200}, reportedCPURequests: []resource.Quantity{}, useMetricsAPI:       true, </ins> expectedConditions: []autoscalingv1.HorizontalPodAutoscalerCondition{ {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "SucceededGetScale"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionFalse, Reason: "FailedGetResourceMetric"},

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: We are starting to use the HPA (horizontal pod autoscaling) for one of our components. What is special about this particular component is that the warm up period is very long (sometimes even more than 30 minutes). During that time, the pod is in Running state but its containers are in Not Ready state. This pod is of course not able to receive any traffic so the metric we use in the HPA is ignoring those pods that are not Ready. The equation used by HPA to calculate the new number of replicas is as follows Our problem is that the equation used by HPA to calculate the new number of replicas is including in the "current replica count" the pods that are "Not Ready" so the value for "new replica count" is not correct. Since this warm up is so long we end up by having a lot of more instances of our services of what it should be necessary. They eventually get deleted once gradually the metrics start going down but that will take a lot of time and a big waste of resources without mentioning that with a changing traffic this HPA will never stabilize What you expected to happen: I would like to have a flag somewhere where I tell HPA to don't include in the "current replica count" the pods that are in "Not Ready" state How to reproduce it (as minimally and precisely as possible): Have a component that takes a lot of time to start and being able to receive traffic. Then define a metric that will ignore the pods that are Not Ready. Finally create a metric that will scale that component based on the mentioned metric Anything else we need to know?: Environment: Kubernetes version 1.7 Cloud provider or hardware configuration: Azure
/sig autoscaling
Hi - thanks for reaching out about this! It looks like this section of the , handles when pods aren't ready. I think you this behavior in , so I'm interested in your thoughts on whats best here. If we do end up adding a flag for this, I'd be happy to write the code :) Also, purely for my own curiosity, I'm curious what type of pod is taking 30 minutes to become ready. If your traffic pattern varies at a rate faster than 30 minutes, it may be difficult for auto-scaling to be helpful.
can you also share a bit more about your HPA?


Question: func TestEmptyCPURequest(t *testing.T) { tc := testCase{ <del> minReplicas:     1, maxReplicas:     5, initialReplicas: 1, desiredReplicas: 1, CPUTarget:       100, reportedLevels:  []uint64{200}, useMetricsAPI:   true, </del> <ins> minReplicas:         1, maxReplicas:         5, initialReplicas:     1, desiredReplicas:     1, CPUTarget:           100, reportedLevels:      []uint64{200}, reportedCPURequests: []resource.Quantity{}, useMetricsAPI:       true, </ins> expectedConditions: []autoscalingv1.HorizontalPodAutoscalerCondition{ {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "SucceededGetScale"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionFalse, Reason: "FailedGetResourceMetric"},

What kind of metric type is it? Based on the description I'm suspecting that it is an Object metric, am I right?
So answer about why our process takes that much time to start. Basically this is a server that in order to do what he needs to do, he needs to load a lot of data in memory (several GBs) and then perform some warm up that are pretty intensive. The idea is that he doesn't need to grab (almost) anything from outside to attend a request so it's latency is extremely low (in the order of millisecond) By the way. Because of this we have another problem which is that there is not an specific flag per HPA object that we could tell k8s to don't try to scale up again passed some time so the new pods have enough time to get ready before the next scale happens. I know that there is a cluster wide flag but we will need to have a different one by resource that we scale. Now to answer You pointed another interesting topic. The metric we use is % Utilization. We are able to calculate the level of utilization of our pods and we want to keep this metric in an specific value. About the flag. I suggested the flag so it doesn't change the existing behavior by default but I honestly think that the default behavior should be to don't include the NonReady pods as part of the calculation
do you think we can add this change to 1.10? it's critical bug/feature for us and we are happy to help with code/testing.
It's really late for 1.10, especially since we don't have any code here, and I'd like to know more about what's going on here, and especially see if it reproduces on Kubernetes 1.9 (we've had HPA controller bugs in the past). FWIW, currently unready pods are treated as consuming 0% of their utilization on scale up, and it was intended that this should prevent such issues. That is, the formula is actually: If the pods are marked as not ready, I find it strange that they would be affecting the scaling that much, unless your other pods are under real heavy load (i.e. I would assume that the unready pods would skew the scaling back down so that the number of pods just stayed constant).
I believe there is different code branch here.


Question: func TestEmptyCPURequest(t *testing.T) { tc := testCase{ <del> minReplicas:     1, maxReplicas:     5, initialReplicas: 1, desiredReplicas: 1, CPUTarget:       100, reportedLevels:  []uint64{200}, useMetricsAPI:   true, </del> <ins> minReplicas:         1, maxReplicas:         5, initialReplicas:     1, desiredReplicas:     1, CPUTarget:           100, reportedLevels:      []uint64{200}, reportedCPURequests: []resource.Quantity{}, useMetricsAPI:       true, </ins> expectedConditions: []autoscalingv1.HorizontalPodAutoscalerCondition{ {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "SucceededGetScale"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionFalse, Reason: "FailedGetResourceMetric"},

metric definition { "type": "Object", "object": { "target": { "kind": "Service", "name": "prometheus-exporter" }, "metricName": "PercentUtilization", "targetValue": "20" } } which should handle as and doesn't check Pod state.
I think proper handling of unready pods is a difficult problem that will require significant work. I suspect we may find out we need to start distinguishing pods that are booting up from pods that are 'broken' and it took us quite a while to solve analogous problem with nodes in Cluster Autoscaler (to be clear: I don't know if we will have to do that, that's just my intuition for now). That being said I agree the formula in GetObjectMetricReplicas will generally lead to scaling-up too much if the pod takes longer to initialize then backoff period in HPA (3 minutes). Assume it takes pod 4 minutes to boot up and we're currently running 2 pods. Metric value is 20 and target is 10. In this case: scales-up to 4 pods (20/10 utilization 2 pods). are starting up, they're not doing anything yet. 3 minutes later HPA scales-up to 8 pods (20/10 utilization 4 pods). If we only counted ready pods the scale-up in 3 would not occur, because the desired replicas would still be 4 (20/10 * 2). My intuition would be that this is the expected behavior of HPA. Also if there is no space in the cluster we will just keep scaling-up until we hit max replicas. The only case where current behavior would be more correct than filtering out unready pods in this formula is if someone uses a metric that is calculated based on total number of replicas. Do we advertise that as the right way to use HPA? If we add to Object metrics in 1.11 this usage pattern will stop making sense. To sum up - I think we will need to change the formula to only count ready pods at some point. I can see how current behavior could be a problem in any use-case except the one described in paragraph above. I'm inclined to agree with and provide some way to change the behavior in 1.10 (a flag or an annotation?), treating it as a bug fix.


Question: func TestEmptyCPURequest(t *testing.T) { tc := testCase{ <del> minReplicas:     1, maxReplicas:     5, initialReplicas: 1, desiredReplicas: 1, CPUTarget:       100, reportedLevels:  []uint64{200}, useMetricsAPI:   true, </del> <ins> minReplicas:         1, maxReplicas:         5, initialReplicas:     1, desiredReplicas:     1, CPUTarget:           100, reportedLevels:      []uint64{200}, reportedCPURequests: []resource.Quantity{}, useMetricsAPI:       true, </ins> expectedConditions: []autoscalingv1.HorizontalPodAutoscalerCondition{ {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "SucceededGetScale"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionFalse, Reason: "FailedGetResourceMetric"},

WDYT
Ah, yeah, I missed that it was object metrics before. I'm not entirely convinced we need a flag here -- it might just make sense to bring Object metrics in line with all the other metrics types. On the other hand, there's a subtlety here -- the other metric types don't quite ignore unready pods -- they just ignore the metrics from unready pods, which causes a heavy bias in the direction of not scaling, but an overwhelming spike can still cause scaling. That behavior is not as achievable using object metrics. I'm inclined to just ignore that subtlety, though, and say that object metrics should ignore unready pods entirely.
I think this is what we should do long term. I was worried about breaking someone who relies on targetValue + averaged metric to get effect similar to targetAverageValue and though about providing something for transitional period, but if you're ok with it I'd rather just change it as you say.
I think it's safe to change as-is -- that feature's still in beta, and it's not an API change, so it shouldn't just hard-break anyone. Plus, we've change stuff like this before (when we did the initial resource metrics change).
As said, we are more than ready to help with the fix/test if required. This since this is a major blocking for us in the implementation of the HPA in our side
is anyone actively working on implementing this? If not, I'm happy to. I think I have a fairly clear picture of the change we need to make and will likely have time early this week to get it done.
(Also, from the conversation above, to we want to tag this as a release blocker for 1.10)?
I probably won't be able to work on this before the end of this week. Happy to do a review if you jump in and implement it.
I don't think this qualifies as a release blocking bug (but I'm willing to have someone try and convince me otherwise ;-)). There doesn't seem to be any behavior here that's explicitly broken -- it's just sub-optimal in some use cases. I'd definitely like to address this in 1.11, though.
it's not release blocker, but to share our experience ;).


Question: func TestEmptyCPURequest(t *testing.T) { tc := testCase{ <del> minReplicas:     1, maxReplicas:     5, initialReplicas: 1, desiredReplicas: 1, CPUTarget:       100, reportedLevels:  []uint64{200}, useMetricsAPI:   true, </del> <ins> minReplicas:         1, maxReplicas:         5, initialReplicas:     1, desiredReplicas:     1, CPUTarget:           100, reportedLevels:      []uint64{200}, reportedCPURequests: []resource.Quantity{}, useMetricsAPI:       true, </ins> expectedConditions: []autoscalingv1.HorizontalPodAutoscalerCondition{ {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "SucceededGetScale"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionFalse, Reason: "FailedGetResourceMetric"},

without this feature we cannot use HPA as is and we need to figure out how to build/run fork (which can be challenging because everything inside controller-manager). also we are planning to upgrade to 1.10 quickly, but not sure for 1.11 (e.g. we are two versions behind now).
Sounds like a plan :) Will try and get this done tomorrow morning!
- I submitted a pr for this - would love your eyes on it so you can see if you think it meets your use case :)
/assign
GitHub didn't allow me to assign the following users: mattjmcnaughton. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I checked the code and it seems good to me. About the comment about scaling down if the readyPods<CurrentPods, we should be fine since our metric is already not including the pods that are not ready.
Is there any advance in this ticket? I'm wondering if this made it to k8s 1.10 or if we need to wait for k8s 1.11?
The PR is lgtm-ed and it will merge once the code freeze is lifted. Meaning it will go to 1.11. To get it into 1.10 you would need to get approval for milestone from (sig-autoscaling lead).


Question: // since volumes should be removed from this list as soon a detach operation // is considered, before the detach operation is triggered). GetVolumesToReportAttached() map[types.NodeName][]v1.AttachedVolume <ins> // GetNodesToUpdateStatusFor returns the map of nodeNames to nodeToUpdateStatusFor GetNodesToUpdateStatusFor() map[types.NodeName]nodeToUpdateStatusFor </ins> } // AttachedVolume represents a volume that is attached to a node.

Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: $ kubectl describe pods test-ipj93 Name: test-ipj93 Namespace: default Node: kubernetes-minion-group-yxra/10.128.0.3 Start Time: Mon, 28 Nov 2016 14:39:28 -0800 Labels: <noneStatus: Pending IP: Controllers: <noneContainers: busybox: Container ID: Image: busybox Image ID: Port: Command: sh -c while true; do echo (hostname)-(cat /proc/stat cut -f 4 -d" "); sleep 30; done QoS Tier: cpu: Burstable memory: BestEffort Requests: cpu: 100m State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test ReadOnly: false default-token-kjoqg: Type: Secret (a volume populated by a Secret) SecretName: default-token-kjoqg Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 57m 57m 1 {default-scheduler } Normal Scheduled Successfully assigned test-ipj93 to kubernetes-minion-group-yxra 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulling pulling image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulled Successfully pulled image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Created Created container with docker id ; Security:[seccomp=unconfined] 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Started Started container with docker id 47m 47m 6 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.


Question: // since volumes should be removed from this list as soon a detach operation // is considered, before the detach operation is triggered). GetVolumesToReportAttached() map[types.NodeName][]v1.AttachedVolume <ins> // GetNodesToUpdateStatusFor returns the map of nodeNames to nodeToUpdateStatusFor GetNodesToUpdateStatusFor() map[types.NodeName]nodeToUpdateStatusFor </ins> } // AttachedVolume represents a volume that is attached to a node.

SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig.


Question: // since volumes should be removed from this list as soon a detach operation // is considered, before the detach operation is triggered). GetVolumesToReportAttached() map[types.NodeName][]v1.AttachedVolume <ins> // GetNodesToUpdateStatusFor returns the map of nodeNames to nodeToUpdateStatusFor GetNodesToUpdateStatusFor() map[types.NodeName]nodeToUpdateStatusFor </ins> } // AttachedVolume represents a volume that is attached to a node.

This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes


Question: livenessProbe := &v1.Probe{ ProbeHandler: v1.ProbeHandler{ GRPC: &v1.GRPCAction{ <del> Port:    2379, </del> <ins> Port:    5000, </ins> Service: nil, }, },

Tests in are using image for gRPC probe tests: should not be restarted with a GRPC liveness probe should be restarted with a GRPC liveness probe We should use image for these tests as it is much smaller size. /sig node /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue KEP:
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Tests in are using image for gRPC probe tests: /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : Tests in are using image for gRPC probe tests: /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi can I work on this issue? I'm new to Kubernetes and want to contribute!
Should I change to in this file?
Yes.


Question: livenessProbe := &v1.Probe{ ProbeHandler: v1.ProbeHandler{ GRPC: &v1.GRPCAction{ <del> Port:    2379, </del> <ins> Port:    5000, </ins> Service: nil, }, },

Also pass proper parameters to agnhost to enable gRPC endpoint, change ports, and run tests to ensure that this worked.
Thanks for your reply! Will let you know if I have more questions as I work
/assign
/assign


Question: e2eskipper.Skipf("this test is intended to be run on a system with at least %d cores per socket", minCoreCount) } <del> return numaNodes, coreCount </del> <ins> threadsPerCore := detectThreadPerCore() return numaNodes, coreCount, threadsPerCore </ins> } func checkMetricValueGreaterThan(value interface{}) types.GomegaMatcher {

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node started to fail since 09-13.
/cc
/assign
/cc
/triage accepted /priority important-soon it's indeed failing too often
/reopen The test is still failing: Now timing out at 120s i.e. 2 minutes as we increased the default timeout . We need to understand the root cause and get to the bottom of this.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Snippet from the build log () corresponding from this failure We see from the above logs that: node has two numa nodes, 8 cpus per NUMA so the root cause of the failure is not because the tests are running on a node that doesn't have multi-numa support. is started and the health check fails a couple of times but eventually succeeds. pod is created with 9 cpus (cpusperNUMA +1 i.e. 8+1) which is expected to fail with Topology Affinity Error as we cannot allocate resources for such a pod from a single NUMAnode. We fail to get matching metrics. It is not clear if the check fails because we don't get kubelet metrics at all or the expected metrics but based on point 6 below it seems like it is because of latter. We can add additional logs for more clarity here. each is executed and Kubelet is restarted and health check is done. are collected from the namespace corresponding to the test pod which shows that the pod started even though it was expected to fail with Topology Affinity Error. We need to understand why the pod got successfully created even though it requested CPUs that couldn't be allocated from a single NUMAnode.


Question: return ctx.Err() } <del> if cmderr != nil { if mounter.withSafeNotMountedBehavior && strings.Contains(string(out), errNotMounted) { klog.V(4).Infof("ignoring 'not mounted' error for %s", path) return nil } return fmt.Errorf("unmount failed: %vnUnmounting arguments: %snOutput: %s", cmderr, path, string(out)) </del> <ins> if err != nil { return checkUmountError(target, command, output, err, withSafeNotMountedBehavior) </ins> } return nil } <del> func forceUmount(path string) error { cmd := exec.Command("umount", "-f", path) out, cmderr := cmd.CombinedOutput() </del> <ins> func forceUmount(target string, withSafeNotMountedBehavior bool) error { command := exec.Command("umount", "-f", target) output, err := command.CombinedOutput() </ins> <del> if cmderr != nil { return fmt.Errorf("unmount failed: %vnUnmounting arguments: %snOutput: %s", cmderr, path, string(out)) </del> <ins> if err != nil { return checkUmountError(target, command, output, err, withSafeNotMountedBehavior) </ins> } return nil } <ins> // checkUmountError checks a result of umount command and determine a return value. func checkUmountError(target string, command *exec.Cmd, output []byte, err error, withSafeNotMountedBehavior bool) error { if err.Error() == errNoChildProcesses { if command.ProcessState.Success() { // We don't consider errNoChildProcesses an error if the process itself succeeded (see - k/k issue #103753). return nil } // Rewrite err with the actual exit error of the process. err = &exec.ExitError{ProcessState: command.ProcessState} } if withSafeNotMountedBehavior && strings.Contains(string(output), errNotMounted) { klog.V(4).Infof("ignoring 'not mounted' error for %s", target) return nil } return fmt.Errorf("unmount failed: %vnUnmounting arguments: %snOutput: %s", err, target, string(output)) } </ins>

A status of Pod mounted NFS PV remain as terminating when a node shutdown and restart. A Pod is completely terminated. Deploy a Pod mounted NFS PV Shutdown the node where the Pod is running Wait for 5 min and the Pod status become Restart the node The Pod status is still and the Pod is never terminated completely <detailsPrepare a NFS Server.(I used ) Deploy a Pod mounted NFS PV like this. Shutdown the node where the Pod is running.(In this case ) And the status of node02 becomes . After 5 min later(by default), the Pod status becomes . (The time a Pod becomes status depends on .) Then, restart the node and the status of node02 becomes . But the Pod status is still and the Pod is never terminated completely. It seems that a faild to delete Pod because of trying to unmount a not mounted directory. Force delete Pod. Delete Pod's directory on the node. </detailsNo response <details</details<detailsnone(on premise) </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig node
It dose not happen using , so it seems to depend on NFS volume plugin.
/assign
I try to build and run kubelet with comment out this logic. Then it seems to get well. related: This change checks the result of instead of the mount point check. However, check logic is not implemented in and (these functions are used in NFS volume plugin), despite it is done in .
/remove-sig node /sig storage
Reopening, fix had to be reverted due to unit tests requiring root permissions ()
I am terribly sorry... Please wait a moment. I'll try to fix it.
no worries, just get to review the rework, making sure the units tests don't require root permissions
Thank you! Fixed in .
Are there any plans to backport this to older Kubernetes versions? This seems like a critical fix.


Question: DockerEndpoint:            dockerEndpoint, DockershimRootDirectory:   "/var/lib/dockershim", DockerExecHandlerName:     "native", <ins> DockerDisableSharedPID:    true, </ins> PodSandboxImage:           defaultPodSandboxImage, ImagePullProgressDeadline: metav1.Duration{Duration: 1 * time.Minute}, RktAPIEndpoint:            defaultRktAPIServiceEndpoint,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --BUG REPORT: What happened: I'm running a simple Kubernetes cluster with master and one node. All basic PODs are deploying successfully (ie. weave-net) but when I try to deploy GlusterFS as a POD - container creation fails with error: This is a result of failing to perform from docker file. The weird part is that if I run this image directly on node through docker it runs smoothly. What you expected to happen: I'd expect to be able to deploy GlusterFS through Kubernetess properly. How to reproduce it (as minimally and precisely as possible): Setup minimal cluster and try to deploy the follwing POD Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: virtualbox OS (e.g. from /etc/os-release):Kernel (e.g. ):Docker versionInstall tools: vagrant Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
One update - I can deploy this POD with no problems on version of k8s.
I am hitting the same bug on a freshly installed k8s cluster running on installed via kubeadm plus weave-net with a currently unsupported docker version. For me, downgrading to a supported docker version helped to resolve this issue (v1.12):
- Thanks for the info! - Well we can confirm that kubernetes is not working properly with Docker 1.29 and 1.3
Reiterating the mentions to trigger a notification: <detailsIn response to : - Thanks for the info!


Question: DockerEndpoint:            dockerEndpoint, DockershimRootDirectory:   "/var/lib/dockershim", DockerExecHandlerName:     "native", <ins> DockerDisableSharedPID:    true, </ins> PodSandboxImage:           defaultPodSandboxImage, ImagePullProgressDeadline: metav1.Duration{Duration: 1 * time.Minute}, RktAPIEndpoint:            defaultRktAPIServiceEndpoint,

- Well we can confirm that kubernetes is not working properly with Docker 1.29 and 1.3 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is not really a bug since kubernetes doesn't support the docker version you use yet, but feel free to send a patch if you find the problem.
I was thinking about your comment and I think it is still a bug since downgrading Kubernetes to 1.6 fixes the problem while having the same setup as described above. So something have changed in 1.7. But yeah. Will try to investigate on my own.
I'm actually getting this same error with 1.7.1 and docker 1.13.1. Anyone have other thoughts on what the issue might be here? I found this so question () but it looks like everything is configured correctly.
In kubernetes 1.7, under pid # 1, there is a "/ pause" process, and since init takes pid 1, it is no longer available. In earlier versions of kubernetes, pid # 1 was free . PID USER TIME COMMAND 1 root 0:00 /pause 7 root 3:41 /tiller Kubernetes now shares a single PID namespace among all containers in a pod when running with docker = 1.13.1. This means processes can now signal processes in other containers in a pod, but it also means that the pattern will cause the pod to be restarted rather than a single container.
I can work around the problem using the option in the kubelet, however this not the desired solution of course.
just FYI, looks like some images have to be run as PID 1.
In 1.7 release, we enabled shared pid namespace if the docker version is docker 1.13 and beyond for debug pod feature. Do anyone know why those images, such as GlusterFS have to run as PID 1? we need to figure out how to resolve this conflict with shared pid namespace. If we cannot, we have to disable that setting for 1.8 since we scheduled to support docker 1.13 in 1.8 timeframe. I am currently assigning you this bug to come up a solution.


Question: DockerEndpoint:            dockerEndpoint, DockershimRootDirectory:   "/var/lib/dockershim", DockerExecHandlerName:     "native", <ins> DockerDisableSharedPID:    true, </ins> PodSandboxImage:           defaultPodSandboxImage, ImagePullProgressDeadline: metav1.Duration{Duration: 1 * time.Minute}, RktAPIEndpoint:            defaultRktAPIServiceEndpoint,

cc/
gluster/gluster-centos is a container image that runs systemd so that it can run ntpd, crond, gssproxy, glusterd & sshd inside a single container. This isn't how Kubernetes is intended to be used and I don't know how much we should go out of our way to support it (this is a question for & seems to be a good write up of the challenges of running systemd inside of docker. It also prevents Kubernetes from doing process management like: reporting state of processes signaling individual processes for graceful restart ingesting and display daemon output (to stdout/err) Ideally we could change this many-processes-in-a-container to be many-containers-in-a-pod, something like this: This doesn't need a privileged container and doesn't run sshd or ntpd, neither of which are needed in Kubernetes. This also doesn't run crond, but that may actually be needed by gluster. I've never used gluster. (I used flags so I didn't have to investigate how to make these processes log to stdout) If you must run systemd in a container, adding this to the container config will get systemd to run: This tells systemd to run in system mode even though it's not PID 1 and disables its chroot detection (which is unrelated to PID 1 but checked when invoked as "systemd", I guess).
afaict, running in a container is well recommended ( and ) at certain scenarios and a common practice. iic, there are enough containers who make use of as its . If shared namespace which introduced in 1.7 breaks this, I believe its a backward compatibility break :(. I would invite here to share his thoughts as well. :)
Those articles are about how to work around shortcomings in vanilla docker that Kubernetes solves natively: resource sharing via pods, containing via SELinux, reaping orphaned zombies, etc. My earlier question was along the lines of how much effort should go into supporting use cases that are not using Kubernetes as intended. (I'm actually asking, not trying to make a point.) Does Kubernetes commit to continuing support for every container image that's ever run on Kubernetes? For how long? Indefinitely? or can we change it over the course of multiple releases? Anyway, this change is compatible with running an init system in a container.


Question: DockerEndpoint:            dockerEndpoint, DockershimRootDirectory:   "/var/lib/dockershim", DockerExecHandlerName:     "native", <ins> DockerDisableSharedPID:    true, </ins> PodSandboxImage:           defaultPodSandboxImage, ImagePullProgressDeadline: metav1.Duration{Duration: 1 * time.Minute}, RktAPIEndpoint:            defaultRktAPIServiceEndpoint,

The problem is with images that assume they will only ever have , which has so far been the case with docker due to a technical limitation, but it was never the case with other runtimes such as rkt, hyperv, etc. Brainstorming other options: shared pid namespace and require users of systemd to set an environment variable in their Kubernetes config (or port the multi-process container to a multi-container pod). shared pid namespace and enable it in a future release where we have the same problem. a solution where pid namespace is configurable per-pod, but not supported indefinitely. It would probably be implemented using annotations, and we will have to modify the Container Runtime Interface to support it. Over the course of multiple releases the default behavior would change but could be reverted based on the presence of an annotation. In a future release, users will have to change an annotation in their config to get the old behavior (rather than the environment variable). In a future future release, the annotation will stop working and users will have to set the environment variable. to supporting per-pod shared vs isolated pid namespaces indefinitely and adding it to the Kubernetes API. It doesn't make sense in some runtimes, so we probably can't require all runtimes to implement it, so it'll likely only see use with Docker. Last time we discussed it in SIG Node, and didn't want to support isolated pid namespaces indefinitely.
Yes running systemd in a container is a critical need for us. There are lots of situations where it is and will be used. Biggest use case if for people simply moving current workloads that run in a VM into a container. I have a service that runs fun under systemd in a VM. If I can simply take the init script and move it whole sale into a container, why wouldn't I want to do this. It's simple and does require a great deal of knowledge by the end user. service containers. Not every project can easily be moved into microservices. But they probably can run inside of containers. If we drop support for systemd running inside of containers, then we loose a big group of users. Getting access to journald information inside of the container. Applications that write to syslog and directly to the journal. Currently in container world without systemd, this information is dropped on the floor.


Question: DockerEndpoint:            dockerEndpoint, DockershimRootDirectory:   "/var/lib/dockershim", DockerExecHandlerName:     "native", <ins> DockerDisableSharedPID:    true, </ins> PodSandboxImage:           defaultPodSandboxImage, ImagePullProgressDeadline: metav1.Duration{Duration: 1 * time.Minute}, RktAPIEndpoint:            defaultRktAPIServiceEndpoint,

if I run systemd and journald, I can actually retrieve and use this information.
If the intension is to force all containers inside of a POD to run with the same PID Namespace? I think this is a mistake since it eliminates another really cool use case. I would love to be able to run two containers inside of a pod, and INIT container with escallated privileges and a locked down container, with no privileges. I want to make sure the locked down container can not see the processes of the privileged container. An example of this, might be a container that loads a kernel module that is needed by the locked down container. Another use case would be a buildah container. The INIT container mounts up a COW file system onto a directory on a location that the locked down container can write, The locked down container writes out the data for his image, when the container completes, the INIT container commits the data to the repo. I believe allowing different containers inside of a POD to run with different security and isolated PIDs should not be removed.
none of the things you've mentioned are removed by this change, save using docker to hide other processes in the pod, and I'm not sure why that's a requirement. Is anyone currently using the "privileged sidecar" pattern you describe or is it hypothetical? It just occurred to me that an interesting compromise here might be to only enable shared pid for multi-container pods. This would enable pod semantics for PID when it would make a difference while drop-in docker monoliths keep the PID 1 behavior they're expecting in docker. I'd have to double check, but I'm pretty sure that behavior could be implemented entirely in the dockershim with no API or CRI changes.
Hypothetical, but I am pushing it for a solution for problems I hear from users fairly frequently, where then need big privileges at start and stop of the pod, but not for the time is is available to the network.
ok, thanks. ftr, I think we should continue to support this pattern as well, and I don't think a shared PID namespace prohibits it. I will bring up this issue at Tuesday's sig node meetup if you'd like to join us. Does the config I mentioned in fix the issue with systemd?


Question: DockerEndpoint:            dockerEndpoint, DockershimRootDirectory:   "/var/lib/dockershim", DockerExecHandlerName:     "native", <ins> DockerDisableSharedPID:    true, </ins> PodSandboxImage:           defaultPodSandboxImage, ImagePullProgressDeadline: metav1.Duration{Duration: 1 * time.Minute}, RktAPIEndpoint:            defaultRktAPIServiceEndpoint,

Copying from a comment made elsewhere:
Update from SIG Node Meetup: Everyone wants to support these use cases, but we're not yet sure the requirements we want to support medium to long term. is going to sync up with and then we'll figure out what we want to support in a future release. For 1.8 the current plan is to leave docker shared pid namespace enabled by default. Cluster administrators can disable it per-node with the --docker-disable-shared-pid flag. I'll see about whether we should add something back to the release notes.
This seems somewhat backwards. Wouldn't it make more sense to go back to supporting systemd use cases by default NOW (by making the shared PID namespace behavior optional or something) until a final decision is reached and can be advertised to current users?
absolutely ! thank you very much
I think that is the best solution is to make the PID Namespace sharing optional.
- optional at what scope - cluster, node, pod. let's sync up before next sig-node and we can revisit if need be before 1.8 goes out. The main question right now is if pid namespace sharing is opt-in or opt-out. We need to weigh pro/cons of both.
I would expect it at the POD, since pods would have different requirements, not sure why you would want to have this at the cluster or node level.
yes it works :)
awesome, thanks! I'd like to better understand the requirements of people objecting to this change ( I think there's a bit of confusion about what has changed, so let's start with some facts: This container image runs on Kubernetes without changes to the image, but it requires (non-obvious) pod configuration. is nothing inherently incompatible between a shared pid namespace and a process manager like systemd. systemd and other init systems work with pid != 1, but may require special invocation. in this issue is specific to Kubernetes. That is, this container image wouldn't work with vanilla docker PID namespace sharing, either. change does not affect securityContext I can think of a few things that are objectionable about the change, but let's state them as requirements. For example, are these requirements for anyone? Kubernetes must run all containers published on docker hub with no additional pod configuration Kubernetes must run all systemd containers, regardless of how they are built, with zero additional pod configuration.


Question: DockerEndpoint:            dockerEndpoint, DockershimRootDirectory:   "/var/lib/dockershim", DockerExecHandlerName:     "native", <ins> DockerDisableSharedPID:    true, </ins> PodSandboxImage:           defaultPodSandboxImage, ImagePullProgressDeadline: metav1.Duration{Duration: 1 * time.Minute}, RktAPIEndpoint:            defaultRktAPIServiceEndpoint,

Kubernetes must run this specific container image with zero additional configuration Kubernetes must officially support isolated PID namespaces between containers in a single pod across all runtimes (docker, rkt, hyperv, etc). Configuration changes are ok, but the flag to systemd cannot be used because the documentation says its not useful. I want to use systemd as intended. Rather than having Kubernetes work around this container image, should we patch the Dockerfile to make it compatible with even the native docker shared-pid, unrelated to Kubernetes?
Configuration changes are ok, but the --system flag to systemd cannot be used because the documentation says its not useful. I want to use systemd as intended. We can ask the systemd guys why they recommend never using the --system flag, but I would guess one of the reasons is that processes showing up that were not created in the ancestry of systemd, could cause confusion. I would also guess that running more then one container with systemd --system would not be supported. Kubernetes must officially support isolated PID namespaces between containers in a single pod. This is required so that you can run different containers with different security constraints in the same pod at the same time. Your last comment makes the seem like it is the default, when actually it is the exception. almost no containers run with shared-pid normally. I guess in a side car container, it makes sense to support this scenario, but forcing this as the ONLY way to run containers is limiting.
This is the part I don't understand. Why? What does PID visibility have to do with security constraints? What am I missing? nor do they run with a shared network namespace, but we impose that on docker containers because that's the pod model.
An update from the last SIG Node: we'll disable this by default in 1.8 and revisit for 1.9 I'll use to track that.
if I have a privileged container process in the same pid namespace as a pid in a non priv container, then the container process in the non priv can attack the priv process. Having them in separate pid namespaces gives you better security. Bottom line, you have a potential change where containers that expect to be run as PID 1, whether legitimately or not, could break when you make this change. Kubernetes in my opinion should make it easy to opt out of this behavior.


Question: DockerEndpoint:            dockerEndpoint, DockershimRootDirectory:   "/var/lib/dockershim", DockerExecHandlerName:     "native", <ins> DockerDisableSharedPID:    true, </ins> PodSandboxImage:           defaultPodSandboxImage, ImagePullProgressDeadline: metav1.Duration{Duration: 1 * time.Minute}, RktAPIEndpoint:            defaultRktAPIServiceEndpoint,

/priority critical-urgent
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details
Update: is pending review and should be to the v1.8 Milestone to resolve this issue.


Question: <ins> // +build linux /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package mount import ( "k8s.io/utils/exec" fakeexec "k8s.io/utils/exec/testing" "testing" ) func TestGetFileSystemSize(t *testing.T) { cmdOutputSuccessXfs := ` statfs.f_bsize = 4096 statfs.f_blocks = 1832448 statfs.f_bavail = 1822366 statfs.f_files = 3670016 statfs.f_ffree = 3670012 statfs.f_flags = 0x1020 geom.bsize = 4096 geom.agcount = 4 geom.agblocks = 458752 geom.datablocks = 1835008 geom.rtblocks = 0 geom.rtextents = 0 geom.rtextsize = 1 geom.sunit = 0 geom.swidth = 0 counts.freedata = 1822372 counts.freertx = 0 counts.freeino = 61 counts.allocino = 64 ` cmdOutputNoDataXfs := ` statfs.f_bsize = 4096 statfs.f_blocks = 1832448 statfs.f_bavail = 1822366 statfs.f_files = 3670016 statfs.f_ffree = 3670012 statfs.f_flags = 0x1020 geom.agcount = 4 geom.agblocks = 458752 geom.rtblocks = 0 geom.rtextents = 0 geom.rtextsize = 1 geom.sunit = 0 geom.swidth = 0 counts.freedata = 1822372 counts.freertx = 0 counts.freeino = 61 counts.allocino = 64 ` cmdOutputSuccessExt4 := ` Filesystem volume name:   cloudimg-rootfs Last mounted on:          / Filesystem UUID:          testUUID Filesystem magic number:  0xEF53 Filesystem revision #:    1 (dynamic) Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit Default mount options:    user_xattr acl Filesystem state:         clean Errors behavior:          Continue Filesystem OS type:       Linux Inode count:              3840000 Block count:              5242880 Reserved block count:     0 Free blocks:              5514413 Free inodes:              3677492 First block:              0 Block size:               4096 Fragment size:            4096 Group descriptor size:    64 Reserved GDT blocks:      252 Blocks per group:         32768 Fragments per group:      32768 Inodes per group:         16000 Inode blocks per group:   1000 Flex block group size:    16 Mount count:              2 Maximum mount count:      -1 Check interval:           0 (<none>) Lifetime writes:          180 GB Reserved blocks uid:      0 (user root) Reserved blocks gid:      0 (group root) First inode:              11 Inode size:	              256 Required extra isize:     32 Desired extra isize:      32 Journal inode:            8 Default directory hash:   half_md4 Directory Hash Seed:      Test Hashing Journal backup:           inode blocks Checksum type:            crc32c Checksum:                 0x57705f62 Journal features:         journal_incompat_revoke journal_64bit journal_checksum_v3 Journal size:             64M Journal length:           16384 Journal sequence:         0x00037109 Journal start:            1 Journal checksum type:    crc32c Journal checksum:         0xb7df3c6e ` cmdOutputNoDataExt4 := `Filesystem volume name:   cloudimg-rootfs Last mounted on:          / Filesystem UUID:          testUUID Filesystem magic number:  0xEF53 Filesystem revision #:    1 (dynamic) Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit Default mount options:    user_xattr acl Filesystem state:         clean Errors behavior:          Continue Filesystem OS type:       Linux Inode count:              3840000 Reserved block count:     0 Free blocks:              5514413 Free inodes:              3677492 First block:              0 Fragment size:            4096 Group descriptor size:    64 Reserved GDT blocks:      252 Blocks per group:         32768 Fragments per group:      32768 Inodes per group:         16000 Inode blocks per group:   1000 Flex block group size:    16 Mount count:              2 Maximum mount count:      -1 Check interval:           0 (<none>) Lifetime writes:          180 GB Reserved blocks uid:      0 (user root) Reserved blocks gid:      0 (group root) First inode:              11 Inode size:	              256 Required extra isize:     32 Desired extra isize:      32 Journal inode:            8 Default directory hash:   half_md4 Directory Hash Seed:      Test Hashing Journal backup:           inode blocks Checksum type:            crc32c Checksum:                 0x57705f62 Journal features:         journal_incompat_revoke journal_64bit journal_checksum_v3 Journal size:             64M Journal length:           16384 Journal sequence:         0x00037109 Journal start:            1 Journal checksum type:    crc32c Journal checksum:         0xb7df3c6e ` testcases := []struct { name        string devicePath  string blocksize   uint64 blockCount  uint64 cmdOutput   string expectError bool fsType      string }{ { name:        "success parse xfs info", devicePath:  "/dev/test1", blocksize:   4096, blockCount:  1835008, cmdOutput:   cmdOutputSuccessXfs, expectError: false, fsType:      "xfs", }, { name:        "block size not present - xfs", devicePath:  "/dev/test1", blocksize:   0, blockCount:  0, cmdOutput:   cmdOutputNoDataXfs, expectError: true, fsType:      "xfs", }, { name:        "success parse ext info", devicePath:  "/dev/test1", blocksize:   4096, blockCount:  5242880, cmdOutput:   cmdOutputSuccessExt4, expectError: false, fsType:      "ext4", }, { name:        "block size not present - ext4", devicePath:  "/dev/test1", blocksize:   0, blockCount:  0, cmdOutput:   cmdOutputNoDataExt4, expectError: true, fsType:      "ext4", }, } for _, test := range testcases { t.Run(test.name, func(t *testing.T) { fcmd := fakeexec.FakeCmd{ CombinedOutputScript: []fakeexec.FakeAction{ func() ([]byte, []byte, error) { return []byte(test.cmdOutput), nil, nil }, }, } fexec := fakeexec.FakeExec{ CommandScript: []fakeexec.FakeCommandAction{ func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, }, } resizefs := ResizeFs{exec: &fexec} var blockSize uint64 var fsSize uint64 var err error switch test.fsType { case "xfs": blockSize, fsSize, err = resizefs.getXFSSize(test.devicePath) case "ext4": blockSize, fsSize, err = resizefs.getExtSize(test.devicePath) } if blockSize != test.blocksize { t.Fatalf("Parse wrong block size value, expect %d, but got %d", test.blocksize, blockSize) } if fsSize != test.blocksize*test.blockCount { t.Fatalf("Parse wrong fs size value, expect %d, but got %d", test.blocksize*test.blockCount, fsSize) } if !test.expectError && err != nil { t.Fatalf("Expect no error but got %v", err) } }) } } func TestNeedResize(t *testing.T) { testcases := []struct { name            string devicePath      string deviceMountPath string deviceSize      string cmdOutputFsType string expectError     bool expectResult    bool }{ { name:            "False - Unsupported fs type", devicePath:      "/dev/test1", deviceMountPath: "/mnt/test1", deviceSize:      "2048", cmdOutputFsType: "TYPE=ntfs", expectError:     true, expectResult:    false, }, } for _, test := range testcases { t.Run(test.name, func(t *testing.T) { fcmd := fakeexec.FakeCmd{ CombinedOutputScript: []fakeexec.FakeAction{ func() ([]byte, []byte, error) { return []byte(test.deviceSize), nil, nil }, func() ([]byte, []byte, error) { return []byte(test.cmdOutputFsType), nil, nil }, }, } fexec := fakeexec.FakeExec{ CommandScript: []fakeexec.FakeCommandAction{ func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, }, } resizefs := ResizeFs{exec: &fexec} needResize, err := resizefs.NeedResize(test.devicePath, test.deviceMountPath) if needResize != test.expectResult { t.Fatalf("Expect result is %v but got %v", test.expectResult, needResize) } if !test.expectError && err != nil { t.Fatalf("Expect no error but got %v", err) } }) } } </ins>

Kubernetes should resize filesystem on a volume when it's created as a clone of a smaller volume / smaller snapshot is restored into it. Example: takes a snapshot of 1 GiB volume with ext3 filesystem on it. restores the snapshot into 2 GiB volume. 2.1 CSI driver restores the snapshot into a bigger volume, but ext3 filesystem still uses only 1 GiB of space. driver should make sure that the filesystem on the volume has been resized before the volume is mounted into a pod. There are few steps needed: [x] Clarify behavior of CSI call, [ ] Update CSI drivers to the new behavior. [x] Move filesystem resize code from kubernetes/kuberenetes to kubernetes/mount-utils, so all CSI drivers can re-use the same code [x] Enhance the code above to compare the volume size from block device perspective and from filesystem perspective. See for proof of concept. [ ] File issues in all community CSI drivers that we care about & fix them. [ ] Update csi-tests to (optionally?) check for volume size when cloning / restoring snapshot to a bigger size.
/assign /sig storage /priority important-soon
True, considering the CSI spec allows restore to a bigger size volume, we should have the support from CO.
This is hard, Kubernets PV controller overwrites on binding. Therefore the external-provisioner cannot create PVC as "marked for resize on node", i.e. with = = the real size of the new volume) and with = size of the original volume. We either need to change the binding mechanism not to overwrite or add a flag (annotation?) that marks the volume for resize on node, even when .
Talked to The external-provisioner could mark for resize on node: Set to the original volume size. Add condition to the PVC. As result, the extenal-provisioner needs to modify , which needs new RBAC and is a breaking change. The PV controller could respect , if set. We discussed and rejected adding a new flag / annotation to PV by the external-provisioner for kubelet to call NodeExpandVolume on the first mount. Such a flag would need to be cleared by kubelet and kubelet can't modify PVs.
From latest discussion last week, I believe the conclusion was to have external-provisioner set to the smaller size.