Question: <ins> ## kubectl kubectl controls the Kubernetes cluster manager ### Commands #### version Print version of client and server Usage: ``` kubectl version [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. -c, --client=false: Client version only (no server required) --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use -h, --help=false: help for version --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": Path to the kubeconfig file to use for CLI requests. --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` #### proxy Run a proxy to the Kubernetes API server. Usage: ``` kubectl proxy [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use -h, --help=false: help for proxy --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": Path to the kubeconfig file to use for CLI requests. --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -p, --port=8001: The port on which to run the proxy -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging -w, --www="": Also serve static files from the given directory under the prefix /static ``` #### get Display one or many resources. Possible resources include pods (po), replication controllers (rc), services (se), minions (mi), or events (ev). If you specify a Go template, you can use any fields defined for the API version you are connecting to the server with. Examples: $ kubectl get pods <list all pods in ps output format> $ kubectl get replicationController 1234-56-7890-234234-456456 <list single replication controller in ps output format> $ kubectl get -o json pod 1234-56-7890-234234-456456 <list single pod in json output format> $ kubectl get rc,services <list replication controllers and services together in ps output format> Usage: ``` kubectl get [(-o|--output=)json|yaml|...] <resource> [<id>] [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use -h, --help=false: help for get --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": Path to the kubeconfig file to use for CLI requests. --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --no-headers=false: When using the default output, don't print headers --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -o, --output="": Output format: json|yaml|template|templatefile --output-version="": Output the formatted object with the given version (default api-version) -l, --selector="": Selector (label query) to filter on -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr -t, --template="": Template string or path to template file to use when -o=template or -o=templatefile. --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging -w, --watch=false: After listing/getting the requested object, watch for changes. --watch-only=false: Watch for changes to the requseted object(s), without listing/getting first. ``` #### describe Show details of a specific resource. This command joins many API calls together to form a detailed description of a given resource. Usage: ``` kubectl describe <resource> <id> [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": Path to the kubeconfig file to use for CLI requests. --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` #### create Create a resource by filename or stdin. JSON and YAML formats are accepted. Examples: $ kubectl create -f pod.json <create a pod using the data in pod.json> $ cat pod.json | kubectl create -f - <create a pod based on the json passed into stdin> Usage: ``` kubectl create -f filename [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use -f, --filename=[]: Filename, directory, or URL to file to use to create the resource -h, --help=false: help for create --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": Path to the kubeconfig file to use for CLI requests. --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` #### update Update a resource by filename or stdin. JSON and YAML formats are accepted. Examples: $ kubectl update -f pod.json <update a pod using the data in pod.json> $ cat pod.json | kubectl update -f - <update a pod based on the json passed into stdin> Usage: ``` kubectl update -f filename [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use -f, --filename="": Filename or URL to file to use to update the resource -h, --help=false: help for update --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": Path to the kubeconfig file to use for CLI requests. --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` #### delete Delete a resource by filename, stdin, resource and id or by resources and label selector. JSON and YAML formats are accepted. If both a filename and command line arguments are passed, the command line arguments are used and the filename is ignored. Note that the delete command does NOT do resource version checks, so if someone submits an update to a resource right when you submit a delete, their update will be lost along with the rest of the resource. Examples: $ kubectl delete -f pod.json <delete a pod using the type and id pod.json> $ cat pod.json | kubectl delete -f - <delete a pod based on the type and id in the json passed into stdin> $ kubectl delete pods,services -l name=myLabel <delete pods and services with label name=myLabel> $ kubectl delete pod 1234-56-7890-234234-456456 <delete a pod with ID 1234-56-7890-234234-456456> Usage: ``` kubectl delete ([-f filename] | (<resource> [(<id> | -l <label>)] [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use -f, --filename=[]: Filename, directory, or URL to a file containing the resource to delete -h, --help=false: help for delete --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": Path to the kubeconfig file to use for CLI requests. --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -l, --selector="": Selector (label query) to filter on -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` #### config config modifies .kubeconfig files using subcommands like "kubectl config set current-context my-context" Usage: ``` kubectl config <subcommand> [flags] kubectl config [command] Available Commands: view                                                                                displays the specified .kubeconfig file or a merged result set-cluster name [server] [insecure-skip-tls-verify] [certificate-authority] [api-version] Sets a cluster entry in .kubeconfig set-credentials name                                                                       Sets a user entry in .kubeconfig set-context name                                                                           Sets a context entry in .kubeconfig set property-name property-value                                                           Sets an individual value in a .kubeconfig file unset property-name                                                                        Unsets an individual value in a .kubeconfig file use-context context-name                                                                   Sets the current-context in a .kubeconfig file Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use --global=false: use the .kubeconfig from /home/username --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": use a particular .kubeconfig file --local=true: use the .kubeconfig in the currect directory --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging Additional help topics: kubectl version       Print version of client and server kubectl proxy         Run a proxy to the Kubernetes API server kubectl get           Display one or many resources kubectl describe      Show details of a specific resource kubectl create        Create a resource by filename or stdin kubectl update        Update a resource by filename or stdin kubectl delete        Delete a resource by filename, stdin or resource and id kubectl config        config modifies .kubeconfig files kubectl namespace     Set and view the current Kubernetes namespace kubectl log           Print the logs for a container in a pod. kubectl rollingupdate Perform a rolling update of the given ReplicationController kubectl run-container Run a particular image on the cluster. Use "kubectl help [command]" for more information about that command. ``` #### config view displays the specified .kubeconfig file or a merged result Usage: ``` kubectl config view [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use --global=false: use the .kubeconfig from /home/username -h, --help=false: help for view --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": use a particular .kubeconfig file --local=true: use the .kubeconfig in the currect directory --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version --merge=false: merge together the full hierarchy of .kubeconfig files -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` #### config set-cluster Sets a cluster entry in .kubeconfig Specifying a name that already exists overwrites that cluster entry. Usage: ``` kubectl config set-cluster name [server] [insecure-skip-tls-verify] [certificate-authority] [api-version] [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": api-version for the cluster entry in .kubeconfig -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": certificate-authority for the cluster entry in .kubeconfig --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use --global=false: use the .kubeconfig from /home/username -h, --help=false: help for set-cluster --insecure-skip-tls-verify=false: insecure-skip-tls-verify for the cluster entry in .kubeconfig --kubeconfig="": use a particular .kubeconfig file --local=true: use the .kubeconfig in the currect directory --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. --server="": server for the cluster entry in .kubeconfig --stderrthreshold=2: logs at or above this threshold go to stderr --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` #### config set-credentials Sets a user entry in .kubeconfig Specifying a name that already exists overwrites that user entry. Usage: ``` kubectl config set-credentials name [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server --auth-path="": auth-path for the user entry in .kubeconfig --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": client-certificate for the user entry in .kubeconfig --client-key="": client-key for the user entry in .kubeconfig --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use --global=false: use the .kubeconfig from /home/username -h, --help=false: help for set-credentials --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": use a particular .kubeconfig file --local=true: use the .kubeconfig in the currect directory --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --token="": token for the user entry in .kubeconfig --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` #### config set-context Sets a context entry in .kubeconfig Specifying a name that already exists overwrites that context entry. Usage: ``` kubectl config set-context name [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": cluster for the context entry in .kubeconfig --context="": The name of the kubeconfig context to use --global=false: use the .kubeconfig from /home/username -h, --help=false: help for set-context --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": use a particular .kubeconfig file --local=true: use the .kubeconfig in the currect directory --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version --namespace="": namespace for the context entry in .kubeconfig --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --token="": Bearer token for authentication to the API server. --user="": user for the context entry in .kubeconfig --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` #### config set Sets an individual value in a .kubeconfig file property-name is a dot delimitted name where each token represents either a attribute name or a map key.  Map keys may not contain dots. property-value is the new value you wish to set. Usage: ``` kubectl config set property-name property-value [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use --global=false: use the .kubeconfig from /home/username -h, --help=false: help for config --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": use a particular .kubeconfig file --local=true: use the .kubeconfig in the currect directory --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` #### config unset Unsets an individual value in a .kubeconfig file property-name is a dot delimitted name where each token represents either a attribute name or a map key.  Map keys may not contain dots. Usage: ``` kubectl config unset property-name [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use --global=false: use the .kubeconfig from /home/username -h, --help=false: help for config --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": use a particular .kubeconfig file --local=true: use the .kubeconfig in the currect directory --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` #### config use-context Sets the current-context in a .kubeconfig file Usage: ``` kubectl config use-context context-name [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use --global=false: use the .kubeconfig from /home/username -h, --help=false: help for config --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": use a particular .kubeconfig file --local=true: use the .kubeconfig in the currect directory --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` #### namespace Set and view the current Kubernetes namespace scope for command line requests. A Kubernetes namespace subdivides the cluster into groups of logically related pods, services, and replication controllers. Examples: $ kubectl namespace Using namespace default $ kubectl namespace other Set current namespace to other Usage: ``` kubectl namespace [<namespace>] [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": Path to the kubeconfig file to use for CLI requests. --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` #### log Print the logs for a container in a pod. If the pod has only one container, the container name is optional Examples: $ kubectl log 123456-7890 ruby-container <returns snapshot of ruby-container logs from pod 123456-7890> $ kubectl log -f 123456-7890 ruby-container <starts streaming of ruby-container logs from pod 123456-7890> Usage: ``` kubectl log [-f] <pod> [<container>] [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use -f, --follow=false: Specify if the logs should be streamed. -h, --help=false: help for log --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": Path to the kubeconfig file to use for CLI requests. --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` #### rollingupdate Perform a rolling update of the given ReplicationController. Replaces named controller with new controller, updating one pod at a time to use the new PodTemplate. The new-controller.json must specify the same namespace as the existing controller and overwrite at least one (common) label in its replicaSelector. Examples: $ kubectl rollingupdate frontend-v1 -f frontend-v2.json <update pods of frontend-v1 using new controller data in frontend-v2.json> $ cat frontend-v2.json | kubectl rollingupdate frontend-v1 -f - <update pods of frontend-v1 using json data passed into stdin> Usage: ``` kubectl rollingupdate <old-controller-name> -f <new-controller.json> [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use -f, --filename="": Filename or URL to file to use to create the new controller -h, --help=false: help for rollingupdate --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": Path to the kubeconfig file to use for CLI requests. --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. --poll-interval="3s": Time delay between polling controller status after update. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr --timeout="5m0s": Max time to wait for a controller to update before giving up. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". --token="": Bearer token for authentication to the API server. --update-period="1m0s": Time to wait between updating pods. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` #### run-container Create and run a particular image, possibly replicated. Creates a replication controller to manage the created container(s) Examples: $ kubectl run-container nginx --image=dockerfile/nginx <starts a single instance of nginx> $ kubectl run-container nginx --image=dockerfile/nginx --replicas=5 <starts a replicated instance of nginx> $ kubectl run-container nginx --image=dockerfile/nginx --dry-run <just print the corresponding API objects, don't actually send them to the apiserver> Usage: ``` kubectl run-container <name> --image=<image> [--replicas=replicas] [--dry-run=<bool>] [flags] Available Flags: --alsologtostderr=false: log to standard error as well as files --api-version="": The API version to use when talking to the server -a, --auth-path="": Path to the auth info file. If missing, prompt the user. Only used if using https. --certificate-authority="": Path to a cert. file for the certificate authority. --client-certificate="": Path to a client key file for TLS. --client-key="": Path to a client key file for TLS. --cluster="": The name of the kubeconfig cluster to use --context="": The name of the kubeconfig context to use --dry-run=false: If true, only print the object that would be sent, don't actually do anything --generator="run-container/v1": The name of the api generator that you want to use.  Default 'run-container-controller-v1' -h, --help=false: help for run-container --image="": The image for the container you wish to run. --insecure-skip-tls-verify=false: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure. --kubeconfig="": Path to the kubeconfig file to use for CLI requests. -l, --labels="": Labels to apply to the pod(s) created by this call to run. --log_backtrace_at=:0: when logging hits line file:N, emit a stack trace --log_dir=: If non-empty, write log files in this directory --log_flush_frequency=5s: Maximum number of seconds between log flushes --logtostderr=true: log to standard error instead of files --match-server-version=false: Require server version to match client version -n, --namespace="": If present, the namespace scope for this CLI request. --no-headers=false: When using the default output, don't print headers --ns-path="/home/username/.kubernetes_ns": Path to the namespace info file that holds the namespace context to use for CLI requests. -o, --output="": Output format: json|yaml|template|templatefile --output-version="": Output the formatted object with the given version (default api-version) -r, --replicas=1: Number of replicas to create for this container. Default 1 -s, --server="": The address of the Kubernetes API server --stderrthreshold=2: logs at or above this threshold go to stderr -t, --template="": Template string or path to template file to use when -o=template or -o=templatefile. --token="": Bearer token for authentication to the API server. --user="": The name of the kubeconfig user to use --v=0: log level for V logs --validate=false: If true, use a schema to validate the input before sending it --vmodule=: comma-separated list of pattern=N settings for file-filtered logging ``` </ins>

Manually updated docs have a tendency to drift from actual implementation. It shouldn't be too hard to add a script to generate the markdown from a . We could then run the script manually or as a commit hook when changes are made to pkg/kubectl.
A big chunk of that is cut/paste from the help output anyway.. Brendan On Jan 9, 2015 4:20 PM, "Jeff Lowdermilk" wrote:
Why are we including the help output? Seems like cobra should do that, or we should just tell the user to ask for help if they want it.
It's useful to see generated markdown of help output for the full command tree, e.g. if you want to link to the syntax on irc or something. Anyway, this was quite easy to do, I'll have a PR shortly.


Question: "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <del> "k8s.io/apimachinery/pkg/fields" </del> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Kubernetes e2e suite.[sig-scheduling] SchedulerPriorities [Serial] Pod should avoid nodes that have avoidPod annotation June 19th 2021 () Testgrid: https://k8s- Prow history: Looks like pods are scheduled where they shouldn't be? : The tests passes in maybe 20% of the builds, but it looks random, so I don't treat it as a flake. At the moment there are also sig-storage tests consistently failing in this job: , They don't look related though. /sig scheduling /kind failing-test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
try to have a look.
try to have a look.
/cc
This is expected. The feature (which was alpha) is now disabled by default. Users can still enable it by switching to a v1beta1 scheduler component config. Let's remove the test.


Question: echo "Starting etcd" kube::etcd::start <ins> # Admission Controllers to invoke prior to persisting objects in cluster ADMISSION_CONTROL=NamespaceLifecycle,NamespaceAutoProvision,LimitRanger,ResourceQuota </ins> APISERVER_LOG=/tmp/kube-apiserver.log sudo -E "${GO_OUT}/kube-apiserver"  --v=${LOG_LEVEL}  <ins> --admission_control="${ADMISSION_CONTROL}"  </ins> --address="${API_HOST}"  --port="${API_PORT}"  --runtime_config=api/v1beta3

Tried to create a service on a namespace 'staging' which doesn't exist on my k8s. Creation succeeded which is odd enough :) Then navigating to the entity also works: GET on /api/v1beta3/namespaces/staging/services/redis-service Navigating to the namespace itself produces 404 'namespaces "staging" not found': /api/v1beta3/namespaces/staging/ Getting all namespaces also doesn't list that namespace which sounds reasonable given the above. I'd expect one of the two: standard approach: a clear error that this namespace doesn't exist when attempting to create an entity on it. preferred imo, but better than nothing: creating silently behind the scenes that namespace cc
Sounds like an error configuring your environment. Is this using a standard provider environment or did you install your own? The kube-Apiserver needs the following flags set for admission_control=NamespaceLifecycle,NamespaceAutoProvision,LimitRanger,ResourceQuota In your case NamespaceAutoProvision must not be running. Sent from my iPhone
Admission control option NamespaceExists does your goal for 1. It is what OpenShift uses. NamespaceAutoProvision does 2. It is used upstream right now, but I think upstream should switch to NamespaceExists and require explicit creation, but that is harder to enable for things like e2e until more work is done. Sent from my iPhone
this is the hack/local-up- I didn't find the admissioncontrol set there at all, perhaps it's a bug in that particular setup?


Question: stepInterval  time.Duration skipIfBlocked bool destChan      chan time.Time <del> fired         bool </del> } func NewFakeClock(t time.Time) *FakeClock {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I was writing a variant of clock.FakeClock, and noticed a couple of bugs in . Look at the pointer comparison in --- the two pointers will never be equal, but the author clearly expected equality when is the index of the fakeClockWaiter corresponding to the fakeTimer. The two pointers will never be equal because one is a pointer into the fakeClock's slice while the other pointer is to a field of the . The consequence of this bug is that the Stop function is ineffective. Also, determines the returned value based on the original copy of the waiter, which --- being a copy --- did not get updated if/when the waiter fired. Note also that does not exercise FakeClock::NewTimer at all. What you expected to happen: I expected FakeClock Timers to work properly. How to reproduce it (as minimally and precisely as possible): Use the TestFakeTimer func introduced in Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
The fake timer's Reset() method also does not work correctly.


Question: export KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. source "${KUBE_ROOT}/hack/lib/init.sh" <ins> git config http.https://gopkg.in.followRedirects true </ins> go get -u gopkg.in/mikedanese/gazel.v13/gazel if ! "${GOPATH}/bin/gazel" -validate -print-diff -root="$(kube::realpath ${KUBE_ROOT})" ; then echo

When I run it locally: Small test case to induce failure: I am using git version 2.11.0, if it makes any difference.
Also applies to
This is caused by the most recent git upgrade and affects all of our deps but gazel is the one we pull the most. how do we mitigate this?
We could possibly set in verify/update-gazel, but that feels a bit iffy because it has to be global. Possibly print out the command to fix it in case the go get fails?
Source code for is here . There doesn't seem to be an open issue for this.
I filed
says
Sorry for the trouble. We'll change a bit to ensure the manual configuration is not necessary.
Was this fixed in
No need to change the configuration anymore. We've changed itself to support the new git quirks.
thanks for fixing this!


Question: return nodes == 1 }, time.Minute, time.Second).Should(gomega.BeTrue()) <del> v1alphaPodResources, err = getV1alpha1NodeDevices(ctx) framework.ExpectNoError(err, "should get node local podresources by accessing the (v1alpha) podresources API endpoint") v1PodResources, err = getV1NodeDevices(ctx) framework.ExpectNoError(err, "should get node local podresources by accessing the (v1) podresources API endpoint") </del> // Before we run the device plugin test, we need to ensure // that the cluster is in a clean state and there are no // pods running on this node. <del> gomega.Expect(v1alphaPodResources.PodResources).To(gomega.BeEmpty(), "should have no pod resources") gomega.Expect(v1PodResources.PodResources).To(gomega.BeEmpty(), "should have no pod resources") </del> <ins> // This is done in a gomega.Eventually with retries since a prior test in a different test suite could've run and the deletion of it's resources may still be in progress. // xref: https://issue.k8s.io/115381 gomega.Eventually(ctx, func(ctx context.Context) error { v1alphaPodResources, err = getV1alpha1NodeDevices(ctx) if err != nil { return fmt.Errorf("failed to get node local podresources by accessing the (v1alpha) podresources API endpoint: %v", err) } v1PodResources, err = getV1NodeDevices(ctx) if err != nil { return fmt.Errorf("failed to get node local podresources by accessing the (v1) podresources API endpoint: %v", err) } if len(v1alphaPodResources.PodResources) > 0 { return fmt.Errorf("expected v1alpha pod resources to be empty, but got non-empty resources: %+v", v1alphaPodResources.PodResources) } if len(v1PodResources.PodResources) > 0 { return fmt.Errorf("expected v1 pod resources to be empty, but got non-empty resources: %+v", v1PodResources.PodResources) } return nil }, f.Timeouts.PodDelete, f.Timeouts.Poll).Should(gomega.Succeed()) </ins> ginkgo.By("Scheduling a sample device plugin pod") data, err := e2etestfiles.Read(SampleDevicePluginDSYAML)

in W0128 10:46:06.683] [FAIL] [sig-node] Device Plugin [Feature:DevicePluginProbe][NodeFeature:DevicePluginProbe][Serial] DevicePlugin [Serial] [Disruptive] [BeforeEach] Can schedule a pod that requires a device The pod was left by the test case not sure No response No response_ /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc for (revert of )
these tests started failing after got merged?
The test passed after rebasing with the latest code. It should be fixed. Thanks.
It failed for some other remaining pods again.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It is still an issue, see It seems that some pods are sometimes not being cleaned up after the test finishes...
Taking a look at Test fails with On kubelet side it was deleted here: The logs show that API server recorded a delete of the pod at but the next test was already running at which is before the pod was deleted. It seems like the test that created namespace (which is ), completed before it deleted all the resources created in the test... edit: The namespace was deleted here: The namespace deletion in the test framework looks async, i.e. I think it will not block to wait until the namespace is deleted... not sure if this is by design or not...
See related slack thread - Namespace deletion is async by design and the test needs to account for this. For this case in particular, since requires no pods to be left over from other tests, we can adjust the BeforeBegin to wait and poll for a bit while all the resources are cleaned up.


Question: for _, item := range testCases { stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) <del> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{item.node}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{node: item.node}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) controller.NodeUpdated(nil, item.node)

What happened: A Pod was scheduled for deletion because a taint w/ effect was observed in the Node the Pod was assigned to, and after the taint is removed from the Node, the Pod scheduled deletion isn't canceled. After scheduling pod deletion, the only cancels pod deletion if all taints are removed from the node, including any taints that the user relies on for their use-cases (example below). What you expected to happen: After the taint that triggered scheduling pod deletion is removed, cancels the scheduled pod deletion. How to reproduce it (as minimally and precisely as possible): A node is tainted w/ , so any workloads already assigned to it which don't tolerate this taint are evicted. A pod is created w/ tolerations: w/ unspecified and w/ . is assigned to . At some point in time after the previous steps took place, is tainted as . gets an update event for and observes two taints. It proceeds to calculate the minimum time (in seconds) that tolerates for the two taints which at this point is 300 seconds, and marks for deletion in ~300 seconds - this happens in-memory, in a timed-queue. At some point in time after the previous steps took place, and before the 300 seconds trigger happen, taint is removed from . gets an update event for and observes only one taint, . It proceeds to calculate the minimum time (in seconds) that tolerates for the this taint which is infinity, and returns, not canceling the previous deletion. It completely ignores the fact that the taint that triggered deletion is no longer observed. Anything else we need to know?: Worked w/ to figure this out after all pods in a node got evicted without apparent reason. Environment: Kubernetes version (use ): 1.17.3 (but looking at the code seems to affect all of 1.17.x and 1.18.x and master. Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): 5.x Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A cc (owners of this code)
There seems to be at least .


Question: for _, item := range testCases { stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) <del> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{item.node}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{node: item.node}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) controller.NodeUpdated(nil, item.node)

You can sort of get a feel for what's going on here by looking at these two code snippets: This is what happens in when the node does not have any NoExecute taints: If you have any NoExecute taints on the node processPodOnNode gets called and this routine is really, really eager to evict pods. So : We are currently working around this issue by changing over our NoExecute taints to NoSchedule.
/sig scheduling node
/cc
/remove-sig scheduling scheduler isn't involved in this case.
Which SIG owns the controller-manager or is the node lifecycle manager (and its NoExecuteTaintManager) owned by sig-node alone?
I think node lifecycle is jointly owned by sig-node and sig-cloud-provider
API machinery owns the mechanics of the controller manager (controller loop setup and management), but the individual controllers are SIG-specific
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Looks like we lost some momentum here and I'd like to avoid this falling into the abyss of bugs that never get fixed. When this bug is triggered the outcome is pretty terrible and once it does happen it is very difficult to dig into and understand what actually happened. Did we ever get this routed to the right owner?
adding sig-node leads for triage/routing. this seems like a clearly reproducible bug with significant correctness issues. /assign
thanks for the detailed report and reproducer. to help accelerate this, do you have a form of your reproducer we could add in the e2e suite (even if its failing initially)?
custom taints are considered disruptive, so it couldn't run in the main suite
It's been a while since I wrote tests for the e2e suite but if you are available over slack in case guidance is needed, I'm willing to put the effort.
Do all suites need to pass for a fix to be merged? I'd assume it is the case, otherwise how would a regression be prevented after the fix lands?


Question: for _, item := range testCases { stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) <del> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{item.node}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{node: item.node}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) controller.NodeUpdated(nil, item.node)

i was more inquiring if there was a test that could automate the reproduction, so even just a linked commit may be helpful for someone that wants to pick this up. i am happy to help if you want to reach out over slack later this week.
looks like a unit test could reproduce the issue, opened a WIP fix in
our code is not publicly accessible, I'm sorry. PR linked by seems to cover both the fix and the test that proves it.
/remove-lifecycle frozen
frozen keeps the bot from auto-closing
Ah, sorry then.


Question: return nil, fmt.Errorf("%s is an incomplete jsonpath template", j.name) } <del> j.cur = []reflect.Value{reflect.ValueOf(data)} </del> <ins> cur := []reflect.Value{reflect.ValueOf(data)} </ins> nodes := j.parser.Root.Nodes fullResult := [][]reflect.Value{} for i := 0; i < len(nodes); i++ { node := nodes[i] <del> results, err := j.walk(j.cur, node) </del> <ins> results, err := j.walk(cur, node) </ins> if err != nil { return nil, err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: (or in my instance) with does not correctly handle nested range statements. For example, one would expect that the following: would result in a list being output similar to this: but instead, results in a list being output like this: What you expected to happen: Nested range statements complete in the order in which they are specified in jsonpath query. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.11.0 Cloud provider or hardware configuration: OpenShift 3.11.141 on vSphere 6.5 OS (e.g: ): RHEL 7.7 Kernel (e.g. ): 3.10.0 Install tools:
/sig cli
/area kubectl
/assign
/sig api-machinery
/remove-sig cli
/remove-area cli /area api
Those labels are not set on the issue: <detailsIn response to : /remove-area cli /area api Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-area kubectl


Question: }, func(client kubeclientset.Interface, obj runtime.Object) error { namespace := obj.(*apiv1.Namespace) <del> err := client.Core().Namespaces().Delete(namespace.Name, &metav1.DeleteOptions{}) </del> <ins> orphanDependents := false err := client.Core().Namespaces().Delete(namespace.Name, &metav1.DeleteOptions{OrphanDependents: &orphanDependents}) </ins> // IsNotFound error is fine since that means the object is deleted already. if errors.IsNotFound(err) { return nil

Problem: When users sends a cascading deletion request to federation i.e they set orphanDependents=false, then federation should also set orphanDependents=false, when it sends the request to underlying clusters.
Fixed by


Question: Done(types.UID) Update(logger klog.Logger, oldPod, newPod *v1.Pod) Delete(pod *v1.Pod) <del> // TODO(sanposhiho): move all PreEnqueueCheck to Requeue and delete it from this parameter eventually. // Some PreEnqueueCheck include event filtering logic based on some in-tree plugins // and it affect badly to other plugins. // See https://github.com/kubernetes/kubernetes/issues/110175 </del> <ins> // Important Note: preCheck shouldn't include anything that depends on the in-tree plugins' logic. // (e.g., filter Pods based on added/updated Node's capacity, etc.) // We know currently some do, but we'll eventually remove them in favor of the scheduling queue hint. </ins> MoveAllToActiveOrBackoffQueue(logger klog.Logger, event framework.ClusterEvent, oldObj, newObj interface{}, preCheck PreEnqueueCheck) AssignedPodAdded(logger klog.Logger, pod *v1.Pod) AssignedPodUpdated(logger klog.Logger, oldPod, newPod *v1.Pod, event framework.ClusterEvent)

In our custom plugin, we want to move pods that is unscheduled by the plugin to activeQ/backoffQ when a new Pod is created. So, we define like this: But, even when a new Pod is created, the Pod that was unscheduled by our custom plugin isn't moved to backoffQ/activeQ. when a new Pod is created, the Pod unscheduled by our custom plugin is moved to backoffQ/activeQ. a custom plugin that has pod events in EventsToRegister. a new Pod that the custom plugin will reject. another new Pod that makes the Pod created in (1) schedulable. can see the Pod created in (1) isn't scheduled immediately (it should've been scheduled immediately after (3) because (3) should move the Pod created in (1) to activeQ/backoffQ.) If I understand correctly and if it's not expected behavior, this bug has a non-small impact, especially in v1.24 because we change to 5 min. v1.23.4 but I think latest master has the same issue. <details</details<details</details<details</details<details</details<details</details
/assign /sig scheduling
Maybe it's only on a newly created unscheduled Pod. We've moved correctly for events related to assigned Pod.
And deleting Node events has the same issue...? I guess we need to run when some Nodes are deleted. In in-tree plugins, only pod topology spread uses the deleting Node event and I guess this bug affects its performance. (Even when some Nodes are deleted, the Pods, that were unscheduled by Pod topology spread, aren't moved to activeQ/backoffQ by that deleting events.)
Hm, it may be a problem that events related to Node and Pod are not always called when an event occurs. (Even if it's intended behavior, at least we need to write this on the comment of .) /retitle scheduler: events in related to Pod and Node don't always work
To summarize: The problem is "not all events related to Pod or Node don't always make move from unschedulableQ to active/backoffQ and it may not be intended for custom-plugin developers because this behavior isn't even documented." I think we should move all unschedulable Pods with all Pod and Node events.


Question: Done(types.UID) Update(logger klog.Logger, oldPod, newPod *v1.Pod) Delete(pod *v1.Pod) <del> // TODO(sanposhiho): move all PreEnqueueCheck to Requeue and delete it from this parameter eventually. // Some PreEnqueueCheck include event filtering logic based on some in-tree plugins // and it affect badly to other plugins. // See https://github.com/kubernetes/kubernetes/issues/110175 </del> <ins> // Important Note: preCheck shouldn't include anything that depends on the in-tree plugins' logic. // (e.g., filter Pods based on added/updated Node's capacity, etc.) // We know currently some do, but we'll eventually remove them in favor of the scheduling queue hint. </ins> MoveAllToActiveOrBackoffQueue(logger klog.Logger, event framework.ClusterEvent, oldObj, newObj interface{}, preCheck PreEnqueueCheck) AssignedPodAdded(logger klog.Logger, pod *v1.Pod) AssignedPodUpdated(logger klog.Logger, oldPod, newPod *v1.Pod, event framework.ClusterEvent)

If concerned about performance degradation, we can extend or , for example, to have the function like ? This problem has a non-small impact, especially in v1.24+ because we change PodMaxUnschedulableQDuration to 5 min. Let's see the details... ↓ We move the unscheduled Pods (rejected by the plugin that has Pod add/update events in ) in or But, the target Pods against this moving are only unschedulable pods which have any affinity term that matches "pod". So,,,, users cannot move from unschedulableQ to activeQ/backoffQ with all assignedPod events? For example, for Pod Topology Spread (it uses Pod All event in ), the new assigned Pod creation may make some Pods, that are rejected by Pod Topology Spread, schedulable. Regardless of whether that unschedulable Pods have any affinity term that matches a newly created assigned pod or not. Of course this is not just a problem for Pod Topology Spread, it could be a problem for user's custom-plugins. I guess we should pass all unscheduled Pods to . Or at least we should document this behavior. No issue on this. We don't move the unschedulable Pods with those events at all. That's a problem... Even when a plugin uses Pod add/update/delete event in and wants to move the rejected Pods with non-scheduled Pod's add/update/delete events, scheduling queue doesn't move the unschedulable Pods with those events. (In our company, we have the custom plugin for and the plugin wants the scheduler queye to move the Pods rejected by the plugin from unschedulableQ to active/backoffQ when a new non-scheduled Pod is created.) We move only unscheduled Pods that pass . checks the Node can run that unscheduled Pod from the points of view of some in-tree filter plugins like noderesources/nodeport/nodeAffinity/nodename. The problem that arises here is similar to "Assigned Pod's add/update events". For example, for Pod Topology Spread (it uses Node Add event in ), the new Node creation may make some Pods, that are rejected by Pod Topology Spread, schedulable. Regardless of the results of against that Node. (In , we change the skew calculation of Pod Topology Spread to exclude Nodes that don't match the node affinity.


Question: Done(types.UID) Update(logger klog.Logger, oldPod, newPod *v1.Pod) Delete(pod *v1.Pod) <del> // TODO(sanposhiho): move all PreEnqueueCheck to Requeue and delete it from this parameter eventually. // Some PreEnqueueCheck include event filtering logic based on some in-tree plugins // and it affect badly to other plugins. // See https://github.com/kubernetes/kubernetes/issues/110175 </del> <ins> // Important Note: preCheck shouldn't include anything that depends on the in-tree plugins' logic. // (e.g., filter Pods based on added/updated Node's capacity, etc.) // We know currently some do, but we'll eventually remove them in favor of the scheduling queue hint. </ins> MoveAllToActiveOrBackoffQueue(logger klog.Logger, event framework.ClusterEvent, oldObj, newObj interface{}, preCheck PreEnqueueCheck) AssignedPodAdded(logger klog.Logger, pod *v1.Pod) AssignedPodUpdated(logger klog.Logger, oldPod, newPod *v1.Pod, event framework.ClusterEvent)

But, after is implemented, the calculation can be configured to include those Nodes.) We don't move the unschedulable Pods with Node delete events... That's a problem. Even when the plugin use Node delete event in , scheduling queue doesn't move the unschedulable Pods with any Node delete events.)
/cc Could you please take a look
These are all intended optimizations. These basic checks prevent the scheduler from retrying unschedulable pods over and and over. So far we haven't had the need to make these checks configurable as they check well established scheduling features. +1 on documenting the behavior, but I'm not sure about changing it or making it configurable (that adds complexity that might not be justified). What is it that you are trying to do?
What is it that you are trying to do? We are trying to remove flushing in queue in , right? Without improving like I suggest, some Pods in unschedulableQ cannot move back to activeQ forever in some cases if we remove flushing. (Actually, our company's custom plugin needs to wait flush because non-assigned Pod's events don't let scheduling queue move unschedulable Pods rejected by that plugin.) So, I think we should simply inform all events to queue and let queue move unschedulable Pods with that events. This can happen in some custom plugins like our company's one and also even in in-tree plugins. For example, Pod Topology Spread (after is implemented). When a user uses and , the creation of Node that doesn't match nodeAffinity/nodeSelector may change the scheduling result.
cc who implemented NodeInclusionPolicy. Can you provide a concrete example? The policy just refers to the topology spread calculations. It doesn't mean that the node doesn't have to match the nodeaffinity to be able to accept the pod. I'm asking what is your custom plugin trying to do. Depending on whether it makes sense, we need to evaluate if it's worth the complexity.
For example, for Pod Topology Spread (it uses Pod All event in EventsToRegister), the new assigned Pod creation may make some Pods, that are rejected by Pod Topology Spread, schedulable This is actually a problem here, 5 mins would be too long for background flush. But it doesn't mean that we should simply shorten the interval time.


Question: Done(types.UID) Update(logger klog.Logger, oldPod, newPod *v1.Pod) Delete(pod *v1.Pod) <del> // TODO(sanposhiho): move all PreEnqueueCheck to Requeue and delete it from this parameter eventually. // Some PreEnqueueCheck include event filtering logic based on some in-tree plugins // and it affect badly to other plugins. // See https://github.com/kubernetes/kubernetes/issues/110175 </del> <ins> // Important Note: preCheck shouldn't include anything that depends on the in-tree plugins' logic. // (e.g., filter Pods based on added/updated Node's capacity, etc.) // We know currently some do, but we'll eventually remove them in favor of the scheduling queue hint. </ins> MoveAllToActiveOrBackoffQueue(logger klog.Logger, event framework.ClusterEvent, oldObj, newObj interface{}, preCheck PreEnqueueCheck) AssignedPodAdded(logger klog.Logger, pod *v1.Pod) AssignedPodUpdated(logger klog.Logger, oldPod, newPod *v1.Pod, event framework.ClusterEvent)

For example, Pod Topology Spread (after is implemented). When a user uses NodeAffinityPolicy: Ignore and WhenUnsatisfiable: DoNotSchedule, the creation of Node that doesn't match nodeAffinity/nodeSelector may change the scheduling result. The skew of a new created node is always 0, so it's still unfit I think.
Our company's custom plugin is like plugin. And like coscheduling's prefilter, our plugin rejects Pods in PreFilter when the number of Pods is less than the defined number. Yes, so, maybe coscheduling plugin has the same issue as our plugin. The coscheduling plugin also uses Pod Add event and I guess it's for non-scheduled Pod's creation. (I don't know much about the coscheduling plugin though.) Oh, that's true Please forget about that example. But, how about like this scenario: user uses and Pod is rejected by Pod Topology Spread because of the skew. e.g. a user uses and TopologyKey is hostname, NodeA/NodeB has 10 Pods each, and NodeC has 8 Pods. adds un-tolerated taint to NodeC skew calculation result should be changed and the Pod rejected in (2) will be allowed by Pod Topology Spread. But the Pod isn't moved from unschedulableQ without flushing because of this issue.
Maybe we can find some pointcuts by the in , e.g. if pod was rejected by PodTopologySpread, we'll when Assigned Pod's add/update events arrived. For customize plugins, we may have to do some refactoring to register the filter func together with the event.
So now NodeC is excluded from calculations. Hence the pod fits in either NodeA or NodeB, which were unchanged. Tricky... I'll have to think about it. We really need those quick checks to pass for performance in most scenarios. /triage accepted
Thanks for bringing this up. Some comments below: You're right. Pods failed by PodTopologySpread may not get re-queued immediately. However, an assigned Pod's add/update event would only impact unschedulable Pods with cross-node scheduling directives. As of today, it's InterPodAffinity and PodTopologySpread.


Question: Done(types.UID) Update(logger klog.Logger, oldPod, newPod *v1.Pod) Delete(pod *v1.Pod) <del> // TODO(sanposhiho): move all PreEnqueueCheck to Requeue and delete it from this parameter eventually. // Some PreEnqueueCheck include event filtering logic based on some in-tree plugins // and it affect badly to other plugins. // See https://github.com/kubernetes/kubernetes/issues/110175 </del> <ins> // Important Note: preCheck shouldn't include anything that depends on the in-tree plugins' logic. // (e.g., filter Pods based on added/updated Node's capacity, etc.) // We know currently some do, but we'll eventually remove them in favor of the scheduling queue hint. </ins> MoveAllToActiveOrBackoffQueue(logger klog.Logger, event framework.ClusterEvent, oldObj, newObj interface{}, preCheck PreEnqueueCheck) AssignedPodAdded(logger klog.Logger, pod *v1.Pod) AssignedPodUpdated(logger klog.Logger, oldPod, newPod *v1.Pod, event framework.ClusterEvent)

So instead of "pass all unscheduled Pods to movePodsToActiveOrBackoffQueue", I'd like to come up with a solution like: It's pretty rare to move pods depending on Unschedulable Pods' events. In your particular case, maybe a proactive call would behave more efficiently. It doesn't sound correlated, is it? As the check will just pre-verify the basic checks which are orthogonal with PodTopologySpread. In other words, it's impossible that a Pod failed can become schedulable eventually. Updated: The example is a valid point... Can you share some use-cases to move Pods upon node's deletion? In the before, I don't think we have any, until the last release the introduction of feature (as in that case deleting a node would impact the number of domains).
to come up with a solution like: If we do so, also, we need to provide the way to pass some functions to decide which Pods should pass to like for out-of-tree plugins? call would behave more efficiently. Ah~, that's a good idea :) Maybe like this (similar to ) user uses WhenUnsatisfiable: DoNotSchedule. Pod is rejected by Pod Topology Spread because of the skew. e.g. a user uses maxSkew: 1 and TopologyKey is hostname, NodeA/NodeB has 10 Pods each, and NodeC has 8 Pods. deletes NodeC skew calculation result should be changed and the Pod rejected in (2) will be allowed by Pod Topology Spread. But the Pod isn't moved from unschedulableQ without flushing because of this issue. So,,, ignoring some events for performance in scheduler side (without providing a way to change this) could be inconvenient for some plugins. What I thought in the above previous comment () is to define "which kind of resource's events should be ignored" in plugin side. (similar to 's idea? ) We can add pre filter functions in like: Then we can set any pre-filter functions for each plugin. And, we can define some default behavior for PreFilterFn and use them when is empty. So, for example, if most of the plugins are considered to prefer to ignore the event for Node that doesn't match nodeAffinity/nodeSelector, we can define this behavior as the default behavior.


Question: Done(types.UID) Update(logger klog.Logger, oldPod, newPod *v1.Pod) Delete(pod *v1.Pod) <del> // TODO(sanposhiho): move all PreEnqueueCheck to Requeue and delete it from this parameter eventually. // Some PreEnqueueCheck include event filtering logic based on some in-tree plugins // and it affect badly to other plugins. // See https://github.com/kubernetes/kubernetes/issues/110175 </del> <ins> // Important Note: preCheck shouldn't include anything that depends on the in-tree plugins' logic. // (e.g., filter Pods based on added/updated Node's capacity, etc.) // We know currently some do, but we'll eventually remove them in favor of the scheduling queue hint. </ins> MoveAllToActiveOrBackoffQueue(logger klog.Logger, event framework.ClusterEvent, oldObj, newObj interface{}, preCheck PreEnqueueCheck) AssignedPodAdded(logger klog.Logger, pod *v1.Pod) AssignedPodUpdated(logger klog.Logger, oldPod, newPod *v1.Pod, event framework.ClusterEvent)

What do you think about this idea?
Thanks. It makes sense to me. To summarize, the following cases doesn't properly re-queue relevant pods: Pod's add/update events -scheduled Pod's add/update/delete events 's add/update/delete events For 1 & 2, I've commented above with solutions. (1 needs core changes similar like podAffinity, 2 may leverage the Activate hook) For 3, with 's examples, I'm convinced this is some area we need to improve. One simple and stratightforward solution is to plumb a hook to let the plugin claim "alright, upon Node events, move the pod failed by me anyways". Then once this knob is registered, we move pods failed by that plugin back unconditionally; instead of proceeding with . This hook should be available to in-tree and out-of-tree plugins. In terms of in-tree plugins, it's just PodTopologySpread (we discussed this in today's sig meeting, even InterPodAffinity doesn't fall into this case) So, as a first step, let's ensure we're on the same page and agree on the ^^ summary. Next, we can better evaluate trade-offs and come up with an optimal solution.
Thanks for discussing this topic in the sig meeting. But, I am still believing that all logic related to a plugin should be controllable from the plugin side and we should not create something that can be achieved in in-tree plugins but not in out-of-tree plugins. So... the benefit of the "customizable PreCheck approach" () isn't only to solve this issue. We can use this feature for other optimization: like improving the whole performance of scheduler by moving only Pods that are more likely to be able to be scheduled by defining more detailed events on with customizable PreCheck. Many plugins have a particular scenario that makes rejected Pods schedulable and the current doesn't have an abundant enough ability of expression for this. (So, today many plugins need to say "register this hoge event on foo resource. This event may make rejected Pods schedulable" on comments in . ) Yes, I watched the sig-meeting recording as well and the opinion that the "customizable PreCheck approach" increases some implementation complexity surely makes sense. But,,, isn't this complexity increasing acceptable even if the above is taken into account?


Question: Done(types.UID) Update(logger klog.Logger, oldPod, newPod *v1.Pod) Delete(pod *v1.Pod) <del> // TODO(sanposhiho): move all PreEnqueueCheck to Requeue and delete it from this parameter eventually. // Some PreEnqueueCheck include event filtering logic based on some in-tree plugins // and it affect badly to other plugins. // See https://github.com/kubernetes/kubernetes/issues/110175 </del> <ins> // Important Note: preCheck shouldn't include anything that depends on the in-tree plugins' logic. // (e.g., filter Pods based on added/updated Node's capacity, etc.) // We know currently some do, but we'll eventually remove them in favor of the scheduling queue hint. </ins> MoveAllToActiveOrBackoffQueue(logger klog.Logger, event framework.ClusterEvent, oldObj, newObj interface{}, preCheck PreEnqueueCheck) AssignedPodAdded(logger klog.Logger, pod *v1.Pod) AssignedPodUpdated(logger klog.Logger, oldPod, newPod *v1.Pod, event framework.ClusterEvent)

EDIT: It is a bad habit of mine to always write long sentences even though I am not very good at English. Summarizing the above: I think it's not a good idea to create something that can be achieved in in-tree plugins but not in out-of-tree plugins. If we need a solid advantage to extend like this, then it has the advantage of optimizing the queue per plugin like described above.
I didn't jump in the thread since it's actively on the discussion. But, the issue is the similar one. (This time, we can solve that issue by changing scheduling queue logic since It's the k/k feature though.)
I'm not sure I fully understand your intention. The slack thread was discussing the AssignedPod update case, so it fallls into case 1 in , which we already agreed to improve in upstream.
Hi Are you working on this? I think this is quite important and if you're not working on this, I can follow.
Sorry, leave here for a while. I wanted to wait for (and then forgot to come back here. ) That proposal is quite similar to what I want to see in the conclusion of this issue; every plugin can judge if each Pod in unschedQ can be moved to schedQ/backoffQ. Do you think it makes sense to add "By which event the pod will be moved back to schedQ/backoffQ" on 's argument? I know and isn't needed in DefaultEnqueue plugin for API, but that's the last missing puzzle piece : If has , enqueue plugin can judge if the event makes it possible to schedule the Pod. For example, I believe we can move ↓ this logic to Pod Affinity plugin by implementing in Pod Affinity plugin. Pod Affinity plugin will remember Pods rejected by itself in . And in , if the given Pod was the one rejected by Pod Affinity, is AssignedPod/Created|Updated and is Pod affinity with the rejected Pod, then move the Pod back to activeQ/backoffQ. Otherwise reject the Pod moving. We can resolve (3) by implementing as well. (by exactly the way you said.↓) My original concern in this issue is that having logic for specific plugins in the scheduler core may prevent out-of-tree plugins' extendability. The scheduler itself should be pure, and shouldn't do anything special for any plugins implementation ideally.


Question: Done(types.UID) Update(logger klog.Logger, oldPod, newPod *v1.Pod) Delete(pod *v1.Pod) <del> // TODO(sanposhiho): move all PreEnqueueCheck to Requeue and delete it from this parameter eventually. // Some PreEnqueueCheck include event filtering logic based on some in-tree plugins // and it affect badly to other plugins. // See https://github.com/kubernetes/kubernetes/issues/110175 </del> <ins> // Important Note: preCheck shouldn't include anything that depends on the in-tree plugins' logic. // (e.g., filter Pods based on added/updated Node's capacity, etc.) // We know currently some do, but we'll eventually remove them in favor of the scheduling queue hint. </ins> MoveAllToActiveOrBackoffQueue(logger klog.Logger, event framework.ClusterEvent, oldObj, newObj interface{}, preCheck PreEnqueueCheck) AssignedPodAdded(logger klog.Logger, pod *v1.Pod) AssignedPodUpdated(logger klog.Logger, oldPod, newPod *v1.Pod, event framework.ClusterEvent)

If we can move all such logic to each plugin, that'd be perfect.
^ is more like a new feature than just a solution to this problem. I created another issue and describe my proposal in detail. Let's have a discussion there.
It can be resolved by moving all preCheck into each plugin's QueueingHint. Let me retitle this issue so that the current status is easy to understand. /retitle scheduler: move all preCheck to QueueingHint
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted


Question: function create-certs { execute-cmd-on-pre-existing-master-with-retries 'sudo cat /etc/kubernetes/admin.conf' > /tmp/kubeconfig <del> # CA_CERT_BASE64, KUBELET_CERT_BASE64 and KUBELET_KEY_BASE64 might be used # in test/kubemark/iks/util.sh # If it becomes clear that the variables are not used anywhere, then we can # remove them: CA_CERT_BASE64=$(grep certificate-authority /tmp/kubeconfig | awk '{print $2}' | head -n 1) KUBELET_CERT_BASE64=$(grep client-certificate-data /tmp/kubeconfig | awk '{print $2}' | head -n 1) KUBELET_KEY_BASE64=$(grep client-key-data /tmp/kubeconfig | awk '{print $2}' | head -n 1) export CA_CERT_BASE64 export KUBELET_CERT_BASE64 export KUBELET_KEY_BASE64 </del> }

While trying to run with a valid ibmcloud account, the following error happened I also tried to run and got the same error. Looks like a compatibility issue. ibmcloud CLI shouldn't return error Run with a valid IBM cloud account. No response I am trying to build kubemark from source <details</details<detailsIBM cloud IKS </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:


Question: runningPod := podPair.RunningPod apiPod := podPair.APIPod <del> if killing.Has(string(runningPod.ID)) { // The pod is already being killed. break </del> <ins> lock.Lock() exists := killing.Has(string(runningPod.ID)) if !exists { killing.Insert(string(runningPod.ID)) </ins> } <del> killing.Insert(string(runningPod.ID)) go func(apiPod *v1.Pod, runningPod *kubecontainer.Pod, ch chan types.UID) { defer func() { ch <- runningPod.ID }() glog.V(2).Infof("Killing unwanted pod %q", runningPod.Name) err := kl.killPod(apiPod, runningPod, nil, nil) if err != nil { glog.Errorf("Failed killing the pod %q: %v", runningPod.Name, err) } }(apiPod, runningPod, resultCh) </del> <ins> lock.Unlock() </ins> <del> case podID := <-resultCh: killing.Delete(string(podID)) </del> <ins> if !exists { go func(apiPod *v1.Pod, runningPod *kubecontainer.Pod) { glog.V(2).Infof("Killing unwanted pod %q", runningPod.Name) err := kl.killPod(apiPod, runningPod, nil, nil) if err != nil { glog.Errorf("Failed killing the pod %q: %v", runningPod.Name, err) } lock.Lock() killing.Delete(string(runningPod.ID)) lock.Unlock() }(apiPod, runningPod) } </ins> } } }

There is a defer calling in this method: defer close(resultCh) It may crash the goroutine created at line 777, which then will make the whole kubelet crash. A simple fix Another simple fix
cc do you think this make sense?
You could just drop the channel and go with a mutex to update the set.
indeed.


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package plugins import ( "fmt" "regexp" "strings" "k8s.io/api/core/v1" storage "k8s.io/api/storage/v1" ) const ( // AzureDiskDriverName is the name of the CSI driver for Azure Disk AzureDiskDriverName = "disk.csi.azure.com" // AzureDiskInTreePluginName is the name of the intree plugin for Azure Disk AzureDiskInTreePluginName = "kubernetes.io/azure-disk" // Parameter names defined in azure disk CSI driver, refer to // https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/docs/driver-parameters.md azureDiskCachingMode = "cachingMode" azureDiskFSType      = "fsType" ) var ( managedDiskPathRE   = regexp.MustCompile(`.*/subscriptions/(?:.*)/resourceGroups/(?:.*)/providers/Microsoft.Compute/disks/(.+)`) unmanagedDiskPathRE = regexp.MustCompile(`http(?:.*)://(?:.*)/vhds/(.+)`) ) var _ InTreePlugin = &azureDiskCSITranslator{} // azureDiskCSITranslator handles translation of PV spec from In-tree // Azure Disk to CSI Azure Disk and vice versa type azureDiskCSITranslator struct{} // NewAzureDiskCSITranslator returns a new instance of azureDiskTranslator func NewAzureDiskCSITranslator() InTreePlugin { return &azureDiskCSITranslator{} } // TranslateInTreeStorageClassParametersToCSI translates InTree Azure Disk storage class parameters to CSI storage class func (t *azureDiskCSITranslator) TranslateInTreeStorageClassToCSI(sc *storage.StorageClass) (*storage.StorageClass, error) { return sc, nil } // TranslateInTreePVToCSI takes a PV with AzureDisk set from in-tree // and converts the AzureDisk source to a CSIPersistentVolumeSource func (t *azureDiskCSITranslator) TranslateInTreePVToCSI(pv *v1.PersistentVolume) (*v1.PersistentVolume, error) { if pv == nil || pv.Spec.AzureDisk == nil { return nil, fmt.Errorf("pv is nil or Azure Disk source not defined on pv") } azureSource := pv.Spec.PersistentVolumeSource.AzureDisk // refer to https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/docs/driver-parameters.md csiSource := &v1.CSIPersistentVolumeSource{ Driver:           AzureDiskDriverName, VolumeHandle:     azureSource.DataDiskURI, ReadOnly:         *azureSource.ReadOnly, FSType:           *azureSource.FSType, VolumeAttributes: map[string]string{}, } if *azureSource.CachingMode != "" { csiSource.VolumeAttributes[azureDiskCachingMode] = string(*azureSource.CachingMode) } if *azureSource.FSType != "" { csiSource.VolumeAttributes[azureDiskFSType] = *azureSource.FSType } pv.Spec.PersistentVolumeSource.AzureDisk = nil pv.Spec.PersistentVolumeSource.CSI = csiSource pv.Spec.AccessModes = backwardCompatibleAccessModes(pv.Spec.AccessModes) return pv, nil } // TranslateCSIPVToInTree takes a PV with CSIPersistentVolumeSource set and // translates the Azure Disk CSI source to a AzureDisk source. func (t *azureDiskCSITranslator) TranslateCSIPVToInTree(pv *v1.PersistentVolume) (*v1.PersistentVolume, error) { if pv == nil || pv.Spec.CSI == nil { return nil, fmt.Errorf("pv is nil or CSI source not defined on pv") } csiSource := pv.Spec.CSI diskURI := csiSource.VolumeHandle diskName, err := getDiskName(diskURI) if err != nil { return nil, err } // refer to https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/docs/driver-parameters.md azureSource := &v1.AzureDiskVolumeSource{ DiskName:    diskName, DataDiskURI: diskURI, FSType:      &csiSource.FSType, ReadOnly:    &csiSource.ReadOnly, } if csiSource.VolumeAttributes != nil { if cachingMode, ok := csiSource.VolumeAttributes[azureDiskCachingMode]; ok { mode := v1.AzureDataDiskCachingMode(cachingMode) azureSource.CachingMode = &mode } if fsType, ok := csiSource.VolumeAttributes[azureDiskFSType]; ok && fsType != "" { azureSource.FSType = &fsType } } pv.Spec.CSI = nil pv.Spec.AzureDisk = azureSource return pv, nil } // CanSupport tests whether the plugin supports a given volume // specification from the API.  The spec pointer should be considered // const. func (t *azureDiskCSITranslator) CanSupport(pv *v1.PersistentVolume) bool { return pv != nil && pv.Spec.AzureDisk != nil } // GetInTreePluginName returns the name of the intree plugin driver func (t *azureDiskCSITranslator) GetInTreePluginName() string { return AzureDiskInTreePluginName } // GetCSIPluginName returns the name of the CSI plugin func (t *azureDiskCSITranslator) GetCSIPluginName() string { return AzureDiskDriverName } func isManagedDisk(diskURI string) bool { if len(diskURI) > 4 && strings.ToLower(diskURI[:4]) == "http" { return false } return true } func getDiskName(diskURI string) (string, error) { diskPathRE := managedDiskPathRE if !isManagedDisk(diskURI) { diskPathRE = unmanagedDiskPathRE } matches := diskPathRE.FindStringSubmatch(diskURI) if len(matches) != 2 { return "", fmt.Errorf("could not get disk name from %s, correct format: %s", diskURI, diskPathRE) } return matches[1], nil } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Azure is not listed as "migratable" in could refer to Why is this needed:
/sig azure /assign


Question: defer framework.DeleteRCAndPods(f.ClientSet, ns, "reserve-all-cpu") framework.ExpectNoError(err) <del> By("creating a new instance of DNS and waiting for DNS to be scheduled") label := labels.SelectorFromSet(labels.Set(map[string]string{"k8s-app": "kube-dns"})) </del> <ins> By("creating a new instance of Dashboard and waiting for Dashboard to be scheduled") label := labels.SelectorFromSet(labels.Set(map[string]string{"k8s-app": "kubernetes-dashboard"})) </ins> listOpts := api.ListOptions{LabelSelector: label} rcs, err := f.ClientSet.Core().ReplicationControllers(api.NamespaceSystem).List(listOpts) framework.ExpectNoError(err)

In , for It("should ensure that critical pod is scheduled in case there is no resources available"), kube-dns is used as the critical pod target and being scaled out and in. However, we plan to enable the dns horizontal autoscaling feature in the near future, such as this . If this feature is turned on, the corresponding autoscaler will fight with this rescheduler e2e and maintain the desired number of replicas. Hence this test will be very likely to fail. So probably we should use another critical pod here, or either create a new critical pod that does not exist before in order to protect the ongoing functionalities.
thanks for letting me know. What is the ETA for enabling horizontal autoscaler for DNS?
I believe it would be enabled no later than the 1.5 release.
Sorry to bother, is there an ETA for this? FYI, as merged, we will upgrade the addon-manager to support rolling update for once is ready to go. After that I will start working on enabling horizontal autoscaler for . These will happen before 1.5 code freeze.
Sorry for the delay. fixes this
Thanks for helping!