Question: fs.BoolVar(&s.EnableLogsHandler, "enable-logs-handler", s.EnableLogsHandler, "If true, install a /logs handler for the apiserver logs.") <ins> fs.MarkDeprecated("enable-logs-handler", "This flag will be removed in v1.19") </ins> // Deprecated in release 1.9 fs.StringVar(&s.SSHUser, "ssh-user", s.SSHUser,

What would you like to be : Deprecate the flag which enables exposing logs of apiserver. Why is this needed: As per , in order to disable accessing apiserver logs in the future, we need to deprecate the current default and wait for the requisite time period.
/sig api-machinery
/assign
I'm in favor of deprecating this. It's potentially disruptive, I think an email blast to kubernetes-dev would be a good idea.
Thanks , I have posted in kubernetes-dev mailing list.
I'm also in favor of removing this functionality from the apiserver. Since this is GA behavior, the deprecation period would be 12 months.
cc


Question: <ins> #!/bin/bash # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Sets up FlexVolume drivers on GCE COS instances using mounting utilities packaged in a Google # Container Registry image. # The user-provided FlexVolume driver(s) must be under /flexvolume of the image filesystem. # For example, the driver k8s/nfs must be located at /flexvolume/k8s~nfs/nfs . # # This script should be used on a clean instance, with no FlexVolume installed. # Should not be run on instances with an existing full or partial installation. # Upon failure, the script will clean up the partial installation automatically. # # Must be executed under /home/kubernetes/bin with sudo. # Warning: kubelet will be restarted upon successful execution. set -o errexit set -o nounset set -o pipefail MOUNTER_IMAGE=${1:-} MOUNTER_PATH=/home/kubernetes/flexvolume_mounter VOLUME_PLUGIN_DIR=/etc/srv/kubernetes/kubelet-plugins/volume/exec usage() { echo "usage: $0 imagename[:tag]" echo "    imagename  Name of a Container Registry image. By default the latest image is used." echo "    :tag       Container Registry image tag." exit 1 } if [ -z ${MOUNTER_IMAGE} ]; then echo "ERROR: No Container Registry mounter image is specified." echo usage fi # Unmounts a mount point lazily. If a mount point does not exist, continue silently, # and without error. umount_silent() { umount -l $1 &> /dev/null || /bin/true } # Waits for kubelet to restart for 1 minute. kubelet_wait() { timeout=60 kubelet_readonly_port=10255 until [[ $timeout -eq 0 ]]; do printf "." if [[ $( curl -s http://localhost:${kubelet_readonly_port}/healthz ) == "ok" ]]; then return 0 fi sleep 1 timeout=$(( timeout-1 )) done # Timed out waiting for kubelet to become healthy. return 1 } flex_clean() { echo echo "An error has occurred. Cleaning up..." echo umount_silent ${VOLUME_PLUGIN_DIR} rm -rf ${VOLUME_PLUGIN_DIR} umount_silent ${MOUNTER_PATH}/var/lib/kubelet umount_silent ${MOUNTER_PATH} rm -rf ${MOUNTER_PATH} if [ -n ${IMAGE_URL:-} ]; then docker rmi -f ${IMAGE_URL} &> /dev/null || /bin/true fi if [ -n ${MOUNTER_DEFAULT_NAME:-} ]; then docker rm -f ${MOUNTER_DEFAULT_NAME} &> /dev/null || /bin/true fi } trap flex_clean ERR # Generates a bash script that wraps all calls to the actual driver inside mount utilities # in the chroot environment. Kubelet sees this script as the FlexVolume driver. generate_chroot_wrapper() { if [ ! -d ${MOUNTER_PATH}/flexvolume ]; then echo "Failed to set up FlexVolume driver: cannot find directory '/flexvolume' in the mount utility image." exit 1 fi for driver_dir in ${MOUNTER_PATH}/flexvolume/*; do if [ -d "$driver_dir" ]; then filecount=$(ls -1 $driver_dir | wc -l) if [ $filecount -gt 1 ]; then echo "ERROR: Expected 1 file in the FlexVolume directory but found $filecount." exit 1 fi driver_file=$( ls $driver_dir | head -n 1 ) # driver_path points to the actual driver inside the mount utility image, # relative to image root. # wrapper_path is the wrapper script location, which is known to kubelet. driver_path=flexvolume/$( basename $driver_dir )/${driver_file} wrapper_dir=${VOLUME_PLUGIN_DIR}/$( basename $driver_dir ) wrapper_path=${wrapper_dir}/${driver_file} mkdir -p $wrapper_dir cat >$wrapper_path <<EOF #!/bin/bash chroot ${MOUNTER_PATH} ${driver_path} "$@" EOF chmod 755 $wrapper_path echo "FlexVolume driver installed at ${wrapper_path}" fi done } echo echo "Importing mount utility image from Container Registry..." echo METADATA=http://metadata.google.internal/computeMetadata/v1 SVC_ACCT_ENDPOINT=$METADATA/instance/service-accounts/default ACCESS_TOKEN=$(curl -s -H 'Metadata-Flavor: Google' $SVC_ACCT_ENDPOINT/token | cut -d'"' -f 4) PROJECT_ID=$(curl -s -H 'Metadata-Flavor: Google' $METADATA/project/project-id) IMAGE_URL=gcr.io/${PROJECT_ID}/${MOUNTER_IMAGE} MOUNTER_DEFAULT_NAME=flexvolume_mounter sudo -u ${SUDO_USER} docker login -u _token -p $ACCESS_TOKEN https://gcr.io > /dev/null sudo -u ${SUDO_USER} docker run --name=${MOUNTER_DEFAULT_NAME} ${IMAGE_URL} docker export ${MOUNTER_DEFAULT_NAME} > /tmp/${MOUNTER_DEFAULT_NAME}.tar docker rm ${MOUNTER_DEFAULT_NAME} > /dev/null docker rmi ${IMAGE_URL} > /dev/null echo echo "Loading mount utilities onto this instance..." echo mkdir ${MOUNTER_PATH} tar xf /tmp/${MOUNTER_DEFAULT_NAME}.tar -C ${MOUNTER_PATH} # Bind the kubelet directory to one under flexvolume_mounter mkdir ${MOUNTER_PATH}/var/lib/kubelet mount --rbind /var/lib/kubelet/ ${MOUNTER_PATH}/var/lib/kubelet mount --make-rshared ${MOUNTER_PATH}/var/lib/kubelet # Remount the flexvolume_mounter environment with /dev enabled. mount --bind ${MOUNTER_PATH} ${MOUNTER_PATH} mount -o remount,dev,exec ${MOUNTER_PATH} echo echo "Setting up FlexVolume driver..." echo mkdir -p ${VOLUME_PLUGIN_DIR} mount --bind ${VOLUME_PLUGIN_DIR} ${VOLUME_PLUGIN_DIR} mount -o remount,exec ${VOLUME_PLUGIN_DIR} generate_chroot_wrapper echo echo "Restarting Kubelet..." echo systemctl restart kubelet.service kubelet_wait if [ $? -eq 0 ]; then echo echo "FlexVolume is ready." else echo "ERROR: Timed out after 1 minute waiting for kubelet restart." fi </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Today there are many obstacles preventing users from setting up FlexVolume on a GCE COS image easily. The default FlexVolume driver directory is not writable. Custom mount utilities can't be installed. Solutions exist for these problems but they are time-consuming to execute manually. On top of these, there are also the common FlexVolume pain points such as manual driver and mount utilities installation, and manually restarting kubelet. Ideally all these tasks should be automated, making it a quick and simple experience to have custom out-of-tree volume plugins. Is this a BUG REPORT or FEATURE REQUEST?: FEATURE REQUEST /kind feature
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig storage
/assign


Question: ContainerID: emptyContainerID, }}, }, <del> map[string]api.ContainerState{ </del> <ins> expectedLastTerminationState: map[string]api.ContainerState{ </ins> "without-reason": {Terminated: &api.ContainerStateTerminated{ ExitCode:    3, ContainerID: emptyContainerID,

is this a Node bug? cc as next oncall to follow up on this.
This is a node bug due to init containers
I may need help debugging - I see:
So I think the problem is in the error case - we're not properly recording that the start failed, and the next time we come in we somehow think the first container succeeded. I'll add some debugging to see if we can't recreate it.
Let's discuss on instead.


Question: ContainerID: emptyContainerID, }}, }, <del> map[string]api.ContainerState{ </del> <ins> expectedLastTerminationState: map[string]api.ContainerState{ </ins> "without-reason": {Terminated: &api.ContainerStateTerminated{ ExitCode:    3, ContainerID: emptyContainerID,

Kubelet issues a docker inspect call after creating a container, but for very short lived containers there is a window in which they could exit before being inspected in the docker manager. If the container PID is 0, we should check the ExitCode and other relevant fields and handle them appropriately. [1] This is the suspected cause of several flakes: /cc
looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: I'd expect the second one to not have run in this case? Node logs:
Init containers have an implicit assumption that container statuses include results of previous SyncPod invocations. If that's not the case we may need to fix that (or at least mitigate it). On May 18, 2016, at 8:28 PM, Prashanth B wrote: looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: May 18 15:58:33.380: INFO: pod-init--1d4b-11e6-80a4- started at 2016-05-18T15:58:04-07:00 (2+1 container statuses recorded) May 18 15:58:33.380: INFO: Init container init1 ready: false, restart count 0 May 18 15:58:33.380: INFO: Init container init2 ready: true, restart count 0 I'd expect the second one to not have run in this case? Node logs: — You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub
This should be true. Is there any evidence that suggests the opposite? only shows conflicting view of whether init1 has run.
If the container exits already and it's not an infra container, kubelet should not error out. Although even if it does, I failed to see how this could affect the readiness of the init container without checking the code further.


Question: ContainerID: emptyContainerID, }}, }, <del> map[string]api.ContainerState{ </del> <ins> expectedLastTerminationState: map[string]api.ContainerState{ </ins> "without-reason": {Terminated: &api.ContainerStateTerminated{ ExitCode:    3, ContainerID: emptyContainerID,

Each syncPod invocation, we inspect the list of init container in order looking for lastTerminationState. There could be a bug in that logic, or if we didn't get the full status, we might make a bad decision. When determining which container to run next, we always look for the last init container in the chain that has success, then run the next container (unless the pod is RestartNever). , Yu-Ju Hong wrote:
Saw this flake again. How do we verify that container status is not lost between syn pods?
... between calls to syncPods
The second sync clearly saw the init1 container, and decided to launch init2. See "InitContainersToKeep" in the message below.
containers to keep can include failed containers. , Yu-Ju Hong wrote:
Discussed on slack. In summary, the problem has to do with how we convert the kubecontainer.PodStatus to the api.PodStatus. If a container is terminated, the function makes the decisions: the pod restart policy is "Always", and there is a "failure reason" recorded for the given container, this means the pod worker tried to (re)start the container but failed. Set the state to "Waiting" and populate the reason. , the container state is simply "terminated". Init containers don't obey the restart policy like regular containers, and the logic will need to be amended to work with them. is working on a PR. In addition to that, kubelet should not error out if a user container exited before it could apply the oom score (which is best effort in this case). fixes that.
Ahh, thanks for digging into the issue. This is the exact the concern I had when I reviewed the initial implementation of init container, which blocks pr for a while. I even asked to help me verify this case, unfortunately we didn't reproduce the issue back then. :-(
I wish I'd seen the flakes in the runs prior to merge.
We didn't notice this until we started running the tests against docker v1.11.1, which is much faster and hit the race condition differently.
I suppose that's a good thing...
It is hard to reproduce this.


Question: ContainerID: emptyContainerID, }}, }, <del> map[string]api.ContainerState{ </del> <ins> expectedLastTerminationState: map[string]api.ContainerState{ </ins> "without-reason": {Terminated: &api.ContainerStateTerminated{ ExitCode:    3, ContainerID: emptyContainerID,

The extra check of last terminated container in your initial code makes me nervous, but we couldn't reproduce it due to that small race windows until we have a faster docker running on the node. :-)
I got "Pods should invoke init containers on a RestartAlways pod" flake (referenced from ): Logs:
All the fixes went in. Please reopen this issue if you see any more flakes.


Question: return err } <del> err = os.Chmod(f.Name(), 0755) </del> <ins> err = os.Chmod(f.Name(), 0660) </ins> if err != nil { return err }

In kubectl v1.8.0+, schema info is cached in the location specified by (defaulting to ), written with world-writeable permissions (rw-rw-rw-). If is specified and pointed at a different location accessible to other users/groups, the written files may be modified by other users/groups and disrupt the kubectl invocation. CVSS score: CVSS:3.0/AV:L/AC:H/PR:L/UI:R/S:U/C:L/I:L/A:N (3.3, low) What versions are affected? kubectl v1.8.0+ What configurations are affected? Invocations that point at world-writeable locations Impact Malformed responses written to the cache directory can disrupt the kubectl invocation Workaround Use the default location in the $HOME directory or point it at a directory that is only accessible to desired users/groups. (original description follows) ==== What happened: The files inside of ".kube/http-cache" are world writeable (rw-rw-rw-). While the default for these files appears to be the home directory, using the "--cache-dir" flag could put these files into a place where world writeable files would allow any user / process to modify the cache files. Modification of the cache files could influence the kubectl utility in a negative way for other users. What you expected to happen: Apply stricter file permissions to the http-cache files. How to reproduce it (as minimally and precisely as possible): Run any generic kubectl command which is successful and then list the cache directory $ kubectl get pods --all-namespaces $ ls -la Anything else we need to know?


Question: return err } <del> err = os.Chmod(f.Name(), 0755) </del> <ins> err = os.Chmod(f.Name(), 0660) </ins> if err != nil { return err }

: I estimate this is a low severity security issue with a CVSS score of "3.3 / CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:L/A:N" - Environment: Linux Kubernetes version (use kubectl version):Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS. Running kube api server in hyperkube. OS (e.g: cat /etc/os-release): NAME="CentOS Linux" VERSION="7.1808 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="7" PRETTYNAME="CentOS Linux 7.1808 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:7" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-7" CENTOSMANTISBTPROJECTVERSION="7" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="7" OSTREEVERSION=7.1808 Kernel (e.g. uname -a): Linux hackit.internal 3.10.0-862.11.6.el7.x8664 SMP Tue Aug 14 21:49:04 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Manual installation. Others: n/a
/sig product-security
Closing this issue. <detailsIn response to : if you think this is serious, send an email thanks!


Question: return err } <del> err = os.Chmod(f.Name(), 0755) </del> <ins> err = os.Chmod(f.Name(), 0660) </ins> if err != nil { return err }

</details
Adjusted the CVSS score to to indicate user interaction is required (since they must specify a non-default cache option pointed at a shared writeable location)
There are two places this should be done: PathPerm and FilePerm should be set here: the temp file and final file should have stricter permissions (probably 0660) here:
/cc
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9.
GitHub didn't allow me to assign the following users: yuchengwu. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
help fixing this is welcome, though it would only be fixed in releases 1.12+, per
/label official-cve-feed (Related to kubernetes/sig-security)


Question: return "default", false, nil } <del> func (inClusterClientConfig) ConfigAccess() ConfigAccess { </del> <ins> func (config *inClusterClientConfig) ConfigAccess() ConfigAccess { </ins> return NewDefaultClientConfigLoadingRules() } // Possible returns true if loading an inside-kubernetes-cluster is possible. <del> func (inClusterClientConfig) Possible() bool { </del> <ins> func (config *inClusterClientConfig) Possible() bool { </ins> fi, err := os.Stat("/var/run/secrets/kubernetes.io/serviceaccount/token") return os.Getenv("KUBERNETES_SERVICE_HOST") != "" && os.Getenv("KUBERNETES_SERVICE_PORT") != "" &&

Found in openshift, but I'm pretty sure the codepaths are the same. That means the token is getting ignored and in-cluster config is being used in preference to flags.
Is a similar issue?
Yep, kind of regression.
Sigh.
Yes, most likely the same cause.
I'm on it.
Unit test :)
Probably will want to cherrypick this into 1.5.
WIP in


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/admission" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api/errors" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api/latest" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/meta" "github.com/GoogleCloudPlatform/kubernetes/pkg/client"

While running one of my soak tests I noticed the error message: I don't see why I should get this error message. Why should it matter if the namespace already exists? It was created as part of the service creation call just beforehand. And the second attempt to create this pod in exactly the same namespace works. Is this a bug?
Running a second time did not produce this error.
I agree it should not matter. The error is produced there, and I suspect there were multiple concurrent calls being made that try to create a resource in a namespace that had not yet existed. We need to update the code in question to swallow the error if the namespace already exists.
FYI - longer term, I would like to eliminate the use of autoprovision of namespace and require you to first create a namespace manually. what is there now is more to preserve existing behavior prior to namespace as object was introduced.


Question: function upload-resources() { swift post kubernetes --read-acl '.r:*,.rlistings' <del> echo "[INFO] Upload ${KUBERNETES_RELEASE_TAR}" swift upload kubernetes ${ROOT}/../../_output/release-tars/${KUBERNETES_RELEASE_TAR}  </del> <ins> locations=( "${ROOT}/../../_output/release-tars/${KUBERNETES_RELEASE_TAR}" "${ROOT}/../../server/${KUBERNETES_RELEASE_TAR}" ) RELEASE_TAR_LOCATION=$( (ls -t "${locations[@]}" 2>/dev/null || true) | head -1 ) RELEASE_TAR_PATH=$(dirname ${RELEASE_TAR_LOCATION}) echo "[INFO] Uploading ${KUBERNETES_RELEASE_TAR}" swift upload kubernetes ${RELEASE_TAR_PATH}/${KUBERNETES_RELEASE_TAR}  </ins> --object-name kubernetes-server.tar.gz <del> echo "[INFO] Upload kubernetes-salt.tar.gz" swift upload kubernetes ${ROOT}/../../_output/release-tars/kubernetes-salt.tar.gz  </del> <ins> echo "[INFO] Uploading kubernetes-salt.tar.gz" swift upload kubernetes ${RELEASE_TAR_PATH}/kubernetes-salt.tar.gz  </ins> --object-name kubernetes-salt.tar.gz }

The path to the server tarball resolves to: is hardcoded, and does not exist in release builds (v1.3.0-beta.2) The file in question is located at: Correcting the path is a workaround for release builds
cc
Hey, sorry about that. I think I got a fix, but I need your help to test it. I'm traveling and my cellular upload speeds are not so great. Please take a look at this and let me know if it solves the issue:


Question: Import-Module -Force C:$Filename } <ins> # Returns true if the ENABLE_STACKDRIVER_WINDOWS or ENABLE_NODE_LOGGING field in kube_env is true. # $KubeEnv is a hash table containing the kube-env metadata keys+values. # ENABLE_NODE_LOGGING is used for legacy Stackdriver Logging, and will be deprecated (always set to False) # soon. ENABLE_STACKDRIVER_WINDOWS is added to indicate whether logging is enabled for windows nodes. function IsLoggingEnabled { param ( [parameter(Mandatory=$true)] [hashtable]$KubeEnv ) if ($KubeEnv.Contains('ENABLE_STACKDRIVER_WINDOWS') -and ` ($KubeEnv['ENABLE_STACKDRIVER_WINDOWS'] -eq 'true')) { return $true } elseif ($KubeEnv.Contains('ENABLE_NODE_LOGGING') -and ` ($KubeEnv['ENABLE_NODE_LOGGING'] -eq 'true')) { return $true } return $false } </ins> try { # Don't use FetchAndImport-ModuleFromMetadata for common.psm1 - the common # module includes variables and functions that any other function may depend

What happened: The stackdriver logging agent is installed and start running no matter logging is enabled/disabled in the cluster or not. What you expected to happen: Do not install and start the logging agent if logging is disabled.
/sig windows
cc


Question: "-A", string(kubeServicesChain), "-m", "comment", "--comment", proxier.ipsetList[kubeClusterIPSet].getComment(), "-m", "set", "--match-set", kubeClusterIPSet, <del> "dst,dst", </del> ) if proxier.masqueradeAll { <del> writeLine(proxier.natRules, append(args, "-j", string(KubeMarkMasqChain))...) </del> <ins> writeLine(proxier.natRules, append(args, "dst,dst", "-j", string(KubeMarkMasqChain))...) </ins> } else if len(proxier.clusterCIDR) > 0 { // This masquerades off-cluster traffic to a service VIP.  The idea // is that you can establish a static route for your Service range, // routing to any node, and that node will bridge into the Service // for you.  Since that might bounce off-node, we masquerade here. // If/when we support "Local" policy for VIPs, we should update this. <del> writeLine(proxier.natRules, append(args, "! -s", proxier.clusterCIDR, "-j", string(KubeMarkMasqChain))...) </del> <ins> writeLine(proxier.natRules, append(args, "dst,dst", "! -s", proxier.clusterCIDR, "-j", string(KubeMarkMasqChain))...) } else { // Masquerade all OUTPUT traffic coming from a service ip. // The kube dummy interface has all service VIPs assigned which // results in the service VIP being picked as the source IP to reach // a VIP. This leads to a connection from VIP:<random port> to // VIP:<service port>. // Always masquerading OUTPUT (node-originating) traffic with a VIP // source ip and service port destination fixes the outgoing connections. writeLine(proxier.natRules, append(args, "src,dst", "-j", string(KubeMarkMasqChain))...) </ins> } }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When using IPVS and not natting (no cluster-cidr specified) requests from the host to service IPs are send with the service ip as the source ip. Let be the kubernetes API IP. Kubeproxy will set up with . This means that traffic from the host (which sees the ) will originate from (best matching ip/network). What you expected to happen: The source ip should be natted to the host ip. How to reproduce it (as minimally and precisely as possible): a ipvs based cluster any cluster-cidr from kube-proxy traffic to a service ip (e.g. http to the kubernetes api server) from the node / host network pod the outgoing interface looking for the service ip () Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: baremetal OS (e.g. from /etc/os-release): Kernel (e.g. ): (based on the Ubuntu 18.04 kernel) Install tools: custom repackaging of kubernetes release tarballs Others: /sig network
We have a similar issue in kube-router FWIW resolution is included in the issue
Ah nice, I reproduced the source IP selection on my local Ubuntu 18.04 without any kubernetes installed (use ip link add / ip addr add and then ping the new dummy IP / tcpdump lo). Might be related to the kernel version (4.15) as you couldn't reproduce it.... Changing the routing table would avoid an extra NAT rule. That sounds interesting but not doable without changing the kubernetes sources Murali Reddy < am Fr., 27. Apr. 2018, 14:06:
let me try with to reproduce. but its no brainer, this issue must be fixed.
getting closer... -CLUSTER-IP is only populated if masquerade-all or cluster-cidr is used: is no OUTPUT masquerading for service IPs if cluster-cidr is not specified (no branch matches): With cluster-cidr enabled I've saved the ipset.... I've disabled cluster-cidr and rebooted (to reset all iptables/ipset/...).


Question: "-A", string(kubeServicesChain), "-m", "comment", "--comment", proxier.ipsetList[kubeClusterIPSet].getComment(), "-m", "set", "--match-set", kubeClusterIPSet, <del> "dst,dst", </del> ) if proxier.masqueradeAll { <del> writeLine(proxier.natRules, append(args, "-j", string(KubeMarkMasqChain))...) </del> <ins> writeLine(proxier.natRules, append(args, "dst,dst", "-j", string(KubeMarkMasqChain))...) </ins> } else if len(proxier.clusterCIDR) > 0 { // This masquerades off-cluster traffic to a service VIP.  The idea // is that you can establish a static route for your Service range, // routing to any node, and that node will bridge into the Service // for you.  Since that might bounce off-node, we masquerade here. // If/when we support "Local" policy for VIPs, we should update this. <del> writeLine(proxier.natRules, append(args, "! -s", proxier.clusterCIDR, "-j", string(KubeMarkMasqChain))...) </del> <ins> writeLine(proxier.natRules, append(args, "dst,dst", "! -s", proxier.clusterCIDR, "-j", string(KubeMarkMasqChain))...) } else { // Masquerade all OUTPUT traffic coming from a service ip. // The kube dummy interface has all service VIPs assigned which // results in the service VIP being picked as the source IP to reach // a VIP. This leads to a connection from VIP:<random port> to // VIP:<service port>. // Always masquerading OUTPUT (node-originating) traffic with a VIP // source ip and service port destination fixes the outgoing connections. writeLine(proxier.natRules, append(args, "src,dst", "-j", string(KubeMarkMasqChain))...) </ins> } }

I think the code should: Always populate Set a masquerade rule for service IPs (src == service IP should not happen on OUTPUT)
How to test this with a local cluster.... the node (install , & modprobe the modules , , , ). a new shell: define a service with manual external endpoints tcpdump on the outgoing interface of the node the service ip the tcpdump, it will show outgoing traffic from_ the service IP:
Can you please take a look at this?
/assign
I assume this issue is related to some specific kernel versions? I just neither can't reproduce it nor can't see the failures in our test grid.
I was aware of there is a PR() in flight, my team will evaluate the effects of that PR. But, please rebase the PR first, thanks!


Question: VolumeNFSServer     = ImageConfig{e2eRegistry, "volume/nfs", "1.0"} VolumeISCSIServer   = ImageConfig{e2eRegistry, "volume/iscsi", "1.0"} VolumeGlusterServer = ImageConfig{e2eRegistry, "volume/gluster", "1.0"} <del> VolumeRBDServer     = ImageConfig{e2eRegistry, "volume/rbd", "1.0"} </del> <ins> VolumeRBDServer     = ImageConfig{e2eRegistry, "volume/rbd", "1.0.1"} </ins> ) func GetE2EImage(image ImageConfig) string {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e tests which use RBD server image fail with What you expected to happen: These tests succeed. How to reproduce it (as minimally and precisely as possible): Run e2e tests which use RBD server image. Anything else we need to know?: /sig storage This issue is introduced by below commit: It change the base image from fedora 26 to fedora 28. In fedora 26, hostname command is installed as a dependency for  However, in fedora 28, it is not installed, as a result, this issue happens due to lack of hostname command. Environment: Kubernetes version (use ): k8s 1.12 alpha
Please see the difference of the behaviour between fedora 26 and 28. [Fedora26] [Fedora28]
Could you help solving this issue? I assume the fix for the code would be as below, but I'm not familiar with the build and push process for the test image. ( in doesn't work.) So, I need your help to make further progress.
(or) Both need to be run from directory
Thank you for letting me know it. It should be s/all-containers/all-container/, but it seems to work. I will test with above patch and above build commands, then make a PR, once I confirm that it works well.
sorry typing from memory :)
I need to ask you two more things: Do we need to increment the version for this image, like 1.0.1, even for this small change? (I guess that and test/images/volume/rbd/VERSION will need to be updated in this case.) How the image push process will be? Does it pushed by CI, if there are a change in the code or docker file, or do someone who have permission will push it manually?
creates below three docker images: Is my understanding right that to make each test access to each arch image, needs to be modified like below? (Without doing so, each test seems to keep trying to access to , without arch.)
For fedora28 image, just installing hostname doesn't seem to work well. Please see the log below and attached . I guess that this is due to the difference of ceph version, which will be beyond my beyond my expertise.


Question: VolumeNFSServer     = ImageConfig{e2eRegistry, "volume/nfs", "1.0"} VolumeISCSIServer   = ImageConfig{e2eRegistry, "volume/iscsi", "1.0"} VolumeGlusterServer = ImageConfig{e2eRegistry, "volume/gluster", "1.0"} <del> VolumeRBDServer     = ImageConfig{e2eRegistry, "volume/rbd", "1.0"} </del> <ins> VolumeRBDServer     = ImageConfig{e2eRegistry, "volume/rbd", "1.0.1"} </ins> ) func GetE2EImage(image ImageConfig) string {

So, could you help debugging this issue? Meanwhile, I will try testing with the image by using fedora26 for workaround.
I confirmed that below commit, which change the base image back to fedora 26, fix this issue. Please see the log . After I checked the image-, I come to think that by pushing the image, each image will be able to be pulled without -arch, so I guess the last parameter for ImageConfig should be false. (I couldn't test it in my environment.) I will fix the commit as so and make a PR once I confirm it. Also, if it will easy to fix with Fedora 28, I will wait for the actual fix and quit making a PR for the workaround.
thanks yes, a quick fix is welcome, looking forward
Thank you for your comment. I will continue with the workaround. I saw a commit for deleting multi arch consideration. I rebased the patch to the commit and made a PR. Please review it.


Question: importpath = "k8s.io/component-base/metrics/testutil", visibility = ["//visibility:public"], deps = [ <ins> "//staging/src/k8s.io/apimachinery/pkg/version:go_default_library", </ins> "//staging/src/k8s.io/component-base/metrics:go_default_library", "//vendor/github.com/prometheus/client_golang/prometheus/testutil:go_default_library", "//vendor/github.com/prometheus/common/expfmt:go_default_library",

What would you like to be : Once a metric be deprecated, we should update the unit test accordingly if there is any. We need to work out a way for a unit test freely select codebase version, and also should make sure the unit test won't fail in the subsequent release. Why is this needed: Deprecated also needs to be protected. /sig instrumentation /assign


Question: // GetInstanceTypeByNodeName gets the instance type by node name. func (ss *scaleSet) GetInstanceTypeByNodeName(name string) (string, error) { <del> _, _, vm, err := ss.getVmssVM(name) </del> <ins> managedByAS, err := ss.isNodeManagedByAvailabilitySet(name) </ins> if err != nil { <del> if err == ErrorNotVmssInstance { glog.V(4).Infof("GetInstanceTypeByNodeName: node %q is managed by availability set", name) // Retry with standard type because nodes are not managed by vmss. return ss.availabilitySet.GetInstanceTypeByNodeName(name) } </del> <ins> glog.Errorf("Failed to check isNodeManagedByAvailabilitySet: %v", err) return "", err } if managedByAS { // vm is managed by availability set. return ss.availabilitySet.GetInstanceTypeByNodeName(name) } _, _, vm, err := ss.getVmssVM(name) if err != nil { </ins> return "", err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: VMAS nodes are reporting error in vmss cluster (vmType set to vmss). What you expected to happen: Node should register correctly in the kubernetes cluster. How to reproduce it (as minimally and precisely as possible): Create a VMAS nodes inside vmss cluster. Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure


Question: # Optional: Enable node logging. ENABLE_NODE_LOGGING=true LOGGING_DESTINATION=elasticsearch # options: elasticsearch, gcp <ins> # Don't require https for registries in our local RFC1918 network EXTRA_DOCKER_OPTS="--insecure-registry 10.0.0.0/8" </ins>

0/8 is with a company's intranet, so it can be insecure. This is similiar to "For GCE, allow insecure registries anywhere in 10.0.0.0/8" Thanks!
Linking to and .
This was


Question: <modelVersion>4.0.0</modelVersion> <groupId>io.k8s.cassandra</groupId> <artifactId>kubernetes-cassandra</artifactId> <del> <version>0.0.4</version> </del> <ins> <version>0.0.5</version> </ins> <build> <sourceDirectory>src</sourceDirectory> <plugins>

What would you like to be : There is a tab in the k8s perf dashaboard for the scheduler perf benchmarks, but it is showing nothing, we should fix that: http://perf- What this dashboard should be showing is multiple runs per day of We can run it every hour, and summarize the 24 runs of each day to show 50%, 90% and 99% percentile. Why is this needed: Having daily runs of the benchmarks is important to track performance regressions. /sig scheduling /assign /priority important-soon
Ref
/cc
The ci job's config is at And one of failed logs could be found at I tried to reproduce it on my local machine(because I do not have Gloud credentials, I run the task directly in the container): However it works well on my local machine. It is so weird. It will be very appreciated if anyone could help reproduce it with Gloud credentials.
can you try and reach out to the test-infra people? ping them on the slack channel perhaps.
yes, I have tried to get help from , he is busy now. I'll try more.
While working on reproducing this in order to file a bug against test-infra folks I think I found a solution. Will send PRs shortly. /assign
/retitle sig-scalability-benchmarks#scheduler fails consistently due to build issues
/sig scalability Testgrid link: https://k8s- Error logs:
A word about the solution, the job uses the deprecated script which is not supported anymore by test-infra oncall / folks. I've rewritten the job definition to use pod utilities and it started working (will require to get merged first).
Thanks
Thanks !


Question: # REVISION provides a version number fo this image and all it's bundled # artifacts. It should start at zero for each LATEST_ETCD_VERSION and increment # for each revision of this image at that etcd version. <del> REVISION?=2 </del> <ins> REVISION?=3 </ins> # IMAGE_TAG Uniquely identifies k8s.gcr.io/etcd docker image with a tag of the form "<etcd-version>-<revision>". IMAGE_TAG=$(LATEST_ETCD_VERSION)-$(REVISION)

The based container images for are shipped with the wrong architecture within the manifest if not amd64: Refers to We updated the images a few minutes ago to buster-v1.3.0, which are now ready to be used as base for the multi-architecture images. If the images contain the right architectures, then it may magically happen that the kube-proxy images are fixed, too. This is because we already use the build syntax for it: /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release


Question: import ( "context" <ins> "fmt" </ins> "testing" "time"

TestRunOneVolumeDetachFailNodeWithReadWriteOnce 4 times since 03-19 23:00 PDT, intermittently ! https://k8s- Still investigating whether this is a tolerable flake or not with more history than testgrid view, or otherwise would appreciate input from anyone with expertise in this test, especially as regards to whether this should be a blocker for tomorrows beta cut. maybe: /sig storage /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is already open for this, looks like it is attempting to fix it
thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. /close
Closing this issue. <detailsIn response to : thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "context" <ins> "fmt" </ins> "testing" "time"

pull-kubernetes-unit TestRunOneVolumeDetachFailNodeWithReadWriteOnce Jan 7th, 2022 Check volume <fake-plugin/volume-nameis reported as attached to node <fail-detach-node, got false, expected true No response_ /sig storage
/cc
this is the top unit test flake according to
cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
I think this test is no longer flaky by checking the test grid, close it now.
Unfortunately this does still seem to be flaky: Here was the output from : /reopen
Reopened this issue. <detailsIn response to : Unfortunately this does still seem to be flaky: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
still the case
How about adding some logging to help troubleshoot this? I looked at it a little, but it is hard to tell which part of the test is failing. PR
/
This is still happening
/triage accepted
From the logs, in the OneVolumeDetachFailNodeWithReadWriteOnce test, succeeded, but the next verification failed. In verifyVolumeAttachedToNode, the attachState is from calling and within asw, it calls . In verifyVolumeReportedAsAttachedToNode, the volumes are obtained from calling , it seems that the node is not in asw.nodesToUpdateStatusFor at this step. Cannot reproduce locally. I tried to run the test 100 times locally with debug.Stack()... 3 * 100 times all passed.
Tried to run stress test on my linux machine as well but still cannot reproduce. Command: Result: Also tried running n process in parallel. Command: Result:
It seems like it happens more frequently.


Question: import ( "context" <ins> "fmt" </ins> "testing" "time"

But I may have no time to dig into the cause. Let's see.
cc as the test author.
The problem is that this test has a lot of dependency on the sleeps that are hardcoded with a very narrow scope, but our CI seems to be able to stress it much more. As soon as you slightly change one of the sleep duration it starts to flake locally The test can not make an assumption that timing is perfect, even more in the sub second range can someone in sig-storage review this test and make it less dependant on those tight sleeps?
According to most failures, it failed at Line 660. Let me run some stress locally. with 250ms sleep with 100ms sleep
I agree this test case is too brittle due to its specific timing dependencies. To fix this, I suggest reworking the test case to: 1) Add retry with timeout to wait for final conditions 2) Remove the intermediate timing-based state transition checks. Instead, we can keep track of state transitions in the Fakes and validate those stats. wdyt?
Friendly ping from 1.27 CI Signal that this test is still a noisy flake on the master-blocking dashboard as CI Signal has flagged it in and


Question: host_platform=$(kube::golang::host_platform) # Use eval to preserve embedded quoted strings. <del> local goflags goldflags gogcflags build_with_coverage </del> <ins> local goflags goldflags gogcflags </ins> eval "goflags=(${GOFLAGS:-})" goldflags="${GOLDFLAGS:-} $(kube::version::ldflags)" gogcflags="${GOGCFLAGS:-}" <del> build_with_coverage="${KUBE_BUILD_WITH_COVERAGE:-}" </del> local -a targets=() local arg

/kind cleanup /sig testing /priority important-longterm This is a followup issue to address the "standard build globals" nit on At the very least I think we shouldn't capture our custom env var. Also think it'd be worth documenting the assumed variables in a list, similar to
cc
/assign


Question: } } <ins> // This function is used canary updates and phased rolling updates of template modifications for partiton1 and delete pod-0 func deletingPodForRollingUpdatePartitionTest(ctx context.Context, f *framework.Framework, c clientset.Interface, ns string, ss *appsv1.StatefulSet) { setHTTPProbe(ss) ss.Spec.UpdateStrategy = appsv1.StatefulSetUpdateStrategy{ Type: appsv1.RollingUpdateStatefulSetStrategyType, RollingUpdate: func() *appsv1.RollingUpdateStatefulSetStrategy { return &appsv1.RollingUpdateStatefulSetStrategy{ Partition: pointer.Int32(1), } }(), } ss, err := c.AppsV1().StatefulSets(ns).Create(ctx, ss, metav1.CreateOptions{}) framework.ExpectNoError(err) e2estatefulset.WaitForRunningAndReady(ctx, c, *ss.Spec.Replicas, ss) ss = waitForStatus(ctx, c, ss) currentRevision, updateRevision := ss.Status.CurrentRevision, ss.Status.UpdateRevision gomega.Expect(currentRevision).To(gomega.Equal(updateRevision), fmt.Sprintf("StatefulSet %s/%s created with update revision %s not equal to current revision %s", ss.Namespace, ss.Name, updateRevision, currentRevision)) pods := e2estatefulset.GetPodList(ctx, c, ss) for i := range pods.Items { gomega.Expect(pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel]).To(gomega.Equal(currentRevision), fmt.Sprintf("Pod %s/%s revision %s is not equal to currentRevision %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel], currentRevision)) } ginkgo.By("Adding finalizer for pod-0") pod0name := getStatefulSetPodNameAtIndex(0, ss) pod0, err := c.CoreV1().Pods(ns).Get(ctx, pod0name, metav1.GetOptions{}) framework.ExpectNoError(err) pod0.Finalizers = append(pod0.Finalizers, testFinalizer) pod0, err = c.CoreV1().Pods(ss.Namespace).Update(ctx, pod0, metav1.UpdateOptions{}) framework.ExpectNoError(err) pods.Items[0] = *pod0 defer e2epod.NewPodClient(f).RemoveFinalizer(ctx, pod0.Name, testFinalizer) ginkgo.By("Updating image on StatefulSet") newImage := NewWebserverImage oldImage := ss.Spec.Template.Spec.Containers[0].Image ginkgo.By(fmt.Sprintf("Updating stateful set template: update image from %s to %s", oldImage, newImage)) gomega.Expect(oldImage).ToNot(gomega.Equal(newImage), "Incorrect test setup: should update to a different image") ss, err = updateStatefulSetWithRetries(ctx, c, ns, ss.Name, func(update *appsv1.StatefulSet) { update.Spec.Template.Spec.Containers[0].Image = newImage }) framework.ExpectNoError(err) ginkgo.By("Creating a new revision") ss = waitForStatus(ctx, c, ss) currentRevision, updateRevision = ss.Status.CurrentRevision, ss.Status.UpdateRevision gomega.Expect(currentRevision).ToNot(gomega.Equal(updateRevision), "Current revision should not equal update revision during rolling update") ginkgo.By("Await for all replicas running, all are updated but pod-0") e2estatefulset.WaitForState(ctx, c, ss, func(set2 *appsv1.StatefulSet, pods2 *v1.PodList) (bool, error) { ss = set2 pods = pods2 if ss.Status.UpdatedReplicas == *ss.Spec.Replicas-1 && ss.Status.Replicas == *ss.Spec.Replicas && ss.Status.ReadyReplicas == *ss.Spec.Replicas { // rolling updated is not completed, because replica 0 isn't ready return true, nil } return false, nil }) ginkgo.By("Verify pod images before pod-0 deletion and recreation") for i := range pods.Items { if i < int(*ss.Spec.UpdateStrategy.RollingUpdate.Partition) { gomega.Expect(pods.Items[i].Spec.Containers[0].Image).To(gomega.Equal(oldImage), fmt.Sprintf("Pod %s/%s has image %s not equal to oldimage image %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Spec.Containers[0].Image, oldImage)) gomega.Expect(pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel]).To(gomega.Equal(currentRevision), fmt.Sprintf("Pod %s/%s has revision %s not equal to current revision %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel], currentRevision)) } else { gomega.Expect(pods.Items[i].Spec.Containers[0].Image).To(gomega.Equal(newImage), fmt.Sprintf("Pod %s/%s has image %s not equal to new image  %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Spec.Containers[0].Image, newImage)) gomega.Expect(pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel]).To(gomega.Equal(updateRevision), fmt.Sprintf("Pod %s/%s has revision %s not equal to new revision %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel], updateRevision)) } } ginkgo.By("Deleting the pod-0 so that kubelet terminates it and StatefulSet controller recreates it") deleteStatefulPodAtIndex(ctx, c, 0, ss) ginkgo.By("Await for two replicas to be updated, while the pod-0 is not running") e2estatefulset.WaitForState(ctx, c, ss, func(set2 *appsv1.StatefulSet, pods2 *v1.PodList) (bool, error) { ss = set2 pods = pods2 return ss.Status.ReadyReplicas == *ss.Spec.Replicas-1, nil }) ginkgo.By(fmt.Sprintf("Removing finalizer from pod-0 (%v/%v) to allow recreation", pod0.Namespace, pod0.Name)) e2epod.NewPodClient(f).RemoveFinalizer(ctx, pod0.Name, testFinalizer) ginkgo.By("Await for recreation of pod-0, so that all replicas are running") e2estatefulset.WaitForState(ctx, c, ss, func(set2 *appsv1.StatefulSet, pods2 *v1.PodList) (bool, error) { ss = set2 pods = pods2 return ss.Status.ReadyReplicas == *ss.Spec.Replicas, nil }) ginkgo.By("Verify pod images after pod-0 deletion and recreation") pods = e2estatefulset.GetPodList(ctx, c, ss) for i := range pods.Items { if i < int(*ss.Spec.UpdateStrategy.RollingUpdate.Partition) { gomega.Expect(pods.Items[i].Spec.Containers[0].Image).To(gomega.Equal(oldImage), fmt.Sprintf("Pod %s/%s has image %s not equal to current image %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Spec.Containers[0].Image, oldImage)) gomega.Expect(pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel]).To(gomega.Equal(currentRevision), fmt.Sprintf("Pod %s/%s has revision %s not equal to current revision %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel], currentRevision)) } else { gomega.Expect(pods.Items[i].Spec.Containers[0].Image).To(gomega.Equal(newImage), fmt.Sprintf("Pod %s/%s has image %s not equal to new image  %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Spec.Containers[0].Image, newImage)) gomega.Expect(pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel]).To(gomega.Equal(updateRevision), fmt.Sprintf("Pod %s/%s has revision %s not equal to new revision %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel], updateRevision)) } } } </ins> // confirmStatefulPodCount asserts that the current number of Pods in ss is count, waiting up to timeout for ss to // scale to count. func confirmStatefulPodCount(ctx context.Context, c clientset.Interface, count int, ss *appsv1.StatefulSet, timeout time.Duration, hard bool) {

My application is clustered and I have 2 pods running. I am using a rolling upgrade strategy and my partitioning number is 1. First I need to upgrade the pod with the ordinal number 1 so I triggered that upgrade with partitioning number 1 and my upgrade gets completed. After this, my pod having the ordinal number 0 gets restarted and it's automatically upgraded to a new image. This is not an expected behavior because still, my partition number is 1 and pod-0 should come up with an older image only. I am facing this issue for the v1.27 cluster. Prior to this, it was working fine. pod-0 should not upgrade to new image until I upgrade the partition number to 0.a Do helm install and upgrade pod-1 only keeping partition number to 1. No response <detailsconsole $ cat /etc/os-release $ uname -a C:wmic os get Caption, Version, BuildNumber, OSArchitecture </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig/api-machinery
/sig apps /sig storage
Hi, Can someone suggest here? As this is introduced in my setup after the v1.27 upgrade? Is there any change in partition logic?
This can be reproduced by using the said STS. 1 - Deploy STS using attached that has a replica count is 2 and updatingstrategy.rolling.partition as 1. 2 - Do edit STS and update the image to 1.15. This will upgrade, pod-1 with image 1.15. 3 - Next delete the pod pod-0 and check that POD-0 came up with a new image which is not the expected behavior. POD 0 should not be updated to a new image until the partition is changed to 0. We have noticed this behavior change in v1.27.


Question: } } <ins> // This function is used canary updates and phased rolling updates of template modifications for partiton1 and delete pod-0 func deletingPodForRollingUpdatePartitionTest(ctx context.Context, f *framework.Framework, c clientset.Interface, ns string, ss *appsv1.StatefulSet) { setHTTPProbe(ss) ss.Spec.UpdateStrategy = appsv1.StatefulSetUpdateStrategy{ Type: appsv1.RollingUpdateStatefulSetStrategyType, RollingUpdate: func() *appsv1.RollingUpdateStatefulSetStrategy { return &appsv1.RollingUpdateStatefulSetStrategy{ Partition: pointer.Int32(1), } }(), } ss, err := c.AppsV1().StatefulSets(ns).Create(ctx, ss, metav1.CreateOptions{}) framework.ExpectNoError(err) e2estatefulset.WaitForRunningAndReady(ctx, c, *ss.Spec.Replicas, ss) ss = waitForStatus(ctx, c, ss) currentRevision, updateRevision := ss.Status.CurrentRevision, ss.Status.UpdateRevision gomega.Expect(currentRevision).To(gomega.Equal(updateRevision), fmt.Sprintf("StatefulSet %s/%s created with update revision %s not equal to current revision %s", ss.Namespace, ss.Name, updateRevision, currentRevision)) pods := e2estatefulset.GetPodList(ctx, c, ss) for i := range pods.Items { gomega.Expect(pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel]).To(gomega.Equal(currentRevision), fmt.Sprintf("Pod %s/%s revision %s is not equal to currentRevision %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel], currentRevision)) } ginkgo.By("Adding finalizer for pod-0") pod0name := getStatefulSetPodNameAtIndex(0, ss) pod0, err := c.CoreV1().Pods(ns).Get(ctx, pod0name, metav1.GetOptions{}) framework.ExpectNoError(err) pod0.Finalizers = append(pod0.Finalizers, testFinalizer) pod0, err = c.CoreV1().Pods(ss.Namespace).Update(ctx, pod0, metav1.UpdateOptions{}) framework.ExpectNoError(err) pods.Items[0] = *pod0 defer e2epod.NewPodClient(f).RemoveFinalizer(ctx, pod0.Name, testFinalizer) ginkgo.By("Updating image on StatefulSet") newImage := NewWebserverImage oldImage := ss.Spec.Template.Spec.Containers[0].Image ginkgo.By(fmt.Sprintf("Updating stateful set template: update image from %s to %s", oldImage, newImage)) gomega.Expect(oldImage).ToNot(gomega.Equal(newImage), "Incorrect test setup: should update to a different image") ss, err = updateStatefulSetWithRetries(ctx, c, ns, ss.Name, func(update *appsv1.StatefulSet) { update.Spec.Template.Spec.Containers[0].Image = newImage }) framework.ExpectNoError(err) ginkgo.By("Creating a new revision") ss = waitForStatus(ctx, c, ss) currentRevision, updateRevision = ss.Status.CurrentRevision, ss.Status.UpdateRevision gomega.Expect(currentRevision).ToNot(gomega.Equal(updateRevision), "Current revision should not equal update revision during rolling update") ginkgo.By("Await for all replicas running, all are updated but pod-0") e2estatefulset.WaitForState(ctx, c, ss, func(set2 *appsv1.StatefulSet, pods2 *v1.PodList) (bool, error) { ss = set2 pods = pods2 if ss.Status.UpdatedReplicas == *ss.Spec.Replicas-1 && ss.Status.Replicas == *ss.Spec.Replicas && ss.Status.ReadyReplicas == *ss.Spec.Replicas { // rolling updated is not completed, because replica 0 isn't ready return true, nil } return false, nil }) ginkgo.By("Verify pod images before pod-0 deletion and recreation") for i := range pods.Items { if i < int(*ss.Spec.UpdateStrategy.RollingUpdate.Partition) { gomega.Expect(pods.Items[i].Spec.Containers[0].Image).To(gomega.Equal(oldImage), fmt.Sprintf("Pod %s/%s has image %s not equal to oldimage image %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Spec.Containers[0].Image, oldImage)) gomega.Expect(pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel]).To(gomega.Equal(currentRevision), fmt.Sprintf("Pod %s/%s has revision %s not equal to current revision %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel], currentRevision)) } else { gomega.Expect(pods.Items[i].Spec.Containers[0].Image).To(gomega.Equal(newImage), fmt.Sprintf("Pod %s/%s has image %s not equal to new image  %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Spec.Containers[0].Image, newImage)) gomega.Expect(pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel]).To(gomega.Equal(updateRevision), fmt.Sprintf("Pod %s/%s has revision %s not equal to new revision %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel], updateRevision)) } } ginkgo.By("Deleting the pod-0 so that kubelet terminates it and StatefulSet controller recreates it") deleteStatefulPodAtIndex(ctx, c, 0, ss) ginkgo.By("Await for two replicas to be updated, while the pod-0 is not running") e2estatefulset.WaitForState(ctx, c, ss, func(set2 *appsv1.StatefulSet, pods2 *v1.PodList) (bool, error) { ss = set2 pods = pods2 return ss.Status.ReadyReplicas == *ss.Spec.Replicas-1, nil }) ginkgo.By(fmt.Sprintf("Removing finalizer from pod-0 (%v/%v) to allow recreation", pod0.Namespace, pod0.Name)) e2epod.NewPodClient(f).RemoveFinalizer(ctx, pod0.Name, testFinalizer) ginkgo.By("Await for recreation of pod-0, so that all replicas are running") e2estatefulset.WaitForState(ctx, c, ss, func(set2 *appsv1.StatefulSet, pods2 *v1.PodList) (bool, error) { ss = set2 pods = pods2 return ss.Status.ReadyReplicas == *ss.Spec.Replicas, nil }) ginkgo.By("Verify pod images after pod-0 deletion and recreation") pods = e2estatefulset.GetPodList(ctx, c, ss) for i := range pods.Items { if i < int(*ss.Spec.UpdateStrategy.RollingUpdate.Partition) { gomega.Expect(pods.Items[i].Spec.Containers[0].Image).To(gomega.Equal(oldImage), fmt.Sprintf("Pod %s/%s has image %s not equal to current image %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Spec.Containers[0].Image, oldImage)) gomega.Expect(pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel]).To(gomega.Equal(currentRevision), fmt.Sprintf("Pod %s/%s has revision %s not equal to current revision %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel], currentRevision)) } else { gomega.Expect(pods.Items[i].Spec.Containers[0].Image).To(gomega.Equal(newImage), fmt.Sprintf("Pod %s/%s has image %s not equal to new image  %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Spec.Containers[0].Image, newImage)) gomega.Expect(pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel]).To(gomega.Equal(updateRevision), fmt.Sprintf("Pod %s/%s has revision %s not equal to new revision %s", pods.Items[i].Namespace, pods.Items[i].Name, pods.Items[i].Labels[appsv1.StatefulSetRevisionLabel], updateRevision)) } } } </ins> // confirmStatefulPodCount asserts that the current number of Pods in ss is count, waiting up to timeout for ss to // scale to count. func confirmStatefulPodCount(ctx context.Context, c clientset.Interface, count int, ss *appsv1.StatefulSet, timeout time.Duration, hard bool) {

Pla suggest : apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx spec: ports: port: 80 name: web clusterIP: None selector: app: nginx apiVersion: apps/v1 kind: StatefulSet metadata: name: web-am spec: updateStrategy: type: RollingUpdate rollingUpdate: partition: 1 serviceName: "nginx" replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: name: nginx args: /bin/sh -c sleep 30;touch /tmp/healthy; sleep 3600 image: nginx:1.14.2 ports: containerPort: 80 name: web


Question: "restartPolicy": "Always" } } <ins> {% endif %} </ins>

For some reason they stopped working and we even don't create manifest file for cluster autoscaler. I used the following command to start the cluster:
It's possible it's related to


Question: basename := filepath.Base(os.Args[0]) if err := commandFor(basename, hyperkubeCommand, allCommandFns).Execute(); err != nil { <del> fmt.Fprintf(os.Stderr, "%vn", err) </del> os.Exit(1) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl version --client=yes command is giving the following error message. Error message is repeated at the start and the end and also containing programming jargon Error: invalid argument "yes" for "--client" flag: strconv.ParseBool: parsing "yes": invalid syntax Examples: Print the client and server versions for the current context kubectl version Options: --client=false: Client version only (no server required). -o, --output='': One of 'yaml' or 'json'. --short=false: Print just the version number. Usage: kubectl version [flags] [options] Use "kubectl options" for a list of global command-line options (applies to all commands). invalid argument "yes" for "--client" flag: strconv.ParseBool: parsing "yes": invalid syntax What you expected to happen: The error message can be improved with one of the two variants invalid argument "yes" for "--client" flag Examples: Print the client and server versions for the current context kubectl version Options: --client=false: Client version only (no server required). -o, --output='': One of 'yaml' or 'json'. --short=false: Print just the version number. Usage: kubectl version [flags] [options] Use "kubectl options" for a list of global command-line options (applies to all commands). (or) Invalid argument "yes" for "--client" flag How to reproduce it (as minimally and precisely as possible): Run the command kubectl version --client=yes Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: AWS OS (e.g: ): Amazon Linux Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
Hi Thanks for posting this issue! It turns out that this duplicate output has been in kubectl, and all the other cli cmds, since . This should be fixed by , and hopefully we can cherry pick it back to at least .


Question: } return true } <ins> // GetWindowsPath get a windows path func GetWindowsPath(path string) string { windowsPath := strings.Replace(path, "/", "", -1) if strings.HasPrefix(windowsPath, "") { windowsPath = "c:" + windowsPath } return windowsPath } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I have tried FlexVolume on a Windows node, it does not work. I am now working on this feature now. Below is the FlexVolume initialization error What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.7.7 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig windows /assign
Detailed steps about how to use flexvolume on Windows with this PR:
cc:


Question: // CheckEtcdServers will attempt to reach all etcd servers once. If any // can be reached, return true. func (con EtcdConnection) CheckEtcdServers() (done bool, err error) { <del> // Attempt to reach every Etcd server in order for _, serverURI := range con.ServerList { host, err := parseServerURI(serverURI) </del> <ins> // Attempt to reach every Etcd server randomly. serverNumber := len(con.ServerList) serverPerms := rand.Perm(serverNumber) for _, index := range serverPerms { host, err := parseServerURI(con.ServerList[index]) </ins> if err != nil { return false, err }

/kind bug What happened: Every time a health check is called on the APIServer via the endpoint, an etcd healthcheck is performed. However, as this just starts at the first server in the list passed to the apiserver binary all the healthchecks generate read load on a single etcd member. What you expected to happen: The first server that is checked should either increment (and wrap around) each time or be randomised to spread load around the cluster
/assign
I think a random check is ok.
/sig api-machinery


Question: } response, err := h.Client.Get(key, false, false) <del> if err != nil && !IsEtcdNotFound(err) { </del> <ins> if err != nil { if IsEtcdNotFound(err) { return nil } </ins> return err }

With the move to running the kube-apiserver in a pod, I am getting errors of the form: The root null pointer is from this line: The line numbers in the stack are from my continuing debug of the original null pointer. We need to protect against a null pointer in this path, which appears to only be invoked when doing a list call that matches a single item based on field selector. Will submit a PR to address.
With proper line numbers:


Question: echo "ZONE='${ZONE}'" echo "MASTER_NAME='${MASTER_NAME}'" echo "MINION_IP_RANGE='${MINION_IP_RANGES[$i]}'" <ins> echo "EXTRA_DOCKER_OPTS='${EXTRA_DOCKER_OPTS}'" </ins> echo "ENABLE_DOCKER_REGISTRY_CACHE='${ENABLE_DOCKER_REGISTRY_CACHE:-false}'" grep -v "^#" "${KUBE_ROOT}/cluster/gce/templates/common.sh" grep -v "^#" "${KUBE_ROOT}/cluster/gce/templates/salt-minion.sh"


Question: return framework.NewStatus(framework.Error, "pod cannot be nil") } if pod.Name == "my-test-pod" { <del> state.Lock() </del> state.Write(framework.StateKey(pod.Name), &stateData{data: "never bind"}) <del> state.Unlock() </del> } return nil }

What happened: kube-scheduler () crashes due to a panic while running : The race condition is triggered when an out-of-tree plugin, that writes to during its Filter phase, is registered through . Access to in general happens from a single goroutine and there are no practical concurrency issues, with the exception of the phase, that parallelizes calls for each node across plugins: A quick glance at the tip of the branch seems to indicate the bug is still present there. What you expected to happen: Ideally access to should be synchronized everywhere for correctness, including by all in-tree plugins, since there are no guarantees about how will call multiple plugins concurrently. It seems to me it is something that was considered at some point, given that has methods. Sadly, almost no in-tree plugins are synchronizing access. For now, I will try to work around the problem by doing a single write to during and storing a data bag structure where I will write state into during (with proper synchronization), instead of writing to directly. How to reproduce it (as minimally and precisely as possible): Install a scheduler plugin that writes to during its phase. In-tree (builtin) plugins are not synchronizing their access to . Environment: Kubernetes version (use ): v1.18.8 Cloud provider or hardware configuration: linux-amd64 (AWS) OS (e.g: ): Ubuntu 18.04.5 LTS (Bionic Beaver) Others: custom kube-scheduler binary built with an out-of-tree plugin using
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
AFAICT all in-tree plugins write to the cycle state in PreFilter and then only read from cycle state in Filter. So this shouldn't be a problem with in-tree plugins as concurrent read access should be fine during RunFilterPlugins. But since your out-of-tree plugin writes in Filter, I can see why this can be a problem.


Question: return framework.NewStatus(framework.Error, "pod cannot be nil") } if pod.Name == "my-test-pod" { <del> state.Lock() </del> state.Write(framework.StateKey(pod.Name), &stateData{data: "never bind"}) <del> state.Unlock() </del> } return nil }

Aside from adding Lock/Unlock in in-tree plugin, offering mutual exclusion on each key as a built-in feature within CycleState could be an alternative solution. This way, reads from in-tree plugins during Filter wouldn't interfere with writes during Filter from out-of-tree plugins.
cc
should we change the Read/Write CycleState methods to be thread safe?
Adding locks without a proper justification would just increase the latency. could you clarify your use case? Why do you need to write anything during Filter?
yeah, we should measure it, but I don't expect it to have significant effect...
Adding locks without a proper justification would just increase the latency. Without contention (which is the case with in-tree plugins), I don't think this is a concern.
Adapting source from : source: essentially, 11.3 ns/Lock-Unlock pair
I remember locks showing up in profiling for PreScore/Score for certain plugins. Although I have to admit that the accesses were not properly implemented. I'd like to hear an answer for before doing any experiments.
+1 to that I'd also like to understand the motivations / requirements of mutating CycleState in Filter().
My particular use-case is that we have resources being dynamically to/removed from Nodes, and those resources need to be allocated (and tracked) in the scheduling loop. Technically I could pre-compute all the available dynamic resources for all Nodes at the phase, and snapshot it, but that would require me to pre-compute everything for all Nodes on all scheduling cycles. Instead, what I'm currently doing is computing what dynamic resources can be utilized on-demand during the phase (kind of a lazy loading of dynamic resources for each pair), and saving state of what extra resources were considered during for each Node that was evaluated. Then, later in the scheduler plugin can lock-in the exact dynamic resources for the particular Node that was selected. I understand the concern with adding extra synchronization overhead without a real need. Even if my use-case is not good enough to justify it, there is another argument for it: technically, the lack of access synchronization on violates the . Even if in the end we decide synchronizing all access is not worth the overhead (for correctness), I would strongly encourage better documentation saying which plugin phases are allowed to modify , and perhaps even forcing a snapshot-read-only copy of to be used by plugins.
Using the scheduler to track resources is not a good idea.


Question: return framework.NewStatus(framework.Error, "pod cannot be nil") } if pod.Name == "my-test-pod" { <del> state.Lock() </del> state.Write(framework.StateKey(pod.Name), &stateData{data: "never bind"}) <del> state.Unlock() </del> } return nil }

The scheduler should be stateless. Can't you track the resources with an external controller or with a device plugin and making it part of the Node spec?
there may be some misunderstanding in my poor choice of words. What I mean by "tracking" is that the scheduler needs to track usage and availability of these extra per-Node resources, to be able to allocate them during placement/scheduling. Exactly the same it does with volumes, cpus, memory, ephemeral disk, etc Doing computations in a controller outside of the scheduling loop would be similar to "precomputing" these resources. In my particular use case I chose a different way because: introduces another point of failure, if the external controller stops pre-computing extra resources and publishing them somewhere (CRD, Node spec, etc) scheduling halts (at least scheduling on these extra resources) Node resources in my use case are volatile (they are coming and going frequently), and the extra latency by an extra controller (vs doing all computations right in the scheduling loop) may be problematic for my use case. Even if you consider my use case not enough to add proper synchronization to , I'd still encourage making it explicit what scheduling phases are allowed to modify it (and even passing in a snapshot read-only version of to phases that are not expected to update it), to avoid confusion and race conditions.
Ok, if you are not tracking the existence of resources in the scheduler, I wouldn't suggest the external controller. Still, have you considered the device plugin framework? That way, the default kube-scheduler could keep track of resources usage. Otherwise, I still don't see why you would need to update the CycleState during Filter. As its name indicates, CycleState only lives during a scheduling cycle.
Regardless of the correctness of usage of CycleState to do what wants, I still think we need to do at least one of the following: that writing to CycleState during Filter will lead to data races, a read-only CycleState to Filter plugins (whether we do this at the interface level or if we simply drop Write requests during Filter is an implementation detail), or make CycleState concurrency safe CycleState as it exists right now allows plugin authors to shoot themselves in the foot IMO.
I'm leaning towards 1, but it's fair to continue the discussion before deciding.
Can you run the "controller" as a goroutine along with your scheduler plugin?


Question: return framework.NewStatus(framework.Error, "pod cannot be nil") } if pod.Name == "my-test-pod" { <del> state.Lock() </del> state.Write(framework.StateKey(pod.Name), &stateData{data: "never bind"}) <del> state.Unlock() </del> } return nil }

So it's not an external controller binary, and hence don't introduce another point of failure. It's pretty like how the default scheduler manages its internal cache. But note that cache is only needed when you need a more efficient data structure. If a {key:value} map is good enough, spinning up a XYZInformer and obtain the latest XYZ resource on-demand is the most recommended way. I think it quite depends on how you design the internal XYZ cache. For example, it should only tweak the cache in an incremental manner - in other words, unchanged resource will be kept unmodified. You can check out the "doubly-pointer snapshot" of scheduler's cache. With above said, if I understand your case correctly, you may need to build a sophisticated XYZ cache, which runs in a separate goroutine and responds to your XYZ resource changes. And then upon each scheduling cycle, take a snapshot (may needs a specific snapshotting algorithm) in PreFilter() and WRITE to CycleState, and then in Filter() READ from CycleState. My personal option is to suggest all plugin developers to read from CycleState in Filter().
nailed it. What you describe is almost exactly what I’m doing today. There are some details particular to my use case (e.g. just tracking current state with an informer and an incremental cache is not enough, because there’s some logic involved in deciding if currently available resources are usable by each pair, and I currently do that computation in as pairs are evaluated). In any case, I just wanted to share my use case to provide some color, but I don’t want what I’m doing distracting us from the more important questions, mentioned by above. My personal preference would be as a very short term next step, then when someone has cycles to do it. I would be OK with punting on until a more strong need shows up.
Note that limiting to only reading from works for me, as long as I can still read my own data structure from it and do writes on it (i.e. don’t deepclone for the phase).
also adds overhead (from copying the state, which can be very expensive).
Could implement just by changing the interface passed to (e.g. ). That wouldn’t be backwards compatible, but very cheap.


Question: return framework.NewStatus(framework.Error, "pod cannot be nil") } if pod.Name == "my-test-pod" { <del> state.Lock() </del> state.Write(framework.StateKey(pod.Name), &stateData{data: "never bind"}) <del> state.Unlock() </del> } return nil }

I guess that would work in the sense that you cannot change the high level values of the CycleState map, but you could still override what's inside them, if they have pointers. But maybe that's ok.
but you could still override what's inside them, if they have pointers. But maybe that's ok. yeah, if it was pointers and if the underlying data needed to be modified, I'd imagine it would have its own Lock/Unlock semantics
for such implementation, the only remaining concern is backwards compatibility. So I would leave that decision to who leads the scheduling-plugins repo.
I rethink 's original problem a little bit: the original racing issue occurs when reading CycleState in in-tree plugins, this indicates one of the (out-of-tree) plugins is (over)writing the most high-level value of the CycleState map. But if we control the locking inside the out-of-tree plugin, and mutate that particular in-place, that would be good enough? Case 3 in the above program demonstrates the correct way to manipulate the specific out-of-tree state. Let me know your thoughts. If it resolves 's concern, we don't need to make any change on the Filter() interface.
Case 3 is how I am working around the problem today to avoid the race. I write a "data bag" once into top level in , then only modify my own inner structure (with proper locking) in , without doing any further modifications to directly (top level) in . That is acceptable, but I'd encourage you to consider making the interface (and documentation) more explicit for plugins, so they avoid the mistake of writing directly into . Without changing anything (no documentation, no readonly interface), it is pretty easy for a plugin to use in a way that you don't want to support.
Thanks for confirming that. Yes, documenting it, or(and) publishing examples would be quite helpful to clarify the semantics. As a wrap-up: a keyed value can be read from CycleState, literally , in any phase. It's always race-free.


Question: return framework.NewStatus(framework.Error, "pod cannot be nil") } if pod.Name == "my-test-pod" { <del> state.Lock() </del> state.Write(framework.StateKey(pod.Name), &stateData{data: "never bind"}) <del> state.Unlock() </del> } return nil }

if you want to mutate the top-level keyed value in CycleState, if it's a phase that CycleState is accessed consecutively, such as PreFilter, PreScore, it's safe to mutate CycleState in any manner if it's a phase that CycleState is accessed parallelly, to be race-free, (for now) you only have 1 option: keep the keyed value's pointer unchanged, and use an internal lock to mutate that pointer obj [TBD] we may come up with changes to add read locks to all in-tree plugins when reading values from CycleState, so that the out-of-tree plugin can surround its mutating logic between and - th mutating logic thereby can be any style Any thoughts on the TBD item? I would assume read locks wouldn't cost too much.
For now, I think the documentation is correct, it clearly says Read() and Write() are not thread-safe: The key thing here is whether in-tree plugin should use RLock/RUnlock or not. If yes, out-of-tree plugins can be freed to use Lock/Unlock. (see the TBD item in the previous comment)
I like the TBD, and optimizing for correctness in how in-tree plugins are accessing . I understand the concern with synchronization overhead, but I would be really surprised if it is the bottleneck in scheduling code.
It can be, when you have 5k-nodes clusters, or bigger.
I don't think it will be expensive, a rlock should be equivalent to an atomic add operation (basically it increments the number of readers). As a comparison, looking up the state from the map is likely to be more expensive.
Of course we are assuming no contention in the default plugins.
This is my impression as well. A vanilla version of scheduler doesn't impose write lock in Filter() and Score(), so should be good.
We can always see with the benchmarks and revert if its not negligible.
I will run the benchmarks to see if there is perf downgrade. /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale This is still relevant, yes?


Question: return framework.NewStatus(framework.Error, "pod cannot be nil") } if pod.Name == "my-test-pod" { <del> state.Lock() </del> state.Write(framework.StateKey(pod.Name), &stateData{data: "never bind"}) <del> state.Unlock() </del> } return nil }

IIUC the volumebinding plugin has been fixed, but external plugins can still shoot themselves in the foot.
/priority backlog
Yes. I'm working on this, late this week I will send out a PR along with perf test result.


Question: Priorities []PriorityPolicy `json:"priorities"` // Holds the information to communicate with the extender(s) ExtenderConfigs []ExtenderConfig `json:"extenders"` <ins> // RequiredDuringScheduling affinity is not symmetric, but there is an implicit PreferredDuringScheduling affinity rule // corresponding to every RequiredDuringScheduling affinity rule. // HardPodAffinitySymmetricWeight represents the weight of implicit PreferredDuringScheduling affinity rule, in the range 1-100. HardPodAffinitySymmetricWeight int `json:"hardPodAffinitySymmetricWeight"` </ins> } type PredicatePolicy struct {

( noticed this) HardPodAffinitySymmetricWeight should be in the , not in KubeSchedulerConfiguration If/when fixing this, we need to be careful to do it in a backward-compatilble way.
Do you mean ? It seems scheduler policy config's args can not passed to FitPredicates; it need to update related interface. For the : mark to be deprecated and set scheduler policy config to HardPodAffinitySymmetricWeight if policy config is empty.
Are you working on this? if no, I can take it.
, nop; please go ahead.
/ / / , When reviewing the PR, there's one question on : the default value of is 1, which means the weight (1) will always add to priorities if term matched (), does it our expectation? And the user has to set it to 0 to disable it. IMO, the default value of should be 0.
The value 1 (instead of 0) was chosen so that if you say A must run with B, there is an automatic preference that B should run with A. This seems like reasonable behavior.
Thanks very much; reasonable case to me. But it seems we can not disable it: if set to 0, the default funcs will set it to default (1); similar result to un-set. Anyway, no user complain this now :).


Question: } } <ins> // TestClusterIPLBInCreateDsrLoadBalancer tests, if the available endpoints are remote, // syncproxyrules only creates ClusterIP Loadbalancer and no NodePort, External IP or IngressIP // loadbalancers will be created. func TestClusterIPLBInCreateDsrLoadBalancer(t *testing.T) { syncPeriod := 30 * time.Second proxier := NewFakeProxier(syncPeriod, syncPeriod, clusterCIDR, "testhost", netutils.ParseIPSloppy("10.0.0.1"), NETWORK_TYPE_OVERLAY) if proxier == nil { t.Error() } svcIP := "10.20.30.41" svcPort := 80 svcNodePort := 3001 svcPortName := proxy.ServicePortName{ NamespacedName: makeNSN("ns1", "svc1"), Port:           "p80", Protocol:       v1.ProtocolTCP, } lbIP := "11.21.31.41" makeServiceMap(proxier, makeTestService(svcPortName.Namespace, svcPortName.Name, func(svc *v1.Service) { svc.Spec.Type = "NodePort" svc.Spec.ClusterIP = svcIP svc.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyLocal svc.Spec.Ports = []v1.ServicePort{{ Name:     svcPortName.Port, Port:     int32(svcPort), Protocol: v1.ProtocolTCP, NodePort: int32(svcNodePort), }} svc.Status.LoadBalancer.Ingress = []v1.LoadBalancerIngress{{ IP: lbIP, }} }), ) tcpProtocol := v1.ProtocolTCP populateEndpointSlices(proxier, makeTestEndpointSlice(svcPortName.Namespace, svcPortName.Name, 1, func(eps *discovery.EndpointSlice) { eps.AddressType = discovery.AddressTypeIPv4 eps.Endpoints = []discovery.Endpoint{{ Addresses: []string{epIpAddressRemote}, NodeName:  pointer.StringPtr("testhost2"), // This will make this endpoint as a remote endpoint }} eps.Ports = []discovery.EndpointPort{{ Name:     pointer.StringPtr(svcPortName.Port), Port:     pointer.Int32(int32(svcPort)), Protocol: &tcpProtocol, }} }), ) proxier.setInitialized(true) proxier.syncProxyRules() svc := proxier.svcPortMap[svcPortName] svcInfo, ok := svc.(*serviceInfo) if !ok { t.Errorf("Failed to cast serviceInfo %q", svcPortName.String()) } else { // Checking ClusterIP Loadbalancer is created if svcInfo.hnsID != guid { t.Errorf("%v does not match %v", svcInfo.hnsID, guid) } // Verifying NodePort Loadbalancer is not created if svcInfo.nodePorthnsID != "" { t.Errorf("NodePortHnsID %v is not empty.", svcInfo.nodePorthnsID) } // Verifying ExternalIP Loadbalancer is not created for _, externalIP := range svcInfo.externalIPs { if externalIP.hnsID != "" { t.Errorf("ExternalLBID %v is not empty.", externalIP.hnsID) } } // Verifying IngressIP Loadbalancer is not created for _, ingressIP := range svcInfo.loadBalancerIngressIPs { if ingressIP.hnsID != "" { t.Errorf("IngressLBID %v is not empty.", ingressIP.hnsID) } } } } </ins> func TestEndpointSlice(t *testing.T) { syncPeriod := 30 * time.Second proxier := NewFakeProxier(syncPeriod, syncPeriod, clusterCIDR, "testhost", netutils.ParseIPSloppy("10.0.0.1"), NETWORK_TYPE_OVERLAY)

I am tracking a changed behavior in networking with kubernetes 1.26. If I create a service of type externalTrafficPolicy: Local, and there are no local endpoints in a node, kube-proxy does not create hns loadbalancers for the clusterIP of that service. With kubernetes 1.25, it creates a loadbalancer for clusterIP (but not externalIP). This is the relevant log message: Looks like this comes with a result from a new function isAllEndpointsTerminating introduced in kubernetes 1.26. In Linux, we can still access clusterIP even if there are no local endpoints for the service. Thoughts? The end result is that in Windows, clients cannot access the clusterIP of such a service with there are no local endpoints.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
cc
Does this issue occur when there are no local endpoints for a given service? If there is a local endpoint, then you can reach the service again, right?
, Also let us know, do you have any InternalTrafficPolicy configured ?
Does this issue occur when there are no local endpoints for a given service? If there is a local endpoint, then you can reach the service again, right? That is correct , Also let us know, do you have any InternalTrafficPolicy configured ? We do not have that configured


Question: VolumeMounts:  volumeMounts, LivenessProbe: componentProbe(8080, "/healthz"), Resources:     componentResources("250m"), <ins> Env:           getProxyEnvVars(), </ins> }, volumes...), kubeControllerManager: componentPod(api.Container{ Name:          kubeControllerManager,

Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? http proxy, httpproxy, httpsproxy, AWS, kubeadm, kubectl, cloud-provider Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): v1.4.4 Environment: Cloud provider or hardware configuration: kubeadm on EC2 instances on AWS OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Linux ip-10-0-133-99 4.4.0-45-generic -Ubuntu SMP Wed Oct 19 14:12:37 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: httpproxy, httpsproxy and noproxy set in /etc/environment, Docker config as well on startup scripts that run What happened: won't complete and gets stuck on it's first deployment. When one looks through the logs it shows that the pods are downloading fine but we are stuck due to the kube-controller-manager continually crashing as it can't contact AWS. Elsewhere in kubernetes I see references to and if the AWS cloud provider could use that it would be able to connect as well. From the POD if I manually set httpproxy and httpsproxy and no_proxy I am able to use the AWS CLI without any issues. What you expected to happen: I would expect to successfully complete and use the environment proxies set up on the system so as not to depend on a default Internet Gateway being connected to the VPC just for AWS access. How to reproduce it (as minimally and precisely as possible): Set up a public VPC with Squid set up as a proxy Set up a private VPC peered to the public VPC Set up a master node and run Set up a regular node and run will never fully initialize. If you remove the --cloud-provider aws` then it will complete and nodes will register. Anything else do we need to know:
what do you have in proxy variables ? (env | grep -i proxy) it tries to connect to (tcp 176.32.118.53:443) directly. You're expecting to connect there via proxy ? then do you have something like in your no_proxy ?


Question: VolumeMounts:  volumeMounts, LivenessProbe: componentProbe(8080, "/healthz"), Resources:     componentResources("250m"), <ins> Env:           getProxyEnvVars(), </ins> }, volumes...), kubeControllerManager: componentPod(api.Container{ Name:          kubeControllerManager,

Unfortunately on the pod itself for the the environment variables aren't passed through. As this is created using I don't have control over the pod definition. So basically none of the *_proxy variables are set inside the pod in the docker image, however if I and export the variables and try to wget that URL it works. Basically I'm hoping we can update to respect and pass the environment variables down to the system pods if they exist, eg, add: Without it I can't stand up a cluster with AWS integration on a completely private VPC and for such a small change it would make a rather big difference.
ok, so request is actually to pass proxy variables into containers. interesting. I'll check how it can be done. BTW, for noproxy value in your example: things like 10.0.0.0/8 would not work on 99% of software that respects noproxy variable. it must be always exact IP match.
you're right that what I'm actually asking for is to pass proxy variables through either from a commandline setting on kubeadm or pick it up from /etc/environment or shell environment that launches kubeadm. I didn't realize it till I dug in to try and think about how one might go about fixing this. At first I thought we would just pick up the variables from the system environment but then I realized kube-controller-manager is deployed in a container and so we'd need to update the yaml file that launches it to contain the environment variables. As I'd like to go to production with this soon hopefully this is a small change that can be .
This issue was moved to kubernetes/kubeadm


Question: ) const execInfoEnv = "KUBERNETES_EXEC_INFO" <ins> const onRotateListWarningLength = 1000 </ins> var scheme = runtime.NewScheme() var codecs = serializer.NewCodecFactory(scheme)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Data race in client-go with below sample code: Data race: What you expected to happen: No data race How to reproduce it (as minimally and precisely as possible): Using above given sample code Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ): OSX
/sig api-machinery
I'd like to take it. /assign
/remove-sig api-machinery /sig auth cli
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then?
Another reminder here that code freeze is in less than two weeks (Nov. 14). We will need to move this issue out of the milestone if isn't going to be resolved before then.


Question: flagNames.InsecureSkipTLSVerify.BindBoolFlag(flags, &clusterInfo.InsecureSkipTLSVerify) } <del> // BindOverrideFlags is a convenience method to bind the specified flags to their associated variables func BindOverrideFlags(overrides *ConfigOverrides, flags *pflag.FlagSet, flagNames ConfigOverrideFlags) { BindAuthInfoFlags(&overrides.AuthInfo, flags, flagNames.AuthOverrideFlags) BindClusterFlags(&overrides.ClusterInfo, flags, flagNames.ClusterOverrideFlags) BindContextFlags(&overrides.Context, flags, flagNames.ContextOverrideFlags) flagNames.CurrentContext.BindStringFlag(flags, &overrides.CurrentContext) flagNames.Timeout.BindStringFlag(flags, &overrides.Timeout) } </del> // BindFlags is a convenience method to bind the specified flags to their associated variables func BindContextFlags(contextInfo *clientcmdapi.Context, flags *pflag.FlagSet, flagNames ContextOverrideFlags) { flagNames.ClusterName.BindStringFlag(flags, &contextInfo.Cluster)

Found in openshift, but I'm pretty sure the codepaths are the same. That means the token is getting ignored and in-cluster config is being used in preference to flags.
Is a similar issue?
Yep, kind of regression.
Sigh.
Yes, most likely the same cause.
I'm on it.
Unit test :)
Probably will want to cherrypick this into 1.5.
WIP in


Question: "github.com/golang/glog" <del> "k8s.io/api/core/v1" </del> apierrors "k8s.io/apimachinery/pkg/api/errors" <del> "k8s.io/apimachinery/pkg/labels" </del> utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait" <del> v1informers "k8s.io/client-go/informers/core/v1" v1listers "k8s.io/client-go/listers/core/v1" </del> "k8s.io/client-go/tools/cache" "k8s.io/client-go/util/workqueue"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Currently APIServiceRegistrationController watch service change and find the related APIServices related if any. Then it AddAPIService/RemoveAPIService for APIAggregator. I dig into the kube-aggregator and find it no need totally, because update APIServices status when related service or endpoint change. So if does not watch services, it will not miss any event for APIServices. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/cc
/kind enhancement /sig api-machinery
/assign
that does not follow. AvailableConditionController modifies the APIService object. APIServiceRegistrationController must still watch the APIService objects to observe those changes.
oh, I misread the description. You're saying APIServiceRegistrationController doesn't need to watch v1.Service objects


Question: Endpoints: []discovery.Endpoint{}, }, }, <ins> "bad-port-name-length": { expectedErrors: 1, endpointSlice: &discovery.EndpointSlice{ ObjectMeta:  standardMeta, AddressType: addressTypePtr(discovery.AddressTypeIP), Ports: []discovery.EndpointPort{{ Name:     utilpointer.StringPtr(strings.Repeat("a", 64)), Protocol: protocolPtr(api.ProtocolTCP), }}, Endpoints: []discovery.Endpoint{}, }, }, </ins> "invalid-port-protocol": { expectedErrors: 1, endpointSlice: &discovery.EndpointSlice{

What happened: No endpoints created in the EndpointSlice for my service due to the port on the service being too long. Logs show: EndpointSlice created has no endpoint: What you expected to happen: Service that works with Endpoints should work with EndpointSlice. If this is intentional, it should be clearly documented How to reproduce it (as minimally and precisely as possible): Create a service with a long port name Environment: Kubernetes version (use ): 1.16 Cloud provider or hardware configuration: kind
/sig networking
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /sig networking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
For a rough measure of impact, Istio had 5 ports that were over this limit (and many more that were right at 15 characters), so I suspect it may not be terribly rare.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This needs to be resolved before endpointslices graduate from alpha
/assign
/remove-triage unresolved