Question:
Similar to this is to test the first contributor experience fails on ubuntu 16.04, docker version 1.12.3:
Got the same error on Linux ubuntu 4.2.0-27-generic ~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x8664 x8664 x86_64 GNU/Linux Docker Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.2.1 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.2.1 Git commit (server): OS/Arch (server): linux/amd64
What git sha of were you building from?
I have cloned the head. I think the reason might be missing package: gccgo-4.8-aarch64-linux-gnu I'm compiling right now and update on result.
were you running or ? runs inside docker, so the gcc error is confusing.
I run: sudo make release in Ubuntu VM running in VMWare workstation if that matters. Are you in kubernetes-users slack channel?
does fail in the same way?
Do you need to run , instead of just ? Can you try running the following:
well, if I don't do sudo docker does not work :) I need to run as root. I'll try the commands as soon as quick-release ends.
ah, gotcha. you can probably fix the docker access issue by following in the docker manual. that's probably not why it's failing, though.
make release still does not work :(
can you please run and then paste the results? ( turns up the debugging verbosity.)
for me: works (using your PR to fixes the gobindata dep) -local Go build- works (docker build) works if you have Docker running. does not work: on OSX did not seem to give more verbosity.
V=4 make release did not help :(
oops, that was not the correct environment variable, sorry. I meant .
here we go:
Aha, I think I figured it out. Sorry for making you rebuild so many times; the issue still isn't super obvious from the logs. Basically, the bug only occurs if you don't have enough memory to do the docker builds in parallel; if that happens, you fall into this . On the parallel side of the branch, we set the platform env vars within a subshell, so each of the cross-compilation builds won't affect each other. On the non-parallel side of the branch, we don't use a subshell. Inside , there's no overrides done for . As a result, we end up incorrectly reusing the value for , , which doesn't recognize the flag being applied for the darwin build. The fix should be easy; I'll get that sent out ASAP. we probably want this fix for 1.5, since if you fall back to non-parallel crossbuilds we're doing the wrong thing.
Ack. Will keep an eye out for the fix. Please mark it as cherrypick-candidate
Thank you guys!
I think it is almost there. Got much further in the build (4 GB of RAM). Towards the end there is a failed test that makes the build fail: This is probably a different bug, as your PR got passed the misconfigured compilation.
FWIW, the worked on ubuntu 16.06. It did ran some tests apparently but the build succeeded.
I just tried with and it worked fine
since we have a PR ready and working, can we please switch this to non-release-blocker?

--env "KUBE_FASTBUILD=${KUBE_FASTBUILD:-false}" --env "KUBE_BUILDER_OS=${OSTYPE:-notdetected}" --env "KUBE_BUILD_PPC64LE=${KUBE_BUILD_PPC64LE}"  # TODO(IBM): remove <ins> --env "KUBE_VERBOSE=${KUBE_VERBOSE}" </ins> ) # If we have stdin we can run interactive.  This allows things like 'shell.sh'


Question:
Similar to this is to test the first contributor experience fails on ubuntu 16.04, docker version 1.12.3:
Got the same error on Linux ubuntu 4.2.0-27-generic ~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x8664 x8664 x86_64 GNU/Linux Docker Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.2.1 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.2.1 Git commit (server): OS/Arch (server): linux/amd64
What git sha of were you building from?
I have cloned the head. I think the reason might be missing package: gccgo-4.8-aarch64-linux-gnu I'm compiling right now and update on result.
were you running or ? runs inside docker, so the gcc error is confusing.
I run: sudo make release in Ubuntu VM running in VMWare workstation if that matters. Are you in kubernetes-users slack channel?
does fail in the same way?
Do you need to run , instead of just ? Can you try running the following:
well, if I don't do sudo docker does not work :) I need to run as root. I'll try the commands as soon as quick-release ends.
ah, gotcha. you can probably fix the docker access issue by following in the docker manual. that's probably not why it's failing, though.
make release still does not work :(
can you please run and then paste the results? ( turns up the debugging verbosity.)
for me: works (using your PR to fixes the gobindata dep) -local Go build- works (docker build) works if you have Docker running. does not work: on OSX did not seem to give more verbosity.
V=4 make release did not help :(
oops, that was not the correct environment variable, sorry. I meant .
here we go:
Aha, I think I figured it out. Sorry for making you rebuild so many times; the issue still isn't super obvious from the logs. Basically, the bug only occurs if you don't have enough memory to do the docker builds in parallel; if that happens, you fall into this . On the parallel side of the branch, we set the platform env vars within a subshell, so each of the cross-compilation builds won't affect each other. On the non-parallel side of the branch, we don't use a subshell. Inside , there's no overrides done for . As a result, we end up incorrectly reusing the value for , , which doesn't recognize the flag being applied for the darwin build. The fix should be easy; I'll get that sent out ASAP. we probably want this fix for 1.5, since if you fall back to non-parallel crossbuilds we're doing the wrong thing.
Ack. Will keep an eye out for the fix. Please mark it as cherrypick-candidate
Thank you guys!
I think it is almost there. Got much further in the build (4 GB of RAM). Towards the end there is a failed test that makes the build fail: This is probably a different bug, as your PR got passed the misconfigured compilation.
FWIW, the worked on ubuntu 16.06. It did ran some tests apparently but the build succeeded.
I just tried with and it worked fine
since we have a PR ready and working, can we please switch this to non-release-blocker?

[[ -n ${1-} ]] || { kube::log::error_exit "!!! Internal error. No platform set in kube::golang::set_platform_envs" } <ins> # make sure we have a clean slate first kube::golang::unset_platform_envs </ins> export GOOS=${platform%/*} export GOARCH=${platform##*/}


Question:
Similar to this is to test the first contributor experience fails on ubuntu 16.04, docker version 1.12.3:
Got the same error on Linux ubuntu 4.2.0-27-generic ~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x8664 x8664 x86_64 GNU/Linux Docker Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.2.1 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.2.1 Git commit (server): OS/Arch (server): linux/amd64
What git sha of were you building from?
I have cloned the head. I think the reason might be missing package: gccgo-4.8-aarch64-linux-gnu I'm compiling right now and update on result.
were you running or ? runs inside docker, so the gcc error is confusing.
I run: sudo make release in Ubuntu VM running in VMWare workstation if that matters. Are you in kubernetes-users slack channel?
does fail in the same way?
Do you need to run , instead of just ? Can you try running the following:
well, if I don't do sudo docker does not work :) I need to run as root. I'll try the commands as soon as quick-release ends.
ah, gotcha. you can probably fix the docker access issue by following in the docker manual. that's probably not why it's failing, though.
make release still does not work :(
can you please run and then paste the results? ( turns up the debugging verbosity.)
for me: works (using your PR to fixes the gobindata dep) -local Go build- works (docker build) works if you have Docker running. does not work: on OSX did not seem to give more verbosity.
V=4 make release did not help :(
oops, that was not the correct environment variable, sorry. I meant .
here we go:
Aha, I think I figured it out. Sorry for making you rebuild so many times; the issue still isn't super obvious from the logs. Basically, the bug only occurs if you don't have enough memory to do the docker builds in parallel; if that happens, you fall into this . On the parallel side of the branch, we set the platform env vars within a subshell, so each of the cross-compilation builds won't affect each other. On the non-parallel side of the branch, we don't use a subshell. Inside , there's no overrides done for . As a result, we end up incorrectly reusing the value for , , which doesn't recognize the flag being applied for the darwin build. The fix should be easy; I'll get that sent out ASAP. we probably want this fix for 1.5, since if you fall back to non-parallel crossbuilds we're doing the wrong thing.
Ack. Will keep an eye out for the fix. Please mark it as cherrypick-candidate
Thank you guys!
I think it is almost there. Got much further in the build (4 GB of RAM). Towards the end there is a failed test that makes the build fail: This is probably a different bug, as your PR got passed the misconfigured compilation.
FWIW, the worked on ubuntu 16.06. It did ran some tests apparently but the build succeeded.
I just tried with and it worked fine
since we have a PR ready and working, can we please switch this to non-release-blocker?

local -a nonstatics=() local -a tests=() <ins> V=2 kube::log::info "Env for ${platform}: GOOS=${GOOS-} GOARCH=${GOARCH-} GOROOT=${GOROOT-} CGO_ENABLED=${CGO_ENABLED-} CC=${CC-}" </ins> for binary in "${binaries[@]}"; do if [[ "${binary}" =~ ".test"$ ]]; then


Question:
Similar to this is to test the first contributor experience fails on ubuntu 16.04, docker version 1.12.3:
Got the same error on Linux ubuntu 4.2.0-27-generic ~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x8664 x8664 x86_64 GNU/Linux Docker Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.2.1 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.2.1 Git commit (server): OS/Arch (server): linux/amd64
What git sha of were you building from?
I have cloned the head. I think the reason might be missing package: gccgo-4.8-aarch64-linux-gnu I'm compiling right now and update on result.
were you running or ? runs inside docker, so the gcc error is confusing.
I run: sudo make release in Ubuntu VM running in VMWare workstation if that matters. Are you in kubernetes-users slack channel?
does fail in the same way?
Do you need to run , instead of just ? Can you try running the following:
well, if I don't do sudo docker does not work :) I need to run as root. I'll try the commands as soon as quick-release ends.
ah, gotcha. you can probably fix the docker access issue by following in the docker manual. that's probably not why it's failing, though.
make release still does not work :(
can you please run and then paste the results? ( turns up the debugging verbosity.)
for me: works (using your PR to fixes the gobindata dep) -local Go build- works (docker build) works if you have Docker running. does not work: on OSX did not seem to give more verbosity.
V=4 make release did not help :(
oops, that was not the correct environment variable, sorry. I meant .
here we go:
Aha, I think I figured it out. Sorry for making you rebuild so many times; the issue still isn't super obvious from the logs. Basically, the bug only occurs if you don't have enough memory to do the docker builds in parallel; if that happens, you fall into this . On the parallel side of the branch, we set the platform env vars within a subshell, so each of the cross-compilation builds won't affect each other. On the non-parallel side of the branch, we don't use a subshell. Inside , there's no overrides done for . As a result, we end up incorrectly reusing the value for , , which doesn't recognize the flag being applied for the darwin build. The fix should be easy; I'll get that sent out ASAP. we probably want this fix for 1.5, since if you fall back to non-parallel crossbuilds we're doing the wrong thing.
Ack. Will keep an eye out for the fix. Please mark it as cherrypick-candidate
Thank you guys!
I think it is almost there. Got much further in the build (4 GB of RAM). Towards the end there is a failed test that makes the build fail: This is probably a different bug, as your PR got passed the misconfigured compilation.
FWIW, the worked on ubuntu 16.06. It did ran some tests apparently but the build succeeded.
I just tried with and it worked fine
since we have a PR ready and working, can we please switch this to non-release-blocker?

else for platform in "${platforms[@]}"; do kube::log::status "Building go targets for ${platform}:" "${targets[@]}" <del> kube::golang::set_platform_envs "${platform}" kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-} </del> <ins> ( kube::golang::set_platform_envs "${platform}" kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-} ) </ins> done fi )


Question:
Similar to this is to test the first contributor experience fails on ubuntu 16.04, docker version 1.12.3:
Got the same error on Linux ubuntu 4.2.0-27-generic ~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x8664 x8664 x86_64 GNU/Linux Docker Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.2.1 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.2.1 Git commit (server): OS/Arch (server): linux/amd64
What git sha of were you building from?
I have cloned the head. I think the reason might be missing package: gccgo-4.8-aarch64-linux-gnu I'm compiling right now and update on result.
were you running or ? runs inside docker, so the gcc error is confusing.
I run: sudo make release in Ubuntu VM running in VMWare workstation if that matters. Are you in kubernetes-users slack channel?
does fail in the same way?
Do you need to run , instead of just ? Can you try running the following:
well, if I don't do sudo docker does not work :) I need to run as root. I'll try the commands as soon as quick-release ends.
ah, gotcha. you can probably fix the docker access issue by following in the docker manual. that's probably not why it's failing, though.
make release still does not work :(
can you please run and then paste the results? ( turns up the debugging verbosity.)
for me: works (using your PR to fixes the gobindata dep) -local Go build- works (docker build) works if you have Docker running. does not work: on OSX did not seem to give more verbosity.
V=4 make release did not help :(
oops, that was not the correct environment variable, sorry. I meant .
here we go:
Aha, I think I figured it out. Sorry for making you rebuild so many times; the issue still isn't super obvious from the logs. Basically, the bug only occurs if you don't have enough memory to do the docker builds in parallel; if that happens, you fall into this . On the parallel side of the branch, we set the platform env vars within a subshell, so each of the cross-compilation builds won't affect each other. On the non-parallel side of the branch, we don't use a subshell. Inside , there's no overrides done for . As a result, we end up incorrectly reusing the value for , , which doesn't recognize the flag being applied for the darwin build. The fix should be easy; I'll get that sent out ASAP. we probably want this fix for 1.5, since if you fall back to non-parallel crossbuilds we're doing the wrong thing.
Ack. Will keep an eye out for the fix. Please mark it as cherrypick-candidate
Thank you guys!
I think it is almost there. Got much further in the build (4 GB of RAM). Towards the end there is a failed test that makes the build fail: This is probably a different bug, as your PR got passed the misconfigured compilation.
FWIW, the worked on ubuntu 16.06. It did ran some tests apparently but the build succeeded.
I just tried with and it worked fine
since we have a PR ready and working, can we please switch this to non-release-blocker?

[[ -n ${1-} ]] || { kube::log::error_exit "!!! Internal error. No platform set in kube::golang::set_platform_envs" } <del> # make sure we have a clean slate first kube::golang::unset_platform_envs </del> export GOOS=${platform%/*} export GOARCH=${platform##*/}


Question:
On kubernetes 1.4.6 taken from releases. Using CentOS 7.2 Kubernetes worked with a 1.2 version. I just uninstalled and redo installation with 1.4.6. SELinux is disabled Everyhting seems to works (controller-manager, kubelet, and so on...) excepting kube-proxy that returns an error: "sysfs not mounted". API is responding (internally and externally), and I can see nodes The command that is launched by systemd is (--v=8 to get a maximum of information): But sysfs is mounted on /sys: I also tried to set /sys as writable (only to be sure, with chmod +x), and also tried to use userspace proxy mode. Note: 1.2.6 version was ok I think that the problem is that sysfs mount "none" as label, not "sysfs".
Note: I can now confirm that the problem comes from the "label". I did this: Then kube-proxy successfully started.
before setting . This logic was introduced in v1.3 by to workaround a known docker issue That's why "1.2.6 version was ok". Because sysfs is mounted as , kubeproxy couldn't find it then reported error. But why sysfs is mounted as ?
I installed kubernetes on a "scaleway" (a very nice cloud provider) VPS cluster. Their VPS uses a init file that mounts fs like that. I opened an issue on their github. But anyway you may mount "none" for tmpfs or any ramfs, sysfs, bugfs. What I think is that the mount listing function that is used in kube-proxy code tries to list the mount "source" but not the "type". For that kind of mountpoint you can use "none" as source.
the problem is here: You should check .Type and not .Device. "Device" is what I named "label" or "source" because I didn't remember the name ;) Sysfs can mount a "none" device as type "sysfs", so maybe you can change. See:
I'll do a PR
SGTM.

for _, mountPoint := range mountPoints { const sysfsDevice = "sysfs" <del> if mountPoint.Device != sysfsDevice { </del> <ins> if mountPoint.Type != sysfsDevice { </ins> continue } // Check whether sysfs is 'rw'


Question:
The test is failing in . https://k8s- https://k8s- https://k8s- The reason is that the test expects that all pause containers () should have . However, in many tests, we are starting test containers with . Those pause containers have . Ideally, containers started in previous test should be cleaned up, so in serial suite it should not affect later test. However, it is not always true when CRI is enabled. For example, in the oom message, we could see: There are 2 problems here: Why are containers not properly cleaned up when CRI enabled? I can't even find which test leaked pause container from the log, because there are many test using pause image, and some of them start a lot of containers, such as density test, restart test. The oom score adj test should not rely on process name . However, I can't find an alternative for now, because we don't report pause container status in our api, we don't even know the container id to inspect with docker or CRI. /cc
This test is still flaking in cri-serial. Some examples: https://k8s- https://k8s- Need to check the logs to see if the symptoms are the same.
It could be that with CRI, it takes kubelet longer to clean up. Or, even worse, that kubelet is not cleaning up properly. We need more information on this on failure. Dumping pods on the node Dumping containers () on the node. Accessing docker directly is not ideal, but it's okay because the test itself only applies to docker. Alternatively, we can get the current running containers from kubelet's /runningPods endpoint. The downside is that this may lag behind the actual state a little bit (seconds). We should also make the test suite run more frequently. It runs once a day, which makes the debugging round-trip overhead too high.
I can reproduce this process leak by running the "Network should recover from ip leaks" on my own node, without CRI enabled. The image used in the test was , and the docker version 1.11.2. The test creates 100 pods, then restarts the docker service by 6 times, with 20s interval in between. After the test ends, there could be multiple processes running left on the node. Those processes were reparented to pid 1, and showed no containers running. /cc
On the side note, the test should not be blindly checking all processes. It should, instead, take a snapshot of the system before the test and only check the new processes created by the tests.
I filed a new issue for the docker bug since it's not unique to CRI and this test is catching it by accident: Leaving this issue to fix the test by not blindly checking all processes:
The docker bug is not a regression, and is tracked in . The problem of the test is not necessarily to be fixed in v1.6, at least it's not release blocker. :)

"//vendor:k8s.io/apimachinery/pkg/runtime/schema", "//vendor:k8s.io/apimachinery/pkg/types", "//vendor:k8s.io/apimachinery/pkg/util/intstr", <ins> "//vendor:k8s.io/apimachinery/pkg/util/sets", </ins> "//vendor:k8s.io/apimachinery/pkg/util/uuid", "//vendor:k8s.io/apimachinery/pkg/watch", "//vendor:k8s.io/client-go/pkg/api",


Question:
The test is failing in . https://k8s- https://k8s- https://k8s- The reason is that the test expects that all pause containers () should have . However, in many tests, we are starting test containers with . Those pause containers have . Ideally, containers started in previous test should be cleaned up, so in serial suite it should not affect later test. However, it is not always true when CRI is enabled. For example, in the oom message, we could see: There are 2 problems here: Why are containers not properly cleaned up when CRI enabled? I can't even find which test leaked pause container from the log, because there are many test using pause image, and some of them start a lot of containers, such as density test, restart test. The oom score adj test should not rely on process name . However, I can't find an alternative for now, because we don't report pause container status in our api, we don't even know the container id to inspect with docker or CRI. /cc
This test is still flaking in cri-serial. Some examples: https://k8s- https://k8s- Need to check the logs to see if the symptoms are the same.
It could be that with CRI, it takes kubelet longer to clean up. Or, even worse, that kubelet is not cleaning up properly. We need more information on this on failure. Dumping pods on the node Dumping containers () on the node. Accessing docker directly is not ideal, but it's okay because the test itself only applies to docker. Alternatively, we can get the current running containers from kubelet's /runningPods endpoint. The downside is that this may lag behind the actual state a little bit (seconds). We should also make the test suite run more frequently. It runs once a day, which makes the debugging round-trip overhead too high.
I can reproduce this process leak by running the "Network should recover from ip leaks" on my own node, without CRI enabled. The image used in the test was , and the docker version 1.11.2. The test creates 100 pods, then restarts the docker service by 6 times, with 20s interval in between. After the test ends, there could be multiple processes running left on the node. Those processes were reparented to pid 1, and showed no containers running. /cc
On the side note, the test should not be blindly checking all processes. It should, instead, take a snapshot of the system before the test and only check the new processes created by the tests.
I filed a new issue for the docker bug since it's not unique to CRI and this test is catching it by accident: Leaving this issue to fix the test by not blindly checking all processes:
The docker bug is not a regression, and is tracked in . The problem of the test is not necessarily to be fixed in v1.6, at least it's not release blocker. :)

"k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/test/e2e/framework"


Question:
The test is failing in . https://k8s- https://k8s- https://k8s- The reason is that the test expects that all pause containers () should have . However, in many tests, we are starting test containers with . Those pause containers have . Ideally, containers started in previous test should be cleaned up, so in serial suite it should not affect later test. However, it is not always true when CRI is enabled. For example, in the oom message, we could see: There are 2 problems here: Why are containers not properly cleaned up when CRI enabled? I can't even find which test leaked pause container from the log, because there are many test using pause image, and some of them start a lot of containers, such as density test, restart test. The oom score adj test should not rely on process name . However, I can't find an alternative for now, because we don't report pause container status in our api, we don't even know the container id to inspect with docker or CRI. /cc
This test is still flaking in cri-serial. Some examples: https://k8s- https://k8s- Need to check the logs to see if the symptoms are the same.
It could be that with CRI, it takes kubelet longer to clean up. Or, even worse, that kubelet is not cleaning up properly. We need more information on this on failure. Dumping pods on the node Dumping containers () on the node. Accessing docker directly is not ideal, but it's okay because the test itself only applies to docker. Alternatively, we can get the current running containers from kubelet's /runningPods endpoint. The downside is that this may lag behind the actual state a little bit (seconds). We should also make the test suite run more frequently. It runs once a day, which makes the debugging round-trip overhead too high.
I can reproduce this process leak by running the "Network should recover from ip leaks" on my own node, without CRI enabled. The image used in the test was , and the docker version 1.11.2. The test creates 100 pods, then restarts the docker service by 6 times, with 20s interval in between. After the test ends, there could be multiple processes running left on the node. Those processes were reparented to pid 1, and showed no containers running. /cc
On the side note, the test should not be blindly checking all processes. It should, instead, take a snapshot of the system before the test and only check the new processes created by the tests.
I filed a new issue for the docker bug since it's not unique to CRI and this test is catching it by accident: Leaving this issue to fix the test by not blindly checking all processes:
The docker bug is not a regression, and is tracked in . The problem of the test is not necessarily to be fixed in v1.6, at least it's not release blocker. :)

}) Context("", func() { It("pod infra containers oom-score-adj should be -998 and best effort container's should be 1000", func() { <del> var err error </del> <ins> // Take a snapshot of existing pause processes. These were // created before this test, and may not be infra // containers. They should be excluded from the test. existingPausePIDs, err := getPidsForProcess("pause", "") Expect(err).To(BeNil(), "failed to list all pause processes on the node") existingPausePIDSet := sets.NewInt(existingPausePIDs...) </ins> podClient := f.PodClient() podName := "besteffort" + string(uuid.NewUUID()) podClient.Create(&v1.Pod{


Question:
The test is failing in . https://k8s- https://k8s- https://k8s- The reason is that the test expects that all pause containers () should have . However, in many tests, we are starting test containers with . Those pause containers have . Ideally, containers started in previous test should be cleaned up, so in serial suite it should not affect later test. However, it is not always true when CRI is enabled. For example, in the oom message, we could see: There are 2 problems here: Why are containers not properly cleaned up when CRI enabled? I can't even find which test leaked pause container from the log, because there are many test using pause image, and some of them start a lot of containers, such as density test, restart test. The oom score adj test should not rely on process name . However, I can't find an alternative for now, because we don't report pause container status in our api, we don't even know the container id to inspect with docker or CRI. /cc
This test is still flaking in cri-serial. Some examples: https://k8s- https://k8s- Need to check the logs to see if the symptoms are the same.
It could be that with CRI, it takes kubelet longer to clean up. Or, even worse, that kubelet is not cleaning up properly. We need more information on this on failure. Dumping pods on the node Dumping containers () on the node. Accessing docker directly is not ideal, but it's okay because the test itself only applies to docker. Alternatively, we can get the current running containers from kubelet's /runningPods endpoint. The downside is that this may lag behind the actual state a little bit (seconds). We should also make the test suite run more frequently. It runs once a day, which makes the debugging round-trip overhead too high.
I can reproduce this process leak by running the "Network should recover from ip leaks" on my own node, without CRI enabled. The image used in the test was , and the docker version 1.11.2. The test creates 100 pods, then restarts the docker service by 6 times, with 20s interval in between. After the test ends, there could be multiple processes running left on the node. Those processes were reparented to pid 1, and showed no containers running. /cc
On the side note, the test should not be blindly checking all processes. It should, instead, take a snapshot of the system before the test and only check the new processes created by the tests.
I filed a new issue for the docker bug since it's not unique to CRI and this test is catching it by accident: Leaving this issue to fix the test by not blindly checking all processes:
The docker bug is not a regression, and is tracked in . The problem of the test is not necessarily to be fixed in v1.6, at least it's not release blocker. :)

return fmt.Errorf("failed to get list of pause pids: %v", err) } for _, pid := range pausePids { <ins> if existingPausePIDSet.Has(pid) { // Not created by this test. Ignore it. continue } </ins> if err := validateOOMScoreAdjSetting(pid, -998); err != nil { return err }


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:38:40Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: None Others: None What happened: When I create a deployment with vSphere cloud provider dynamic provisioning, it creates a volume spec with volume path - [datastore] kubevols/volume which contains a white space in it. In this case the cloud provider queries the getDeviceNameFromMount() to return the path of the volume mounted. Since getDeviceNameFromMount() queries the filesystem to get the mount references, it returns a volume path "[datastore]040kubevols/volume". Later the kubelet searches for this volume path in both the actual and desired states. Th actual and desired states contains volume with path "[datastore] kubevols/volume". So, it couldn't find such volume path and therefore kubernetes stalls unable to make any progress further similar to one described in . Also similar issue described in What you expected to happen: All the pods in the recreated deployment comes up. How to reproduce it (as minimally and precisely as possible): See above. Anything else do we need to know: None
cc

"fmt" "os" "path" <ins> "strings" </ins> "github.com/golang/glog" "k8s.io/kubernetes/pkg/api/resource"


Question:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:38:40Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-29T01:32:42Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: None Others: None What happened: When I create a deployment with vSphere cloud provider dynamic provisioning, it creates a volume spec with volume path - [datastore] kubevols/volume which contains a white space in it. In this case the cloud provider queries the getDeviceNameFromMount() to return the path of the volume mounted. Since getDeviceNameFromMount() queries the filesystem to get the mount references, it returns a volume path "[datastore]040kubevols/volume". Later the kubelet searches for this volume path in both the actual and desired states. Th actual and desired states contains volume with path "[datastore] kubevols/volume". So, it couldn't find such volume path and therefore kubernetes stalls unable to make any progress further similar to one described in . Also similar issue described in What you expected to happen: All the pods in the recreated deployment comes up. How to reproduce it (as minimally and precisely as possible): See above. Anything else do we need to know: None
cc

if err != nil { return nil, err } <ins> volumePath = strings.Replace(volumePath, "040", " ", -1) </ins> glog.V(5).Infof("vSphere volume path is %q", volumePath) vsphereVolume := &v1.Volume{ Name: volumeName,


Question: From etcd3: May need to retry poststarthook a few times to avoid bootstrapping failures like this.

"//pkg/registry/rbac/rolebinding/etcd:go_default_library", "//pkg/registry/rbac/rolebinding/policybased:go_default_library", "//pkg/util/runtime:go_default_library", <ins> "//pkg/util/wait:go_default_library", </ins> "//plugin/pkg/auth/authorizer/rbac/bootstrappolicy:go_default_library", "//vendor:github.com/golang/glog", ],


Question: From etcd3: May need to retry poststarthook a few times to avoid bootstrapping failures like this.

import ( "fmt" "sync" <ins> "time" </ins> "github.com/golang/glog"


Question: From etcd3: May need to retry poststarthook a few times to avoid bootstrapping failures like this.

rolebindingetcd "k8s.io/kubernetes/pkg/registry/rbac/rolebinding/etcd" rolebindingpolicybased "k8s.io/kubernetes/pkg/registry/rbac/rolebinding/policybased" utilruntime "k8s.io/kubernetes/pkg/util/runtime" <ins> "k8s.io/kubernetes/pkg/util/wait" </ins> "k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac/bootstrappolicy" )


Question: From etcd3: May need to retry poststarthook a few times to avoid bootstrapping failures like this.

} func PostStartHook(hookContext genericapiserver.PostStartHookContext) error { <del> clientset, err := rbacclient.NewForConfig(hookContext.LoopbackClientConfig) if err != nil { utilruntime.HandleError(fmt.Errorf("unable to initialize clusterroles: %v", err)) return nil } existingClusterRoles, err := clientset.ClusterRoles().List(api.ListOptions{}) if err != nil { utilruntime.HandleError(fmt.Errorf("unable to initialize clusterroles: %v", err)) return nil } // if clusterroles already exist, then assume we don't have work to do because we've already // initialized or another API server has started this task if len(existingClusterRoles.Items) > 0 { return nil } for _, clusterRole := range append(bootstrappolicy.ClusterRoles(), bootstrappolicy.ControllerRoles()...) { if _, err := clientset.ClusterRoles().Create(&clusterRole); err != nil { // don't fail on failures, try to create as many as you can </del> <ins> // intializing roles is really important.  On some e2e runs, we've seen cases where etcd is down when the server // starts, the roles don't initialize, and nothing works. err := wait.Poll(1*time.Second, 30*time.Second, func() (done bool, err error) { clientset, err := rbacclient.NewForConfig(hookContext.LoopbackClientConfig) if err != nil { </ins> utilruntime.HandleError(fmt.Errorf("unable to initialize clusterroles: %v", err)) <del> continue </del> <ins> return false, nil </ins> } <del> glog.Infof("Created clusterrole.%s/%s", rbac.GroupName, clusterRole.Name) } existingClusterRoleBindings, err := clientset.ClusterRoleBindings().List(api.ListOptions{}) if err != nil { utilruntime.HandleError(fmt.Errorf("unable to initialize clusterrolebindings: %v", err)) return nil } // if clusterrolebindings already exist, then assume we don't have work to do because we've already // initialized or another API server has started this task if len(existingClusterRoleBindings.Items) > 0 { return nil } </del> <ins> existingClusterRoles, err := clientset.ClusterRoles().List(api.ListOptions{}) if err != nil { utilruntime.HandleError(fmt.Errorf("unable to initialize clusterroles: %v", err)) return false, nil } // only initialized on empty etcd if len(existingClusterRoles.Items) == 0 { for _, clusterRole := range append(bootstrappolicy.ClusterRoles(), bootstrappolicy.ControllerRoles()...) { if _, err := clientset.ClusterRoles().Create(&clusterRole); err != nil { // don't fail on failures, try to create as many as you can utilruntime.HandleError(fmt.Errorf("unable to initialize clusterroles: %v", err)) continue } glog.Infof("Created clusterrole.%s/%s", rbac.GroupName, clusterRole.Name) } } </ins> <del> for _, clusterRoleBinding := range append(bootstrappolicy.ClusterRoleBindings(), bootstrappolicy.ControllerRoleBindings()...) { if _, err := clientset.ClusterRoleBindings().Create(&clusterRoleBinding); err != nil { // don't fail on failures, try to create as many as you can </del> <ins> existingClusterRoleBindings, err := clientset.ClusterRoleBindings().List(api.ListOptions{}) if err != nil { </ins> utilruntime.HandleError(fmt.Errorf("unable to initialize clusterrolebindings: %v", err)) <del> continue </del> <ins> return false, nil } // only initialized on empty etcd if len(existingClusterRoleBindings.Items) == 0 { for _, clusterRoleBinding := range append(bootstrappolicy.ClusterRoleBindings(), bootstrappolicy.ControllerRoleBindings()...) { if _, err := clientset.ClusterRoleBindings().Create(&clusterRoleBinding); err != nil { // don't fail on failures, try to create as many as you can utilruntime.HandleError(fmt.Errorf("unable to initialize clusterrolebindings: %v", err)) continue } glog.Infof("Created clusterrolebinding.%s/%s", rbac.GroupName, clusterRoleBinding.Name) } </ins> } <del> glog.Infof("Created clusterrolebinding.%s/%s", rbac.GroupName, clusterRoleBinding.Name) </del> <ins> return true, nil }) // if we're never able to make it through intialization, kill the API server if err != nil { return fmt.Errorf("unable to initialize roles: %v", err) </ins> } return nil


Question:
adler when brute-forced turns out to break really fast (). There are a couple of places we use it in the repo: We should probably replace it with fnv. I tested fnv and it seems to be slightly more expensive but also a lot more stable than adler:
Just work out a patch to fix this, please take a look.
assigning this to you since the only place left to use adler is in and it's a legacy helper.
/assign
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. If the required changes are not made within 3 days, the issue will be moved out of the v1.8 milestone. kind: Must specify at most one of ['kind/bug', 'kind/feature', 'kind/cleanup']. priority: Must specify at most one of ['priority/critical-urgent', 'priority/important-soon', 'priority/important-longterm']. <detailsAdditional instructions available <a href=""here</a</details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This was removed in

package controller import ( <del> "hash/adler32" </del> <ins> "hash/fnv" </ins> "sync" "github.com/golang/groupcache/lru"


Question:
adler when brute-forced turns out to break really fast (). There are a couple of places we use it in the repo: We should probably replace it with fnv. I tested fnv and it seems to be slightly more expensive but also a lot more stable than adler:
Just work out a patch to fix this, please take a look.
assigning this to you since the only place left to use adler is in and it's a legacy helper.
/assign
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. If the required changes are not made within 3 days, the issue will be moved out of the v1.8 milestone. kind: Must specify at most one of ['kind/bug', 'kind/feature', 'kind/cleanup']. priority: Must specify at most one of ['priority/critical-urgent', 'priority/important-soon', 'priority/important-longterm']. <detailsAdditional instructions available <a href=""here</a</details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This was removed in

// Since we match objects by namespace and Labels/Selector, so if two objects have the same namespace and labels, // they will have the same key. func keyFunc(obj objectWithMeta) uint64 { <del> hash := adler32.New() </del> <ins> hash := fnv.New32a() </ins> hashutil.DeepHashObject(hash, &equivalenceLabelObj{ namespace: obj.GetNamespace(), labels:    obj.GetLabels(),


Question:
adler when brute-forced turns out to break really fast (). There are a couple of places we use it in the repo: We should probably replace it with fnv. I tested fnv and it seems to be slightly more expensive but also a lot more stable than adler:
Just work out a patch to fix this, please take a look.
assigning this to you since the only place left to use adler is in and it's a legacy helper.
/assign
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. If the required changes are not made within 3 days, the issue will be moved out of the v1.8 milestone. kind: Must specify at most one of ['kind/bug', 'kind/feature', 'kind/cleanup']. priority: Must specify at most one of ['priority/critical-urgent', 'priority/important-soon', 'priority/important-longterm']. <detailsAdditional instructions available <a href=""here</a</details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This was removed in

import ( "bufio" "fmt" <del> "hash/adler32" </del> <ins> "hash/fnv" </ins> "io" "os" "os/exec"


Question:
adler when brute-forced turns out to break really fast (). There are a couple of places we use it in the repo: We should probably replace it with fnv. I tested fnv and it seems to be slightly more expensive but also a lot more stable than adler:
Just work out a patch to fix this, please take a look.
assigning this to you since the only place left to use adler is in and it's a legacy helper.
/assign
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. If the required changes are not made within 3 days, the issue will be moved out of the v1.8 milestone. kind: Must specify at most one of ['kind/bug', 'kind/feature', 'kind/cleanup']. priority: Must specify at most one of ['priority/critical-urgent', 'priority/important-soon', 'priority/important-longterm']. <detailsAdditional instructions available <a href=""here</a</details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This was removed in

} func readProcMountsFrom(file io.Reader, out *[]MountPoint) (uint32, error) { <del> hash := adler32.New() </del> <ins> hash := fnv.New32a() </ins> scanner := bufio.NewReader(file) for { line, err := scanner.ReadString('n')


Question:
adler when brute-forced turns out to break really fast (). There are a couple of places we use it in the repo: We should probably replace it with fnv. I tested fnv and it seems to be slightly more expensive but also a lot more stable than adler:
Just work out a patch to fix this, please take a look.
assigning this to you since the only place left to use adler is in and it's a legacy helper.
/assign
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. If the required changes are not made within 3 days, the issue will be moved out of the v1.8 milestone. kind: Must specify at most one of ['kind/bug', 'kind/feature', 'kind/cleanup']. priority: Must specify at most one of ['priority/critical-urgent', 'priority/important-soon', 'priority/important-longterm']. <detailsAdditional instructions available <a href=""here</a</details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This was removed in

/dev/1    /path/to/1   type1	flags 1 1 /dev/2 /path/to/2 type2 flags,1,2=3 2 2 ` <ins> // NOTE: readProcMountsFrom has been updated to using fnv.New32a() </ins> hash, err := readProcMountsFrom(strings.NewReader(successCase), nil) if err != nil { t.Errorf("expected success") } <del> if hash != 0xa3522051 { t.Errorf("expected 0xa3522051, got %#x", hash) </del> <ins> if hash != 0xa290ff0b { t.Errorf("expected 0xa290ff0b, got %#x", hash) </ins> } mounts := []MountPoint{} hash, err = readProcMountsFrom(strings.NewReader(successCase), &mounts) if err != nil { t.Errorf("expected success") } <del> if hash != 0xa3522051 { t.Errorf("expected 0xa3522051, got %#x", hash) </del> <ins> if hash != 0xa290ff0b { t.Errorf("expected 0xa290ff0b, got %#x", hash) </ins> } if len(mounts) != 3 { t.Fatalf("expected 3 mounts, got %d", len(mounts))


Question:
adler when brute-forced turns out to break really fast (). There are a couple of places we use it in the repo: We should probably replace it with fnv. I tested fnv and it seems to be slightly more expensive but also a lot more stable than adler:
Just work out a patch to fix this, please take a look.
assigning this to you since the only place left to use adler is in and it's a legacy helper.
/assign
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. If the required changes are not made within 3 days, the issue will be moved out of the v1.8 milestone. kind: Must specify at most one of ['kind/bug', 'kind/feature', 'kind/cleanup']. priority: Must specify at most one of ['priority/critical-urgent', 'priority/important-soon', 'priority/important-longterm']. <detailsAdditional instructions available <a href=""here</a</details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This was removed in

package core import ( <del> "hash/adler32" </del> <ins> "hash/fnv" </ins> "github.com/golang/groupcache/lru"


Question:
adler when brute-forced turns out to break really fast (). There are a couple of places we use it in the repo: We should probably replace it with fnv. I tested fnv and it seems to be slightly more expensive but also a lot more stable than adler:
Just work out a patch to fix this, please take a look.
assigning this to you since the only place left to use adler is in and it's a legacy helper.
/assign
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. If the required changes are not made within 3 days, the issue will be moved out of the v1.8 milestone. kind: Must specify at most one of ['kind/bug', 'kind/feature', 'kind/cleanup']. priority: Must specify at most one of ['priority/critical-urgent', 'priority/important-soon', 'priority/important-longterm']. <detailsAdditional instructions available <a href=""here</a</details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This was removed in

// hashEquivalencePod returns the hash of equivalence pod. func (ec *EquivalenceCache) hashEquivalencePod(pod *v1.Pod) uint64 { equivalencePod := ec.getEquivalencePod(pod) <del> hash := adler32.New() </del> <ins> hash := fnv.New32a() </ins> hashutil.DeepHashObject(hash, equivalencePod) return uint64(hash.Sum32()) }


Question:
Our scalability tests started failing on Thu last week: https://k8s- [250 run passed, 251 failed and since then everything failed] I started looking into it and it seems that something happened to how we are handling watches (i.e. I'm seeing a huge number of watches failing with "err too old" errors (which I didn't see before).
As an example in run 258 of kubemark-scale, I'm seeing this log in the test (after 5 minutes of running test); This is extremely strange and unexpected. I looked into apiserver logs, and there is expected number of POST and PUT operations on pods (and no DELETE and PATCH operations until then). Until then, there were alltogethere ~60.000 operations on pods. However, in 5000-node cluster, the size of watchcache of pods is supposed to be 250.000, which means that at that moment we should have all the history from the beginning in watch cache. And still we are observing those errors. Something must have changed around that recently...
As an example, logs from apiserver from a given node: This means something is really wrong: the first watch started at 45, then after it finished, we tried starting watch from rv= and it failed with "rv too old" the list, returned rv= from which we started the next watch. If the watch cache size for pods would be 250.000 (50*cluster size), there there is no way that with current rv=, the rv= would be outside history window...
I had a hypothesis that it might be the problem of breaking watch to etcd. But in that case there should be a warning "Terminating all watchers" log in apiserver, and there is no such log. It also seems that the size of cache is being set correctly...
So I tried running large kubemark locally and it seems that the problem is that the size of cache is too small. I few more logs, and here is what I'm seeing: So it seems that we are correctly initializing cachesizes, but then watch cache capacity is 100 (which explains why we are seeing all those "rv too old" errors. I will try to debug why the capacity is so small.
OK - I found it. contains the fix.

type cacheWatcher struct { sync.Mutex copier  runtime.ObjectCopier <del> input   chan *watchCacheEvent </del> <ins> input   chan watchCacheEvent </ins> result  chan watch.Event done    chan struct{} filter  watchFilterFunc


Question:
Our scalability tests started failing on Thu last week: https://k8s- [250 run passed, 251 failed and since then everything failed] I started looking into it and it seems that something happened to how we are handling watches (i.e. I'm seeing a huge number of watches failing with "err too old" errors (which I didn't see before).
As an example in run 258 of kubemark-scale, I'm seeing this log in the test (after 5 minutes of running test); This is extremely strange and unexpected. I looked into apiserver logs, and there is expected number of POST and PUT operations on pods (and no DELETE and PATCH operations until then). Until then, there were alltogethere ~60.000 operations on pods. However, in 5000-node cluster, the size of watchcache of pods is supposed to be 250.000, which means that at that moment we should have all the history from the beginning in watch cache. And still we are observing those errors. Something must have changed around that recently...
As an example, logs from apiserver from a given node: This means something is really wrong: the first watch started at 45, then after it finished, we tried starting watch from rv= and it failed with "rv too old" the list, returned rv= from which we started the next watch. If the watch cache size for pods would be 250.000 (50*cluster size), there there is no way that with current rv=, the rv= would be outside history window...
I had a hypothesis that it might be the problem of breaking watch to etcd. But in that case there should be a warning "Terminating all watchers" log in apiserver, and there is no such log. It also seems that the size of cache is being set correctly...
So I tried running large kubemark locally and it seems that the problem is that the size of cache is too small. I few more logs, and here is what I'm seeing: So it seems that we are correctly initializing cachesizes, but then watch cache capacity is 100 (which explains why we are seeing all those "rv too old" errors. I will try to debug why the capacity is so small.
OK - I found it. contains the fix.

func newCacheWatcher(copier runtime.ObjectCopier, resourceVersion uint64, chanSize int, initEvents []*watchCacheEvent, filter watchFilterFunc, forget func(bool)) *cacheWatcher { watcher := &cacheWatcher{ copier:  copier, <del> input:   make(chan *watchCacheEvent, chanSize), </del> <ins> input:   make(chan watchCacheEvent, chanSize), </ins> result:  make(chan watch.Event, chanSize), done:    make(chan struct{}), filter:  filter,


Question:
Our scalability tests started failing on Thu last week: https://k8s- [250 run passed, 251 failed and since then everything failed] I started looking into it and it seems that something happened to how we are handling watches (i.e. I'm seeing a huge number of watches failing with "err too old" errors (which I didn't see before).
As an example in run 258 of kubemark-scale, I'm seeing this log in the test (after 5 minutes of running test); This is extremely strange and unexpected. I looked into apiserver logs, and there is expected number of POST and PUT operations on pods (and no DELETE and PATCH operations until then). Until then, there were alltogethere ~60.000 operations on pods. However, in 5000-node cluster, the size of watchcache of pods is supposed to be 250.000, which means that at that moment we should have all the history from the beginning in watch cache. And still we are observing those errors. Something must have changed around that recently...
As an example, logs from apiserver from a given node: This means something is really wrong: the first watch started at 45, then after it finished, we tried starting watch from rv= and it failed with "rv too old" the list, returned rv= from which we started the next watch. If the watch cache size for pods would be 250.000 (50*cluster size), there there is no way that with current rv=, the rv= would be outside history window...
I had a hypothesis that it might be the problem of breaking watch to etcd. But in that case there should be a warning "Terminating all watchers" log in apiserver, and there is no such log. It also seems that the size of cache is being set correctly...
So I tried running large kubemark locally and it seems that the problem is that the size of cache is too small. I few more logs, and here is what I'm seeing: So it seems that we are correctly initializing cachesizes, but then watch cache capacity is 100 (which explains why we are seeing all those "rv too old" errors. I will try to debug why the capacity is so small.
OK - I found it. contains the fix.

func (c *cacheWatcher) add(event *watchCacheEvent, budget *timeBudget) { // Try to send the event immediately, without blocking. select { <del> case c.input <- event: </del> <ins> case c.input <- *event: </ins> return default: }


Question:
Our scalability tests started failing on Thu last week: https://k8s- [250 run passed, 251 failed and since then everything failed] I started looking into it and it seems that something happened to how we are handling watches (i.e. I'm seeing a huge number of watches failing with "err too old" errors (which I didn't see before).
As an example in run 258 of kubemark-scale, I'm seeing this log in the test (after 5 minutes of running test); This is extremely strange and unexpected. I looked into apiserver logs, and there is expected number of POST and PUT operations on pods (and no DELETE and PATCH operations until then). Until then, there were alltogethere ~60.000 operations on pods. However, in 5000-node cluster, the size of watchcache of pods is supposed to be 250.000, which means that at that moment we should have all the history from the beginning in watch cache. And still we are observing those errors. Something must have changed around that recently...
As an example, logs from apiserver from a given node: This means something is really wrong: the first watch started at 45, then after it finished, we tried starting watch from rv= and it failed with "rv too old" the list, returned rv= from which we started the next watch. If the watch cache size for pods would be 250.000 (50*cluster size), there there is no way that with current rv=, the rv= would be outside history window...
I had a hypothesis that it might be the problem of breaking watch to etcd. But in that case there should be a warning "Terminating all watchers" log in apiserver, and there is no such log. It also seems that the size of cache is being set correctly...
So I tried running large kubemark locally and it seems that the problem is that the size of cache is too small. I few more logs, and here is what I'm seeing: So it seems that we are correctly initializing cachesizes, but then watch cache capacity is 100 (which explains why we are seeing all those "rv too old" errors. I will try to debug why the capacity is so small.
OK - I found it. contains the fix.

defer timerPool.Put(t) select { <del> case c.input <- event: </del> <ins> case c.input <- *event: </ins> stopped := t.Stop() if !stopped { // Consume triggered (but not yet received) timer event


Question:
Our scalability tests started failing on Thu last week: https://k8s- [250 run passed, 251 failed and since then everything failed] I started looking into it and it seems that something happened to how we are handling watches (i.e. I'm seeing a huge number of watches failing with "err too old" errors (which I didn't see before).
As an example in run 258 of kubemark-scale, I'm seeing this log in the test (after 5 minutes of running test); This is extremely strange and unexpected. I looked into apiserver logs, and there is expected number of POST and PUT operations on pods (and no DELETE and PATCH operations until then). Until then, there were alltogethere ~60.000 operations on pods. However, in 5000-node cluster, the size of watchcache of pods is supposed to be 250.000, which means that at that moment we should have all the history from the beginning in watch cache. And still we are observing those errors. Something must have changed around that recently...
As an example, logs from apiserver from a given node: This means something is really wrong: the first watch started at 45, then after it finished, we tried starting watch from rv= and it failed with "rv too old" the list, returned rv= from which we started the next watch. If the watch cache size for pods would be 250.000 (50*cluster size), there there is no way that with current rv=, the rv= would be outside history window...
I had a hypothesis that it might be the problem of breaking watch to etcd. But in that case there should be a warning "Terminating all watchers" log in apiserver, and there is no such log. It also seems that the size of cache is being set correctly...
So I tried running large kubemark locally and it seems that the problem is that the size of cache is too small. I few more logs, and here is what I'm seeing: So it seems that we are correctly initializing cachesizes, but then watch cache capacity is 100 (which explains why we are seeing all those "rv too old" errors. I will try to debug why the capacity is so small.
OK - I found it. contains the fix.

} // only send events newer than resourceVersion if event.ResourceVersion > resourceVersion { <del> c.sendWatchCacheEvent(event) </del> <ins> c.sendWatchCacheEvent(&event) </ins> } } }


Question:
The description of says: However the implementation will interpret the name as a regex. So for example "..." will match any three-letter string. This is not what I understand by "exact match".
Will open an PR fix this comment

return utilerrors.NewAggregate(errList) } <del> // Find process(es) with a specified name (exact match) </del> <ins> // Find process(es) with a specified name (regexp match) </ins> // and return their pid(s) func PidOf(name string) ([]int, error) { if len(name) == 0 {


Question: If in line line is executed also for step 2.2.1 if the tool failed during first run of upgrade. I guess we should also check for the step and compare it to version.

exit 1 fi <ins> echo "$(date +'%Y-%m-%d %H:%M:%S') Detecting if migration is needed" </ins> if [ "${TARGET_STORAGE}" != "etcd2" -a "${TARGET_STORAGE}" != "etcd3" ]; then echo "Not supported version of storage: ${TARGET_STORAGE}" exit 1


Question: If in line line is executed also for step 2.2.1 if the tool failed during first run of upgrade. I guess we should also check for the step and compare it to version.

CURRENT_VERSION=${step} echo "${CURRENT_VERSION}/${CURRENT_STORAGE}" > "${DATA_DIRECTORY}/${VERSION_FILE}" fi <del> if [ "$(echo ${CURRENT_VERSION} | cut -c1-2)" = "3." -a "${CURRENT_STORAGE}" = "etcd2" -a "${TARGET_STORAGE}" = "etcd3" ]; then </del> <ins> if [ "$(echo ${CURRENT_VERSION} | cut -c1-2)" = "3." -a "${CURRENT_VERSION}" = "${step}" -a "${CURRENT_STORAGE}" = "etcd2" -a "${TARGET_STORAGE}" = "etcd3" ]; then </ins> # If it is the first 3.x release in the list and we are migrating # also from 'etcd2' to 'etcd3', do the migration now. echo "Performing etcd2 -> etcd3 migration"


Question: If in line line is executed also for step 2.2.1 if the tool failed during first run of upgrade. I guess we should also check for the step and compare it to version.

CURRENT_VERSION="2.3.7" echo "${CURRENT_VERSION}/${CURRENT_STORAGE}" > "${DATA_DIRECTORY}/${VERSION_FILE}" fi <ins> echo "$(date +'%Y-%m-%d %H:%M:%S') Migration finished" </ins>


Question: At the very least it should be clear when (timestamp) the migration started and finished.


Question: At the very least it should be clear when (timestamp) the migration started and finished.


Question: At the very least it should be clear when (timestamp) the migration started and finished.


Question:
https://k8s- do you know the statefulset owners?
cc
I did not see this happening until recently.
The controller misses that the first pod transitioned to unready and proceeds to create the second pod. The test uses a client to hit the api server and will scale up the statefulset as soon as the broken pod is observed but the controller uses caches so it may observe the scaled up statefulset quicker than the updated pod.
I will update the test to ensure that we wait until the unreadiness is observed on StatefulSet.Status.Replicas field prior to the scaling operation
SGTM

} } <del> func (s *StatefulSetTester) waitForStatus(ss *apps.StatefulSet, expectedReplicas int32) { </del> <ins> // WaitForStatus waits for the ss.Status.Replicas to be equal to expectedReplicas func (s *StatefulSetTester) WaitForStatus(ss *apps.StatefulSet, expectedReplicas int32) { </ins> Logf("Waiting for statefulset status.replicas updated to %d", expectedReplicas) ns, name := ss.Namespace, ss.Name


Question:
https://k8s- do you know the statefulset owners?
cc
I did not see this happening until recently.
The controller misses that the first pod transitioned to unready and proceeds to create the second pod. The test uses a client to hit the api server and will scale up the statefulset as soon as the broken pod is observed but the controller uses caches so it may observe the scaled up statefulset quicker than the updated pod.
I will update the test to ensure that we wait until the unreadiness is observed on StatefulSet.Status.Replicas field prior to the scaling operation
SGTM

if err != nil { return false, err } <ins> if *ssGet.Status.ObservedGeneration < ss.Generation { return false, nil } </ins> if ssGet.Status.Replicas != expectedReplicas { Logf("Waiting for stateful set status to become %d, currently %d", expectedReplicas, ssGet.Status.Replicas) return false, nil


Question:
https://k8s- do you know the statefulset owners?
cc
I did not see this happening until recently.
The controller misses that the first pod transitioned to unready and proceeds to create the second pod. The test uses a client to hit the api server and will scale up the statefulset as soon as the broken pod is observed but the controller uses caches so it may observe the scaled up statefulset quicker than the updated pod.
I will update the test to ensure that we wait until the unreadiness is observed on StatefulSet.Status.Replicas field prior to the scaling operation
SGTM

if err := sst.Scale(&ss, 0); err != nil { errList = append(errList, fmt.Sprintf("%v", err)) } <del> sst.waitForStatus(&ss, 0) </del> <ins> sst.WaitForStatus(&ss, 0) </ins> Logf("Deleting statefulset %v", ss.Name) if err := c.Apps().StatefulSets(ss.Namespace).Delete(ss.Name, nil); err != nil { errList = append(errList, fmt.Sprintf("%v", err))


Question:
https://k8s- do you know the statefulset owners?
cc
I did not see this happening until recently.
The controller misses that the first pod transitioned to unready and proceeds to create the second pod. The test uses a client to hit the api server and will scale up the statefulset as soon as the broken pod is observed but the controller uses caches so it may observe the scaled up statefulset quicker than the updated pod.
I will update the test to ensure that we wait until the unreadiness is observed on StatefulSet.Status.Replicas field prior to the scaling operation
SGTM

By("Before scale up finished setting 2nd pod to be not ready by breaking readiness probe") sst.BreakProbe(ss, testProbe) <ins> sst.WaitForStatus(ss, 0) </ins> sst.WaitForRunningAndNotReady(2, ss) By("Continue scale operation after the 2nd pod, and scaling down to 1 replica")


Question:
https://k8s- do you know the statefulset owners?
cc
I did not see this happening until recently.
The controller misses that the first pod transitioned to unready and proceeds to create the second pod. The test uses a client to hit the api server and will scale up the statefulset as soon as the broken pod is observed but the controller uses caches so it may observe the scaled up statefulset quicker than the updated pod.
I will update the test to ensure that we wait until the unreadiness is observed on StatefulSet.Status.Replicas field prior to the scaling operation
SGTM

By("Confirming that stateful set scale up will halt with unhealthy stateful pod") sst.BreakProbe(ss, testProbe) sst.WaitForRunningAndNotReady(*ss.Spec.Replicas, ss) <ins> sst.WaitForStatus(ss, 0) </ins> sst.UpdateReplicas(ss, 3) sst.ConfirmStatefulPodCount(1, ss, 10*time.Second)


Question:
https://k8s- do you know the statefulset owners?
cc
I did not see this happening until recently.
The controller misses that the first pod transitioned to unready and proceeds to create the second pod. The test uses a client to hit the api server and will scale up the statefulset as soon as the broken pod is observed but the controller uses caches so it may observe the scaled up statefulset quicker than the updated pod.
I will update the test to ensure that we wait until the unreadiness is observed on StatefulSet.Status.Replicas field prior to the scaling operation
SGTM

Expect(err).NotTo(HaveOccurred()) sst.BreakProbe(ss, testProbe) <ins> sst.WaitForStatus(ss, 0) </ins> sst.WaitForRunningAndNotReady(3, ss) sst.UpdateReplicas(ss, 0) sst.ConfirmStatefulPodCount(3, ss, 10*time.Second)


Question:
https://k8s- replica set controller maybe?
The rc seems to be updated almost a second after the failure. Maybe the test needs to retry
And the real log for the failure is
/assign flakes PRs at least once a day The can filter down to focus on a single test:
For when you have some time.
Reassign to you
Output of has been change in 1.6. That's why upgrade tests are failing. But I'm not sure why it is not failing 100% from the triage board. Do you know why? Output of 1.5 or earlier: Output in 1.6 or later:
The timeout is now. Is it reasonable to give it a little more time?
That timeout waits for the pods to run based on the API server. The rc controller is using its own cache so it may lag a bit before it actually updates the rc status. The test needs to retry for some time if the output we get back from describe does not match.
In other word, should also have its own retries.
SGTM. Going to send a fix today.
FYI, this test continues to flake PRs at least once per day:
should it show up in
Hm, this seems to fail in the upgrade tests. this failure seems to be related to an image name change as you already pointed out before. Mind finding what changed the test to use that image and fix the image name?
It is actually the key name has been changed in the key-value pair, not the value. As I pointed out in before expected key: now expected key:
Hm, this is unfortunate. These kubectl tests should really be unit tests.
the 24 hr test history is largely focused on failures right now, and this is highlighting flakes specifically (aka which tests are both passing and failing at the same commit the most often). Please follow the reports I'm sending to kubernetes-dev and related sigs. Also you can generate this info yourself using PS: also try out for finding failures (I'm sending out a separate email highlighting the biggest sources of failure). PPS: we are in the process of creating infrastructure to automate creating these reports, but automation never works as well as a fellow human asking for attention.
We still need to patch the release-1.4 and release-1.5 branch to make the upgrade tests passing. I will send PRs later this week.
We didn't fix
The change mentioned in was introduced in 1.6. The change covers describer and kubectl e2e test. After discussing with we think we are not going to backport the changes to 1.4 and 1.5, instead we are going to skip the tests in version skew cases. WDYT?
should fix this issue
looks like the PR is resolved, should we close?
Please wait until fix to be cherrypick into 1.4.
Sure! Which PR is this?
Is it possible to cherrypick to 1.4 by just changing the milestone to 1.4?
no you need to create a new PR with the target branch of 1.4
Ready to close?
Yup. /close

}) // Rc <del> output := framework.RunKubectlOrDie("describe", "rc", "redis-master", nsFlag) </del> requiredStrings := [][]string{ {"Name:", "redis-master"}, {"Namespace:", ns},


Question:
https://k8s- replica set controller maybe?
The rc seems to be updated almost a second after the failure. Maybe the test needs to retry
And the real log for the failure is
/assign flakes PRs at least once a day The can filter down to focus on a single test:
For when you have some time.
Reassign to you
Output of has been change in 1.6. That's why upgrade tests are failing. But I'm not sure why it is not failing 100% from the triage board. Do you know why? Output of 1.5 or earlier: Output in 1.6 or later:
The timeout is now. Is it reasonable to give it a little more time?
That timeout waits for the pods to run based on the API server. The rc controller is using its own cache so it may lag a bit before it actually updates the rc status. The test needs to retry for some time if the output we get back from describe does not match.
In other word, should also have its own retries.
SGTM. Going to send a fix today.
FYI, this test continues to flake PRs at least once per day:
should it show up in
Hm, this seems to fail in the upgrade tests. this failure seems to be related to an image name change as you already pointed out before. Mind finding what changed the test to use that image and fix the image name?
It is actually the key name has been changed in the key-value pair, not the value. As I pointed out in before expected key: now expected key:
Hm, this is unfortunate. These kubectl tests should really be unit tests.
the 24 hr test history is largely focused on failures right now, and this is highlighting flakes specifically (aka which tests are both passing and failing at the same commit the most often). Please follow the reports I'm sending to kubernetes-dev and related sigs. Also you can generate this info yourself using PS: also try out for finding failures (I'm sending out a separate email highlighting the biggest sources of failure). PPS: we are in the process of creating infrastructure to automate creating these reports, but automation never works as well as a fellow human asking for attention.
We still need to patch the release-1.4 and release-1.5 branch to make the upgrade tests passing. I will send PRs later this week.
We didn't fix
The change mentioned in was introduced in 1.6. The change covers describer and kubectl e2e test. After discussing with we think we are not going to backport the changes to 1.4 and 1.5, instead we are going to skip the tests in version skew cases. WDYT?
should fix this issue
looks like the PR is resolved, should we close?
Please wait until fix to be cherrypick into 1.4.
Sure! Which PR is this?
Is it possible to cherrypick to 1.4 by just changing the milestone to 1.4?
no you need to create a new PR with the target branch of 1.4
Ready to close?
Yup. /close

{"Pod Template:"}, {"Image:", redisImage}, {"Events:"}} <del> checkOutput(output, requiredStrings) </del> <ins> checkKubectlOutputWithRetry(requiredStrings, "describe", "rc", "redis-master", nsFlag) </ins> // Service <del> output = framework.RunKubectlOrDie("describe", "service", "redis-master", nsFlag) </del> <ins> output := framework.RunKubectlOrDie("describe", "service", "redis-master", nsFlag) </ins> requiredStrings = [][]string{ {"Name:", "redis-master"}, {"Namespace:", ns},


Question:
https://k8s- replica set controller maybe?
The rc seems to be updated almost a second after the failure. Maybe the test needs to retry
And the real log for the failure is
/assign flakes PRs at least once a day The can filter down to focus on a single test:
For when you have some time.
Reassign to you
Output of has been change in 1.6. That's why upgrade tests are failing. But I'm not sure why it is not failing 100% from the triage board. Do you know why? Output of 1.5 or earlier: Output in 1.6 or later:
The timeout is now. Is it reasonable to give it a little more time?
That timeout waits for the pods to run based on the API server. The rc controller is using its own cache so it may lag a bit before it actually updates the rc status. The test needs to retry for some time if the output we get back from describe does not match.
In other word, should also have its own retries.
SGTM. Going to send a fix today.
FYI, this test continues to flake PRs at least once per day:
should it show up in
Hm, this seems to fail in the upgrade tests. this failure seems to be related to an image name change as you already pointed out before. Mind finding what changed the test to use that image and fix the image name?
It is actually the key name has been changed in the key-value pair, not the value. As I pointed out in before expected key: now expected key:
Hm, this is unfortunate. These kubectl tests should really be unit tests.
the 24 hr test history is largely focused on failures right now, and this is highlighting flakes specifically (aka which tests are both passing and failing at the same commit the most often). Please follow the reports I'm sending to kubernetes-dev and related sigs. Also you can generate this info yourself using PS: also try out for finding failures (I'm sending out a separate email highlighting the biggest sources of failure). PPS: we are in the process of creating infrastructure to automate creating these reports, but automation never works as well as a fellow human asking for attention.
We still need to patch the release-1.4 and release-1.5 branch to make the upgrade tests passing. I will send PRs later this week.
We didn't fix
The change mentioned in was introduced in 1.6. The change covers describer and kubectl e2e test. After discussing with we think we are not going to backport the changes to 1.4 and 1.5, instead we are going to skip the tests in version skew cases. WDYT?
should fix this issue
looks like the PR is resolved, should we close?
Please wait until fix to be cherrypick into 1.4.
Sure! Which PR is this?
Is it possible to cherrypick to 1.4 by just changing the milestone to 1.4?
no you need to create a new PR with the target branch of 1.4
Ready to close?
Yup. /close

}) // Checks whether the output split by line contains the required elements. <del> func checkOutput(output string, required [][]string) { </del> <ins> func checkOutputReturnError(output string, required [][]string) error { </ins> outputLines := strings.Split(output, "n") currentLine := 0 for _, requirement := range required {


Question:
https://k8s- replica set controller maybe?
The rc seems to be updated almost a second after the failure. Maybe the test needs to retry
And the real log for the failure is
/assign flakes PRs at least once a day The can filter down to focus on a single test:
For when you have some time.
Reassign to you
Output of has been change in 1.6. That's why upgrade tests are failing. But I'm not sure why it is not failing 100% from the triage board. Do you know why? Output of 1.5 or earlier: Output in 1.6 or later:
The timeout is now. Is it reasonable to give it a little more time?
That timeout waits for the pods to run based on the API server. The rc controller is using its own cache so it may lag a bit before it actually updates the rc status. The test needs to retry for some time if the output we get back from describe does not match.
In other word, should also have its own retries.
SGTM. Going to send a fix today.
FYI, this test continues to flake PRs at least once per day:
should it show up in
Hm, this seems to fail in the upgrade tests. this failure seems to be related to an image name change as you already pointed out before. Mind finding what changed the test to use that image and fix the image name?
It is actually the key name has been changed in the key-value pair, not the value. As I pointed out in before expected key: now expected key:
Hm, this is unfortunate. These kubectl tests should really be unit tests.
the 24 hr test history is largely focused on failures right now, and this is highlighting flakes specifically (aka which tests are both passing and failing at the same commit the most often). Please follow the reports I'm sending to kubernetes-dev and related sigs. Also you can generate this info yourself using PS: also try out for finding failures (I'm sending out a separate email highlighting the biggest sources of failure). PPS: we are in the process of creating infrastructure to automate creating these reports, but automation never works as well as a fellow human asking for attention.
We still need to patch the release-1.4 and release-1.5 branch to make the upgrade tests passing. I will send PRs later this week.
We didn't fix
The change mentioned in was introduced in 1.6. The change covers describer and kubectl e2e test. After discussing with we think we are not going to backport the changes to 1.4 and 1.5, instead we are going to skip the tests in version skew cases. WDYT?
should fix this issue
looks like the PR is resolved, should we close?
Please wait until fix to be cherrypick into 1.4.
Sure! Which PR is this?
Is it possible to cherrypick to 1.4 by just changing the milestone to 1.4?
no you need to create a new PR with the target branch of 1.4
Ready to close?
Yup. /close

currentLine++ } if currentLine == len(outputLines) { <del> framework.Failf("Failed to find %s in %s", requirement[0], output) </del> <ins> return fmt.Errorf("failed to find %s in %s", requirement[0], output) </ins> } for _, item := range requirement[1:] { if !strings.Contains(outputLines[currentLine], item) { <del> framework.Failf("Failed to find %s in %s", item, outputLines[currentLine]) </del> <ins> return fmt.Errorf("failed to find %s in %s", item, outputLines[currentLine]) </ins> } } } <ins> return nil } func checkOutput(output string, required [][]string) { err := checkOutputReturnError(output, required) if err != nil { framework.Failf("%v", err) } } func checkKubectlOutputWithRetry(required [][]string, args ...string) { var pollErr error wait.PollImmediate(time.Second, time.Minute, func() (bool, error) { output := framework.RunKubectlOrDie(args...) err := checkOutputReturnError(output, required) if err != nil { pollErr = err return false, nil } pollErr = nil return true, nil }) if pollErr != nil { framework.Failf("%v", pollErr) } return </ins> } func getAPIVersions(apiEndpoint string) (*metav1.APIVersions, error) {


Question:
We need to gracefully handle the case when a Deployment specifies an absolute value of MaxUnavailable larger than its spec.replicas. I don't think we want to add validation for this, simply collapsing maxUnavailable to spec.replicas would suffice.
Are you working on this? I am glad to take this if not.
feel free to work on it

} // Error caught by validation _, maxUnavailable, _ := ResolveFenceposts(deployment.Spec.Strategy.RollingUpdate.MaxSurge, deployment.Spec.Strategy.RollingUpdate.MaxUnavailable, *(deployment.Spec.Replicas)) <ins> if maxUnavailable > *deployment.Spec.Replicas { return *deployment.Spec.Replicas } </ins> return maxUnavailable }


Question:
We need to gracefully handle the case when a Deployment specifies an absolute value of MaxUnavailable larger than its spec.replicas. I don't think we want to add validation for this, simply collapsing maxUnavailable to spec.replicas would suffice.
Are you working on this? I am glad to take this if not.
feel free to work on it

} } } <ins> func TestMaxUnavailable(t *testing.T) { deployment := func(replicas int32, maxUnavailable intstr.IntOrString) extensions.Deployment { return extensions.Deployment{ Spec: extensions.DeploymentSpec{ Replicas: func(i int32) *int32 { return &i }(replicas), Strategy: extensions.DeploymentStrategy{ RollingUpdate: &extensions.RollingUpdateDeployment{ MaxSurge:       func(i int) *intstr.IntOrString { x := intstr.FromInt(i); return &x }(int(1)), MaxUnavailable: &maxUnavailable, }, Type: extensions.RollingUpdateDeploymentStrategyType, }, }, } } tests := []struct { name       string deployment extensions.Deployment expected   int32 }{ { name:       "maxUnavailable less than replicas", deployment: deployment(10, intstr.FromInt(5)), expected:   int32(5), }, { name:       "maxUnavailable equal replicas", deployment: deployment(10, intstr.FromInt(10)), expected:   int32(10), }, { name:       "maxUnavailable greater than replicas", deployment: deployment(5, intstr.FromInt(10)), expected:   int32(5), }, { name:       "maxUnavailable with replicas is 0", deployment: deployment(0, intstr.FromInt(10)), expected:   int32(0), }, { name: "maxUnavailable with Recreate deployment strategy", deployment: extensions.Deployment{ Spec: extensions.DeploymentSpec{ Strategy: extensions.DeploymentStrategy{ Type: extensions.RecreateDeploymentStrategyType, }, }, }, expected: int32(0), }, { name:       "maxUnavailable less than replicas with percents", deployment: deployment(10, intstr.FromString("50%")), expected:   int32(5), }, { name:       "maxUnavailable equal replicas with percents", deployment: deployment(10, intstr.FromString("100%")), expected:   int32(10), }, { name:       "maxUnavailable greater than replicas with percents", deployment: deployment(5, intstr.FromString("100%")), expected:   int32(5), }, } for _, test := range tests { t.Log(test.name) maxUnavailable := MaxUnavailable(test.deployment) if test.expected != maxUnavailable { t.Fatalf("expected:%v, got:%v", test.expected, maxUnavailable) } } } </ins>


Question: When patching, in the type decoded from ToUnstructured mismatch with the one decoded form json.Unmarshal for numbers. Example: port: 80 in original object has type int64 port: 80 in original object has type float64 This is the root cause of

package handlers import ( <del> "encoding/json" </del> "fmt" "k8s.io/apimachinery/pkg/conversion/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/json" </ins> "k8s.io/apimachinery/pkg/util/strategicpatch" "github.com/evanphx/json-patch"


Question: When patching, in the type decoded from ToUnstructured mismatch with the one decoded form json.Unmarshal for numbers. Example: port: 80 in original object has type int64 port: 80 in original object has type float64 This is the root cause of

} <ins> func TestNumberConversion(t *testing.T) { codec := api.Codecs.LegacyCodec(schema.GroupVersion{Version: "v1"}) currentVersionedObject := &v1.Service{ TypeMeta:   metav1.TypeMeta{Kind: "Service", APIVersion: "v1"}, ObjectMeta: metav1.ObjectMeta{Name: "test-service"}, Spec: v1.ServiceSpec{ Ports: []v1.ServicePort{ { Port:     80, Protocol: "TCP", NodePort: 31678, }, }, }, } versionedObjToUpdate := &v1.Service{} versionedObj := &v1.Service{} patchJS := []byte(`{"spec":{"ports":[{"port":80,"nodePort":31789}]}}`) _, _, err := strategicPatchObject(codec, currentVersionedObject, patchJS, versionedObjToUpdate, versionedObj) if err != nil { t.Fatal(err) } ports := versionedObjToUpdate.Spec.Ports if len(ports) != 1 || ports[0].Port != 80 || ports[0].NodePort != 31789 { t.Fatal(errors.New("Ports failed to merge because of number conversion issue")) } } </ins> func TestPatchResourceWithVersionConflict(t *testing.T) { namespace := "bar" name := "foo"


Question: When patching, in the type decoded from ToUnstructured mismatch with the one decoded form json.Unmarshal for numbers. Example: port: 80 in original object has type int64 port: 80 in original object has type float64 This is the root cause of

"//vendor:k8s.io/apimachinery/pkg/runtime/serializer/streaming", "//vendor:k8s.io/apimachinery/pkg/types", "//vendor:k8s.io/apimachinery/pkg/util/httpstream", <ins> "//vendor:k8s.io/apimachinery/pkg/util/json", </ins> "//vendor:k8s.io/apimachinery/pkg/util/mergepatch", "//vendor:k8s.io/apimachinery/pkg/util/net", "//vendor:k8s.io/apimachinery/pkg/util/runtime",


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Kubernetes version (use ): master What happened: The functionality is hidden in a large function, and not explicit, testable. What you expected to happen: The logic of should read more clearly, with tests for individual filters.
has a PR for this .
Few more things include removing limits in entries like
yeah, im not sure why the thing is there, after all they are just appending to the slice and the limit wont have an effect , from what i can tell, will it? probably good idea to remove unless im missing something.
Its part of part of PR too.

// Returns selectors of services, RCs and RSs matching the given pod. func getSelectors(pod *v1.Pod, sl algorithm.ServiceLister, cl algorithm.ControllerLister, rsl algorithm.ReplicaSetLister, ssl algorithm.StatefulSetLister) []labels.Selector { <del> selectors := make([]labels.Selector, 0, 3) </del> <ins> var selectors []labels.Selector </ins> if services, err := sl.GetPodServices(pod); err == nil { for _, service := range services { selectors = append(selectors, labels.SelectorFromSet(service.Spec.Selector))


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Kubernetes version (use ): master What happened: The functionality is hidden in a large function, and not explicit, testable. What you expected to happen: The logic of should read more clearly, with tests for individual filters.
has a PR for this .
Few more things include removing limits in entries like
yeah, im not sure why the thing is there, after all they are just appending to the slice and the limit wont have an effect , from what i can tell, will it? probably good idea to remove unless im missing something.
Its part of part of PR too.

return antiAffinity.CalculateAntiAffinityPriority } <ins> // Classifies nodes into ones with labels and without labels. func (s *ServiceAntiAffinity) getNodeClassificationByLabels(nodes []*v1.Node) (map[string]string, []string) { labeledNodes := map[string]string{} nonLabeledNodes := []string{} for _, node := range nodes { if labels.Set(node.Labels).Has(s.label) { label := labels.Set(node.Labels).Get(s.label) labeledNodes[node.Name] = label } else { nonLabeledNodes = append(nonLabeledNodes, node.Name) } } return labeledNodes, nonLabeledNodes } </ins> // CalculateAntiAffinityPriority spreads pods by minimizing the number of pods belonging to the same service // on machines with the same value for a particular label. // The label to be considered is provided to the struct (ServiceAntiAffinity).


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Kubernetes version (use ): master What happened: The functionality is hidden in a large function, and not explicit, testable. What you expected to happen: The logic of should read more clearly, with tests for individual filters.
has a PR for this .
Few more things include removing limits in entries like
yeah, im not sure why the thing is there, after all they are just appending to the slice and the limit wont have an effect , from what i can tell, will it? probably good idea to remove unless im missing something.
Its part of part of PR too.

} // separate out the nodes that have the label from the ones that don't <del> otherNodes := []string{} labeledNodes := map[string]string{} for _, node := range nodes { if labels.Set(node.Labels).Has(s.label) { label := labels.Set(node.Labels).Get(s.label) labeledNodes[node.Name] = label } else { otherNodes = append(otherNodes, node.Name) } } </del> <ins> labeledNodes, nonLabeledNodes := s.getNodeClassificationByLabels(nodes) </ins> podCounts := map[string]int{} for _, pod := range nsServicePods { label, exists := labeledNodes[pod.Spec.NodeName]


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Kubernetes version (use ): master What happened: The functionality is hidden in a large function, and not explicit, testable. What you expected to happen: The logic of should read more clearly, with tests for individual filters.
has a PR for this .
Few more things include removing limits in entries like
yeah, im not sure why the thing is there, after all they are just appending to the slice and the limit wont have an effect , from what i can tell, will it? probably good idea to remove unless im missing something.
Its part of part of PR too.

} podCounts[label]++ } numServicePods := len(nsServicePods) result := []schedulerapi.HostPriority{} //score int - scale of 0-maxPriority


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Kubernetes version (use ): master What happened: The functionality is hidden in a large function, and not explicit, testable. What you expected to happen: The logic of should read more clearly, with tests for individual filters.
has a PR for this .
Few more things include removing limits in entries like
yeah, im not sure why the thing is there, after all they are just appending to the slice and the limit wont have an effect , from what i can tell, will it? probably good idea to remove unless im missing something.
Its part of part of PR too.

result = append(result, schedulerapi.HostPriority{Host: node, Score: int(fScore)}) } // add the open nodes with a score of 0 <del> for _, node := range otherNodes { </del> <ins> for _, node := range nonLabeledNodes { </ins> result = append(result, schedulerapi.HostPriority{Host: node, Score: 0}) } return result, nil }


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Kubernetes version (use ): master What happened: The functionality is hidden in a large function, and not explicit, testable. What you expected to happen: The logic of should read more clearly, with tests for individual filters.
has a PR for this .
Few more things include removing limits in entries like
yeah, im not sure why the thing is there, after all they are just appending to the slice and the limit wont have an effect , from what i can tell, will it? probably good idea to remove unless im missing something.
Its part of part of PR too.

} } <ins> func TestGetNodeClassificationByLabels(t *testing.T) { const machine01 = "machine01" const machine02 = "machine02" const zoneA = "zoneA" zone1 := map[string]string{ "zone": zoneA, } labeledNodes := map[string]map[string]string{ machine01: zone1, } expectedNonLabeledNodes := []string{machine02} serviceAffinity := ServiceAntiAffinity{label: "zone"} newLabeledNodes, noNonLabeledNodes := serviceAffinity.getNodeClassificationByLabels(makeLabeledNodeList(labeledNodes)) noLabeledNodes, newnonLabeledNodes := serviceAffinity.getNodeClassificationByLabels(makeNodeList(expectedNonLabeledNodes)) label, _ := newLabeledNodes[machine01] if label != zoneA && len(noNonLabeledNodes) != 0 { t.Errorf("Expected only labeled node with label zoneA and no noNonLabeledNodes") } if len(noLabeledNodes) != 0 && newnonLabeledNodes[0] != machine02 { t.Errorf("Expected only non labled nodes") } } </ins> func makeLabeledNodeList(nodeMap map[string]map[string]string) []*v1.Node { nodes := make([]*v1.Node, 0, len(nodeMap)) for nodeName, labels := range nodeMap {