Question:
Both the GKE and Parallel tests recently had a run where they timed out. (after 90 minutes!) This is like 10x longer than the parallel run should be taking.
And the build project, too:
any ideas?
p0
yup, investigating this is next on my queue, right after I figure out why the PR jenkins is failing so many runs.
This happened again: In fact I've seen at least few cases like that: "at some point tests just stop and completely nothing is happening to the end" The same happened here - completely nothing happened during last hour of the run,
While the failure rates of the individual test jobs is not very high (e.g. kubernetes-e2e-gce-parallel : MTTF (Last 7 Days) 7 hr 27 min ... the fact that we gate auto-merging on about 7 of these jobs means that a merge fails almost every hour due to this.
I looked at 4 of the builds in kubernetes-e2e-gce-parallel, comparing the test cases marked "N/A" (i.e. no test result given) instead of their usual "PASSED" or "SKIPPED" report. So far, there's been only one test case that's been consistently N/A in all four of the builds I looked at: I'm not sure what might be happening, but I'm tempted to disable this test in parallel runs now while we investigate what's actually wrong.
OK, that test has been disabled. I'm hoping that the PR builds recover (along with some git fixes as referenced in ). Now on to figuring out why this is causing us problems...
Had a successful failure in build 8902. Grepping out the Ginkgo node number shows this test being the problem: So it looks like it got stuck in ? Is there anything in here that should hang without deadline?
I had another run get stuck in exactly the same place: What's worrying is that another parallel build (http://go/k8s-test/job/kubernetes-e2e-gce-parallel/7960/) timed out again early this morning, but this networking test was not enabled. This suggests that there's still something else triggering the problem.
I may have a partial fix for this in .
Have we seen this since merged? Can we close this bug?
- I think we had a lot of other problems during that time. I would suggest leaving it open during the weekend and close it on Monday if we don't see any problems.
I don't think we've seen this in a while. Please reopen if we're still having mysterious timeouts in Jenkins.
We're seeing this again in . It looks like someone put "Networking should function for intra-pod communication" test back in the parallel run recently. I think that we should revert whatever that PR was.
That test had been fine. More worrying is that my fix from seems to have vanished.
OK, looks like that was removed in , referencing issue . I'm not sure if the same issue is happening again (where operations mysteriously hang forever) or if it's something else. any thoughts?
I think your answer is here: The timeout field, and all references to it in clients were removed, based on the fact that clients should use request.Timeout() instead, but the clients were not updated to do that. I will send a PR shortly.
It seems like very few of the e2e tests are using this method, but we aren't seeing issues on most of the tests. I wonder if there's something in particular we're hitting with the networking test?
Could this be this magic number in ? I guess this means the checks only happen for about a 75 seconds. If one or two of the nodes are spinning up very slowly, this will trip over . Just a thought. Either way I think this needs to be configurable, I'll work on that angle.
I agree this "15" should probably be bigger. However, I'm pretty sure, that this shouldn't cause the test hang forever - it may cause test failures, but hanging forever test shouldn't happen...
Anyway - I've sent out out for review - we will see if that helps...
OK - it won't help, because it doesn't work :( - I asked if there is any reasonably easy way to do it...
Another instance: Again - the "Networking should function for intra-pod communication" is among tests that didn't run...
assigning this issue to you since you're working on a PR
- sure, although I'm not sure if this will solve the issue...
you can assign back to me if it doesn't :)
Unfortunately my PR didn't help. This just happened again: - reassigning back to you
Another instance:
And again:
Are folks still seeing this? I'm having a really hard time getting it to occur on-demand.
OK, I did find another timeout - The list of suspicious tests does not include the networking test this time: Kubectl client Kubectl patch should add annotations for pods in rc [Conformance] Kubectl client Simple pod should support exec through an HTTP proxy Pods should be submitted and removed [Conformance] Secrets should be consumable from pods [Conformance] Will have to keep searching for builds timing out to see if any new pattern emerges.
Looking more carefully, is another timeout. (I really wish it'd fail on my debug PR...) Suspicious tests: Job should scale a job down Kubectl client Simple pod should support exec through an HTTP proxy I'm gonna start suspecting that Kubectl HTTP proxy test now... are we doing anything in this test that might be blocking?
has a very curious failure: only 3 tests ran at all. It seems we are running the cluster-sanity checks in for every Ginkgo node, so there are probably cases like this run where it'll race and fail in mysterious ways (or possibly even deadlock)? We should probably use and (as documented ) to run the sanity checks, cleanup, and log dumping only once, rather than once per Ginkgo node. This may solve the mysterious timeouts.
Bumping down to P1 because this seems pretty rare nowadays (but I'll still address the issues I noted).
It seems like this issue is way too general: jobs timing out without a clear reason should have an action of making that reason clear. Is there any action we can take there?
I think the fixes I mentioned in will address the mysterious timeouts.
Happened here:
Hoping this one is gone for good, but I'll track and see.
Reopening; saw this again with
Final lines of that before being killed: Another run which timed out is kubernetes-pull-build-test-e2e-gce/. Final lines from that: Plus my earlier suspicions in make me think the kubectl proxy test is lacking a necessary timeout somewhere.
Moving discussion of that test case to .
With open, can we close this in favor of ?
Closing for now. If fixing doesn't fix, I'll keep looking...

flag.StringVar(&testContext.OutputPrintType, "output-print-type", "hr", "Comma separated list: 'hr' for human readable summaries 'json' for JSON ones.") } <del> func TestE2E(t *testing.T) { util.ReallyCrash = true util.InitLogs() defer util.FlushLogs() if testContext.ReportDir != "" { if err := os.MkdirAll(testContext.ReportDir, 0755); err != nil { glog.Errorf("Failed creating report directory: %v", err) } defer CoreDump(testContext.ReportDir) } if testContext.Provider == "" { </del> <ins> // setupProviderConfig validates and sets up cloudConfig based on testContext.Provider. func setupProviderConfig() error { switch testContext.Provider { case "": </ins> glog.Info("The --provider flag is not set.  Treating as a conformance test.  Some tests may not be run.") <del> } if testContext.Provider == "gce" || testContext.Provider == "gke" { </del> <ins> case "gce", "gke": </ins> var err error Logf("Fetching cloud provider for %qrn", testContext.Provider) var tokenSource oauth2.TokenSource


Question:
Both the GKE and Parallel tests recently had a run where they timed out. (after 90 minutes!) This is like 10x longer than the parallel run should be taking.
And the build project, too:
any ideas?
p0
yup, investigating this is next on my queue, right after I figure out why the PR jenkins is failing so many runs.
This happened again: In fact I've seen at least few cases like that: "at some point tests just stop and completely nothing is happening to the end" The same happened here - completely nothing happened during last hour of the run,
While the failure rates of the individual test jobs is not very high (e.g. kubernetes-e2e-gce-parallel : MTTF (Last 7 Days) 7 hr 27 min ... the fact that we gate auto-merging on about 7 of these jobs means that a merge fails almost every hour due to this.
I looked at 4 of the builds in kubernetes-e2e-gce-parallel, comparing the test cases marked "N/A" (i.e. no test result given) instead of their usual "PASSED" or "SKIPPED" report. So far, there's been only one test case that's been consistently N/A in all four of the builds I looked at: I'm not sure what might be happening, but I'm tempted to disable this test in parallel runs now while we investigate what's actually wrong.
OK, that test has been disabled. I'm hoping that the PR builds recover (along with some git fixes as referenced in ). Now on to figuring out why this is causing us problems...
Had a successful failure in build 8902. Grepping out the Ginkgo node number shows this test being the problem: So it looks like it got stuck in ? Is there anything in here that should hang without deadline?
I had another run get stuck in exactly the same place: What's worrying is that another parallel build (http://go/k8s-test/job/kubernetes-e2e-gce-parallel/7960/) timed out again early this morning, but this networking test was not enabled. This suggests that there's still something else triggering the problem.
I may have a partial fix for this in .
Have we seen this since merged? Can we close this bug?
- I think we had a lot of other problems during that time. I would suggest leaving it open during the weekend and close it on Monday if we don't see any problems.
I don't think we've seen this in a while. Please reopen if we're still having mysterious timeouts in Jenkins.
We're seeing this again in . It looks like someone put "Networking should function for intra-pod communication" test back in the parallel run recently. I think that we should revert whatever that PR was.
That test had been fine. More worrying is that my fix from seems to have vanished.
OK, looks like that was removed in , referencing issue . I'm not sure if the same issue is happening again (where operations mysteriously hang forever) or if it's something else. any thoughts?
I think your answer is here: The timeout field, and all references to it in clients were removed, based on the fact that clients should use request.Timeout() instead, but the clients were not updated to do that. I will send a PR shortly.
It seems like very few of the e2e tests are using this method, but we aren't seeing issues on most of the tests. I wonder if there's something in particular we're hitting with the networking test?
Could this be this magic number in ? I guess this means the checks only happen for about a 75 seconds. If one or two of the nodes are spinning up very slowly, this will trip over . Just a thought. Either way I think this needs to be configurable, I'll work on that angle.
I agree this "15" should probably be bigger. However, I'm pretty sure, that this shouldn't cause the test hang forever - it may cause test failures, but hanging forever test shouldn't happen...
Anyway - I've sent out out for review - we will see if that helps...
OK - it won't help, because it doesn't work :( - I asked if there is any reasonably easy way to do it...
Another instance: Again - the "Networking should function for intra-pod communication" is among tests that didn't run...
assigning this issue to you since you're working on a PR
- sure, although I'm not sure if this will solve the issue...
you can assign back to me if it doesn't :)
Unfortunately my PR didn't help. This just happened again: - reassigning back to you
Another instance:
And again:
Are folks still seeing this? I'm having a really hard time getting it to occur on-demand.
OK, I did find another timeout - The list of suspicious tests does not include the networking test this time: Kubectl client Kubectl patch should add annotations for pods in rc [Conformance] Kubectl client Simple pod should support exec through an HTTP proxy Pods should be submitted and removed [Conformance] Secrets should be consumable from pods [Conformance] Will have to keep searching for builds timing out to see if any new pattern emerges.
Looking more carefully, is another timeout. (I really wish it'd fail on my debug PR...) Suspicious tests: Job should scale a job down Kubectl client Simple pod should support exec through an HTTP proxy I'm gonna start suspecting that Kubectl HTTP proxy test now... are we doing anything in this test that might be blocking?
has a very curious failure: only 3 tests ran at all. It seems we are running the cluster-sanity checks in for every Ginkgo node, so there are probably cases like this run where it'll race and fail in mysterious ways (or possibly even deadlock)? We should probably use and (as documented ) to run the sanity checks, cleanup, and log dumping only once, rather than once per Ginkgo node. This may solve the mysterious timeouts.
Bumping down to P1 because this seems pretty rare nowadays (but I'll still address the issues I noted).
It seems like this issue is way too general: jobs timing out without a clear reason should have an action of making that reason clear. Is there any action we can take there?
I think the fixes I mentioned in will address the mysterious timeouts.
Happened here:
Hoping this one is gone for good, but I'll track and see.
Reopening; saw this again with
Final lines of that before being killed: Another run which timed out is kubernetes-pull-build-test-e2e-gce/. Final lines from that: Plus my earlier suspicions in make me think the kubectl proxy test is lacking a necessary timeout somewhere.
Moving discussion of that test case to .
With open, can we close this in favor of ?
Closing for now. If fixing doesn't fix, I'll keep looking...

zone := testContext.CloudConfig.Zone region, err := gcecloud.GetGCERegion(zone) if err != nil { <del> glog.Fatalf("error parsing GCE region from zone %q: %v", zone, err) </del> <ins> return fmt.Errorf("error parsing GCE/GKE region from zone %q: %v", zone, err) </ins> } managedZones := []string{zone} // Only single-zone for now cloudConfig.Provider, err = gcecloud.CreateGCECloud(testContext.CloudConfig.ProjectID, region, zone, managedZones, "" /* networkUrl */, tokenSource, false /* useMetadataServer */) if err != nil { <del> glog.Fatal("Error building GCE provider: ", err) </del> <ins> return fmt.Errorf("Error building GCE/GKE provider: ", err) </ins> } <del> } if testContext.Provider == "aws" { </del> <ins> case "aws": </ins> awsConfig := "[Global]n" if cloudConfig.Zone == "" { <del> glog.Fatal("gce-zone must be specified for AWS") </del> <ins> return fmt.Errorf("gce-zone must be specified for AWS") </ins> } awsConfig += fmt.Sprintf("Zone=%sn", cloudConfig.Zone) if cloudConfig.ClusterTag == "" { <del> glog.Fatal("--cluster-tag must be specified for AWS") </del> <ins> return fmt.Errorf("--cluster-tag must be specified for AWS") </ins> } awsConfig += fmt.Sprintf("KubernetesClusterTag=%sn", cloudConfig.ClusterTag) var err error cloudConfig.Provider, err = cloudprovider.GetCloudProvider(testContext.Provider, strings.NewReader(awsConfig)) if err != nil { <del> glog.Fatal("Error building AWS provider: ", err) </del> <ins> return fmt.Errorf("Error building AWS provider: ", err) </ins> } <del> } // Disable skipped tests unless they are explicitly requested. if config.GinkgoConfig.FocusString == "" && config.GinkgoConfig.SkipString == "" { // TODO(ihmccreery) Remove [Skipped] once all [Skipped] labels have been reclassified. config.GinkgoConfig.SkipString = `[Flaky]|[Skipped]|[Feature:.+]` </del> } <del> gomega.RegisterFailHandler(ginkgo.Fail) c, err := loadClient() if err != nil { glog.Fatal("Error loading client: ", err) } </del> <ins> return nil } // There are certain operations we only want to run once per overall test invocation // (such as deleting old namespaces, or verifying that all system pods are running. // Because of the way Ginkgo runs tests in parallel, we must use SynchronizedBeforeSuite // to ensure that these operations only run on the first parallel Ginkgo node. // // This function takes two parameters: one function which runs on only the first Ginkgo node, // returning an opaque byte array, and then a second function which runs on all Ginkgo nodes, // accepting the byte array. var _ = ginkgo.SynchronizedBeforeSuite(func() []byte { // Run only on Ginkgo node 1 </ins> // Delete any namespaces except default and kube-system. This ensures no // lingering resources are left over from a previous test run. if testContext.CleanStart { <ins> c, err := loadClient() if err != nil { glog.Fatal("Error loading client: ", err) } </ins> deleted, err := deleteNamespaces(c, nil /* deleteFilter */, []string{api.NamespaceSystem, api.NamespaceDefault}) if err != nil { <del> t.Errorf("Error deleting orphaned namespaces: %v", err) </del> <ins> Failf("Error deleting orphaned namespaces: %v", err) </ins> } glog.Infof("Waiting for deletion of the following namespaces: %v", deleted) if err := waitForNamespacesDeleted(c, deleted, namespaceCleanupTimeout); err != nil { <del> glog.Fatalf("Failed to delete orphaned namespaces %v: %v", deleted, err) </del> <ins> Failf("Failed to delete orphaned namespaces %v: %v", deleted, err) </ins> } }


Question:
Both the GKE and Parallel tests recently had a run where they timed out. (after 90 minutes!) This is like 10x longer than the parallel run should be taking.
And the build project, too:
any ideas?
p0
yup, investigating this is next on my queue, right after I figure out why the PR jenkins is failing so many runs.
This happened again: In fact I've seen at least few cases like that: "at some point tests just stop and completely nothing is happening to the end" The same happened here - completely nothing happened during last hour of the run,
While the failure rates of the individual test jobs is not very high (e.g. kubernetes-e2e-gce-parallel : MTTF (Last 7 Days) 7 hr 27 min ... the fact that we gate auto-merging on about 7 of these jobs means that a merge fails almost every hour due to this.
I looked at 4 of the builds in kubernetes-e2e-gce-parallel, comparing the test cases marked "N/A" (i.e. no test result given) instead of their usual "PASSED" or "SKIPPED" report. So far, there's been only one test case that's been consistently N/A in all four of the builds I looked at: I'm not sure what might be happening, but I'm tempted to disable this test in parallel runs now while we investigate what's actually wrong.
OK, that test has been disabled. I'm hoping that the PR builds recover (along with some git fixes as referenced in ). Now on to figuring out why this is causing us problems...
Had a successful failure in build 8902. Grepping out the Ginkgo node number shows this test being the problem: So it looks like it got stuck in ? Is there anything in here that should hang without deadline?
I had another run get stuck in exactly the same place: What's worrying is that another parallel build (http://go/k8s-test/job/kubernetes-e2e-gce-parallel/7960/) timed out again early this morning, but this networking test was not enabled. This suggests that there's still something else triggering the problem.
I may have a partial fix for this in .
Have we seen this since merged? Can we close this bug?
- I think we had a lot of other problems during that time. I would suggest leaving it open during the weekend and close it on Monday if we don't see any problems.
I don't think we've seen this in a while. Please reopen if we're still having mysterious timeouts in Jenkins.
We're seeing this again in . It looks like someone put "Networking should function for intra-pod communication" test back in the parallel run recently. I think that we should revert whatever that PR was.
That test had been fine. More worrying is that my fix from seems to have vanished.
OK, looks like that was removed in , referencing issue . I'm not sure if the same issue is happening again (where operations mysteriously hang forever) or if it's something else. any thoughts?
I think your answer is here: The timeout field, and all references to it in clients were removed, based on the fact that clients should use request.Timeout() instead, but the clients were not updated to do that. I will send a PR shortly.
It seems like very few of the e2e tests are using this method, but we aren't seeing issues on most of the tests. I wonder if there's something in particular we're hitting with the networking test?
Could this be this magic number in ? I guess this means the checks only happen for about a 75 seconds. If one or two of the nodes are spinning up very slowly, this will trip over . Just a thought. Either way I think this needs to be configurable, I'll work on that angle.
I agree this "15" should probably be bigger. However, I'm pretty sure, that this shouldn't cause the test hang forever - it may cause test failures, but hanging forever test shouldn't happen...
Anyway - I've sent out out for review - we will see if that helps...
OK - it won't help, because it doesn't work :( - I asked if there is any reasonably easy way to do it...
Another instance: Again - the "Networking should function for intra-pod communication" is among tests that didn't run...
assigning this issue to you since you're working on a PR
- sure, although I'm not sure if this will solve the issue...
you can assign back to me if it doesn't :)
Unfortunately my PR didn't help. This just happened again: - reassigning back to you
Another instance:
And again:
Are folks still seeing this? I'm having a really hard time getting it to occur on-demand.
OK, I did find another timeout - The list of suspicious tests does not include the networking test this time: Kubectl client Kubectl patch should add annotations for pods in rc [Conformance] Kubectl client Simple pod should support exec through an HTTP proxy Pods should be submitted and removed [Conformance] Secrets should be consumable from pods [Conformance] Will have to keep searching for builds timing out to see if any new pattern emerges.
Looking more carefully, is another timeout. (I really wish it'd fail on my debug PR...) Suspicious tests: Job should scale a job down Kubectl client Simple pod should support exec through an HTTP proxy I'm gonna start suspecting that Kubectl HTTP proxy test now... are we doing anything in this test that might be blocking?
has a very curious failure: only 3 tests ran at all. It seems we are running the cluster-sanity checks in for every Ginkgo node, so there are probably cases like this run where it'll race and fail in mysterious ways (or possibly even deadlock)? We should probably use and (as documented ) to run the sanity checks, cleanup, and log dumping only once, rather than once per Ginkgo node. This may solve the mysterious timeouts.
Bumping down to P1 because this seems pretty rare nowadays (but I'll still address the issues I noted).
It seems like this issue is way too general: jobs timing out without a clear reason should have an action of making that reason clear. Is there any action we can take there?
I think the fixes I mentioned in will address the mysterious timeouts.
Happened here:
Hoping this one is gone for good, but I'll track and see.
Reopening; saw this again with
Final lines of that before being killed: Another run which timed out is kubernetes-pull-build-test-e2e-gce/. Final lines from that: Plus my earlier suspicions in make me think the kubectl proxy test is lacking a necessary timeout somewhere.
Moving discussion of that test case to .
With open, can we close this in favor of ?
Closing for now. If fixing doesn't fix, I'll keep looking...

// test pods from running, and tests that ensure all pods are running and // ready will fail). if err := waitForPodsRunningReady(api.NamespaceSystem, testContext.MinStartupPods, podStartupTimeout); err != nil { <del> t.Errorf("Error waiting for all pods to be running and ready: %v", err) return </del> <ins> Failf("Error waiting for all pods to be running and ready: %v", err) </ins> } <ins> return nil }, func(data []byte) { // Run on all Ginkgo nodes }) // Similar to SynchornizedBeforeSuite, we want to run some operations only once (such as collecting cluster logs). // Here, the order of functions is reversed; first, the function which runs everywhere, // and then the function that only runs on the first Ginkgo node. var _ = ginkgo.SynchronizedAfterSuite(func() { // Run on all Ginkgo nodes }, func() { // Run only Ginkgo on node 1 if testContext.ReportDir != "" { CoreDump(testContext.ReportDir) } }) // TestE2E checks configuration parameters (specified through flags) and then runs // E2E tests using the Ginkgo runner. // If a "report directory" is specified, one or more JUnit test reports will be // generated in this directory, and cluster logs will also be saved. // This function is called on each Ginkgo node in parallel mode. func TestE2E(t *testing.T) { util.ReallyCrash = true util.InitLogs() defer util.FlushLogs() // We must call setupProviderConfig first since SynchronizedBeforeSuite needs // cloudConfig to be set up already. if err := setupProviderConfig(); err != nil { glog.Fatalf(err.Error()) } gomega.RegisterFailHandler(ginkgo.Fail) // Disable skipped tests unless they are explicitly requested. if config.GinkgoConfig.FocusString == "" && config.GinkgoConfig.SkipString == "" { // TODO(ihmccreery) Remove [Skipped] once all [Skipped] labels have been reclassified. config.GinkgoConfig.SkipString = `[Flaky]|[Skipped]|[Feature:.+]` } </ins> // Run tests through the Ginkgo runner with output to console + JUnit for Jenkins var r []ginkgo.Reporter if testContext.ReportDir != "" { <del> r = append(r, reporters.NewJUnitReporter(path.Join(testContext.ReportDir, fmt.Sprintf("junit_%02d.xml", config.GinkgoConfig.ParallelNode)))) </del> <ins> // TODO: we should probably only be trying to create this directory once // rather than once-per-Ginkgo-node. if err := os.MkdirAll(testContext.ReportDir, 0755); err != nil { glog.Errorf("Failed creating report directory: %v", err) } else { r = append(r, reporters.NewJUnitReporter(path.Join(testContext.ReportDir, fmt.Sprintf("junit_%02d.xml", config.GinkgoConfig.ParallelNode)))) } </ins> } <del> glog.Infof("Starting e2e run; %q", runId) </del> <ins> glog.Infof("Starting e2e run %q on Ginkgo node %d", runId, config.GinkgoConfig.ParallelNode) </ins> ginkgo.RunSpecsWithDefaultAndCustomReporters(t, "Kubernetes e2e suite", r) }


Question: I0921 20:36:33. ] Could not connect to D-Bus system bus: %sdial unix /var/run/dbus/systembussocket: connect: no such file or directory

func (runner *runner) connectToFirewallD() { bus, err := runner.dbus.SystemBus() if err != nil { <del> glog.V(1).Info("Could not connect to D-Bus system bus: %s", err) </del> <ins> glog.V(1).Infof("Could not connect to D-Bus system bus: %s", err) </ins> return }


Question:
From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct. Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now? I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016

// GetOriginalConfiguration retrieves the original configuration of the object // from the annotation, or nil if no annotation was found. <del> func GetOriginalConfiguration(info *resource.Info) ([]byte, error) { annots, err := info.Mapping.MetadataAccessor.Annotations(info.Object) </del> <ins> func GetOriginalConfiguration(mapping *meta.RESTMapping, obj runtime.Object) ([]byte, error) { annots, err := mapping.MetadataAccessor.Annotations(obj) </ins> if err != nil { return nil, err }


Question:
From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct. Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now? I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016

// UpdateApplyAnnotation calls CreateApplyAnnotation if the last applied // configuration annotation is already present. Otherwise, it does nothing. func UpdateApplyAnnotation(info *resource.Info, codec runtime.Encoder) error { <del> if original, err := GetOriginalConfiguration(info); err != nil || len(original) <= 0 { </del> <ins> if original, err := GetOriginalConfiguration(info.Mapping, info.Object); err != nil || len(original) <= 0 { </ins> return err } return CreateApplyAnnotation(info, codec)


Question:
From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct. Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now? I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016

import ( "fmt" "io" <ins> "time" "github.com/jonboulle/clockwork" </ins> "github.com/spf13/cobra" "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/errors" <ins> "k8s.io/kubernetes/pkg/api/meta" </ins> "k8s.io/kubernetes/pkg/kubectl" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" "k8s.io/kubernetes/pkg/kubectl/resource"


Question:
From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct. Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now? I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016

} const ( <ins> // maxPatchRetry is the maximum number of conflicts retry for during a patch operation before returning failure maxPatchRetry = 5 // backOffPeriod is the period to back off when apply patch resutls in error. backOffPeriod = 1 * time.Second // how many times we can retry before back off triesBeforeBackOff = 1 ) const ( </ins> apply_long = `Apply a configuration to a resource by filename or stdin. The resource will be created if it doesn't exist yet. To use 'apply', always create the resource initially with either 'apply' or 'create --save-config'.


Question:
From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct. Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now? I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016

return nil } <del> // Serialize the current configuration of the object from the server. current, err := runtime.Encode(encoder, info.Object) if err != nil { return cmdutil.AddSourceToErr(fmt.Sprintf("serializing current configuration from:n%vnfor:", info), info.Source, err) } // Retrieve the original configuration of the object from the annotation. original, err := kubectl.GetOriginalConfiguration(info) if err != nil { return cmdutil.AddSourceToErr(fmt.Sprintf("retrieving original configuration from:n%vnfor:", info), info.Source, err) } // Create the versioned struct from the original from the server for // strategic patch. // TODO: Move all structs in apply to use raw data. Can be done once // builder has a RawResult method which delivers raw data instead of // internal objects. versionedObject, _, err := decoder.Decode(current, nil, nil) if err != nil { return cmdutil.AddSourceToErr(fmt.Sprintf("converting encoded server-side object back to versioned struct:n%vnfor:", info), info.Source, err) } // Compute a three way strategic merge patch to send to server. patch, err := strategicpatch.CreateThreeWayMergePatch(original, modified, current, versionedObject, true) if err != nil { format := "creating patch with:noriginal:n%snmodified:n%sncurrent:n%snfrom:n%vnfor:" return cmdutil.AddSourceToErr(fmt.Sprintf(format, original, modified, current, info), info.Source, err) } </del> helper := resource.NewHelper(info.Client, info.Mapping) <del> _, err = helper.Patch(info.Namespace, info.Name, api.StrategicMergePatchType, patch) </del> <ins> patcher := NewPatcher(encoder, decoder, info.Mapping, helper) patchBytes, err := patcher.patch(info.Object, modified, info.Source, info.Namespace, info.Name) </ins> if err != nil { <del> return cmdutil.AddSourceToErr(fmt.Sprintf("applying patch:n%snto:n%vnfor:", patch, info), info.Source, err) </del> <ins> return cmdutil.AddSourceToErr(fmt.Sprintf("applying patch:n%snto:n%vnfor:", patchBytes, info), info.Source, err) </ins> } if cmdutil.ShouldRecord(cmd, info) { <del> patch, err = cmdutil.ChangeResourcePatch(info, f.Command()) </del> <ins> patch, err := cmdutil.ChangeResourcePatch(info, f.Command()) </ins> if err != nil { return err }


Question:
From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct. Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now? I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016

return nil } <ins> type patcher struct { encoder runtime.Encoder decoder runtime.Decoder mapping *meta.RESTMapping helper  *resource.Helper backOff clockwork.Clock } func NewPatcher(encoder runtime.Encoder, decoder runtime.Decoder, mapping *meta.RESTMapping, helper *resource.Helper) *patcher { return &patcher{ encoder: encoder, decoder: decoder, mapping: mapping, helper:  helper, backOff: clockwork.NewRealClock(), } } func (p *patcher) patchSimple(obj runtime.Object, modified []byte, source, namespace, name string) ([]byte, error) { // Serialize the current configuration of the object from the server. current, err := runtime.Encode(p.encoder, obj) if err != nil { return nil, cmdutil.AddSourceToErr(fmt.Sprintf("serializing current configuration from:n%vnfor:", obj), source, err) } // Retrieve the original configuration of the object from the annotation. original, err := kubectl.GetOriginalConfiguration(p.mapping, obj) if err != nil { return nil, cmdutil.AddSourceToErr(fmt.Sprintf("retrieving original configuration from:n%vnfor:", obj), source, err) } // Create the versioned struct from the original from the server for // strategic patch. // TODO: Move all structs in apply to use raw data. Can be done once // builder has a RawResult method which delivers raw data instead of // internal objects. versionedObject, _, err := p.decoder.Decode(current, nil, nil) if err != nil { return nil, cmdutil.AddSourceToErr(fmt.Sprintf("converting encoded server-side object back to versioned struct:n%vnfor:", obj), source, err) } // Compute a three way strategic merge patch to send to server. patch, err := strategicpatch.CreateThreeWayMergePatch(original, modified, current, versionedObject, true) if err != nil { format := "creating patch with:noriginal:n%snmodified:n%sncurrent:n%snfor:" return nil, cmdutil.AddSourceToErr(fmt.Sprintf(format, original, modified, current), source, err) } _, err = p.helper.Patch(namespace, name, api.StrategicMergePatchType, patch) return patch, err } func (p *patcher) patch(current runtime.Object, modified []byte, source, namespace, name string) ([]byte, error) { var getErr error patchBytes, err := p.patchSimple(current, modified, source, namespace, name) for i := 1; i <= maxPatchRetry && errors.IsConflict(err); i++ { if i > triesBeforeBackOff { p.backOff.Sleep(backOffPeriod) } current, getErr = p.helper.Get(namespace, name, false) if getErr != nil { return nil, getErr } patchBytes, err = p.patchSimple(current, modified, source, namespace, name) } return patchBytes, err } </ins>


Question:
From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct. Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now? I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016

import ( "bytes" "encoding/json" <ins> "fmt" </ins> "io/ioutil" "net/http" "os"


Question:
From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct. Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now? I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016

"k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/annotations" <ins> kubeerr "k8s.io/kubernetes/pkg/api/errors" </ins> "k8s.io/kubernetes/pkg/api/meta" <ins> "k8s.io/kubernetes/pkg/api/unversioned" </ins> "k8s.io/kubernetes/pkg/client/unversioned/fake" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" "k8s.io/kubernetes/pkg/runtime"


Question:
From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct. Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now? I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016

} } <ins> func TestApplyRetry(t *testing.T) { initTestErrorHandler(t) nameRC, currentRC := readAndAnnotateReplicationController(t, filenameRC) pathRC := "/namespaces/test/replicationcontrollers/" + nameRC firstPatch := true retry := false getCount := 0 f, tf, codec := NewAPIFactory() tf.Printer = &testPrinter{} tf.Client = &fake.RESTClient{ Codec: codec, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { switch p, m := req.URL.Path, req.Method; { case p == pathRC && m == "GET": getCount++ bodyRC := ioutil.NopCloser(bytes.NewReader(currentRC)) return &http.Response{StatusCode: 200, Header: defaultHeader(), Body: bodyRC}, nil case p == pathRC && m == "PATCH": if firstPatch { firstPatch = false statusErr := kubeerr.NewConflict(unversioned.GroupResource{Group: "", Resource: "rc"}, "test-rc", fmt.Errorf("the object has been modified. Please apply at first.")) bodyBytes, _ := json.Marshal(statusErr) bodyErr := ioutil.NopCloser(bytes.NewReader(bodyBytes)) return &http.Response{StatusCode: http.StatusConflict, Header: defaultHeader(), Body: bodyErr}, nil } retry = true validatePatchApplication(t, req) bodyRC := ioutil.NopCloser(bytes.NewReader(currentRC)) return &http.Response{StatusCode: 200, Header: defaultHeader(), Body: bodyRC}, nil default: t.Fatalf("unexpected request: %#vn%#v", req.URL, req) return nil, nil } }), } tf.Namespace = "test" buf := bytes.NewBuffer([]byte{}) cmd := NewCmdApply(f, buf) cmd.Flags().Set("filename", filenameRC) cmd.Flags().Set("output", "name") cmd.Run(cmd, []string{}) if !retry || getCount != 2 { t.Fatalf("apply didn't retry when get conflict error") } // uses the name from the file, not the response expectRC := "replicationcontroller/" + nameRC + "n" if buf.String() != expectRC { t.Fatalf("unexpected output: %snexpected: %s", buf.String(), expectRC) } } </ins> func TestApplyNonExistObject(t *testing.T) { nameRC, currentRC := readAndAnnotateReplicationController(t, filenameRC) pathRC := "/namespaces/test/replicationcontrollers"


Question:
We should be testing the reliability of a long-running cluster. These tests kill/restart components or nodes in the cluster, defeating the purpose of a soak cluster.
Example tests: NodessResize Services.restarting DaemonRestart NodessResize Reboot Etcd failure
Agreed. Some of them already seem to be skipped: GCESOAKCONTINUOUSSKIPTESTS=( "Density.30spods" "Elasticsearch" "Etcd.SIGKILL" "externalsloadsbalancer" "identicallysnamedsservices" "networkspartition" "Services.Typesgoessfrom" ) Q
FYI
It'd be hard to prevent regression as more tests are . It might be better if we explicitly define a set of DISRUPTIVETESTS, and include them wherever needed, e.g., GCEPARALLELSKIPTESTS and GCESOAKCONTINUOUSSKIPTESTS. That way, people who are adding tests will be aware of the category. WDYT?
Agreed, that seems like a good idea. Want to send in a PR? Or at least open an issue?

"experimentalsresourcesusagestracking" # Expect --max-pods=100 ) <ins> # Tests which kills or restarts components and/or nodes. DISRUPTIVE_TESTS=( "DaemonRestart" "Etcdsfailure" "NodessResize" "Reboot" "Services.*restarting" ) </ins> # The following tests are known to be flaky, and are thus run only in their own # -flaky- build variants. GCE_FLAKY_TESTS=(


Question:
We should be testing the reliability of a long-running cluster. These tests kill/restart components or nodes in the cluster, defeating the purpose of a soak cluster.
Example tests: NodessResize Services.restarting DaemonRestart NodessResize Reboot Etcd failure
Agreed. Some of them already seem to be skipped: GCESOAKCONTINUOUSSKIPTESTS=( "Density.30spods" "Elasticsearch" "Etcd.SIGKILL" "externalsloadsbalancer" "identicallysnamedsservices" "networkspartition" "Services.Typesgoessfrom" ) Q
FYI
It'd be hard to prevent regression as more tests are . It might be better if we explicitly define a set of DISRUPTIVETESTS, and include them wherever needed, e.g., GCEPARALLELSKIPTESTS and GCESOAKCONTINUOUSSKIPTESTS. That way, people who are adding tests will be aware of the category. WDYT?
Agreed, that seems like a good idea. Want to send in a PR? Or at least open an issue?

# Tests which are not able to be run in parallel. GCE_PARALLEL_SKIP_TESTS=( <del> "Etcd" "NetworkingNew" </del> "NodessNetwork" <del> "NodessResize" </del> "MaxPods" "Resourcesusagesofssystemscontainers" "SchedulerPredicates" <del> "Services.*restarting" </del> "resourcesusagestracking" <ins> "${DISRUPTIVE_TESTS[@]}" </ins> ) # Tests which are known to be flaky when run in parallel.


Question:
We should be testing the reliability of a long-running cluster. These tests kill/restart components or nodes in the cluster, defeating the purpose of a soak cluster.
Example tests: NodessResize Services.restarting DaemonRestart NodessResize Reboot Etcd failure
Agreed. Some of them already seem to be skipped: GCESOAKCONTINUOUSSKIPTESTS=( "Density.30spods" "Elasticsearch" "Etcd.SIGKILL" "externalsloadsbalancer" "identicallysnamedsservices" "networkspartition" "Services.Typesgoessfrom" ) Q
FYI
It'd be hard to prevent regression as more tests are . It might be better if we explicitly define a set of DISRUPTIVETESTS, and include them wherever needed, e.g., GCEPARALLELSKIPTESTS and GCESOAKCONTINUOUSSKIPTESTS. That way, people who are adding tests will be aware of the category. WDYT?
Agreed, that seems like a good idea. Want to send in a PR? Or at least open an issue?

GCE_SOAK_CONTINUOUS_SKIP_TESTS=( "Density.*30spods" "Elasticsearch" <del> "Etcd.*SIGKILL" </del> "externalsloadsbalancer" "identicallysnamedsservices" "networkspartition" "Services.*Typesgoessfrom" <ins> "${DISRUPTIVE_TESTS[@]}"       # avoid component restarts. </ins> ) GCE_RELEASE_SKIP_TESTS=(


Question:
The desciption of --all-namespaces is listing the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with --namespace." But when I used the following command: error: namespace may not be empty when retrieving a resource by name no namespace set on resource pods "counter" The error messages are unclear, and the second message isn't labeled as an error even.
And why couldn't use --all-namespaces flag?
You're asking for a specific pod name (counter) in all namespaces, which is a nonsensical query. The pod name is not a grep-like filter, it's a specific name. Yeah, the errors could be better. Describe could work, but potentially generates a LOT of information and API calls.
But I found the following API is supported now. Get The result is: has the more information. Do we need to support the following kubectl command(using the above API) or just return clear error messages? get pods counter --all-namespaces
I'd be ok with this being implemented via the field selector.
The issue is still here :). What's our expected result? I'd like to contribute this.
, after check the code, the URL was built by ; do you mean we should update it; or avoid to use , instead, build url () directly.
After checking the code, it's better to add fieldSelector to , is created for fieldSelector. Will submit a PR based on .
getting an object with a particular name across all namespaces seems like something we shouldn't encourage... it breaks the assumption of namespace independence.
, so we'll only update message for this issue? for example:
Disagree. Need easy way to troubleshouting the problem "why pods is stuck in Pending status". It's tiring for every pod to find who created it and in which namespace. In the logs there is only the name of the pod.
Now, with field selectors we can do To further my understanding, I'd ask:- doesn't itself break independence? One can search by label across all namespaces, it feels funny for by name to be special.

# Clean up kubectl delete namespace my-namespace <del> ############## </del> <ins> ###################### </ins> # Pods in Namespaces # <del> ############## </del> <ins> ###################### </ins> ### Create a new namespace # Pre-condition: the other namespace does not exist


Question:
The desciption of --all-namespaces is listing the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with --namespace." But when I used the following command: error: namespace may not be empty when retrieving a resource by name no namespace set on resource pods "counter" The error messages are unclear, and the second message isn't labeled as an error even.
And why couldn't use --all-namespaces flag?
You're asking for a specific pod name (counter) in all namespaces, which is a nonsensical query. The pod name is not a grep-like filter, it's a specific name. Yeah, the errors could be better. Describe could work, but potentially generates a LOT of information and API calls.
But I found the following API is supported now. Get The result is: has the more information. Do we need to support the following kubectl command(using the above API) or just return clear error messages? get pods counter --all-namespaces
I'd be ok with this being implemented via the field selector.
The issue is still here :). What's our expected result? I'd like to contribute this.
, after check the code, the URL was built by ; do you mean we should update it; or avoid to use , instead, build url () directly.
After checking the code, it's better to add fieldSelector to , is created for fieldSelector. Will submit a PR based on .
getting an object with a particular name across all namespaces seems like something we shouldn't encourage... it breaks the assumption of namespace independence.
, so we'll only update message for this issue? for example:
Disagree. Need easy way to troubleshouting the problem "why pods is stuck in Pending status". It's tiring for every pod to find who created it and in which namespace. In the logs there is only the name of the pod.
Now, with field selectors we can do To further my understanding, I'd ask:- doesn't itself break independence? One can search by label across all namespaces, it feels funny for by name to be special.

kube::test::get_object_assert 'pods --namespace=other' "{{range.items}}{{$id_field}}:{{end}}" 'valid-pod:' # Post-condition: verify shorthand `-n other` has the same results as `--namespace=other` kube::test::get_object_assert 'pods -n other' "{{range.items}}{{$id_field}}:{{end}}" 'valid-pod:' <ins> # Post-condition: a resource cannot be retrieved by name across all namespaces output_message=$(! kubectl get "${kube_flags[@]}" pod valid-pod --all-namespaces 2>&1) kube::test::if_has_string "${output_message}" "a resource cannot be retrieved by name across all namespaces" </ins> ### Delete POD valid-pod in specific namespace # Pre-condition: valid-pod POD exists


Question:
The desciption of --all-namespaces is listing the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with --namespace." But when I used the following command: error: namespace may not be empty when retrieving a resource by name no namespace set on resource pods "counter" The error messages are unclear, and the second message isn't labeled as an error even.
And why couldn't use --all-namespaces flag?
You're asking for a specific pod name (counter) in all namespaces, which is a nonsensical query. The pod name is not a grep-like filter, it's a specific name. Yeah, the errors could be better. Describe could work, but potentially generates a LOT of information and API calls.
But I found the following API is supported now. Get The result is: has the more information. Do we need to support the following kubectl command(using the above API) or just return clear error messages? get pods counter --all-namespaces
I'd be ok with this being implemented via the field selector.
The issue is still here :). What's our expected result? I'd like to contribute this.
, after check the code, the URL was built by ; do you mean we should update it; or avoid to use , instead, build url () directly.
After checking the code, it's better to add fieldSelector to , is created for fieldSelector. Will submit a PR based on .
getting an object with a particular name across all namespaces seems like something we shouldn't encourage... it breaks the assumption of namespace independence.
, so we'll only update message for this issue? for example:
Disagree. Need easy way to troubleshouting the problem "why pods is stuck in Pending status". It's tiring for every pod to find who created it and in which namespace. In the logs there is only the name of the pod.
Now, with field selectors we can do To further my understanding, I'd ask:- doesn't itself break independence? One can search by label across all namespaces, it feels funny for by name to be special.

# Clean up kubectl delete namespace other <del> ############## </del> <ins> ########### </ins> # Secrets # <del> ############## </del> <ins> ########### </ins> ### Create a new namespace # Pre-condition: the test-secrets namespace does not exist