Question:
See - currently there's just one test that triggers Unreserve through a pre-bind failure.
/sig scheduling
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I shall work on it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/assign
i saw commented the only test code, do you have any idea why this going on?
well, I found some functions missing, maybe because of the code refactor.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale

} } <del> // // TestUnreserveReservePlugin tests invocation of the Unreserve operation in // // reserve plugins through failures in execution points such as pre-bind. Also // // tests that the order of invocation of Unreserve operation is executed in the // // reverse order of invocation of the Reserve operation. // func TestReservePluginUnreserve(t *testing.T) { // 	tests := []struct { // 		name             string // 		failReserve      bool // 		failReserveIndex int // 		failPreBind      bool // 	}{ // 		{ // 			name:             "fail reserve", // 			failReserve:      true, // 			failReserveIndex: 1, // 		}, // 		{ // 			name:        "fail preBind", // 			failPreBind: true, // 		}, // 		{ // 			name: "pass everything", // 		}, // 	} // 	for _, test := range tests { // 		t.Run(test.name, func(t *testing.T) { // 			numReservePlugins := 3 // 			pluginInvokeEventChan := make(chan pluginInvokeEvent, numReservePlugins) // 			preBindPlugin := &PreBindPlugin{ // 				failPreBind: true, // 			} // 			var reservePlugins []*ReservePlugin // 			for i := 0; i < numReservePlugins; i++ { // 				reservePlugins = append(reservePlugins, &ReservePlugin{ // 					name:                  fmt.Sprintf("%s-%d", reservePluginName, i), // 					pluginInvokeEventChan: pluginInvokeEventChan, // 				}) // 			} // 			registry := frameworkruntime.Registry{ // 				// TODO(#92229): test more failure points that would trigger Unreserve in // 				// reserve plugins than just one pre-bind plugin. // 				preBindPluginName: newPlugin(preBindPlugin), // 			} // 			for _, pl := range reservePlugins { // 				registry[pl.Name()] = newPlugin(pl) // 			} // 			// Setup initial reserve and prebind plugin for testing. // 			prof := schedulerconfig.KubeSchedulerProfile{ // 				SchedulerName: v1.DefaultSchedulerName, // 				Plugins: &schedulerconfig.Plugins{ // 					Reserve: schedulerconfig.PluginSet{ // 						// filled by looping over reservePlugins // 					}, // 					PreBind: schedulerconfig.PluginSet{ // 						Enabled: []schedulerconfig.Plugin{ // 							{ // 								Name: preBindPluginName, // 							}, // 						}, // 					}, // 				}, // 			} // 			for _, pl := range reservePlugins { // 				prof.Plugins.Reserve.Enabled = append(prof.Plugins.Reserve.Enabled, schedulerconfig.Plugin{ // 					Name: pl.Name(), // 				}) // 			} // 			// Create the master and the scheduler with the test plugin set. // 			testCtx := initTestSchedulerForFrameworkTest(t, testutils.InitTestMaster(t, "reserve-plugin-unreserve", nil), 2, // 				scheduler.WithProfiles(prof), // 				scheduler.WithFrameworkOutOfTreeRegistry(registry)) // 			defer testutils.CleanupTest(t, testCtx) // 			preBindPlugin.failPreBind = test.failPreBind // 			if test.failReserve { // 				reservePlugins[test.failReserveIndex].failReserve = true // 			} // 			// Create a best effort pod. // 			pod, err := createPausePod(testCtx.ClientSet, // 				initPausePod(&pausePodConfig{Name: "test-pod", Namespace: testCtx.NS.Name})) // 			if err != nil { // 				t.Errorf("Error while creating a test pod: %v", err) // 			} // 			if test.failPreBind || test.failReserve { // 				if err = wait.Poll(10*time.Millisecond, 30*time.Second, podSchedulingError(testCtx.ClientSet, pod.Namespace, pod.Name)); err != nil { // 					t.Errorf("Expected a scheduling error, but didn't get it: %v", err) // 				} // 				for i := numReservePlugins - 1; i >= 0; i-- { // 					select { // 					case event := <-pluginInvokeEventChan: // 						expectedPluginName := reservePlugins[i].Name() // 						if expectedPluginName != event.pluginName { // 							t.Errorf("event.pluginName = %s, want %s", event.pluginName, expectedPluginName) // 						} // 					case <-time.After(time.Second * 30): // 						t.Errorf("pluginInvokeEventChan receive timed out") // 					} // 				} // 			} else { // 				if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, pod); err != nil { // 					t.Errorf("Expected the pod to be scheduled, got an error: %v", err) // 				} // 				for i, pl := range reservePlugins { // 					if pl.numUnreserveCalled != 0 { // 						t.Errorf("reservePlugins[%d].numUnreserveCalled = %d, want 0", i, pl.numUnreserveCalled) // 					} // 				} // 			} // 			testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{pod}) // 		}) // 	} // } </del> <ins> // TestUnReserveReservePlugins tests invocation of the Unreserve operation in // reserve plugins through failures in execution points such as pre-bind. Also // tests that the order of invocation of Unreserve operation is executed in the // reverse order of invocation of the Reserve operation. func TestUnReserveReservePlugins(t *testing.T) { tests := []struct { name          string plugins       []*ReservePlugin fail          bool failPluginIdx int }{ { name:          "The first Reserve plugin fails", failPluginIdx: 0, plugins: []*ReservePlugin{ { name:        "failedReservePlugin1", failReserve: true, }, { name:        "succeedReservePlugin", failReserve: false, }, { name:        "failedReservePlugin2", failReserve: true, }, }, fail: true, }, { name:          "The middle Reserve plugin fails", failPluginIdx: 1, plugins: []*ReservePlugin{ { name:        "succeedReservePlugin1", failReserve: false, }, { name:        "failedReservePlugin1", failReserve: true, }, { name:        "succeedReservePlugin2", failReserve: false, }, }, fail: true, }, { name:          "The last Reserve plugin fails", failPluginIdx: 2, plugins: []*ReservePlugin{ { name:        "succeedReservePlugin1", failReserve: false, }, { name:        "succeedReservePlugin2", failReserve: false, }, { name:        "failedReservePlugin1", failReserve: true, }, }, fail: true, }, { name:          "The Reserve plugins succeed", failPluginIdx: -1, plugins: []*ReservePlugin{ { name:        "succeedReservePlugin1", failReserve: false, }, { name:        "succeedReservePlugin2", failReserve: false, }, { name:        "succeedReservePlugin3", failReserve: false, }, }, fail: false, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { var pls []framework.Plugin for _, pl := range test.plugins { pls = append(pls, pl) } registry, prof := initRegistryAndConfig(t, pls...) // Create the API server and the scheduler with the test plugin set. testCtx := initTestSchedulerForFrameworkTest( t, testutils.InitTestAPIServer(t, "unreserve-reserve-plugin", nil), 2, scheduler.WithProfiles(prof), scheduler.WithFrameworkOutOfTreeRegistry(registry)) defer testutils.CleanupTest(t, testCtx) // Create a best effort pod. podName := "test-pod" pod, err := createPausePod(testCtx.ClientSet, initPausePod(&pausePodConfig{Name: podName, Namespace: testCtx.NS.Name})) if err != nil { t.Errorf("Error while creating a test pod: %v", err) } if test.fail { if err = wait.Poll(10*time.Millisecond, 30*time.Second, podSchedulingError(testCtx.ClientSet, pod.Namespace, pod.Name)); err != nil { t.Errorf("Expected a reasons other than Unschedulable, but got: %v", err) } for i, pl := range test.plugins { if i <= test.failPluginIdx { if pl.numReserveCalled != 1 { t.Errorf("Reserve Plugins %s numReserveCalled = %d, want 1.", pl.name, pl.numReserveCalled) } } else { if pl.numReserveCalled != 0 { t.Errorf("Reserve Plugins %s numReserveCalled = %d, want 0.", pl.name, pl.numReserveCalled) } } if pl.numUnreserveCalled != 1 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 1.", pl.name, pl.numUnreserveCalled) } } } else { if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, pod); err != nil { t.Errorf("Expected the pod to be scheduled. error: %v", err) } for _, pl := range test.plugins { if pl.numReserveCalled != 1 { t.Errorf("Reserve Plugin %s numReserveCalled = %d, want 1.", pl.name, pl.numReserveCalled) } if pl.numUnreserveCalled != 0 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 0.", pl.name, pl.numUnreserveCalled) } } } testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{pod}) }) } } // TestUnReservePermitPlugins tests unreserve of Permit plugins. func TestUnReservePermitPlugins(t *testing.T) { tests := []struct { name   string plugin *PermitPlugin fail   bool }{ { name: "All Reserve plugins passed, but a Permit plugin was rejected", fail: true, plugin: &PermitPlugin{ name:         "rejectedPermitPlugin", rejectPermit: true, }, }, { name: "All Reserve plugins passed, but a Permit plugin timeout in waiting", fail: true, plugin: &PermitPlugin{ name:          "timeoutPermitPlugin", timeoutPermit: true, }, }, { name: "The Permit plugin succeed", fail: false, plugin: &PermitPlugin{ name: "succeedPermitPlugin", }, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { reservePlugin := &ReservePlugin{ name:        "reservePlugin", failReserve: false, } registry, profile := initRegistryAndConfig(t, []framework.Plugin{test.plugin, reservePlugin}...) // Create the API server and the scheduler with the test plugin set. testCtx := initTestSchedulerForFrameworkTest( t, testutils.InitTestAPIServer(t, "unreserve-reserve-plugin", nil), 2, scheduler.WithProfiles(profile), scheduler.WithFrameworkOutOfTreeRegistry(registry)) defer testutils.CleanupTest(t, testCtx) // Create a best effort pod. podName := "test-pod" pod, err := createPausePod(testCtx.ClientSet, initPausePod(&pausePodConfig{Name: podName, Namespace: testCtx.NS.Name})) if err != nil { t.Errorf("Error while creating a test pod: %v", err) } if test.fail { if err = waitForPodUnschedulable(testCtx.ClientSet, pod); err != nil { t.Errorf("Didn't expect the pod to be scheduled. error: %v", err) } // Verify the Reserve Plugins if reservePlugin.numUnreserveCalled != 1 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 1.", reservePlugin.name, reservePlugin.numUnreserveCalled) } } else { if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, pod); err != nil { t.Errorf("Expected the pod to be scheduled. error: %v", err) } // Verify the Reserve Plugins if reservePlugin.numUnreserveCalled != 0 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 0.", reservePlugin.name, reservePlugin.numUnreserveCalled) } } if test.plugin.numPermitCalled != 1 { t.Errorf("Expected the Permit plugin to be called.") } testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{pod}) }) } } // TestUnReservePreBindPlugins tests unreserve of Prebind plugins. func TestUnReservePreBindPlugins(t *testing.T) { tests := []struct { name   string plugin *PreBindPlugin fail   bool }{ { name: "All Reserve plugins passed, but a PreBind plugin failed", fail: true, plugin: &PreBindPlugin{ podUIDs:       make(map[types.UID]struct{}), rejectPreBind: true, }, }, { name:   "All Reserve plugins passed, and PreBind plugin succeed", fail:   false, plugin: &PreBindPlugin{podUIDs: make(map[types.UID]struct{})}, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { reservePlugin := &ReservePlugin{ name:        "reservePlugin", failReserve: false, } registry, profile := initRegistryAndConfig(t, []framework.Plugin{test.plugin, reservePlugin}...) // Create the API server and the scheduler with the test plugin set. testCtx := initTestSchedulerForFrameworkTest( t, testutils.InitTestAPIServer(t, "unreserve-prebind-plugin", nil), 2, scheduler.WithProfiles(profile), scheduler.WithFrameworkOutOfTreeRegistry(registry)) defer testutils.CleanupTest(t, testCtx) // Create a pause pod. podName := "test-pod" pod, err := createPausePod(testCtx.ClientSet, initPausePod(&pausePodConfig{Name: podName, Namespace: testCtx.NS.Name})) if err != nil { t.Errorf("Error while creating a test pod: %v", err) } if test.fail { if err = wait.Poll(10*time.Millisecond, 30*time.Second, podSchedulingError(testCtx.ClientSet, pod.Namespace, pod.Name)); err != nil { t.Errorf("Expected a reasons other than Unschedulable, but got: %v", err) } // Verify the Reserve Plugins if reservePlugin.numUnreserveCalled != 1 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 1.", reservePlugin.name, reservePlugin.numUnreserveCalled) } } else { if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, pod); err != nil { t.Errorf("Expected the pod to be scheduled. error: %v", err) } // Verify the Reserve Plugins if reservePlugin.numUnreserveCalled != 0 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 0.", reservePlugin.name, reservePlugin.numUnreserveCalled) } } if test.plugin.numPreBindCalled != 1 { t.Errorf("Expected the Prebind plugin to be called.") } testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{pod}) }) } } // TestUnReserveBindPlugins tests unreserve of Bind plugins. func TestUnReserveBindPlugins(t *testing.T) { tests := []struct { name   string plugin *BindPlugin fail   bool }{ { name:   "All Reserve plugins passed, and Bind plugin succeed", fail:   false, plugin: &BindPlugin{name: "SucceedBindPlugin"}, }, { name:   "All Reserve plugins passed, but a Bind plugin failed", fail:   false, plugin: &BindPlugin{name: "FailedBindPlugin"}, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { reservePlugin := &ReservePlugin{ name:        "reservePlugin", failReserve: false, } registry, profile := initRegistryAndConfig(t, []framework.Plugin{test.plugin, reservePlugin}...) apiCtx := testutils.InitTestAPIServer(t, "unreserve-bind-plugin", nil) test.plugin.client = apiCtx.ClientSet // Create the scheduler with the test plugin set. testCtx := initTestSchedulerForFrameworkTest( t, apiCtx, 2, scheduler.WithProfiles(profile), scheduler.WithFrameworkOutOfTreeRegistry(registry)) defer testutils.CleanupTest(t, testCtx) // Create a pause pod. podName := "test-pod" pod, err := createPausePod(testCtx.ClientSet, initPausePod(&pausePodConfig{Name: podName, Namespace: testCtx.NS.Name})) if err != nil { t.Errorf("Error while creating a test pod: %v", err) } if test.fail { if err = wait.Poll(10*time.Millisecond, 30*time.Second, podSchedulingError(testCtx.ClientSet, pod.Namespace, pod.Name)); err != nil { t.Errorf("Expected a reasons other than Unschedulable, but got: %v", err) } // Verify the Reserve Plugins if reservePlugin.numUnreserveCalled != 1 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 1.", reservePlugin.name, reservePlugin.numUnreserveCalled) } } else { if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, pod); err != nil { t.Errorf("Expected the pod to be scheduled. error: %v", err) } // Verify the Reserve Plugins if reservePlugin.numUnreserveCalled != 0 { t.Errorf("Reserve Plugin %s numUnreserveCalled = %d, want 0.", reservePlugin.name, reservePlugin.numUnreserveCalled) } } if test.plugin.numBindCalled != 1 { t.Errorf("Expected the Prebind plugin to be called.") } testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{pod}) }) } } </ins> type pluginInvokeEvent struct { pluginName string


Question:
See - currently there's just one test that triggers Unreserve through a pre-bind failure.
/sig scheduling
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I shall work on it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/assign
i saw commented the only test code, do you have any idea why this going on?
well, I found some functions missing, maybe because of the code refactor.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale

// TestBindPlugin tests invocation of bind plugins. func TestBindPlugin(t *testing.T) { testContext := testutils.InitTestAPIServer(t, "bind-plugin", nil) <del> bindPlugin1 := &BindPlugin{PluginName: "bind-plugin-1", client: testContext.ClientSet} bindPlugin2 := &BindPlugin{PluginName: "bind-plugin-2", client: testContext.ClientSet} </del> <ins> bindPlugin1 := &BindPlugin{name: "bind-plugin-1", client: testContext.ClientSet} bindPlugin2 := &BindPlugin{name: "bind-plugin-2", client: testContext.ClientSet} </ins> reservePlugin := &ReservePlugin{name: "mock-reserve-plugin"} postBindPlugin := &PostBindPlugin{name: "mock-post-bind-plugin"} // Create a plugin registry for testing. Register reserve, bind, and


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : The ebtables utility in is missing 3 useful features: method in its Interface table chain Why is this needed: The missing features are useful to allow for: rule deletion as there is a way to add (ensure) a rule but not removing it currently. manipulating the broute table and brouting chain The first feature would also align ebtables with iptables, as a DeleteRule method is available there.
/sig network
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Is there some feature/bugfix you're working on in kubernetes that needs this?
Yes, in kubernetes/dns (node-cache) actually. That PR is working, but it's only in my fork of kubernetes/dns at the moment as it depends on this. It solves this: Adding this code to ebtables would help me avoid re-inventing the wheel there.
... not sure if you've seen that ... but I have created a PR to solve this issue already:
Yes, I saw the PR first. So, kubernetes/kubernetes isn't really intended to be vendored into other repos and kubernetes/dns is being bad here. It's dubious to add methods to that kubernetes itself isn't using because someone else might just end up removing them later when they notice that they're not being used... OTOH the PR needs access to the internals of the ebtables code so you couldn't just add it to kubernetes/dns directly without forking all of ...
Thanks for your comments. I saw iptables used that way in node-cache and I thought it would be good for consistency to go down the same route with ebtables. To be honest I am not very keen on forking the entire package, for obvious reasons, but I can understand your reservations about this change.
/assign
Hi Did you manage to discuss this?
yes, sorry, didn't get a chance to get back to this on Friday. (I /lgtm'ed the PR) /remove-triage unresolved
Thanks!

const ( TableNAT    Table = "nat" TableFilter Table = "filter" <ins> TableBroute Table = "broute" </ins> ) type Chain string


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : The ebtables utility in is missing 3 useful features: method in its Interface table chain Why is this needed: The missing features are useful to allow for: rule deletion as there is a way to add (ensure) a rule but not removing it currently. manipulating the broute table and brouting chain The first feature would also align ebtables with iptables, as a DeleteRule method is available there.
/sig network
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Is there some feature/bugfix you're working on in kubernetes that needs this?
Yes, in kubernetes/dns (node-cache) actually. That PR is working, but it's only in my fork of kubernetes/dns at the moment as it depends on this. It solves this: Adding this code to ebtables would help me avoid re-inventing the wheel there.
... not sure if you've seen that ... but I have created a PR to solve this issue already:
Yes, I saw the PR first. So, kubernetes/kubernetes isn't really intended to be vendored into other repos and kubernetes/dns is being bad here. It's dubious to add methods to that kubernetes itself isn't using because someone else might just end up removing them later when they notice that they're not being used... OTOH the PR needs access to the internals of the ebtables code so you couldn't just add it to kubernetes/dns directly without forking all of ...
Thanks for your comments. I saw iptables used that way in node-cache and I thought it would be good for consistency to go down the same route with ebtables. To be honest I am not very keen on forking the entire package, for obvious reasons, but I can understand your reservations about this change.
/assign
Hi Did you manage to discuss this?
yes, sorry, didn't get a chance to get back to this on Friday. (I /lgtm'ed the PR) /remove-triage unresolved
Thanks!

ChainPrerouting  Chain = "PREROUTING" ChainOutput      Chain = "OUTPUT" ChainInput       Chain = "INPUT" <ins> ChainBrouting    Chain = "BROUTING" </ins> ) type operation string


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : The ebtables utility in is missing 3 useful features: method in its Interface table chain Why is this needed: The missing features are useful to allow for: rule deletion as there is a way to add (ensure) a rule but not removing it currently. manipulating the broute table and brouting chain The first feature would also align ebtables with iptables, as a DeleteRule method is available there.
/sig network
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Is there some feature/bugfix you're working on in kubernetes that needs this?
Yes, in kubernetes/dns (node-cache) actually. That PR is working, but it's only in my fork of kubernetes/dns at the moment as it depends on this. It solves this: Adding this code to ebtables would help me avoid re-inventing the wheel there.
... not sure if you've seen that ... but I have created a PR to solve this issue already:
Yes, I saw the PR first. So, kubernetes/kubernetes isn't really intended to be vendored into other repos and kubernetes/dns is being bad here. It's dubious to add methods to that kubernetes itself isn't using because someone else might just end up removing them later when they notice that they're not being used... OTOH the PR needs access to the internals of the ebtables code so you couldn't just add it to kubernetes/dns directly without forking all of ...
Thanks for your comments. I saw iptables used that way in node-cache and I thought it would be good for consistency to go down the same route with ebtables. To be honest I am not very keen on forking the entire package, for obvious reasons, but I can understand your reservations about this change.
/assign
Hi Did you manage to discuss this?
yes, sorry, didn't get a chance to get back to this on Friday. (I /lgtm'ed the PR) /remove-triage unresolved
Thanks!

// Input args must follow the format and sequence of ebtables list output. Otherwise, EnsureRule will always create // new rules and causing duplicates. EnsureRule(position RulePosition, table Table, chain Chain, args ...string) (bool, error) <ins> // DeleteRule checks if the specified rule is present and, if so, deletes it. DeleteRule(table Table, chain Chain, args ...string) error </ins> // EnsureChain checks if the specified chain is present and, if not, creates it.  If the rule existed, return true. EnsureChain(table Table, chain Chain) (bool, error) // DeleteChain deletes the specified chain.  If the chain did not exist, return error.


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : The ebtables utility in is missing 3 useful features: method in its Interface table chain Why is this needed: The missing features are useful to allow for: rule deletion as there is a way to add (ensure) a rule but not removing it currently. manipulating the broute table and brouting chain The first feature would also align ebtables with iptables, as a DeleteRule method is available there.
/sig network
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Is there some feature/bugfix you're working on in kubernetes that needs this?
Yes, in kubernetes/dns (node-cache) actually. That PR is working, but it's only in my fork of kubernetes/dns at the moment as it depends on this. It solves this: Adding this code to ebtables would help me avoid re-inventing the wheel there.
... not sure if you've seen that ... but I have created a PR to solve this issue already:
Yes, I saw the PR first. So, kubernetes/kubernetes isn't really intended to be vendored into other repos and kubernetes/dns is being bad here. It's dubious to add methods to that kubernetes itself isn't using because someone else might just end up removing them later when they notice that they're not being used... OTOH the PR needs access to the internals of the ebtables code so you couldn't just add it to kubernetes/dns directly without forking all of ...
Thanks for your comments. I saw iptables used that way in node-cache and I thought it would be good for consistency to go down the same route with ebtables. To be honest I am not very keen on forking the entire package, for obvious reasons, but I can understand your reservations about this change.
/assign
Hi Did you manage to discuss this?
yes, sorry, didn't get a chance to get back to this on Friday. (I /lgtm'ed the PR) /remove-triage unresolved
Thanks!

return exist, nil } <ins> func (runner *runner) DeleteRule(table Table, chain Chain, args ...string) error { exist := true fullArgs := makeFullArgs(table, opListChain, chain, fullMac) out, err := runner.exec.Command(cmdebtables, fullArgs...).CombinedOutput() if err != nil { exist = false } else { exist = checkIfRuleExists(string(out), args...) } if !exist { return nil } fullArgs = makeFullArgs(table, opDeleteRule, chain, args...) out, err = runner.exec.Command(cmdebtables, fullArgs...).CombinedOutput() if err != nil { return fmt.Errorf("Failed to delete rule: %v, output: %s", err, out) } return nil } </ins> func (runner *runner) EnsureChain(table Table, chain Chain) (bool, error) { exist := true


Question:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : The ebtables utility in is missing 3 useful features: method in its Interface table chain Why is this needed: The missing features are useful to allow for: rule deletion as there is a way to add (ensure) a rule but not removing it currently. manipulating the broute table and brouting chain The first feature would also align ebtables with iptables, as a DeleteRule method is available there.
/sig network
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Is there some feature/bugfix you're working on in kubernetes that needs this?
Yes, in kubernetes/dns (node-cache) actually. That PR is working, but it's only in my fork of kubernetes/dns at the moment as it depends on this. It solves this: Adding this code to ebtables would help me avoid re-inventing the wheel there.
... not sure if you've seen that ... but I have created a PR to solve this issue already:
Yes, I saw the PR first. So, kubernetes/kubernetes isn't really intended to be vendored into other repos and kubernetes/dns is being bad here. It's dubious to add methods to that kubernetes itself isn't using because someone else might just end up removing them later when they notice that they're not being used... OTOH the PR needs access to the internals of the ebtables code so you couldn't just add it to kubernetes/dns directly without forking all of ...
Thanks for your comments. I saw iptables used that way in node-cache and I thought it would be good for consistency to go down the same route with ebtables. To be honest I am not very keen on forking the entire package, for obvious reasons, but I can understand your reservations about this change.
/assign
Hi Did you manage to discuss this?
yes, sorry, didn't get a chance to get back to this on Friday. (I /lgtm'ed the PR) /remove-triage unresolved
Thanks!

t.Errorf("expected error: %q", errStr) } } <ins> func TestDeleteRule(t *testing.T) { fcmd := fakeexec.FakeCmd{ CombinedOutputScript: []fakeexec.FakeAction{ // Exists func() ([]byte, []byte, error) { return []byte(`Bridge table: filter Bridge chain: OUTPUT, entries: 4, policy: ACCEPT -j TEST `), nil, nil }, // Fail to delete func() ([]byte, []byte, error) { return nil, nil, &fakeexec.FakeExitError{Status: 2} }, // Does not Exists. func() ([]byte, []byte, error) { return []byte(`Bridge table: filter Bridge chain: TEST, entries: 0, policy: ACCEPT`), nil, nil }, }, } fexec := fakeexec.FakeExec{ CommandScript: []fakeexec.FakeCommandAction{ func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, }, } runner := New(&fexec) err := runner.DeleteRule(TableFilter, ChainOutput, "-j", "TEST") errStr := "Failed to delete rule: exit 2, output: " if err == nil || err.Error() != errStr { t.Errorf("expected error: %q", errStr) } err = runner.DeleteRule(TableFilter, ChainOutput, "-j", "TEST") if err != nil { t.Errorf("expected err = nil") } } </ins>


Question:
/sig testing /sig release
I'll work on this one /assign
thanks! /remove-sig release (on second thought this is just SIG Testing)
/lifecycle active

touch "staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go" echo > api/api-rules/violation_exceptions.list echo > api/api-rules/codegen_violation_exceptions.list <del> if make generated_files >/dev/null; then </del> <ins> if make generated_files >/dev/null 2>&1; then </ins> echo "Expected make generated_files to fail with API violations." echo "" exit 1


Question:
What happened: Running manually includes this file in LICENSES. Running complains if it is present. The file was incorrectly dropped in This needs to be resolved before 1.19 ships /priority important-soon /milestone v1.19 /cc
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
fixed in

<ins> = vendor/go.opencensus.io licensed under: = Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!)  The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. = vendor/go.opencensus.io/LICENSE 175792518e4ac015ab6696d16c4f607e </ins>


Question:
What happened: Running manually includes this file in LICENSES. Running complains if it is present. The file was incorrectly dropped in This needs to be resolved before 1.19 ships /priority important-soon /milestone v1.19 /cc
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
fixed in

continue fi # Skip a directory if 1) it has no files and 2) all the subdirectories contain a go.mod file. <del> if [[ -z "$(find "${DEPS_DIR}/${PACKAGE}" -mindepth 1 -maxdepth 1 -type f)" ]]; then </del> <ins> if [[ -z "$(find "${DEPS_DIR}/${PACKAGE}/" -mindepth 1 -maxdepth 1 -type f)" ]]; then </ins> misses_go_mod=false while read -d "" -r SUBDIR; do if [[ ! -e "${SUBDIR}/go.mod" ]]; then misses_go_mod=true break fi <del> done < <(find "${DEPS_DIR}/${PACKAGE}" -mindepth 1 -maxdepth 1 -type d -print0) </del> <ins> done < <(find "${DEPS_DIR}/${PACKAGE}/" -mindepth 1 -maxdepth 1 -type d -print0) </ins> if [[ $misses_go_mod = false ]]; then echo "${PACKAGE} has no files, skipping" >&2 continue


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet fails to create kubepods cgroup on larger machine with 512 cpus, fails with the following error: What you expected to happen: Run kubelet properly without any error How to reproduce it (as minimally and precisely as possible): Start the kubelet on a machine which has a more processors Anything else we need to know?: Environment: Kubernetes version (use ): 18.5 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
When I changed CgroupDriver to cgroupfs, kubelet is running is fine:
And here is the sample code which is creating this issue:
so this is essentially a runc/libcontainer problem or a problem with the host configuration - e.g. the host having less than the requested ? this line: hints that your snippet is for an older version of runc. rc8 renamed it to "Manager": is that the version of the kubelet too? k/k 1.18 is using rc10: so ideally the match test for kubelet 1.18.5 should be against rc10 of runc. and current master of k/k is at rc90: are you able to reproduce the same with rc90?
I tried running with master kublet which does use rc90 build and kubelet fails with the same error: complete log: I think this is happening because we are trying to create a cgroup more than a supported range for the Linux kernel: 2^18 = is the max value we can use for the CPUShare, so wondering whether can add logic into the kubelet to address this issue when we run on bigger machines(more than 256 processors)
/cc
interesting snippet in moby: ! where
i think we have a misconception here that CPU shares == # of CPU. "cpu shares" is a weighting proportion to give a container more resources in comparison to other running containers: the reason for 2^18 in the kernel is to have a larger integer scale to work with. unlike docker that has (nanocpu / fractional cpus) (): does not have such an option. but it does expose (CpusetCpus) which accepts a such as . i have a wild guess that this same problem is not present if you use dockershim. to see more on what is happening you could: run the kubelet with --v=4 or more. patch the kubelet to add debug code.
sorry for the late reply. I think the issue is not in libcontainer, but in the Kubelet asking for a out of range value for cpu shares. I am not sure how it makes sense to set the number of shares in relation to the number of available CPUs, since it is a weight and not a limit. The error you are seeing is coming straight from systemd: Since there is already a check for the minimum value, I think it makes sense to add also a check for the maximum allowed (while we figure out why cpu shares are used this way). Could you give a try to the current patch? I've nothing close to 512 cores to validate it really solves the issue you've seen :smile: I'll open a PR once we are sure it solves the problem
You are right, this is coming from the systemd library which is getting used in the libcontainer With the above patch, I see kubelet is running fine but I don't think this is just good enough because while requesting the cpus in the pod is in terms of milicores which is 1000th of 1 cpu which is getting converted into and assigned to the pod's cgroup. e.g: if the user requests a cpu of which 1024/ = ~2 cpu share out 512 which brakes the current logic in the code. I think we need to add an additional logic for allocation based on the number of cpus present in the machine. Simple math should work fine if cpus are more than 256 in the system: e.g: if the request is 1024m 1024/(()/) = 512 shares
the value doesn't really limit the cpu time a process gets. It is a weight used for the cgroups at the same level. It looks to me that the Kubelet is using this mechanism simply to maintain some sort of priorities among the different pods (i.e. if pod A has double millicpus than pod B, then A gets double weight than B). Before adding more complexity, I'd make sure why it is done this way, I think it is just wrong to put in relation to the number of cores. The cpu shares assigned to each pod though are not dependent on the number of cores, but they are converted directly from the spec file. You've reported the issue on systemd, and it happens since systemd adds another check that is not present with cgroupfs or the kernel. The kernel does the same clamping I've proposed above: so the patch is just to make sure the systemd cgroup driver behaves in the same way as cgroupfs; and I think it doesn't have any impact on the pods
I've opened a PR here:
Thanks for the PR, my only worry about - how k8s will prioritize when we have two pods requesting more than the MaxShares how this logic will prioritize it!
for now it works in the same way as in the cgroupfs case. Pod requests are not in relationship with the # of cores, that AFAICS happens only when setting up the node cgroups. I think a static allocation would work better for such cgroups, as anyway is a weight.

) const ( <del> // Taken from lmctfy https://github.com/google/lmctfy/blob/master/lmctfy/controllers/cpu_controller.cc MinShares     = 2 </del> <ins> // These limits are defined in the kernel: // https://github.com/torvalds/linux/blob/0bddd227f3dc55975e2b8dfa7fc6f959b062a2c7/kernel/sched/sched.h#L427-L428 MinShares = 2 MaxShares = 262144 </ins> SharesPerCPU  = 1024 MilliCPUToCPU = 1000


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet fails to create kubepods cgroup on larger machine with 512 cpus, fails with the following error: What you expected to happen: Run kubelet properly without any error How to reproduce it (as minimally and precisely as possible): Start the kubelet on a machine which has a more processors Anything else we need to know?: Environment: Kubernetes version (use ): 18.5 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
When I changed CgroupDriver to cgroupfs, kubelet is running is fine:
And here is the sample code which is creating this issue:
so this is essentially a runc/libcontainer problem or a problem with the host configuration - e.g. the host having less than the requested ? this line: hints that your snippet is for an older version of runc. rc8 renamed it to "Manager": is that the version of the kubelet too? k/k 1.18 is using rc10: so ideally the match test for kubelet 1.18.5 should be against rc10 of runc. and current master of k/k is at rc90: are you able to reproduce the same with rc90?
I tried running with master kublet which does use rc90 build and kubelet fails with the same error: complete log: I think this is happening because we are trying to create a cgroup more than a supported range for the Linux kernel: 2^18 = is the max value we can use for the CPUShare, so wondering whether can add logic into the kubelet to address this issue when we run on bigger machines(more than 256 processors)
/cc
interesting snippet in moby: ! where
i think we have a misconception here that CPU shares == # of CPU. "cpu shares" is a weighting proportion to give a container more resources in comparison to other running containers: the reason for 2^18 in the kernel is to have a larger integer scale to work with. unlike docker that has (nanocpu / fractional cpus) (): does not have such an option. but it does expose (CpusetCpus) which accepts a such as . i have a wild guess that this same problem is not present if you use dockershim. to see more on what is happening you could: run the kubelet with --v=4 or more. patch the kubelet to add debug code.
sorry for the late reply. I think the issue is not in libcontainer, but in the Kubelet asking for a out of range value for cpu shares. I am not sure how it makes sense to set the number of shares in relation to the number of available CPUs, since it is a weight and not a limit. The error you are seeing is coming straight from systemd: Since there is already a check for the minimum value, I think it makes sense to add also a check for the maximum allowed (while we figure out why cpu shares are used this way). Could you give a try to the current patch? I've nothing close to 512 cores to validate it really solves the issue you've seen :smile: I'll open a PR once we are sure it solves the problem
You are right, this is coming from the systemd library which is getting used in the libcontainer With the above patch, I see kubelet is running fine but I don't think this is just good enough because while requesting the cpus in the pod is in terms of milicores which is 1000th of 1 cpu which is getting converted into and assigned to the pod's cgroup. e.g: if the user requests a cpu of which 1024/ = ~2 cpu share out 512 which brakes the current logic in the code. I think we need to add an additional logic for allocation based on the number of cpus present in the machine. Simple math should work fine if cpus are more than 256 in the system: e.g: if the request is 1024m 1024/(()/) = 512 shares
the value doesn't really limit the cpu time a process gets. It is a weight used for the cgroups at the same level. It looks to me that the Kubelet is using this mechanism simply to maintain some sort of priorities among the different pods (i.e. if pod A has double millicpus than pod B, then A gets double weight than B). Before adding more complexity, I'd make sure why it is done this way, I think it is just wrong to put in relation to the number of cores. The cpu shares assigned to each pod though are not dependent on the number of cores, but they are converted directly from the spec file. You've reported the issue on systemd, and it happens since systemd adds another check that is not present with cgroupfs or the kernel. The kernel does the same clamping I've proposed above: so the patch is just to make sure the systemd cgroup driver behaves in the same way as cgroupfs; and I think it doesn't have any impact on the pods
I've opened a PR here:
Thanks for the PR, my only worry about - how k8s will prioritize when we have two pods requesting more than the MaxShares how this logic will prioritize it!
for now it works in the same way as in the cgroupfs case. Pod requests are not in relationship with the # of cores, that AFAICS happens only when setting up the node cgroups. I think a static allocation would work better for such cgroups, as anyway is a weight.

if shares < MinShares { return MinShares } <ins> if shares > MaxShares { return MaxShares } </ins> return uint64(shares) }


Question:
These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions

"reason_cache.go", "runonce.go", "runtime.go", <ins> "time_cache.go", </ins> "util.go", "volume_host.go", ],


Question:
These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions

"pod_workers_test.go", "reason_cache_test.go", "runonce_test.go", <ins> "time_cache_test.go", </ins> ], embed = [":go_default_library"], deps = [


Question:
These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions


Question:
These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions

experimentalHostUserNamespaceDefaulting: utilfeature.DefaultFeatureGate.Enabled(features.ExperimentalHostUserNamespaceDefaultingGate), keepTerminatedPodVolumes:                keepTerminatedPodVolumes, nodeStatusMaxImages:                     nodeStatusMaxImages, <ins> lastContainerStartedTime:                newTimeCache(), </ins> } if klet.cloud != nil {


Question:
These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions

// lastStatusReportTime is the time when node status was last reported. lastStatusReportTime time.Time <ins> // lastContainerStartedTime is the time of the last ContainerStarted event observed per pod lastContainerStartedTime *timeCache </ins> // syncNodeStatusMux is a lock on updating the node status, because this path is not thread-safe. // This lock is used by Kubelet.syncNodeStatus function and shouldn't be used anywhere else. syncNodeStatusMux sync.Mutex


Question:
These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions

} kl.podWorkers.ForgetWorker(pod.UID) <ins> // make sure our runtimeCache is at least as fresh as the last container started event we observed. // this ensures we correctly send graceful deletion signals to all containers we've reported started. if lastContainerStarted, ok := kl.lastContainerStartedTime.Get(pod.UID); ok { if err := kl.runtimeCache.ForceUpdateIfOlder(lastContainerStarted); err != nil { return fmt.Errorf("error updating containers: %v", err) } } </ins> // Runtime cache may not have been updated to with the pod, but it's okay // because the periodic cleanup routine will attempt to delete again later. runningPods, err := kl.runtimeCache.GetPods()


Question:
These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions

kl.sourcesReady.AddSource(u.Source) case e := <-plegCh: <ins> if e.Type == pleg.ContainerStarted { // record the most recent time we observed a container start for this pod. // this lets us selectively invalidate the runtimeCache when processing a delete for this pod // to make sure we don't miss handling graceful termination for containers we reported as having started. kl.lastContainerStartedTime.Add(e.ID, time.Now()) } </ins> if isSyncPodWorthy(e) { // PLEG event for a pod; sync it. if pod, ok := kl.podManager.GetPodByUID(e.ID); ok {


Question:
These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions

<ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package kubelet import ( "sync" "time" "github.com/golang/groupcache/lru" "k8s.io/apimachinery/pkg/types" ) // timeCache stores a time keyed by uid type timeCache struct { lock  sync.RWMutex cache *lru.Cache } // maxTimeCacheEntries is the cache entry number in lru cache. 1000 is a proper number // for our 100 pods per node target. If we support more pods per node in the future, we // may want to increase the number. const maxTimeCacheEntries = 1000 func newTimeCache() *timeCache { return &timeCache{cache: lru.New(maxTimeCacheEntries)} } func (c *timeCache) Add(uid types.UID, t time.Time) { c.lock.Lock() defer c.lock.Unlock() c.cache.Add(uid, t) } func (c *timeCache) Remove(uid types.UID) { c.lock.Lock() defer c.lock.Unlock() c.cache.Remove(uid) } func (c *timeCache) Get(uid types.UID) (time.Time, bool) { c.lock.RLock() defer c.lock.RUnlock() value, ok := c.cache.Get(uid) if !ok { return time.Time{}, false } t, ok := value.(time.Time) if !ok { return time.Time{}, false } return t, true } </ins>


Question:
These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions

<ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package kubelet import ( "testing" "time" "github.com/golang/groupcache/lru" ) func TestTimeCache(t *testing.T) { cache := &timeCache{cache: lru.New(2)} if a, ok := cache.Get("123"); ok { t.Errorf("expected cache miss, got %v, %v", a, ok) } now := time.Now() soon := now.Add(time.Minute) cache.Add("now", now) cache.Add("soon", soon) if a, ok := cache.Get("now"); !ok || !a.Equal(now) { t.Errorf("expected cache hit matching %v, got %v, %v", now, a, ok) } if a, ok := cache.Get("soon"); !ok || !a.Equal(soon) { t.Errorf("expected cache hit matching %v, got %v, %v", soon, a, ok) } then := now.Add(-time.Minute) cache.Add("then", then) if a, ok := cache.Get("now"); ok { t.Errorf("expected cache miss from oldest evicted value, got %v, %v", a, ok) } if a, ok := cache.Get("soon"); !ok || !a.Equal(soon) { t.Errorf("expected cache hit matching %v, got %v, %v", soon, a, ok) } if a, ok := cache.Get("then"); !ok || !a.Equal(then) { t.Errorf("expected cache hit matching %v, got %v, %v", then, a, ok) } } </ins>


Question:
Which jobs are failing: ci-kubernetes-node-kubelet-conformance Which test(s) are failing: Variable Expansion should succeed in writing subpaths in container Variable Expansion should verify that a failing subpath expansion can be modified during the lifecycle of a container Since when has it been failing: Since Fri Jul 10 15:41:46 2020 -0700, when was merged Testgrid link: https://k8s- Reason for failure: Used DeletePropagationForeground option to delete test pod introduced by Anything else we need to know: The test pod stuck in Terminating state until 5m timeout of DeletePodWithWait expired. It happens with empty pod as well. Here is a simplified ginkgo test case that triggers this failure: The pod phase is "Failed" and container reason is "Error" for some reason: Reverting above commit or changing delete option to DeletePropagationBackground should fix this.
/cc
/sig node
/sig storage cc
/assign
I think change just triggered the underlying issue. DeletePropagationForeground option doesn't work as expected it seems.
does the ci-kubernetes-node-kubelet-conformance job run the controller-manager? the kube-controller-manager is responsible for deleting objects with a foreground finalizer on them, but I don't see any kube-controller-manager logs gathered in the artifacts for one of those test failures -
I could reproduce the issue for commit (before 'generic ephemeral inline volumes' PR was merged) + 'use foreground deletion' commit.
yeah, I tend to think switching that e2e helper method for all callers was not a good idea
it looks like node e2e runs only etcd, api server and namespace controller: I don't think it makes sense to run controller-manager just to fix this failure. Would modifying test framework to set DeletePropagationBackground option for e2e node tests be correct solution?
For a quick fix, I would suggest just reverting the test change for now. Long term, yes, it should be configurable per test.
Also do we have plans to make this job a blocking presubmit test so we can find these problems earlier?
I agree, we should revert , both to avoid changing behavior for existing callers and to match how pods are normally deleted
if we need a PR for that I can prepare it.
that would be great, thanks
putting my hat on, probably not? more hour+ tests that run on every PR are a non-goal
The SIG / OWNERs should enable testgrid alerts on failures, if periodic testing is not sufficient you can use a postsubmit to run whenever we merge code. We're not looking to rely further on presubmit, as is we're running too much there leading to excessive resource usage and inability to merge fixes like this in a timely manner between one thing or the other flaking heavily (and our odds get much worse with each new presubmit)

} // DeletePodWithWait deletes the passed-in pod and waits for the pod to be terminated. Resilient to the pod <del> // not existing. Also waits for all owned resources to be deleted. </del> <ins> // not existing. </ins> func DeletePodWithWait(c clientset.Interface, pod *v1.Pod) error { if pod == nil { return nil


Question:
Which jobs are failing: ci-kubernetes-node-kubelet-conformance Which test(s) are failing: Variable Expansion should succeed in writing subpaths in container Variable Expansion should verify that a failing subpath expansion can be modified during the lifecycle of a container Since when has it been failing: Since Fri Jul 10 15:41:46 2020 -0700, when was merged Testgrid link: https://k8s- Reason for failure: Used DeletePropagationForeground option to delete test pod introduced by Anything else we need to know: The test pod stuck in Terminating state until 5m timeout of DeletePodWithWait expired. It happens with empty pod as well. Here is a simplified ginkgo test case that triggers this failure: The pod phase is "Failed" and container reason is "Error" for some reason: Reverting above commit or changing delete option to DeletePropagationBackground should fix this.
/cc
/sig node
/sig storage cc
/assign
I think change just triggered the underlying issue. DeletePropagationForeground option doesn't work as expected it seems.
does the ci-kubernetes-node-kubelet-conformance job run the controller-manager? the kube-controller-manager is responsible for deleting objects with a foreground finalizer on them, but I don't see any kube-controller-manager logs gathered in the artifacts for one of those test failures -
I could reproduce the issue for commit (before 'generic ephemeral inline volumes' PR was merged) + 'use foreground deletion' commit.
yeah, I tend to think switching that e2e helper method for all callers was not a good idea
it looks like node e2e runs only etcd, api server and namespace controller: I don't think it makes sense to run controller-manager just to fix this failure. Would modifying test framework to set DeletePropagationBackground option for e2e node tests be correct solution?
For a quick fix, I would suggest just reverting the test change for now. Long term, yes, it should be configurable per test.
Also do we have plans to make this job a blocking presubmit test so we can find these problems earlier?
I agree, we should revert , both to avoid changing behavior for existing callers and to match how pods are normally deleted
if we need a PR for that I can prepare it.
that would be great, thanks
putting my hat on, probably not? more hour+ tests that run on every PR are a non-goal
The SIG / OWNERs should enable testgrid alerts on failures, if periodic testing is not sufficient you can use a postsubmit to run whenever we merge code. We're not looking to rely further on presubmit, as is we're running too much there leading to excessive resource usage and inability to merge fixes like this in a timely manner between one thing or the other flaking heavily (and our odds get much worse with each new presubmit)

} // DeletePodWithWaitByName deletes the named and namespaced pod and waits for the pod to be terminated. Resilient to the pod <del> // not existing. Also waits for all owned resources to be deleted. </del> <ins> // not existing. </ins> func DeletePodWithWaitByName(c clientset.Interface, podName, podNamespace string) error { e2elog.Logf("Deleting pod %q in namespace %q", podName, podNamespace) <del> deletionPolicy := metav1.DeletePropagationForeground err := c.CoreV1().Pods(podNamespace).Delete(context.TODO(), podName, metav1.DeleteOptions{ // If the pod is the owner of some resources (like ephemeral inline volumes), // then we want to be sure that those are also gone before we return. // Blocking pod deletion via metav1.DeletePropagationForeground achieves that. PropagationPolicy: &deletionPolicy, }) </del> <ins> err := c.CoreV1().Pods(podNamespace).Delete(context.TODO(), podName, metav1.DeleteOptions{}) </ins> if err != nil { if apierrors.IsNotFound(err) { return nil // assume pod was already deleted


Question:
Which jobs are flaking: ci-kubernetes-kind-e2e-parallel pr:pull-kubernetes-e2e-gce pr:pull-kubernetes-e2e-gce-ubuntu-containerd pr:pull-kubernetes-e2e-gce-ubuntu-containerd ci-kubernetes-kind-ipv6-e2e-parallel Which test(s) are flaking: Second flakiest test in the e2e job according to Testgrid link: Reason for failure: Anything else we need to know: /sig network /assign
needs investigation since we're switching kube-proxy to endpointslice in this release
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
made me think of , but that's in already.
It's possible it's a test timeout that's just too close, but we need verification of that checking the controller and apiserver logs to make sure there isn't a latent bug
Thanks for catching this! It looks like the timeouts are actually associated with waiting for 2 pods to be ready, not actually anything related to EndpointSlice functionality - timeout is here: Given that, it seems like the obvious solution would be to update the Services to set to and update this block to wait for the Pods to have IPs, not for them to be ready. If that sounds reasonable, I'll work on getting a fix in tomorrow.
/remove-triage unresolved
If that is still exercises the same conditions (I vaguely remember publishNotReadyAddresses changing code paths in the endpoints controller... does it do the same in the endpointslice controller?), that seems reasonable to me.
I think the code path is very similar here, the only real difference should be how the EndpointSlice controller calculates readiness. I've gone ahead and filed a PR that uses this approach, hopefully it helps:
This flaked again this morning /reopen
Reopened this issue. <detailsIn response to : This flaked again this morning Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This should be /priority critical-urgent given
The top level error message does not make this clear but the issue here is that too many EndpointSlices are being created - 3 instead of 2.
Interesting... that condition should not persist for O(minutes), right? that seems like it might be a bug, not just a test issue
There's potential that it's not consistently merging 2 ports with 1 endpoint into a single slice and instead creating a slice per port. It also could represent a larger bug. Trying to find a way to consistently reproduce this.
Despite my best efforts I have been unable to reproduce this on my own e2e cluster. I've had 10 test runs going at the same time repeatedly without any luck, any ideas for how to reproduce flaky test failures local would be welcome. In the meantime, I've created a follow up PR that will provide more helpful information if/when this test fails again: My leading theory right now is that this could be a caching issue. The EndpointSlice controller runs a sync, sees that we don't have an EndpointSlice for the given port combination in cache and then goes ahead and creates one, even though a matching EndpointSlice already exists and has not been cached yet. That would result in an annoying situation where an essentially duplicate EndpointSlice was created due to a stale cache. I would think the EndpointSliceTracker would resolve this issue for us though, so not sure. Getting the above PR in will help validate this theory. It's worth noting that if this is actually a duplicate EndpointSlice, clients like kube-proxy will handle this seamlessly because they already have to handle potential duplication in EndpointSlices. That would potentially explain why a test failure like this does not seem to be causing other things to break.
It appears that my theory was correct as witnessed here: I create a . The key points I've noticed so far: 1 EndpointSlice references pod2 (example-named-port-f7vq5) 2 EndpointSlices reference pod1 (example-named-port-fkgcc and example-named-port-vfp2q) 2 EndpointSlices were created at "2020-07-28T04:57:51Z": example-named-port-f7vq5 (references pod2) example-named-port-vfp2q (references pod1) 1 EndpointSlice was created 1 second later ("2020-07-28T04:57:52Z") example-named-port-fkgcc (references pod1) All 3 EndpointSlices are: On generation 1 - no updates have occurred With a state of ready: true The 2 EndpointSlices referencing pod1 are effectively identical  Of course this is something the EndpointSliceTracker should be handling for us, but even that approach seems less than ideal here. It wouldn't actually prevent the duplicate EndpointSlices from being created, it should just be cleaning up duplicates after they happen. This combined with the race conditions referenced in make me think we may want to find a way to slow down the sync cycle for each Service here. This also feels at least tangentially related to the need for better rate limiting in both this and the Endpoints controller discussed in . /cc
Hmm - actually I don't understand why this is happening. Maybe I'm missing something, but I don't think slowing down controller is needed to solve this problem. Even though the controller usually is not lagging my more than 1s, it is possible that it can be lagging by a minute or more in pathological cases. So we can't just rely on the fact that it's guaranteed to be synced. Can you explain it a bit deeper why this is happening? i.e. some more detailed timeline like: T1: state of cache is XYZ T2: controller computes the ES x1, x2 T3: state of cache in ABC T4: controller computes ES y1 [why?]
Agree, I'm not in favor of slowing down the controller globally to address this. Delaying resync of a particular service until we observe the existence of new endpointslices we know we created (with some max delay) could be reasonable.
That's exactly where I was hoping to end up here, although I'm not quite sure how to do that cleanly yet. To be clear, this is not just to fix this flaky test. This flaky test is a symptom of a much larger and more annoying issue with the current implementation. The EndpointSlice controller is consistently running into errors where it's trying to update EndpointSlices that are stale. This is caused by follow up calls to running before the informer cache is up to date, which happens all the time during rolling updates where rapid churn of EndpointSlices is expected. Although this eventually resolves itself, it's causing a lot more load on API Server than necessary and can potentially result in some bad states where an update fails but a delete succeeds. In this specific instance, it looks like a syncService run is occurring after the first 2 EndpointSlices have been created but before the one referencing pod1 is in the informer cache. That results in the controller creating a new EndpointSlice to reference pod1 and thus we end up with a duplicate. If anything in the future triggers another syncService call that would be resolved. I'd initially thought that EndpointSliceTracker would be thing to trigger that follow up syncService, but I was wrong. That tracker just tracks the latest resource version for each EndpointSlice the controller has created/updated and compares that with EndpointSlices received through create/update events. If the controller doesn't think that EndpointSlice should exist or the controller's resource version for that slice does not match it triggers another syncService. In this case neither of those conditions would be met so no additional syncService call would be triggered. So that's a long way of saying that I think our best possible solution here is to introduce some delay between when Services can be synced. I think the easiest solution that would solve 99% of our problems would be to add something like a 1s delay in between syncs simply by delaying when was called. This would not be ideal, but it would result in an immediate improvement and be easy to backport. It's worth noting that although duplicate EndpointSlices like this or even attempts to update a stale EndpointSlice are not ideal, they are not actually invalid and will eventually resolve themselves. Consumers of this API already have to do some form of deduping. With that said, I'm hopeful that there's a clean way to implement a solution that actually waited for each EndpointSlice change the controller has made to be reflected in the cache because that would be a better solution. I'll spend some time looking into possible implementations here.
I've created a draft PR that contains my initial progress towards the approach I described above. I don't particularly love this approach so I'm very open to other ways we could solve this issue. It's also still very much a WIP and has significant work remaining. I opened the PR early just so I could get some initial feedback on the direction and potentially inspire other ideas around approaches we could take here.
is still the best approach I can come up with here. It is ready for review if anyone has time to take a look.
Quick update here, I've limited the scope here to the following changes as discussed in Update the flaky test to consider duplicate EndpointSlices valid. Update the docs to make it very clear that consumers of the EndpointSlice API will have to do deduplication (as kube-proxy already does). Update the controller error handling to return immediately if a create, update, or delete errors (in that order). Reviews would be appreciated on all of the above.
Reopening since is not yet merged.
Thanks for catching that Closing now that is merged. /close
Closing this issue. <detailsIn response to : Thanks for catching that Closing now that is merged. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

namespace := "default" name := "kubernetes" endpoints, err := cs.CoreV1().Endpoints(namespace).Get(context.TODO(), name, metav1.GetOptions{}) <del> framework.ExpectNoError(err) </del> <ins> framework.ExpectNoError(err, "error creating Endpoints resource") </ins> if len(endpoints.Subsets) != 1 { framework.Failf("Expected 1 subset in endpoints, got %d: %#v", len(endpoints.Subsets), endpoints.Subsets) } endpointSubset := endpoints.Subsets[0] endpointSlice, err := cs.DiscoveryV1beta1().EndpointSlices(namespace).Get(context.TODO(), name, metav1.GetOptions{}) <del> framework.ExpectNoError(err) </del> <ins> framework.ExpectNoError(err, "error creating EndpointSlice resource") </ins> if len(endpointSlice.Ports) != len(endpointSubset.Ports) { framework.Failf("Expected EndpointSlice to have %d ports, got %d: %#v", len(endpointSubset.Ports), len(endpointSlice.Ports), endpointSlice.Ports) }