Question:
Graduate seccomp to GA, as defined on the respective . This is an umbrella issue to track all outstanding work. [x] API Server changes (MERGED) [x] Kubelet changes (MERGED) [x] Mark flag as deprecated (MERGED) [x] Update PSP to support seccomp fields (MERGED) [x] Document Kubernetes seccomp support (MERGED)[x] Add seccomp least privilege for docker sandbox (MERGED) [x] Add seccomp least privilege for kuberuntime (MERGED) ) /area kubelet /area kubelet-api /sig node /area security /assign
/cc
just a heads up that we have concluded all the core items for seccomp to go GA. There are still 2 pending PRs that ideally would also go in 1.19, but we are waiting for them to be reviewed: and
needs to be rebased apparently and isn't just waiting in the merge pool as verify is not passing.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I think the warning mechanism was blocked by a not existing API. This is now tracked in
/remove-lifecycle stale
Updated the description to given that the warning mechanism is now set to that milestone.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/priority backlog
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/lifecycle frozen What is the remaining work here?
I guess we have to follow-up on and implement: I can take care of that if we agree that we want to ship that in 1.27. /cc

} } <del> func TestApplySeccompVersionSkew(t *testing.T) { const containerName = "container" testProfile := "test" for _, test := range []struct { description string pod         *api.Pod validation  func(*testing.T, *api.Pod) }{ { description: "Security context nil", pod:         &api.Pod{}, validation: func(t *testing.T, pod *api.Pod) { require.NotNil(t, pod) }, }, { description: "Security context not nil", pod: &api.Pod{ Spec: api.PodSpec{SecurityContext: &api.PodSecurityContext{}}, }, validation: func(t *testing.T, pod *api.Pod) { require.NotNil(t, pod) }, }, { description: "Field set and no annotation present", pod: &api.Pod{ Spec: api.PodSpec{ SecurityContext: &api.PodSecurityContext{ SeccompProfile: &api.SeccompProfile{ Type: api.SeccompProfileTypeUnconfined, }, }, }, }, validation: func(t *testing.T, pod *api.Pod) { require.Len(t, pod.Annotations, 0) }, }, { description: "Annotation 'unconfined' and no field present", pod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ v1.SeccompPodAnnotationKey: v1.SeccompProfileNameUnconfined, }, }, Spec: api.PodSpec{}, }, validation: func(t *testing.T, pod *api.Pod) { require.Equal(t, api.SeccompProfileTypeUnconfined, pod.Spec.SecurityContext.SeccompProfile.Type) require.Nil(t, pod.Spec.SecurityContext.SeccompProfile.LocalhostProfile) }, }, { description: "Annotation 'runtime/default' and no field present", pod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ v1.SeccompPodAnnotationKey: v1.SeccompProfileRuntimeDefault, }, }, Spec: api.PodSpec{SecurityContext: &api.PodSecurityContext{}}, }, validation: func(t *testing.T, pod *api.Pod) { require.Equal(t, api.SeccompProfileTypeRuntimeDefault, pod.Spec.SecurityContext.SeccompProfile.Type) require.Nil(t, pod.Spec.SecurityContext.SeccompProfile.LocalhostProfile) }, }, { description: "Annotation 'docker/default' and no field present", pod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ v1.SeccompPodAnnotationKey: v1.DeprecatedSeccompProfileDockerDefault, }, }, Spec: api.PodSpec{SecurityContext: &api.PodSecurityContext{}}, }, validation: func(t *testing.T, pod *api.Pod) { require.Equal(t, api.SeccompProfileTypeRuntimeDefault, pod.Spec.SecurityContext.SeccompProfile.Type) require.Nil(t, pod.Spec.SecurityContext.SeccompProfile.LocalhostProfile) }, }, { description: "Annotation 'localhost/test' and no field present", pod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ v1.SeccompPodAnnotationKey: v1.SeccompLocalhostProfileNamePrefix + testProfile, }, }, Spec: api.PodSpec{SecurityContext: &api.PodSecurityContext{}}, }, validation: func(t *testing.T, pod *api.Pod) { require.Equal(t, api.SeccompProfileTypeLocalhost, pod.Spec.SecurityContext.SeccompProfile.Type) require.Equal(t, testProfile, *pod.Spec.SecurityContext.SeccompProfile.LocalhostProfile) }, }, { description: "Annotation 'localhost/' has zero length", pod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ v1.SeccompPodAnnotationKey: v1.SeccompLocalhostProfileNamePrefix, }, }, Spec: api.PodSpec{SecurityContext: &api.PodSecurityContext{}}, }, validation: func(t *testing.T, pod *api.Pod) { require.Nil(t, pod.Spec.SecurityContext.SeccompProfile) }, }, { description: "Security context nil (container)", pod: &api.Pod{ Spec: api.PodSpec{ Containers: []api.Container{{}}, }, }, validation: func(t *testing.T, pod *api.Pod) { require.NotNil(t, pod) }, }, { description: "Security context not nil (container)", pod: &api.Pod{ Spec: api.PodSpec{ Containers: []api.Container{{ SecurityContext: &api.SecurityContext{}, }}, }, }, validation: func(t *testing.T, pod *api.Pod) { require.NotNil(t, pod) }, }, { description: "Field set and no annotation present (container)", pod: &api.Pod{ Spec: api.PodSpec{ Containers: []api.Container{ { Name: containerName, SecurityContext: &api.SecurityContext{ SeccompProfile: &api.SeccompProfile{ Type: api.SeccompProfileTypeUnconfined, }, }, }, }, }, }, validation: func(t *testing.T, pod *api.Pod) { require.Len(t, pod.Annotations, 0) }, }, { description: "Multiple containers with fields (container)", pod: &api.Pod{ Spec: api.PodSpec{ Containers: []api.Container{ { Name: containerName + "1", SecurityContext: &api.SecurityContext{ SeccompProfile: &api.SeccompProfile{ Type: api.SeccompProfileTypeUnconfined, }, }, }, { Name: containerName + "2", }, { Name: containerName + "3", SecurityContext: &api.SecurityContext{ SeccompProfile: &api.SeccompProfile{ Type: api.SeccompProfileTypeRuntimeDefault, }, }, }, }, }, }, validation: func(t *testing.T, pod *api.Pod) { require.Len(t, pod.Annotations, 0) }, }, { description: "Annotation 'unconfined' and no field present (container)", pod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ v1.SeccompContainerAnnotationKeyPrefix + containerName: v1.SeccompProfileNameUnconfined, }, }, Spec: api.PodSpec{ Containers: []api.Container{{ Name: containerName, }}, }, }, validation: func(t *testing.T, pod *api.Pod) { require.Equal(t, api.SeccompProfileTypeUnconfined, pod.Spec.Containers[0].SecurityContext.SeccompProfile.Type) require.Nil(t, pod.Spec.Containers[0].SecurityContext.SeccompProfile.LocalhostProfile) }, }, { description: "Annotation 'runtime/default' and no field present (container)", pod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ v1.SeccompContainerAnnotationKeyPrefix + containerName: v1.SeccompProfileRuntimeDefault, }, }, Spec: api.PodSpec{ Containers: []api.Container{{ Name:            containerName, SecurityContext: &api.SecurityContext{}, }}, }, }, validation: func(t *testing.T, pod *api.Pod) { require.Equal(t, api.SeccompProfileTypeRuntimeDefault, pod.Spec.Containers[0].SecurityContext.SeccompProfile.Type) require.Nil(t, pod.Spec.Containers[0].SecurityContext.SeccompProfile.LocalhostProfile) }, }, { description: "Annotation 'docker/default' and no field present (container)", pod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ v1.SeccompContainerAnnotationKeyPrefix + containerName: v1.DeprecatedSeccompProfileDockerDefault, }, }, Spec: api.PodSpec{ Containers: []api.Container{{ Name:            containerName, SecurityContext: &api.SecurityContext{}, }}, }, }, validation: func(t *testing.T, pod *api.Pod) { require.Equal(t, api.SeccompProfileTypeRuntimeDefault, pod.Spec.Containers[0].SecurityContext.SeccompProfile.Type) require.Nil(t, pod.Spec.Containers[0].SecurityContext.SeccompProfile.LocalhostProfile) }, }, { description: "Multiple containers by annotations (container)", pod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ v1.SeccompContainerAnnotationKeyPrefix + containerName + "1": v1.SeccompLocalhostProfileNamePrefix + testProfile, v1.SeccompContainerAnnotationKeyPrefix + containerName + "3": v1.SeccompProfileRuntimeDefault, }, }, Spec: api.PodSpec{ Containers: []api.Container{ {Name: containerName + "1"}, {Name: containerName + "2"}, {Name: containerName + "3"}, }, }, }, validation: func(t *testing.T, pod *api.Pod) { require.Equal(t, api.SeccompProfileTypeLocalhost, pod.Spec.Containers[0].SecurityContext.SeccompProfile.Type) require.Equal(t, testProfile, *pod.Spec.Containers[0].SecurityContext.SeccompProfile.LocalhostProfile) require.Equal(t, api.SeccompProfileTypeRuntimeDefault, pod.Spec.Containers[2].SecurityContext.SeccompProfile.Type) }, }, { description: "Annotation 'localhost/test' and no field present (container)", pod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ v1.SeccompContainerAnnotationKeyPrefix + containerName: v1.SeccompLocalhostProfileNamePrefix + testProfile, }, }, Spec: api.PodSpec{ Containers: []api.Container{{ Name:            containerName, SecurityContext: &api.SecurityContext{}, }}, }, }, validation: func(t *testing.T, pod *api.Pod) { require.Equal(t, api.SeccompProfileTypeLocalhost, pod.Spec.Containers[0].SecurityContext.SeccompProfile.Type) require.Equal(t, testProfile, *pod.Spec.Containers[0].SecurityContext.SeccompProfile.LocalhostProfile) }, }, } { output := &api.Pod{ ObjectMeta: metav1.ObjectMeta{Annotations: map[string]string{}}, } for i, ctr := range test.pod.Spec.Containers { output.Spec.Containers = append(output.Spec.Containers, api.Container{}) if ctr.SecurityContext != nil && ctr.SecurityContext.SeccompProfile != nil { output.Spec.Containers[i].SecurityContext = &api.SecurityContext{ SeccompProfile: &api.SeccompProfile{ Type:             api.SeccompProfileType(ctr.SecurityContext.SeccompProfile.Type), LocalhostProfile: ctr.SecurityContext.SeccompProfile.LocalhostProfile, }, } } } applySeccompVersionSkew(test.pod) test.validation(t, test.pod) } } </del> func newPodWithHugePageValue(resourceName api.ResourceName, value resource.Quantity) *api.Pod { return &api.Pod{


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I created several Workloads which require up to ten ports for both UDP and TCP protocol. Now my logs get spammed with . What you expected to happen: should consider -combination instead of only . How to reproduce it (as minimally and precisely as possible): Deploy any service with a node-port with both protocols, e.g.: Anything else we need to know?: This was already acknowledged by here: However, as this thread addresses a different bug and seems kind of stuck, i'd like to suggest that this might be addressed in a new context. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Deployed via Rancher v2.4.3
/sig api-machinery
/remove-sig api-machinery /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
How did you get into this situation? The allocator should not have granted the same node port twice?
Oh right, that issue. The only real way out is to de-allocate the NodePorts and then re-allocate them, I think
As this happens for all of my ports in the said Production Application (the one where my log gets spammed) as well as complete fresh installation (where i verified the bug), i think there's no "way out" except fixing the bug. As said, for me it's necessary to assign both protocols to the same port. This might be a niche-use-case, but checking the it's actually very likely for ports to allow tcp AND udp. I'm not sure about the history of this, but it might be that issue was the origin, which was adressed in PR ("Allow services which use same port, different protocol to use the same nodePort for both"). But this PR doesn't contain the adjustment of the said , so still considers the behaviour as invalid, while it's actually allowed and supported.
/assign
Ahh, I thought you meant that you had accidentally been allocated the same nodePorts and I was very confused when you said "all of them". I have reproduced this issue, and I can see it by inspection. The obvious fix would be to collect ports split by protocol, but I think that's not right since we don't store it split. An even simpler fix would be for to build a . Then for each protocol, for each port, build that contains all the port numbers, but remove any duplicates ONLY IF they are duplicated across protocols. Duplicates in the same protocol should be retained (to report an error later). I don't have time to tackle this tonight (and write a test), but it seems like a good bug.
perhaps I can fix this problem /assign
things may be a little more complicated. Right now the is not aware of the protocol of the port so I guess simply rewriting function to return is not enough, the port allocator will still return on same node port even with different protocol. perhaps we need to refactor the to make it protocol-aware.
I don't think we need to do make the allocator be protocol aware. IIRC we allow you to use the same nodeport on different protocols ONLY within a single service. This means (I think) that we can do hyper-local deduplication as we collect the nodeports for each service. If we find the same nodeport number with different protocols, just remove the duplicates because it is allowed. I.e. a service that looks like: return - deduped. if ANY OTHER service tried to use it would be an error at the "rebuilt" allocator Likewise if a single Service claims to use the same nodeport number more than once on the same protocol, it should be an error: return - NOT deduped.
obviously, we want a great unit test here :P
ah got it! thanks for your detailed explanation!
Hi! I have filed a PR to resolve this issue, could you take a look?


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I created several Workloads which require up to ten ports for both UDP and TCP protocol. Now my logs get spammed with . What you expected to happen: should consider -combination instead of only . How to reproduce it (as minimally and precisely as possible): Deploy any service with a node-port with both protocols, e.g.: Anything else we need to know?: This was already acknowledged by here: However, as this thread addresses a different bug and seems kind of stuck, i'd like to suggest that this might be addressed in a new context. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Deployed via Rancher v2.4.3
/sig api-machinery
/remove-sig api-machinery /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
How did you get into this situation? The allocator should not have granted the same node port twice?
Oh right, that issue. The only real way out is to de-allocate the NodePorts and then re-allocate them, I think
As this happens for all of my ports in the said Production Application (the one where my log gets spammed) as well as complete fresh installation (where i verified the bug), i think there's no "way out" except fixing the bug. As said, for me it's necessary to assign both protocols to the same port. This might be a niche-use-case, but checking the it's actually very likely for ports to allow tcp AND udp. I'm not sure about the history of this, but it might be that issue was the origin, which was adressed in PR ("Allow services which use same port, different protocol to use the same nodePort for both"). But this PR doesn't contain the adjustment of the said , so still considers the behaviour as invalid, while it's actually allowed and supported.
/assign
Ahh, I thought you meant that you had accidentally been allocated the same nodePorts and I was very confused when you said "all of them". I have reproduced this issue, and I can see it by inspection. The obvious fix would be to collect ports split by protocol, but I think that's not right since we don't store it split. An even simpler fix would be for to build a . Then for each protocol, for each port, build that contains all the port numbers, but remove any duplicates ONLY IF they are duplicated across protocols. Duplicates in the same protocol should be retained (to report an error later). I don't have time to tackle this tonight (and write a test), but it seems like a good bug.
perhaps I can fix this problem /assign
things may be a little more complicated. Right now the is not aware of the protocol of the port so I guess simply rewriting function to return is not enough, the port allocator will still return on same node port even with different protocol. perhaps we need to refactor the to make it protocol-aware.
I don't think we need to do make the allocator be protocol aware. IIRC we allow you to use the same nodeport on different protocols ONLY within a single service. This means (I think) that we can do hyper-local deduplication as we collect the nodeports for each service. If we find the same nodeport number with different protocols, just remove the duplicates because it is allowed. I.e. a service that looks like: return - deduped. if ANY OTHER service tried to use it would be an error at the "rebuilt" allocator Likewise if a single Service claims to use the same nodeport number more than once on the same protocol, it should be an error: return - NOT deduped.
obviously, we want a great unit test here :P
ah got it! thanks for your detailed explanation!
Hi! I have filed a PR to resolve this issue, could you take a look?

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/net" "k8s.io/apimachinery/pkg/util/runtime" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/wait" corev1client "k8s.io/client-go/kubernetes/typed/core/v1" "k8s.io/client-go/tools/record"


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I created several Workloads which require up to ten ports for both UDP and TCP protocol. Now my logs get spammed with . What you expected to happen: should consider -combination instead of only . How to reproduce it (as minimally and precisely as possible): Deploy any service with a node-port with both protocols, e.g.: Anything else we need to know?: This was already acknowledged by here: However, as this thread addresses a different bug and seems kind of stuck, i'd like to suggest that this might be addressed in a new context. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Deployed via Rancher v2.4.3
/sig api-machinery
/remove-sig api-machinery /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
How did you get into this situation? The allocator should not have granted the same node port twice?
Oh right, that issue. The only real way out is to de-allocate the NodePorts and then re-allocate them, I think
As this happens for all of my ports in the said Production Application (the one where my log gets spammed) as well as complete fresh installation (where i verified the bug), i think there's no "way out" except fixing the bug. As said, for me it's necessary to assign both protocols to the same port. This might be a niche-use-case, but checking the it's actually very likely for ports to allow tcp AND udp. I'm not sure about the history of this, but it might be that issue was the origin, which was adressed in PR ("Allow services which use same port, different protocol to use the same nodePort for both"). But this PR doesn't contain the adjustment of the said , so still considers the behaviour as invalid, while it's actually allowed and supported.
/assign
Ahh, I thought you meant that you had accidentally been allocated the same nodePorts and I was very confused when you said "all of them". I have reproduced this issue, and I can see it by inspection. The obvious fix would be to collect ports split by protocol, but I think that's not right since we don't store it split. An even simpler fix would be for to build a . Then for each protocol, for each port, build that contains all the port numbers, but remove any duplicates ONLY IF they are duplicated across protocols. Duplicates in the same protocol should be retained (to report an error later). I don't have time to tackle this tonight (and write a test), but it seems like a good bug.
perhaps I can fix this problem /assign
things may be a little more complicated. Right now the is not aware of the protocol of the port so I guess simply rewriting function to return is not enough, the port allocator will still return on same node port even with different protocol. perhaps we need to refactor the to make it protocol-aware.
I don't think we need to do make the allocator be protocol aware. IIRC we allow you to use the same nodeport on different protocols ONLY within a single service. This means (I think) that we can do hyper-local deduplication as we collect the nodeports for each service. If we find the same nodeport number with different protocols, just remove the duplicates because it is allowed. I.e. a service that looks like: return - deduped. if ANY OTHER service tried to use it would be an error at the "rebuilt" allocator Likewise if a single Service claims to use the same nodeport number more than once on the same protocol, it should be an error: return - NOT deduped.
obviously, we want a great unit test here :P
ah got it! thanks for your detailed explanation!
Hi! I have filed a PR to resolve this issue, could you take a look?

return nil } <ins> // collectServiceNodePorts returns nodePorts specified in the Service. // Please note that: //   1. same nodePort with *same* protocol will be duplicated as it is //   2. same nodePort with *different* protocol will be deduplicated </ins> func collectServiceNodePorts(service *corev1.Service) []int { <del> servicePorts := []int{} for i := range service.Spec.Ports { servicePort := &service.Spec.Ports[i] if servicePort.NodePort != 0 { servicePorts = append(servicePorts, int(servicePort.NodePort)) </del> <ins> var servicePorts []int // map from nodePort to set of protocols seen := make(map[int]sets.String) for _, port := range service.Spec.Ports { nodePort := int(port.NodePort) if nodePort == 0 { continue } proto := string(port.Protocol) s := seen[nodePort] if s == nil { // have not seen this nodePort before s = sets.NewString(proto) servicePorts = append(servicePorts, nodePort) } else if s.Has(proto) { // same nodePort with same protocol servicePorts = append(servicePorts, nodePort) } else { // same nodePort with different protocol s.Insert(proto) </ins> } <ins> seen[nodePort] = s </ins> } <del> if service.Spec.HealthCheckNodePort != 0 { servicePorts = append(servicePorts, int(service.Spec.HealthCheckNodePort)) </del> <ins> healthPort := int(service.Spec.HealthCheckNodePort) if healthPort != 0 { s := seen[healthPort] // TODO: is it safe to assume the protocol is always TCP? if s == nil || s.Has(string(corev1.ProtocolTCP)) { servicePorts = append(servicePorts, healthPort) } </ins> } return servicePorts


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I created several Workloads which require up to ten ports for both UDP and TCP protocol. Now my logs get spammed with . What you expected to happen: should consider -combination instead of only . How to reproduce it (as minimally and precisely as possible): Deploy any service with a node-port with both protocols, e.g.: Anything else we need to know?: This was already acknowledged by here: However, as this thread addresses a different bug and seems kind of stuck, i'd like to suggest that this might be addressed in a new context. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Deployed via Rancher v2.4.3
/sig api-machinery
/remove-sig api-machinery /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
How did you get into this situation? The allocator should not have granted the same node port twice?
Oh right, that issue. The only real way out is to de-allocate the NodePorts and then re-allocate them, I think
As this happens for all of my ports in the said Production Application (the one where my log gets spammed) as well as complete fresh installation (where i verified the bug), i think there's no "way out" except fixing the bug. As said, for me it's necessary to assign both protocols to the same port. This might be a niche-use-case, but checking the it's actually very likely for ports to allow tcp AND udp. I'm not sure about the history of this, but it might be that issue was the origin, which was adressed in PR ("Allow services which use same port, different protocol to use the same nodePort for both"). But this PR doesn't contain the adjustment of the said , so still considers the behaviour as invalid, while it's actually allowed and supported.
/assign
Ahh, I thought you meant that you had accidentally been allocated the same nodePorts and I was very confused when you said "all of them". I have reproduced this issue, and I can see it by inspection. The obvious fix would be to collect ports split by protocol, but I think that's not right since we don't store it split. An even simpler fix would be for to build a . Then for each protocol, for each port, build that contains all the port numbers, but remove any duplicates ONLY IF they are duplicated across protocols. Duplicates in the same protocol should be retained (to report an error later). I don't have time to tackle this tonight (and write a test), but it seems like a good bug.
perhaps I can fix this problem /assign
things may be a little more complicated. Right now the is not aware of the protocol of the port so I guess simply rewriting function to return is not enough, the port allocator will still return on same node port even with different protocol. perhaps we need to refactor the to make it protocol-aware.
I don't think we need to do make the allocator be protocol aware. IIRC we allow you to use the same nodeport on different protocols ONLY within a single service. This means (I think) that we can do hyper-local deduplication as we collect the nodeports for each service. If we find the same nodeport number with different protocols, just remove the duplicates because it is allowed. I.e. a service that looks like: return - deduped. if ANY OTHER service tried to use it would be an error at the "rebuilt" allocator Likewise if a single Service claims to use the same nodeport number more than once on the same protocol, it should be an error: return - NOT deduped.
obviously, we want a great unit test here :P
ah got it! thanks for your detailed explanation!
Hi! I have filed a PR to resolve this issue, could you take a look?

import ( "fmt" <ins> "reflect" "sort" </ins> "strings" "testing"


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I created several Workloads which require up to ten ports for both UDP and TCP protocol. Now my logs get spammed with . What you expected to happen: should consider -combination instead of only . How to reproduce it (as minimally and precisely as possible): Deploy any service with a node-port with both protocols, e.g.: Anything else we need to know?: This was already acknowledged by here: However, as this thread addresses a different bug and seems kind of stuck, i'd like to suggest that this might be addressed in a new context. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Deployed via Rancher v2.4.3
/sig api-machinery
/remove-sig api-machinery /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
How did you get into this situation? The allocator should not have granted the same node port twice?
Oh right, that issue. The only real way out is to de-allocate the NodePorts and then re-allocate them, I think
As this happens for all of my ports in the said Production Application (the one where my log gets spammed) as well as complete fresh installation (where i verified the bug), i think there's no "way out" except fixing the bug. As said, for me it's necessary to assign both protocols to the same port. This might be a niche-use-case, but checking the it's actually very likely for ports to allow tcp AND udp. I'm not sure about the history of this, but it might be that issue was the origin, which was adressed in PR ("Allow services which use same port, different protocol to use the same nodePort for both"). But this PR doesn't contain the adjustment of the said , so still considers the behaviour as invalid, while it's actually allowed and supported.
/assign
Ahh, I thought you meant that you had accidentally been allocated the same nodePorts and I was very confused when you said "all of them". I have reproduced this issue, and I can see it by inspection. The obvious fix would be to collect ports split by protocol, but I think that's not right since we don't store it split. An even simpler fix would be for to build a . Then for each protocol, for each port, build that contains all the port numbers, but remove any duplicates ONLY IF they are duplicated across protocols. Duplicates in the same protocol should be retained (to report an error later). I don't have time to tackle this tonight (and write a test), but it seems like a good bug.
perhaps I can fix this problem /assign
things may be a little more complicated. Right now the is not aware of the protocol of the port so I guess simply rewriting function to return is not enough, the port allocator will still return on same node port even with different protocol. perhaps we need to refactor the to make it protocol-aware.
I don't think we need to do make the allocator be protocol aware. IIRC we allow you to use the same nodeport on different protocols ONLY within a single service. This means (I think) that we can do hyper-local deduplication as we collect the nodeports for each service. If we find the same nodeport number with different protocols, just remove the duplicates because it is allowed. I.e. a service that looks like: return - deduped. if ANY OTHER service tried to use it would be an error at the "rebuilt" allocator Likewise if a single Service claims to use the same nodeport number more than once on the same protocol, it should be an error: return - NOT deduped.
obviously, we want a great unit test here :P
ah got it! thanks for your detailed explanation!
Hi! I have filed a PR to resolve this issue, could you take a look?

t.Errorf("unexpected portallocator state: %d free", free) } } <ins> func TestCollectServiceNodePorts(t *testing.T) { tests := []struct { name        string serviceSpec corev1.ServiceSpec expected    []int }{ { name: "no duplicated nodePorts", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolTCP}, {NodePort: 112, Protocol: corev1.ProtocolUDP}, {NodePort: 113, Protocol: corev1.ProtocolUDP}, }, }, expected: []int{111, 112, 113}, }, { name: "duplicated nodePort with TCP protocol", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolTCP}, {NodePort: 111, Protocol: corev1.ProtocolTCP}, {NodePort: 112, Protocol: corev1.ProtocolUDP}, }, }, expected: []int{111, 111, 112}, }, { name: "duplicated nodePort with UDP protocol", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolUDP}, {NodePort: 111, Protocol: corev1.ProtocolUDP}, {NodePort: 112, Protocol: corev1.ProtocolTCP}, }, }, expected: []int{111, 111, 112}, }, { name: "duplicated nodePort with different protocol", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolTCP}, {NodePort: 112, Protocol: corev1.ProtocolTCP}, {NodePort: 111, Protocol: corev1.ProtocolUDP}, }, }, expected: []int{111, 112}, }, { name: "no duplicated port(with health check port)", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolTCP}, {NodePort: 112, Protocol: corev1.ProtocolUDP}, }, HealthCheckNodePort: 113, }, expected: []int{111, 112, 113}, }, { name: "nodePort has different protocol with duplicated health check port", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolUDP}, {NodePort: 112, Protocol: corev1.ProtocolTCP}, }, HealthCheckNodePort: 111, }, expected: []int{111, 112}, }, { name: "nodePort has same protocol as duplicated health check port", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolUDP}, {NodePort: 112, Protocol: corev1.ProtocolTCP}, }, HealthCheckNodePort: 112, }, expected: []int{111, 112, 112}, }, } for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { ports := collectServiceNodePorts(&corev1.Service{ ObjectMeta: metav1.ObjectMeta{Namespace: "one", Name: "one"}, Spec:       tc.serviceSpec, }) sort.Ints(ports) if !reflect.DeepEqual(tc.expected, ports) { t.Fatalf("Invalid resultnexpected: %vngot: %v", tc.expected, ports) } }) } } </ins>


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: add a CRD resource with kind only a single character while cause kube-apiserver crashslice bounds out of range What you expected to happen: add validation to limit CRD kind definition or fix the bug of slice out of bounds How to reproduce it (as minimally and precisely as possible): apiVersion: kind: CustomResourceDefinition metadata: name: spec: group: version: v1 names: kind: R plural: rs singular: r scope: Cluster Anything else we need to know?: apiserverlogs: 10:05:32. 6 ] Shutting down OpenAPI controller E6606 10:05:32. 6 ] Observed a panic: runtime.boundsError{x:-1, y:1, signed:true, code:0x3(runtime error: slice bounds out of range [-1:]) goroutine 1670 [running]: (0x41458a0, Gxc00ed66d40) /workspace/anago-v1.17.G- +0xa3 (0x0, 0x0, 0x0) /workspace/anago-v1.17.G- +0x82 panic(0x41458a0, Oxc0Oed66d40) +0x1b2 (0x7493232, 0x1, 0x447d778, 0x1, 0x130, 0xc00ed13680) /workspace/anago-v1.17.0- +0x2b8 ("builder).descriptionFor(€xc007b351f0, 0x0, 0x0, 0x4481982, 0x6, Gxc00ed06d00, 0x18) /workspace/anago-v1.17.0- +0x3c6 ("builder).buildRoute(GxcGG7b351fG, 0xc00ed66d00, 0x18, 0x0, 0x0, 0x447f0F3, 0x4, 0x447f71b, 0x4, 0x4481982, ...) /workspace/anago-v1.17.0- .go :281 +0x197 (0xc66b7bcfd0, 0xc005d24a3c, 0x2, 0x101, 0x1, 0x2, Gxc60b76aec8) /workspace/anago-v1.17.0-rc .2.10+70132b0F130acc/src/k8s .io/kubernetes/output/dockerized/go/src/k8s .io/kubernetes/vendor/k8s .io/apiextensions-apiserver/pkg/controller/openapi/builder/builder .go :146 +0x49d (0xc66b7bcfd0, 0x0, Oxb, Gx447f1db, 0x4, 0x1) /workspace/anago-v1.17.0- +0x1a8 ("Controller).Run(0xc0012c8050, Gxc0G1d61200, 0xc007a6d1e0, 0xc069cced80) /workspace/anago-v1.17.0- +0x2da created by /workspace/anago-v1.17.0- .go :236 +0x244 panic: runtime error: slice bounds out of range [-1:] [recovered] panic: runtime error: slice bounds out of range [-1:] Environment: Kubernetes version (use ): 1.18.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig apps
/assign
There is a typo in the subject.

// GetArticleForNoun returns the article needed for the given noun. func GetArticleForNoun(noun string, padding string) string { <del> if noun[len(noun)-2:] != "ss" && noun[len(noun)-1:] == "s" { </del> <ins> if !strings.HasSuffix(noun, "ss") && strings.HasSuffix(noun, "s") { </ins> // Plurals don't have an article. // Don't catch words like class return fmt.Sprintf("%v", padding)


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: add a CRD resource with kind only a single character while cause kube-apiserver crashslice bounds out of range What you expected to happen: add validation to limit CRD kind definition or fix the bug of slice out of bounds How to reproduce it (as minimally and precisely as possible): apiVersion: kind: CustomResourceDefinition metadata: name: spec: group: version: v1 names: kind: R plural: rs singular: r scope: Cluster Anything else we need to know?: apiserverlogs: 10:05:32. 6 ] Shutting down OpenAPI controller E6606 10:05:32. 6 ] Observed a panic: runtime.boundsError{x:-1, y:1, signed:true, code:0x3(runtime error: slice bounds out of range [-1:]) goroutine 1670 [running]: (0x41458a0, Gxc00ed66d40) /workspace/anago-v1.17.G- +0xa3 (0x0, 0x0, 0x0) /workspace/anago-v1.17.G- +0x82 panic(0x41458a0, Oxc0Oed66d40) +0x1b2 (0x7493232, 0x1, 0x447d778, 0x1, 0x130, 0xc00ed13680) /workspace/anago-v1.17.0- +0x2b8 ("builder).descriptionFor(€xc007b351f0, 0x0, 0x0, 0x4481982, 0x6, Gxc00ed06d00, 0x18) /workspace/anago-v1.17.0- +0x3c6 ("builder).buildRoute(GxcGG7b351fG, 0xc00ed66d00, 0x18, 0x0, 0x0, 0x447f0F3, 0x4, 0x447f71b, 0x4, 0x4481982, ...) /workspace/anago-v1.17.0- .go :281 +0x197 (0xc66b7bcfd0, 0xc005d24a3c, 0x2, 0x101, 0x1, 0x2, Gxc60b76aec8) /workspace/anago-v1.17.0-rc .2.10+70132b0F130acc/src/k8s .io/kubernetes/output/dockerized/go/src/k8s .io/kubernetes/vendor/k8s .io/apiextensions-apiserver/pkg/controller/openapi/builder/builder .go :146 +0x49d (0xc66b7bcfd0, 0x0, Oxb, Gx447f1db, 0x4, 0x1) /workspace/anago-v1.17.0- +0x1a8 ("Controller).Run(0xc0012c8050, Gxc0G1d61200, 0xc007a6d1e0, 0xc069cced80) /workspace/anago-v1.17.0- +0x2da created by /workspace/anago-v1.17.0- .go :236 +0x244 panic: runtime error: slice bounds out of range [-1:] [recovered] panic: runtime error: slice bounds out of range [-1:] Environment: Kubernetes version (use ): 1.18.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig apps
/assign
There is a typo in the subject.

padding: " ", want:    " a ", }, <ins> { noun:    "S", padding: " ", want:    " a ", }, { noun:    "O", padding: " ", want:    " an ", }, </ins> } for _, tt := range tests { if got := GetArticleForNoun(tt.noun, tt.padding); got != tt.want {


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig testing /cc /priority critical-urgent /milestone v1.19
/assign
I see the PR merged, however, the job still failing in the recent run 22:31 PDT Failures for bootstrap-e2e-minion-group (if any): W0611 06:08:56.190] 2020/06/11 06:08:56 Step './cluster/log-dump/log- /workspace/artifacts' finished in 2m14.442417098s W0611 06:08:44.597] ERROR: () [/usr/bin/scp] exited with return code [1]. W0611 06:08:46.070] scp: No such file or directory W0611 06:08:46.070] scp: /var/log/node-problem- No such file or directory W0611 06:08:46.072] scp: No such file or directory W0611 06:08:46.072] scp: No such file or directory W0611 06:08:46.073] scp: No such file or directory W0611 06:08:46.074] scp: /var/log/node-problem- No such file or directory W0611 06:08:46.075] scp: No such file or directory W0611 06:08:46.075] scp: No such file or directory W0611 06:08:46.079] ERROR: () [/usr/bin/scp] exited with return code [1]. W0611 06:08:46.084] ERROR: () [/usr/bin/scp] exited with return code [1]. W0611 06:08:51.917] INSTANCEGROUPS=bootstrap-e2e-minion-group W0611 06:08:51.918] NODENAMES=bootstrap-e2e-minion-group-dq3m bootstrap-e2e-minion-group-gjk0 bootstrap-e2e-minion-group-qpxw I0611 06:08:54.017] Failures for bootstrap-e2e-minion-group (if any): W0611 06:08:56.190] 2020/06/11 06:08:56 Step './cluster/log-dump/log- /workspace/artifacts' finished in 2m14.442417098s W0611 06:08:56.191] 2020/06/11 06:08:56 Running: ./hack/e2e-internal/e2e-
I see the issue closed by the PR however the job still failing since 06-10 16:31 PDT
looks like we just turned green :)
cool! :)
Yes it's indeed ... thanks for the support!

# Set default values with override if [[ "${CONTAINER_RUNTIME}" == "docker" ]]; then <del> export CONTAINER_RUNTIME_ENDPOINT=${KUBE_CONTAINER_RUNTIME_ENDPOINT:-unix:///var/run/docker.sock} </del> <ins> export CONTAINER_RUNTIME_ENDPOINT=${KUBE_CONTAINER_RUNTIME_ENDPOINT:-unix:///var/run/dockershim.sock} </ins> export CONTAINER_RUNTIME_NAME=${KUBE_CONTAINER_RUNTIME_NAME:-docker} export LOAD_IMAGE_COMMAND=${KUBE_LOAD_IMAGE_COMMAND:-} elif [[ "${CONTAINER_RUNTIME}" == "containerd" ]]; then


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: https://k8s- Which test(s) are failing: sig-network] SCTP [Feature:SCTP] [LinuxOnly] should create a Pod with SCTP HostPort Since when has it been failing: Test [sig-network] SCTP [Feature:SCTP] [LinuxOnly] should create a Pod with SCTP HostPort has failed 3 times since it last passed at commit/. It started failing between commit/ (Tue Jun 09 13:25 PDT) and commit/ (Wed Jun 10 01:26 PDT): Testgrid link: https://k8s- Reason for failure: Anything else we need to know: I'm not sure if we already did it but there was a proposal to change from dockershim to containerd for the GCE jobs
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc
GitHub didn't allow me to assign the following users: janosi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yes, that would break us... the test checks "are we using kubenet", but it didn't occur to me that we would need to deal with kubenet with CRI instead of kubenet with dockershim. It should be easy to make it check for both the dockershim rule and the CNI portmap rule though...
yes, according to the "Changes" link on the dashboard, containerd replaced docker in GCE right before the failures.
Let's see if I can make it :)
So actually the test is pretty generic (it just looks for and ) and it seems like it ought to pass correctly with CNI portmap too. I'm not sure what's failing. You may need to add some debugging...
OK, so as I can see, containerd's CRI implementation just ignores all HostPort definitions that do not have TCP or UDP Protocol value. I.e. containerd will not send HostPort mapping definitions to the CNI if the protocol is not TCP or UDP.
seems this is the reason why sctp was skipped
Yes, exactly. Introduced right after SCTP support was merged in k8s. Just found this in the package: Though not sure it "knows" the right runtime at runtime, because there is no user of it in :)
just curious, why do we skip in containerd and allow it with dockershim?
Hmm, do you mean why do not we skip in case of containerd? The reason is that I did not remember that containerd PR about the SCTP support. Let's see if I can skip it.
Hm, that is not true. My search engine was wrong. After a restart I found usage of that in e2e_node
Has a bug been filed upstream in containerd? Just skipping the tests seems like the wrong approach here.
sorry, I meant why we are not allowing it in containerd, does this feature should work in all container runtimes? CRIO, docker and containerd? if affirmative, we can submit a PR to containerd/cri to allow SCTP, I can do it if you don't have time
Excuse me, I misunderstood your question. Yep, why not. I mean, it could work in all container runtimes. As I understand the containerd implementation is based on a misunderstanding: that the host port mapping loads the SCTP kernel module and that would be bad for those apps that want to use user space SCTP. But as long as there is no SCTP socket created on a node the SCTP kernel module is not loaded.
I can initiate a discussion with containerd about this.
seems this dates back to the original SCTP KEP, seems like it should be a pretty simple fix (?) thanks for filing the bug and for the details there!
CRI-O seems to have the same issue:
Just confirming you want this filter to be gone now. Or do you need the filter to be manageable in some way.. filtered by a config switch?
For TCP and UDP HostPorts, the runtime is expected to do two things: a listening socket on that port so the kernel will think it's in use and not allocate it to anyone else iptables rules (or whatever) to route traffic from the host port to the container port For SCTP HostPorts, the runtime should do the second part (iptables rules), but should not do the first part (listening socket), for two reasons: Some users want to use "userspace SCTP", in which case they will be referring to SCTP at the Kubernetes level but need to keep the kernel module from ever being loaded. (This is discussed more .) If is not loaded when Kubernetes starts, we want to avoid having Kubernetes itself cause it to be auto-loaded, for security reasons. (More discussion of this .) So, runtimes should be filtering out SCTP for the opening-listening-sockets part, but not for the making-iptables-rules part. (Some implementations, notably CNI , don't actually do the opening-listening-sockets part, so they wouldn't be filtering at all.)
Exactly. The current state in the runtimes: Docker (dockershim, actually): OK. The logic was implemented together with the basic SCTP feature, so listening on the host port is filtered for SCTP, but the iptables rules are created Containerd (CRI, actually): the patch which was applied 2 years ago filters out every SCTP host port handling, because it filters SCTP ports from the list of port mappings that are passed to CNI (portmap) later. Actually, containerd (CRI) does not perform the "listen" part at all, so the patch is not needed at all CRI-O: the patch applied 2 years ago filters out everything. CRI-O imports the host port manager implementation from kubelet/dockershim. For this reason it has the logic that can filter out SCTP ports from the "listen" part, but still applies the iptables rules. I.e. the patch is not needed in CRI-O either.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale The test is flaky now :thinking: https://k8s-
I guess it is a timing issue? Do you think some extra polling rounds would help? (I did not disappear, but currently I am very loaded with my regular job...)
I've submitted a PR to fix it and another one to test the sctp connectivity, please review them, I've pinged you there :smile:

err := cs.CoreV1().Pods(f.Namespace.Name).Delete(context.TODO(), podName, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete pod: %s in namespace: %s", podName, f.Namespace.Name) }() <del> ginkgo.By("dumping iptables rules on the node") </del> <ins> // wait until host port manager syncs rules </ins> cmd = "sudo iptables-save" <del> framework.Logf("Executing cmd %q on node %v", cmd, node.Name) result, err := hostExec.IssueCommandWithResult(cmd, node) if err != nil { framework.Failf("Interrogation of iptables rules failed on node %v", node.Name) </del> <ins> if framework.TestContext.ClusterIsIPv6() { cmd = "sudo ip6tables-save" </ins> } <ins> err = wait.PollImmediate(framework.Poll, framework.PollShortTimeout, func() (bool, error) { framework.Logf("Executing cmd %q on node %v", cmd, node.Name) result, err := hostExec.IssueCommandWithResult(cmd, node) if err != nil { framework.Logf("Interrogation of iptables rules failed on node %v", node.Name) return false, nil } </ins> <del> ginkgo.By("checking that iptables contains the necessary iptables rules") found := false for _, line := range strings.Split(result, "n") { if strings.Contains(line, "-p sctp") && strings.Contains(line, "--dport 5060") { found = true break </del> <ins> for _, line := range strings.Split(result, "n") { if strings.Contains(line, "-p sctp") && strings.Contains(line, "--dport 5060") { return true, nil } </ins> } <del> } if !found { </del> <ins> framework.Logf("retrying ... not hostport sctp iptables rules found on node %v", node.Name) return false, nil }) if err != nil { </ins> framework.Failf("iptables rules are not set for a pod with sctp hostport") } ginkgo.By("validating sctp module is still not loaded")


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: https://k8s- Which test(s) are failing: sig-network] SCTP [Feature:SCTP] [LinuxOnly] should create a Pod with SCTP HostPort Since when has it been failing: Test [sig-network] SCTP [Feature:SCTP] [LinuxOnly] should create a Pod with SCTP HostPort has failed 3 times since it last passed at commit/. It started failing between commit/ (Tue Jun 09 13:25 PDT) and commit/ (Wed Jun 10 01:26 PDT): Testgrid link: https://k8s- Reason for failure: Anything else we need to know: I'm not sure if we already did it but there was a proposal to change from dockershim to containerd for the GCE jobs
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc
GitHub didn't allow me to assign the following users: janosi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yes, that would break us... the test checks "are we using kubenet", but it didn't occur to me that we would need to deal with kubenet with CRI instead of kubenet with dockershim. It should be easy to make it check for both the dockershim rule and the CNI portmap rule though...
yes, according to the "Changes" link on the dashboard, containerd replaced docker in GCE right before the failures.
Let's see if I can make it :)
So actually the test is pretty generic (it just looks for and ) and it seems like it ought to pass correctly with CNI portmap too. I'm not sure what's failing. You may need to add some debugging...
OK, so as I can see, containerd's CRI implementation just ignores all HostPort definitions that do not have TCP or UDP Protocol value. I.e. containerd will not send HostPort mapping definitions to the CNI if the protocol is not TCP or UDP.
seems this is the reason why sctp was skipped
Yes, exactly. Introduced right after SCTP support was merged in k8s. Just found this in the package: Though not sure it "knows" the right runtime at runtime, because there is no user of it in :)
just curious, why do we skip in containerd and allow it with dockershim?
Hmm, do you mean why do not we skip in case of containerd? The reason is that I did not remember that containerd PR about the SCTP support. Let's see if I can skip it.
Hm, that is not true. My search engine was wrong. After a restart I found usage of that in e2e_node
Has a bug been filed upstream in containerd? Just skipping the tests seems like the wrong approach here.
sorry, I meant why we are not allowing it in containerd, does this feature should work in all container runtimes? CRIO, docker and containerd? if affirmative, we can submit a PR to containerd/cri to allow SCTP, I can do it if you don't have time
Excuse me, I misunderstood your question. Yep, why not. I mean, it could work in all container runtimes. As I understand the containerd implementation is based on a misunderstanding: that the host port mapping loads the SCTP kernel module and that would be bad for those apps that want to use user space SCTP. But as long as there is no SCTP socket created on a node the SCTP kernel module is not loaded.
I can initiate a discussion with containerd about this.
seems this dates back to the original SCTP KEP, seems like it should be a pretty simple fix (?) thanks for filing the bug and for the details there!
CRI-O seems to have the same issue:
Just confirming you want this filter to be gone now. Or do you need the filter to be manageable in some way.. filtered by a config switch?
For TCP and UDP HostPorts, the runtime is expected to do two things: a listening socket on that port so the kernel will think it's in use and not allocate it to anyone else iptables rules (or whatever) to route traffic from the host port to the container port For SCTP HostPorts, the runtime should do the second part (iptables rules), but should not do the first part (listening socket), for two reasons: Some users want to use "userspace SCTP", in which case they will be referring to SCTP at the Kubernetes level but need to keep the kernel module from ever being loaded. (This is discussed more .) If is not loaded when Kubernetes starts, we want to avoid having Kubernetes itself cause it to be auto-loaded, for security reasons. (More discussion of this .) So, runtimes should be filtering out SCTP for the opening-listening-sockets part, but not for the making-iptables-rules part. (Some implementations, notably CNI , don't actually do the opening-listening-sockets part, so they wouldn't be filtering at all.)
Exactly. The current state in the runtimes: Docker (dockershim, actually): OK. The logic was implemented together with the basic SCTP feature, so listening on the host port is filtered for SCTP, but the iptables rules are created Containerd (CRI, actually): the patch which was applied 2 years ago filters out every SCTP host port handling, because it filters SCTP ports from the list of port mappings that are passed to CNI (portmap) later. Actually, containerd (CRI) does not perform the "listen" part at all, so the patch is not needed at all CRI-O: the patch applied 2 years ago filters out everything. CRI-O imports the host port manager implementation from kubelet/dockershim. For this reason it has the logic that can filter out SCTP ports from the "listen" part, but still applies the iptables rules. I.e. the patch is not needed in CRI-O either.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale The test is flaky now :thinking: https://k8s-
I guess it is a timing issue? Do you think some extra polling rounds would help? (I did not disappear, but currently I am very loaded with my regular job...)
I've submitted a PR to fix it and another one to test the sctp connectivity, please review them, I've pinged you there :smile:

err = e2enetwork.WaitForService(f.ClientSet, ns, serviceName, true, 5*time.Second, e2eservice.TestTimeout) framework.ExpectNoError(err, fmt.Sprintf("error while waiting for service:%s err: %v", serviceName, err)) <del> ginkgo.By("dumping iptables rules on a node") </del> hostExec := utils.NewHostExec(f) defer hostExec.Cleanup() node, err := e2enode.GetRandomReadySchedulableNode(cs) framework.ExpectNoError(err) cmd := "sudo iptables-save" <del> framework.Logf("Executing cmd %q on node %v", cmd, node.Name) result, err := hostExec.IssueCommandWithResult(cmd, node) if err != nil { framework.Failf("Interrogation of iptables rules failed on node %v", node.Name) </del> <ins> if framework.TestContext.ClusterIsIPv6() { cmd = "sudo ip6tables-save" </ins> } <ins> err = wait.PollImmediate(framework.Poll, e2eservice.KubeProxyLagTimeout, func() (bool, error) { framework.Logf("Executing cmd %q on node %v", cmd, node.Name) result, err := hostExec.IssueCommandWithResult(cmd, node) if err != nil { framework.Logf("Interrogation of iptables rules failed on node %v", node.Name) return false, nil } </ins> <del> ginkgo.By("checking that iptables contains the necessary iptables rules") kubeService := false for _, line := range strings.Split(result, "n") { if strings.Contains(line, "-A KUBE-SERVICES") && strings.Contains(line, "-p sctp") { kubeService = true break </del> <ins> for _, line := range strings.Split(result, "n") { if strings.Contains(line, "-A KUBE-SERVICES") && strings.Contains(line, "-p sctp") { return true, nil } </ins> } <del> } if !kubeService { </del> <ins> framework.Logf("retrying ... no iptables rules found for service with sctp ports on node %v", node.Name) return false, nil }) if err != nil { </ins> framework.Failf("iptables rules are not set for a clusterip service with sctp ports") } ginkgo.By("validating sctp module is still not loaded")


Question: The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the apimachinery repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery

# DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> cjcullen joelsmith liggitt philips </del> <ins> caesarxuchao deads2k lavalamp </ins> sttts <del> tallclair </del>


Question: The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the apimachinery repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery

# DO NOT REPORT SECURITY VULNERABILITIES DIRECTLY TO THESE NAMES, FOLLOW THE # INSTRUCTIONS AT https://kubernetes.io/security/ <del> cjcullen joelsmith liggitt philips </del> <ins> cheftako deads2k lavalamp </ins> sttts <del> tallclair </del>


Question: The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the sample-apiserver repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


Question: The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the sample-apiserver repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-e2e-aks-engine-azure-master-windows-2004 Which test(s) are failing: [] Container Runtime blackbox test when running a container with a new image should be able to pull from private registry with secret [NodeConformance] Since when has it been failing: since enabling the Windows 2004 SAC release tests Testgrid link: Reason for failure: The manifest for is missing the new windows server version 2004. This has occurred before: I looks like we need someone with appropriate permissions to update the manifests once the images are ready. Anything else we need to know: There was some work done in to automate some of this. Is there something missing? There are a few other tests that are failing in this job but should be covered by updating /cc /sig windows
The test passed 2x consecutively. We can close this now. /close
Closing this issue. <detailsIn response to : The test passed 2x consecutively. We can close this now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

"port": 10001, "labels": { "name": "redisslave" <del> } </del> <ins> }, </ins> "selector": { "name": "redisslave" }


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: Discovery should validate PreferredVersion for each APIGroup Testgrid link: Reason for failure: The test does not handle API groups which are or removed dynamically during the run of the test. Just because an API group is present in at the beginning of the test does not mean or will be present by the end of the test. This is particularly true for the dynamic/custom-resource API groups registered/unregistered during the e2e test run /sig api-machinery /assign
mitigation opened at
Saw this today in It seems to happen relatively often Likely due to the CRD created in: Should the ValidatingAdmissionPolicy test CRD have its group name template changed to match the existing test exclusion for groups ending in /cc

import ( "context" <ins> "strings" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" utilversion "k8s.io/apimachinery/pkg/util/version" "k8s.io/apiserver/pkg/endpoints/discovery"


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: Discovery should validate PreferredVersion for each APIGroup Testgrid link: Reason for failure: The test does not handle API groups which are or removed dynamically during the run of the test. Just because an API group is present in at the beginning of the test does not mean or will be present by the end of the test. This is particularly true for the dynamic/custom-resource API groups registered/unregistered during the e2e test run /sig api-machinery /assign
mitigation opened at
Saw this today in It seems to happen relatively often Likely due to the CRD created in: Should the ValidatingAdmissionPolicy test CRD have its group name template changed to match the existing test exclusion for groups ending in /cc

framework.ExpectNotEqual(len(list.Groups), 0, "Missing APIGroups") for _, group := range list.Groups { <ins> if strings.HasSuffix(group.Name, ".example.com") { // ignore known example dynamic API groups that are added/removed during the e2e test run continue } </ins> framework.Logf("Checking APIGroup: %v", group.Name) // locate APIGroup endpoint


Question:
Which jobs are flaking: ci-kubernetes-node-kubelet-serial-hugepages Which test(s) are flaking: all Testgrid link: https://k8s- Reason for failure:
/sig node
/assign /cc
GitHub didn't allow me to assign the following users: cynepco3hahue. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
testgrid alert:
/cc
I started to look into logs and the issue that one of the kubelet restarts fails on: I will try to reproduce the issue locally.
any news? Have you managed to reproduce the issue locally?
I already have a fix that is waiting for merge -

func restartKubelet() { kubeletServiceName := findRunningKubletServiceName() <del> stdout, err := exec.Command("sudo", "systemctl", "restart", kubeletServiceName).CombinedOutput() </del> <ins> // reset the kubelet service start-limit-hit stdout, err := exec.Command("sudo", "systemctl", "reset-failed", kubeletServiceName).CombinedOutput() framework.ExpectNoError(err, "Failed to reset kubelet start-limit-hit with systemctl: %v, %v", err, stdout) stdout, err = exec.Command("sudo", "systemctl", "restart", kubeletServiceName).CombinedOutput() </ins> framework.ExpectNoError(err, "Failed to restart kubelet with systemctl: %v, %v", err, stdout) }


Question:
What happened: This issue was originally reported via the Kubernetes HackerOne bug bounty by HackerOne user but we have decided that due to the high privileges required, it can be addressed publicly. A user able to create CRDs could create a malicious CRD such that listing CRs will cause enormous amounts of CPU usage on the API server. What you expected to happen: The function should not expand strings with more than two consecutive periods. Either it should collapse additional repeated periods or it should return an error. How to reproduce it (as minimally and precisely as possible): /kind bug /area apiserver /area security /committee product-security /priority important-longterm
How about we take it a step further by rejecting the jsonpath which contains when parsing it? as more than 2 periods seem to make no sense IMO
good idea. I have posted to do what you suggest.
/sig api-machinery
/cc

{"allarray", "{.Book[*].Author}", storeData, "Nigel Rees Evelyn Waugh Herman Melville", false}, {"allfields", `{range .Bicycle[*]}{ "{" }{ @.* }{ "} " }{end}`, storeData, "{red 19.95 true} {green 20.01 false} ", false}, {"recurfields", "{..Price}", storeData, "8.95 12.99 8.99 19.95 20.01", false}, <ins> {"recurdotfields", "{...Price}", storeData, "8.95 12.99 8.99 19.95 20.01", false}, {"superrecurfields", "{............................................................Price}", storeData, "", true}, </ins> {"allstructsSlice", "{.Bicycle}", storeData, `[{"Color":"red","Price":19.95,"IsNew":true},{"Color":"green","Price":20.01,"IsNew":false}]`, false}, {"allstructs", `{range .Bicycle[*]}{ @ }{ " " }{end}`, storeData,


Question:
What happened: This issue was originally reported via the Kubernetes HackerOne bug bounty by HackerOne user but we have decided that due to the high privileges required, it can be addressed publicly. A user able to create CRDs could create a malicious CRD such that listing CRs will cause enormous amounts of CPU usage on the API server. What you expected to happen: The function should not expand strings with more than two consecutive periods. Either it should collapse additional repeated periods or it should return an error. How to reproduce it (as minimally and precisely as possible): /kind bug /area apiserver /area security /committee product-security /priority important-longterm
How about we take it a step further by rejecting the jsonpath which contains when parsing it? as more than 2 periods seem to make no sense IMO
good idea. I have posted to do what you suggest.
/sig api-machinery
/cc

return p.parseInsideAction(cur) } <del> // parseRecursive scans the recursive desent operator .. </del> <ins> // parseRecursive scans the recursive descent operator .. </ins> func (p *Parser) parseRecursive(cur *ListNode) error { <ins> if lastIndex := len(cur.Nodes) - 1; lastIndex >= 0 && cur.Nodes[lastIndex].Type() == NodeRecursive { return fmt.Errorf("invalid multiple recursive descent") } </ins> p.pos += len("..") p.consumeText() cur.append(newRecursive())


Question:
What happened: This issue was originally reported via the Kubernetes HackerOne bug bounty by HackerOne user but we have decided that due to the high privileges required, it can be addressed publicly. A user able to create CRDs could create a malicious CRD such that listing CRs will cause enormous amounts of CPU usage on the API server. What you expected to happen: The function should not expand strings with more than two consecutive periods. Either it should collapse additional repeated periods or it should return an error. How to reproduce it (as minimally and precisely as possible): /kind bug /area apiserver /area security /committee product-security /priority important-longterm
How about we take it a step further by rejecting the jsonpath which contains when parsing it? as more than 2 periods seem to make no sense IMO
good idea. I have posted to do what you suggest.
/sig api-machinery
/cc

{"invalid number", "{+12.3.0}", "cannot parse number +12.3.0"}, {"unterminated array", "{[1}", "unterminated array"}, {"unterminated filter", "{[?(.price]}", "unterminated filter"}, <ins> {"invalid multiple recursive descent", "{........}", "invalid multiple recursive descent"}, </ins> } for _, test := range failParserTests { _, err := Parse(test.name, test.text)


Question:
Looks like this test is never able to succeed because service does not exist: /kind failing-test /sig instrumentation
possibly related to
/assign
/assign
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
where did we get with this?
Looks like those tests didn't work for at least half a year and nobody cared... Should we delete this e2e test as well? If so I can help with that.
Yes, let's remove them. Would appreciate the help! /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Still failing: /remove-lifecycle stale

kube-up echo "... calling validate-cluster" >&2 <del> if [[ "${EXIT_ON_WEAK_ERROR}" == "true" ]]; then validate-cluster else validate-cluster validate_result="$?" if [[ ${validate_result} != "0" ]]; then if [[ "${validate_result}" == "1" ]]; then exit 1 elif [[ "${validate_result}" == "2" ]]; then echo "...ignoring non-fatal errors in validate-cluster" >&2 else echo "Got unknown validate result: ${validate_result}" fi </del> <ins> # Override errexit (validate-cluster) && validate_result="$?" || validate_result="$?" # We have two different failure modes from validate cluster: # - 1: fatal error - cluster won't be working correctly # - 2: weak error - something went wrong, but cluster probably will be working correctly # We always exit in case 1), but if EXIT_ON_WEAK_ERROR != true, then we don't fail on 2). if [[ "${validate_result}" == "1" ]]; then exit 1 elif [[ "${validate_result}" == "2" ]]; then if [[ "${EXIT_ON_WEAK_ERROR}" == "true" ]]; then exit 1; else echo "...ignoring non-fatal errors in validate-cluster" >&2 </ins> fi fi


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For each NLB created, K8s creates an ingress rule on the node group's security group for each port exposed for each CIDR attached to the VPC to allow health checks to succeed. AWS limits the number of rules per security group to 60 by default. This quota is adjustable but multiplied by the quota for security groups per network interface cannot exceed 1000. I have a VPC with 16 CIDR blocks attached to it and a quota for rules per security group set to 80. Each NLB created adds 18 new rules (16 for Health Check, 1 for Client, and 1 for MTU) to the node group's security groups. This high number of unnecessary rules created makes it impossible to create more than 4 NLBs per node group (if each one has a single port exposed). What you expected to happen: For the health check to succeed, the NLB must be able to ping the node port for the service on the cluster's node group. So the node group's security group just needs an inbound rule for each ENI of the NLB, making the number of rules necessary on the cluster's node group directly proportional to the number of AZs the NLB is configured. How to reproduce it (as minimally and precisely as possible): Create a VPC on AWS with 8 CIDRs attached to it; Create a K8s Cluster; Create a Network Load Balancer with 6 ports exposed; If the default quota of rules per security group has not been changed, the NLB creation will fail with RulesPerSecurityGroupLimitExceeded. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/area provider/aws

return listener, nil } <ins> func (c *Cloud) getSubnetCidrs(subnetIDs []string) ([]string, error) { request := &ec2.DescribeSubnetsInput{} for _, subnetID := range subnetIDs { request.SubnetIds = append(request.SubnetIds, aws.String(subnetID)) } subnets, err := c.ec2.DescribeSubnets(request) if err != nil { return nil, fmt.Errorf("error querying Subnet for ELB: %q", err) } if len(subnets) != len(subnetIDs) { return nil, fmt.Errorf("error querying Subnet for ELB, got %d subnets for %v", len(subnets), subnetIDs) } cidrs := make([]string, 0, len(subnets)) for _, subnet := range subnets { cidrs = append(cidrs, aws.StringValue(subnet.CidrBlock)) } return cidrs, nil } </ins> // EnsureLoadBalancer implements LoadBalancer.EnsureLoadBalancer func (c *Cloud) EnsureLoadBalancer(ctx context.Context, clusterName string, apiService *v1.Service, nodes []*v1.Node) (*v1.LoadBalancerStatus, error) { annotations := apiService.Annotations


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For each NLB created, K8s creates an ingress rule on the node group's security group for each port exposed for each CIDR attached to the VPC to allow health checks to succeed. AWS limits the number of rules per security group to 60 by default. This quota is adjustable but multiplied by the quota for security groups per network interface cannot exceed 1000. I have a VPC with 16 CIDR blocks attached to it and a quota for rules per security group set to 80. Each NLB created adds 18 new rules (16 for Health Check, 1 for Client, and 1 for MTU) to the node group's security groups. This high number of unnecessary rules created makes it impossible to create more than 4 NLBs per node group (if each one has a single port exposed). What you expected to happen: For the health check to succeed, the NLB must be able to ping the node port for the service on the cluster's node group. So the node group's security group just needs an inbound rule for each ENI of the NLB, making the number of rules necessary on the cluster's node group directly proportional to the number of AZs the NLB is configured. How to reproduce it (as minimally and precisely as possible): Create a VPC on AWS with 8 CIDRs attached to it; Create a K8s Cluster; Create a Network Load Balancer with 6 ports exposed; If the default quota of rules per security group has not been changed, the NLB creation will fail with RulesPerSecurityGroupLimitExceeded. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/area provider/aws

return nil, err } <ins> subnetCidrs, err := c.getSubnetCidrs(subnetIDs) if err != nil { klog.Errorf("Error getting subnet cidrs: %q", err) return nil, err } </ins> sourceRangeCidrs := []string{} for cidr := range sourceRanges { sourceRangeCidrs = append(sourceRangeCidrs, cidr)


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For each NLB created, K8s creates an ingress rule on the node group's security group for each port exposed for each CIDR attached to the VPC to allow health checks to succeed. AWS limits the number of rules per security group to 60 by default. This quota is adjustable but multiplied by the quota for security groups per network interface cannot exceed 1000. I have a VPC with 16 CIDR blocks attached to it and a quota for rules per security group set to 80. Each NLB created adds 18 new rules (16 for Health Check, 1 for Client, and 1 for MTU) to the node group's security groups. This high number of unnecessary rules created makes it impossible to create more than 4 NLBs per node group (if each one has a single port exposed). What you expected to happen: For the health check to succeed, the NLB must be able to ping the node port for the service on the cluster's node group. So the node group's security group just needs an inbound rule for each ENI of the NLB, making the number of rules necessary on the cluster's node group directly proportional to the number of AZs the NLB is configured. How to reproduce it (as minimally and precisely as possible): Create a VPC on AWS with 8 CIDRs attached to it; Create a K8s Cluster; Create a Network Load Balancer with 6 ports exposed; If the default quota of rules per security group has not been changed, the NLB creation will fail with RulesPerSecurityGroupLimitExceeded. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/area provider/aws

sourceRangeCidrs = append(sourceRangeCidrs, "0.0.0.0/0") } <del> err = c.updateInstanceSecurityGroupsForNLB(loadBalancerName, instances, sourceRangeCidrs, v2Mappings) </del> <ins> err = c.updateInstanceSecurityGroupsForNLB(loadBalancerName, instances, subnetCidrs, sourceRangeCidrs, v2Mappings) </ins> if err != nil { klog.Warningf("Error opening ingress rules for the load balancer to the instances: %q", err) return nil, err


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For each NLB created, K8s creates an ingress rule on the node group's security group for each port exposed for each CIDR attached to the VPC to allow health checks to succeed. AWS limits the number of rules per security group to 60 by default. This quota is adjustable but multiplied by the quota for security groups per network interface cannot exceed 1000. I have a VPC with 16 CIDR blocks attached to it and a quota for rules per security group set to 80. Each NLB created adds 18 new rules (16 for Health Check, 1 for Client, and 1 for MTU) to the node group's security groups. This high number of unnecessary rules created makes it impossible to create more than 4 NLBs per node group (if each one has a single port exposed). What you expected to happen: For the health check to succeed, the NLB must be able to ping the node port for the service on the cluster's node group. So the node group's security group just needs an inbound rule for each ENI of the NLB, making the number of rules necessary on the cluster's node group directly proportional to the number of AZs the NLB is configured. How to reproduce it (as minimally and precisely as possible): Create a VPC on AWS with 8 CIDRs attached to it; Create a K8s Cluster; Create a Network Load Balancer with 6 ports exposed; If the default quota of rules per security group has not been changed, the NLB creation will fail with RulesPerSecurityGroupLimitExceeded. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/area provider/aws

} } <del> return c.updateInstanceSecurityGroupsForNLB(loadBalancerName, nil, nil, nil) </del> <ins> return c.updateInstanceSecurityGroupsForNLB(loadBalancerName, nil, nil, nil, nil) </ins> } lb, err := c.describeLoadBalancer(loadBalancerName)


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For each NLB created, K8s creates an ingress rule on the node group's security group for each port exposed for each CIDR attached to the VPC to allow health checks to succeed. AWS limits the number of rules per security group to 60 by default. This quota is adjustable but multiplied by the quota for security groups per network interface cannot exceed 1000. I have a VPC with 16 CIDR blocks attached to it and a quota for rules per security group set to 80. Each NLB created adds 18 new rules (16 for Health Check, 1 for Client, and 1 for MTU) to the node group's security groups. This high number of unnecessary rules created makes it impossible to create more than 4 NLBs per node group (if each one has a single port exposed). What you expected to happen: For the health check to succeed, the NLB must be able to ping the node port for the service on the cluster's node group. So the node group's security group just needs an inbound rule for each ENI of the NLB, making the number of rules necessary on the cluster's node group directly proportional to the number of AZs the NLB is configured. How to reproduce it (as minimally and precisely as possible): Create a VPC on AWS with 8 CIDRs attached to it; Create a K8s Cluster; Create a Network Load Balancer with 6 ports exposed; If the default quota of rules per security group has not been changed, the NLB creation will fail with RulesPerSecurityGroupLimitExceeded. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/area provider/aws

return targetGroup, nil } <del> func (c *Cloud) getVpcCidrBlocks() ([]string, error) { vpcs, err := c.ec2.DescribeVpcs(&ec2.DescribeVpcsInput{ VpcIds: []*string{aws.String(c.vpcID)}, }) if err != nil { return nil, fmt.Errorf("error querying VPC for ELB: %q", err) } if len(vpcs.Vpcs) != 1 { return nil, fmt.Errorf("error querying VPC for ELB, got %d vpcs for %s", len(vpcs.Vpcs), c.vpcID) } cidrBlocks := make([]string, 0, len(vpcs.Vpcs[0].CidrBlockAssociationSet)) for _, cidr := range vpcs.Vpcs[0].CidrBlockAssociationSet { if aws.StringValue(cidr.CidrBlockState.State) != ec2.VpcCidrBlockStateCodeAssociated { continue } cidrBlocks = append(cidrBlocks, aws.StringValue(cidr.CidrBlock)) } return cidrBlocks, nil } </del> // updateInstanceSecurityGroupsForNLB will adjust securityGroup's settings to allow inbound traffic into instances from clientCIDRs and portMappings. // TIP: if either instances or clientCIDRs or portMappings are nil, then the securityGroup rules for lbName are cleared. <del> func (c *Cloud) updateInstanceSecurityGroupsForNLB(lbName string, instances map[InstanceID]*ec2.Instance, clientCIDRs []string, portMappings []nlbPortMapping) error { </del> <ins> func (c *Cloud) updateInstanceSecurityGroupsForNLB(lbName string, instances map[InstanceID]*ec2.Instance, subnetCIDRs []string, clientCIDRs []string, portMappings []nlbPortMapping) error { </ins> if c.cfg.Global.DisableSecurityGroupIngress { return nil }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For each NLB created, K8s creates an ingress rule on the node group's security group for each port exposed for each CIDR attached to the VPC to allow health checks to succeed. AWS limits the number of rules per security group to 60 by default. This quota is adjustable but multiplied by the quota for security groups per network interface cannot exceed 1000. I have a VPC with 16 CIDR blocks attached to it and a quota for rules per security group set to 80. Each NLB created adds 18 new rules (16 for Health Check, 1 for Client, and 1 for MTU) to the node group's security groups. This high number of unnecessary rules created makes it impossible to create more than 4 NLBs per node group (if each one has a single port exposed). What you expected to happen: For the health check to succeed, the NLB must be able to ping the node port for the service on the cluster's node group. So the node group's security group just needs an inbound rule for each ENI of the NLB, making the number of rules necessary on the cluster's node group directly proportional to the number of AZs the NLB is configured. How to reproduce it (as minimally and precisely as possible): Create a VPC on AWS with 8 CIDRs attached to it; Create a K8s Cluster; Create a Network Load Balancer with 6 ports exposed; If the default quota of rules per security group has not been changed, the NLB creation will fail with RulesPerSecurityGroupLimitExceeded. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/area provider/aws

} clientRuleAnnotation := fmt.Sprintf("%s=%s", NLBClientRuleDescription, lbName) healthRuleAnnotation := fmt.Sprintf("%s=%s", NLBHealthCheckRuleDescription, lbName) <del> vpcCIDRs, err := c.getVpcCidrBlocks() if err != nil { return err } </del> for sgID, sg := range clusterSGs { sgPerms := NewIPPermissionSet(sg.IpPermissions...).Ungroup() if desiredSGIDs.Has(sgID) { <del> if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, healthRuleAnnotation, "tcp", healthCheckPorts, vpcCIDRs); err != nil { </del> <ins> if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, healthRuleAnnotation, "tcp", healthCheckPorts, subnetCIDRs); err != nil { </ins> return err } if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, clientRuleAnnotation, clientProtocol, clientPorts, clientCIDRs); err != nil {