Question: // +groupName=wardle.k8s.io // Package api is the internal version of the API. <del> package wardle </del> <ins> package wardle // import "k8s.io/sample-apiserver/pkg/apis/wardle" </ins>

/lifecycle rotten
I guess this is still not done, was supposed to partially fix this issue. /reopen
Reopened this issue. <detailsIn response to : I guess this is still not done, was supposed to partially fix this issue. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /area code-organization


Question: t.Errorf("No PDB was deleted from the cache: %v", err) } } <ins> // TestSchedulerInformers tests that scheduler receives informer events and updates its cache when // pods are scheduled by other schedulers. func TestSchedulerInformers(t *testing.T) { // Initialize scheduler. context := initTest(t, "scheduler-informer") defer cleanupTest(t, context) cs := context.clientSet defaultPodRes := &v1.ResourceRequirements{Requests: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(200, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(200, resource.BinarySI)}, } defaultNodeRes := &v1.ResourceList{ v1.ResourcePods:   *resource.NewQuantity(32, resource.DecimalSI), v1.ResourceCPU:    *resource.NewMilliQuantity(500, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(500, resource.BinarySI), } type nodeConfig struct { name string res  *v1.ResourceList } tests := []struct { description         string nodes               []*nodeConfig existingPods        []*v1.Pod pod                 *v1.Pod preemptedPodIndexes map[int]struct{} }{ { description: "Pod cannot be scheduled when node is occupied by pods scheduled by other schedulers", nodes:       []*nodeConfig{{name: "node-1", res: defaultNodeRes}}, existingPods: []*v1.Pod{ initPausePod(context.clientSet, &pausePodConfig{ Name:          "pod1", Namespace:     context.ns.Name, Resources:     defaultPodRes, Labels:        map[string]string{"foo": "bar"}, NodeName:      "node-1", SchedulerName: "foo-scheduler", }), initPausePod(context.clientSet, &pausePodConfig{ Name:          "pod2", Namespace:     context.ns.Name, Resources:     defaultPodRes, Labels:        map[string]string{"foo": "bar"}, NodeName:      "node-1", SchedulerName: "bar-scheduler", }), }, pod: initPausePod(cs, &pausePodConfig{ Name:      "unschedulable-pod", Namespace: context.ns.Name, Resources: defaultPodRes, }), preemptedPodIndexes: map[int]struct{}{2: {}}, }, } for _, test := range tests { for _, nodeConf := range test.nodes { _, err := createNode(cs, nodeConf.name, nodeConf.res) if err != nil { t.Fatalf("Error creating node %v: %v", nodeConf.name, err) } } pods := make([]*v1.Pod, len(test.existingPods)) var err error // Create and run existingPods. for i, p := range test.existingPods { if pods[i], err = runPausePod(cs, p); err != nil { t.Fatalf("Test [%v]: Error running pause pod: %v", test.description, err) } } // Create the new "pod". unschedulable, err := createPausePod(cs, test.pod) if err != nil { t.Errorf("Error while creating new pod: %v", err) } if err := waitForPodUnschedulable(cs, unschedulable); err != nil { t.Errorf("Pod %v got scheduled: %v", unschedulable.Name, err) } // Cleanup pods = append(pods, unschedulable) cleanupPods(cs, t, pods) cs.PolicyV1beta1().PodDisruptionBudgets(context.ns.Name).DeleteCollection(nil, metav1.ListOptions{}) cs.CoreV1().Nodes().DeleteCollection(nil, metav1.ListOptions{}) } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Scheduler's pod informer has a selector that . This is incorrect. Scheduler must watch all the pods of the cluster, including those scheduled by other schedulers in order to update its cache of the state of the cluster. Otherwise, it will miss important events and the amount of available resources, pods on nodes, ... which are needed for scheduler to work correctly, will not be updated properly. /sig scheduling /assign
cc
For updating it cache, shouldn't the default scheduler rely on node info/status? Also its not very clear, what information the default scheduler needs from other pods (not to be scheduled by it) that it makes decision on? Just trying to understand the issue more clearly.
Scheduler watches informer events to update its cache. Node status is not enough to update all the information that scheduler caches. Scheduler must know what pods are placed on what nodes. For example, in order to check inter-pod affinity/anti-affinity it needs to know the placement of all the pods, including those scheduled by other schedulers.
But this information is present here: So it seems that your PR will now make sure to get the scheduled pods' information directly from the pod informer.
So when I check the history, I see it was here:
have we seen any specific issues due to this change (like between default scheduler and DS controller)?
Yes, but scheduler relies on its pod informers to keep this location up to date. That's exactly why it is important for the scheduler to receive updates for all the pods in the cluster. Yes. We missed the problem at the time. This is problematic only in clusters that use multiple schedulers. I initially thought that DS controller could cause issues as well, but DS controller does not set , so it must not cause an issue. We have seen instances of scheduler's missing certain events in cluster and as a result scheduler's cache getting stale, but I doubt if it is related to this particular issue. This only affects clusters with more than one scheduler. If you run the test that I have written in with the head, it will fail. That tests simulate a multi-scheduler scenario.


Question: } }) <del> ginkgo.It("A set of valid responses are returned for both pod and service ProxyWithPath", func() { </del> <ins> /* Release: v1.21 Testname: Proxy, validate ProxyWithPath responses Description: Attempt to create a pod and a service. A set of pod and service endpoints MUST be accessed via ProxyWithPath using a list of http methods. A valid response MUST be returned for each endpoint. */ framework.ConformanceIt("A set of valid responses are returned for both pod and service ProxyWithPath", func() { </ins> ns := f.Namespace.Name msg := "foo"

[x] two weeks soak start date : testgrid-link [x] two weeks soak end date : [ ] test promotion pr : kubernetes/kubernetes#? According to this APIsnoop query, there are still some remaining ProxyWithPath endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a pod with a static label using the agnhost image Confirm that the pod is in the running phase Iterate through a list of http methods and check the response from the porter app Confirm that each response code is 200 OK If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance Where/How does PATCH/HEAD/OPTIONS requests for the following endpoints line up with the API reference? connectCoreV1PatchNamespacedPodProxyWithPath connectCoreV1HeadNamespacedPodProxyWithPath connectCoreV1OptionsNamespacedPodProxyWithPath Pod v1 core: Proxy Operations the following endpoints are listed as valid endpoints; connectCoreV1HeadNamespacedPodProxyWithPath connectCoreV1OptionsNamespacedPodProxyWithPath These requests are getting logged as a verb. Line : api/openapi-spec Line : api/openapi-spec Looking at there doesn't look to be a mapping for the above methods. According to this APIsnoop query, there are still some remaining ProxyWithPath endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a pod with a static label using the agnhost image Confirm that the pod is in the running phase Create a service and link it to the pod Iterate through a list of http methods and check the response from the porter app Confirm that each response code is 200 OK If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance.


Question: } }) <del> ginkgo.It("A set of valid responses are returned for both pod and service ProxyWithPath", func() { </del> <ins> /* Release: v1.21 Testname: Proxy, validate ProxyWithPath responses Description: Attempt to create a pod and a service. A set of pod and service endpoints MUST be accessed via ProxyWithPath using a list of http methods. A valid response MUST be returned for each endpoint. */ framework.ConformanceIt("A set of valid responses are returned for both pod and service ProxyWithPath", func() { </ins> ns := f.Namespace.Name msg := "foo"

The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/re-open
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ``` /* Testname: Kubelet-OutputToLogs <ins> Release: v1.9 </ins> Description: By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs. */

What happened: The Conformance Document currently has a static header that is displayed from a file. The release information in the header should reflect proper release number so that the document reflect the current state. What you expected to happen: Should show proper current release information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
in reaction to comments is still hardcoded to mention v1.9 at the top, could we parameterize this?
/sig testing /area conformance /release-note-none
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
/priority backlog
This seems to have been addressed at Should we just close this?
No, specifically did not address this issue, this should stay open. See


Question: // conditions is the set of conditions required for this autoscaler to scale its target, // and indicates whether or not those conditions are met. <ins> // +optional </ins> repeated HorizontalPodAutoscalerCondition conditions = 6; }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Serializing a v2beta2 HorizontalPodAutoscaler with (and ) produces yaml that cannot be applied due to a ValidationError on the field which is under the optional field . A valid HPA serialized with should be able to be applied to the cluster with no validation errors, without filling out the optional field. Other valid objects (such as Namespaces and Deployments) can be applied directly, without filling out their optional fields. Run this to generate the serialized yaml (same outcome with both and functions) Running the go code should yield this yaml: Writing the above yaml to a file and applying it (with ) will result in: Kubernetes version (use ): Server: v1.20.2, Client: v1.20.4 go version: go1.16 darwin/amd64 has: cc cc
/sig api-machinery /sig autoscaling /sig cli
/assign /triage accepted
marking conditions as +optional in the API type would resolve the client-side apply error
It would remove the validation error, but I'd be concerned about any side effects or unwanted outcomes when moving that field from required to optional. A somewhat analogous field is required. But if you anticipate this won't change much else, that will work.
Client-side validation is non-authoritative, and problematic for the reasons described in this issue ... the client is disallowing a request the server would actually have no problems with. No field the server would permit to be empty (e.g. succeeds with ) should be marked as required for this reason. fwiw, openapi schema generation marks fields tagged as as optional:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: // conditions is the set of conditions required for this autoscaler to scale its target, // and indicates whether or not those conditions are met. <ins> // +optional </ins> repeated HorizontalPodAutoscalerCondition conditions = 6; }

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten


Question: } var cgroupRoots []string <del> cgroupRoots = append(cgroupRoots, cm.NodeAllocatableRoot(s.CgroupRoot, s.CgroupDriver)) </del> <ins> nodeAllocatableRoot := cm.NodeAllocatableRoot(s.CgroupRoot, s.CgroupsPerQOS, s.CgroupDriver) cgroupRoots = append(cgroupRoots, nodeAllocatableRoot) </ins> kubeletCgroup, err := cm.GetKubeletContainer(s.KubeletCgroups) if err != nil { klog.Warningf("failed to get the kubelet's cgroup: %v.  Kubelet system container metrics may be missing.", err)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: k8s 1.17.2 config kubelet with those parameters cgroupDriver: cgroupfs cgroupRoot: /docker cgroupsPerQOS: false kubelet log always prints those E0303 17:25:03. ] Failed to get system container stats for "/docker": failed to get cgroup stats for "/docker": failed to get container info for "/docker": unknown container "/docker" E0303 17:25:03. ] eviction manager: failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics What you expected to happen: kubelet does not print those error information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node


Question: Key:    algorithm.TaintNodeNetworkUnavailable, Effect: v1.TaintEffectNoSchedule, } <ins> notReadyTaint := &v1.Taint{ Key:    algorithm.TaintNodeNotReady, Effect: v1.TaintEffectNoSchedule, } </ins> tests := []struct { Name           string

Is this a BUG REPORT or FEATURE REQUEST?: What happened: admission control plugin in apiserver. a deployment with 1 replica. The pod is successfully launched. By default the admission control plugin will add following toleration in the pod. the node where the above pod is running. of image locality policy and above toleration in the pod, scheduler still choose the node to launch the pod. The pod is in state and failed to create. What you expected to happen: The pod should be scheduled to healthy node and created successfully. How to reproduce it (as minimally and precisely as possible): See above steps. Anything else we need to know?: Environment: Kubernetes version (use ): v1.10 Cloud provider or hardware configuration: openstack OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-121-generic Install tools: N/A Others:
/sig scheduler
/sig scheduling /sig node
I think that we should avoid scheduling pods to nodes which is not ready even it can tolerate and , comments
As the pod already claimed to tolerate NotReady, this is an expected behavior. P.S. If the eviction dose not happen after 5m, that should be the bug. Please reopen the issue and modify the description. This is what tolerations is designed for :) And there are many use cases for pod to be scheduled to NotReady nodes (mostly during cluster deployment).
I think maybe you can configure effect to avoid scheduling to those nodes.
scheduler still choose the NotReady node to launch the pod For running, the toleration is used for graceful termination by ; if is enabled, it's better to also taint node with for .
Wondering how the pod can start to run if the Node is not ready at deployment time? Can you specify a use case here?
It won't need to start, system pod pending will be expected :) The point is, freedom for users to decide what tolerations they want matters here. I don't think it's reasonable to combine with in this case.
Perhaps, add to default toleration of admission control plugin could eliminate the confusion.
We should align the behavior when enabled or disabled : if is disabled, will not schedule pod to not-ready node. if is enabled, we should also taint node with .


Question: Key:    algorithm.TaintNodeNetworkUnavailable, Effect: v1.TaintEffectNoSchedule, } <ins> notReadyTaint := &v1.Taint{ Key:    algorithm.TaintNodeNotReady, Effect: v1.TaintEffectNoSchedule, } </ins> tests := []struct { Name           string

with timeout is used for graceful termination of 'running pods' for feature. We should not re-schedule pod to not-ready unless tolerating it explicitly.
reopen this for more discussion :)
Let me extracted the mixed problems in this issue :-) First one: These two default tolerations attached to the pod by means: if the node is tainted as or , wait for 300s before the pod is evicted from the node. While according to the issue description, is working as expected (pods are not evicted immediately). This part should be closed. Second one: The discussion raised is actually about taint during eviction: if TaintNodesByCondition is enabled, we should also taint node with NotReady:NoSchedule. My thoughts is: it's reasonable to taint node with as well for consistency consideration (but I'm hesitate if only for eliminating confusion as pods are expected to pending on NotReady nodes) I think it should be clearer now.
I open a PR If TaintNodesByCondition is enable, taint node with NotReady:NoSchedule and Unreachable:NoSchedule.
, we did not catch 1.11 release, so this issue will be handled in 1.12 :)
in 1.9.7enable the TaintNodesByCondition and taint nodee.g. node01 with NotReady:NoSchedulethen pod also can not be scheduled to node01 even when node01 is healthy. What you expected to happen: The pod should be scheduled to node01 when it is healthy node and not be scheduled to node01 when it is not ready. Thanks
How did you do that? by kubectl? The fix is not ready until 1.12.
YesI tried "kubectl taint nodes node01 ",and also tried "kubectl taint nodes node01 "
this feature did not handle taint manually
Ok, thanks a lot! So I can not solve the problem manually now and what I should do is just waiting for the version 1.12, right?


Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "strings" "time" <ins> v1 "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/wait" clientset "k8s.io/client-go/kubernetes" <ins> "k8s.io/kubernetes/test/e2e/framework" "github.com/onsi/ginkgo" "k8s.io/apimachinery/pkg/types" </ins> ) <ins> // Action is a function to be performed by the system. type Action func() error var _ = ginkgo.Describe("[sig-api-machinery] Events", func() { f := framework.NewDefaultFramework("events") ginkgo.It("should ensure that an event can be fetched, patched, deleted, and listed", func() { eventTestName := "event-test" ginkgo.By("creating a test event") // create a test event in test namespace _, err := f.ClientSet.CoreV1().Events(f.Namespace.Name).Create(context.TODO(), &v1.Event{ ObjectMeta: metav1.ObjectMeta{ Name: eventTestName, Labels: map[string]string{ "testevent-constant": "true", }, }, Message: "This is a test event", Reason:  "Test", Type:    "Normal", Count:   1, InvolvedObject: v1.ObjectReference{ Namespace: f.Namespace.Name, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create test event") ginkgo.By("listing all events in all namespaces") // get a list of Events in all namespaces to ensure endpoint coverage eventsList, err := f.ClientSet.CoreV1().Events("").List(context.TODO(), metav1.ListOptions{ LabelSelector: "testevent-constant=true", }) framework.ExpectNoError(err, "failed list all events") foundCreatedEvent := false var eventCreatedName string for _, val := range eventsList.Items { if val.ObjectMeta.Name == eventTestName && val.ObjectMeta.Namespace == f.Namespace.Name { foundCreatedEvent = true eventCreatedName = val.ObjectMeta.Name break } } framework.ExpectEqual(foundCreatedEvent, true, "unable to find the test event") ginkgo.By("patching the test event") // patch the event's message eventPatchMessage := "This is a test event - patched" eventPatch, err := json.Marshal(map[string]interface{}{ "message": eventPatchMessage, }) framework.ExpectNoError(err, "failed to marshal the patch JSON payload") _, err = f.ClientSet.CoreV1().Events(f.Namespace.Name).Patch(context.TODO(), eventTestName, types.StrategicMergePatchType, []byte(eventPatch), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch the test event") ginkgo.By("fetching the test event") // get event by name event, err := f.ClientSet.CoreV1().Events(f.Namespace.Name).Get(context.TODO(), eventCreatedName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to fetch the test event") framework.ExpectEqual(event.Message, eventPatchMessage, "test event message does not match patch message") ginkgo.By("deleting the test event") // delete original event err = f.ClientSet.CoreV1().Events(f.Namespace.Name).Delete(context.TODO(), eventCreatedName, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete the test event") ginkgo.By("listing all events in all namespaces") // get a list of Events list namespace eventsList, err = f.ClientSet.CoreV1().Events("").List(context.TODO(), metav1.ListOptions{ LabelSelector: "testevent-constant=true", }) framework.ExpectNoError(err, "fail to list all events") foundCreatedEvent = false for _, val := range eventsList.Items { if val.ObjectMeta.Name == eventTestName && val.ObjectMeta.Namespace == f.Namespace.Name { foundCreatedEvent = true break } } framework.ExpectEqual(foundCreatedEvent, false, "should not have found test event after deletion") }) }) </ins> // WaitTimeoutForEvent waits the given timeout duration for an event to occur. func WaitTimeoutForEvent(c clientset.Interface, namespace, eventSelector, msg string, timeout time.Duration) error { interval := 2 * time.Second

This issue proposes the writing of a new test for the API endponts , , , , , and with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to these query results from APIsnoop, we are lacking in tests for the following endpoints related to Events We chose the following 5 endpoints for this mock test: - - - Test outline: Event in namespace default with custom fields all events in all namespaces find event created in in response of the content of the event created in events by name in namespace default ensure that the event updated from event by name all events in namespace ensure event in namespace default was deleted This mock test uses client-go's standard functions for talking to the Event area of the API. Discover useragents Display endpoints hit by the test Show the change in coverage after the mock test If a test with these calls gets merged in, conformance will go up by 5 endpoints. /sig architecture /area conformance
lgtm according to the SIG-Arch conformance office hours 17th December 2019.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen can you put the checkboxes at the top for your apisnoop org pr, the test, and link to the promotion PR? I think we are ready.
/open
Reopened this issue. <detailsIn response to : /reopen can you put the checkboxes at the top for your apisnoop org pr, the test, and link to the promotion PR? I think we are ready. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/retitle Promote CoreV1Event test - +5 coverage
/assign
This test was written before the tickets were stored in APISnoop branches. Unsure what to do with it in regards to the APISnoop PR


Question: ginkgo.By("Make client connect") options := nat.CloseWaitClientOptions{ <del> RemoteAddr: fmt.Sprintf("%v:%v", serverNodeInfo.nodeIP, testDaemonTCPPort), TimeoutSeconds:        timeoutSeconds, PostFinTimeoutSeconds: 0, </del> <ins> RemoteAddr:            net.JoinHostPort(serverNodeInfo.nodeIP, strconv.Itoa(testDaemonTCPPort)), TimeoutSeconds:        deadlineTimeoutSeconds, PostFinTimeoutSeconds: postFinTimeoutSeconds, </ins> LeakConnection:        true, }

Which jobs are failing: kops-aws and many other jobs Which test(s) are failing: This isn't the actual test, but the text we're consistently seeing is Since when has it been failing: Testgrid links: (ref: ) Reason for failure: TBD Anything else we need to know: Very rarely tests pass when one of the tests passes: Seems related to , that was merged about the same time:
/sig cluster-lifecycle /sig aws
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cluster-lifecycle /sig aws Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent this is merge blocking
/assign /cc /cc /cc
The PR we think we want to revert
is also failing, I don't think this is aws-specific
/cc
/assign
it failed with kind too :/ with the same message
Opened to revert
/unassign /sig network /sig testing /remove-sig cluster-lifecycle
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This may be the actual cause PR to address But we're going with revert first to be sure we've identified the offending PR
I think I've identified the root cause
Revert brought us back to green: Awaiting root cause fix
/remove-triage unresolved
/reopen merged but we're now seeing multiple tests fail since it merged Looking at it went red between these commits:
Reopened this issue. <detailsIn response to : /reopen merged but we're now seeing multiple tests fail since it merged Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is an attempt to fix
reverting in
is the failure cluster I suspect will go down once the revert goes in
/close Failure cluster has gone down, calling this done <img width="998" alt="Screen Shot 2020-01-10 at 9 56 17 AM" src="https://user-"
Closing this issue.


Question: ginkgo.By("Make client connect") options := nat.CloseWaitClientOptions{ <del> RemoteAddr: fmt.Sprintf("%v:%v", serverNodeInfo.nodeIP, testDaemonTCPPort), TimeoutSeconds:        timeoutSeconds, PostFinTimeoutSeconds: 0, </del> <ins> RemoteAddr:            net.JoinHostPort(serverNodeInfo.nodeIP, strconv.Itoa(testDaemonTCPPort)), TimeoutSeconds:        deadlineTimeoutSeconds, PostFinTimeoutSeconds: postFinTimeoutSeconds, </ins> LeakConnection:        true, }

<detailsIn response to : /close Failure cluster has gone down, calling this done <img width="998" alt="Screen Shot 2020-01-10 at 9 56 17 AM" src="https://user-"</details


Question: } oldStatus, found := oldStatuses[container.Name] if found { <del> if isInitContainer && oldStatus.State.Terminated != nil { </del> <ins> if oldStatus.State.Terminated != nil { </ins> // Do not update status on terminated init containers as // they be removed at any time. status = &oldStatus

After , we observed pods moving from phase to . Presents on master and release-1.8 as it was picked in test script has an command for basically doing a CLI watch or you can look in the kubelet log to see the phase transition from to Thanks to for helping isolate this :+1: /sig node
/assign
taking a look. although at first glance it is strange this didnt happen with evicted pods prior to this.
Thanks. FYI, I did confirm that reverting the mentioned PR on master corrected the issue.
This showed up breaking a job controller that depended on the pod not transitioning back to pending. It's possible this can cause applications to fail / run multiple times in potentially dangerous ways. I'm not sure we should allow a status update that moves back to a non terminal state via validation. It's not a legal transition as we have defined phase. We should separately put a guard in place that will report a clear error to clients.
after thinking on it, I think that the situation with eviction is a little different in that the containers are terminated but they are not immediately cleaned up. It is the proximity between pod termination time and container removal time that is uncovering the issue, I think. AFAICT, some path is attempting to get the container status after the container has been removed. When this fails, something in the status manager gets messed up and it under the impression that the pod is new and the status needs to be reinitialized.
Here are the relevant logs showing the bug: Still trying to figure out what caused the status update.
Ok, I figured out why this is happening. When we compute a container's status, we assume that . So deleting a pod's containers before the pod object is removed causes the state to return to pending.
Maybe we should expand this that I to prevent init container status reset to include all containers with an oldStatus
Would that prevent updating a container that crashed and restarted? Edit: nvm, I think that might work
I a comment at
i also commented here that it would be good if syncPod in status manager protected against an invalid transfer of phase... see: can we do that in this same PR?
My hope is to cherrypick this to 1.8. Is that check something we want to cherrypick as well?


Question: } oldStatus, found := oldStatuses[container.Name] if found { <del> if isInitContainer && oldStatus.State.Terminated != nil { </del> <ins> if oldStatus.State.Terminated != nil { </ins> // Do not update status on terminated init containers as // they be removed at any time. status = &oldStatus

i dont see why not... absent that its hard to know this fix actually fixes it... or that we dont regress.
do we want to do it in or ? is on the other side of a channel where a stack trace on the caller of could be more informative.
both? i just know that we get the latest state as told to api server on the other end of the channel...


Question: Containers: []v1.Container{ { Name:            "e2e-net-exec", <del> Image:           imageutils.GetE2EImage(imageutils.DebianIptables), </del> <ins> Image:           imageutils.GetE2EImage(imageutils.DistrolessIptables), </ins> ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"sleep", "600"}, SecurityContext: &v1.SecurityContext{

Istio folks have a iptables image here that is quite small In the past, we did talk about this, for example wrote it well what we could try: We could even do for running the hack script we have for switching the symlinks when vs : kube-proxy image is definitely one of the main causes of the cost we incur in our GCR repository, so we should definitely give this a shot again. /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is a sig/network-only issue. A gives; The looks like this (narroved): If I don't misinterpret this, K8s uses a unified server-image build and it should not be different for . The size of the binary for amd64 in is; It is stripped and can't easily be "made smaller" I guess.
We (Istio) would definitely be interested in sharing ownership (hopefully by upsteaming to distroless mainline) in the image we have if there is interest. Some other projects have expressed similar needs as well
!
kube-proxy make up 9% of current image data traffic
I threw together this little repo (based on our next-gen distroless infrastructure): It pushed which on x64 is roughly another 5x smaller (musl vs. glibc): If folks are interested in this, I'd be happy to create a repo under which is where we've been standing up new images, or you can create your own from pretty easily.
debian based please!! we have KEPs and stuff written that lay down that direction:
For a possible approach + above in that issue and for "distroless" debian based images with arbitrary C/C++ binaries + deps. Note that we need things besides just the distro binaries (see comment linked). and is pretty straightforward for one way to handle the debian binaries part.
I don't even know how we build the kube-proxy image. Where does that happen and form which dockerfile?


Question: Containers: []v1.Container{ { Name:            "e2e-net-exec", <del> Image:           imageutils.GetE2EImage(imageutils.DebianIptables), </del> <ins> Image:           imageutils.GetE2EImage(imageutils.DistrolessIptables), </ins> ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"sleep", "600"}, SecurityContext: &v1.SecurityContext{

Note we need iptables, iptables-save and iptables-restore but also ipvsadm, shell, conntrack, ipset, and maybe other tools
It used to be in kubernetes/kubernetes under build/images, it's now in (Dockerfile etc there)
That's the base - where is the logic to turn that into a kube-proxy image?
Turning that image into a kube-proxy image is just adding the kube-proxy binary with a dockerfile and entirely too much bash which ultimately just does: (except api-server, where we do a setcap in )
/triage accepted Will take a look between this and next week
It doesn't even need bash; the version of the script in intentionally avoids bash-specific features (and gets tested against Alpine where /bin/sh is busybox (which I guess is "ash"?)). And the version in kubernetes/release appears to be just a stripped-down version of that script. (But actually, you should update the script to be based on the latest version anyway, to get the new logic in , in preparation for .)
Probably per one of the things we don't want to do right now is jump into a musl based image like Alpine, right? So I would take the case that we want to use distroless, and copy all the binaries we need to inside that container (conntrack, ipvs, iptables-*, etc etc). per iptables-wrapper, you said it's not bash specific so maybe we can use distroless + the smaller available shell? I will check some approaches here before jumping into changing the release/dockerfiles/whatever scripts, and check if I can run some comformance tests in a distroless k/proxy. Maybe starting with a good binary list is a way to cross check the binaries + libraries required for this distroless container. I did a list with the installed packages per kproxy needs, and some ldd on it: Will start working on something to assemble an image just with the binaries and required libraries, and check size + libraries
I agree no musl/alpine On Sun, Apr 17, 2022, 11:38 AM Ricardo Katz
If you identify the list of binaries, has logic to handle ldd for a binary (e.g. ignore vdso) + keep the copyright and package version info via dpkg.


Question: Containers: []v1.Container{ { Name:            "e2e-net-exec", <del> Image:           imageutils.GetE2EImage(imageutils.DebianIptables), </del> <ins> Image:           imageutils.GetE2EImage(imageutils.DistrolessIptables), </ins> ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"sleep", "600"}, SecurityContext: &v1.SecurityContext{

We probably need a few more not in the list above for the other proxy modes?
Thanks this is gonna be very helpful! I will work on it using the script you provided during the week!
/cc
From script and some changes: There's a pretty good reduction of size (50%). But this is without kproxy binary, that is almost 40Mb. I will open a Draft PR in k/release so we can discuss over there the approach. BTW, anyone knows why kube-proxy binary is that big? Never digged into that, but curious about it.
https://dr- !
(updated with conntrack-tools) Why? The binary is statically linked so it does not need or use , it only needs some binaries to be executable and it doesn't matter if they are built with musl. They just have to work. I have sucessfully built and executed my own tests, both with proxy-mode and , using an image built with this : The image size on cluster including the binary (~40MB) is; (I use a private local registry) Compare to;
what about the different iptables-nft vs iptables-legacy binaries?
I have assumed that takes care of that (executed by the Dockerfile).
Re size: Are DWARF2 symbols stripped? Ideally k8s should produce artifacts without DWARF2 symbols and distribute them separatelly. Or produce the optional -debug artifacts. This can cut GCR/GCS costs, but introduce some debug/profiling complexity. The Go symbol table should be always included. Related
IIRC wrapper in prod needs both binaries existing in the container, and will use this in a way to check if the host machine is using iptables-legacy or iptables-nft.
Both exist in Alpine; The first example in is Alpine. can you please confirm that it works with Alpine?
I have altered my testenv so I can switch between legacy and nft and verified that the tests work with both and that shows the same in the container and the K8s node, e.g;
Side remark; to my suprise is not needed in the kube-proxy image
Tested the with K8s e2e;
There are / we're legal reasons to avoid Alpine. Have those been resolved? I haven't heard anything.


Question: Containers: []v1.Container{ { Name:            "e2e-net-exec", <del> Image:           imageutils.GetE2EImage(imageutils.DebianIptables), </del> <ins> Image:           imageutils.GetE2EImage(imageutils.DistrolessIptables), </ins> ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"sleep", "600"}, SecurityContext: &v1.SecurityContext{

On Tue, Apr 19, 2022, 11:31 AM Lars Ekman
i don't think so. it's status quo on Alpine which is to avoid it totally.
Can I read about those somewhere? Are the legal reasons about Alpine? Or about musl?
Not my personal decision, and I'm not a lawyer let alone an OSS license expert but ... sharing what I've heard on the subject: IIRC for example Alpine does not make available the source code (AFAICT Alpine publishes source code for the packaging scripts, but not the package sources), or even the LICENSE of binary packages, which is problematic for compliance with licenses ... (Various common license require including the license text, and / or making source code available). Normal alpine images don't appear to install these doc packages: So those images don't even ship with license / copyright info files for packages which is ... not a great start. I'm not sure where a good public reference is for this, Google OSPO has some relevant docs internally about following license requirements when producing container images. So far the best public doc on the topic I've found is
Thanks!
TL:DR;
I wasn't suggesting switching to Alpine, I was just saying that I unit test the iptables-wrappers scripts against Alpine, which doesn't use bash, and so I know the scripts don't require bash. Yeah, I don't know the details of how distroless works, but if we're talking about debian packages then it looks like is 1/10 the size of (212k vs 2285k)
In Alpine (and Debian, and probably most distros other than Fedora), the "iptables" package includes both the legacy and nft binaries, so that's the only dependency.
Made a comment on bandwidth that may be better in this issue rahter than the PR;
Want to give a quick update on this (EDIT: I wrote this was going to be quick update, but ended up not being! sorry!): The image generation is working "fine", see: This process generates a temporary that can be used later with and/or changed in Kubernetes code to trigger the tests (see ) Due to the way iptables-wrapper-installer works, it calls iptables-legacy and iptables-nft to check if they are working fine.


Question: Containers: []v1.Container{ { Name:            "e2e-net-exec", <del> Image:           imageutils.GetE2EImage(imageutils.DebianIptables), </del> <ins> Image:           imageutils.GetE2EImage(imageutils.DistrolessIptables), </ins> ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"sleep", "600"}, SecurityContext: &v1.SecurityContext{

We need to skip it (via --skip-integrity-check) in arch not amd64, as those are really specific kernel features and when called from arm64, as an example the command ends in an error. We need /bin/sleep to be present in distroless as well, due specifically to this image being used in this test: cloud-build timeout needs to be increased, it takes a long time for arm/s390/etc to build this The image can be optimized a bit more, removing some steps like linking the iptables binaries (as pointed by and removing some binaries like wc (as pointed by ) but right now I didn't wanted to change the image build AND the iptables-wrapper script. My take here would be: let's see now with sig-release if the build process is fine, get this image promoted and used before v1.25 All the tests but verify ran fine (verify due to dependency check, etc etc) - Here are the main tests with the distroless image: kind - kind ipv6 - gce ipvs - performance 100 nodes - fyi. I can bring this to next sig-net meeting, wanted to check: from sig/release perspective, is the PR fine? I will make some more changes here, but wanted even to check if this should be called debian-iptables:distroless or something like from sig/net if we want to move forward migrating the iptables-wrapper to shell script this release or not. I tend of agree with Rob that maybe we are getting too many changes in one release, and we can plan this for v1.26 (the PR is already there, I need to check Dan's comments only) Thanks
/assign
/reopen let's close it once we switch to the new image
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, Could you suggest if it's ok to use already? That would be super cool to use the image as a base for NodeLocalDNS (as it uses iptables). I already did manual smoke test in cluster - all is working correctly. Thanks!


Question: Containers: []v1.Container{ { Name:            "e2e-net-exec", <del> Image:           imageutils.GetE2EImage(imageutils.DebianIptables), </del> <ins> Image:           imageutils.GetE2EImage(imageutils.DistrolessIptables), </ins> ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"sleep", "600"}, SecurityContext: &v1.SecurityContext{

Let's introduce it gradually, I suggest to wait until the image is promoted for kube-proxy
I've lost track of this, what do we need to do to promote it for kube-proxy?
I think Ricardo was on it But if someone is busy and others have time, let's sync next monday on slack and organize the work, so someone can get this to the finish line, there is no need to overload one person if there are other wanting to help
I didn’t saw there was a comment there. I can fix that, and wanted to plan with you folks next steps. I agree with Antonio and don’t want to block this on my availability :) I will check suggestion tomorrow and as we get this merged I’m more than happy to handle to someone else! On Sat, 16 Jul 2022 at 07:21 Antonio Ojea
I've opened a new PR to do the proper promotion. As soon as this is promoted I think we can start testing and migrating to distroless :) From here I think I can handle to someone else as I wont have much time and attention to this specific effort and I know this is important :) I'm still willing to migrate iptables-wrapper to go code (we started it in kubernetes-sigs/iptables-wrapper) so I will probably finish fixing Dan's reviews and we can discuss the distroless v0.2.0 being based entirely in Go :)
we should probably nudge a few of the subprojects to migrate, we just migrated KIND but I'm sure we have at least a few more out there like ip-masq-agent
I'm migrating nodelocaldns. I've got a couple of questions: nodelocaldns uses registry . Should I switch it to ? Is considered sort of a prod image registry? a multi-arch image? Because currently we use is it better to keep up with the new tags for ? So that we could quickly upgrade base image version as well? Thanks for the work!
YES! Make happy! This registry is actually a "proxy" that sig-k8s-infra is maintaining so people get directed to the closest Image registry, based on their cloud provider. As an example, with this users in AWS wont need to go to GCR to pull their image Yes, it is. It follows the same build pattern of debian-iptables. See Well, this is a good question.


Question: Containers: []v1.Container{ { Name:            "e2e-net-exec", <del> Image:           imageutils.GetE2EImage(imageutils.DebianIptables), </del> <ins> Image:           imageutils.GetE2EImage(imageutils.DistrolessIptables), </ins> ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"sleep", "600"}, SecurityContext: &v1.SecurityContext{

AFAIK we don't have an event / notification system that sends a message (in Slack, as an example) whenever a base image gets promoted. It would be cool to have at least for debian-base, debian-iptables and distroless-iptables, probably it's worth to question sig-release on this :) ( maybe know)
Thanks for the answer! Regarding the - we use it as a compile image for dnsmasq. Will this image still be supported or should I find something other to replace this compile image?
Short answer: I imagine/suppose there is no deprecation plan for this image right now, as it is used as the compilation base for a bunch of other things (and it is base of etcd image) Long answer: In case of distroless-iptables the rationale was: we need to reduce costs (and by reducing costs, reducing image size) we need to use something with a smaller footprint and reduce the amount of packages, and as a result the amount of CVEs in image package. kube-proxy exists in all nodes (unless you use a CNI that implements its features) so it is a good candidate I guess right now debian-base will still be used (I don't remember on the top of my head if other kubernetes components are already running using distroless or compiled based on debian-base, as far as I can see at least etcd image depends on debian-base () so probably any effort on this image should be announced.


Question: print "readonly {var}={value}".format(var = k, value = pipes.quote(str(v))) ''' < "${kube_env_yaml}") <del> # Infer master status from presence in node pool if [[ $(hostname) = ${NODE_INSTANCE_PREFIX}* ]]; then KUBERNETES_MASTER="false" else </del> <ins> # Infer master status from hostname if [[ $(hostname) == "${INSTANCE_PREFIX}-master" ]]; then </ins> KUBERNETES_MASTER="true" <ins> else KUBERNETES_MASTER="false" </ins> fi }

Node 2's kubelet logs: controller-manager side: I have a cluster to hand off if someone wants to debug. It's not a transient state: node 2 just failed to ever post a valid status. cc cc (who looked at it briefly with me)
I will take a look soon.
Ok, why kubelet on node 2 talks to ip address of master instead of hostname? root ps -ef | grep kubelet root 1 3 17:35 ? 00:10:45 /usr/local/bin/kubelet --apiservers=https://10.240.49.47 --config=/etc/kubernetes/manifests --allowprivileged=False --v=2 --clusterdns=10.51.240.10 -- It looks like a cluster up issue to me, not a node problem.
It looks like failed to translate the master name into the fqdn (which is done ).
Fascinating. I think we're seeing instance-DNS flakiness, because that name is supposed to get converted ASAP. But what's weirder is ... that's not even the master:
The is definitely right:
Okay, this is messed up. Here's node 1, which is happy: Here's our unhappy node .... that decided to bring itself as a master:
haha... I checked kube-env and did suspect unhappy node treat itself as master node based on those variables. But I didn't check salt state :-)
I ... just can't get there. By the time we get here: , node-2 has set: My best guess is that flaked here: . I think we're seeing a systemic issue where we're racing with GCE network configuration.
I'm going to work up a PR that hopefully improves the reliability of this section to some extent.
<a href=""<img src="" title="made at "/</a
My working theory is that we raced with the configuration done normally by the GCE NetworkManager, leaving us in a bogus state when we made that check.
Why are we relying on inferring the master/node status from the hostname. It should be obvious from the variables set in the kube-env if you are the master or not. Masters don't get passed so why not just check that? Making sure DNS is working is probably also good for reliability...
On GKE, we actually pass the same kube-env on both sides for simplicity.


Question: print "readonly {var}={value}".format(var = k, value = pipes.quote(str(v))) ''' < "${kube_env_yaml}") <del> # Infer master status from presence in node pool if [[ $(hostname) = ${NODE_INSTANCE_PREFIX}* ]]; then KUBERNETES_MASTER="false" else </del> <ins> # Infer master status from hostname if [[ $(hostname) == "${INSTANCE_PREFIX}-master" ]]; then </ins> KUBERNETES_MASTER="true" <ins> else KUBERNETES_MASTER="false" </ins> fi }

We can obviously change that, but I was going for a change that didn't involve a deployment change on our side, too.


Question: d.ActiveConnections = int(native.Uint16(attr.Value)) case ipvsDestAttrInactiveConnections: d.InactiveConnections = int(native.Uint16(attr.Value)) <del> case ipvsSvcAttrStats: </del> <ins> case ipvsDestAttrStats: </ins> stats, err := assembleStats(attr.Value) if err != nil { return nil, err

Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error.


Question: d.ActiveConnections = int(native.Uint16(attr.Value)) case ipvsDestAttrInactiveConnections: d.InactiveConnections = int(native.Uint16(attr.Value)) <del> case ipvsSvcAttrStats: </del> <ins> case ipvsDestAttrStats: </ins> stats, err := assembleStats(attr.Value) if err != nil { return nil, err

I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO.


Question: d.ActiveConnections = int(native.Uint16(attr.Value)) case ipvsDestAttrInactiveConnections: d.InactiveConnections = int(native.Uint16(attr.Value)) <del> case ipvsSvcAttrStats: </del> <ins> case ipvsDestAttrStats: </ins> stats, err := assembleStats(attr.Value) if err != nil { return nil, err

However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).


Question: d.ActiveConnections = int(native.Uint16(attr.Value)) case ipvsDestAttrInactiveConnections: d.InactiveConnections = int(native.Uint16(attr.Value)) <del> case ipvsSvcAttrStats: </del> <ins> case ipvsDestAttrStats: </ins> stats, err := assembleStats(attr.Value) if err != nil { return nil, err

FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks


Question: git add "${VERSION_FILE}" git commit -m "Kubernetes version ${NEW_VERSION}-dev" <ins> echo "+++ Constructing backmerge branches" function return_to_kansas { git checkout -f "${current_branch}" } trap return_to_kansas EXIT backmerge="v${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH}-merge-to-master" backmergetmp="${backmerge}-tmp-$(date +%s)" # Now we create a temporary branch to revert the doc commit, then # create the backmerge branch for the convenience of the user. git checkout -b "${backmergetmp}" git revert "${dochash}" --no-edit git checkout -b "${backmerge}" "${fetch_remote}/master" git merge -s recursive -X ours "${backmergetmp}" -m "${VERSION_MAJOR}.${VERSION_MINOR}.${VERSION_PATCH} merge to master" git checkout "${current_branch}" git branch -D "${backmergetmp}" </ins> echo "" echo "Success you must now:" echo ""

doesn't work with our existing release flow. I thought it did, but, from cc closing out .
Actually, I just realized my last suggestion in isn't the worst. The pull requests to master are essentially only there to get the tags backmerged, so we can construct a release branch like so: and then a release branch backmerge branch like so: and it'll still work.


Question: partition = strconv.Itoa(int(pdSource.Partition)) } <del> pv := &v1.PersistentVolume{ </del> <ins> var am v1.PersistentVolumeAccessMode if pdSource.ReadOnly { am = v1.ReadOnlyMany } else { am = v1.ReadWriteOnce } return &v1.PersistentVolume{ </ins> ObjectMeta: metav1.ObjectMeta{ // A.K.A InnerVolumeSpecName required to match for Unmount Name: volume.Name,

When running CSI migration tests focussing on multi-attach scearnios with GCE PD inline volumes that are read-only, the following error pops up in the logs and the second attachment takes 6 mins: For read-only inline volumes, the above warning/error around is unexpected with CSI migration enabled and needs to be root-caused/resolved. What happened: CSI migration tests for read-only inline GCE PD volumes occasionally fails with the above messages in the logs when executing tests like What you expected to happen: The tests and multi-attach scenarios to execute smoothly with no warnings in the logs. How to reproduce it (as minimally and precisely as possible): Execute with focus set to and run a few times. Anything else we need to know?: This has been observed with CSI migration enabled and the tests exercising read-only inline volume scenarios. Environment: Kubernetes version (use ): v1.16 Cloud provider or hardware configuration: GCE OS (e.g: ): Kernel (e.g. ): Install tools: kube- Network plugin and version (if this is a network-related bug): Others:
/cc
/sig storage
Looks like hard coding access modes to ReadWriteOnce during translation for GCE PD at is causing isMultiAttachForbidden to return leading down to We need to set the access mode to ReadOnlyMany for in-tree inline volumes like in GCE PD (as well as others like EBS that support multi-attach) depending on instead of always setting it to ReadWriteOnce.
I can pick up this fix /assign


Question: } func (g *gcpAuthProvider) WrapTransport(rt http.RoundTripper) http.RoundTripper { <del> return &oauth2.Transport{ Source: g.tokenSource, Base:   rt, } </del> <ins> return &conditionalTransport{&oauth2.Transport{Source: g.tokenSource, Base: rt}} </ins> } func (g *gcpAuthProvider) Login() error { return nil }

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubectl token BUG REPORT Kubernetes version (use ): kubectl version Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): macOS Sierra 10.12.4 Kernel (e.g. ): uname -a Darwin jacobsimpson- 16.5.0 Darwin Kernel Version 16.5.0: Fri Mar 3 16:52:33 PST 2017; root:xnu-3789.51.2~3/RELEASEX8664 x8664 Install tools: Others: What happened: I set up a second user, then I use to authenticate as the second user. The parameter is ignored and the authentication token from the file is used instead, so I'm authenticated as the user configured in the config file, not as the user set on the command line parameters. I further verified this by commenting out the section in the config file with the authentication token and then the parameter starts to work as expected. What you expected to happen: I expect the command line parameters, when present, to override the config file settings. How to reproduce it (as minimally and precisely as possible): At this point, the error message should be indicating kubectl tried to authenticate with the account. Something like this: Now edit , search for the section for your cluster and comment out the whole section for the test cluster. Now it should be possible to run these two commands. The one using the user_token will fail with the error message indicating the user name.


Question: } func (g *gcpAuthProvider) WrapTransport(rt http.RoundTripper) http.RoundTripper { <del> return &oauth2.Transport{ Source: g.tokenSource, Base:   rt, } </del> <ins> return &conditionalTransport{&oauth2.Transport{Source: g.tokenSource, Base: rt}} </ins> } func (g *gcpAuthProvider) Login() error { return nil }

Anything else we need to know:
The token arg on the command line overrides the bearer token field in the config file, but I'm not sure the precedence of the token arg and the auth provider in the config file. cc
All auth providers currently override the "Authorization" header even if it's set. Should be an easy-ish check to add. Long term we probably want to have the auth provider stop wrapping the transport and just act like a token source , that way some central code would handle the HTTP headers and we wouldn't have to enforce this on a per plugin basis.
So the tokens from command line should be used, right? if so I can send a pr to fix this.
the client auth plugins shouldn't override the "Authorization" headers. Yeah feel free to send a PR.
Is this a regression or why 1.7?
bug is pretty self contained in the GCP client auth plugin and the fix is short Not strictly a regression, but something that popped up over this cycle as a bug impacting users. Is that reason enough to get it in during the 1.7 freeze?


Question: if resizeError != nil { klog.Errorf("MountVolume.NodeExpandVolume failed with %v", resizeError) <ins> // Resize failed. To make sure NodeExpand is re-tried again on the next attempt // *before* SetUp(), mark the mounted device as uncertain. markDeviceUncertainErr := actualStateOfWorld.MarkDeviceAsUncertain( volumeToMount.VolumeName, devicePath, deviceMountPath) if markDeviceUncertainErr != nil { // just log, return the resizeError error instead klog.Infof(volumeToMount.GenerateMsgDetailed( "MountVolume.MountDevice failed to mark volume as uncertain", markDeviceUncertainErr.Error())) } </ins> return volumeToMount.GenerateError("MountVolume.MountDevice failed while expanding volume", resizeError) } }

What happened: A custom CSI driver implementing returns a failure response, indicating that it did not complete its intended steps to prepare the volume for the mounting pod. Let's say that these steps involve a filesystem at the target directory, data from a service, and that data to a set of files in the mounted filesystem. If the first filesystem mounting step succeeds, but either of the subsequent two steps fail, the volume is still not ready for use by the pod. However, if this CSI driver returns a failure response, and the CSI mounter comes back and, before invoking the RPC again, it finds a filesystem already mounted at the target directory, , and ceases further calls to to allow the CSI driver to try again. What you expected to happen: If a CSI driver returns a failure response to a RPC, Kubernetes should honor that declaration of failure and invoke the RPC again, disallowing the mounting pod from starting until this RPC succeeds. Even though it's possible to partially work around the current behavior by having the driver mount the filesystem as late as possible, and then attempting to unmount it if fails to populate the volume with its intended data, doing so makes writing drivers more complicated and forces them to do extra work. How to reproduce it (as minimally and precisely as possible): In a CSI driver (even one that serves just ephemeral, inline volumes), implement to mount a filesystem at the target path, then return a failure response. Confirm that Kubernetes does not invoke again, because there's now a filesystem mounted at the target path. Anything else we need to know?: This topic came up for discussion in the "Kubernetes" Slack team. There, helped investigate this complaint, and to discuss changing this behavior. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Others: csi-node-driver-registrar container image tag: v1.0.2
/sig storage
While the plugin should try to clean up as much as possible, I agree that kubelet should not be assuming that if there's a mount point, the publish is successful. NodePublish is supposed to be idempotent so it should theoretically be safe to call it when it's already mounted, although there is the possibility that plugins may have written against the K8s behavior.


Question: if resizeError != nil { klog.Errorf("MountVolume.NodeExpandVolume failed with %v", resizeError) <ins> // Resize failed. To make sure NodeExpand is re-tried again on the next attempt // *before* SetUp(), mark the mounted device as uncertain. markDeviceUncertainErr := actualStateOfWorld.MarkDeviceAsUncertain( volumeToMount.VolumeName, devicePath, deviceMountPath) if markDeviceUncertainErr != nil { // just log, return the resizeError error instead klog.Infof(volumeToMount.GenerateMsgDetailed( "MountVolume.MountDevice failed to mark volume as uncertain", markDeviceUncertainErr.Error())) } </ins> return volumeToMount.GenerateError("MountVolume.MountDevice failed while expanding volume", resizeError) } }

For the mount point checking here before calling NodePublishVolume, I am wondering it is just for performance optimization, or it has other purposes. Have concern about some drivers might depend on this check if they are not implemented with idenpotency.
Those drivers would be operating in violation of the specification. I'd prefer that we don't penalize already conforming and yet-to-be written drivers in order to accommodate these existing drivers—if they even exist.
More fallout in last night's testing: I introduced a bug into my driver that caused it to panic during ("Writing that data to a set of files in the mounted filesystem"). My handler function is set up to unmount a successfully mounted volume via if it doesn't complete steps 2 and 3 above, in order to deal with this issue's principal complaint. My driver container would crash due to this panic. The kubelet would call on my driver twice, it would crash twice, but the pod still started running. I'm not yet sure whether or not the filesystem got unmounted successfully via the on the first call, but somehow the pod started despite my handler function never completing successfully. I'm going to try to get some more debugging time on this later today to see if I can figure out what the kubelet was doing there.
Studying my code more, I see that my function only unmounted the mounted directory when the containing handler function was returning an error. When unwinding due to a panic, the named return value of type I was inspecting was still nil. Revising the criteria in that function covers both failure reasons (returning an and unwinding due to a panic). Regardless, this further underscores the problem: If a driver makes it as far as mounting a filesystem at the target directory, but fails afterward—including the process crashing and never responding positively to the RPC—the mounting pod will start. That's likely to then cause the pod to fail, for lack of the files it expected to find mounted.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.


Question: if resizeError != nil { klog.Errorf("MountVolume.NodeExpandVolume failed with %v", resizeError) <ins> // Resize failed. To make sure NodeExpand is re-tried again on the next attempt // *before* SetUp(), mark the mounted device as uncertain. markDeviceUncertainErr := actualStateOfWorld.MarkDeviceAsUncertain( volumeToMount.VolumeName, devicePath, deviceMountPath) if markDeviceUncertainErr != nil { // just log, return the resizeError error instead klog.Infof(volumeToMount.GenerateMsgDetailed( "MountVolume.MountDevice failed to mark volume as uncertain", markDeviceUncertainErr.Error())) } </ins> return volumeToMount.GenerateError("MountVolume.MountDevice failed while expanding volume", resizeError) } }

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"), }, }, <ins> "tcpBeLiberal": { SchemaProps: spec.SchemaProps{ Description: "tcpBeLiberal, if true, kube-proxy will configure conntrack to run in liberal mode for TCP connections and packets with out-of-window sequence numbers won't be marked INVALID.", Default:     false, Type:        []string{"boolean"}, Format:      "", }, }, </ins> "udpTimeout": { SchemaProps: spec.SchemaProps{ Description: "udpTimeout is how long an idle UDP conntrack entry in UNREPLIED state will remain in the conntrack table (e.g. '30s'). Must be greater than 0 to set.",

What happened: When kube-proxy is configured with a cluster-cidr range, it drops packets with an INVALID conntrack state for all sources on the KUBE-FORWARD chain. This is meant to prevent spurious retransmits in long-lived TCP connections to a service IP, but it prevents clusters from having asymmetrical routing on private subnets with services outside of a cluster. What you expected to happen: Packets on the KUBE-FORWARD chain with an INVALID conntrack state should be dropped for only sources inside of the cluster-cidr if set. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: Change to drop INVALID conntrack packets was in Backing out that commit and compiling is my current workaround. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I am reading but I don't fully grok what caused the problem in the first place, or why we think "only drop if it was in-cluster" is a correct fix? Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. Is this a conntrack bug? Should it be applying the DNAT?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I can try to give a bit more context to our setup, but the knowledge I have around this all dates back to Feb, unfortuately, so I may have forgotten details. We were replacing our mesos cluster with a kubernetes cluster in Feb and using a custom ingress solution (HAProxy w/an outside process configuring it for pod ips / mesos ip-ports) for both. The ingress nodes lived in the private subnet, but outside of the kubernetes cluster cidr range. When we upgraded from kubernetes 1.14 to kubernetes 1.15 (the first version with PR ) our ingress could no longer talk to any pod ips.


Question: Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"), }, }, <ins> "tcpBeLiberal": { SchemaProps: spec.SchemaProps{ Description: "tcpBeLiberal, if true, kube-proxy will configure conntrack to run in liberal mode for TCP connections and packets with out-of-window sequence numbers won't be marked INVALID.", Default:     false, Type:        []string{"boolean"}, Format:      "", }, }, </ins> "udpTimeout": { SchemaProps: spec.SchemaProps{ Description: "udpTimeout is how long an idle UDP conntrack entry in UNREPLIED state will remain in the conntrack table (e.g. '30s'). Must be greater than 0 to set.",

I discovered that the KUBE-FORWARD chain had a rule to drop invalid packets, which was in Compiling kubernetes 1.15 without this commit fixed our problems, and we have been carrying that patch forward ever since. I would like to get rid of our forked kubernetes, hence this ticket. When I was writing this ticket up, I also found also mentions the same issue with the iptables dropping INVALID conntrack state packets near the end, regarding asymmetrical routing. Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. I think you are correct, I can imagine packets from outside of a cluster could exhibit the same behavior as packets inside the cluster and cause spurious resets. However, I am wondering if kube-proxy should be managing packets from outside of the cluster on a node... That seems outside of the scope of kube-proxy.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I took a stab at implementing what I think is the correct behavior for kube-proxy in .
/assign
I think that another option is to set
I don't understand how that will help me here. The packets with a ctstate INVALID are not generated due to conntrack saturation (), but due to asymmetric routing in our network. See for an example of a similar setup.
are the packets on your asymmetric routing setup marked as invalid if you set ? it should only mark the ones that are out of window per the description
/triage accepted
I'd love to pin this down, but I don't have the means to set up a repro. Is there a way I can set up a way to experiment?
We were bit by this too after revisiting an old service that had been unused since the heady days of pre-, and would also like to see the proposed patch merged (or a configuration option to prevent insertion of the INVALID state drop rule - either suits us). I unfortunately don't have any useful input as to setting up for reproduction. In our case we have a pretty odd anycast set up with a small pool of (public) addresses managed by calico.


Question: Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"), }, }, <ins> "tcpBeLiberal": { SchemaProps: spec.SchemaProps{ Description: "tcpBeLiberal, if true, kube-proxy will configure conntrack to run in liberal mode for TCP connections and packets with out-of-window sequence numbers won't be marked INVALID.", Default:     false, Type:        []string{"boolean"}, Format:      "", }, }, </ins> "udpTimeout": { SchemaProps: spec.SchemaProps{ Description: "udpTimeout is how long an idle UDP conntrack entry in UNREPLIED state will remain in the conntrack table (e.g. '30s'). Must be greater than 0 to set.",

However I do seem to have a dirty workaround for small clusters. By setting notrack (ctstate UNTRACKED?) in the raw table for packets in our specific ranges that would otherwise match ctstate INVALID, the packets aren't dropped. Which may be of help to someone else who finds this issue.
We are seeing something very similar on large file downloads and our issue exactly fits this one's subject. These RESET (corrected after are happening intermittently and I am seeing similar behavior mentioned in the blog mentioned below. I am unsure if someone has found a workaround for this issue or not, so yeah my +1 to this issue being looked up by the community.
We're using as a solution for the connection reset we were seeing and might fix this issue.
Let me try to avoid that this thread derails and see if I can get current status: -proxy install rules to drop INVALID cstate connections, this solves the ~REJECT~ RESET problems are some scenarios, with asymmetric routing per example, that this rules may drop unrelated traffic. The request is to remove the rule that drops those INVALID states or to try to restrict it, and that can cause that people start to have the ~REJECT~ RESET problems and drops again. If we have a reproducer we can be more precise in the final solution, but removing one rule without any strong evidence in kube-proxy is very risky As a workaround for the asymetric routing problem, I suggested to set the nfconntracktcpbeliberal on the kernel: I think that will solve the problems, it will be good if someone can test it and provide feedback
No sadly, its my bad, it is not REJECT, it is RESET that's happening and conntrack has nothing to do with it, mostly (I hope). Kubeproxy is doing what is intended but specifically when there are external agents in play, the packet doesn't sends an ACK back to the response of SYN-ACK. Our case, we hit A to initiate a download, A calls B to get the action completed to generate download data for the user finally via their browser. A and B both are on k8s, we saw A starts and a complete cycle goes for sometime until there multiple parallel downloads requested where few gets completed and few gets a RESET.


Question: Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"), }, }, <ins> "tcpBeLiberal": { SchemaProps: spec.SchemaProps{ Description: "tcpBeLiberal, if true, kube-proxy will configure conntrack to run in liberal mode for TCP connections and packets with out-of-window sequence numbers won't be marked INVALID.", Default:     false, Type:        []string{"boolean"}, Format:      "", }, }, </ins> "udpTimeout": { SchemaProps: spec.SchemaProps{ Description: "udpTimeout is how long an idle UDP conntrack entry in UNREPLIED state will remain in the conntrack table (e.g. '30s'). Must be greater than 0 to set.",

On primary debugging, it appears that RESET was getting invoked by the pod but pod is just sending a RST-ACK not RST itself. The missing packet is never found in the default route, unless we see that there is a chance where ACK is missed and we found some lines in tcpdump with missing sequence for ACKs. Thats why i doubt my issue is related here. One more weird coincidence is few downloads have spurious transmission.
Thankyou, I will take a look, but mostly I would like to wait before using it, debug more to understand the actual cause and then act on it.
sorry , I wanted to say RESET, thanks for catching it. If you think is different and you can provide information and how to reproduce it, I suggest you to open a new issue
I will sure do, when it is different. For now I would like to watch and submit my findings if it is related. PS: sysctl -w net.netfilter.nfconntracktcpbeliberal=1 is already implemented and we are testing if it does something helpful. so far it doesn't looks like it. thanks for your help.
/area kube-proxy
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I have the feeling we are also affected by this with a standard calico installation: every node announces a /122 IPv6 network to the upstream router the upstream router uses ECMP to route the packet back packets that are sent from* the pod and returned via asymmetric path from outside seem to be dropped
and if you set ?
Same behaviour with set on all nodes, the outgoing pod traffic does not return. And I instantly see packets going back into the drop rule:
:thinking:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.


Question: Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"), }, }, <ins> "tcpBeLiberal": { SchemaProps: spec.SchemaProps{ Description: "tcpBeLiberal, if true, kube-proxy will configure conntrack to run in liberal mode for TCP connections and packets with out-of-window sequence numbers won't be marked INVALID.", Default:     false, Type:        []string{"boolean"}, Format:      "", }, }, </ins> "udpTimeout": { SchemaProps: spec.SchemaProps{ Description: "udpTimeout is how long an idle UDP conntrack entry in UNREPLIED state will remain in the conntrack table (e.g. '30s'). Must be greater than 0 to set.",

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, are you still working on this? Same problem happened in our clusters, just because of asymmetrical routing.
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Also the command suggested by is helping us, but I still think this should be a bug of kube-proxy.
The problem here is: what happens if the solutions are exclusive or we regress? The addition of this rule fixed a legit bug that was documented and there is also a nice blog explaining the details Removing this rule without a repro and understanding what is the root cause of this issue is not really the best way to proceed. I think that people affected should try to find the root cause and explain the problem or just provide a way to reproduce it, so we can fix it
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


Question: Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"), }, }, <ins> "tcpBeLiberal": { SchemaProps: spec.SchemaProps{ Description: "tcpBeLiberal, if true, kube-proxy will configure conntrack to run in liberal mode for TCP connections and packets with out-of-window sequence numbers won't be marked INVALID.", Default:     false, Type:        []string{"boolean"}, Format:      "", }, }, </ins> "udpTimeout": { SchemaProps: spec.SchemaProps{ Description: "udpTimeout is how long an idle UDP conntrack entry in UNREPLIED state will remain in the conntrack table (e.g. '30s'). Must be greater than 0 to set.",

/close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The same problem happened in our cluster, the same as because of asymmetrical routing. the command suggested by doesn't work for us, and I delete the iptables rule and add new rule(refer to ), it works I think that makes sense I'll put up a reproduction of the gist link later on /reopen
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The main problem is how to be completely sure we don't regress, this comment from Tim is spot on
I'm trying to check that non-local connections will still reset without the INVALID Rule
I think the patch in is wrong; it changes the rule from "if INVALID then drop" to "if INVALID and from a pod, then drop" so that it doesn't affect non-kubernetes traffic, but I think the rule we actually need is "if INVALID and to a service, then drop". In particular, I believe we need to make sure that the rule applies to traffic with a cluster-external source and an in-cluster destination. It's possible that changing it to "if INVALID and to a pod, then drop" would also work. I skimmed and some linked bugs elsewhere but I was unable to confirm whether that rule would cover the reporters' use cases.
I think that is only INVALID and to a pod
In my case, cluster-external source to service is both ok with the INVALID rule or without the INVALID rule. It seems the rule doesn't affect on access service.


Question: Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"), }, }, <ins> "tcpBeLiberal": { SchemaProps: spec.SchemaProps{ Description: "tcpBeLiberal, if true, kube-proxy will configure conntrack to run in liberal mode for TCP connections and packets with out-of-window sequence numbers won't be marked INVALID.", Default:     false, Type:        []string{"boolean"}, Format:      "", }, }, </ins> "udpTimeout": { SchemaProps: spec.SchemaProps{ Description: "udpTimeout is how long an idle UDP conntrack entry in UNREPLIED state will remain in the conntrack table (e.g. '30s'). Must be greater than 0 to set.",

I add this rule but it doesn't work
So, based on that, the bug happens when a reply packet fails to get un-DNAT'ed, causing the client to then send a RST to the server, which matches the source/destination IP that the server knows of. But this could happen if you had a service with external IPs too; in that case the reply packet would be from the external endpoint IP to the node IP (because the packet would have been masqueraded), and if conntrack didn't unmap it correctly then the node would send a RST to the external IP, breaking the connection in just the same way. In this case there is no good way kube-proxy can recognize the packet as being its own, other than by using conntrack, which presumably won't work if the packet is INVALID?
It seems that there is no better way to solve this for external sources accessing the cluster CIDR, is it right? If so, Can we have a flag to decide whether to create The invalid rule? It would be nice that It's up to the user to decide whether to create the rule.
There's related discussion in . I think we should create whatever iptables rules we can that will fix up our own traffic without interfering with other traffic, and also document the sysctl that admins can use to change conntrack behavior to work around the problem. So probably, we should change the "drop if INVALID" rule to "drop if INVALID and to a pod", and also address the problem in , but not try to solve the problem with external IPs I mentioned in because we don't have a good way of fixing that without potentially breaking other things. (And it's kind of way more of an edge case anyway.) Note that can currently output rules for "from a pod", but not for "to a pod", so that will need some tweaking. (FWIW will also eventually need the ability to output nftables rules rather than iptables rules so that's another thing to think about if someone is going to change its API.)
So in the absence of a way to fix the problem with external IPs, can we have a flag to decide whether to create this rule or not? If true (default), create the drop if INVALID and to a pod rule. Otherwise, We do not create the rule.


Question: Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"), }, }, <ins> "tcpBeLiberal": { SchemaProps: spec.SchemaProps{ Description: "tcpBeLiberal, if true, kube-proxy will configure conntrack to run in liberal mode for TCP connections and packets with out-of-window sequence numbers won't be marked INVALID.", Default:     false, Type:        []string{"boolean"}, Format:      "", }, }, </ins> "udpTimeout": { SchemaProps: spec.SchemaProps{ Description: "udpTimeout is how long an idle UDP conntrack entry in UNREPLIED state will remain in the conntrack table (e.g. '30s'). Must be greater than 0 to set.",

I believe that even without this rule we can solve this problem by way of sysctl, right? I think we should do the following things We need a flag to disable the rule We should change the "drop if INVALID" rule to "drop if INVALID and to a pod" We document sysctl as a way to solve the problem.
If the rule is "drop if INVALID and to a pod", we shouldn't need a flag to disable it; it won't affect non-pod traffic, and it won't interfere with users that want to set the sysctl instead. (Setting the sysctl will cause packets to not be marked as INVALID so the rule we add just won't get hit.)
OK, drop if INVALID and to a pod is help us identify cluster traffic, right? We only drop cluster traffic if it's invalid, but this will not solve the problem of external IP. So we need a flag to disable the rule is drop if INVALID?
The same problem happened in our cluster, the same as because of asymmetrical routing None of the above methods solve the problem in our ENV. Are there any other workarounds?
Can you execute the following command on each node? I think this is a workaround.
Thank you for your reply I also tried removing this rule. However, k8s will reconfigure the deleted rules to the system at some point in time.
I worded that badly: I meant "external IP" as in "the endpoints are cluster-external IPs rather than pod IPs", not as in . So that would apply only to people using kube-proxy to load-balance traffic from clients outside the cluster to servers outside the cluster, which is a pretty weird use case. And nobody ever reported the packet RST bug for that case anyway. So I was arguing that we can just ignore that case. So: change the existing "drop if invalid" rule to "drop if invalid and to a pod" document the sysctl We don't need an option to disable the drop rule, because it won't affect non-k8s traffic, and it will automatically become a no-op if someone decides to use the sysctl instead.
Thank you for your response. I'd love to take this one. But I have a question, How to define k8s traffic?


Question: Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"), }, }, <ins> "tcpBeLiberal": { SchemaProps: spec.SchemaProps{ Description: "tcpBeLiberal, if true, kube-proxy will configure conntrack to run in liberal mode for TCP connections and packets with out-of-window sequence numbers won't be marked INVALID.", Default:     false, Type:        []string{"boolean"}, Format:      "", }, }, </ins> "udpTimeout": { SchemaProps: spec.SchemaProps{ Description: "udpTimeout is how long an idle UDP conntrack entry in UNREPLIED state will remain in the conntrack table (e.g. '30s'). Must be greater than 0 to set.",

If a pod in a cluster, we call it an underlay pod, its IP is not allocated from cluster CIDR, this is a real underlay IP, when it accesses other underlay pods in the cluster or pods that allocate IP from cluster CIDR(We call it overlay pod), is this k8s traffic? In our cluster, multiple types of CNI pods are implemented through multus running in a cluster, because of the problem of asymmetrical routing and the rule that makes them inaccessible, it seems that the above method will not solve this problem.
For kube-proxy, this is done by the . If you are in a cluster where there is no simple way to discriminate pod from non-pod traffic, then you'd have to run kube-proxy with , and then kube-proxy will see that returns and so it won't try to distinguish local from non-local traffic, and will do some things (like internal/external traffic policy) slightly differently as a result. So in this case, if there was no functioning localDetector, it would also not create the "drop invalid" rule
Thanks for the details, this is probably the case we want.
/assign
I did not understand how you could get asymmetric routing in K8s, but reading up more carefully I saw: If I understand this correctly, this is a load-balancer outside K8s that load-balance directly to POD addresses. To get asymmetric routing some kind of Direct Server Return (DSR) must also be used. So, to get asymmetric routing: from an external source is directed to an external LB LB is configured with POD-IPs as targets and selects one packet is routed to a K8s node, e.g. with ECMP. But is may not be the node where the target POD is executing packet is handled by the POD, but the response packet may be directed to some other GW This will give asymmetric routing. But it's not the every-day K8s network setup for sure. Given the above it's fairly easy to reproduce the problem. You don't even have to setup a load-balancer. The biggest problem may be that you need two ways in to the cluster. Example: ! I setup direct routes that ensures asymmetric routing, which can be checked with on vm-201 and vm-202.