Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened. But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)

ctx, cancel := watchtools.ContextWithOptionalTimeout(context.Background(), timeout) defer cancel() <del> preconditionFunc := func(store cache.Store) (bool, error) { _, exists, err := store.Get(&metav1.ObjectMeta{Namespace: ns, Name: name}) if err != nil { return true, err } if !exists { // We need to make sure we see the object in the cache before we start waiting for events // or we would be waiting for the timeout if such object didn't exist. // (e.g. it was deleted before we started informers so they wouldn't even see the delete event) return true, errors.NewNotFound(corev1.Resource("pods"), name) } return false, nil } </del> fieldSelector := fields.OneTermEqualSelector("metadata.name", name).String() lw := &cache.ListWatch{ ListFunc: func(options metav1.ListOptions) (runtime.Object, error) {


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened. But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)

intr := interrupt.New(nil, cancel) var result *corev1.Pod err := intr.Run(func() error { <del> ev, err := watchtools.UntilWithSync(ctx, lw, &corev1.Pod{}, preconditionFunc, func(ev watch.Event) (bool, error) { return exitCondition(ev) }) </del> <ins> ev, err := watchtools.UntilWithSync(ctx, lw, &corev1.Pod{}, nil, exitCondition) </ins> if ev != nil { result = ev.Object.(*corev1.Pod) }


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened. But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)

return false, nil } <ins> // podSucceeded returns true if the pod has run to completion, false if the pod has not yet // reached running state, or an error in any other case. func podSucceeded(event watch.Event) (bool, error) { switch event.Type { case watch.Deleted: return false, errors.NewNotFound(schema.GroupResource{Resource: "pods"}, "") } switch t := event.Object.(type) { case *corev1.Pod: return t.Status.Phase == corev1.PodSucceeded, nil } return false, nil } </ins> // podRunningAndReady returns true if the pod is running and ready, false if the pod has not // yet reached those states, returns ErrPodCompleted if the pod has run to completion, or // an error in any other case.


Question:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened. But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)

_, err = framework.NewKubectlCommand(ns, nsFlag, "run", "-i", "--image="+busyboxImage, "--restart=OnFailure", "failure-2", "--", "/bin/sh", "-c", "cat && exit 42"). WithStdinData("abcd1234"). Exec() <del> framework.ExpectNoError(err) </del> <ins> ee, ok = err.(uexec.ExitError) framework.ExpectEqual(ok, true) if !strings.Contains(ee.String(), "timed out") { framework.Failf("Missing expected 'timed out' error, got: %#v", ee) } </ins> ginkgo.By("running a failing command without --restart=Never, but with --rm") _, err = framework.NewKubectlCommand(ns, nsFlag, "run", "-i", "--image="+busyboxImage, "--restart=OnFailure", "--rm", "failure-3", "--", "/bin/sh", "-c", "cat && exit 42"). WithStdinData("abcd1234"). Exec() <del> framework.ExpectNoError(err) </del> <ins> ee, ok = err.(uexec.ExitError) framework.ExpectEqual(ok, true) if !strings.Contains(ee.String(), "timed out") { framework.Failf("Missing expected 'timed out' error, got: %#v", ee) } </ins> e2epod.WaitForPodToDisappear(f.ClientSet, ns, "failure-3", labels.Everything(), 2*time.Second, wait.ForeverTestTimeout) ginkgo.By("running a failing command with --leave-stdin-open")


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: , from whatever combination of inputs we stuck into it, segfaulted. The segfault, which originates from a slightly modified fork of v1.17, looks as such: Translated to master, where the problem appears to be present, the segfault originates from this call: And that is called from here: This seems odd, because The only pointer involved in the segfaulting call is itself The caller nil-checks before calling a method on . I did some digging and found , which is a fix for this problem. However, this fix appears to be undone in . There doesn't seem to be a reason for this to have been changed in the latter PR. What you expected to happen: should not segfault from any bizarre combination of user input. If the situation is invalid, it should fail gracefully. How to reproduce it (as minimally and precisely as possible): Honestly, I'm not 100% sure what combination of inputs we crammed into , but I can locate them and post them if necessary. Anything else we need to know?: I don't feel confident opening a PR to fix this myself, but in addition to recreating the fix from , a comment should be explaining why there is this weird, seemingly pointless layer of indirection when assigning to the interface, so that this doesn't get un-fixed again.
Maybe decompose the above statement into multiple lines which should give us more idea which field was nil.
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
can't be nil, as it is of type , which is not an interface or pointer is an interface. However, when built from the exported constructor, , it is set to a literal , which is likewise not a pointer Therefore, the only remaining field is itself. Plus, I don't quite recall my go segfault stack trace syntax, but I'm relatively sure that indicates that the first argument to the method is , nil, and that the first argument to methods is the object being called on.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
When config.HealthzBindAddress is empty, we would encounter the scenario depicted in the stack trace. I have proposed a fix.
/assign
That commit, , is part of the PR that I believe introduced the problem. I strongly believe the problem to be one of a non-nil interface with a nil receiver. This is the only way that could be nil in the snippet of code from . Before PR (the changes from which are present in the version I ran), there was this odd looking set of code: The variable healthzServer, which is a pointer of type , was assigned to the variable , which is an interface of type . Then, later on, the variable was passed to , as an argument of type . This looks unexpected, and it appears that the variable is extraneous. After the changes in PR , the variable was removed, and was passed directly to as the argument for interface type . The former set of changes, with the intermediate variable, are meant to avoid the situation of a non-nil interface with a nil value. If the variable is never set in the conditional statement, then it is a nil interface. If is passed directly, then the result is the argument to is an an interface who's value is and who's concrete type is . This means that is called with as the receiver. I strongly believe this to be the root cause of the issue, but I do not know enough about kubernetes to open a PR with the correct fix, which would be to reinstate the variable (after the renaming in , this would be a variable of type ) to use as a go-between, to once again avoid this situation of a non-nil interface with a nil receiver. I believe that the removal of the variable was out of a misunderstanding of its purpose. If my assessment of the problem is incorrect, please let me know so I can understand where my logic is wrong. It is very possible that I am misunderstanding the phenomenon of a non-nil interface with a nil receiver. I would open a PR myself, but I do not know enough about Kubernetes to be confident in my ability to correctly fix and test this problem.
/remove-triage unresolved

UseEndpointSlices      bool OOMScoreAdj            *int32 ConfigSyncPeriod       time.Duration <del> HealthzServer          *healthcheck.ProxierHealthServer </del> <ins> HealthzServer          healthcheck.ProxierHealthUpdater </ins> } // createClients creates a kube client and an event client from the given config and masterOverride.


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: , from whatever combination of inputs we stuck into it, segfaulted. The segfault, which originates from a slightly modified fork of v1.17, looks as such: Translated to master, where the problem appears to be present, the segfault originates from this call: And that is called from here: This seems odd, because The only pointer involved in the segfaulting call is itself The caller nil-checks before calling a method on . I did some digging and found , which is a fix for this problem. However, this fix appears to be undone in . There doesn't seem to be a reason for this to have been changed in the latter PR. What you expected to happen: should not segfault from any bizarre combination of user input. If the situation is invalid, it should fail gracefully. How to reproduce it (as minimally and precisely as possible): Honestly, I'm not 100% sure what combination of inputs we crammed into , but I can locate them and post them if necessary. Anything else we need to know?: I don't feel confident opening a PR to fix this myself, but in addition to recreating the fix from , a comment should be explaining why there is this weird, seemingly pointless layer of indirection when assigning to the interface, so that this doesn't get un-fixed again.
Maybe decompose the above statement into multiple lines which should give us more idea which field was nil.
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
can't be nil, as it is of type , which is not an interface or pointer is an interface. However, when built from the exported constructor, , it is set to a literal , which is likewise not a pointer Therefore, the only remaining field is itself. Plus, I don't quite recall my go segfault stack trace syntax, but I'm relatively sure that indicates that the first argument to the method is , nil, and that the first argument to methods is the object being called on.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
When config.HealthzBindAddress is empty, we would encounter the scenario depicted in the stack trace. I have proposed a fix.
/assign
That commit, , is part of the PR that I believe introduced the problem. I strongly believe the problem to be one of a non-nil interface with a nil receiver. This is the only way that could be nil in the snippet of code from . Before PR (the changes from which are present in the version I ran), there was this odd looking set of code: The variable healthzServer, which is a pointer of type , was assigned to the variable , which is an interface of type . Then, later on, the variable was passed to , as an argument of type . This looks unexpected, and it appears that the variable is extraneous. After the changes in PR , the variable was removed, and was passed directly to as the argument for interface type . The former set of changes, with the intermediate variable, are meant to avoid the situation of a non-nil interface with a nil value. If the variable is never set in the conditional statement, then it is a nil interface. If is passed directly, then the result is the argument to is an an interface who's value is and who's concrete type is . This means that is called with as the receiver. I strongly believe this to be the root cause of the issue, but I do not know enough about kubernetes to open a PR with the correct fix, which would be to reinstate the variable (after the renaming in , this would be a variable of type ) to use as a go-between, to once again avoid this situation of a non-nil interface with a nil receiver. I believe that the removal of the variable was out of a misunderstanding of its purpose. If my assessment of the problem is incorrect, please let me know so I can understand where my logic is wrong. It is very possible that I am misunderstanding the phenomenon of a non-nil interface with a nil receiver. I would open a PR myself, but I do not know enough about Kubernetes to be confident in my ability to correctly fix and test this problem.
/remove-triage unresolved

Namespace: "", } <del> var healthzServer *healthcheck.ProxierHealthServer </del> <ins> var healthzServer healthcheck.ProxierHealthUpdater </ins> if len(config.HealthzBindAddress) > 0 { healthzServer = healthcheck.NewProxierHealthServer(config.HealthzBindAddress, 2*config.IPTables.SyncPeriod.Duration, recorder, nodeRef) }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: , from whatever combination of inputs we stuck into it, segfaulted. The segfault, which originates from a slightly modified fork of v1.17, looks as such: Translated to master, where the problem appears to be present, the segfault originates from this call: And that is called from here: This seems odd, because The only pointer involved in the segfaulting call is itself The caller nil-checks before calling a method on . I did some digging and found , which is a fix for this problem. However, this fix appears to be undone in . There doesn't seem to be a reason for this to have been changed in the latter PR. What you expected to happen: should not segfault from any bizarre combination of user input. If the situation is invalid, it should fail gracefully. How to reproduce it (as minimally and precisely as possible): Honestly, I'm not 100% sure what combination of inputs we crammed into , but I can locate them and post them if necessary. Anything else we need to know?: I don't feel confident opening a PR to fix this myself, but in addition to recreating the fix from , a comment should be explaining why there is this weird, seemingly pointless layer of indirection when assigning to the interface, so that this doesn't get un-fixed again.
Maybe decompose the above statement into multiple lines which should give us more idea which field was nil.
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
can't be nil, as it is of type , which is not an interface or pointer is an interface. However, when built from the exported constructor, , it is set to a literal , which is likewise not a pointer Therefore, the only remaining field is itself. Plus, I don't quite recall my go segfault stack trace syntax, but I'm relatively sure that indicates that the first argument to the method is , nil, and that the first argument to methods is the object being called on.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
When config.HealthzBindAddress is empty, we would encounter the scenario depicted in the stack trace. I have proposed a fix.
/assign
That commit, , is part of the PR that I believe introduced the problem. I strongly believe the problem to be one of a non-nil interface with a nil receiver. This is the only way that could be nil in the snippet of code from . Before PR (the changes from which are present in the version I ran), there was this odd looking set of code: The variable healthzServer, which is a pointer of type , was assigned to the variable , which is an interface of type . Then, later on, the variable was passed to , as an argument of type . This looks unexpected, and it appears that the variable is extraneous. After the changes in PR , the variable was removed, and was passed directly to as the argument for interface type . The former set of changes, with the intermediate variable, are meant to avoid the situation of a non-nil interface with a nil value. If the variable is never set in the conditional statement, then it is a nil interface. If is passed directly, then the result is the argument to is an an interface who's value is and who's concrete type is . This means that is called with as the receiver. I strongly believe this to be the root cause of the issue, but I do not know enough about kubernetes to open a PR with the correct fix, which would be to reinstate the variable (after the renaming in , this would be a variable of type ) to use as a go-between, to once again avoid this situation of a non-nil interface with a nil receiver. I believe that the removal of the variable was out of a misunderstanding of its purpose. If my assessment of the problem is incorrect, please let me know so I can understand where my logic is wrong. It is very possible that I am misunderstanding the phenomenon of a non-nil interface with a nil receiver. I would open a PR myself, but I do not know enough about Kubernetes to be confident in my ability to correctly fix and test this problem.
/remove-triage unresolved

// Updated should be called when the proxier has successfully updated the service // rules to reflect the current state. Updated() <ins> // Run starts the healthz http server and returns. Run() </ins> } <del> var _ ProxierHealthUpdater = &ProxierHealthServer{} </del> <ins> var _ ProxierHealthUpdater = &proxierHealthServer{} </ins> <del> // ProxierHealthServer returns 200 "OK" by default. It verifies that the delay between </del> <ins> // proxierHealthServer returns 200 "OK" by default. It verifies that the delay between </ins> // QueuedUpdate() calls and Updated() calls never exceeds healthTimeout. <del> type ProxierHealthServer struct { </del> <ins> type proxierHealthServer struct { </ins> listener    listener httpFactory httpServerFactory clock       clock.Clock


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: , from whatever combination of inputs we stuck into it, segfaulted. The segfault, which originates from a slightly modified fork of v1.17, looks as such: Translated to master, where the problem appears to be present, the segfault originates from this call: And that is called from here: This seems odd, because The only pointer involved in the segfaulting call is itself The caller nil-checks before calling a method on . I did some digging and found , which is a fix for this problem. However, this fix appears to be undone in . There doesn't seem to be a reason for this to have been changed in the latter PR. What you expected to happen: should not segfault from any bizarre combination of user input. If the situation is invalid, it should fail gracefully. How to reproduce it (as minimally and precisely as possible): Honestly, I'm not 100% sure what combination of inputs we crammed into , but I can locate them and post them if necessary. Anything else we need to know?: I don't feel confident opening a PR to fix this myself, but in addition to recreating the fix from , a comment should be explaining why there is this weird, seemingly pointless layer of indirection when assigning to the interface, so that this doesn't get un-fixed again.
Maybe decompose the above statement into multiple lines which should give us more idea which field was nil.
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
can't be nil, as it is of type , which is not an interface or pointer is an interface. However, when built from the exported constructor, , it is set to a literal , which is likewise not a pointer Therefore, the only remaining field is itself. Plus, I don't quite recall my go segfault stack trace syntax, but I'm relatively sure that indicates that the first argument to the method is , nil, and that the first argument to methods is the object being called on.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
When config.HealthzBindAddress is empty, we would encounter the scenario depicted in the stack trace. I have proposed a fix.
/assign
That commit, , is part of the PR that I believe introduced the problem. I strongly believe the problem to be one of a non-nil interface with a nil receiver. This is the only way that could be nil in the snippet of code from . Before PR (the changes from which are present in the version I ran), there was this odd looking set of code: The variable healthzServer, which is a pointer of type , was assigned to the variable , which is an interface of type . Then, later on, the variable was passed to , as an argument of type . This looks unexpected, and it appears that the variable is extraneous. After the changes in PR , the variable was removed, and was passed directly to as the argument for interface type . The former set of changes, with the intermediate variable, are meant to avoid the situation of a non-nil interface with a nil value. If the variable is never set in the conditional statement, then it is a nil interface. If is passed directly, then the result is the argument to is an an interface who's value is and who's concrete type is . This means that is called with as the receiver. I strongly believe this to be the root cause of the issue, but I do not know enough about kubernetes to open a PR with the correct fix, which would be to reinstate the variable (after the renaming in , this would be a variable of type ) to use as a go-between, to once again avoid this situation of a non-nil interface with a nil receiver. I believe that the removal of the variable was out of a misunderstanding of its purpose. If my assessment of the problem is incorrect, please let me know so I can understand where my logic is wrong. It is very possible that I am misunderstanding the phenomenon of a non-nil interface with a nil receiver. I would open a PR myself, but I do not know enough about Kubernetes to be confident in my ability to correctly fix and test this problem.
/remove-triage unresolved

} // NewProxierHealthServer returns a proxier health http server. <del> func NewProxierHealthServer(addr string, healthTimeout time.Duration, recorder record.EventRecorder, nodeRef *v1.ObjectReference) *ProxierHealthServer { </del> <ins> func NewProxierHealthServer(addr string, healthTimeout time.Duration, recorder record.EventRecorder, nodeRef *v1.ObjectReference) ProxierHealthUpdater { </ins> return newProxierHealthServer(stdNetListener{}, stdHTTPServerFactory{}, clock.RealClock{}, addr, healthTimeout, recorder, nodeRef) } <del> func newProxierHealthServer(listener listener, httpServerFactory httpServerFactory, c clock.Clock, addr string, healthTimeout time.Duration, recorder record.EventRecorder, nodeRef *v1.ObjectReference) *ProxierHealthServer { return &ProxierHealthServer{ </del> <ins> func newProxierHealthServer(listener listener, httpServerFactory httpServerFactory, c clock.Clock, addr string, healthTimeout time.Duration, recorder record.EventRecorder, nodeRef *v1.ObjectReference) *proxierHealthServer { return &proxierHealthServer{ </ins> listener:      listener, httpFactory:   httpServerFactory, clock:         c,


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: , from whatever combination of inputs we stuck into it, segfaulted. The segfault, which originates from a slightly modified fork of v1.17, looks as such: Translated to master, where the problem appears to be present, the segfault originates from this call: And that is called from here: This seems odd, because The only pointer involved in the segfaulting call is itself The caller nil-checks before calling a method on . I did some digging and found , which is a fix for this problem. However, this fix appears to be undone in . There doesn't seem to be a reason for this to have been changed in the latter PR. What you expected to happen: should not segfault from any bizarre combination of user input. If the situation is invalid, it should fail gracefully. How to reproduce it (as minimally and precisely as possible): Honestly, I'm not 100% sure what combination of inputs we crammed into , but I can locate them and post them if necessary. Anything else we need to know?: I don't feel confident opening a PR to fix this myself, but in addition to recreating the fix from , a comment should be explaining why there is this weird, seemingly pointless layer of indirection when assigning to the interface, so that this doesn't get un-fixed again.
Maybe decompose the above statement into multiple lines which should give us more idea which field was nil.
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
can't be nil, as it is of type , which is not an interface or pointer is an interface. However, when built from the exported constructor, , it is set to a literal , which is likewise not a pointer Therefore, the only remaining field is itself. Plus, I don't quite recall my go segfault stack trace syntax, but I'm relatively sure that indicates that the first argument to the method is , nil, and that the first argument to methods is the object being called on.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
When config.HealthzBindAddress is empty, we would encounter the scenario depicted in the stack trace. I have proposed a fix.
/assign
That commit, , is part of the PR that I believe introduced the problem. I strongly believe the problem to be one of a non-nil interface with a nil receiver. This is the only way that could be nil in the snippet of code from . Before PR (the changes from which are present in the version I ran), there was this odd looking set of code: The variable healthzServer, which is a pointer of type , was assigned to the variable , which is an interface of type . Then, later on, the variable was passed to , as an argument of type . This looks unexpected, and it appears that the variable is extraneous. After the changes in PR , the variable was removed, and was passed directly to as the argument for interface type . The former set of changes, with the intermediate variable, are meant to avoid the situation of a non-nil interface with a nil value. If the variable is never set in the conditional statement, then it is a nil interface. If is passed directly, then the result is the argument to is an an interface who's value is and who's concrete type is . This means that is called with as the receiver. I strongly believe this to be the root cause of the issue, but I do not know enough about kubernetes to open a PR with the correct fix, which would be to reinstate the variable (after the renaming in , this would be a variable of type ) to use as a go-between, to once again avoid this situation of a non-nil interface with a nil receiver. I believe that the removal of the variable was out of a misunderstanding of its purpose. If my assessment of the problem is incorrect, please let me know so I can understand where my logic is wrong. It is very possible that I am misunderstanding the phenomenon of a non-nil interface with a nil receiver. I would open a PR myself, but I do not know enough about Kubernetes to be confident in my ability to correctly fix and test this problem.
/remove-triage unresolved

} // Updated updates the lastUpdated timestamp. <del> func (hs *ProxierHealthServer) Updated() { </del> <ins> func (hs *proxierHealthServer) Updated() { </ins> hs.lastUpdated.Store(hs.clock.Now()) } // QueuedUpdate updates the lastQueued timestamp. <del> func (hs *ProxierHealthServer) QueuedUpdate() { </del> <ins> func (hs *proxierHealthServer) QueuedUpdate() { </ins> hs.lastQueued.Store(hs.clock.Now()) } // Run starts the healthz http server and returns. <del> func (hs *ProxierHealthServer) Run() { </del> <ins> func (hs *proxierHealthServer) Run() { </ins> serveMux := http.NewServeMux() serveMux.Handle("/healthz", healthzHandler{hs: hs}) server := hs.httpFactory.New(hs.addr, serveMux)


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: , from whatever combination of inputs we stuck into it, segfaulted. The segfault, which originates from a slightly modified fork of v1.17, looks as such: Translated to master, where the problem appears to be present, the segfault originates from this call: And that is called from here: This seems odd, because The only pointer involved in the segfaulting call is itself The caller nil-checks before calling a method on . I did some digging and found , which is a fix for this problem. However, this fix appears to be undone in . There doesn't seem to be a reason for this to have been changed in the latter PR. What you expected to happen: should not segfault from any bizarre combination of user input. If the situation is invalid, it should fail gracefully. How to reproduce it (as minimally and precisely as possible): Honestly, I'm not 100% sure what combination of inputs we crammed into , but I can locate them and post them if necessary. Anything else we need to know?: I don't feel confident opening a PR to fix this myself, but in addition to recreating the fix from , a comment should be explaining why there is this weird, seemingly pointless layer of indirection when assigning to the interface, so that this doesn't get un-fixed again.
Maybe decompose the above statement into multiple lines which should give us more idea which field was nil.
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
can't be nil, as it is of type , which is not an interface or pointer is an interface. However, when built from the exported constructor, , it is set to a literal , which is likewise not a pointer Therefore, the only remaining field is itself. Plus, I don't quite recall my go segfault stack trace syntax, but I'm relatively sure that indicates that the first argument to the method is , nil, and that the first argument to methods is the object being called on.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
When config.HealthzBindAddress is empty, we would encounter the scenario depicted in the stack trace. I have proposed a fix.
/assign
That commit, , is part of the PR that I believe introduced the problem. I strongly believe the problem to be one of a non-nil interface with a nil receiver. This is the only way that could be nil in the snippet of code from . Before PR (the changes from which are present in the version I ran), there was this odd looking set of code: The variable healthzServer, which is a pointer of type , was assigned to the variable , which is an interface of type . Then, later on, the variable was passed to , as an argument of type . This looks unexpected, and it appears that the variable is extraneous. After the changes in PR , the variable was removed, and was passed directly to as the argument for interface type . The former set of changes, with the intermediate variable, are meant to avoid the situation of a non-nil interface with a nil value. If the variable is never set in the conditional statement, then it is a nil interface. If is passed directly, then the result is the argument to is an an interface who's value is and who's concrete type is . This means that is called with as the receiver. I strongly believe this to be the root cause of the issue, but I do not know enough about kubernetes to open a PR with the correct fix, which would be to reinstate the variable (after the renaming in , this would be a variable of type ) to use as a go-between, to once again avoid this situation of a non-nil interface with a nil receiver. I believe that the removal of the variable was out of a misunderstanding of its purpose. If my assessment of the problem is incorrect, please let me know so I can understand where my logic is wrong. It is very possible that I am misunderstanding the phenomenon of a non-nil interface with a nil receiver. I would open a PR myself, but I do not know enough about Kubernetes to be confident in my ability to correctly fix and test this problem.
/remove-triage unresolved

} type healthzHandler struct { <del> hs *ProxierHealthServer </del> <ins> hs *proxierHealthServer </ins> } func (h healthzHandler) ServeHTTP(resp http.ResponseWriter, req *http.Request) {


Question:
Seen in the KIND e2e test: a) maybe we shouldn't log this more than 1x per second or something? b) maybe the error could make it clearer that the problem is the user's data? c) maybe we could fix the test triggering this? /sig api-machinery /wg apply Logs that I saw this in:
The 'duplicate entries for key' error came from validatingObjectWalker#visitListItems.
/assign
Yeah, they have multiple environment variables with the same name. I don't think that should be an error though ...
/cc
At the same time, it's not really useful to have multiple times the same variable and is probably a recipe for trouble.
It's definitely user error. Spamming apiserver logs about it doesn't help things, though.
I don't want to hide it, it's really not supposed to happen :-) it's not really user-data though. The fact that the error triggers here rather than in the actual validation means that there is a discrepancy between our validation and the regular validation. That's probably a problem that we have to fix. In that specific example, we can see that Kubernetes allows multiple environment variable with the same name, while we have assumed that it's not OK. It's a very legitimate error and we have to think about it to fix it. I think the test is relevant, and people out there are doing that. We could consider doing ratcheting if we want to stop allowing multiple variables, but I don't think that's how we want to fix that problem.
For history: should fix that.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
We've definitely improved that, maybe let's close it for now.
/reopen We still see this happening. Kube users get nervous, I would too. Please fix the root cause.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

// trigger a new deployment framework.Logf("%02d: triggering a new rollout for deployment %q", i, deployment.Name) deployment, err = e2edeployment.UpdateDeploymentWithRetries(c, ns, deployment.Name, func(update *appsv1.Deployment) { <del> newEnv := v1.EnvVar{Name: "A", Value: fmt.Sprintf("%d", i)} </del> <ins> newEnv := v1.EnvVar{Name: fmt.Sprintf("A%d", i), Value: fmt.Sprintf("%d", i)} </ins> update.Spec.Template.Spec.Containers[0].Env = append(update.Spec.Template.Spec.Containers[0].Env, newEnv) randomScale(update, i) })


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: when i installed k8s using kubeadm and my custom config like then output is It is a bug that the default value(Node,RBAC) is entered but a warning message is displayed. A function must be to compare two string slices. What you expected to happen: If the default value and custom value are the same, no warning message should be displayed. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 17.3 Install tools: kubeadm /area kubeadm /feature bug /sig cluster-lifecycle
/lifecycle active

// only return the user provided mode if at least one was valid if len(mode) > 0 { <del> klog.Warningf("the default kube-apiserver authorization-mode is %q; using %q", strings.Join(defaultMode, ","), strings.Join(mode, ","), ) </del> <ins> if !compareAuthzModes(defaultMode, mode) { klog.Warningf("the default kube-apiserver authorization-mode is %q; using %q", strings.Join(defaultMode, ","), strings.Join(mode, ","), ) } </ins> return strings.Join(mode, ",") } } return strings.Join(defaultMode, ",") } <ins> // compareAuthzModes compares two given authz modes and returns false if they do not match func compareAuthzModes(a, b []string) bool { if len(a) != len(b) { return false } for i, m := range a { if m != b[i] { return false } } return true } </ins> func isValidAuthzMode(authzMode string) bool { allModes := []string{ kubeadmconstants.ModeNode,


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: when i installed k8s using kubeadm and my custom config like then output is It is a bug that the default value(Node,RBAC) is entered but a warning message is displayed. A function must be to compare two string slices. What you expected to happen: If the default value and custom value are the same, no warning message should be displayed. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 17.3 Install tools: kubeadm /area kubeadm /feature bug /sig cluster-lifecycle
/lifecycle active

}) } } <ins> func TestCompareAuthzModes(t *testing.T) { var tests = []struct { name   string modesA []string modesB []string equal  bool }{ { name:   "modes match", modesA: []string{"a", "b", "c"}, modesB: []string{"a", "b", "c"}, equal:  true, }, { name:   "modes order does not match", modesA: []string{"a", "c", "b"}, modesB: []string{"a", "b", "c"}, }, { name:   "modes do not match; A has less modes", modesA: []string{"a", "b"}, modesB: []string{"a", "b", "c"}, }, { name:   "modes do not match; B has less modes", modesA: []string{"a", "b", "c"}, modesB: []string{"a", "b"}, }, } for _, rt := range tests { t.Run(rt.name, func(t *testing.T) { equal := compareAuthzModes(rt.modesA, rt.modesB) if equal != rt.equal { t.Errorf("failed compareAuthzModes:nexpected:n%vnsaw:n%v", rt.equal, equal) } }) } } </ins>


Question:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestStressingCascadingDeletion Testgrid link: Reason for failure: Maybe related: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig api-machinery
I'm seeing lots of these.
post-submit failures (all of these are flakes):
first failure seen in postsubmit is , part of ... preceeding many runs looked clean
I verified the delete options as seen by the server are identical before and after , so I don't think it is a functional change related to that PR
notably, many other tests in this suite started flaking on 3/7 and after ()
figured it out, dropped the detection of empty deletion options which would trigger the test to wait for all pods to be observed by the GC controller

} orphan := false switch { <ins> case options.OrphanDependents == nil && options.PropagationPolicy == nil && len(initialFinalizers) == 0: // if there are no deletion options, the default policy for replication controllers is orphan orphan = true </ins> case options.OrphanDependents != nil: // if the deletion options explicitly specify whether to orphan, that controls orphan = *options.OrphanDependents


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If gid is configured in Pod's securityContext.fsGroup, csi inline will panic. log: E0320 16:50:59. ] Operation for """ ("-ae1b-4b5c-be91-")" failed. No retries permitted until 2020-03-20 16:51:31. +0800 CST m=+. (durationBeforeRetry 32s). Error: "recovered from panic "runtime error: invalid memory address or nil pointer dereference". (err=<nil) Call stack:ngoroutine [running]:(0xc001497e70)n +0xb5npanic(0x3b713e0, 0x7bd2050)n +0x1b5(csiMountMgr).applyFSGroup(0xc0005372c0, 0xc001a2f100, 0x4, 0xc00205ed70, 0x44, 0x0)n +0x64(csiMountMgr).SetUpAt(0xc0005372c0, 0xc000b6cb80, 0x78, 0xc00205ed70, 0x0, 0x0, 0x0)n +0x1180(csiMountMgr).SetUp(0xc0005372c0, 0xc00205ed70, 0x0, 0x0, 0x0)n +0x4d(operationGenerator).GenerateMountVolumeFunc.func1(0x8, 0x453f9f8, 0xc001497e70, 0xc0025ba750)n +0x4a4(GeneratedOperations).Run(0xc0009742a0, 0x452f628, 0xc000d56480, 0xc00135a050, 0x45)n +0xaf(nestedPendingOperations).Run.func1(0xc000d56480, 0xc00135a050, 0x45, 0xc001106d50, 0x24, 0xc0009742a0, 0x0, 0x0, 0x0, 0x0)n +0xcbncreated by (*nestedPendingOperations).Runn +0x378n" code: If using inline volume, c.spec.PersistentVolume will be nil What you expected to happen: fsGroup is successfully applied Environment: Kubernetes version (use ): 1.16.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig sig/storage
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig/storage Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
ran into this bug recently on v1.18, and I don't see anything that would fix this issue on later releases either. /reopen /lifecycle freeze /assign
Reopened this issue. <detailsIn response to : ran into this bug recently on v1.18, and I don't see anything that would fix this issue on later releases either. /reopen /lifecycle freeze /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/lifecycle frozen
I tested this scenario with the hostpath driver on k8s 1.24 (built from master) on a local VM using a slightly modified version of the examples/csi-app- spec ( fsGroup and fsType): And this does not panic kubelet It does not panic because of this check: Which was in However, I do believe we can and should support fsgroup for CSI inline volumes. Instead of checking the PV spec (which is not set for inline volumes) we could pass the directly into , which will be RWO for inline volumes.
Sorry, it was not obvious that the panic was already fixed when I first looked at this. Re-purposing this bug to address the issue that fsgroup still does not get applied for inline volumes. /retitle CSI inline volumes do not apply fsGroup
THX, also link to this
/retitle CSI inline volumes should support fsGroup

return false } <del> if c.spec.PersistentVolume == nil { klog.V(4).Info(log("mounter.SetupAt Warning: skipping fsGroup permission change, no access mode available. The volume may only be accessible to root users.")) return false } if c.spec.PersistentVolume.Spec.AccessModes == nil { klog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, access modes not provided")) return false } if !hasReadWriteOnce(c.spec.PersistentVolume.Spec.AccessModes) { klog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, only support ReadWriteOnce access mode")) return false </del> <ins> if c.spec.PersistentVolume != nil { if c.spec.PersistentVolume.Spec.AccessModes == nil { klog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, access modes not provided")) return false } if !hasReadWriteOnce(c.spec.PersistentVolume.Spec.AccessModes) { klog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, only support ReadWriteOnce access mode")) return false } return true } else if c.spec.Volume != nil && c.spec.Volume.CSI != nil { if !utilfeature.DefaultFeatureGate.Enabled(features.CSIInlineVolume) { klog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, CSIInlineVolume feature required")) return false } // Inline CSI volumes are always mounted with RWO AccessMode by SetUpAt return true </ins> } <del> return true </del> <ins> klog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, unsupported volume type")) return false </ins> } // isDirMounted returns the !notMounted result from IsLikelyNotMountPoint check


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If gid is configured in Pod's securityContext.fsGroup, csi inline will panic. log: E0320 16:50:59. ] Operation for """ ("-ae1b-4b5c-be91-")" failed. No retries permitted until 2020-03-20 16:51:31. +0800 CST m=+. (durationBeforeRetry 32s). Error: "recovered from panic "runtime error: invalid memory address or nil pointer dereference". (err=<nil) Call stack:ngoroutine [running]:(0xc001497e70)n +0xb5npanic(0x3b713e0, 0x7bd2050)n +0x1b5(csiMountMgr).applyFSGroup(0xc0005372c0, 0xc001a2f100, 0x4, 0xc00205ed70, 0x44, 0x0)n +0x64(csiMountMgr).SetUpAt(0xc0005372c0, 0xc000b6cb80, 0x78, 0xc00205ed70, 0x0, 0x0, 0x0)n +0x1180(csiMountMgr).SetUp(0xc0005372c0, 0xc00205ed70, 0x0, 0x0, 0x0)n +0x4d(operationGenerator).GenerateMountVolumeFunc.func1(0x8, 0x453f9f8, 0xc001497e70, 0xc0025ba750)n +0x4a4(GeneratedOperations).Run(0xc0009742a0, 0x452f628, 0xc000d56480, 0xc00135a050, 0x45)n +0xaf(nestedPendingOperations).Run.func1(0xc000d56480, 0xc00135a050, 0x45, 0xc001106d50, 0x24, 0xc0009742a0, 0x0, 0x0, 0x0, 0x0)n +0xcbncreated by (*nestedPendingOperations).Runn +0x378n" code: If using inline volume, c.spec.PersistentVolume will be nil What you expected to happen: fsGroup is successfully applied Environment: Kubernetes version (use ): 1.16.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig sig/storage
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig/storage Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
ran into this bug recently on v1.18, and I don't see anything that would fix this issue on later releases either. /reopen /lifecycle freeze /assign
Reopened this issue. <detailsIn response to : ran into this bug recently on v1.18, and I don't see anything that would fix this issue on later releases either. /reopen /lifecycle freeze /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/lifecycle frozen
I tested this scenario with the hostpath driver on k8s 1.24 (built from master) on a local VM using a slightly modified version of the examples/csi-app- spec ( fsGroup and fsType): And this does not panic kubelet It does not panic because of this check: Which was in However, I do believe we can and should support fsgroup for CSI inline volumes. Instead of checking the PV spec (which is not set for inline volumes) we could pass the directly into , which will be RWO for inline volumes.
Sorry, it was not obvious that the panic was already fixed when I first looked at this. Re-purposing this bug to address the issue that fsgroup still does not get applied for inline volumes. /retitle CSI inline volumes do not apply fsGroup
THX, also link to this
/retitle CSI inline volumes should support fsGroup

}, want: true, }, <ins> { name: "driverPolicy is ReadWriteOnceWithFSTypeFSGroupPolicy with CSI inline volume", args: args{ fsGroup:      new(int64), fsType:       "ext4", driverPolicy: storage.ReadWriteOnceWithFSTypeFSGroupPolicy, }, fields: fields{ spec: volume.NewSpecFromVolume(&api.Volume{ VolumeSource: api.VolumeSource{ CSI: &api.CSIVolumeSource{ Driver: testDriver, }, }, }), }, want: true, }, </ins> } for _, tt := range tests {


Question:
After reading up on some corner cases with etcd client behavior (connections, streams, etc.) I came across the following option that has been introduced in etcd a while ago: From the official etcd docs on client behavior: Source: I do not see this option being used in the API server when is . Is there code in the API server that deals with these cases (etcd/API server node paritioned) and disconnects clients (REST, SDK) appropriately? Or are we at risk of having dangling (hanging) consumer-side client watches due to not using this option in the etcd client? Related discussions: /sig api-machinery cc/
was this something that was considered already?
We should be using for watches. (see also the ). It should just need to be . It would also be valuable to confirm that the behavior works as expected both when establishing watches and when a etcd member is partitioned after the watch has been established.
If anyone wants to own fixing this please self assign, else feel free to assign to me.
For this method: It seems we should do:
Thanks! This is a good question. I am not exactly sure what is the semantics on kubernetes watch. It is correct that without , etcd server will not cancel watch stream when itself is isolated from the cluster. However, when this happens, other requests to etcd (including health checks on etcd) will fail. So the control plane is already in a bad state and is likely to be replaced / repaired either automatically or manually. IIUC, this means apiserver will restart, the its clients will restart their watches.
Oh I just saw Joe's earlier comment. If using aligns with k8s watch semantics, we should use it:)
i'd support moving to 3.4.5 to pick this up!
I am happy to contribute but want to do some more testing and validation on the current behavior of and with the proposed fix. Please give me some days.
Quick update: after some quick tests it seems that does not solve the issue with an active after the connected etcd endpoint becomes partitioned. I think we have the following scenarios to consider: 1) Single node API server/etcd 2) Quorum-based cluster (e.g. 3 nodes) where K8s control plane is colocated with the etcd nodes (e.g. one API server and one etcd instance on one node) 3) Quorum-based cluster (e.g. 3 nodes) where K8s control plane is not colocated with the etcd endpoints (separate nodes for K8s control plane and etcd nodes) We can rule out (1) because technically a partition with just one node in the cluster (K8s control plane and etcd) cannot happen. (3) is mostly addressed with existing gRPC timeouts and keep-alives so when an API server connected to an etcd node that gets partitioned this API server will reconnect to other reachable etcd nodes (endpoints). However, this is only true if the API server itself is not within that partition (e.g. multi-zone deployment where a whole zone becomes partitioned; read on for my observations). (2) is the interesting one: an API server is connected to an etcd node and both become partitioned from the majority. But the API server has active watches against this etcd node and serves these (Kubernetes) watches to Kubernetes clients (controllers, etc.). I was expecting that the gets evaluated even after establishing the . But it seems my assumption is not correct, i.e. it is only evaluated (checked) when is called. At least I did not see an existing being dropped after partitioning the nodes from the quorum (I did check the underlying gRPC transport status and it remained in stayed for as long as the partition was not resolved. So what does that mean for the issue raised here? One could argue that if more than just one etcd node is partitioned, i.e. also the API server and perhaps other Kubernetes controllers running in that partitioned environment (e.g. a zone), it makes no difference using because: the clients (custom controllers, etc.) are also partitioned and thus cannot connect to any other API server anyways; would be nice to return an error here instead of hanging but the system won't self-heal during the partition even if the clients (custom controllers) would not be in that partition, the corresponding API server could not be reached anymore and client-go session handling should address this (e.g. reconnect to another API server) you write: I did not see an existing watch being dropped, e.g. using this simple code: Once I partition the etcd node including this client it will not drop the . Looking at the definition of it seems that this is performed only during establishing the but not reevaluated as long as the client has a successful connection to the etcd node (e.g. node is partitioned but not dead). Since I am by no means an expert on the etcd semantics here, I might be doing something wrong. It would be great to get some clarification whether this issue (I raised, I know :) ...) needs to be addressed, documented, or can actually be addressed at all with the current implementation of .
Please see analysis above. should this work when the etcd + apiserver are on the same node or not?
Thanks for the careful analysis My take on this is: For case (2) where the apiserver and etcd are co-located (and the apiserver only talks to that etcd), I don't see any reason why dropping the watch would help. In this type of setup, when etcd becomes partitioned, the co-located apiserver becomes unavailable and apiserver clients should route away from that apiserver entirely until the partition is healed. Even if the apiserver were to drop the watch, it is still unavailable. For case (3) the behavior is already better than I expected. optimizes the case slightly by ensuring the initial watch connection is against a non-partitioned etcd, so it's still worth enabling, but not urgently needed since failover is already in place. We should document in etcd that only applies to establishing the initial watch connection, the documentation is currently misleading.
It is correct that without WithRequireLeader, etcd server will not cancel watch stream when itself is isolated from the cluster This is correct. Edit: Sorry I missread the test. IIUC, this is not correct. After receiving the watch request with , etcd server will continue to monitor its leader for this watch stream. When there is leader lost observed for "maxNoLeaderCnt election timeouts", server will cancel the watch stream.
Thanks that means the documentation is correct then. So while we haven't found any pressing issues that would fix, it does do what it should do, enabling it in the kube-apiserver won't make anything worse, and might make failover slightly faster?
Make sense to me.
BTW, I just did a quick test. an etcd cluster of 3 members: A, B and C. member B and C some time, the etcdctl command prints
Sorry for my delayed reply, wanted to do some more tests before replying. I can confirm this scenario. However, the tests that I did and which caught my attention because it was not what I expected from reading the doc string was to not shut down the peers but isolate one peer, i.e. simulate a partition which technically is also a "lost leader" scenario. In this case I saw the watches not being closed (neither for nor programmatically). Not sure if this is a bug or by design but I think it's more likely to have partitions than loosing the whole cluster where a majority of nodes is terminated/crashed (stop-failure). If you think it's worth and not expected behavior I am happy to open an issue in the etcd repo and provide more details. Even though in etcd makes it sound should address the partition scenario. Note: I'm currently reviewing the network partition tests in etcd which state that the results I'm seeing are not expected. Update: looks like I'm experiencing what was reported a while ago in but it's unclear to me whether it's fixed, unfixable or what the workaround (from K8s perspective) would be.
I'm in favor of you opening a new against etcd for the watch not being closed in the lost leader scenario (and linking to this discussion). If it turns out it is a duplicate we can always mark as such.
has some detailed watch behaviors.
FYI. There was a bug in . The fix was backported to = v3.3.19 and v3.4.5
Thank you and for your feedback. After reviewing the docs and I think the issue with and how to deal with partitions is still open (see TODO below). The scenario where clients (Watchers or any client indirectly consuming from an etcd ) would hang (forever?) is when the client gets partitioned with the corresponding partitioned etcd node OR when a whole zone/dc is partitioned. Since etcd picks safety (consistency) over liveness (availability) in this case at least not data corruption occurs but the overall system would be stuck when using watches instead of PUT/GET RPCs with timeouts. It would be ideal to throw a useful error, e.g. "lost leader", instead of hanging in "gRPC TRANSIENT_ERROR" state (which is what I'm seeing). Until this is resolved in etcd, perhaps the docs could be updated as is a bit misleading here. I will comment on the etcd issue above and file a PR for the K8s watcher implementation to make use of as it is still a useful option when (re)establishing the stream. Depending on the environment, API server instances could have a health/liveness probe (or systemd equivalent) which could restart the process in case of a partition. Upon reconnect to the etcd node with the option it would immediately fail and error out accordingly.

// The filter doesn't filter out any object. wc.internalPred = storage.Everything } <del> wc.ctx, wc.cancel = context.WithCancel(ctx) </del> <ins> // The etcd server waits until it cannot find a leader for 3 election // timeouts to cancel existing streams. 3 is currently a hard coded // constant. The election timeout defaults to 1000ms. If the cluster is // healthy, when the leader is stopped, the leadership transfer should be // smooth. (leader transfers its leadership before stopping). If leader is // hard killed, other servers will take an election timeout to realize // leader lost and start campaign. wc.ctx, wc.cancel = context.WithCancel(clientv3.WithRequireLeader(ctx)) </ins> return wc }


Question:
Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error. I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO. However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).
FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks

github.com/lpabon/godbc v0.1.1 // indirect github.com/magiconair/properties v1.8.1 // indirect github.com/miekg/dns v1.1.4 <del> github.com/moby/ipvs v1.0.0 </del> <ins> github.com/moby/ipvs v1.0.1 </ins> github.com/mohae/deepcopy v0.0.0-20170603005431-491d3605edfb // indirect github.com/mrunalp/fileutils v0.0.0-20171103030105-7d4729fb3618 // indirect github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822


Question:
Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error. I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO. However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).
FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks

github.com/mitchellh/go-homedir => github.com/mitchellh/go-homedir v1.1.0 github.com/mitchellh/go-wordwrap => github.com/mitchellh/go-wordwrap v1.0.0 github.com/mitchellh/mapstructure => github.com/mitchellh/mapstructure v1.1.2 <del> github.com/moby/ipvs => github.com/moby/ipvs v1.0.0 </del> <ins> github.com/moby/ipvs => github.com/moby/ipvs v1.0.1 </ins> github.com/moby/term => github.com/moby/term v0.0.0-20200312100748-672ec06f55cd github.com/modern-go/concurrent => github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd github.com/modern-go/reflect2 => github.com/modern-go/reflect2 v1.0.1


Question:
Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error. I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO. However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).
FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks

github.com/mitchellh/go-wordwrap v1.0.0/go.mod h1:ZXFpozHsX6DPmq2I0TCekCxypsnAUbP2oI0UX1GXzOo= github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= <del> github.com/moby/ipvs v1.0.0 h1:89i8bPaL2DC0cjyRDv0QQOYUOU4fujziJmhF4ca/mtY= github.com/moby/ipvs v1.0.0/go.mod h1:2pngiyseZbIKXNv7hsKj3O9UEz30c53MT9005gt2hxQ= </del> <ins> github.com/moby/ipvs v1.0.1 h1:aoZ7fhLTXgDbzVrAnvV+XbKOU8kOET7B3+xULDF/1o0= github.com/moby/ipvs v1.0.1/go.mod h1:2pngiyseZbIKXNv7hsKj3O9UEz30c53MT9005gt2hxQ= </ins> github.com/moby/term v0.0.0-20200312100748-672ec06f55cd h1:aY7OQNf2XqY/JQ6qREWamhI/81os/agb2BAGpcx5yWI= github.com/moby/term v0.0.0-20200312100748-672ec06f55cd/go.mod h1:DdlQx2hp0Ss5/fLikoLlEeIYiATotOjgB//nb973jeo= github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=