Question:
Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error. I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO. However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).
FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks

import ( "bytes" "encoding/binary" <ins> "errors" </ins> "fmt" "net" "os/exec"


Question:
Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error. I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO. However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).
FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks

d.ActiveConnections = int(native.Uint16(attr.Value)) case ipvsDestAttrInactiveConnections: d.InactiveConnections = int(native.Uint16(attr.Value)) <del> case ipvsSvcAttrStats: </del> <ins> case ipvsDestAttrStats: </ins> stats, err := assembleStats(attr.Value) if err != nil { return nil, err


Question:
Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error. I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO. However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).
FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks

} } <ins> // in older kernels (< 3.18), the destination address family attribute doesn't exist so we must // assume it based on the destination address provided. if d.AddressFamily == 0 { // we can't check the address family using net stdlib because netlink returns // IPv4 addresses as the first 4 bytes in a []byte of length 16 where as // stdlib expects it as the last 4 bytes. addressFamily, err := getIPFamily(addressBytes) if err != nil { return nil, err } d.AddressFamily = addressFamily } </ins> // parse Address after parse AddressFamily incase of parseIP error if addressBytes != nil { ip, err := parseIP(addressBytes, d.AddressFamily)


Question:
Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error. I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO. However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).
FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks

return &d, nil } <ins> // getIPFamily parses the IP family based on raw data from netlink. // For AF_INET, netlink will set the first 4 bytes with trailing zeros //   10.0.0.1 -> [10 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0] // For AF_INET6, the full 16 byte array is used: //   2001:db8:3c4d:15::1a00 -> [32 1 13 184 60 77 0 21 0 0 0 0 0 0 26 0] func getIPFamily(address []byte) (uint16, error) { if len(address) == 4 { return syscall.AF_INET, nil } if isZeros(address) { return 0, errors.New("could not parse IP family from address data") } // assume IPv4 if first 4 bytes are non-zero but rest of the data is trailing zeros if !isZeros(address[:4]) && isZeros(address[4:]) { return syscall.AF_INET, nil } return syscall.AF_INET6, nil } func isZeros(b []byte) bool { for i := 0; i < len(b); i++ { if b[i] != 0 { return false } } return true } </ins> // parseDestination given a ipvs netlink response this function will respond with a valid destination entry, an error otherwise func (i *Handle) parseDestination(msg []byte) (*Destination, error) { var dst *Destination


Question:
Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error. I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO. However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).
FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks

github.com/mitchellh/go-wordwrap # github.com/mitchellh/mapstructure v1.1.2 => github.com/mitchellh/mapstructure v1.1.2 github.com/mitchellh/mapstructure <del> # github.com/moby/ipvs v1.0.0 => github.com/moby/ipvs v1.0.0 </del> <ins> # github.com/moby/ipvs v1.0.1 => github.com/moby/ipvs v1.0.1 </ins> github.com/moby/ipvs # github.com/moby/term v0.0.0-20200312100748-672ec06f55cd => github.com/moby/term v0.0.0-20200312100748-672ec06f55cd github.com/moby/term


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestSubresourcePatch Testgrid link: Reason for failure: Unknown. A diff with the original object would be helpful. Anything else we need to know: /sig api-machinery
OK Trying to reproduce, see what's going on.
Clearly this is because SSA is updating the tracked entry's time. It happens when the two request don't happen in the same second. Having a between the two requests makes this into a definitive error.
/wg apply
This seems to be because of . A repeated patch like will always trigger a new managedFields entry to be created, since it's always going to change the object (status or spec will be wiped depending on the endpoint), so the patch will always cause the object to change, which will always cause the entry to be created.
/assign
/assign
/assign
Still seeing this, bumped into it over in
deflake in while the underlying issue is being worked on in

expectInt64(t, patchedNoxuInstance.UnstructuredContent(), 999, "status", "num") // .status.num should be 999 expectInt64(t, patchedNoxuInstance.UnstructuredContent(), 10, "spec", "num")    // .spec.num should remain 10 <del> rv, found, err := unstructured.NestedString(patchedNoxuInstance.UnstructuredContent(), "metadata", "resourceVersion") if err != nil { t.Fatal(err) } if !found { t.Fatalf("metadata.resourceVersion not found") } // this call waits for the resourceVersion to be reached in the cache before returning. // We need to do this because the patch gets its initial object from the storage, and the cache serves that. // If it is out of date, then our initial patch is applied to an old resource version, which conflicts // and then the updated object shows a conflicting diff, which permanently fails the patch. // This gives expected stability in the patch without retrying on an known number of conflicts below in the test. // See https://issue.k8s.io/42644 _, err = noxuResourceClient.Get(context.TODO(), "foo", metav1.GetOptions{ResourceVersion: patchedNoxuInstance.GetResourceVersion()}) if err != nil { t.Fatalf("unexpected error: %v", err) } </del> // no-op patch <del> t.Logf("Patching .status.num again to 999") patchedNoxuInstance, err = noxuResourceClient.Patch(context.TODO(), "foo", types.MergePatchType, patch, metav1.PatchOptions{}, "status") if err != nil { t.Fatalf("unexpected error: %v", err) </del> <ins> rv := "" found := false // TODO: remove this retry once http://issue.k8s.io/75564 is resolved, and expect the resourceVersion to remain unchanged 100% of the time. // server-side-apply incorrectly considers spec fields in patches submitted to /status when updating managedFields timestamps, so this patch is racy: // if it spans a 1-second boundary from the last write, server-side-apply updates the managedField timestamp and increments resourceVersion. for i := 0; i < 10; i++ { rv, found, err = unstructured.NestedString(patchedNoxuInstance.UnstructuredContent(), "metadata", "resourceVersion") if err != nil { t.Fatal(err) } if !found { t.Fatalf("metadata.resourceVersion not found") } t.Logf("Patching .status.num again to 999") patchedNoxuInstance, err = noxuResourceClient.Patch(context.TODO(), "foo", types.MergePatchType, patch, metav1.PatchOptions{}, "status") if err != nil { t.Fatalf("unexpected error: %v", err) } // make sure no-op patch does not increment resourceVersion expectInt64(t, patchedNoxuInstance.UnstructuredContent(), 999, "status", "num") expectInt64(t, patchedNoxuInstance.UnstructuredContent(), 10, "spec", "num") if patchedNoxuInstance.GetResourceVersion() == rv { break } t.Logf("resource version changed - retrying") </ins> } <del> // make sure no-op patch does not increment resourceVersion expectInt64(t, patchedNoxuInstance.UnstructuredContent(), 999, "status", "num") expectInt64(t, patchedNoxuInstance.UnstructuredContent(), 10, "spec", "num") </del> expectString(t, patchedNoxuInstance.UnstructuredContent(), rv, "metadata", "resourceVersion") // empty patch


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestSubresourcePatch Testgrid link: Reason for failure: Unknown. A diff with the original object would be helpful. Anything else we need to know: /sig api-machinery
OK Trying to reproduce, see what's going on.
Clearly this is because SSA is updating the tracked entry's time. It happens when the two request don't happen in the same second. Having a between the two requests makes this into a definitive error.
/wg apply
This seems to be because of . A repeated patch like will always trigger a new managedFields entry to be created, since it's always going to change the object (status or spec will be wiped depending on the endpoint), so the patch will always cause the object to change, which will always cause the entry to be created.
/assign
/assign
/assign
Still seeing this, bumped into it over in
deflake in while the underlying issue is being worked on in

t.Fatalf("metadata.resourceVersion not found") } <del> // this call waits for the resourceVersion to be reached in the cache before returning. // We need to do this because the patch gets its initial object from the storage, and the cache serves that. // If it is out of date, then our initial patch is applied to an old resource version, which conflicts // and then the updated object shows a conflicting diff, which permanently fails the patch. // This gives expected stability in the patch without retrying on an known number of conflicts below in the test. // See https://issue.k8s.io/42644 _, err = noxuResourceClient.Get(context.TODO(), "foo", metav1.GetOptions{ResourceVersion: patchedNoxuInstance.GetResourceVersion()}) if err != nil { t.Fatalf("unexpected error: %v", err) } </del> // Scale.Spec.Replicas = 7 but Scale.Status.Replicas should remain 0 gottenScale, err := scaleClient.Scales("not-the-default").Get(context.TODO(), groupResource, "foo", metav1.GetOptions{}) if err != nil {


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When creating a new cluster using , panics with SIGSEGV on first run: Note that after the crash the pod restarts, and the second run is successful: and everything seems to be working as expected. What you expected to happen: starts successfully the first time. How to reproduce it (as minimally and precisely as possible): Create a new cluster using , with in mode. Anything else we need to know?: Environment: Kubernetes version (use ): 1.18.0 Cloud provider or hardware configuration: Hetzner VPS OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 5.3.0-45-generic (Ubuntu's default HWE kernel) Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others: using CRI and mode Please let me know if you need any additional information.
/sig network
/assign
According to the panic output, the crashed code is For some reason, might be nil, we should always check it before using.
Do we understand WHY it is nil?
One more thing I've noticed while fiddling with the cluster is that if I remove the node and re-join it, starts just fine, so I'm guessing it has to do with the initial state of the server / ipvs.
That would be my guess too. And that sort of race is worth fixing :)
do you see anything in the logs in the form: ??
I think somewhere here we're failing to initialize the IPVS interface. We probably shouldn't ignore the error from also :grimacing:
Nope, I posted the (full) logs in the issue description.
Thanks, will look into it deeper
do you happen to have a start up script that loads the necessary IPVS modules or is it preloaded on your nodes? Wondering if there could be a race between when the modules get loaded and when proxy starts up. Either way we should log the error ().
Neither, but if you give me the module list I can try to load them before creating the cluster to see if I still get the panic.
For kernel <4.19: For = 4.19:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
I can confirm that, if I explicitly preload the modules, starts successfully the first time.
According to this log , we can confirm second check success, missing modules loaded after first failed check. Because missing modules loaded but not put to

} ipvsModules := utilipvs.GetRequiredIPVSModules(kernelVersion) <del> var bmods []string </del> <ins> var bmods, lmods []string </ins> // Find out loaded kernel modules. If this is a full static kernel it will try to verify if the module is compiled using /boot/config-KERNELVERSION modulesFile, err := os.Open("/proc/modules")


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When creating a new cluster using , panics with SIGSEGV on first run: Note that after the crash the pod restarts, and the second run is successful: and everything seems to be working as expected. What you expected to happen: starts successfully the first time. How to reproduce it (as minimally and precisely as possible): Create a new cluster using , with in mode. Anything else we need to know?: Environment: Kubernetes version (use ): 1.18.0 Cloud provider or hardware configuration: Hetzner VPS OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 5.3.0-45-generic (Ubuntu's default HWE kernel) Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others: using CRI and mode Please let me know if you need any additional information.
/sig network
/assign
According to the panic output, the crashed code is For some reason, might be nil, we should always check it before using.
Do we understand WHY it is nil?
One more thing I've noticed while fiddling with the cluster is that if I remove the node and re-join it, starts just fine, so I'm guessing it has to do with the initial state of the server / ipvs.
That would be my guess too. And that sort of race is worth fixing :)
do you see anything in the logs in the form: ??
I think somewhere here we're failing to initialize the IPVS interface. We probably shouldn't ignore the error from also :grimacing:
Nope, I posted the (full) logs in the issue description.
Thanks, will look into it deeper
do you happen to have a start up script that loads the necessary IPVS modules or is it preloaded on your nodes? Wondering if there could be a race between when the modules get loaded and when proxy starts up. Either way we should log the error ().
Neither, but if you give me the module list I can try to load them before creating the cluster to see if I still get the panic.
For kernel <4.19: For = 4.19:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
I can confirm that, if I explicitly preload the modules, starts successfully the first time.
According to this log , we can confirm second check success, missing modules loaded after first failed check. Because missing modules loaded but not put to

if err != nil { klog.Warningf("Failed to load kernel module %v with modprobe. "+ "You can ignore this message when kube-proxy is running inside container without mounting /lib/modules", module) <ins> } else { lmods = append(lmods, module) </ins> } } } <del> return append(mods, bmods...), nil </del> <ins> mods = append(mods, bmods...) mods = append(mods, lmods...) return mods, nil </ins> } // getFirstColumn reads all the content from r into memory and return a


Question:
Hi, I have a issue with a dualstack cluster for my Kubernetes Dashboard deployment. I opened a ticket here (), but I am informed that it is not related to Kubernetes Dashboard. What is the problem? Errors in kube-api: The service is created with the correct ipFamily (=IPv6). My endpoint is fully in IPv6. Indeed, my pod has one IP in IPv4 and another in IPv6. Do you have also the issue? Kubernetes version: 1.18 OS: RHEL7
/sig network /sig api-machinery
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think the following should fix the error you saw:
/remove-sig api-machinery

if pod == nil { return fmt.Errorf("pod is missing, skipping (%s/%s)", addr.TargetRef.Namespace, addr.TargetRef.Name) } <del> if pod.Status.PodIPs[0].IP != addr.IP { return fmt.Errorf("pod ip doesn't match endpoint ip, skipping: %s vs %s (%s/%s)", pod.Status.PodIPs[0].IP, addr.IP, addr.TargetRef.Namespace, addr.TargetRef.Name) </del> <ins> for _, podIP := range pod.Status.PodIPs { if podIP.IP == addr.IP { return nil } </ins> } <del> return nil </del> <ins> return fmt.Errorf("pod ip(s) doesn't match endpoint ip, skipping: %v vs %s (%s/%s)", pod.Status.PodIPs, addr.IP, addr.TargetRef.Namespace, addr.TargetRef.Name) </ins> } // This is O(N), but we expect haystack to be small;


Question:
Hi, I have a issue with a dualstack cluster for my Kubernetes Dashboard deployment. I opened a ticket here (), but I am informed that it is not related to Kubernetes Dashboard. What is the problem? Errors in kube-api: The service is created with the correct ipFamily (=IPv6). My endpoint is fully in IPv6. Indeed, my pod has one IP in IPv4 and another in IPv6. Do you have also the issue? Kubernetes version: 1.18 OS: RHEL7
/sig network /sig api-machinery
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think the following should fix the error you saw:
/remove-sig api-machinery

Ports:     []api.EndpointPort{{Name: "", Port: 80}, {Name: "p", Port: 93}}, }}, }, <ins> { ObjectMeta: metav1.ObjectMeta{ Name:      "foo-second-ip", Namespace: metav1.NamespaceDefault, }, Subsets: []api.EndpointSubset{{ Addresses: []api.EndpointAddress{{IP: "2001:db7::", TargetRef: &api.ObjectReference{Name: "foo", Namespace: metav1.NamespaceDefault}}}, Ports:     []api.EndpointPort{{Name: "", Port: 80}, {Name: "p", Port: 93}}, }}, }, </ins> }, } pods := &api.PodList{


Question:
Hi, I have a issue with a dualstack cluster for my Kubernetes Dashboard deployment. I opened a ticket here (), but I am informed that it is not related to Kubernetes Dashboard. What is the problem? Errors in kube-api: The service is created with the correct ipFamily (=IPv6). My endpoint is fully in IPv6. Indeed, my pod has one IP in IPv4 and another in IPv6. Do you have also the issue? Kubernetes version: 1.18 OS: RHEL7
/sig network /sig api-machinery
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think the following should fix the error you saw:
/remove-sig api-machinery

t.Errorf("Expected %v, but got %v", e, a) } <ins> // Test a simple id (using second ip). location, _, err = redirector.ResourceLocation(ctx, "foo-second-ip") if err != nil { t.Errorf("Unexpected error: %v", err) } if location == nil { t.Errorf("Unexpected nil: %v", location) } if e, a := "//[2001:db7::]:80", location.String(); e != a { t.Errorf("Expected %v, but got %v", e, a) } </ins> // Test a name + port. location, _, err = redirector.ResourceLocation(ctx, "foo:p") if err != nil {


Question:
Hi, I have a issue with a dualstack cluster for my Kubernetes Dashboard deployment. I opened a ticket here (), but I am informed that it is not related to Kubernetes Dashboard. What is the problem? Errors in kube-api: The service is created with the correct ipFamily (=IPv6). My endpoint is fully in IPv6. Indeed, my pod has one IP in IPv4 and another in IPv6. Do you have also the issue? Kubernetes version: 1.18 OS: RHEL7
/sig network /sig api-machinery
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think the following should fix the error you saw:
/remove-sig api-machinery

t.Errorf("Expected %v, but got %v", e, a) } <ins> // Test a name + port (using second ip). location, _, err = redirector.ResourceLocation(ctx, "foo-second-ip:p") if err != nil { t.Errorf("Unexpected error: %v", err) } if location == nil { t.Errorf("Unexpected nil: %v", location) } if e, a := "//[2001:db7::]:93", location.String(); e != a { t.Errorf("Expected %v, but got %v", e, a) } </ins> // Test a name + port number (service port 93 -> target port 80) location, _, err = redirector.ResourceLocation(ctx, "foo:93") if err != nil {


Question:
Hi, I have a issue with a dualstack cluster for my Kubernetes Dashboard deployment. I opened a ticket here (), but I am informed that it is not related to Kubernetes Dashboard. What is the problem? Errors in kube-api: The service is created with the correct ipFamily (=IPv6). My endpoint is fully in IPv6. Indeed, my pod has one IP in IPv4 and another in IPv6. Do you have also the issue? Kubernetes version: 1.18 OS: RHEL7
/sig network /sig api-machinery
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think the following should fix the error you saw:
/remove-sig api-machinery

t.Errorf("Expected %v, but got %v", e, a) } <ins> // Test a name + port number (service port 93 -> target port 80, using second ip) location, _, err = redirector.ResourceLocation(ctx, "foo-second-ip:93") if err != nil { t.Errorf("Unexpected error: %v", err) } if location == nil { t.Errorf("Unexpected nil: %v", location) } if e, a := "//[2001:db7::]:80", location.String(); e != a { t.Errorf("Expected %v, but got %v", e, a) } </ins> // Test a name + port number (service port 9393 -> target port "p" -> endpoint port 93) location, _, err = redirector.ResourceLocation(ctx, "foo:9393") if err != nil {


Question:
Hi, I have a issue with a dualstack cluster for my Kubernetes Dashboard deployment. I opened a ticket here (), but I am informed that it is not related to Kubernetes Dashboard. What is the problem? Errors in kube-api: The service is created with the correct ipFamily (=IPv6). My endpoint is fully in IPv6. Indeed, my pod has one IP in IPv4 and another in IPv6. Do you have also the issue? Kubernetes version: 1.18 OS: RHEL7
/sig network /sig api-machinery
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think the following should fix the error you saw:
/remove-sig api-machinery

t.Errorf("Expected %v, but got %v", e, a) } <ins> // Test a name + port number (service port 9393 -> target port "p" -> endpoint port 93, using second ip) location, _, err = redirector.ResourceLocation(ctx, "foo-second-ip:9393") if err != nil { t.Errorf("Unexpected error: %v", err) } if location == nil { t.Errorf("Unexpected nil: %v", location) } if e, a := "//[2001:db7::]:93", location.String(); e != a { t.Errorf("Expected %v, but got %v", e, a) } </ins> // Test a scheme + name + port. location, _, err = redirector.ResourceLocation(ctx, "https:foo:p") if err != nil {


Question:
Hi, I have a issue with a dualstack cluster for my Kubernetes Dashboard deployment. I opened a ticket here (), but I am informed that it is not related to Kubernetes Dashboard. What is the problem? Errors in kube-api: The service is created with the correct ipFamily (=IPv6). My endpoint is fully in IPv6. Indeed, my pod has one IP in IPv4 and another in IPv6. Do you have also the issue? Kubernetes version: 1.18 OS: RHEL7
/sig network /sig api-machinery
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think the following should fix the error you saw:
/remove-sig api-machinery

t.Errorf("Expected %v, but got %v", e, a) } <ins> // Test a scheme + name + port (using second ip). location, _, err = redirector.ResourceLocation(ctx, "https:foo-second-ip:p") if err != nil { t.Errorf("Unexpected error: %v", err) } if location == nil { t.Errorf("Unexpected nil: %v", location) } if e, a := "https://[2001:db7::]:93", location.String(); e != a { t.Errorf("Expected %v, but got %v", e, a) } </ins> // Test a non-existent name + port. location, _, err = redirector.ResourceLocation(ctx, "foo:q") if err == nil { t.Errorf("Unexpected nil error") } <ins> // Test a non-existent name + port (using second ip). location, _, err = redirector.ResourceLocation(ctx, "foo-second-ip:q") if err == nil { t.Errorf("Unexpected nil error") } </ins> // Test error path if _, _, err = redirector.ResourceLocation(ctx, "bar"); err == nil { t.Errorf("unexpected nil error")


Question:
What happened: In a dual-stack cluster when specifying both ipv6 and ipv4 addresses in the entries are inserted in ipvs for the ipv6 nodeports but with an invalid ipv4 address; The example shows the ipv4 loopback address but any ipv4 address in will cause a faulty entry. What you expected to happen: No ipv4 entry shall be inserted for the ipv6 nodeport. How to reproduce it (as minimally and precisely as possible): Configure with both ipv4 and ipv6 addresses Bring up a dual-stack cluster with proxy-mode Create a service with ipFamily:IPv6 Do Anything else we need to know?: Proxy-mode=iptables seems to work. With no ipv4 entries are inserted for ipv6 nodeports. In it is recommended to use but doing so makes kube-proxy enter rules for address (in addition to "127.0.0.1"). While these entries cause no harm they can't be used and should not be there. These entries are created for both proxy-mode "iptables" and "ipvs". This should not be handled in this issue but in another (new) issue if needed. Example; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Kvm's on Ubuntu OS (e.g: ): xcluster Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): calico/cni:v3.12.0 Others: /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/area ipvs /area ipv6
/remove-triage unresolved /assign I take this myself if no one objects.
The ipv4 address is obviously a ipv4-converted ipv6 from .
According to the code here (), we may need to filter the before passing it to .
Looks that that's it. Please go ahead and make a PR, or I can do it if you prefer. I will help test the PR of course. Here are some review comments right away; Use v1.IPFamily as param to the filter function rather than The filter function seem usable and generic and may be placed in some "util" package, but leave it as-is and let the maintainers decide that.
Hi feel free to send a PR please, as you have took this already :) I'd like to help review it!
The "addresses" are really CIDRs which make the fix even simpler.

serviceHealthServer := healthcheck.NewServiceHealthServer(hostname, recorder) isIPv6 := ipt.IsIPv6() <ins> var incorrectAddresses []string nodePortAddresses, incorrectAddresses = utilproxy.FilterIncorrectCIDRVersion(nodePortAddresses, isIPv6) if len(incorrectAddresses) > 0 { klog.Warning("NodePortAddresses of wrong family; ", incorrectAddresses) } </ins> proxier := &Proxier{ portsMap:                 make(map[utilproxy.LocalPort]utilproxy.Closeable), serviceMap:               make(proxy.ServiceMap),


Question:
What happened: In a dual-stack cluster when specifying both ipv6 and ipv4 addresses in the entries are inserted in ipvs for the ipv6 nodeports but with an invalid ipv4 address; The example shows the ipv4 loopback address but any ipv4 address in will cause a faulty entry. What you expected to happen: No ipv4 entry shall be inserted for the ipv6 nodeport. How to reproduce it (as minimally and precisely as possible): Configure with both ipv4 and ipv6 addresses Bring up a dual-stack cluster with proxy-mode Create a service with ipFamily:IPv6 Do Anything else we need to know?: Proxy-mode=iptables seems to work. With no ipv4 entries are inserted for ipv6 nodeports. In it is recommended to use but doing so makes kube-proxy enter rules for address (in addition to "127.0.0.1"). While these entries cause no harm they can't be used and should not be there. These entries are created for both proxy-mode "iptables" and "ipvs". This should not be handled in this issue but in another (new) issue if needed. Example; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Kvm's on Ubuntu OS (e.g: ): xcluster Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): calico/cni:v3.12.0 Others: /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/area ipvs /area ipv6
/remove-triage unresolved /assign I take this myself if no one objects.
The ipv4 address is obviously a ipv4-converted ipv6 from .
According to the code here (), we may need to filter the before passing it to .
Looks that that's it. Please go ahead and make a PR, or I can do it if you prefer. I will help test the PR of course. Here are some review comments right away; Use v1.IPFamily as param to the filter function rather than The filter function seem usable and generic and may be placed in some "util" package, but leave it as-is and let the maintainers decide that.
Hi feel free to send a PR please, as you have took this already :) I'd like to help review it!
The "addresses" are really CIDRs which make the fix even simpler.

nodePortAddresses []string, ) (proxy.Provider, error) { // Create an ipv4 instance of the single-stack proxier <ins> nodePortAddresses4, nodePortAddresses6 := utilproxy.FilterIncorrectCIDRVersion(nodePortAddresses, false) </ins> ipv4Proxier, err := NewProxier(ipt[0], sysctl, exec, syncPeriod, minSyncPeriod, masqueradeAll, masqueradeBit, localDetectors[0], hostname, <del> nodeIP[0], recorder, healthzServer, nodePortAddresses) </del> <ins> nodeIP[0], recorder, healthzServer, nodePortAddresses4) </ins> if err != nil { return nil, fmt.Errorf("unable to create ipv4 proxier: %v", err) } ipv6Proxier, err := NewProxier(ipt[1], sysctl, exec, syncPeriod, minSyncPeriod, masqueradeAll, masqueradeBit, localDetectors[1], hostname, <del> nodeIP[1], recorder, healthzServer, nodePortAddresses) </del> <ins> nodeIP[1], recorder, healthzServer, nodePortAddresses6) </ins> if err != nil { return nil, fmt.Errorf("unable to create ipv6 proxier: %v", err) }


Question:
What happened: In a dual-stack cluster when specifying both ipv6 and ipv4 addresses in the entries are inserted in ipvs for the ipv6 nodeports but with an invalid ipv4 address; The example shows the ipv4 loopback address but any ipv4 address in will cause a faulty entry. What you expected to happen: No ipv4 entry shall be inserted for the ipv6 nodeport. How to reproduce it (as minimally and precisely as possible): Configure with both ipv4 and ipv6 addresses Bring up a dual-stack cluster with proxy-mode Create a service with ipFamily:IPv6 Do Anything else we need to know?: Proxy-mode=iptables seems to work. With no ipv4 entries are inserted for ipv6 nodeports. In it is recommended to use but doing so makes kube-proxy enter rules for address (in addition to "127.0.0.1"). While these entries cause no harm they can't be used and should not be there. These entries are created for both proxy-mode "iptables" and "ipvs". This should not be handled in this issue but in another (new) issue if needed. Example; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Kvm's on Ubuntu OS (e.g: ): xcluster Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): calico/cni:v3.12.0 Others: /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/area ipvs /area ipv6
/remove-triage unresolved /assign I take this myself if no one objects.
The ipv4 address is obviously a ipv4-converted ipv6 from .
According to the code here (), we may need to filter the before passing it to .
Looks that that's it. Please go ahead and make a PR, or I can do it if you prefer. I will help test the PR of course. Here are some review comments right away; Use v1.IPFamily as param to the filter function rather than The filter function seem usable and generic and may be placed in some "util" package, but leave it as-is and let the maintainers decide that.
Hi feel free to send a PR please, as you have took this already :) I'd like to help review it!
The "addresses" are really CIDRs which make the fix even simpler.

endpointSlicesEnabled := utilfeature.DefaultFeatureGate.Enabled(features.EndpointSliceProxying) <ins> var incorrectAddresses []string nodePortAddresses, incorrectAddresses = utilproxy.FilterIncorrectCIDRVersion(nodePortAddresses, isIPv6) if len(incorrectAddresses) > 0 { klog.Warning("NodePortAddresses of wrong family; ", incorrectAddresses) } </ins> proxier := &Proxier{ portsMap:              make(map[utilproxy.LocalPort]utilproxy.Closeable), serviceMap:            make(proxy.ServiceMap),


Question:
What happened: In a dual-stack cluster when specifying both ipv6 and ipv4 addresses in the entries are inserted in ipvs for the ipv6 nodeports but with an invalid ipv4 address; The example shows the ipv4 loopback address but any ipv4 address in will cause a faulty entry. What you expected to happen: No ipv4 entry shall be inserted for the ipv6 nodeport. How to reproduce it (as minimally and precisely as possible): Configure with both ipv4 and ipv6 addresses Bring up a dual-stack cluster with proxy-mode Create a service with ipFamily:IPv6 Do Anything else we need to know?: Proxy-mode=iptables seems to work. With no ipv4 entries are inserted for ipv6 nodeports. In it is recommended to use but doing so makes kube-proxy enter rules for address (in addition to "127.0.0.1"). While these entries cause no harm they can't be used and should not be there. These entries are created for both proxy-mode "iptables" and "ipvs". This should not be handled in this issue but in another (new) issue if needed. Example; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Kvm's on Ubuntu OS (e.g: ): xcluster Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): calico/cni:v3.12.0 Others: /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/area ipvs /area ipv6
/remove-triage unresolved /assign I take this myself if no one objects.
The ipv4 address is obviously a ipv4-converted ipv6 from .
According to the code here (), we may need to filter the before passing it to .
Looks that that's it. Please go ahead and make a PR, or I can do it if you prefer. I will help test the PR of course. Here are some review comments right away; Use v1.IPFamily as param to the filter function rather than The filter function seem usable and generic and may be placed in some "util" package, but leave it as-is and let the maintainers decide that.
Hi feel free to send a PR please, as you have took this already :) I'd like to help review it!
The "addresses" are really CIDRs which make the fix even simpler.

safeIpset := newSafeIpset(ipset) <ins> nodePortAddresses4, nodePortAddresses6 := utilproxy.FilterIncorrectCIDRVersion(nodePortAddresses, false) </ins> // Create an ipv4 instance of the single-stack proxier ipv4Proxier, err := NewProxier(ipt[0], ipvs, safeIpset, sysctl, exec, syncPeriod, minSyncPeriod, filterCIDRs(false, excludeCIDRs), strictARP, tcpTimeout, tcpFinTimeout, udpTimeout, masqueradeAll, masqueradeBit, localDetectors[0], hostname, nodeIP[0], <del> recorder, healthzServer, scheduler, nodePortAddresses, kernelHandler) </del> <ins> recorder, healthzServer, scheduler, nodePortAddresses4, kernelHandler) </ins> if err != nil { return nil, fmt.Errorf("unable to create ipv4 proxier: %v", err) }


Question:
What happened: In a dual-stack cluster when specifying both ipv6 and ipv4 addresses in the entries are inserted in ipvs for the ipv6 nodeports but with an invalid ipv4 address; The example shows the ipv4 loopback address but any ipv4 address in will cause a faulty entry. What you expected to happen: No ipv4 entry shall be inserted for the ipv6 nodeport. How to reproduce it (as minimally and precisely as possible): Configure with both ipv4 and ipv6 addresses Bring up a dual-stack cluster with proxy-mode Create a service with ipFamily:IPv6 Do Anything else we need to know?: Proxy-mode=iptables seems to work. With no ipv4 entries are inserted for ipv6 nodeports. In it is recommended to use but doing so makes kube-proxy enter rules for address (in addition to "127.0.0.1"). While these entries cause no harm they can't be used and should not be there. These entries are created for both proxy-mode "iptables" and "ipvs". This should not be handled in this issue but in another (new) issue if needed. Example; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Kvm's on Ubuntu OS (e.g: ): xcluster Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): calico/cni:v3.12.0 Others: /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/area ipvs /area ipv6
/remove-triage unresolved /assign I take this myself if no one objects.
The ipv4 address is obviously a ipv4-converted ipv6 from .
According to the code here (), we may need to filter the before passing it to .
Looks that that's it. Please go ahead and make a PR, or I can do it if you prefer. I will help test the PR of course. Here are some review comments right away; Use v1.IPFamily as param to the filter function rather than The filter function seem usable and generic and may be placed in some "util" package, but leave it as-is and let the maintainers decide that.
Hi feel free to send a PR please, as you have took this already :) I'd like to help review it!
The "addresses" are really CIDRs which make the fix even simpler.

exec, syncPeriod, minSyncPeriod, filterCIDRs(true, excludeCIDRs), strictARP, tcpTimeout, tcpFinTimeout, udpTimeout, masqueradeAll, masqueradeBit, localDetectors[1], hostname, nodeIP[1], <del> nil, nil, scheduler, nodePortAddresses, kernelHandler) </del> <ins> nil, nil, scheduler, nodePortAddresses6, kernelHandler) </ins> if err != nil { return nil, fmt.Errorf("unable to create ipv6 proxier: %v", err) }


Question:
What happened: In handler#handleSummary, return value from request.Request.ParseForm() is ignored. What you expected to happen: The error return should be checked.
/sig node
Hi this seems to be easy to fix. Could I solve this?
Please go ahead.

} <ins> func TestFailedParseParamsSummaryHandler(t *testing.T) { fw := newServerTest() defer fw.testHTTPServer.Close() resp, err := http.Post(fw.testHTTPServer.URL+"/stats/summary", "invalid/content/type", nil) assert.NoError(t, err) defer resp.Body.Close() v, err := ioutil.ReadAll(resp.Body) assert.NoError(t, err) assert.Equal(t, http.StatusInternalServerError, resp.StatusCode) assert.Contains(t, string(v), "parse form failed") } </ins> func TestTrimURLPath(t *testing.T) { tests := []struct { path, expected string


Question:
What happened: In handler#handleSummary, return value from request.Request.ParseForm() is ignored. What you expected to happen: The error return should be checked.
/sig node
Hi this seems to be easy to fix. Could I solve this?
Please go ahead.

"//staging/src/k8s.io/component-base/metrics:go_default_library", "//vendor/github.com/emicklei/go-restful:go_default_library", "//vendor/github.com/google/cadvisor/info/v1:go_default_library", <ins> "//vendor/github.com/pkg/errors:go_default_library", </ins> "//vendor/k8s.io/klog:go_default_library", ], )


Question:
What happened: In handler#handleSummary, return value from request.Request.ParseForm() is ignored. What you expected to happen: The error return should be checked.
/sig node
Hi this seems to be easy to fix. Could I solve this?
Please go ahead.

restful "github.com/emicklei/go-restful" cadvisorapi "github.com/google/cadvisor/info/v1" <ins> "github.com/pkg/errors" </ins> "k8s.io/klog" "k8s.io/api/core/v1"


Question:
What happened: In handler#handleSummary, return value from request.Request.ParseForm() is ignored. What you expected to happen: The error return should be checked.
/sig node
Hi this seems to be easy to fix. Could I solve this?
Please go ahead.

// If "only_cpu_and_memory" GET param is true then only cpu and memory is returned in response. func (h *handler) handleSummary(request *restful.Request, response *restful.Response) { onlyCPUAndMemory := false <del> request.Request.ParseForm() </del> <ins> err := request.Request.ParseForm() if err != nil { handleError(response, "/stats/summary", errors.Wrapf(err, "parse form failed")) return } </ins> if onlyCluAndMemoryParam, found := request.Request.Form["only_cpu_and_memory"]; found && len(onlyCluAndMemoryParam) == 1 && onlyCluAndMemoryParam[0] == "true" { onlyCPUAndMemory = true } var summary *statsapi.Summary <del> var err error </del> if onlyCPUAndMemory { summary, err = h.summaryProvider.GetCPUAndMemoryStats() } else {


Question:
What would you like to be : Please see Implement Instances::NodeAddresses so it doesn't ignore the node input. Why is this needed: The current implementation assumes it will be only run on the Kubelet. That is no longer a valid assumption and in fact it the nearish future it will no longer be run on the Kubelet.
/sig cloud-provider /area provider/gcp
This issue is about the GCE implementation found at This problem cannot be fixed in the cloud node controller.
According to is indeed only used for the address of the calling instance. Maybe we should use instead? Note that implements
Or, we should definitely remove the if its TODO item is no longer true.
I believe the TODO is no longer true as there is also a call in

} // NodeAddresses is an implementation of Instances.NodeAddresses. <del> func (g *Cloud) NodeAddresses(_ context.Context, _ types.NodeName) ([]v1.NodeAddress, error) { internalIP, err := metadata.Get("instance/network-interfaces/0/ip") if err != nil { return nil, fmt.Errorf("couldn't get internal IP: %v", err) </del> <ins> func (g *Cloud) NodeAddresses(ctx context.Context, nodeName types.NodeName) ([]v1.NodeAddress, error) { timeoutCtx, cancel := context.WithTimeout(ctx, 1*time.Hour) defer cancel() instanceName := string(nodeName) if g.useMetadataServer { // Use metadata server if possible if g.isCurrentInstance(instanceName) { internalIP, err := metadata.Get("instance/network-interfaces/0/ip") if err != nil { return nil, fmt.Errorf("couldn't get internal IP: %v", err) } externalIP, err := metadata.Get("instance/network-interfaces/0/access-configs/0/external-ip") if err != nil { return nil, fmt.Errorf("couldn't get external IP: %v", err) } addresses := []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: internalIP}, {Type: v1.NodeExternalIP, Address: externalIP}, } if internalDNSFull, err := metadata.Get("instance/hostname"); err != nil { klog.Warningf("couldn't get full internal DNS name: %v", err) } else { addresses = append(addresses, v1.NodeAddress{Type: v1.NodeInternalDNS, Address: internalDNSFull}, v1.NodeAddress{Type: v1.NodeHostName, Address: internalDNSFull}, ) } return addresses, nil } </ins> } <del> externalIP, err := metadata.Get("instance/network-interfaces/0/access-configs/0/external-ip") </del> <ins> // Use GCE API instanceObj, err := g.getInstanceByName(instanceName) </ins> if err != nil { <del> return nil, fmt.Errorf("couldn't get external IP: %v", err) } addresses := []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: internalIP}, {Type: v1.NodeExternalIP, Address: externalIP}, </del> <ins> return nil, fmt.Errorf("couldn't get instance details: %v", err) </ins> } <del> if internalDNSFull, err := metadata.Get("instance/hostname"); err != nil { klog.Warningf("couldn't get full internal DNS name: %v", err) } else { addresses = append(addresses, v1.NodeAddress{Type: v1.NodeInternalDNS, Address: internalDNSFull}, v1.NodeAddress{Type: v1.NodeHostName, Address: internalDNSFull}, ) </del> <ins> instance, err := g.c.Instances().Get(timeoutCtx, meta.ZonalKey(canonicalizeInstanceName(instanceObj.Name), instanceObj.Zone)) if err != nil { return []v1.NodeAddress{}, fmt.Errorf("error while querying for instance: %v", err) </ins> } <del> return addresses, nil </del> <ins> return nodeAddressesFromInstance(instance) </ins> } // NodeAddressesByProviderID will not be called from the node that is requesting this ID.


Question:
What happened: In , there is mismatch between method name and the method in logs. e.g. RemoteRuntimeService#Attach has the following: What you expected to happen: The method names should be consistent with what appear in the logs. Otherwise people would be confused when troubleshooting. /sig node
could i try to fix this?Thanks
Yes. fixes method name for ContainerStats. You can cover the rest. thanks
/assign

if resp.Url == "" { errorMessage := "URL is not set" <del> klog.Errorf("Exec failed: %s", errorMessage) </del> <ins> klog.Errorf("Attach failed: %s", errorMessage) </ins> return nil, errors.New(errorMessage) } return resp, nil


Question:
What happened: In , there is mismatch between method name and the method in logs. e.g. RemoteRuntimeService#Attach has the following: What you expected to happen: The method names should be consistent with what appear in the logs. Otherwise people would be confused when troubleshooting. /sig node
could i try to fix this?Thanks
Yes. fixes method name for ContainerStats. You can cover the rest. thanks
/assign

if resp.Url == "" { errorMessage := "URL is not set" <del> klog.Errorf("Exec failed: %s", errorMessage) </del> <ins> klog.Errorf("PortForward failed: %s", errorMessage) </ins> return nil, errors.New(errorMessage) }


Question:
For external cloud provider - Kubelet setting the TaintExternalCloudProvider taint: The results in calling function in cloud node-controller: This (in a healthy case) sends 4 GCE API calls to: node addresses: (as part of getNodeModifier call) instance type: (as part of getNodeModifiers call) instance: (as part of updateNodeAddresses call) node addresses (again): (as part of updateNodeAddresses call) All those requests are read requests and could be replaced by a single call. We could do that by introducing a simple struct like: and adding a method (or rather replacing the existing InstanceTypeByProviderID which is used only for this purpose) to Instances cloud-provider interface; With that change, we would be able to replace all that 4 calls with just a single API call. This would be useful especially for creating large clusters (like 5k nodes), which may saturate the quota for API calls for quite some time. This is quite significant size of refactoring, though it should be relatively mechanical task to do. /sig cloud-provider /sig scalability /priority important-longterm /help-wanted - FYI
The InstanceMetadataByProviderID would be to every cloud provider, such as aws, azure, etc. right ?
/assign Hi , if you're interested in it, I can work with you.
Yes - we need to do that (which is why I wrote it's quite significant refectoring). It also makes reviews harder, because we will need to ask maintainers of all cloud-providers. So it would be good to separate changes to different cloud providers into separate commits.
I have one question: and first query by providerID, if failed then query by nodeName. While the logic in is converse(if providerID is nil, get providerID by nodeName, then ensure its existence by providerID). If we merge these three method into , how to deal with the query sequence of nodeName and providerID?
I bet it's a consequence of different people adding different code paths. Those conceptually seem to be the same to me. Unifying this will be additional benefit from this change imho. So I would say that choose whatever is easier to implement.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I think and are the only PRs left for this. One other optimization that can be made is to cache the instance metadata but I think that's a provider specific implementation
Plus enabling it for GCE if we decide to remove it from the first PR. But yeah, in general I agree.
- can you please address ?
Yes, I have seen that.
I think this issue has been resolved, feel free to reopen it. /close
Closing this issue. <detailsIn response to : I think this issue has been resolved, feel free to reopen it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

} // InstancesV2 returns an implementation of InstancesV2 for Google Compute Engine. <del> // TODO: implement ONLY for external cloud provider </del> <ins> // Implement ONLY for external cloud provider </ins> func (g *Cloud) InstancesV2() (cloudprovider.InstancesV2, bool) { <del> return nil, false </del> <ins> return g, true </ins> } // Zones returns an implementation of Zones for Google Compute Engine.


Question:
For external cloud provider - Kubelet setting the TaintExternalCloudProvider taint: The results in calling function in cloud node-controller: This (in a healthy case) sends 4 GCE API calls to: node addresses: (as part of getNodeModifier call) instance type: (as part of getNodeModifiers call) instance: (as part of updateNodeAddresses call) node addresses (again): (as part of updateNodeAddresses call) All those requests are read requests and could be replaced by a single call. We could do that by introducing a simple struct like: and adding a method (or rather replacing the existing InstanceTypeByProviderID which is used only for this purpose) to Instances cloud-provider interface; With that change, we would be able to replace all that 4 calls with just a single API call. This would be useful especially for creating large clusters (like 5k nodes), which may saturate the quota for API calls for quite some time. This is quite significant size of refactoring, though it should be relatively mechanical task to do. /sig cloud-provider /sig scalability /priority important-longterm /help-wanted - FYI
The InstanceMetadataByProviderID would be to every cloud provider, such as aws, azure, etc. right ?
/assign Hi , if you're interested in it, I can work with you.
Yes - we need to do that (which is why I wrote it's quite significant refectoring). It also makes reviews harder, because we will need to ask maintainers of all cloud-providers. So it would be good to separate changes to different cloud providers into separate commits.
I have one question: and first query by providerID, if failed then query by nodeName. While the logic in is converse(if providerID is nil, get providerID by nodeName, then ensure its existence by providerID). If we merge these three method into , how to deal with the query sequence of nodeName and providerID?
I bet it's a consequence of different people adding different code paths. Those conceptually seem to be the same to me. Unifying this will be additional benefit from this change imho. So I would say that choose whatever is easier to implement.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I think and are the only PRs left for this. One other optimization that can be made is to cache the instance metadata but I think that's a provider specific implementation
Plus enabling it for GCE if we decide to remove it from the first PR. But yeah, in general I agree.
- can you please address ?
Yes, I have seen that.
I think this issue has been resolved, feel free to reopen it. /close
Closing this issue. <detailsIn response to : I think this issue has been resolved, feel free to reopen it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

return false, cloudprovider.NotImplemented } <ins> // InstanceShutdown returns true if the instance is in safe state to detach volumes func (g *Cloud) InstanceShutdown(ctx context.Context, node *v1.Node) (bool, error) { return false, cloudprovider.NotImplemented } </ins> func nodeAddressesFromInstance(instance *compute.Instance) ([]v1.NodeAddress, error) { if len(instance.NetworkInterfaces) < 1 { return nil, fmt.Errorf("could not find network interfaces for instanceID %q", instance.Id)


Question:
For external cloud provider - Kubelet setting the TaintExternalCloudProvider taint: The results in calling function in cloud node-controller: This (in a healthy case) sends 4 GCE API calls to: node addresses: (as part of getNodeModifier call) instance type: (as part of getNodeModifiers call) instance: (as part of updateNodeAddresses call) node addresses (again): (as part of updateNodeAddresses call) All those requests are read requests and could be replaced by a single call. We could do that by introducing a simple struct like: and adding a method (or rather replacing the existing InstanceTypeByProviderID which is used only for this purpose) to Instances cloud-provider interface; With that change, we would be able to replace all that 4 calls with just a single API call. This would be useful especially for creating large clusters (like 5k nodes), which may saturate the quota for API calls for quite some time. This is quite significant size of refactoring, though it should be relatively mechanical task to do. /sig cloud-provider /sig scalability /priority important-longterm /help-wanted - FYI
The InstanceMetadataByProviderID would be to every cloud provider, such as aws, azure, etc. right ?
/assign Hi , if you're interested in it, I can work with you.
Yes - we need to do that (which is why I wrote it's quite significant refectoring). It also makes reviews harder, because we will need to ask maintainers of all cloud-providers. So it would be good to separate changes to different cloud providers into separate commits.
I have one question: and first query by providerID, if failed then query by nodeName. While the logic in is converse(if providerID is nil, get providerID by nodeName, then ensure its existence by providerID). If we merge these three method into , how to deal with the query sequence of nodeName and providerID?
I bet it's a consequence of different people adding different code paths. Those conceptually seem to be the same to me. Unifying this will be additional benefit from this change imho. So I would say that choose whatever is easier to implement.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I think and are the only PRs left for this. One other optimization that can be made is to cache the instance metadata but I think that's a provider specific implementation
Plus enabling it for GCE if we decide to remove it from the first PR. But yeah, in general I agree.
- can you please address ?
Yes, I have seen that.
I think this issue has been resolved, feel free to reopen it. /close
Closing this issue. <detailsIn response to : I think this issue has been resolved, feel free to reopen it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

return true, nil } <ins> // InstanceExists returns true if the instance with the given provider id still exists and is running. // If false is returned with no error, the instance will be immediately deleted by the cloud controller manager. func (g *Cloud) InstanceExists(ctx context.Context, node *v1.Node) (bool, error) { providerID := node.Spec.ProviderID if providerID == "" { var err error if providerID, err = g.InstanceID(ctx, types.NodeName(node.Name)); err != nil { return false, err } } return g.InstanceExistsByProviderID(ctx, providerID) } // InstanceMetadata returns metadata of the specified instance. func (g *Cloud) InstanceMetadata(ctx context.Context, node *v1.Node) (*cloudprovider.InstanceMetadata, error) { timeoutCtx, cancel := context.WithTimeout(ctx, 1*time.Hour) defer cancel() providerID := node.Spec.ProviderID if providerID == "" { var err error if providerID, err = g.InstanceID(ctx, types.NodeName(node.Name)); err != nil { return nil, err } } _, zone, name, err := splitProviderID(providerID) if err != nil { return nil, err } region, err := GetGCERegion(zone) if err != nil { return nil, err } instance, err := g.c.Instances().Get(timeoutCtx, meta.ZonalKey(canonicalizeInstanceName(name), zone)) if err != nil { return nil, fmt.Errorf("error while querying for providerID %q: %v", providerID, err) } addresses, err := nodeAddressesFromInstance(instance) if err != nil { return nil, err } return &cloudprovider.InstanceMetadata{ ProviderID:    providerID, InstanceType:  lastComponent(instance.MachineType), NodeAddresses: addresses, Zone:          zone, Region:        region, }, nil } </ins> // InstanceID returns the cloud provider ID of the node with the specified NodeName. func (g *Cloud) InstanceID(ctx context.Context, nodeName types.NodeName) (string, error) { instanceName := mapNodeNameToInstanceName(nodeName)


Question:
We have some scripts that install docker/containerd etc and are having trouble installing on focal based images as does not yet support focal fossa. So in we have a patch that uses debs for focal. Additional context, please see ci test failures in and This issue is to track fixing the script back up when focal support is to
/sig release /sig testing
/priority important-soon /area release-eng
/assign

software-properties-common  lsb-release <del> # focal repo for docker is not yet available, so we use bonic for now # https://github.com/kubernetes/kubernetes/issues/90709 </del> release=$(lsb_release -cs) <del> if [ "$release" == "focal" ]; then release="bionic"; fi </del> # Add the Docker apt-repository curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg


Question:
We have some scripts that install docker/containerd etc and are having trouble installing on focal based images as does not yet support focal fossa. So in we have a patch that uses debs for focal. Additional context, please see ci test failures in and This issue is to track fixing the script back up when focal support is to
/sig release /sig testing
/priority important-soon /area release-eng
/assign

software-properties-common  lsb-release <del> # focal repo for docker is not yet available, so we use bonic for now # https://github.com/kubernetes/kubernetes/issues/90709 </del> release=$(lsb_release -cs) <del> if [ "$release" == "focal" ]; then release="bionic"; fi </del> # Add the Docker apt-repository (as we install containerd from there) curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg


Question: <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: sig-windows jobs: Which test(s) are failing: [sig-network] Services should be able to change the type from ClusterIP to ExternalName [Conformance] [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] Since when has it been failing: It was a false positive until recently. Testgrid link: Reason for failure: Partially Qualified Domain Names are not currently supported on Windows containers. The tests are checking PQDNs for validation. The test was passing until recently because was still returning the exit code 0, even if the given DNS name was not found: [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] [BeforeEach] [sig-network] Services STEP: Creating a kubernetes client Apr 28 21:24:38.587: INFO: kubeConfig: /home/ubuntu/kube/config200424 STEP: Building a namespace api object, basename services STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in services-2633 STEP: Waiting for a default service account to be provisioned in namespace [BeforeEach] [sig-network] Services [It] should be able to change the type from NodePort to ExternalName [Conformance] STEP: creating a service nodeport-service with the type=NodePort in namespace services-2633 STEP: Creating active service to test reachability when its FQDN is referred as externalName for another service STEP: creating service externalsvc in namespace services-2633 STEP: creating replication controller externalsvc in namespace services-2633 I0428 21:24:42. ] Created replication controller with name: externalsvc, namespace: services-2633, replica count: 2 I0428 21:24:45. ] externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:48. ] externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:51. ] externalsvc Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:54. ] externalsvc Pods: 2 out of 2 created, 2 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady STEP: changing the NodePort service to type=ExternalName Apr 28 21:24:55.544: INFO: Creating new exec pod Apr 28 21:25:06.371: INFO: Running '/home/ubuntu/rune2e/kubernetes/output/dockerized/bin/linux/amd64/kubectl --server=https://kubetest--83ef-11ea-ae60- --kubeconfig=/home/ubuntu/kube/config200424 exec --namespace=services-2633 agnhost-pod -- /bin/sh -x -c nslookup nodeport-service' Apr 28 21:25:09.610: INFO: stderr: "+ nslookup nodeport-servicen kube- can't find nodeport-service: Non-existent domainrrn" Apr 28 21:25:09.610: INFO: stdout: "Server: kube-rnAddress: 10.0.0.10rnrn" STEP: deleting ReplicationController externalsvc in namespace services-2633, will wait for the garbage collector to delete the pods Apr 28 21:25:10.429: INFO: Deleting ReplicationController externalsvc took: 423.628373ms Apr 28 21:25:10.830: INFO: Terminating ReplicationController externalsvc pods took: 400.686463ms Apr 28 21:25:15.792: INFO: Cleaning up the NodePort to ExternalName test service [AfterEach] [sig-network] Services Apr 28 21:25:16.230: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready STEP: Destroying namespace "services-2633" for this suite. [AfterEach] [sig-network] Services â¢{"msg":"PASSED [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance]","total":2,"completed":2,"skipped":2913,"failed":0} The issue was observed with an updated, which doesn't return 0 if the given DNS name was not found. Anything else we need to know*: /sig testing /sig windows

// checkExternalServiceReachability ensures service of type externalName resolves to IP address and no fake externalName is set // FQDN of kubernetes is used as externalName(for air tight platforms). func (j *TestJig) checkExternalServiceReachability(svc *v1.Service, pod *v1.Pod) error { <ins> // NOTE(claudiub): Windows does not support PQDN. svcName := fmt.Sprintf("%s.%s.svc.%s", svc.Name, svc.Namespace, framework.TestContext.ClusterDNSDomain) </ins> // Service must resolve to IP <del> cmd := fmt.Sprintf("nslookup %s", svc.Name) _, err := framework.RunHostCmd(pod.Namespace, pod.Name, cmd) if err != nil { </del> <ins> cmd := fmt.Sprintf("nslookup %s", svcName) _, stderr, err := framework.RunHostCmdWithFullOutput(pod.Namespace, pod.Name, cmd) // NOTE(claudiub): nslookup may return 0 on Windows, even though the DNS name was not found. In this case, // we can check stderr for the error. if err != nil || (framework.NodeOSDistroIs("windows") && strings.Contains(stderr, fmt.Sprintf("can't find %s", svcName))) { </ins> return fmt.Errorf("ExternalName service %q must resolve to IP", pod.Namespace+"/"+pod.Name) } return nil


Question: <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: sig-windows jobs: Which test(s) are failing: [sig-network] Services should be able to change the type from ClusterIP to ExternalName [Conformance] [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] Since when has it been failing: It was a false positive until recently. Testgrid link: Reason for failure: Partially Qualified Domain Names are not currently supported on Windows containers. The tests are checking PQDNs for validation. The test was passing until recently because was still returning the exit code 0, even if the given DNS name was not found: [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance] [BeforeEach] [sig-network] Services STEP: Creating a kubernetes client Apr 28 21:24:38.587: INFO: kubeConfig: /home/ubuntu/kube/config200424 STEP: Building a namespace api object, basename services STEP: Binding the e2e-test-privileged-psp PodSecurityPolicy to the default service account in services-2633 STEP: Waiting for a default service account to be provisioned in namespace [BeforeEach] [sig-network] Services [It] should be able to change the type from NodePort to ExternalName [Conformance] STEP: creating a service nodeport-service with the type=NodePort in namespace services-2633 STEP: Creating active service to test reachability when its FQDN is referred as externalName for another service STEP: creating service externalsvc in namespace services-2633 STEP: creating replication controller externalsvc in namespace services-2633 I0428 21:24:42. ] Created replication controller with name: externalsvc, namespace: services-2633, replica count: 2 I0428 21:24:45. ] externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:48. ] externalsvc Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:51. ] externalsvc Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady I0428 21:24:54. ] externalsvc Pods: 2 out of 2 created, 2 running, 0 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady STEP: changing the NodePort service to type=ExternalName Apr 28 21:24:55.544: INFO: Creating new exec pod Apr 28 21:25:06.371: INFO: Running '/home/ubuntu/rune2e/kubernetes/output/dockerized/bin/linux/amd64/kubectl --server=https://kubetest--83ef-11ea-ae60- --kubeconfig=/home/ubuntu/kube/config200424 exec --namespace=services-2633 agnhost-pod -- /bin/sh -x -c nslookup nodeport-service' Apr 28 21:25:09.610: INFO: stderr: "+ nslookup nodeport-servicen kube- can't find nodeport-service: Non-existent domainrrn" Apr 28 21:25:09.610: INFO: stdout: "Server: kube-rnAddress: 10.0.0.10rnrn" STEP: deleting ReplicationController externalsvc in namespace services-2633, will wait for the garbage collector to delete the pods Apr 28 21:25:10.429: INFO: Deleting ReplicationController externalsvc took: 423.628373ms Apr 28 21:25:10.830: INFO: Terminating ReplicationController externalsvc pods took: 400.686463ms Apr 28 21:25:15.792: INFO: Cleaning up the NodePort to ExternalName test service [AfterEach] [sig-network] Services Apr 28 21:25:16.230: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready STEP: Destroying namespace "services-2633" for this suite. [AfterEach] [sig-network] Services â¢{"msg":"PASSED [sig-network] Services should be able to change the type from NodePort to ExternalName [Conformance]","total":2,"completed":2,"skipped":2913,"failed":0} The issue was observed with an updated, which doesn't return 0 if the given DNS name was not found. Anything else we need to know*: /sig testing /sig windows

// Exec runs the kubectl executable. func (b KubectlBuilder) Exec() (string, error) { <ins> stdout, _, err := b.ExecWithFullOutput() return stdout, err } // ExecWithFullOutput runs the kubectl executable, and returns the stdout and stderr. func (b KubectlBuilder) ExecWithFullOutput() (string, string, error) { </ins> var stdout, stderr bytes.Buffer cmd := b.cmd cmd.Stdout, cmd.Stderr = &stdout, &stderr Logf("Running '%s %s'", cmd.Path, strings.Join(cmd.Args[1:], " ")) // skip arg[0] as it is printed separately if err := cmd.Start(); err != nil { <del> return "", fmt.Errorf("error starting %v:nCommand stdout:n%vnstderr:n%vnerror:n%v", cmd, cmd.Stdout, cmd.Stderr, err) </del> <ins> return "", "", fmt.Errorf("error starting %v:nCommand stdout:n%vnstderr:n%vnerror:n%v", cmd, cmd.Stdout, cmd.Stderr, err) </ins> } errCh := make(chan error, 1) go func() {