Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-kubernetes-e2e-ubuntu-gce [sig-storage] Subpath Container restart should verify that container can restart successfully after configmaps modified links to appreciated links to specific failures in spyglass appreciated Triage shows that this has been flaking quite frequently: It has flaked in 5 of the last 10 runs: /sig storage /priority important-soon <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Noticed some more flakes on this one
Yep, looks like this is still flaking quite frequently: Going to re-ping SIG Storage.
/assign
Test is flaking maybe half of the time. ~This test is failing in gce-ubuntu-master-default - not sure if it's failing in any other dashboards, nor whether this is run anywhere else.~ edit: This is failing on Ubuntu only. Not sure if this is limited to master branch. Fix to the original issue was in since 1.19. This is a subpath test which a configmap a container that mounts the configmap a liveness probe and waits for the container to restart. liveness probe and waits for the container to stop restarting. The test failed at the last step - the container never stopped restarting. Timeout is 2min. The liveness probe continuously checks whether a "probeFile" exists in (an emptyDir), and if it doesn't exist, whether the configmap file contains the modified value. The probeFile essentially acts as a condition which guards the probe from checking the configmap file. The test fails the liveness probe by removing the probeFile without updating the configmap just yet. It then fixes the liveness probe by updating the configmap to the modified value. So if the container keeps restarting, the liveness probe could've been failing continuously within 2min, due to one of the following: (? not sure about this one) Command can't be executed (for example if container can't start) configmap file was never updated. Within 2min, the pod only restarted 5 times. The liveness probe's period is 2s, initial delay is 1s, and failure threshold is 1, so a restart due to liveness probe failure is triggered every 3s. The image is busybox. The command is just , so it's unlikely for the container to have crashed. There were no liveness probe unhealthy events after the test fixes the probe, and yet it continued to restart. No more events after probe is fixed.
Could we check if the configmap value was updated for sure to rule out that scenario ? I have not tried to run an end to end test locally :) so am not sure how to debug it
Will do! I ran the test in a GKE Ubuntu cluster running 1.19.6 and couldn't reproduce the failure. I'm now trying to run a cluster on GCE with Ubuntu nodes running k8s master, but haven't been successful after trying a number of techniques. If anyone happens to know how to do this let me know; I'll keep asking around otherwise.
I got a repro on a GKE cluster backed by COS. After the test failed the container ends up in Running state eventually, and the configmap file outputs the expected modified value. The fix is to increase the test timeout. PR coming up.

// Check that container has restarted ginkgo.By("Waiting for container to restart") restarts := int32(0) <del> err = wait.PollImmediate(10*time.Second, 2*time.Minute, func() (bool, error) { </del> <ins> err = wait.PollImmediate(10*time.Second, framework.PodStartTimeout, func() (bool, error) { </ins> pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(context.TODO(), pod.Name, metav1.GetOptions{}) if err != nil { return false, err


Question:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-kubernetes-e2e-ubuntu-gce [sig-storage] Subpath Container restart should verify that container can restart successfully after configmaps modified links to appreciated links to specific failures in spyglass appreciated Triage shows that this has been flaking quite frequently: It has flaked in 5 of the last 10 runs: /sig storage /priority important-soon <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Noticed some more flakes on this one
Yep, looks like this is still flaking quite frequently: Going to re-ping SIG Storage.
/assign
Test is flaking maybe half of the time. ~This test is failing in gce-ubuntu-master-default - not sure if it's failing in any other dashboards, nor whether this is run anywhere else.~ edit: This is failing on Ubuntu only. Not sure if this is limited to master branch. Fix to the original issue was in since 1.19. This is a subpath test which a configmap a container that mounts the configmap a liveness probe and waits for the container to restart. liveness probe and waits for the container to stop restarting. The test failed at the last step - the container never stopped restarting. Timeout is 2min. The liveness probe continuously checks whether a "probeFile" exists in (an emptyDir), and if it doesn't exist, whether the configmap file contains the modified value. The probeFile essentially acts as a condition which guards the probe from checking the configmap file. The test fails the liveness probe by removing the probeFile without updating the configmap just yet. It then fixes the liveness probe by updating the configmap to the modified value. So if the container keeps restarting, the liveness probe could've been failing continuously within 2min, due to one of the following: (? not sure about this one) Command can't be executed (for example if container can't start) configmap file was never updated. Within 2min, the pod only restarted 5 times. The liveness probe's period is 2s, initial delay is 1s, and failure threshold is 1, so a restart due to liveness probe failure is triggered every 3s. The image is busybox. The command is just , so it's unlikely for the container to have crashed. There were no liveness probe unhealthy events after the test fixes the probe, and yet it continued to restart. No more events after probe is fixed.
Could we check if the configmap value was updated for sure to rule out that scenario ? I have not tried to run an end to end test locally :) so am not sure how to debug it
Will do! I ran the test in a GKE Ubuntu cluster running 1.19.6 and couldn't reproduce the failure. I'm now trying to run a cluster on GCE with Ubuntu nodes running k8s master, but haven't been successful after trying a number of techniques. If anyone happens to know how to do this let me know; I'll keep asking around otherwise.
I got a repro on a GKE cluster backed by COS. After the test failed the container ends up in Running state eventually, and the configmap file outputs the expected modified value. The fix is to increase the test timeout. PR coming up.

ginkgo.By("Waiting for container to stop restarting") stableCount := int(0) stableThreshold := int(time.Minute / framework.Poll) <del> err = wait.PollImmediate(framework.Poll, 2*time.Minute, func() (bool, error) { </del> <ins> err = wait.PollImmediate(framework.Poll, framework.PodStartTimeout, func() (bool, error) { </ins> pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(context.TODO(), pod.Name, metav1.GetOptions{}) if err != nil { return false, err


Question:
found this in while finding beta API dependencies for 1.22. This will block from running at all in 1.22 I think this is for this deployment: , and in ingress-gce I found
/sig network /triage accepted I pinged some GCE sig-network folks in the PR
given that uses extensions/v1beta1 Deployments, that particular manifest seems unused
once we land the fix in , anyone know how to update the image we use in e2e tests?
/assign The update to use v1 APIs is merged to in What version should be tagged we can pick up in Does need to be picked back to one or more release branches?
It would be helpful to get a version containing the fix tagged, so the manifest can be updated in to get e2e signal even before opens for 1.22.
Ack. We are planning to cut a release soon. Once the release is cut, I will update the k/k to use that
what's the ETA for cutting ingress-gce release? I think it might be related to failing pull-kubernetes-e2e-gce-100-performance run in

var ( // TODO: Deprecate gitMajor and gitMinor, use only gitVersion instead. gitMajor     string = "0"              // major version, always numeric <del> gitMinor     string = "2+"             // minor version, numeric possibly followed by "+" gitVersion   string = "v0.2-dev"       // version from git, output of $(git describe) </del> <ins> gitMinor     string = "3+"             // minor version, numeric possibly followed by "+" gitVersion   string = "v0.3-dev"       // version from git, output of $(git describe) </ins> gitCommit    string = ""               // sha1 from git, output of $(git rev-parse HEAD) gitTreeState string = "not a git tree" // state of git tree, either "clean" or "dirty" )


Question:
found this in while finding beta API dependencies for 1.22. This will block from running at all in 1.22 I think this is for this deployment: , and in ingress-gce I found
/sig network /triage accepted I pinged some GCE sig-network folks in the PR
given that uses extensions/v1beta1 Deployments, that particular manifest seems unused
once we land the fix in , anyone know how to update the image we use in e2e tests?
/assign The update to use v1 APIs is merged to in What version should be tagged we can pick up in Does need to be picked back to one or more release branches?
It would be helpful to get a version containing the fix tagged, so the manifest can be updated in to get e2e signal even before opens for 1.22.
Ack. We are planning to cut a release soon. Once the release is cut, I will update the k/k to use that
what's the ETA for cutting ingress-gce release? I think it might be related to failing pull-kubernetes-e2e-gce-100-performance run in

terminationGracePeriodSeconds: 600 hostNetwork: true containers: <del> - image: gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64:v1.11.0 </del> <ins> - image: gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64:v1.12.0 </ins> livenessProbe: httpGet: path: /healthz


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In , client-go don't share transport when c.Dial is not nil, but kubelet custorm Dial by default, After PR is integrated, the connections between kubelet and kube-apiserver in our cluster with 4000 nodes increases by five times. kubelet share transport by default, one kubelet only keep one connection to kube-apiserver. start kubelet with rotate-certificates=false Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ):EulerOS 2.9 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /sig node
/triage accepted /assign
how many connections between kubelet and kube apiserver on one node after PR in your case? it's mentioned it increases by five times, does that mean before the PR, there is one connection, after the PR, there are 5 or 6 connections?
before the PR, there is one connection, after the PR, there 3 connections, there are some customization capabilities in our product, and we'll have another two connections. In our kubelet startup parameters, rotate-certificates=false
Some increase was expected, since multiple REST clients constructed from a config with a custom dialer cannot safely share a transport, and client-go constructs a REST client for each API group/version accessed. Something like would be required to rework how client-go constructs clients to start sharing REST clients above the transport level between API groups/versions. I think this is working as intended until client-go client construction is reworked.
do you know where in the code the 5 connections are created? are they all safe to share the transport? also are the 5 connections all active all the time? if not, IdleConnTimeout should cause the connection to be closed.
I test the code on master branch, if I change rotateCertificates to false, there are 6 connections between kubelet and kube-apiserver ! !
transport will share when config.Transport not nil if rotateCertificates is true, config.Transport will been set in kubeletcertificate.UpdateTransport but if rotateCertificates is false, config.Transport will not been set.
/area kubelet Any update on this issue?
I will work on this issue
Hi if rotate-certificates=false kubelet customizes the dialer to provide a closeAllConns to close the connection when the connection is dead but not been closed. In , there is a discuss about another solution after client-go has HTTP 2.0 health check, so we don't need this change. So we can solve the current problem by reverting . I have submitted to revert , and this change requires your final confirmation. Thanks you.
are we guaranteed kubelets always speak to kube-apiserver over http/2? if they use a load-balancer that is not http/2 capable, isn't the "close all connections" fix still required as a backstop?
I thought http1.1 wasn't affected because it will always try to connect a new connection
I'm not confident in that. The original issue dated back to before client-go defaulted to http/2, and we definitely saw issues with stuck TCP connections then.
yeah, I could reproduce it with http1.1, however, it can be solved using the transport CloseIdleConnections method please take a look , the last commit needs more work, but I think that the 2 first commits exposeing the CloseIdleConnections() are useful
ok, this seems to work , but it will be nice to have more feedback would you be able to test and see if this solve your issue?
Hi aojea, I have test on v1.21.1 before merge , there are 6 connections to kube-apiserver from kubelet ! after merge , there is only 1 connection to kube-apiserver from kubelet ! so, can solve my issue. thank you very much.

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" <ins> utilnet "k8s.io/apimachinery/pkg/util/net" </ins> utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/wait"


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In , client-go don't share transport when c.Dial is not nil, but kubelet custorm Dial by default, After PR is integrated, the connections between kubelet and kube-apiserver in our cluster with 4000 nodes increases by five times. kubelet share transport by default, one kubelet only keep one connection to kube-apiserver. start kubelet with rotate-certificates=false Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ):EulerOS 2.9 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /sig node
/triage accepted /assign
how many connections between kubelet and kube apiserver on one node after PR in your case? it's mentioned it increases by five times, does that mean before the PR, there is one connection, after the PR, there are 5 or 6 connections?
before the PR, there is one connection, after the PR, there 3 connections, there are some customization capabilities in our product, and we'll have another two connections. In our kubelet startup parameters, rotate-certificates=false
Some increase was expected, since multiple REST clients constructed from a config with a custom dialer cannot safely share a transport, and client-go constructs a REST client for each API group/version accessed. Something like would be required to rework how client-go constructs clients to start sharing REST clients above the transport level between API groups/versions. I think this is working as intended until client-go client construction is reworked.
do you know where in the code the 5 connections are created? are they all safe to share the transport? also are the 5 connections all active all the time? if not, IdleConnTimeout should cause the connection to be closed.
I test the code on master branch, if I change rotateCertificates to false, there are 6 connections between kubelet and kube-apiserver ! !
transport will share when config.Transport not nil if rotateCertificates is true, config.Transport will been set in kubeletcertificate.UpdateTransport but if rotateCertificates is false, config.Transport will not been set.
/area kubelet Any update on this issue?
I will work on this issue
Hi if rotate-certificates=false kubelet customizes the dialer to provide a closeAllConns to close the connection when the connection is dead but not been closed. In , there is a discuss about another solution after client-go has HTTP 2.0 health check, so we don't need this change. So we can solve the current problem by reverting . I have submitted to revert , and this change requires your final confirmation. Thanks you.
are we guaranteed kubelets always speak to kube-apiserver over http/2? if they use a load-balancer that is not http/2 capable, isn't the "close all connections" fix still required as a backstop?
I thought http1.1 wasn't affected because it will always try to connect a new connection
I'm not confident in that. The original issue dated back to before client-go defaulted to http/2, and we definitely saw issues with stuck TCP connections then.
yeah, I could reproduce it with http1.1, however, it can be solved using the transport CloseIdleConnections method please take a look , the last commit needs more work, but I think that the 2 first commits exposeing the CloseIdleConnections() are useful
ok, this seems to work , but it will be nice to have more feedback would you be able to test and see if this solve your issue?
Hi aojea, I have test on v1.21.1 before merge , there are 6 connections to kube-apiserver from kubelet ! after merge , there is only 1 connection to kube-apiserver from kubelet ! so, can solve my issue. thank you very much.

} kubeClientConfigOverrides(s, clientConfig) <del> closeAllConns, err := updateDialer(clientConfig) if err != nil { return nil, nil, err </del> <ins> // Kubelet needs to be able to recover from stale http connections. // HTTP2 has a mechanism to detect broken connections by sending periodical pings. // HTTP1 only can have one persistent connection, and it will close all Idle connections // once the Kubelet heartbeat fails. However, since there are many edge cases that we can't // control, users can still opt-in to the previous behavior for closing the connections by // setting the environment variable DISABLE_HTTP2. var closeAllConns func() if s := os.Getenv("DISABLE_HTTP2"); len(s) > 0 { klog.InfoS("HTTP2 has been explicitly disabled, updating Kubelet client Dialer to forcefully close active connections on heartbeat failures") closeAllConns, err = updateDialer(clientConfig) if err != nil { return nil, nil, err } } else { closeAllConns = func() { utilnet.CloseIdleConnectionsFor(clientConfig.Transport) } </ins> } return clientConfig, closeAllConns, nil }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In , client-go don't share transport when c.Dial is not nil, but kubelet custorm Dial by default, After PR is integrated, the connections between kubelet and kube-apiserver in our cluster with 4000 nodes increases by five times. kubelet share transport by default, one kubelet only keep one connection to kube-apiserver. start kubelet with rotate-certificates=false Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ):EulerOS 2.9 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /sig node
/triage accepted /assign
how many connections between kubelet and kube apiserver on one node after PR in your case? it's mentioned it increases by five times, does that mean before the PR, there is one connection, after the PR, there are 5 or 6 connections?
before the PR, there is one connection, after the PR, there 3 connections, there are some customization capabilities in our product, and we'll have another two connections. In our kubelet startup parameters, rotate-certificates=false
Some increase was expected, since multiple REST clients constructed from a config with a custom dialer cannot safely share a transport, and client-go constructs a REST client for each API group/version accessed. Something like would be required to rework how client-go constructs clients to start sharing REST clients above the transport level between API groups/versions. I think this is working as intended until client-go client construction is reworked.
do you know where in the code the 5 connections are created? are they all safe to share the transport? also are the 5 connections all active all the time? if not, IdleConnTimeout should cause the connection to be closed.
I test the code on master branch, if I change rotateCertificates to false, there are 6 connections between kubelet and kube-apiserver ! !
transport will share when config.Transport not nil if rotateCertificates is true, config.Transport will been set in kubeletcertificate.UpdateTransport but if rotateCertificates is false, config.Transport will not been set.
/area kubelet Any update on this issue?
I will work on this issue
Hi if rotate-certificates=false kubelet customizes the dialer to provide a closeAllConns to close the connection when the connection is dead but not been closed. In , there is a discuss about another solution after client-go has HTTP 2.0 health check, so we don't need this change. So we can solve the current problem by reverting . I have submitted to revert , and this change requires your final confirmation. Thanks you.
are we guaranteed kubelets always speak to kube-apiserver over http/2? if they use a load-balancer that is not http/2 capable, isn't the "close all connections" fix still required as a backstop?
I thought http1.1 wasn't affected because it will always try to connect a new connection
I'm not confident in that. The original issue dated back to before client-go defaulted to http/2, and we definitely saw issues with stuck TCP connections then.
yeah, I could reproduce it with http1.1, however, it can be solved using the transport CloseIdleConnections method please take a look , the last commit needs more work, but I think that the 2 first commits exposeing the CloseIdleConnections() are useful
ok, this seems to work , but it will be nice to have more feedback would you be able to test and see if this solve your issue?
Hi aojea, I have test on v1.21.1 before merge , there are 6 connections to kube-apiserver from kubelet ! after merge , there is only 1 connection to kube-apiserver from kubelet ! so, can solve my issue. thank you very much.

} } <ins> // CloseIdleConnectionsFor close idles connections for the Transport. // If the Transport is wrapped it iterates over the wrapped round trippers // until it finds one that implements the CloseIdleConnections method. // If the Transport does not have a CloseIdleConnections method // then this function does nothing. func CloseIdleConnectionsFor(transport http.RoundTripper) { if transport == nil { return } type closeIdler interface { CloseIdleConnections() } switch transport := transport.(type) { case closeIdler: transport.CloseIdleConnections() case RoundTripperWrapper: CloseIdleConnectionsFor(transport.WrappedRoundTripper()) default: klog.Warningf("unknown transport type: %T", transport) } } </ins> type TLSClientConfigHolder interface { TLSClientConfig() *tls.Config }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In , client-go don't share transport when c.Dial is not nil, but kubelet custorm Dial by default, After PR is integrated, the connections between kubelet and kube-apiserver in our cluster with 4000 nodes increases by five times. kubelet share transport by default, one kubelet only keep one connection to kube-apiserver. start kubelet with rotate-certificates=false Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ):EulerOS 2.9 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /sig node
/triage accepted /assign
how many connections between kubelet and kube apiserver on one node after PR in your case? it's mentioned it increases by five times, does that mean before the PR, there is one connection, after the PR, there are 5 or 6 connections?
before the PR, there is one connection, after the PR, there 3 connections, there are some customization capabilities in our product, and we'll have another two connections. In our kubelet startup parameters, rotate-certificates=false
Some increase was expected, since multiple REST clients constructed from a config with a custom dialer cannot safely share a transport, and client-go constructs a REST client for each API group/version accessed. Something like would be required to rework how client-go constructs clients to start sharing REST clients above the transport level between API groups/versions. I think this is working as intended until client-go client construction is reworked.
do you know where in the code the 5 connections are created? are they all safe to share the transport? also are the 5 connections all active all the time? if not, IdleConnTimeout should cause the connection to be closed.
I test the code on master branch, if I change rotateCertificates to false, there are 6 connections between kubelet and kube-apiserver ! !
transport will share when config.Transport not nil if rotateCertificates is true, config.Transport will been set in kubeletcertificate.UpdateTransport but if rotateCertificates is false, config.Transport will not been set.
/area kubelet Any update on this issue?
I will work on this issue
Hi if rotate-certificates=false kubelet customizes the dialer to provide a closeAllConns to close the connection when the connection is dead but not been closed. In , there is a discuss about another solution after client-go has HTTP 2.0 health check, so we don't need this change. So we can solve the current problem by reverting . I have submitted to revert , and this change requires your final confirmation. Thanks you.
are we guaranteed kubelets always speak to kube-apiserver over http/2? if they use a load-balancer that is not http/2 capable, isn't the "close all connections" fix still required as a backstop?
I thought http1.1 wasn't affected because it will always try to connect a new connection
I'm not confident in that. The original issue dated back to before client-go defaulted to http/2, and we definitely saw issues with stuck TCP connections then.
yeah, I could reproduce it with http1.1, however, it can be solved using the transport CloseIdleConnections method please take a look , the last commit needs more work, but I think that the 2 first commits exposeing the CloseIdleConnections() are useful
ok, this seems to work , but it will be nice to have more feedback would you be able to test and see if this solve your issue?
Hi aojea, I have test on v1.21.1 before merge , there are 6 connections to kube-apiserver from kubelet ! after merge , there is only 1 connection to kube-apiserver from kubelet ! so, can solve my issue. thank you very much.

"k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/runtime/serializer" utilnet "k8s.io/apimachinery/pkg/util/net" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> ) type tcpLB struct {


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In , client-go don't share transport when c.Dial is not nil, but kubelet custorm Dial by default, After PR is integrated, the connections between kubelet and kube-apiserver in our cluster with 4000 nodes increases by five times. kubelet share transport by default, one kubelet only keep one connection to kube-apiserver. start kubelet with rotate-certificates=false Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ):EulerOS 2.9 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /sig node
/triage accepted /assign
how many connections between kubelet and kube apiserver on one node after PR in your case? it's mentioned it increases by five times, does that mean before the PR, there is one connection, after the PR, there are 5 or 6 connections?
before the PR, there is one connection, after the PR, there 3 connections, there are some customization capabilities in our product, and we'll have another two connections. In our kubelet startup parameters, rotate-certificates=false
Some increase was expected, since multiple REST clients constructed from a config with a custom dialer cannot safely share a transport, and client-go constructs a REST client for each API group/version accessed. Something like would be required to rework how client-go constructs clients to start sharing REST clients above the transport level between API groups/versions. I think this is working as intended until client-go client construction is reworked.
do you know where in the code the 5 connections are created? are they all safe to share the transport? also are the 5 connections all active all the time? if not, IdleConnTimeout should cause the connection to be closed.
I test the code on master branch, if I change rotateCertificates to false, there are 6 connections between kubelet and kube-apiserver ! !
transport will share when config.Transport not nil if rotateCertificates is true, config.Transport will been set in kubeletcertificate.UpdateTransport but if rotateCertificates is false, config.Transport will not been set.
/area kubelet Any update on this issue?
I will work on this issue
Hi if rotate-certificates=false kubelet customizes the dialer to provide a closeAllConns to close the connection when the connection is dead but not been closed. In , there is a discuss about another solution after client-go has HTTP 2.0 health check, so we don't need this change. So we can solve the current problem by reverting . I have submitted to revert , and this change requires your final confirmation. Thanks you.
are we guaranteed kubelets always speak to kube-apiserver over http/2? if they use a load-balancer that is not http/2 capable, isn't the "close all connections" fix still required as a backstop?
I thought http1.1 wasn't affected because it will always try to connect a new connection
I'm not confident in that. The original issue dated back to before client-go defaulted to http/2, and we definitely saw issues with stuck TCP connections then.
yeah, I could reproduce it with http1.1, however, it can be solved using the transport CloseIdleConnections method please take a look , the last commit needs more work, but I think that the 2 first commits exposeing the CloseIdleConnections() are useful
ok, this seems to work , but it will be nice to have more feedback would you be able to test and see if this solve your issue?
Hi aojea, I have test on v1.21.1 before merge , there are 6 connections to kube-apiserver from kubelet ! after merge , there is only 1 connection to kube-apiserver from kubelet ! so, can solve my issue. thank you very much.

} } <ins> // 1. connect to https server with http1.1 using a TCP proxy // 2. the connection has keepalive enabled so it will be reused // 3. break the TCP connection stopping the proxy // 4. close the idle connection to force creating a new connection // 5. count that there are 2 connection to the server (we didn't reuse the original connection) func TestReconnectBrokenTCP_HTTP1(t *testing.T) { ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "Hello, %s", r.Proto) })) ts.EnableHTTP2 = false ts.StartTLS() defer ts.Close() u, err := url.Parse(ts.URL) if err != nil { t.Fatalf("failed to parse URL from %q: %v", ts.URL, err) } lb := newLB(t, u.Host) defer lb.ln.Close() stopCh := make(chan struct{}) go lb.serve(stopCh) transport, ok := ts.Client().Transport.(*http.Transport) if !ok { t.Fatal("failed to assert *http.Transport") } config := &Config{ Host:      "https://" + lb.ln.Addr().String(), Transport: utilnet.SetTransportDefaults(transport), // large timeout, otherwise the broken connection will be cleaned by it Timeout: wait.ForeverTestTimeout, // These fields are required to create a REST client. ContentConfig: ContentConfig{ GroupVersion:         &schema.GroupVersion{}, NegotiatedSerializer: &serializer.CodecFactory{}, }, } config.TLSClientConfig.NextProtos = []string{"http/1.1"} client, err := RESTClientFor(config) if err != nil { t.Fatalf("failed to create REST client: %v", err) } data, err := client.Get().AbsPath("/").DoRaw(context.TODO()) if err != nil { t.Fatalf("unexpected err: %s: %v", data, err) } if string(data) != "Hello, HTTP/1.1" { t.Fatalf("unexpected response: %s", data) } // Deliberately let the LB stop proxying traffic for the current // connection. This mimics a broken TCP connection that's not properly // closed. close(stopCh) stopCh = make(chan struct{}) go lb.serve(stopCh) // Close the idle connections utilnet.CloseIdleConnectionsFor(client.Client.Transport) // If the client didn't close the idle connections, the broken connection // would still be in the connection pool, the following request would // then reuse the broken connection instead of creating a new one, and // thus would fail. data, err = client.Get().AbsPath("/").DoRaw(context.TODO()) if err != nil { t.Fatalf("unexpected err: %v", err) } if string(data) != "Hello, HTTP/1.1" { t.Fatalf("unexpected response: %s", data) } dials := atomic.LoadInt32(&lb.dials) if dials != 2 { t.Fatalf("expected %d dials, got %d", 2, dials) } } // 1. connect to https server with http1.1 using a TCP proxy making the connection to timeout // 2. the connection has keepalive enabled so it will be reused // 3. close the in-flight connection to force creating a new connection // 4. count that there are 2 connection on the LB but only one succeeds func TestReconnectBrokenTCPInFlight_HTTP1(t *testing.T) { done := make(chan struct{}) defer close(done) received := make(chan struct{}) ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if r.URL.Path == "/hang" { conn, _, _ := w.(http.Hijacker).Hijack() close(received) <-done conn.Close() } fmt.Fprintf(w, "Hello, %s", r.Proto) })) ts.EnableHTTP2 = false ts.StartTLS() defer ts.Close() u, err := url.Parse(ts.URL) if err != nil { t.Fatalf("failed to parse URL from %q: %v", ts.URL, err) } lb := newLB(t, u.Host) defer lb.ln.Close() stopCh := make(chan struct{}) go lb.serve(stopCh) transport, ok := ts.Client().Transport.(*http.Transport) if !ok { t.Fatal("failed to assert *http.Transport") } config := &Config{ Host:      "https://" + lb.ln.Addr().String(), Transport: utilnet.SetTransportDefaults(transport), // Use something extraordinary large to not hit the timeout Timeout: wait.ForeverTestTimeout, // These fields are required to create a REST client. ContentConfig: ContentConfig{ GroupVersion:         &schema.GroupVersion{}, NegotiatedSerializer: &serializer.CodecFactory{}, }, } config.TLSClientConfig.NextProtos = []string{"http/1.1"} client, err := RESTClientFor(config) if err != nil { t.Fatalf("failed to create REST client: %v", err) } // The request will connect, hang and eventually time out // but we can use a context to close once the test is done // we are only interested in have an inflight connection ctx, cancel := context.WithCancel(context.Background()) reqErrCh := make(chan error, 1) defer close(reqErrCh) go func() { _, err = client.Get().AbsPath("/hang").DoRaw(ctx) reqErrCh <- err }() // wait until it connect to the server select { case <-received: case <-time.After(wait.ForeverTestTimeout): t.Fatal("Test timed out waiting for first request to fail") } // Deliberately let the LB stop proxying traffic for the current // connection. This mimics a broken TCP connection that's not properly // closed. close(stopCh) stopCh = make(chan struct{}) go lb.serve(stopCh) // New request will fail if tries to reuse the connection data, err := client.Get().AbsPath("/").DoRaw(context.Background()) if err != nil { t.Fatalf("unexpected err: %v", err) } if string(data) != "Hello, HTTP/1.1" { t.Fatalf("unexpected response: %s", data) } dials := atomic.LoadInt32(&lb.dials) if dials != 2 { t.Fatalf("expected %d dials, got %d", 2, dials) } // cancel the in-flight connection cancel() select { case <-reqErrCh: if err == nil { t.Fatal("Connection succeeded but was expected to timeout") } case <-time.After(10 * time.Second): t.Fatal("Test timed out waiting for the request to fail") } } </ins> func TestRestClientTimeout(t *testing.T) { ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { time.Sleep(2 * time.Second)


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In , client-go don't share transport when c.Dial is not nil, but kubelet custorm Dial by default, After PR is integrated, the connections between kubelet and kube-apiserver in our cluster with 4000 nodes increases by five times. kubelet share transport by default, one kubelet only keep one connection to kube-apiserver. start kubelet with rotate-certificates=false Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ):EulerOS 2.9 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /sig node
/triage accepted /assign
how many connections between kubelet and kube apiserver on one node after PR in your case? it's mentioned it increases by five times, does that mean before the PR, there is one connection, after the PR, there are 5 or 6 connections?
before the PR, there is one connection, after the PR, there 3 connections, there are some customization capabilities in our product, and we'll have another two connections. In our kubelet startup parameters, rotate-certificates=false
Some increase was expected, since multiple REST clients constructed from a config with a custom dialer cannot safely share a transport, and client-go constructs a REST client for each API group/version accessed. Something like would be required to rework how client-go constructs clients to start sharing REST clients above the transport level between API groups/versions. I think this is working as intended until client-go client construction is reworked.
do you know where in the code the 5 connections are created? are they all safe to share the transport? also are the 5 connections all active all the time? if not, IdleConnTimeout should cause the connection to be closed.
I test the code on master branch, if I change rotateCertificates to false, there are 6 connections between kubelet and kube-apiserver ! !
transport will share when config.Transport not nil if rotateCertificates is true, config.Transport will been set in kubeletcertificate.UpdateTransport but if rotateCertificates is false, config.Transport will not been set.
/area kubelet Any update on this issue?
I will work on this issue
Hi if rotate-certificates=false kubelet customizes the dialer to provide a closeAllConns to close the connection when the connection is dead but not been closed. In , there is a discuss about another solution after client-go has HTTP 2.0 health check, so we don't need this change. So we can solve the current problem by reverting . I have submitted to revert , and this change requires your final confirmation. Thanks you.
are we guaranteed kubelets always speak to kube-apiserver over http/2? if they use a load-balancer that is not http/2 capable, isn't the "close all connections" fix still required as a backstop?
I thought http1.1 wasn't affected because it will always try to connect a new connection
I'm not confident in that. The original issue dated back to before client-go defaulted to http/2, and we definitely saw issues with stuck TCP connections then.
yeah, I could reproduce it with http1.1, however, it can be solved using the transport CloseIdleConnections method please take a look , the last commit needs more work, but I think that the 2 first commits exposeing the CloseIdleConnections() are useful
ok, this seems to work , but it will be nice to have more feedback would you be able to test and see if this solve your issue?
Hi aojea, I have test on v1.21.1 before merge , there are 6 connections to kube-apiserver from kubelet ! after merge , there is only 1 connection to kube-apiserver from kubelet ! so, can solve my issue. thank you very much.

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/runtime/serializer" </ins> "k8s.io/apimachinery/pkg/runtime/serializer/streaming" "k8s.io/apimachinery/pkg/util/diff" "k8s.io/apimachinery/pkg/util/httpstream" "k8s.io/apimachinery/pkg/util/intstr" <ins> utilnet "k8s.io/apimachinery/pkg/util/net" </ins> "k8s.io/apimachinery/pkg/watch" "k8s.io/client-go/kubernetes/scheme" restclientwatch "k8s.io/client-go/rest/watch"


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In , client-go don't share transport when c.Dial is not nil, but kubelet custorm Dial by default, After PR is integrated, the connections between kubelet and kube-apiserver in our cluster with 4000 nodes increases by five times. kubelet share transport by default, one kubelet only keep one connection to kube-apiserver. start kubelet with rotate-certificates=false Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ):EulerOS 2.9 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /sig node
/triage accepted /assign
how many connections between kubelet and kube apiserver on one node after PR in your case? it's mentioned it increases by five times, does that mean before the PR, there is one connection, after the PR, there are 5 or 6 connections?
before the PR, there is one connection, after the PR, there 3 connections, there are some customization capabilities in our product, and we'll have another two connections. In our kubelet startup parameters, rotate-certificates=false
Some increase was expected, since multiple REST clients constructed from a config with a custom dialer cannot safely share a transport, and client-go constructs a REST client for each API group/version accessed. Something like would be required to rework how client-go constructs clients to start sharing REST clients above the transport level between API groups/versions. I think this is working as intended until client-go client construction is reworked.
do you know where in the code the 5 connections are created? are they all safe to share the transport? also are the 5 connections all active all the time? if not, IdleConnTimeout should cause the connection to be closed.
I test the code on master branch, if I change rotateCertificates to false, there are 6 connections between kubelet and kube-apiserver ! !
transport will share when config.Transport not nil if rotateCertificates is true, config.Transport will been set in kubeletcertificate.UpdateTransport but if rotateCertificates is false, config.Transport will not been set.
/area kubelet Any update on this issue?
I will work on this issue
Hi if rotate-certificates=false kubelet customizes the dialer to provide a closeAllConns to close the connection when the connection is dead but not been closed. In , there is a discuss about another solution after client-go has HTTP 2.0 health check, so we don't need this change. So we can solve the current problem by reverting . I have submitted to revert , and this change requires your final confirmation. Thanks you.
are we guaranteed kubelets always speak to kube-apiserver over http/2? if they use a load-balancer that is not http/2 capable, isn't the "close all connections" fix still required as a backstop?
I thought http1.1 wasn't affected because it will always try to connect a new connection
I'm not confident in that. The original issue dated back to before client-go defaulted to http/2, and we definitely saw issues with stuck TCP connections then.
yeah, I could reproduce it with http1.1, however, it can be solved using the transport CloseIdleConnections method please take a look , the last commit needs more work, but I think that the 2 first commits exposeing the CloseIdleConnections() are useful
ok, this seems to work , but it will be nice to have more feedback would you be able to test and see if this solve your issue?
Hi aojea, I have test on v1.21.1 before merge , there are 6 connections to kube-apiserver from kubelet ! after merge , there is only 1 connection to kube-apiserver from kubelet ! so, can solve my issue. thank you very much.

func testRESTClientWithConfig(t testing.TB, srv *httptest.Server, contentConfig ClientContentConfig) *RESTClient { base, _ := url.Parse("http://localhost") <ins> var c *http.Client </ins> if srv != nil { var err error base, err = url.Parse(srv.URL) if err != nil { t.Fatalf("failed to parse test URL: %v", err) } <ins> c = srv.Client() </ins> } versionedAPIPath := defaultResourcePathWithPrefix("", "", "", "") <del> client, err := NewRESTClient(base, versionedAPIPath, contentConfig, nil, nil) </del> <ins> client, err := NewRESTClient(base, versionedAPIPath, contentConfig, nil, c) </ins> if err != nil { t.Fatalf("failed to create a client: %v", err) }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In , client-go don't share transport when c.Dial is not nil, but kubelet custorm Dial by default, After PR is integrated, the connections between kubelet and kube-apiserver in our cluster with 4000 nodes increases by five times. kubelet share transport by default, one kubelet only keep one connection to kube-apiserver. start kubelet with rotate-certificates=false Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ):EulerOS 2.9 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /sig node
/triage accepted /assign
how many connections between kubelet and kube apiserver on one node after PR in your case? it's mentioned it increases by five times, does that mean before the PR, there is one connection, after the PR, there are 5 or 6 connections?
before the PR, there is one connection, after the PR, there 3 connections, there are some customization capabilities in our product, and we'll have another two connections. In our kubelet startup parameters, rotate-certificates=false
Some increase was expected, since multiple REST clients constructed from a config with a custom dialer cannot safely share a transport, and client-go constructs a REST client for each API group/version accessed. Something like would be required to rework how client-go constructs clients to start sharing REST clients above the transport level between API groups/versions. I think this is working as intended until client-go client construction is reworked.
do you know where in the code the 5 connections are created? are they all safe to share the transport? also are the 5 connections all active all the time? if not, IdleConnTimeout should cause the connection to be closed.
I test the code on master branch, if I change rotateCertificates to false, there are 6 connections between kubelet and kube-apiserver ! !
transport will share when config.Transport not nil if rotateCertificates is true, config.Transport will been set in kubeletcertificate.UpdateTransport but if rotateCertificates is false, config.Transport will not been set.
/area kubelet Any update on this issue?
I will work on this issue
Hi if rotate-certificates=false kubelet customizes the dialer to provide a closeAllConns to close the connection when the connection is dead but not been closed. In , there is a discuss about another solution after client-go has HTTP 2.0 health check, so we don't need this change. So we can solve the current problem by reverting . I have submitted to revert , and this change requires your final confirmation. Thanks you.
are we guaranteed kubelets always speak to kube-apiserver over http/2? if they use a load-balancer that is not http/2 capable, isn't the "close all connections" fix still required as a backstop?
I thought http1.1 wasn't affected because it will always try to connect a new connection
I'm not confident in that. The original issue dated back to before client-go defaulted to http/2, and we definitely saw issues with stuck TCP connections then.
yeah, I could reproduce it with http1.1, however, it can be solved using the transport CloseIdleConnections method please take a look , the last commit needs more work, but I think that the 2 first commits exposeing the CloseIdleConnections() are useful
ok, this seems to work , but it will be nice to have more feedback would you be able to test and see if this solve your issue?
Hi aojea, I have test on v1.21.1 before merge , there are 6 connections to kube-apiserver from kubelet ! after merge , there is only 1 connection to kube-apiserver from kubelet ! so, can solve my issue. thank you very much.

}) } } <ins> func TestReuseRequest(t *testing.T) { var tests = []struct { name        string enableHTTP2 bool }{ {"HTTP1", false}, {"HTTP2", true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write([]byte(r.RemoteAddr)) })) ts.EnableHTTP2 = tt.enableHTTP2 ts.StartTLS() defer ts.Close() ctx, cancel := context.WithCancel(context.Background()) defer cancel() c := testRESTClient(t, ts) req1, err := c.Verb("GET"). Prefix("foo"). DoRaw(ctx) if err != nil { t.Fatalf("Unexpected error: %v", err) } req2, err := c.Verb("GET"). Prefix("foo"). DoRaw(ctx) if err != nil { t.Fatalf("Unexpected error: %v", err) } if string(req1) != string(req2) { t.Fatalf("Expected %v to be equal to %v", string(req1), string(req2)) } }) } } func TestHTTP1DoNotReuseRequestAfterTimeout(t *testing.T) { var tests = []struct { name        string enableHTTP2 bool }{ {"HTTP1", false}, {"HTTP2", true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { done := make(chan struct{}) ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { t.Logf("TEST Connected from %v on %vn", r.RemoteAddr, r.URL.Path) if r.URL.Path == "/hang" { t.Logf("TEST hanging %vn", r.RemoteAddr) <-done } w.Write([]byte(r.RemoteAddr)) })) ts.EnableHTTP2 = tt.enableHTTP2 ts.StartTLS() defer ts.Close() // close hanging connection before shutting down the http server defer close(done) ctx, cancel := context.WithCancel(context.Background()) defer cancel() transport, ok := ts.Client().Transport.(*http.Transport) if !ok { t.Fatalf("failed to assert *http.Transport") } config := &Config{ Host:      ts.URL, Transport: utilnet.SetTransportDefaults(transport), Timeout:   100 * time.Millisecond, // These fields are required to create a REST client. ContentConfig: ContentConfig{ GroupVersion:         &schema.GroupVersion{}, NegotiatedSerializer: &serializer.CodecFactory{}, }, } if !tt.enableHTTP2 { config.TLSClientConfig.NextProtos = []string{"http/1.1"} } c, err := RESTClientFor(config) if err != nil { t.Fatalf("failed to create REST client: %v", err) } req1, err := c.Verb("GET"). Prefix("foo"). DoRaw(ctx) if err != nil { t.Fatalf("Unexpected error: %v", err) } _, err = c.Verb("GET"). Prefix("/hang"). DoRaw(ctx) if err == nil { t.Fatalf("Expected error") } req2, err := c.Verb("GET"). Prefix("foo"). DoRaw(ctx) if err != nil { t.Fatalf("Unexpected error: %v", err) } // http1 doesn't reuse the connection after it times if tt.enableHTTP2 != (string(req1) == string(req2)) { if tt.enableHTTP2 { t.Fatalf("Expected %v to be the same as %v", string(req1), string(req2)) } else { t.Fatalf("Expected %v to be different to %v", string(req1), string(req2)) } } }) } } func TestTransportConcurrency(t *testing.T) { const numReqs = 10 var tests = []struct { name        string enableHTTP2 bool }{ {"HTTP1", false}, {"HTTP2", true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { ts := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { t.Logf("Connected from %v %v", r.RemoteAddr, r.URL) fmt.Fprintf(w, "%v", r.FormValue("echo")) })) ts.EnableHTTP2 = tt.enableHTTP2 ts.StartTLS() defer ts.Close() var wg sync.WaitGroup wg.Add(numReqs) c := testRESTClient(t, ts) reqs := make(chan string) defer close(reqs) for i := 0; i < 4; i++ { go func() { for req := range reqs { res, err := c.Get().Param("echo", req).DoRaw(context.Background()) if err != nil { t.Errorf("error on req %s: %v", req, err) wg.Done() continue } if string(res) != req { t.Errorf("body of req %s = %q; want %q", req, res, req) } wg.Done() } }() } for i := 0; i < numReqs; i++ { reqs <- fmt.Sprintf("request-%d", i) } wg.Wait() }) } } </ins>


Question:
is wrong as it says default label can be specified, but KEP and other lines of changelog mentions annotation. Changelog is correct and consistent about default container feature for kubectl. I'm still not able to get this feature to work actually for . Kubernetes version (use ): v1.21.0
/triage accepted thanks for reporting early!
/remove-kind bug /kind documentation
/sig release

- [Deprecation of PodSecurityPolicy](#deprecation-of-podsecuritypolicy) - [Kubernetes API Reference Documentation](#kubernetes-api-reference-documentation) - [Kustomize Updates in Kubectl](#kustomize-updates-in-kubectl) <del> - [Default Container Labels](#default-container-labels) </del> <ins> - [Default Container Annotation](#default-container-annotation) </ins> - [Immutable Secrets and ConfigMaps](#immutable-secrets-and-configmaps) - [Structured Logging in Kubelet](#structured-logging-in-kubelet) - [Storage Capacity Tracking](#storage-capacity-tracking)


Question:
is wrong as it says default label can be specified, but KEP and other lines of changelog mentions annotation. Changelog is correct and consistent about default container feature for kubectl. I'm still not able to get this feature to work actually for . Kubernetes version (use ): v1.21.0
/triage accepted thanks for reporting early!
/remove-kind bug /kind documentation
/sig release

[Kustomize](https://github.com/kubernetes-sigs/kustomize) version in kubectl had a jump from v2.0.3 to [v4.0.5](https://github.com/kubernetes/kubernetes/pull/98946). Kustomize is now treated as a library and future updates will be less sporadic. <del> ### Default Container Labels </del> <ins> ### Default Container Annotation </ins> <del> Pod with multiple containers can use `kubectl.kubernetes.io/default-container` label to have a container preselected for kubectl commands. More can be read in [KEP-2227](https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/2227-kubectl-default-container/README.md). </del> <ins> Pod with multiple containers can use `kubectl.kubernetes.io/default-container` annotation to have a container preselected for kubectl commands. More can be read in [KEP-2227](https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/2227-kubectl-default-container/README.md). </ins> ### Immutable Secrets and ConfigMaps


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --panic as below when an Endpoint doesn't have zone. should print zone as if an Endpoint's zone is unset. not set "" for Nodes. Pods and export them as a service. kubectl to describe the corresponding EndpointSlice. Kubernetes version (use ): v1.21.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /sig cli /assign

w.Write(LEVEL_2, "NodeName:t%sn", nodeNameText) zoneText := "<unset>" <del> if endpoint.NodeName != nil { </del> <ins> if endpoint.Zone != nil { </ins> zoneText = *endpoint.Zone } w.Write(LEVEL_2, "Zone:t%sn", zoneText)


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --panic as below when an Endpoint doesn't have zone. should print zone as if an Endpoint's zone is unset. not set "" for Nodes. Pods and export them as a service. kubectl to describe the corresponding EndpointSlice. Kubernetes version (use ): v1.21.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /sig cli /assign

Addresses:  []string{"1.2.3.6", "1.2.3.7"}, Conditions: discoveryv1.EndpointConditions{Ready: utilpointer.BoolPtr(true)}, TargetRef:  &corev1.ObjectReference{Kind: "Pod", Name: "test-124"}, <ins> NodeName:   utilpointer.StringPtr("node-2"), </ins> }, }, Ports: []discoveryv1.EndpointPort{


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --panic as below when an Endpoint doesn't have zone. should print zone as if an Endpoint's zone is unset. not set "" for Nodes. Pods and export them as a service. kubectl to describe the corresponding EndpointSlice. Kubernetes version (use ): v1.21.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /sig cli /assign

Ready:    true Hostname:   <unset> TargetRef:  Pod/test-124 <del> NodeName:   <unset> </del> <ins> NodeName:   node-2 </ins> Zone:       <unset> Events:         <none>` + "n", },


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The default fluent-bit configuration for GCE Windows nodes produces unparsed JSON logs and does not forward container logs. Container log output is produced. fluent-bit on a given windows image. a GCE Windows e2e cluster and view the stackdriver output for pods. Kubernetes version (use ): 1.20+ Cloud provider or hardware configuration: GKE/GCE OS (e.g: ): Windows Kernel (e.g. ): Any version Install tools: fluent-bit Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows

# Windows path it'll use it instead of the default unzipper. # See: https://github.com/containerd/containerd/issues/1896 Add-MachineEnvironmentPath -Path $PIGZ_ROOT <del> # Add process exclusion for Windows Defender to boost performance. </del> <ins> # Add process exclusion for Windows Defender to boost performance. </ins> Add-MpPreference -ExclusionProcess "$PIGZ_ROOTunpigz.exe" Log-Output "Installed Pigz $PIGZ_VERSION" } else {


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The default fluent-bit configuration for GCE Windows nodes produces unparsed JSON logs and does not forward container logs. Container log output is produced. fluent-bit on a given windows image. a GCE Windows e2e cluster and view the stackdriver output for pods. Kubernetes version (use ): 1.20+ Cloud provider or hardware configuration: GKE/GCE OS (e.g: ): Windows Kernel (e.g. ): Any version Install tools: fluent-bit Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows

# TODO(pjh): move the logging agent code below into a separate # module; it was put here temporarily to avoid disrupting the file layout in # the K8s release machinery. <del> $LOGGINGAGENT_VERSION = '1.6.0' </del> <ins> $LOGGINGAGENT_VERSION = '1.7.3' </ins> $LOGGINGAGENT_ROOT = 'C:fluent-bit' $LOGGINGAGENT_SERVICE = 'fluent-bit' $LOGGINGAGENT_CMDLINE = '*fluent-bit.exe*' <del> $LOGGINGEXPORTER_VERSION = 'v0.10.3' </del> <ins> $LOGGINGEXPORTER_VERSION = 'v0.16.2' </ins> $LOGGINGEXPORTER_ROOT = 'C:flb-exporter' $LOGGINGEXPORTER_SERVICE = 'flb-exporter' $LOGGINGEXPORTER_CMDLINE = '*flb-exporter.exe*'


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The default fluent-bit configuration for GCE Windows nodes produces unparsed JSON logs and does not forward container logs. Container log output is produced. fluent-bit on a given windows image. a GCE Windows e2e cluster and view the stackdriver output for pods. Kubernetes version (use ): 1.20+ Cloud provider or hardware configuration: GKE/GCE OS (e.g: ): Windows Kernel (e.g. ): Any version Install tools: fluent-bit Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows

$fluentbit_parser_file = "$LOGGINGAGENT_ROOTconfparsers.conf" $PARSERS_CONFIG | Out-File -FilePath $fluentbit_parser_file -Encoding ASCII <ins> # Create directory for all the log position files. New-Item -Type Directory -Path "/var/run/google-fluentbit/pos-files/" </ins> Log-Output "Wrote logging config to $fluentbit_parser_file" }


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The default fluent-bit configuration for GCE Windows nodes produces unparsed JSON logs and does not forward container logs. Container log output is produced. fluent-bit on a given windows image. a GCE Windows e2e cluster and view the stackdriver output for pods. Kubernetes version (use ): 1.20+ Cloud provider or hardware configuration: GKE/GCE OS (e.g: ): Windows Kernel (e.g. ): Any version Install tools: fluent-bit Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows

[SERVICE] Flush         5 Grace         120 <del> Log_Level     debug </del> <ins> Log_Level     info </ins> Log_File      /var/log/fluentbit.log Daemon        off Parsers_File  parsers.conf


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The default fluent-bit configuration for GCE Windows nodes produces unparsed JSON logs and does not forward container logs. Container log output is produced. fluent-bit on a given windows image. a GCE Windows e2e cluster and view the stackdriver output for pods. Kubernetes version (use ): 1.20+ Cloud provider or hardware configuration: GKE/GCE OS (e.g: ): Windows Kernel (e.g. ): Any version Install tools: fluent-bit Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows

# # storage.backlog.mem_limit 5M [INPUT] Name         winlog Interval_Sec 2 # Channels Setup,Windows PowerShell Channels     application,system,security Tag          winevent.raw <del> DB           winlog.sqlite   # </del> <ins> DB           /var/run/google-fluentbit/pos-files/winlog.db </ins> # Json Log Example: # {"log":"[info:2016-02-16T16:04:05.930-08:00] Some log text heren","stream":"stdout","time":"2016-02-17T00:04:05.931087621Z"}


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The default fluent-bit configuration for GCE Windows nodes produces unparsed JSON logs and does not forward container logs. Container log output is produced. fluent-bit on a given windows image. a GCE Windows e2e cluster and view the stackdriver output for pods. Kubernetes version (use ): 1.20+ Cloud provider or hardware configuration: GKE/GCE OS (e.g: ): Windows Kernel (e.g. ): Any version Install tools: fluent-bit Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows

Alias            kube_containers Tag              kube_<namespace_name>_<pod_name>_<container_name> Tag_Regex        (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)- <del> Path             /var/log/containers/*.log </del> Mem_Buf_Limit    5MB Skip_Long_Lines  On Refresh_Interval 5 <del> DB               flb_kube.db # Settings from fluentd missing here. # tag reform.* # format json # time_key time # time_format %Y-%m-%dT%H:%M:%S.%NZ </del> <ins> Path             C:varlogcontainers*.log DB               /var/run/google-fluentbit/pos-files/flb_kube.db [FILTER] Name         parser Match        kube_* Key_Name     log Reserve_Data True Parser       docker Parser       containerd </ins> # Example: <del> # I0204 07:32:30.020537    3368 server.go:1048] POST /stats/container/: (13.972191ms) 200 [[Go-http-client/1.1] 10.244.1.3:40537] </del> <ins> # I0928 03:15:50.440223    4880 main.go:51] Starting CSI-Proxy Server ... </ins> [INPUT] Name             tail <del> Alias            kubelet Tag              kubelet #Multiline        on #Multiline_Flush  5 </del> <ins> Alias            csi-proxy Tag              csi-proxy </ins> Mem_Buf_Limit    5MB Skip_Long_Lines  On Refresh_Interval 5 <del> Path /etc/kubernetes/logs/kubelet.log DB               /etc/kubernetes/logs/gcp-kubelet.db # Copied from fluentbit config. How is this used ? In match stages ? Parser_Firstline /^wd{4}/ Parser_1         ^(?<severity>w)(?<time>d{4} [^s]*)s+(?<pid>d+)s+(?<source>[^ ]]+)] (?<message>.*)/ # missing from fluentbit #   time_format %m%d %H:%M:%S.%N </del> <ins> Path             /etc/kubernetes/logs/csi-proxy.log DB               /var/run/google-fluentbit/pos-files/csi-proxy.db Multiline        On Parser_Firstline glog </ins> <del> # Example: # I0928 03:15:50.440223    4880 main.go:51] Starting CSI-Proxy Server ... </del> <ins> # I1118 21:26:53.975789       6 proxier.go:1096] Port "nodePort for kube-system/default-http-backend:http" (:31429/tcp) was open before and is still needed </ins> [INPUT] Name             tail <del> Alias            csi-proxy Tag              csi-proxy #Multiline        on #Multiline_Flush  5 </del> <ins> Alias            kube-proxy Tag              kube-proxy </ins> Mem_Buf_Limit    5MB Skip_Long_Lines  On Refresh_Interval 5 <del> Path             /etc/kubernetes/logs/csi-proxy.log DB               /etc/kubernetes/logs/gcp-csi-proxy.db # Copied from fluentbit config. How is this used ? In match stages ? Parser_Firstline /^wd{4}/ Parser_1         ^(?<severity>w)(?<time>d{4} [^s]*)s+(?<pid>d+)s+(?<source>[^ ]]+)] (?<message>.*)/ # missing from fluentbit #   time_format %m%d %H:%M:%S.%N </del> <ins> Path             /etc/kubernetes/logs/kube-proxy.log DB               /var/run/google-fluentbit/pos-files/kube-proxy.db Multiline        On Parser_Firstline glog </ins> # Example: # time="2019-12-10T21:27:59.836946700Z" level=info msg="loading plugin "io.containerd.grpc.v1.cri"..." type=io.containerd.grpc.v1 [INPUT] Name             tail Alias            container-runtime <del> Tag container-runtime #Multiline        on #Multiline_Flush  5 </del> <ins> Tag              container-runtime </ins> Mem_Buf_Limit    5MB Skip_Long_Lines  On Refresh_Interval 5 Path             /etc/kubernetes/logs/containerd.log <del> DB               /etc/kubernetes/logs/gcp-containerd.log.pos # Copied from fluentbit config. How is this used ? In match stages ? Parser_Firstline /^wd{4}/ Parser_1         ^(?<severity>w)(?<time>d{4} [^s]*)s+(?<pid>d+)s+(?<source>[^ ]]+)] (?<message>.*)/ </del> <ins> DB               /var/run/google-fluentbit/pos-files/container-runtime.db # TODO: Add custom parser for containerd logs once format is settled. </ins> <del> # missing from fluentbit #   time_format %m%d %H:%M:%S.%N # I1118 21:26:53.975789       6 proxier.go:1096] Port "nodePort for kube-system/default-http-backend:http" (:31429/tcp) was open before and is still needed </del> <ins> # Example: # I0204 07:32:30.020537    3368 server.go:1048] POST /stats/container/: (13.972191ms) 200 [[Go-http-client/1.1] 10.244.1.3:40537] </ins> [INPUT] Name             tail <del> Alias            kube-proxy Tag              kube-proxy #Multiline        on #Multiline_Flush  5 </del> <ins> Alias            kubelet Tag              kubelet </ins> Mem_Buf_Limit    5MB Skip_Long_Lines  On Refresh_Interval 5 <del> Path             /etc/kubernetes/logs/kube-proxy.log DB               /etc/kubernetes/logs/gcp-kubeproxy.db # Copied from fluentbit config. How is this used ? In match stages ? Parser_Firstline /^wd{4}/ Parser_1         ^(?<severity>w)(?<time>d{4} [^s]*)s+(?<pid>d+)s+(?<source>[^ ]]+)] (?<message>.*)/ # missing from fluentbit #   time_format %m%d %H:%M:%S.%N </del> <ins> Path             /etc/kubernetes/logs/kubelet.log DB               /var/run/google-fluentbit/pos-files/kubelet.db Multiline        On Parser_Firstline glog </ins> [FILTER] Name        modify Match       * Hard_rename log message <del> # [OUTPUT] #    Name        http #    Match       * #    Host        127.0.0.1 #    Port        2021 #    URI         /logs #    header_tag  FLUENT-TAG #    Format      msgpack #    Retry_Limit 2 </del> <ins> [FILTER] Name         parser Match        kube_* Key_Name     message Reserve_Data True Parser       glog Parser       json </ins> [OUTPUT] <del> name  stackdriver match * </del> <ins> Name        http Match       * Host        127.0.0.1 Port        2021 URI         /logs header_tag  FLUENT-TAG Format      msgpack Retry_Limit 2 </ins> '@ # Fluentbit parsers config file $PARSERS_CONFIG = @' [PARSER] Name        docker Format      json


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The default fluent-bit configuration for GCE Windows nodes produces unparsed JSON logs and does not forward container logs. Container log output is produced. fluent-bit on a given windows image. a GCE Windows e2e cluster and view the stackdriver output for pods. Kubernetes version (use ): 1.20+ Cloud provider or hardware configuration: GKE/GCE OS (e.g: ): Windows Kernel (e.g. ): Any version Install tools: fluent-bit Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows

Time_Key    timestamp Time_Format %Y-%m-%dT%H:%M:%S.%L%z <del> # ---------- [PARSER] Name   json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name         docker Format       json Time_Key     time Time_Format  %Y-%m-%dT%H:%M:%S.%L Time_Keep    On </del> [PARSER] Name        syslog-rfc5424 Format      regex


Question:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The default fluent-bit configuration for GCE Windows nodes produces unparsed JSON logs and does not forward container logs. Container log output is produced. fluent-bit on a given windows image. a GCE Windows e2e cluster and view the stackdriver output for pods. Kubernetes version (use ): 1.20+ Cloud provider or hardware configuration: GKE/GCE OS (e.g: ): Windows Kernel (e.g. ): Any version Install tools: fluent-bit Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows

$PARSERS_CONFIG | Out-File -FilePath $fluentbit_parser_file -Encoding ASCII # Create directory for all the log position files. <del> New-Item -Type Directory -Path "/var/run/google-fluentbit/pos-files/" </del> <ins> New-Item -Type Directory -Path "/var/run/google-fluentbit/pos-files/" -Force | Out-Null </ins> Log-Output "Wrote logging config to $fluentbit_parser_file" }


Question:
I am aware that the following is a corner case that does not occur in day-to-day business, but it took me a long while to debug, so I thought I would raise this issue to save others from the same pain. Deploying the following job with empty imagePullSecrets leads to it being stuck in forever, and status looking the following way: The status should look something like: This would allow my autoscaler to kick in and add the corresponding nodes. Run Kubernetes version (use ):Cloud provider or hardware configuration: AWS EKS OS (e.g: ): MacOSX 11.3 Kernel (e.g. ): Darwin Kernel Version 20.4.0 Install tools: brew
/sig scheduling (I assume)
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig scheduling (I assume) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Scheduler log
Should we instead forbid empty items in ? Is there a use case for using as ?
For me personally that would be totally fine I had no use case for it. It was more produced by chance as I wasn't aware that I had to handle the case separately. An error which led to an early failure would have been fine by me and solved all my problems in tracking this down.
/assign as
Forbidding it is the appropriate solution. However, we can't do it for backwards compatibility of the API. /triage accepted

func updatePod(client clientset.Interface, pod *v1.Pod, condition *v1.PodCondition, nominatedNode string) error { klog.V(3).InfoS("Updating pod condition", "pod", klog.KObj(pod), "conditionType", condition.Type, "conditionStatus", condition.Status, "conditionReason", condition.Reason) <del> podCopy := pod.DeepCopy() </del> <ins> podStatusCopy := pod.Status.DeepCopy() </ins> // NominatedNodeName is updated only if we are trying to set it, and the value is // different from the existing one. <del> if !podutil.UpdatePodCondition(&podCopy.Status, condition) && </del> <ins> if !podutil.UpdatePodCondition(podStatusCopy, condition) && </ins> (len(nominatedNode) == 0 || pod.Status.NominatedNodeName == nominatedNode) { return nil } if nominatedNode != "" { <del> podCopy.Status.NominatedNodeName = nominatedNode </del> <ins> podStatusCopy.NominatedNodeName = nominatedNode </ins> } <del> return util.PatchPod(client, pod, podCopy) </del> <ins> return util.PatchPodStatus(client, pod, podStatusCopy) </ins> } // assume signals to the cache that a pod is already in the cache, so that binding can be asynchronous.


Question:
I am aware that the following is a corner case that does not occur in day-to-day business, but it took me a long while to debug, so I thought I would raise this issue to save others from the same pain. Deploying the following job with empty imagePullSecrets leads to it being stuck in forever, and status looking the following way: The status should look something like: This would allow my autoscaler to kick in and add the corresponding nodes. Run Kubernetes version (use ):Cloud provider or hardware configuration: AWS EKS OS (e.g: ): MacOSX 11.3 Kernel (e.g. ): Darwin Kernel Version 20.4.0 Install tools: brew
/sig scheduling (I assume)
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig scheduling (I assume) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Scheduler log
Should we instead forbid empty items in ? Is there a use case for using as ?
For me personally that would be totally fine I had no use case for it. It was more produced by chance as I wasn't aware that I had to handle the case separately. An error which led to an early failure would have been fine by me and solved all my problems in tracking this down.
/assign as
Forbidding it is the appropriate solution. However, we can't do it for backwards compatibility of the API. /triage accepted

return GetPodStartTime(pod1).Before(GetPodStartTime(pod2)) } <del> // PatchPod calculates the delta bytes change from <old> to <new>, </del> <ins> // PatchPodStatus calculates the delta bytes change from <old.Status> to <newStatus>, </ins> // and then submit a request to API server to patch the pod changes. <del> func PatchPod(cs kubernetes.Interface, old *v1.Pod, new *v1.Pod) error { oldData, err := json.Marshal(old) </del> <ins> func PatchPodStatus(cs kubernetes.Interface, old *v1.Pod, newStatus *v1.PodStatus) error { if newStatus == nil { return nil } oldData, err := json.Marshal(v1.Pod{Status: old.Status}) </ins> if err != nil { return err } <del> newData, err := json.Marshal(new) </del> <ins> newData, err := json.Marshal(v1.Pod{Status: *newStatus}) </ins> if err != nil { return err }


Question:
I am aware that the following is a corner case that does not occur in day-to-day business, but it took me a long while to debug, so I thought I would raise this issue to save others from the same pain. Deploying the following job with empty imagePullSecrets leads to it being stuck in forever, and status looking the following way: The status should look something like: This would allow my autoscaler to kick in and add the corresponding nodes. Run Kubernetes version (use ):Cloud provider or hardware configuration: AWS EKS OS (e.g: ): MacOSX 11.3 Kernel (e.g. ): Darwin Kernel Version 20.4.0 Install tools: brew
/sig scheduling (I assume)
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig scheduling (I assume) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Scheduler log
Should we instead forbid empty items in ? Is there a use case for using as ?
For me personally that would be totally fine I had no use case for it. It was more produced by chance as I wasn't aware that I had to handle the case separately. An error which led to an early failure would have been fine by me and solved all my problems in tracking this down.
/assign as
Forbidding it is the appropriate solution. However, we can't do it for backwards compatibility of the API. /triage accepted

if len(p.Status.NominatedNodeName) == 0 { continue } <del> podCopy := p.DeepCopy() podCopy.Status.NominatedNodeName = "" if err := PatchPod(cs, p, podCopy); err != nil { </del> <ins> podStatusCopy := p.Status.DeepCopy() podStatusCopy.NominatedNodeName = "" if err := PatchPodStatus(cs, p, podStatusCopy); err != nil { </ins> errs = append(errs, err) } }


Question:
I am aware that the following is a corner case that does not occur in day-to-day business, but it took me a long while to debug, so I thought I would raise this issue to save others from the same pain. Deploying the following job with empty imagePullSecrets leads to it being stuck in forever, and status looking the following way: The status should look something like: This would allow my autoscaler to kick in and add the corresponding nodes. Run Kubernetes version (use ):Cloud provider or hardware configuration: AWS EKS OS (e.g: ): MacOSX 11.3 Kernel (e.g. ): Darwin Kernel Version 20.4.0 Install tools: brew
/sig scheduling (I assume)
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig scheduling (I assume) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Scheduler log
Should we instead forbid empty items in ? Is there a use case for using as ?
For me personally that would be totally fine I had no use case for it. It was more produced by chance as I wasn't aware that I had to handle the case separately. An error which led to an early failure would have been fine by me and solved all my problems in tracking this down.
/assign as
Forbidding it is the appropriate solution. However, we can't do it for backwards compatibility of the API. /triage accepted

package util import ( <ins> "context" </ins> "fmt" <ins> "github.com/google/go-cmp/cmp" </ins> "testing" "time"


Question:
I am aware that the following is a corner case that does not occur in day-to-day business, but it took me a long while to debug, so I thought I would raise this issue to save others from the same pain. Deploying the following job with empty imagePullSecrets leads to it being stuck in forever, and status looking the following way: The status should look something like: This would allow my autoscaler to kick in and add the corresponding nodes. Run Kubernetes version (use ):Cloud provider or hardware configuration: AWS EKS OS (e.g: ): MacOSX 11.3 Kernel (e.g. ): Darwin Kernel Version 20.4.0 Install tools: brew
/sig scheduling (I assume)
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig scheduling (I assume) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Scheduler log
Should we instead forbid empty items in ? Is there a use case for using as ?
For me personally that would be totally fine I had no use case for it. It was more produced by chance as I wasn't aware that I had to handle the case separately. An error which led to an early failure would have been fine by me and solved all my problems in tracking this down.
/assign as
Forbidding it is the appropriate solution. However, we can't do it for backwards compatibility of the API. /triage accepted

}) } } <ins> func TestPatchPodStatus(t *testing.T) { tests := []struct { name           string pod            v1.Pod statusToUpdate v1.PodStatus }{ { name: "Should update pod conditions successfully", pod: v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: "ns", Name:      "pod1", }, Spec: v1.PodSpec{ ImagePullSecrets: []v1.LocalObjectReference{{Name: "foo"}}, }, }, statusToUpdate: v1.PodStatus{ Conditions: []v1.PodCondition{ { Type:   v1.PodScheduled, Status: v1.ConditionFalse, }, }, }, }, { // ref: #101697, #94626 - ImagePullSecrets are allowed to have empty secret names // which would fail the 2-way merge patch generation on Pod patches // due to the mergeKey being the name field name: "Should update pod conditions successfully on a pod Spec with secrets with empty name", pod: v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: "ns", Name:      "pod2", }, Spec: v1.PodSpec{ // this will serialize to imagePullSecrets:[{}] ImagePullSecrets: make([]v1.LocalObjectReference, 1), }, }, statusToUpdate: v1.PodStatus{ Conditions: []v1.PodCondition{ { Type:   v1.PodScheduled, Status: v1.ConditionFalse, }, }, }, }, } client := clientsetfake.NewSimpleClientset() for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { _, err := client.CoreV1().Pods(tc.pod.Namespace).Create(context.TODO(), &tc.pod, metav1.CreateOptions{}) if err != nil { t.Fatal(err) } err = PatchPodStatus(client, &tc.pod, &tc.statusToUpdate) if err != nil { t.Fatal(err) } retrievedPod, err := client.CoreV1().Pods(tc.pod.Namespace).Get(context.TODO(), tc.pod.Name, metav1.GetOptions{}) if err != nil { t.Fatal(err) } if diff := cmp.Diff(tc.statusToUpdate, retrievedPod.Status); diff != "" { t.Errorf("unexpected pod status (-want,+got):n%s", diff) } }) } } </ins>


Question:
This issue is a bucket placeholder for collaborating on the "Known Issues" additions for the 1.22 Release Notes(). If you know of issues or API changes that are going out in 1.22, please comment here so that we can coordinate incorporating information about these changes in the Release Notes. /assign /sig release /milestone v1.22
GitHub didn't allow me to assign the following users: simcard0000. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This issue is a bucket placeholder for collaborating on the "Known Issues" additions for the 1.22 Release Notes(). If you know of issues or API changes that are going out in 1.19, please comment here so that we can coordinate incorporating information about these changes in the Release Notes. /sig release /milestone v1.22 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
1.19 or 1.22 ?
/assign
For the 1.21 example in , should be changed to
Please add the following to the Known Issues section of the 1.22 Release Notes: CC
Can you also mention the CSIMigrationVSphere feature gate in the note as well?
what should I mention about the CSIMigrationVSphere feature gate? If someone uses vSphere CSI Driver, he/she should not upgrade to v1.22 whether CSIMigrationVSphere is on or off. If someone does not use vSphere CSI Driver, then we also don't need to worry about this feature gate.
If the feature gate is enabled, they should also hold off on upgrading to 1.22
The main concern as far as k8s release is - it is intree to CSI migration of vSphere volumes that is broken in 1.22. So I think our release note should say something like:

<ins> # Maintainers Eric Paris <eparis@redhat.com> </ins>


Question:
This issue is a bucket placeholder for collaborating on the "Known Issues" additions for the 1.22 Release Notes(). If you know of issues or API changes that are going out in 1.22, please comment here so that we can coordinate incorporating information about these changes in the Release Notes. /assign /sig release /milestone v1.22
GitHub didn't allow me to assign the following users: simcard0000. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This issue is a bucket placeholder for collaborating on the "Known Issues" additions for the 1.22 Release Notes(). If you know of issues or API changes that are going out in 1.19, please comment here so that we can coordinate incorporating information about these changes in the Release Notes. /sig release /milestone v1.22 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
1.19 or 1.22 ?
/assign
For the 1.21 example in , should be changed to
Please add the following to the Known Issues section of the 1.22 Release Notes: CC
Can you also mention the CSIMigrationVSphere feature gate in the note as well?
what should I mention about the CSIMigrationVSphere feature gate? If someone uses vSphere CSI Driver, he/she should not upgrade to v1.22 whether CSIMigrationVSphere is on or off. If someone does not use vSphere CSI Driver, then we also don't need to worry about this feature gate.
If the feature gate is enabled, they should also hold off on upgrading to 1.22
The main concern as far as k8s release is - it is intree to CSI migration of vSphere volumes that is broken in 1.22. So I think our release note should say something like:

<ins> #!bash # # bash completion file for core kubecfg commands # # This script provides completion of non replication controller options # # To enable the completions either: #  - place this file in /etc/bash_completion.d #  or #  - copy this file and add the line below to your .bashrc after #    bash completion features are loaded #     . kubecfg # # Note: # Currently, the completions will not work if the apiserver daemon is not # running on localhost on the standard port 8080 __contains_word () { local w word=$1; shift for w in "$@"; do [[ $w = "$word" ]] && return done return 1 } # This should be provided by the bash-completions, but give a really simple # stoopid version just in case.  It works most of the time. if ! declare -F _get_comp_words_by_ref >/dev/null 2>&1; then _get_comp_words_by_ref () { while [ $# -gt 0 ]; do case "$1" in cur) cur=${COMP_WORDS[COMP_CWORD]} ;; prev) prev=${COMP_WORDS[COMP_CWORD-1]} ;; words) words=("${COMP_WORDS[@]}") ;; cword) cword=$COMP_CWORD ;; -n) shift # we don't handle excludes ;; esac shift done } fi __has_service() { local i for ((i=0; i < cword; i++)); do local word=${words[i]} # strip everything after a / so things like pods/[id] match word=${word%%/*} if __contains_word "${word}" "${services[@]}" && ! __contains_word "${words[i-1]}" "${opts[@]}"; then return 0 fi done return 1 } # call kubecfg list $1, # exclude blank lines # skip the header stuff kubecfg prints on the first 2 lines # append $1/ to the first column and use that in compgen __kubecfg_parse_list() { local kubecfg_output if kubecfg_output=$(kubecfg list "$1" 2>/dev/null); then out=($(echo "${kubecfg_output}" | awk -v prefix="$1" '/^$/ {next} NR > 2 {print prefix"/"$1}')) COMPREPLY=( $( compgen -W "${out[*]}" -- "$cur" ) ) fi } _kubecfg_specific_service_match() { case "$cur" in pods/*) __kubecfg_parse_list pods ;; minions/*) __kubecfg_parse_list minions ;; replicationControllers/*) __kubecfg_parse_list replicationControllers ;; services/*) __kubecfg_parse_list services ;; *) if __has_service; then return 0 fi compopt -o nospace COMPREPLY=( $( compgen -S / -W "${services[*]}" -- "$cur" ) ) ;; esac } _kubecfg_service_match() { if __has_service; then return 0 fi COMPREPLY=( $( compgen -W "${services[*]}" -- "$cur" ) ) } _kubecfg() { local opts=( -h -c ) local create_services=(pods replicationControllers services) local update_services=(replicationControllers) local all_services=(pods replicationControllers services minions) local services=("${all_services[@]}") local json_commands=(create update) local all_commands=(create update get list delete stop rm rollingupdate resize) local commands=("${all_commands[@]}") COMPREPLY=() local command local cur prev words cword _get_comp_words_by_ref -n : cur prev words cword if __contains_word "$prev" "${opts[@]}"; then case $prev in -c) _filedir '@(json|yml|yaml)' return 0 ;; -h) return 0 ;; esac fi if [[ "$cur" = -* ]]; then COMPREPLY=( $(compgen -W "${opts[*]}" -- "$cur") ) return 0 fi # if you passed -c, you are limited to create or update if __contains_word "-c" "${words[@]}"; then services=("${create_services[@]}" "${update_services[@]}") commands=("${json_commands[@]}") fi # figure out which command they are running, remembering that arguments to # options don't count as the command!  So a hostname named 'create' won't # trip things up local i for ((i=0; i < cword; i++)); do if __contains_word "${words[i]}" "${commands[@]}" && ! __contains_word "${words[i-1]}" "${opts[@]}"; then command=${words[i]} break fi done # tell the list of possible commands if [[ -z ${command} ]]; then COMPREPLY=( $( compgen -W "${commands[*]}" -- "$cur" ) ) return 0 fi # remove services which you can't update given your command if [[ ${command} == "create" ]]; then services=("${create_services[@]}") elif [[ ${command} == "update" ]]; then services=("${update_services[@]}") fi case $command in create | list) _kubecfg_service_match ;; update | get | delete) _kubecfg_specific_service_match ;; *) ;; esac return 0 } complete -F _kubecfg kubecfg # ex: ts=4 sw=4 et filetype=sh </ins>