Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

go controller.Run(1, stopCh) } <ins> if controller, ok := s.Cert.(dynamiccertificates.ControllerRunner); ok { // runonce to be sure that we have a value. if err := controller.RunOnce(); err != nil { return nil, err } go controller.Run(1, stopCh) } </ins> // runonce to be sure that we have a value. if err := dynamicCertificateController.RunOnce(); err != nil {


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

package podlogs import ( <ins> "bytes" </ins> "crypto/tls" <ins> "crypto/x509" "encoding/base64" </ins> "io/ioutil" "net/url" <ins> "path" </ins> "strings" "testing" "time" "k8s.io/apiserver/pkg/server/dynamiccertificates" "k8s.io/kubernetes/cmd/kube-apiserver/app/options" "k8s.io/kubernetes/test/integration/framework" )


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

"client_ca.go", "dynamic_cafile_content.go", "dynamic_serving_content.go", <ins> "dynamic_sni_content.go", </ins> "named_certificates.go", "static_content.go", "tlsconfig.go",


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

<ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package dynamiccertificates // DynamicFileSNIContent provides a SNICertKeyContentProvider that can dynamically react to new file content type DynamicFileSNIContent struct { *DynamicFileServingContent sniNames []string } var _ Notifier = &DynamicFileSNIContent{} var _ SNICertKeyContentProvider = &DynamicFileSNIContent{} var _ ControllerRunner = &DynamicFileSNIContent{} // NewDynamicSNIContentFromFiles returns a dynamic SNICertKeyContentProvider based on a cert and key filename and explicit names func NewDynamicSNIContentFromFiles(purpose, certFile, keyFile string, sniNames ...string) (*DynamicFileSNIContent, error) { servingContent, err := NewDynamicServingContentFromFiles(purpose, certFile, keyFile) if err != nil { return nil, err } ret := &DynamicFileSNIContent{ DynamicFileServingContent: servingContent, sniNames:                  sniNames, } if err := ret.loadServingCert(); err != nil { return nil, err } return ret, nil } // SNINames returns explicitly set SNI names for the certificate. These are not dynamic. func (c *DynamicFileSNIContent) SNINames() []string { return c.sniNames } </ins>


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

// load SNI certs namedTLSCerts := make([]dynamiccertificates.SNICertKeyContentProvider, 0, len(s.SNICertKeys)) for _, nck := range s.SNICertKeys { <del> tlsCert, err := dynamiccertificates.NewStaticSNICertKeyContentFromFiles(nck.CertFile, nck.KeyFile, nck.Names...) </del> <ins> tlsCert, err := dynamiccertificates.NewDynamicSNIContentFromFiles("sni-serving-cert", nck.CertFile, nck.KeyFile, nck.Names...) </ins> namedTLSCerts = append(namedTLSCerts, tlsCert) if err != nil { return fmt.Errorf("failed to load SNI cert and key: %v", err)


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

go controller.Run(1, stopCh) } <ins> for _, sniCert := range s.SNICerts { if notifier, ok := sniCert.(dynamiccertificates.Notifier); ok { notifier.AddListener(dynamicCertificateController) } if controller, ok := sniCert.(dynamiccertificates.ControllerRunner); ok { // runonce to be sure that we have a value. if err := controller.RunOnce(); err != nil { return nil, err } go controller.Run(1, stopCh) } } </ins> // runonce to be sure that we have a value. if err := dynamicCertificateController.RunOnce(); err != nil {


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

deps = [ "//cmd/kube-apiserver/app/options:go_default_library", "//staging/src/k8s.io/apiserver/pkg/server/dynamiccertificates:go_default_library", <ins> "//staging/src/k8s.io/component-base/cli/flag:go_default_library", </ins> "//test/integration/framework:go_default_library", ], )


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

"time" "k8s.io/apiserver/pkg/server/dynamiccertificates" <ins> "k8s.io/component-base/cli/flag" </ins> "k8s.io/kubernetes/cmd/kube-apiserver/app/options" "k8s.io/kubernetes/test/integration/framework" )


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

} } } <ins> func TestSNICert(t *testing.T) { stopCh := make(chan struct{}) defer close(stopCh) var servingCertPath string _, kubeconfig := framework.StartTestServer(t, stopCh, framework.TestServerSetup{ ModifyServerRunOptions: func(opts *options.ServerRunOptions) { opts.GenericServerRunOptions.MaxRequestBodyBytes = 1024 * 1024 servingCertPath = opts.SecureServing.ServerCert.CertDirectory if err := ioutil.WriteFile(path.Join(servingCertPath, "foo.key"), anotherServerKey, 0644); err != nil { t.Fatal(err) } if err := ioutil.WriteFile(path.Join(servingCertPath, "foo.crt"), anotherServerCert, 0644); err != nil { t.Fatal(err) } dynamiccertificates.FileRefreshDuration = 1 * time.Second opts.SecureServing.SNICertKeys = []flag.NamedCertKey{{ Names:    []string{"foo"}, CertFile: path.Join(servingCertPath, "foo.crt"), KeyFile:  path.Join(servingCertPath, "foo.key"), }} }, }) apiserverURL, err := url.Parse(kubeconfig.Host) if err != nil { t.Fatal(err) } // When we run this the second time, we know which one we are expecting. acceptableCerts := [][]byte{} tlsConfig := &tls.Config{ InsecureSkipVerify: true, VerifyPeerCertificate: func(rawCerts [][]byte, verifiedChains [][]*x509.Certificate) error { acceptableCerts = make([][]byte, 0, len(rawCerts)) for _, r := range rawCerts { acceptableCerts = append(acceptableCerts, r) } return nil }, ServerName: "foo", } conn, err := tls.Dial("tcp", apiserverURL.Host, tlsConfig) if err != nil { t.Fatal(err) } conn.Close() cert, err := tls.LoadX509KeyPair(path.Join(servingCertPath, "foo.crt"), path.Join(servingCertPath, "foo.key")) if err != nil { t.Fatal(err) } expectedCerts := cert.Certificate if len(expectedCerts) != len(acceptableCerts) { var certs []string for _, a := range acceptableCerts { certs = append(certs, base64.StdEncoding.EncodeToString(a)) } t.Fatalf("Unexpected number of certs: %v", strings.Join(certs, ":")) } for i := range expectedCerts { if !bytes.Equal(acceptableCerts[i], expectedCerts[i]) { t.Errorf("expected %q, got %q", base64.StdEncoding.EncodeToString(expectedCerts[i]), base64.StdEncoding.EncodeToString(acceptableCerts[i])) } } if err := ioutil.WriteFile(path.Join(servingCertPath, "foo.key"), serverKey, 0644); err != nil { t.Fatal(err) } if err := ioutil.WriteFile(path.Join(servingCertPath, "foo.crt"), serverCert, 0644); err != nil { t.Fatal(err) } time.Sleep(4 * time.Second) conn2, err := tls.Dial("tcp", apiserverURL.Host, tlsConfig) if err != nil { t.Fatal(err) } conn2.Close() cert, err = tls.X509KeyPair(serverCert, serverKey) if err != nil { t.Fatal(err) } expectedCerts = cert.Certificate if len(expectedCerts) != len(acceptableCerts) { var certs []string for _, a := range acceptableCerts { certs = append(certs, base64.StdEncoding.EncodeToString(a)) } t.Fatalf("Unexpected number of certs: %v", strings.Join(certs, ":")) } for i := range expectedCerts { if !bytes.Equal(acceptableCerts[i], expectedCerts[i]) { t.Errorf("expected %q, got %q", base64.StdEncoding.EncodeToString(expectedCerts[i]), base64.StdEncoding.EncodeToString(acceptableCerts[i])) } } } </ins>


Question:
/kind bug What happened: Kubelet uses "iptables" instead of "ip6tables" in an ipv6-only cluster. This causes failed traffic for services (and probably a lot of other problems). For instance the chain is created in but is missing in ; When a is assigned the fails; What you expected to happen: All components should use in a ipv6-only cluster. should work in an ipv6-only cluster. How to reproduce it (as minimally and precisely as possible): Start a ipv6-only cluster Check iptables Optional; try a service type: LoadBalancer Anything else we need to know?: is hard-coded in finc in "" Environment: Kubernetes version (use ):Cloud provider or hardware configuration: kvm VMs on Ubuntu 16.04 OS (e.g. from /etc/os-release): Own BusyBox system Kernel (e.g. ):Install tools: None Others:
/sig network
/area ipv6
Thanks for reporting this problem. I'll take a look.
is a link to the code with the hard-coded ipv4 protocol.
Thanks, I will send a pr to fix this issue.
/assign
I see this issue too. Also there is some related discussion with issue .
I just pulled the latest from master, and gave a try with IPv6 cluster, but I still see the error, when using kube-proxy + IPVS: E0829 16:51:56. 1 ] Failed to execute iptables-restore: exit status 2 (ip6tables-restore v1.4.21: The protocol family of set KUBE-LOOP-BACK is IPv4, which is not applicable. Were you able to try this with kube-proxy+IPVS?
Sorry, I didn't try ipv6 cluster with ipvs proxier. Those two issue are related, but not exactly the same issue. Seems we already have conclusion in , nobody working on that?

} httpClient := &http.Client{} parsedNodeIP := net.ParseIP(nodeIP) <ins> protocol := utilipt.ProtocolIpv4 if parsedNodeIP != nil && parsedNodeIP.To4() == nil { glog.V(0).Infof("IPv6 node IP (%s), assume IPv6 operation", nodeIP) protocol = utilipt.ProtocolIpv6 } </ins> klet := &Kubelet{ hostname:                       hostname,


Question:
/kind bug What happened: Kubelet uses "iptables" instead of "ip6tables" in an ipv6-only cluster. This causes failed traffic for services (and probably a lot of other problems). For instance the chain is created in but is missing in ; When a is assigned the fails; What you expected to happen: All components should use in a ipv6-only cluster. should work in an ipv6-only cluster. How to reproduce it (as minimally and precisely as possible): Start a ipv6-only cluster Check iptables Optional; try a service type: LoadBalancer Anything else we need to know?: is hard-coded in finc in "" Environment: Kubernetes version (use ):Cloud provider or hardware configuration: kvm VMs on Ubuntu 16.04 OS (e.g. from /etc/os-release): Own BusyBox system Kernel (e.g. ):Install tools: None Others:
/sig network
/area ipv6
Thanks for reporting this problem. I'll take a look.
is a link to the code with the hard-coded ipv4 protocol.
Thanks, I will send a pr to fix this issue.
/assign
I see this issue too. Also there is some related discussion with issue .
I just pulled the latest from master, and gave a try with IPv6 cluster, but I still see the error, when using kube-proxy + IPVS: E0829 16:51:56. 1 ] Failed to execute iptables-restore: exit status 2 (ip6tables-restore v1.4.21: The protocol family of set KUBE-LOOP-BACK is IPv4, which is not applicable. Were you able to try this with kube-proxy+IPVS?
Sorry, I didn't try ipv6 cluster with ipvs proxier. Those two issue are related, but not exactly the same issue. Seems we already have conclusion in , nobody working on that?

nodeIPValidator: validateNodeIP, clock:           clock.RealClock{}, enableControllerAttachDetach:            kubeCfg.EnableControllerAttachDetach, <del> iptClient:                               utilipt.New(utilexec.New(), utildbus.New(), utilipt.ProtocolIpv4), </del> <ins> iptClient:                               utilipt.New(utilexec.New(), utildbus.New(), protocol), </ins> makeIPTablesUtilChains:                  kubeCfg.MakeIPTablesUtilChains, iptablesMasqueradeBit:                   int(kubeCfg.IPTablesMasqueradeBit), iptablesDropBit:                         int(kubeCfg.IPTablesDropBit),


Question:
/kind bug What happened: From doest not find kubeconfig file What you expected to happen: To find kubeconfig file How to reproduce it (as minimally and precisely as possible): from conformance file Anything else we need to know?: Line 185 : Starting kubelet with args But does not exist. Using from line 80 seems to work properly Environment: Kubernetes version (use ):Cloud provider or hardware configuration: hardware configuration OS (e.g. from /etc/os-release): Fedora 28 Server Kernel (e.g. ): Linux 4.17.12-200.fc28.x86_64
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area conformance

# CNI_BIN_DIR is the path to network plugin config files. CNI_BIN_DIR=${CNI_BIN_DIR:-""} <del> # KUBELET_KUBECONFIG_DIR is the path to a dir for the kubelet's kubeconfig file </del> <ins> # KUBELET_KUBECONFIG is the path to a kubeconfig file, specifying how to connect to the API server. </ins> KUBELET_KUBECONFIG=${KUBELET_KUBECONFIG:-"/var/lib/kubelet/kubeconfig"} # Creates a kubeconfig file for the kubelet.


Question:
/kind bug What happened: From doest not find kubeconfig file What you expected to happen: To find kubeconfig file How to reproduce it (as minimally and precisely as possible): from conformance file Anything else we need to know?: Line 185 : Starting kubelet with args But does not exist. Using from line 80 seems to work properly Environment: Kubernetes version (use ):Cloud provider or hardware configuration: hardware configuration OS (e.g. from /etc/os-release): Fedora 28 Server Kernel (e.g. ): Linux 4.17.12-200.fc28.x86_64
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area conformance

file_check_frequency=10s pod_cidr=10.100.0.0/24 log_level=4 <del> start_kubelet --kubeconfig "${KUBELET_KUBECONFIG_DIR}/kubelet.kubeconfig"  </del> <ins> start_kubelet --kubeconfig ${KUBELET_KUBECONFIG}  </ins> --volume-stats-agg-period $volume_stats_agg_period  --allow-privileged=$allow_privileged  --serialize-image-pulls=$serialize_image_pulls


Question:
/kind cleanup /sig testing /priority important-longterm This is a followup issue to address the "standard build globals" nit on At the very least I think we shouldn't capture our custom env var. Also think it'd be worth documenting the assumed variables in a list, similar to
cc
/assign

# Arguments: a list of kubernetes packages to build. # Expected variables: ${build_args} should be set to an array of Go build arguments. # In addition, ${package} and ${platform} should have been set earlier, and if <del> # ${build_with_coverage} is set, coverage instrumentation will be enabled. </del> <ins> # ${KUBE_BUILD_WITH_COVERAGE} is set, coverage instrumentation will be enabled. </ins> # # Invokes Go to actually build some packages. If coverage is disabled, simply invokes # go install. If coverage is enabled, builds covered binaries using go test, temporarily # producing the required unit test files and then cleaning up after itself. # Non-covered binaries are then built using go install as usual. kube::golang::build_some_binaries() { <del> if [[ -n "${build_with_coverage:-}" ]]; then </del> <ins> if [[ -n "${KUBE_BUILD_WITH_COVERAGE:-}" ]]; then </ins> local -a uncovered=() for package in "$@"; do if kube::golang::is_instrumented_package "${package}"; then


Question:
/kind cleanup /sig testing /priority important-longterm This is a followup issue to address the "standard build globals" nit on At the very least I think we shouldn't capture our custom env var. Also think it'd be worth documenting the assumed variables in a list, similar to
cc
/assign

host_platform=$(kube::golang::host_platform) # Use eval to preserve embedded quoted strings. <del> local goflags goldflags gogcflags build_with_coverage </del> <ins> local goflags goldflags gogcflags </ins> eval "goflags=(${GOFLAGS:-})" goldflags="${GOLDFLAGS:-} $(kube::version::ldflags)" gogcflags="${GOGCFLAGS:-}" <del> build_with_coverage="${KUBE_BUILD_WITH_COVERAGE:-}" </del> local -a targets=() local arg


Question:
We should consider updating the command to wait on all resources for a given kind. For example: would wait for every pod in the current namespace to be deleted. /sig cli cc
Interestingly, the error text implies that this is already supported: Yet when you try to use it, it declares Would really like the wait functionality given some much needed TLC. Still no way to determine what conditions are supported for various resources (eg. I can say and it just waits for the timeout like that resource might actually potato at some point).
I'm looking into adding the --all flag to wait, sounds like a nice feature to have :)

PrintFlags:       genericclioptions.NewPrintFlags("condition met"), ResourceBuilderFlags: genericclioptions.NewResourceBuilderFlags(). WithLabelSelector(""). <ins> WithFieldSelector(""). WithAll(false). </ins> WithAllNamespaces(false). WithAll(false). WithLatest(),


Question:
We should consider updating the command to wait on all resources for a given kind. For example: would wait for every pod in the current namespace to be deleted. /sig cli cc
Interestingly, the error text implies that this is already supported: Yet when you try to use it, it declares Would really like the wait functionality given some much needed TLC. Still no way to determine what conditions are supported for various resources (eg. I can say and it just waits for the timeout like that resource might actually potato at some point).
I'm looking into adding the --all flag to wait, sounds like a nice feature to have :)

flags := NewWaitFlags(restClientGetter, streams) cmd := &cobra.Command{ <del> Use:                   "wait resource.group/name [--for=delete|--for condition=available]", </del> <ins> Use:     "wait ([-f FILENAME] | resource.group/resource.name | resource.group [(-l label | --all)]) [--for=delete|--for condition=available]", Short:   "Experimental: Wait for a specific condition on one or many resources.", Long:    waitLong, Example: waitExample, </ins> DisableFlagsInUseLine: true, <del> Short:                 "Experimental: Wait for a specific condition on one or many resources.", Long:                  waitLong, Example:               waitExample, </del> Run: func(cmd *cobra.Command, args []string) { o, err := flags.ToOptions(args) cmdutil.CheckErr(err)


Question:
We should consider updating the command to wait on all resources for a given kind. For example: would wait for every pod in the current namespace to be deleted. /sig cli cc
Interestingly, the error text implies that this is already supported: Yet when you try to use it, it declares Would really like the wait functionality given some much needed TLC. Still no way to determine what conditions are supported for various resources (eg. I can say and it just waits for the timeout like that resource might actually potato at some point).
I'm looking into adding the --all flag to wait, sounds like a nice feature to have :)

}, }, { <ins> name: "handles watch delete multiple", infos: []*resource.Info{ { Mapping: &meta.RESTMapping{ Resource: schema.GroupVersionResource{Group: "group", Version: "version", Resource: "theresource-1"}, }, Name:      "name-foo-1", Namespace: "ns-foo", }, { Mapping: &meta.RESTMapping{ Resource: schema.GroupVersionResource{Group: "group", Version: "version", Resource: "theresource-2"}, }, Name:      "name-foo-2", Namespace: "ns-foo", }, }, fakeClient: func() *dynamicfakeclient.FakeDynamicClient { fakeClient := dynamicfakeclient.NewSimpleDynamicClient(scheme) fakeClient.PrependReactor("get", "theresource-1", func(action clienttesting.Action) (handled bool, ret runtime.Object, err error) { return true, newUnstructured("group/version", "TheKind", "ns-foo", "name-foo-1"), nil }) fakeClient.PrependReactor("get", "theresource-2", func(action clienttesting.Action) (handled bool, ret runtime.Object, err error) { return true, newUnstructured("group/version", "TheKind", "ns-foo", "name-foo-2"), nil }) fakeClient.PrependWatchReactor("theresource-1", func(action clienttesting.Action) (handled bool, ret watch.Interface, err error) { fakeWatch := watch.NewRaceFreeFake() fakeWatch.Action(watch.Deleted, newUnstructured("group/version", "TheKind", "ns-foo", "name-foo-1")) return true, fakeWatch, nil }) fakeClient.PrependWatchReactor("theresource-2", func(action clienttesting.Action) (handled bool, ret watch.Interface, err error) { fakeWatch := watch.NewRaceFreeFake() fakeWatch.Action(watch.Deleted, newUnstructured("group/version", "TheKind", "ns-foo", "name-foo-2")) return true, fakeWatch, nil }) return fakeClient }, timeout: 10 * time.Second, validateActions: func(t *testing.T, actions []clienttesting.Action) { if len(actions) != 2 { t.Fatal(spew.Sdump(actions)) } if !actions[0].Matches("list", "theresource-1") { t.Error(spew.Sdump(actions)) } if !actions[1].Matches("list", "theresource-2") { t.Error(spew.Sdump(actions)) } }, }, { </ins> name: "ignores watch error", infos: []*resource.Info{ {


Question:
We should consider updating the command to wait on all resources for a given kind. For example: would wait for every pod in the current namespace to be deleted. /sig cli cc
Interestingly, the error text implies that this is already supported: Yet when you try to use it, it declares Would really like the wait functionality given some much needed TLC. Still no way to determine what conditions are supported for various resources (eg. I can say and it just waits for the timeout like that resource might actually potato at some point).
I'm looking into adding the --all flag to wait, sounds like a nice feature to have :)

source "${KUBE_ROOT}/test/cmd/storage.sh" source "${KUBE_ROOT}/test/cmd/template-output.sh" source "${KUBE_ROOT}/test/cmd/version.sh" <ins> source "${KUBE_ROOT}/test/cmd/wait.sh" </ins> ETCD_HOST=${ETCD_HOST:-127.0.0.1}


Question:
We should consider updating the command to wait on all resources for a given kind. For example: would wait for every pod in the current namespace to be deleted. /sig cli cc
Interestingly, the error text implies that this is already supported: Yet when you try to use it, it declares Would really like the wait functionality given some much needed TLC. Still no way to determine what conditions are supported for various resources (eg. I can say and it just waits for the timeout like that resource might actually potato at some point).
I'm looking into adding the --all flag to wait, sounds like a nice feature to have :)

################# record_command run_impersonation_tests <ins> #################### # kubectl wait     # #################### record_command run_wait_tests </ins> kube::test::clear_all if [[ -n "${foundError}" ]]; then


Question:
We should consider updating the command to wait on all resources for a given kind. For example: would wait for every pod in the current namespace to be deleted. /sig cli cc
Interestingly, the error text implies that this is already supported: Yet when you try to use it, it declares Would really like the wait functionality given some much needed TLC. Still no way to determine what conditions are supported for various resources (eg. I can say and it just waits for the timeout like that resource might actually potato at some point).
I'm looking into adding the --all flag to wait, sounds like a nice feature to have :)

<ins> #!/usr/bin/env bash # Copyright 2018 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail run_wait_tests() { set -o nounset set -o errexit kube::log::status "Testing kubectl wait" create_and_use_new_namespace ### Wait for deletion using --all flag # create test data kubectl create deployment test-1 --image=busybox kubectl create deployment test-2 --image=busybox # Post-Condition: deployments exists kube::test::get_object_assert "deployments" "{{range .items}}{{.metadata.name}},{{end}}" 'test-1,test-2,' # Delete all deployments async to kubectl wait ( sleep 2 && kubectl delete deployment --all ) & # Command: Wait for all deployments to be deleted output_message=$(kubectl wait deployment --for=delete --all) # Post-Condition: Wait was successful kube::test::if_has_string "${output_message}" 'test-1 condition met' kube::test::if_has_string "${output_message}" 'test-2 condition met' set +o nounset set +o errexit } </ins>  No newline at end of file


Question:
/kind bug /sig release when we use , we must make sure xxx is not nil. So before we use it, we should check err value first. Such as:
I think even if the file is nil , maybe It doesn't matter
Sometimes it does matter, while can't handle nil correctly. Maybe it is just a good practise.
Although File.Close() can handle cases like this, serverConn.Close() will cause panic to happen. Anyway, these are bugs that exist. /assign

<del> 1.0 </del> <ins> 1.1 </ins>


Question:
/kind bug /sig release when we use , we must make sure xxx is not nil. So before we use it, we should check err value first. Such as:
I think even if the file is nil , maybe It doesn't matter
Sometimes it does matter, while can't handle nil correctly. Maybe it is just a good practise.
Although File.Close() can handle cases like this, serverConn.Close() will cause panic to happen. Anyway, these are bugs that exist. /assign

serverAddress, err := net.ResolveUDPAddr("udp", fmt.Sprintf(":%d", udpPort)) assertNoError(err) serverConn, err := net.ListenUDP("udp", serverAddress) <ins> assertNoError(err) </ins> defer serverConn.Close() buf := make([]byte, 1024)


Question:
/kind bug /sig release when we use , we must make sure xxx is not nil. So before we use it, we should check err value first. Such as:
I think even if the file is nil , maybe It doesn't matter
Sometimes it does matter, while can't handle nil correctly. Maybe it is just a good practise.
Although File.Close() can handle cases like this, serverConn.Close() will cause panic to happen. Anyway, these are bugs that exist. /assign

spec: containers: - name: netexec <del> image: gcr.io/kubernetes-e2e-test-images/netexec-amd64:1.0 </del> <ins> image: gcr.io/kubernetes-e2e-test-images/netexec-amd64:1.1 </ins> ports: - containerPort: 8080 protocol: TCP


Question:
/kind bug /sig release when we use , we must make sure xxx is not nil. So before we use it, we should check err value first. Such as:
I think even if the file is nil , maybe It doesn't matter
Sometimes it does matter, while can't handle nil correctly. Maybe it is just a good practise.
Although File.Close() can handle cases like this, serverConn.Close() will cause panic to happen. Anyway, these are bugs that exist. /assign

MounttestUser            = ImageConfig{e2eRegistry, "mounttest-user", "1.0"} Nautilus                 = ImageConfig{e2eRegistry, "nautilus", "1.0"} Net                      = ImageConfig{e2eRegistry, "net", "1.0"} <del> Netexec                  = ImageConfig{e2eRegistry, "netexec", "1.0"} </del> <ins> Netexec                  = ImageConfig{e2eRegistry, "netexec", "1.1"} </ins> Nettest                  = ImageConfig{e2eRegistry, "nettest", "1.0"} Nginx                    = ImageConfig{dockerLibraryRegistry, "nginx", "1.14-alpine"} NginxNew                 = ImageConfig{dockerLibraryRegistry, "nginx", "1.15-alpine"}


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind feature Alpine fixed a RCE vulnerability in apk in 3.8.1 () Update the base image used for addon-manager () to pick it up. Note: I'd send a PR if it depended directly on alpine image, not bashell/alpine.
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assing
Ping

<ins> ### Version 8.8  (Mon October 1 2018 Zihong Zheng <zihongz@google.com>) - Update to use debian-base:0.3.2. </ins> ### Version 8.7  (Tue September 4 2018 Zihong Zheng <zihongz@google.com>) - Support extra `--prune-whitelist` resources in kube-addon-manager. - Update kubectl to v1.10.7.


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind feature Alpine fixed a RCE vulnerability in apk in 3.8.1 () Update the base image used for addon-manager () to pick it up. Note: I'd send a PR if it depended directly on alpine image, not bashell/alpine.
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assing
Ping

FROM BASEIMAGE <ins> RUN clean-install bash </ins> ADD kube-addons.sh /opt/ ADD namespace.yaml /opt/ ADD kubectl /usr/local/bin/


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind feature Alpine fixed a RCE vulnerability in apk in 3.8.1 () Update the base image used for addon-manager () to pick it up. Note: I'd send a PR if it depended directly on alpine image, not bashell/alpine.
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assing
Ping

IMAGE=staging-k8s.gcr.io/kube-addon-manager ARCH?=amd64 TEMP_DIR:=$(shell mktemp -d) <del> VERSION=v8.7 </del> <ins> VERSION=v8.8 </ins> KUBECTL_VERSION?=v1.10.7 <del> ifeq ($(ARCH),amd64) BASEIMAGE?=bashell/alpine-bash endif ifeq ($(ARCH),arm) BASEIMAGE?=arm32v7/debian endif ifeq ($(ARCH),arm64) BASEIMAGE?=arm64v8/debian endif ifeq ($(ARCH),ppc64le) BASEIMAGE?=ppc64le/debian endif ifeq ($(ARCH),s390x) BASEIMAGE?=s390x/debian endif </del> <ins> BASEIMAGE=k8s.gcr.io/debian-base-$(ARCH):0.3.2 </ins> .PHONY: build push


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: "[sig-storage] HostPath [It] should support existing single file subPath" test failed on environments which don't support external IPs like What you expected to happen: The test can pass successfully. How to reproduce it (as minimally and precisely as possible): just kick: Environment: Kubernetes version (use ): v1.11.1 Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.4 LTS Kernel (e.g. ): 4.4.0-134-generic Install tools: kubeadm Others:
/sig storage

// Create the subPath directory on the host existing := path.Join(source.Path, subPath) <del> externalIP, err := framework.GetNodeExternalIP(&nodeList.Items[0]) </del> <ins> nodeIP, err := framework.GetNodeExternalIP(&nodeList.Items[0]) if err != nil { nodeIP, err = framework.GetNodeInternalIP(&nodeList.Items[0]) } </ins> framework.ExpectNoError(err) <del> result, err := framework.SSH(fmt.Sprintf("mkdir -p %s", existing), externalIP, framework.TestContext.Provider) </del> <ins> result, err := framework.SSH(fmt.Sprintf("mkdir -p %s", existing), nodeIP, framework.TestContext.Provider) </ins> framework.LogSSHResult(result) framework.ExpectNoError(err) if result.Code != 0 {


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: "[sig-storage] HostPath [It] should support existing single file subPath" test failed on environments which don't support external IPs like What you expected to happen: The test can pass successfully. How to reproduce it (as minimally and precisely as possible): just kick: Environment: Kubernetes version (use ): v1.11.1 Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.4 LTS Kernel (e.g. ): 4.4.0-134-generic Install tools: kubeadm Others:
/sig storage

// Create the subPath file on the host existing := path.Join(source.Path, subPath) <del> externalIP, err := framework.GetNodeExternalIP(&nodeList.Items[0]) </del> <ins> nodeIP, err := framework.GetNodeExternalIP(&nodeList.Items[0]) if err != nil { nodeIP, err = framework.GetNodeInternalIP(&nodeList.Items[0]) } </ins> framework.ExpectNoError(err) <del> result, err := framework.SSH(fmt.Sprintf("echo "mount-tester new file" > %s", existing), externalIP, framework.TestContext.Provider) </del> <ins> result, err := framework.SSH(fmt.Sprintf("echo "mount-tester new file" > %s", existing), nodeIP, framework.TestContext.Provider) </ins> framework.LogSSHResult(result) framework.ExpectNoError(err) if result.Code != 0 {


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --BUG REPORT: /kind bug What happened: Kubernetes distro testing v1.12.0-rc.2 shows in setups with self-hosted control planes (i.e. apiserver, controller-manager, scheduler run as pods) that previously worked on v1.11.3 and prior, the kube-controller-manager now crash loops. Notably, the kube-controller-manager is run with: which has been fine through to v1.11.3. Full . What you expected to happen: Controller manager may not assume it has root. Or should clarify if an additional mount is now required. May have been introduced in cc How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0-rc.2 Cloud provider or hardware configuration: GCE, DigitalOcean, AWS OS (e.g. from /etc/os-release): Container Linux 1855.4.0 Kernel (e.g. ): 4.14.67-coreos Install tools: Typhoon
Seems like distros like Typhoon, Tectonic, and Tectonic/Openshift may work around this explicitly passing some other path to kube-controller-manager's or creating an emptyDir mount for . Need to understand the expectations around these certs (multiple pods, restarts, etc.) A release note seems needed to cover the new requirements. Appears this was also noticed by juju in
/sig release
cc
WDYT? Can we get folks to write a note to cover this problem in known issues?
I've an mount to address that crashloop, which seems to work preliminary (). was able to proceed with self-signed cert creation and hit a new bug I've filed separately EDIT: Perhaps a suitable release note for this change is:
the cert should not need to be persisted to disk. I don't think adding a writeable directory requirement just for a self-signed cert is a very good idea /assign
Possible fixes enable by default if no cert is provided, rather than self-signing the self-sign cert in memory only, no disk persistence needed 't enable by default at all 2 would be ideal but is likely too big a change at this point. 3 would be unfortunate. 1 could be a good compromise if the code change is contained.
It's up to the SIG if this is a 1.12.0 blocker, or a Known Issue and 1.12.1 material. I'm inclined to think the latter.
thanks for the clarifications. I would prefer to avoid my current workaround of giving it a writeable location so those all sound fine to me. is there a full story around these controller-manager changes somewhere? I could mint actual certs, mount from a secret, update clients. If there's merit. Like, how big of a deal is it that this is now always enabled vs before? I'll probably read through the various diffs after vacation.
If this is going to get some text in release notes Known Issues (seems like it should), somebody needs to draft the text and make a suggestion of it in ASAP.
/assign
Do we want to fix this for 1.12.1 (on friday)?
Self signing in memory means you get a different cert each time it starts. It's probably better than nothing but it's not how things are supposed to work (it's too inconvenient to manually verify the cert out of band in places users should trust it). OTOH if you want to do that, maybe you'll provide a writable directory in the first place, or even a real cert. So I guess in-memory is probably OK.
Second thought: is a terrible default for the cert path. It should have the word "certs" in the path somewhere. If folks (inadvisably) run apiserver and controller manager on the same system, they should at least not both try to use certs from the same place, or worse, write them there. I suggest as the default.
What components are now verifying these certs or supposed to be? I'd expect something to be failing since random certs are generated each time the pod restarts, but don't see any impact (implying this isn't yet improving security). Seems like controller-manager just an HTTPS endpoint, but the HTTP one is still what's used right?
I guess that aligns with the comment
The autogenerated certs only have SANs for localhost and 127.0.0.1.
The auto generated certs are self signed, right? If so, and callers won't be able to validate the signer, which SANs they have don't really matter.
Yeah, they're self-signed. So looks like the 1.12 controller-manager serves on HTTP/ (always on) and HTTPS/ (unless --secure-port=0). I guess the point of the original change was so some client could now choose to use the HTTPS endpoint with insecureSkipVerify. Main reason I was interested in the SAN is to get some clue about how a client would be expected to access these controller-manager pods (for contrast, apiserver SAN has DNS:kubernetes and an internal record that always resolves to a kube-apiserve pod).
is this updated for the openstack cloud provider too?
This issue wasn't related to any particular cloud provider. The cloud provider / hardware config field is just a typical part of filing an issue in case it had been related.
I have a writeable (I've tried other writeable directories too.) I'm encountering this: Should I post elsewhere in a new thread, or continue here?

"k8s.io/kubernetes/pkg/api/legacyscheme" "k8s.io/kubernetes/pkg/controller" "k8s.io/kubernetes/pkg/master/ports" // add the kubernetes feature gates _ "k8s.io/kubernetes/pkg/features" )