Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When I created a crd e.g.: Then I used . I found that the first time the result will be with an error: And as for the second time it will be normal: And it can be stably reproduced every time I created a new CRD. What you expected to happen: kubectl returns the first time I requested the new CRD instead of an error. Anything else we need to know?: I also found that requesting short name e.g. will be the same. It returns an error the first time but becomes normal the second time. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /kind bug
/area custom-resources
Confirmed that could show up even the crd has status /assign
/sig api-machinery Haven't looked into the problem yet but could be related:
Looks like this happens because cache file in .kube/http-cache/ or .kube/cache/ is not synced with kube-apiserver. delete all files under .kube/http-cache/ and .kube/cache/ will cause a sync and the error will not show up. This looks like a cli issue.
fyi
Tried to fix in cli:
/cc
/assign
this seems similar to
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is /shrug
Can you put this into words? What's the actual issue?
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is Can you put this into words? What's the actual issue? SRY, I should have put it clearer.. I just mean that this issue is sort of fundamental and obviously has been existing for a long time.
/remove-shrug
/unshrug looks like an important problem we want to solve (?)
¯()/¯ <detailsIn response to : /unshrug Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten ouch..
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
encountered same issue, as suggested the error will not show up.
any progress?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Here is a little more info, and here is how I am able to reproduce it (kubectl version 1.18.0 and cluster version 1.19.0-beta.2): A couple interesting things i noted... If I deleted my and re-ran it, I could not reproduce it. It is only the first time the short name was EVER used that this occurs. I only tried from one client machine, so I'm not sure this is cluster-specific or client-specific. If I used a different short name, like 'fo2' and re-ran the example, then it reproduces the problem. You can also run after creating the CRD and it has the same effect as deleting your . EDIT TO ADD: This doesn't seem to be a timing thing... if I sleep for 10 seconds after creating the CRD, it still has the problem.
The root of the problem is that the is run before the discovery. So in this particular case this line continues to b/c read cached data and never reached out to server. Whereas discovery mapper in will get a not found error and will proceed invalidating cache and re-fetching discovery data and return that back. In the subsequent call the shortcut expander is already aware of the new shortcut so it works just fine. We'd need to teach shortcut expander to re-fetch discovery data, but I'm not sure that's desirable, or we could at least warn a user about this situation.
See for possible approach.
Any new progress on this ?
lists a candidate solution. This bug is causing steadily failing tests. I'll talk to a few people to see if this can get any traction. do you know who can get this moving? It sounds like you have identified the problem and have proposed a solution.

func (c *fakeDiscoveryClient) OpenAPIV3() openapi.Client { panic("implement me") } <ins> type fakeCachedDiscoveryClient struct { discovery.DiscoveryInterface freshHandler      func() bool invalidateHandler func() } var _ discovery.CachedDiscoveryInterface = &fakeCachedDiscoveryClient{} func (c *fakeCachedDiscoveryClient) Fresh() bool { if c.freshHandler != nil { return c.freshHandler() } return true } func (c *fakeCachedDiscoveryClient) Invalidate() { if c.invalidateHandler != nil { c.invalidateHandler() } } </ins>


Question:
/kind bug What happened: When I try to create a priorityclass object like: It fails with And when I try create it with yaml file like: It works. What you expected to happen: Create a priorityclass object with kubectl.
/sig cli
Do you have --runtime- set in kube-apiserver?
No, if set this, it works.
Agree this is a bug. v1beta1 should be preferred.
/sig scheduling

"//staging/src/k8s.io/api/policy/v1beta1:go_default_library", "//staging/src/k8s.io/api/rbac/v1:go_default_library", "//staging/src/k8s.io/api/rbac/v1beta1:go_default_library", <del> "//staging/src/k8s.io/api/scheduling/v1alpha1:go_default_library", </del> <ins> "//staging/src/k8s.io/api/scheduling/v1beta1:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/api/equality:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/meta:go_default_library",


Question:
/kind bug What happened: When I try to create a priorityclass object like: It fails with And when I try create it with yaml file like: It works. What you expected to happen: Create a priorityclass object with kubectl.
/sig cli
Do you have --runtime- set in kube-apiserver?
No, if set this, it works.
Agree this is a bug. v1beta1 should be preferred.
/sig scheduling

"//staging/src/k8s.io/api/policy/v1beta1:go_default_library", "//staging/src/k8s.io/api/rbac/v1:go_default_library", "//staging/src/k8s.io/api/rbac/v1beta1:go_default_library", <del> "//staging/src/k8s.io/api/scheduling/v1alpha1:go_default_library", </del> <ins> "//staging/src/k8s.io/api/scheduling/v1beta1:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/meta:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/resource:go_default_library",


Question:
/kind bug What happened: When I try to create a priorityclass object like: It fails with And when I try create it with yaml file like: It works. What you expected to happen: Create a priorityclass object with kubectl.
/sig cli
Do you have --runtime- set in kube-apiserver?
No, if set this, it works.
Agree this is a bug. v1beta1 should be preferred.
/sig scheduling

import ( "fmt" <del> scheduling "k8s.io/api/scheduling/v1alpha1" </del> <ins> scheduling "k8s.io/api/scheduling/v1beta1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" )


Question:
/kind bug What happened: When I try to create a priorityclass object like: It fails with And when I try create it with yaml file like: It works. What you expected to happen: Create a priorityclass object with kubectl.
/sig cli
Do you have --runtime- set in kube-apiserver?
No, if set this, it works.
Agree this is a bug. v1beta1 should be preferred.
/sig scheduling

package kubectl import ( <del> scheduling "k8s.io/api/scheduling/v1alpha1" </del> <ins> scheduling "k8s.io/api/scheduling/v1beta1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "reflect"


Question:
In the function, there is a that always runs and closes the channel when returning. This means that instead of running periodically, the image garbage collection will be terminated immediately when the function returns. The bug was introduced by in 1.11 /cc
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
BTW, when the threshold is set to 100% (the use case that the PR originally wanted to fix), the code'd try sending to a closed channel and triggers a panic.
i will check it soon, will submit one patch immediately, thanks
does this bug applies to v1.11.2 ? or v.1.11.2 includes the fix?

} }, ContainerGCPeriod, wait.NeverStop) <del> stopChan := make(chan struct{}) defer close(stopChan) </del> // when the high threshold is set to 100, stub the image GC manager if kl.kubeletConfiguration.ImageGCHighThresholdPercent == 100 { glog.V(2).Infof("ImageGCHighThresholdPercent is set 100, Disable image GC") <del> go func() { stopChan <- struct{}{} }() </del> <ins> return </ins> } prevImageGCFailed := false


Question:
In the function, there is a that always runs and closes the channel when returning. This means that instead of running periodically, the image garbage collection will be terminated immediately when the function returns. The bug was introduced by in 1.11 /cc
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
BTW, when the threshold is set to 100% (the use case that the PR originally wanted to fix), the code'd try sending to a closed channel and triggers a panic.
i will check it soon, will submit one patch immediately, thanks
does this bug applies to v1.11.2 ? or v.1.11.2 includes the fix?

glog.V(vLevel).Infof("Image garbage collection succeeded") } <del> }, ImageGCPeriod, stopChan) </del> <ins> }, ImageGCPeriod, wait.NeverStop) </ins> } // initializeModules will initialize internal modules that do not require the container runtime to be up.


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --/kind feature The field essentially has the type , but currently there is no validation for set and map semantics, i.e. unique elements / keys. Having this validation at API level removes logic to validate at the volume plugin level. edit: this check will not be in cases where this makes it difficult to add other fields and make this field optional. /sig api-machinery /sig storage
/assign
Thanks for catching this. Would you be able to implement this for this release? We want to try to go beta if we can, and we won't be able to strengthen validation once that happens.
Yes will do!

return allErrs } <del> // validateTopologySelectorLabelRequirement tests that the specified TopologySelectorLabelRequirement fields has valid data func validateTopologySelectorLabelRequirement(rq core.TopologySelectorLabelRequirement, fldPath *field.Path) field.ErrorList { </del> <ins> // validateTopologySelectorLabelRequirement tests that the specified TopologySelectorLabelRequirement fields has valid data, // and constructs a set containing all of its Values. func validateTopologySelectorLabelRequirement(rq core.TopologySelectorLabelRequirement, fldPath *field.Path) (sets.String, field.ErrorList) { </ins> allErrs := field.ErrorList{} <ins> valueSet := make(sets.String) valuesPath := fldPath.Child("values") </ins> if len(rq.Values) == 0 { <del> allErrs = append(allErrs, field.Forbidden(fldPath.Child("values"), "must specify as least one value")) </del> <ins> allErrs = append(allErrs, field.Required(valuesPath, "")) </ins> } <ins> // Validate set property of Values field for i, value := range rq.Values { if valueSet.Has(value) { allErrs = append(allErrs, field.Duplicate(valuesPath.Index(i), value)) } valueSet.Insert(value) } </ins> allErrs = append(allErrs, unversionedvalidation.ValidateLabelName(rq.Key, fldPath.Child("key"))...) <del> return allErrs </del> <ins> return valueSet, allErrs </ins> } <del> // ValidateTopologySelectorTerm tests that the specified topology selector term has valid data func ValidateTopologySelectorTerm(term core.TopologySelectorTerm, fldPath *field.Path) field.ErrorList { </del> <ins> // ValidateTopologySelectorTerm tests that the specified topology selector term has valid data, // and constructs a map representing the term in raw form. func ValidateTopologySelectorTerm(term core.TopologySelectorTerm, fldPath *field.Path) (map[string]sets.String, field.ErrorList) { </ins> allErrs := field.ErrorList{} <ins> exprMap := make(map[string]sets.String) exprPath := fldPath.Child("matchLabelExpressions") </ins> if utilfeature.DefaultFeatureGate.Enabled(features.DynamicProvisioningScheduling) { <ins> // Allow empty MatchLabelExpressions, in case this field becomes optional in the future. </ins> for i, req := range term.MatchLabelExpressions { <del> allErrs = append(allErrs, validateTopologySelectorLabelRequirement(req, fldPath.Child("matchLabelExpressions").Index(i))...) </del> <ins> idxPath := exprPath.Index(i) valueSet, exprErrs := validateTopologySelectorLabelRequirement(req, idxPath) allErrs = append(allErrs, exprErrs...) // Validate no duplicate keys exist. if _, exists := exprMap[req.Key]; exists { allErrs = append(allErrs, field.Duplicate(idxPath.Child("key"), req.Key)) } exprMap[req.Key] = valueSet </ins> } } else if len(term.MatchLabelExpressions) != 0 { allErrs = append(allErrs, field.Forbidden(fldPath, "field is disabled by feature-gate DynamicProvisioningScheduling")) } <del> return allErrs </del> <ins> return exprMap, allErrs </ins> } // ValidateAvoidPodsInNodeAnnotations tests that the serialized AvoidPods in Node.Annotations has valid data


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --/kind feature The field essentially has the type , but currently there is no validation for set and map semantics, i.e. unique elements / keys. Having this validation at API level removes logic to validate at the volume plugin level. edit: this check will not be in cases where this makes it difficult to add other fields and make this field optional. /sig api-machinery /sig storage
/assign
Thanks for catching this. Would you be able to implement this for this release? We want to try to go beta if we can, and we won't be able to strengthen validation once that happens.
Yes will do!

importpath = "k8s.io/kubernetes/pkg/apis/storage/validation", deps = [ "//pkg/apis/core:go_default_library", <ins> "//pkg/apis/core/helper:go_default_library", </ins> "//pkg/apis/core/validation:go_default_library", "//pkg/apis/storage:go_default_library", "//pkg/features:go_default_library",


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --/kind feature The field essentially has the type , but currently there is no validation for set and map semantics, i.e. unique elements / keys. Having this validation at API level removes logic to validate at the volume plugin level. edit: this check will not be in cases where this makes it difficult to add other fields and make this field optional. /sig api-machinery /sig storage
/assign
Thanks for catching this. Would you be able to implement this for this release? We want to try to go beta if we can, and we won't be able to strengthen validation once that happens.
Yes will do!

"k8s.io/apimachinery/pkg/util/validation/field" utilfeature "k8s.io/apiserver/pkg/util/feature" api "k8s.io/kubernetes/pkg/apis/core" <ins> "k8s.io/kubernetes/pkg/apis/core/helper" </ins> apivalidation "k8s.io/kubernetes/pkg/apis/core/validation" "k8s.io/kubernetes/pkg/apis/storage" "k8s.io/kubernetes/pkg/features"


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --/kind feature The field essentially has the type , but currently there is no validation for set and map semantics, i.e. unique elements / keys. Having this validation at API level removes logic to validate at the volume plugin level. edit: this check will not be in cases where this makes it difficult to add other fields and make this field optional. /sig api-machinery /sig storage
/assign
Thanks for catching this. Would you be able to implement this for this release? We want to try to go beta if we can, and we won't be able to strengthen validation once that happens.
Yes will do!

allErrs = append(allErrs, field.Forbidden(fldPath, "field is disabled by feature-gate DynamicProvisioningScheduling")) } <ins> rawTopologies := make([]map[string]sets.String, len(topologies)) </ins> for i, term := range topologies { <del> allErrs = append(allErrs, apivalidation.ValidateTopologySelectorTerm(term, fldPath.Index(i))...) </del> <ins> idxPath := fldPath.Index(i) exprMap, termErrs := apivalidation.ValidateTopologySelectorTerm(term, fldPath.Index(i)) allErrs = append(allErrs, termErrs...) // TODO (verult) consider improving runtime for _, t := range rawTopologies { if helper.Semantic.DeepEqual(exprMap, t) { allErrs = append(allErrs, field.Duplicate(idxPath.Child("matchLabelExpressions"), "")) } } rawTopologies = append(rawTopologies, exprMap) </ins> } return allErrs


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --/kind feature The field essentially has the type , but currently there is no validation for set and map semantics, i.e. unique elements / keys. Having this validation at API level removes logic to validate at the volume plugin level. edit: this check will not be in cases where this makes it difficult to add other fields and make this field optional. /sig api-machinery /sig storage
/assign
Thanks for catching this. Would you be able to implement this for this release? We want to try to go beta if we can, and we won't be able to strengthen validation once that happens.
Yes will do!

}, } <ins> topologyDupValues := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node1", "node1"}, }, }, }, } topologyMultiValues := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node1", "node2"}, }, }, }, } topologyEmptyMatchLabelExpressions := []api.TopologySelectorTerm{ { MatchLabelExpressions: nil, }, } topologyDupKeys := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node1"}, }, { Key:    "kubernetes.io/hostname", Values: []string{"node2"}, }, }, }, } topologyMultiTerm := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node1"}, }, }, }, { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node2"}, }, }, }, } topologyDupTermsIdentical := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone1"}, }, { Key:    "kubernetes.io/hostname", Values: []string{"node1"}, }, }, }, { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone1"}, }, { Key:    "kubernetes.io/hostname", Values: []string{"node1"}, }, }, }, } topologyExprsOneSameOneDiff := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone1"}, }, { Key:    "kubernetes.io/hostname", Values: []string{"node1"}, }, }, }, { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone1"}, }, { Key:    "kubernetes.io/hostname", Values: []string{"node2"}, }, }, }, } topologyValuesOneSameOneDiff := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node1", "node2"}, }, }, }, { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node1", "node3"}, }, }, }, } topologyDupTermsDiffExprOrder := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node1"}, }, { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone1"}, }, }, }, { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone1"}, }, { Key:    "kubernetes.io/hostname", Values: []string{"node1"}, }, }, }, } topologyDupTermsDiffValueOrder := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone1", "zone2"}, }, }, }, { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone2", "zone1"}, }, }, }, } </ins> cases := map[string]bindingTest{ "no topology": { class:         makeClass(nil, nil),


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --/kind feature The field essentially has the type , but currently there is no validation for set and map semantics, i.e. unique elements / keys. Having this validation at API level removes logic to validate at the volume plugin level. edit: this check will not be in cases where this makes it difficult to add other fields and make this field optional. /sig api-machinery /sig storage
/assign
Thanks for catching this. Would you be able to implement this for this release? We want to try to go beta if we can, and we won't be able to strengthen validation once that happens.
Yes will do!

class:         makeClass(nil, topologyLackOfValues), shouldSucceed: false, }, <ins> "duplicate TopologySelectorRequirement values": { class:         makeClass(nil, topologyDupValues), shouldSucceed: false, }, "multiple TopologySelectorRequirement values": { class:         makeClass(nil, topologyMultiValues), shouldSucceed: true, }, "empty MatchLabelExpressions": { class:         makeClass(nil, topologyEmptyMatchLabelExpressions), shouldSucceed: false, }, "duplicate MatchLabelExpression keys": { class:         makeClass(nil, topologyDupKeys), shouldSucceed: false, }, "duplicate MatchLabelExpression keys but across separate terms": { class:         makeClass(nil, topologyMultiTerm), shouldSucceed: true, }, "duplicate AllowedTopologies terms - identical": { class:         makeClass(nil, topologyDupTermsIdentical), shouldSucceed: false, }, "two AllowedTopologies terms, with a pair of the same MatchLabelExpressions and a pair of different ones": { class:         makeClass(nil, topologyExprsOneSameOneDiff), shouldSucceed: true, }, "two AllowedTopologies terms, with a pair of the same Values and a pair of different ones": { class:         makeClass(nil, topologyValuesOneSameOneDiff), shouldSucceed: true, }, "duplicate AllowedTopologies terms - different MatchLabelExpressions order": { class:         makeClass(nil, topologyDupTermsDiffExprOrder), shouldSucceed: false, }, "duplicate AllowedTopologies terms - different TopologySelectorRequirement values order": { class:         makeClass(nil, topologyDupTermsDiffValueOrder), shouldSucceed: false, }, } // Disable VolumeScheduling so nil VolumeBindingMode doesn't fail to validate. err := utilfeature.DefaultFeatureGate.Set("VolumeScheduling=false") if err != nil { t.Fatalf("Failed to disable feature gate for VolumeScheduling: %v", err) </ins> } // TODO: remove when feature gate not required <del> err := utilfeature.DefaultFeatureGate.Set("DynamicProvisioningScheduling=true") </del> <ins> err = utilfeature.DefaultFeatureGate.Set("DynamicProvisioningScheduling=true") </ins> if err != nil { t.Fatalf("Failed to enable feature gate for DynamicProvisioningScheduling: %v", err) }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: unit tests failed when run under What you expected to happen: unit tests pass How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I've seen this on both docker for mac and linux, with varying uids (501 and 1000). This patch makes the package's tests pass: If this is acceptable, I will open a PR. Please let me know!
LGTM
PR:
/sig cluster-lifecycle
/assign
/assign

readonly ENABLE_LEGACY_ABAC=false readonly ETC_MANIFESTS=${KUBE_HOME}/etc/kubernetes/manifests readonly KUBE_API_SERVER_DOCKER_TAG=v1.11.0-alpha.0.1808_3c7452dc11645d-dirty <del> readonly LOG_OWNER_USER=$(whoami) </del> <ins> readonly LOG_OWNER_USER=$(id -un) </ins> readonly LOG_OWNER_GROUP=$(id -gn) ENCRYPTION_PROVIDER_CONFIG={{.EncryptionProviderConfig}} ENCRYPTION_PROVIDER_CONFIG_PATH={{.EncryptionProviderConfigPath}}


Question:
/kind bug What happened: From I expected kubectl apply to support deleting a resource and recreating it, if changes cannot be applied. kubectl replace works: What you expected to happen: How to reproduce it (as minimally and precisely as possible): ` Change clusterIP to any IP address. Anything else we need to know?: Environment: Kubernetes version (use ): client v1.11.0, cluster v1.11.1 Cloud provider or hardware configuration: QEMU/KVM OS (e.g. from /etc/os-release): Cluster CentOS Linux release 7.5.1804 (Core), client macOS Kernel (e.g. ): 3.10.0-862.6.3.el7.x86_64 Install tools: kubeadm Others:
/cc
/assign

<ins> apiVersion: v1 kind: Service metadata: name: a spec: selector: app: test clusterIP: None ports: - port: 80 </ins>


Question:
/kind bug What happened: From I expected kubectl apply to support deleting a resource and recreating it, if changes cannot be applied. kubectl replace works: What you expected to happen: How to reproduce it (as minimally and precisely as possible): ` Change clusterIP to any IP address. Anything else we need to know?: Environment: Kubernetes version (use ): client v1.11.0, cluster v1.11.1 Cloud provider or hardware configuration: QEMU/KVM OS (e.g. from /etc/os-release): Cluster CentOS Linux release 7.5.1804 (Core), client macOS Kernel (e.g. ): 3.10.0-862.6.3.el7.x86_64 Install tools: kubeadm Others:
/cc
/assign

<ins> apiVersion: v1 kind: Service metadata: name: a spec: selector: app: test clusterIP: 10.0.0.12 ports: - port: 80 </ins>


Question:
/kind bug What happened: From I expected kubectl apply to support deleting a resource and recreating it, if changes cannot be applied. kubectl replace works: What you expected to happen: How to reproduce it (as minimally and precisely as possible): ` Change clusterIP to any IP address. Anything else we need to know?: Environment: Kubernetes version (use ): client v1.11.0, cluster v1.11.1 Cloud provider or hardware configuration: QEMU/KVM OS (e.g. from /etc/os-release): Cluster CentOS Linux release 7.5.1804 (Core), client macOS Kernel (e.g. ): 3.10.0-862.6.3.el7.x86_64 Install tools: kubeadm Others:
/cc
/assign

} patchBytes, patchObject, err = p.patchSimple(current, modified, source, namespace, name, errOut) } <del> if err != nil && errors.IsConflict(err) && p.force { </del> <ins> if err != nil && (errors.IsConflict(err) || errors.IsInvalid(err)) && p.force { </ins> patchBytes, patchObject, err = p.deleteAndCreate(current, modified, namespace, name) } return patchBytes, patchObject, err


Question:
/kind bug What happened: From I expected kubectl apply to support deleting a resource and recreating it, if changes cannot be applied. kubectl replace works: What you expected to happen: How to reproduce it (as minimally and precisely as possible): ` Change clusterIP to any IP address. Anything else we need to know?: Environment: Kubernetes version (use ): client v1.11.0, cluster v1.11.1 Cloud provider or hardware configuration: QEMU/KVM OS (e.g. from /etc/os-release): Cluster CentOS Linux release 7.5.1804 (Core), client macOS Kernel (e.g. ): 3.10.0-862.6.3.el7.x86_64 Install tools: kubeadm Others:
/cc
/assign

# cleanup kubectl delete svc prune-svc 2>&1 "${kube_flags[@]}" <ins> ## kubectl apply -f some.yml --force # Pre-condition: no service exists kube::test::get_object_assert services "{{range.items}}{{$id_field}}:{{end}}" '' # apply service a kubectl apply -f hack/testdata/service-revision1.yaml "${kube_flags[@]}" # check right service exists kube::test::get_object_assert 'services a' "{{${id_field}}}" 'a' # change immutable field and apply service a output_message=$(! kubectl apply -f hack/testdata/service-revision2.yaml 2>&1 "${kube_flags[@]}") kube::test::if_has_string "${output_message}" 'field is immutable' # apply --force to recreate resources for immutable fields kubectl apply -f hack/testdata/service-revision2.yaml --force "${kube_flags[@]}" # check immutable field exists kube::test::get_object_assert 'services a' "{{.spec.clusterIP}}" '10.0.0.12' # cleanup kubectl delete -f hack/testdata/service-revision2.yaml "${kube_flags[@]}" </ins> set +o nounset set +o errexit }


Question:
is a library yet is is instead of release tag. Please consider vendoring by tag whenever possible. Thanks.
We vendored it that way because the commit that fixed a bug (which we needed to be fixed and vendored back in) was This was merged on June 13. The release 1.1.4 () was released on July 1. If we want to update it to 1.1.4, I believe it should be fine. Between the commit we have vendored right now and the latest release, there are only three commits: adds tests merges (?) the case-sensitive fix - I'm not really sure what this commit is doing. fixes renaming when naming strategy is set (looks fine). Also, there are also no licensing changes. opinions?
/sig api-machinery
+1 to what said, it was a fire drill and 1.11.0 release was in trouble. Though in hind sight, we should have opened a issue to track that. something better to do next time. cc
I'm indifferent to updating this. I'm okay with moving up if it doesn't break anything, or I'm fine with leaving things as is. We don't require deps to match semver. We can version out any commit in a deps tree, as we rely on our vendored copy.
It is always a good idea to match semver and it should be a default policy. Even if it doesn't matter to you, it should, because it is a best practice. Moreover downstream it matters much more as dependencies may be un-vendored in favour of tested re-usable packages.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Someone please take care of this or bot will eventually close this valid issue without action...
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/lifecycle rotten
/remove-lifecycle rotten PR open (thanks,
oops! sorry, I meant to remove it. thanks!
Thanks.
please do feel free to open issues/prs of this sort early in the release cycle, so we can work through things needed well before the milestones/freeze etc. we are getting some heat about updating dependencies and we kind of settled on some guidelines here: if you think that needs updating, we could definitely debate that as well :) (on a PR against that doc)
it have nothing to do with release cycle or justification for upgrade if/when dependency has been vendored incorrectly. It is basically a vendoring bug. No debate is necessary although that kind of issues usually meet some resistance and even hostility here... Bugs are filed when they are discovered, regardless of release cycle. It would be wrong to sit on the bug without reporting it in order to synchronise with release cycle.

}, { "ImportPath": "github.com/json-iterator/go", <del> "Comment": "1.1.3-22-gf2b4162afba355", "Rev": "f2b4162afba35581b6d4a50d3b8f34e33c144682" </del> <ins> "Comment": "1.1.4", "Rev": "ab8a2e0c74be9d3be70b3184d9acc634935ded82" </ins> }, { "ImportPath": "github.com/jteeuwen/go-bindata",


Question:
is a library yet is is instead of release tag. Please consider vendoring by tag whenever possible. Thanks.
We vendored it that way because the commit that fixed a bug (which we needed to be fixed and vendored back in) was This was merged on June 13. The release 1.1.4 () was released on July 1. If we want to update it to 1.1.4, I believe it should be fine. Between the commit we have vendored right now and the latest release, there are only three commits: adds tests merges (?) the case-sensitive fix - I'm not really sure what this commit is doing. fixes renaming when naming strategy is set (looks fine). Also, there are also no licensing changes. opinions?
/sig api-machinery
+1 to what said, it was a fire drill and 1.11.0 release was in trouble. Though in hind sight, we should have opened a issue to track that. something better to do next time. cc
I'm indifferent to updating this. I'm okay with moving up if it doesn't break anything, or I'm fine with leaving things as is. We don't require deps to match semver. We can version out any commit in a deps tree, as we rely on our vendored copy.
It is always a good idea to match semver and it should be a default policy. Even if it doesn't matter to you, it should, because it is a best practice. Moreover downstream it matters much more as dependencies may be un-vendored in favour of tested re-usable packages.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Someone please take care of this or bot will eventually close this valid issue without action...
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/lifecycle rotten
/remove-lifecycle rotten PR open (thanks,
oops! sorry, I meant to remove it. thanks!
Thanks.
please do feel free to open issues/prs of this sort early in the release cycle, so we can work through things needed well before the milestones/freeze etc. we are getting some heat about updating dependencies and we kind of settled on some guidelines here: if you think that needs updating, we could definitely debate that as well :) (on a PR against that doc)
it have nothing to do with release cycle or justification for upgrade if/when dependency has been vendored incorrectly. It is basically a vendoring bug. No debate is necessary although that kind of issues usually meet some resistance and even hostility here... Bugs are filed when they are discovered, regardless of release cycle. It would be wrong to sit on the bug without reporting it in order to synchronise with release cycle.

}, { "ImportPath": "github.com/json-iterator/go", <del> "Rev": "f2b4162afba35581b6d4a50d3b8f34e33c144682" </del> <ins> "Rev": "ab8a2e0c74be9d3be70b3184d9acc634935ded82" </ins> }, { "ImportPath": "github.com/mailru/easyjson/buffer",


Question:
is a library yet is is instead of release tag. Please consider vendoring by tag whenever possible. Thanks.
We vendored it that way because the commit that fixed a bug (which we needed to be fixed and vendored back in) was This was merged on June 13. The release 1.1.4 () was released on July 1. If we want to update it to 1.1.4, I believe it should be fine. Between the commit we have vendored right now and the latest release, there are only three commits: adds tests merges (?) the case-sensitive fix - I'm not really sure what this commit is doing. fixes renaming when naming strategy is set (looks fine). Also, there are also no licensing changes. opinions?
/sig api-machinery
+1 to what said, it was a fire drill and 1.11.0 release was in trouble. Though in hind sight, we should have opened a issue to track that. something better to do next time. cc
I'm indifferent to updating this. I'm okay with moving up if it doesn't break anything, or I'm fine with leaving things as is. We don't require deps to match semver. We can version out any commit in a deps tree, as we rely on our vendored copy.
It is always a good idea to match semver and it should be a default policy. Even if it doesn't matter to you, it should, because it is a best practice. Moreover downstream it matters much more as dependencies may be un-vendored in favour of tested re-usable packages.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Someone please take care of this or bot will eventually close this valid issue without action...
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/lifecycle rotten
/remove-lifecycle rotten PR open (thanks,
oops! sorry, I meant to remove it. thanks!
Thanks.
please do feel free to open issues/prs of this sort early in the release cycle, so we can work through things needed well before the milestones/freeze etc. we are getting some heat about updating dependencies and we kind of settled on some guidelines here: if you think that needs updating, we could definitely debate that as well :) (on a PR against that doc)
it have nothing to do with release cycle or justification for upgrade if/when dependency has been vendored incorrectly. It is basically a vendoring bug. No debate is necessary although that kind of issues usually meet some resistance and even hostility here... Bugs are filed when they are discovered, regardless of release cycle. It would be wrong to sit on the bug without reporting it in order to synchronise with release cycle.

}, { "ImportPath": "github.com/json-iterator/go", <del> "Rev": "f2b4162afba35581b6d4a50d3b8f34e33c144682" </del> <ins> "Rev": "ab8a2e0c74be9d3be70b3184d9acc634935ded82" </ins> }, { "ImportPath": "github.com/modern-go/concurrent",


Question:
is a library yet is is instead of release tag. Please consider vendoring by tag whenever possible. Thanks.
We vendored it that way because the commit that fixed a bug (which we needed to be fixed and vendored back in) was This was merged on June 13. The release 1.1.4 () was released on July 1. If we want to update it to 1.1.4, I believe it should be fine. Between the commit we have vendored right now and the latest release, there are only three commits: adds tests merges (?) the case-sensitive fix - I'm not really sure what this commit is doing. fixes renaming when naming strategy is set (looks fine). Also, there are also no licensing changes. opinions?
/sig api-machinery
+1 to what said, it was a fire drill and 1.11.0 release was in trouble. Though in hind sight, we should have opened a issue to track that. something better to do next time. cc
I'm indifferent to updating this. I'm okay with moving up if it doesn't break anything, or I'm fine with leaving things as is. We don't require deps to match semver. We can version out any commit in a deps tree, as we rely on our vendored copy.
It is always a good idea to match semver and it should be a default policy. Even if it doesn't matter to you, it should, because it is a best practice. Moreover downstream it matters much more as dependencies may be un-vendored in favour of tested re-usable packages.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Someone please take care of this or bot will eventually close this valid issue without action...
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/lifecycle rotten
/remove-lifecycle rotten PR open (thanks,
oops! sorry, I meant to remove it. thanks!
Thanks.
please do feel free to open issues/prs of this sort early in the release cycle, so we can work through things needed well before the milestones/freeze etc. we are getting some heat about updating dependencies and we kind of settled on some guidelines here: if you think that needs updating, we could definitely debate that as well :) (on a PR against that doc)
it have nothing to do with release cycle or justification for upgrade if/when dependency has been vendored incorrectly. It is basically a vendoring bug. No debate is necessary although that kind of issues usually meet some resistance and even hostility here... Bugs are filed when they are discovered, regardless of release cycle. It would be wrong to sit on the bug without reporting it in order to synchronise with release cycle.

[[projects]] <del> name = "github.com/json-iterator/go" packages = ["."] revision = "ca39e5af3ece67bbcda3d0f4f56a8e24d9f2dad4" version = "1.1.3" [[projects]] </del> name = "github.com/modern-go/concurrent" packages = ["."] revision = "e0a39a4cb4216ea8db28e22a69f4ec25610d513a"


Question:
is a library yet is is instead of release tag. Please consider vendoring by tag whenever possible. Thanks.
We vendored it that way because the commit that fixed a bug (which we needed to be fixed and vendored back in) was This was merged on June 13. The release 1.1.4 () was released on July 1. If we want to update it to 1.1.4, I believe it should be fine. Between the commit we have vendored right now and the latest release, there are only three commits: adds tests merges (?) the case-sensitive fix - I'm not really sure what this commit is doing. fixes renaming when naming strategy is set (looks fine). Also, there are also no licensing changes. opinions?
/sig api-machinery
+1 to what said, it was a fire drill and 1.11.0 release was in trouble. Though in hind sight, we should have opened a issue to track that. something better to do next time. cc
I'm indifferent to updating this. I'm okay with moving up if it doesn't break anything, or I'm fine with leaving things as is. We don't require deps to match semver. We can version out any commit in a deps tree, as we rely on our vendored copy.
It is always a good idea to match semver and it should be a default policy. Even if it doesn't matter to you, it should, because it is a best practice. Moreover downstream it matters much more as dependencies may be un-vendored in favour of tested re-usable packages.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Someone please take care of this or bot will eventually close this valid issue without action...
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/lifecycle rotten
/remove-lifecycle rotten PR open (thanks,
oops! sorry, I meant to remove it. thanks!
Thanks.
please do feel free to open issues/prs of this sort early in the release cycle, so we can work through things needed well before the milestones/freeze etc. we are getting some heat about updating dependencies and we kind of settled on some guidelines here: if you think that needs updating, we could definitely debate that as well :) (on a PR against that doc)
it have nothing to do with release cycle or justification for upgrade if/when dependency has been vendored incorrectly. It is basically a vendoring bug. No debate is necessary although that kind of issues usually meet some resistance and even hostility here... Bugs are filed when they are discovered, regardless of release cycle. It would be wrong to sit on the bug without reporting it in order to synchronise with release cycle.

[[projects]] name = "github.com/modern-go/reflect2" packages = ["."] <del> revision = "1df9eeb2bb81f327b96228865c5687bc2194af3f" version = "1.0.0" </del> <ins> revision = "4b7aa43c6742a2c18fdef89dd197aaae7dac7ccd" version = "1.0.1" </ins> [solve-meta] analyzer-name = "dep" analyzer-version = 1 <del> inputs-digest = "56a0b9e9e61d2bc8af5e1b68537401b7f4d60805eda3d107058f3171aa5cf793" </del> <ins> inputs-digest = "ea54a775e5a354cb015502d2e7aa4b74230fc77e894f34a838b268c25ec8eeb8" </ins> solver-name = "gps-cdcl" solver-version = 1


Question:
is a library yet is is instead of release tag. Please consider vendoring by tag whenever possible. Thanks.
We vendored it that way because the commit that fixed a bug (which we needed to be fixed and vendored back in) was This was merged on June 13. The release 1.1.4 () was released on July 1. If we want to update it to 1.1.4, I believe it should be fine. Between the commit we have vendored right now and the latest release, there are only three commits: adds tests merges (?) the case-sensitive fix - I'm not really sure what this commit is doing. fixes renaming when naming strategy is set (looks fine). Also, there are also no licensing changes. opinions?
/sig api-machinery
+1 to what said, it was a fire drill and 1.11.0 release was in trouble. Though in hind sight, we should have opened a issue to track that. something better to do next time. cc
I'm indifferent to updating this. I'm okay with moving up if it doesn't break anything, or I'm fine with leaving things as is. We don't require deps to match semver. We can version out any commit in a deps tree, as we rely on our vendored copy.
It is always a good idea to match semver and it should be a default policy. Even if it doesn't matter to you, it should, because it is a best practice. Moreover downstream it matters much more as dependencies may be un-vendored in favour of tested re-usable packages.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Someone please take care of this or bot will eventually close this valid issue without action...
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/lifecycle rotten
/remove-lifecycle rotten PR open (thanks,
oops! sorry, I meant to remove it. thanks!
Thanks.
please do feel free to open issues/prs of this sort early in the release cycle, so we can work through things needed well before the milestones/freeze etc. we are getting some heat about updating dependencies and we kind of settled on some guidelines here: if you think that needs updating, we could definitely debate that as well :) (on a PR against that doc)
it have nothing to do with release cycle or justification for upgrade if/when dependency has been vendored incorrectly. It is basically a vendoring bug. No debate is necessary although that kind of issues usually meet some resistance and even hostility here... Bugs are filed when they are discovered, regardless of release cycle. It would be wrong to sit on the bug without reporting it in order to synchronise with release cycle.

[[constraint]] name = "github.com/modern-go/reflect2" <del> version = "1.0.0" </del> <ins> version = "1.0.1" </ins>


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

srcs = [ "cert_key.go", "client_ca.go", <del> "dynamicfile_content.go", </del> <ins> "dynamic_cafile_content.go", "dynamic_serving_content.go", </ins> "named_certificates.go", "static_content.go", "tlsconfig.go",


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

<ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package dynamiccertificates import ( "bytes" "crypto/x509" "fmt" "io/ioutil" "sync/atomic" "time" "k8s.io/client-go/util/cert" utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/util/workqueue" "k8s.io/klog" ) // FileRefreshDuration is exposed so that integration tests can crank up the reload speed. var FileRefreshDuration = 1 * time.Minute // Listener is an interface to use to notify interested parties of a change. type Listener interface { // Enqueue should be called when an input may have changed Enqueue() } // Notifier is a way to add listeners type Notifier interface { // AddListener is adds a listener to be notified of potential input changes AddListener(listener Listener) } // ControllerRunner is a generic interface for starting a controller type ControllerRunner interface { // RunOnce runs the sync loop a single time.  This useful for synchronous priming RunOnce() error // Run should be called a go .Run Run(workers int, stopCh <-chan struct{}) } // DynamicFileCAContent provies a CAContentProvider that can dynamically react to new file content // It also fulfills the authenticator interface to provide verifyoptions type DynamicFileCAContent struct { name string // filename is the name the file to read. filename string // caBundle is a caBundleAndVerifier that contains the last read, non-zero length content of the file caBundle atomic.Value listeners []Listener // queue only ever has one item, but it has nice error handling backoff/retry semantics queue workqueue.RateLimitingInterface } var _ Notifier = &DynamicFileCAContent{} var _ CAContentProvider = &DynamicFileCAContent{} var _ ControllerRunner = &DynamicFileCAContent{} type caBundleAndVerifier struct { caBundle      []byte verifyOptions x509.VerifyOptions } // NewDynamicCAContentFromFile returns a CAContentProvider based on a filename that automatically reloads content func NewDynamicCAContentFromFile(purpose, filename string) (*DynamicFileCAContent, error) { if len(filename) == 0 { return nil, fmt.Errorf("missing filename for ca bundle") } name := fmt.Sprintf("%s::%s", purpose, filename) ret := &DynamicFileCAContent{ name:     name, filename: filename, queue:    workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), fmt.Sprintf("DynamicCABundle-%s", purpose)), } if err := ret.loadCABundle(); err != nil { return nil, err } return ret, nil } // AddListener adds a listener to be notified when the CA content changes. func (c *DynamicFileCAContent) AddListener(listener Listener) { c.listeners = append(c.listeners, listener) } // loadCABundle determines the next set of content for the file. func (c *DynamicFileCAContent) loadCABundle() error { caBundle, err := ioutil.ReadFile(c.filename) if err != nil { return err } if len(caBundle) == 0 { return fmt.Errorf("missing content for CA bundle %q", c.Name()) } // check to see if we have a change. If the values are the same, do nothing. if !c.hasCAChanged(caBundle) { return nil } caBundleAndVerifier, err := newCABundleAndVerifier(c.Name(), caBundle) if err != nil { return err } c.caBundle.Store(caBundleAndVerifier) for _, listener := range c.listeners { listener.Enqueue() } return nil } // hasCAChanged returns true if the caBundle is different than the current. func (c *DynamicFileCAContent) hasCAChanged(caBundle []byte) bool { uncastExisting := c.caBundle.Load() if uncastExisting == nil { return true } // check to see if we have a change. If the values are the same, do nothing. existing, ok := uncastExisting.(*caBundleAndVerifier) if !ok { return true } if !bytes.Equal(existing.caBundle, caBundle) { return true } return false } // RunOnce runs a single sync loop func (c *DynamicFileCAContent) RunOnce() error { return c.loadCABundle() } // Run starts the kube-apiserver and blocks until stopCh is closed. func (c *DynamicFileCAContent) Run(workers int, stopCh <-chan struct{}) { defer utilruntime.HandleCrash() defer c.queue.ShutDown() klog.Infof("Starting %s", c.name) defer klog.Infof("Shutting down %s", c.name) // doesn't matter what workers say, only start one. go wait.Until(c.runWorker, time.Second, stopCh) // start timer that rechecks every minute, just in case.  this also serves to prime the controller quickly. _ = wait.PollImmediateUntil(FileRefreshDuration, func() (bool, error) { c.queue.Add(workItemKey) return false, nil }, stopCh) // TODO this can be wired to an fsnotifier as well. <-stopCh } func (c *DynamicFileCAContent) runWorker() { for c.processNextWorkItem() { } } func (c *DynamicFileCAContent) processNextWorkItem() bool { dsKey, quit := c.queue.Get() if quit { return false } defer c.queue.Done(dsKey) err := c.loadCABundle() if err == nil { c.queue.Forget(dsKey) return true } utilruntime.HandleError(fmt.Errorf("%v failed with : %v", dsKey, err)) c.queue.AddRateLimited(dsKey) return true } // Name is just an identifier func (c *DynamicFileCAContent) Name() string { return c.name } // CurrentCABundleContent provides ca bundle byte content func (c *DynamicFileCAContent) CurrentCABundleContent() (cabundle []byte) { return c.caBundle.Load().(*caBundleAndVerifier).caBundle } // VerifyOptions provides verifyoptions compatible with authenticators func (c *DynamicFileCAContent) VerifyOptions() x509.VerifyOptions { return c.caBundle.Load().(*caBundleAndVerifier).verifyOptions } // newVerifyOptions creates a new verification func from a file.  It reads the content and then fails. // It will return a nil function if you pass an empty CA file. func newCABundleAndVerifier(name string, caBundle []byte) (*caBundleAndVerifier, error) { if len(caBundle) == 0 { return nil, fmt.Errorf("missing content for CA bundle %q", name) } // Wrap with an x509 verifier var err error verifyOptions := defaultVerifyOptions() verifyOptions.Roots, err = cert.NewPoolFromBytes(caBundle) if err != nil { return nil, fmt.Errorf("error loading CA bundle for %q: %v", name, err) } return &caBundleAndVerifier{ caBundle:      caBundle, verifyOptions: verifyOptions, }, nil } // defaultVerifyOptions returns VerifyOptions that use the system root certificates, current time, // and requires certificates to be valid for client auth (x509.ExtKeyUsageClientAuth) func defaultVerifyOptions() x509.VerifyOptions { return x509.VerifyOptions{ KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, } } </ins>


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

<ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package dynamiccertificates import ( "crypto/tls" "fmt" "io/ioutil" "sync/atomic" "time" utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/util/workqueue" "k8s.io/klog" ) // DynamicFileServingContent provides a CertKeyContentProvider that can dynamically react to new file content type DynamicFileServingContent struct { name string // certFile is the name of the certificate file to read. certFile string // keyFile is the name of the key file to read. keyFile string // servingCert is a certKeyContent that contains the last read, non-zero length content of the key and cert servingCert atomic.Value listeners []Listener // queue only ever has one item, but it has nice error handling backoff/retry semantics queue workqueue.RateLimitingInterface } var _ Notifier = &DynamicFileServingContent{} var _ CertKeyContentProvider = &DynamicFileServingContent{} var _ ControllerRunner = &DynamicFileServingContent{} // NewDynamicServingContentFromFiles returns a dynamic CertKeyContentProvider based on a cert and key filename func NewDynamicServingContentFromFiles(purpose, certFile, keyFile string) (*DynamicFileServingContent, error) { if len(certFile) == 0 || len(keyFile) == 0 { return nil, fmt.Errorf("missing filename for serving cert") } name := fmt.Sprintf("%s::%s::%s", purpose, certFile, keyFile) ret := &DynamicFileServingContent{ name:     name, certFile: certFile, keyFile:  keyFile, queue:    workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), fmt.Sprintf("DynamicCABundle-%s", purpose)), } if err := ret.loadServingCert(); err != nil { return nil, err } return ret, nil } // AddListener adds a listener to be notified when the serving cert content changes. func (c *DynamicFileServingContent) AddListener(listener Listener) { c.listeners = append(c.listeners, listener) } // loadServingCert determines the next set of content for the file. func (c *DynamicFileServingContent) loadServingCert() error { cert, err := ioutil.ReadFile(c.certFile) if err != nil { return err } key, err := ioutil.ReadFile(c.keyFile) if err != nil { return err } if len(cert) == 0 || len(key) == 0 { return fmt.Errorf("missing content for serving cert %q", c.Name()) } // Ensure that the key matches the cert and both are valid _, err = tls.X509KeyPair(cert, key) if err != nil { return err } newCertKey := &certKeyContent{ cert: cert, key:  key, } // check to see if we have a change. If the values are the same, do nothing. existing, ok := c.servingCert.Load().(*certKeyContent) if ok && existing != nil && existing.Equal(newCertKey) { return nil } c.servingCert.Store(newCertKey) for _, listener := range c.listeners { listener.Enqueue() } return nil } // RunOnce runs a single sync loop func (c *DynamicFileServingContent) RunOnce() error { return c.loadServingCert() } // Run starts the controller and blocks until stopCh is closed. func (c *DynamicFileServingContent) Run(workers int, stopCh <-chan struct{}) { defer utilruntime.HandleCrash() defer c.queue.ShutDown() klog.Infof("Starting %s", c.name) defer klog.Infof("Shutting down %s", c.name) // doesn't matter what workers say, only start one. go wait.Until(c.runWorker, time.Second, stopCh) // start timer that rechecks every minute, just in case.  this also serves to prime the controller quickly. _ = wait.PollImmediateUntil(FileRefreshDuration, func() (bool, error) { c.queue.Add(workItemKey) return false, nil }, stopCh) // TODO this can be wired to an fsnotifier as well. <-stopCh } func (c *DynamicFileServingContent) runWorker() { for c.processNextWorkItem() { } } func (c *DynamicFileServingContent) processNextWorkItem() bool { dsKey, quit := c.queue.Get() if quit { return false } defer c.queue.Done(dsKey) err := c.loadServingCert() if err == nil { c.queue.Forget(dsKey) return true } utilruntime.HandleError(fmt.Errorf("%v failed with : %v", dsKey, err)) c.queue.AddRateLimited(dsKey) return true } // Name is just an identifier func (c *DynamicFileServingContent) Name() string { return c.name } // CurrentCertKeyContent provides serving cert byte content func (c *DynamicFileServingContent) CurrentCertKeyContent() ([]byte, []byte) { certKeyContent := c.servingCert.Load().(*certKeyContent) return certKeyContent.cert, certKeyContent.key } </ins>


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

<del> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package dynamiccertificates import ( "bytes" "crypto/x509" "fmt" "io/ioutil" "sync/atomic" "time" "k8s.io/client-go/util/cert" utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/util/workqueue" "k8s.io/klog" ) // FileRefreshDuration is exposed so that integration tests can crank up the reload speed. var FileRefreshDuration = 1 * time.Minute // Listener is an interface to use to notify interested parties of a change. type Listener interface { // Enqueue should be called when an input may have changed Enqueue() } // Notifier is a way to add listeners type Notifier interface { // AddListener is adds a listener to be notified of potential input changes AddListener(listener Listener) } // ControllerRunner is a generic interface for starting a controller type ControllerRunner interface { // RunOnce runs the sync loop a single time.  This useful for synchronous priming RunOnce() error // Run should be called a go .Run Run(workers int, stopCh <-chan struct{}) } // DynamicFileCAContent provies a CAContentProvider that can dynamically react to new file content // It also fulfills the authenticator interface to provide verifyoptions type DynamicFileCAContent struct { name string // filename is the name the file to read. filename string // caBundle is a caBundleAndVerifier that contains the last read, non-zero length content of the file caBundle atomic.Value listeners []Listener // queue only ever has one item, but it has nice error handling backoff/retry semantics queue workqueue.RateLimitingInterface } var _ Notifier = &DynamicFileCAContent{} var _ CAContentProvider = &DynamicFileCAContent{} var _ ControllerRunner = &DynamicFileCAContent{} type caBundleAndVerifier struct { caBundle      []byte verifyOptions x509.VerifyOptions } // NewDynamicCAContentFromFile returns a CAContentProvider based on a filename that automatically reloads content func NewDynamicCAContentFromFile(purpose, filename string) (*DynamicFileCAContent, error) { if len(filename) == 0 { return nil, fmt.Errorf("missing filename for ca bundle") } name := fmt.Sprintf("%s::%s", purpose, filename) ret := &DynamicFileCAContent{ name:     name, filename: filename, queue:    workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), fmt.Sprintf("DynamicCABundle-%s", purpose)), } if err := ret.loadCABundle(); err != nil { return nil, err } return ret, nil } // AddListener adds a listener to be notified when the CA content changes. func (c *DynamicFileCAContent) AddListener(listener Listener) { c.listeners = append(c.listeners, listener) } // loadCABundle determines the next set of content for the file. func (c *DynamicFileCAContent) loadCABundle() error { caBundle, err := ioutil.ReadFile(c.filename) if err != nil { return err } if len(caBundle) == 0 { return fmt.Errorf("missing content for CA bundle %q", c.Name()) } // check to see if we have a change. If the values are the same, do nothing. if !c.hasCAChanged(caBundle) { return nil } caBundleAndVerifier, err := newCABundleAndVerifier(c.Name(), caBundle) if err != nil { return err } c.caBundle.Store(caBundleAndVerifier) for _, listener := range c.listeners { listener.Enqueue() } return nil } // hasCAChanged returns true if the caBundle is different than the current. func (c *DynamicFileCAContent) hasCAChanged(caBundle []byte) bool { uncastExisting := c.caBundle.Load() if uncastExisting == nil { return true } // check to see if we have a change. If the values are the same, do nothing. existing, ok := uncastExisting.(*caBundleAndVerifier) if !ok { return true } if !bytes.Equal(existing.caBundle, caBundle) { return true } return false } // RunOnce runs a single sync loop func (c *DynamicFileCAContent) RunOnce() error { return c.loadCABundle() } // Run starts the kube-apiserver and blocks until stopCh is closed. func (c *DynamicFileCAContent) Run(workers int, stopCh <-chan struct{}) { defer utilruntime.HandleCrash() defer c.queue.ShutDown() klog.Infof("Starting %s", c.name) defer klog.Infof("Shutting down %s", c.name) // doesn't matter what workers say, only start one. go wait.Until(c.runWorker, time.Second, stopCh) // start timer that rechecks every minute, just in case.  this also serves to prime the controller quickly. _ = wait.PollImmediateUntil(FileRefreshDuration, func() (bool, error) { c.queue.Add(workItemKey) return false, nil }, stopCh) // TODO this can be wired to an fsnotifier as well. <-stopCh } func (c *DynamicFileCAContent) runWorker() { for c.processNextWorkItem() { } } func (c *DynamicFileCAContent) processNextWorkItem() bool { dsKey, quit := c.queue.Get() if quit { return false } defer c.queue.Done(dsKey) err := c.loadCABundle() if err == nil { c.queue.Forget(dsKey) return true } utilruntime.HandleError(fmt.Errorf("%v failed with : %v", dsKey, err)) c.queue.AddRateLimited(dsKey) return true } // Name is just an identifier func (c *DynamicFileCAContent) Name() string { return c.name } // CurrentCABundleContent provides ca bundle byte content func (c *DynamicFileCAContent) CurrentCABundleContent() (cabundle []byte) { return c.caBundle.Load().(*caBundleAndVerifier).caBundle } // VerifyOptions provides verifyoptions compatible with authenticators func (c *DynamicFileCAContent) VerifyOptions() x509.VerifyOptions { return c.caBundle.Load().(*caBundleAndVerifier).verifyOptions } // newVerifyOptions creates a new verification func from a file.  It reads the content and then fails. // It will return a nil function if you pass an empty CA file. func newCABundleAndVerifier(name string, caBundle []byte) (*caBundleAndVerifier, error) { if len(caBundle) == 0 { return nil, fmt.Errorf("missing content for CA bundle %q", name) } // Wrap with an x509 verifier var err error verifyOptions := defaultVerifyOptions() verifyOptions.Roots, err = cert.NewPoolFromBytes(caBundle) if err != nil { return nil, fmt.Errorf("error loading CA bundle for %q: %v", name, err) } return &caBundleAndVerifier{ caBundle:      caBundle, verifyOptions: verifyOptions, }, nil } // defaultVerifyOptions returns VerifyOptions that use the system root certificates, current time, // and requires certificates to be valid for client auth (x509.ExtKeyUsageClientAuth) func defaultVerifyOptions() x509.VerifyOptions { return x509.VerifyOptions{ KeyUsages: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, } } </del>


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

// load main cert if len(serverCertFile) != 0 || len(serverKeyFile) != 0 { var err error <del> c.Cert, err = dynamiccertificates.NewStaticCertKeyContentFromFiles(serverCertFile, serverKeyFile) </del> <ins> c.Cert, err = dynamiccertificates.NewDynamicServingContentFromFiles("serving-cert", serverCertFile, serverKeyFile) </ins> if err != nil { return err }


Question:
When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then. It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

if notifier, ok := s.ClientCA.(dynamiccertificates.Notifier); ok { notifier.AddListener(dynamicCertificateController) } <ins> if notifier, ok := s.Cert.(dynamiccertificates.Notifier); ok { notifier.AddListener(dynamicCertificateController) } </ins> // start controllers if possible if controller, ok := s.ClientCA.(dynamiccertificates.ControllerRunner); ok { // runonce to be sure that we have a value.