Question: Float32 float32 // a base64 encoded characters ByteArray []byte <ins> // a member with an extension // +k8s:openapi-gen=x-kubernetes-member-tag:member_test WithExtension string </ins> } `) if err != nil {

how describing Nodes fetches Pod information. I still think that the solution is compatible with the richer solution you proposed and there is not much additional cost in doing it. Howv What would you think of supporting a set of optional ALPHA extensions for "display-columns" and "describe-columns" allowing any consumers of the swagger to print the objects. In theory it could also be used by the UI to provide a summary view without having to do separate lookups. As an ALPHA feature we could try it for supporting sig-service-catalog and revisit it in 1.7. If we consider it successful, the solution outlined in could become another consumer of the display-columns tag? I like this approach. My biggest concern here is being able to get this into 1.6. Since kubectl supports +/-1 version skew, we need will need to maintain the old logic for at least 1 full release after we add support for the new swagger based logic. That puts us at 6 months from now to fully migrate if we don't have any swagger extensions in 1.6.
I think that's a nice to have in the long run - but then people will say "why are the columns different when the server prints them from the local print"? And most of the pod columns in no way can represent the full complexity of what they print with swagger, so the best effort local printing is going to be really, really, really best effort. Also, local print is a really specialized case - if we just did the generic metadata print that would probably be ok. So while I agree it's valid, it seems like a lot of work for very little practical value. Again, not opposed, just don't want to do two things halfway and then have nothing. So just biased towards not adding more client complexity in the near term. So you want to hack this in for service catalog in 1.6 on TPR? I feel like that's a lot of hacks - the generic printer stuff at least gives you name and namespace, and I'd rather do nothing in 1.7 for generic client printer rather than jump ahead. I won't stop you if you want to spend time and effort on it, but it's a lot of specialized work that will then be neglected while we do the server side stuff.


Question: Float32 float32 // a base64 encoded characters ByteArray []byte <ins> // a member with an extension // +k8s:openapi-gen=x-kubernetes-member-tag:member_test WithExtension string </ins> } `) if err != nil {

My intend here was avoiding repeated code for adding each of these tags somewhere (in openapi-gen code or some root level place). Let me distinguish these tags in two categories with respect to OpenAPI code generator: that need some sort of processing that affects OpenAPI spec (e.g. +optional removes the field from OpenAPI field) that need no more processing and will be directly put into OpenAPI spec (e.g. +gvk that we don't have but we can define as a way to define Group/Version/Kind of an object vs register method) My argument is about the second one. I think patch strategy here falls into this category. The question is only about naming it. Naming Option 1: +patch-strategy=... This looks cleaner but we need to define an alias somewhere to tell generator this is the same as x-kubernetes-patch-strategy. The alias definition and the tags are in two places. Putting them in the generator code make the code non-generic and we don't want to do that. Putting these aliases somewhere as an argument to the generator begs for bugs as people that look at the type and add this tag to their types need to know (maybe by reading a doc, but who do that) that it needs to pass the alias to the generator. I, in 6 month, will look at a type and see a tag and copy the tag to my type and expect it to work that will not with this option. Naming Option 2: use something generic for these type of tags. after all generator does not need to do any processing. I suggest +openapi-tag:tag=value that will result in "x-kubernetes-tag: value" in OpenAPI spec. There is no alias defined anywhere and without any code change or alias copying or any knowledge about passing any argument, a tag can be to OpenAPI spec. The value of these tags and their use cases are outside of the scope of what I am arguing about. Anyway a solution is better than no solution, I would suggest we continue with Option 2 as Alpha and change it later if we see fit. Adding an aliasing mechanism to this option later should be easy.
On patch strategy, I had assumed we'd read the value from what's in the struct tag, rather than requiring a net new tag?


Question: Float32 float32 // a base64 encoded characters ByteArray []byte <ins> // a member with an extension // +k8s:openapi-gen=x-kubernetes-member-tag:member_test WithExtension string </ins> } `) if err != nil {

, Mehdy Bohlool <:
The get stuff is less urgent for me as long as we aren't doing even more hacky things to work around the current limitations, and can make a commitment to delivering it in a specific release (sounds like 1.7). For service catalog there was a plan to fork kubectl, compile the service catalog types into the binary, and provide the fork as a separate binary. I have some concerns with this approach. I would rather provide a temporary hack in the code that can provide a consistent user experience from day one and re-implement later vs having users get used to downloading separate binaries for each aggregated apiserver. I have no plans to do any work for on this for the core Kubernetes types, and can punt on it until the scope of the work is understood to be relatively small and easily upgraded to the server-side get.


Question: // get a typed object for this GVK if we need to apply a strategic merge patch obj, err := creater.New(gvk) if err != nil { <del> return nil, fmt.Errorf("cannot apply strategic merge patch for %s locally, try --type merge", gvk.String()) </del> <ins> return nil, fmt.Errorf("strategic merge patch is not supported for %s locally, try --type merge", gvk.String()) </ins> } return strategicpatch.StrategicMergePatch(originalJS, patchJS, obj)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Attempting to add a finalizer to a custom resource using a strategic merge patch fails: kubectl sends the request with a media type () that the server rejects: What you expected to happen: kubectl sends the patch with a media type that the server accepts. How to reproduce it (as minimally and precisely as possible): Apply the example custom resource definition from the sample controller project: Create a sample custom resource: Attempt to patch the CR: Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-12-18T12:09:25Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-12-18T12:00:47Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: libvirt virtual machine OS (e.g: ): NAME="Ubuntu" VERSION="20.04.1 LTS (Focal Fossa)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 20.04.1 LTS" VERSIONID="20.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" PRIVACYPOLICYURL="" VERSIONCODENAME=focal UBUNTUCODENAME=focal Kernel (e.g. ): Linux egw 5.4.0-56-generic -Ubuntu SMP Mon Nov 23 19:20:19 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
/sig cli
/assign Will look into this. You can refer to and the quota above. "strategic merge patch format is not supported" for custom resource.


Question: // get a typed object for this GVK if we need to apply a strategic merge patch obj, err := creater.New(gvk) if err != nil { <del> return nil, fmt.Errorf("cannot apply strategic merge patch for %s locally, try --type merge", gvk.String()) </del> <ins> return nil, fmt.Errorf("strategic merge patch is not supported for %s locally, try --type merge", gvk.String()) </ins> } return strategicpatch.StrategicMergePatch(originalJS, patchJS, obj)

The error message is not that clear.
When you run it with , it suggests to use --type=merge and it works. uses uses
This seems to work as expected. Do you think we need to make the server response more friendly as the message of .
Thank you, Paco, for looking into this! The problem is that merge patch is unable to do some important operations (for example, safely removing a finalizer from an object's metadata) so it's not a substitute for strategic merge. The error message suggesting that the client switch to merge patch is unhelpful, but a clear message indicating that strategic merge patch is unsupported would be helpful, and documentation that strategic merge patch is unsupported for Custom Resource objects would also be helpful. For most use cases the proper workaround is update with retry, not merge patch. Thank you!
/unassign
this is a topic for sig api machinery. /sig api-machinery
/cc /triage accepted
Do you think we need to make the server response more friendly as the message of kubectl client --dry-run=client. Apologies for the really late response on this. I've created to improve the message. /assign


Question: if nodeConditionsChanged(newNode, oldNode) { events = append(events, queue.NodeConditionChange) } <ins> if nodeAnnotationsChanged(newNode, oldNode) { events = append(events, queue.NodeAnnotationChange) } </ins> return events } func nodeAllocatableChanged(newNode *v1.Node, oldNode *v1.Node) bool { <del> return !reflect.DeepEqual(oldNode.Status.Allocatable, newNode.Status.Allocatable) </del> <ins> return !equality.Semantic.DeepEqual(oldNode.Status.Allocatable, newNode.Status.Allocatable) </ins> } func nodeLabelsChanged(newNode *v1.Node, oldNode *v1.Node) bool { <del> return !reflect.DeepEqual(oldNode.GetLabels(), newNode.GetLabels()) </del> <ins> return !equality.Semantic.DeepEqual(oldNode.GetLabels(), newNode.GetLabels()) </ins> } func nodeTaintsChanged(newNode *v1.Node, oldNode *v1.Node) bool { <del> return !reflect.DeepEqual(newNode.Spec.Taints, oldNode.Spec.Taints) </del> <ins> return !equality.Semantic.DeepEqual(newNode.Spec.Taints, oldNode.Spec.Taints) </ins> } func nodeConditionsChanged(newNode *v1.Node, oldNode *v1.Node) bool {

Due to limited character space, I've stored additional information in annotations on nodes. My custom scheduling plugins read these annotations to schedule Pods. However, when a Pod fails to be scheduled on a node, I want it to be removed from the Unschedulable queue. This can be achieved by modifying the annotations of the node, allowing the Pod to be rescheduled. Unfortunately, the eventHandler of the scheduler only supports five types of node events, and it does not include the nodeAnnotationsChanged event. Consequently, I am currently unable to trigger rescheduling by changing the annotations of nodes. /sig scheduling Unschedulable pods will be rescheduled after annotations of nodes changed. None No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign
It's more like a new feature, not bug because it doesn't lead to any bad behavior on the upstream. /remove-kind bug /kind feature
My concern is node annotation is arbitrary, change a node annotation doesn't imply a pod is schedulable. But for other events, like node labels, node taints, they're the constraints we consider in scheduling. They're different, I'm afraid this will lead to invalid scheduling attempts. Who is responsible for change the annotations in you scenario?
This enhancement has no impact on the in-tree plugins / upstream scheduler, right? Only (custom) plugins, that want to reschedule Pods via the node annotation change, will register event to EventsToRegister.
I have a controller capable of modifying node annotations to indicate node's current availability. And my custom plugins need to be aware of these changes.
I guess we can still be impacted when scheduling pod failed because of no fit node rather than failed by the plugin.


Question: if nodeConditionsChanged(newNode, oldNode) { events = append(events, queue.NodeConditionChange) } <ins> if nodeAnnotationsChanged(newNode, oldNode) { events = append(events, queue.NodeAnnotationChange) } </ins> return events } func nodeAllocatableChanged(newNode *v1.Node, oldNode *v1.Node) bool { <del> return !reflect.DeepEqual(oldNode.Status.Allocatable, newNode.Status.Allocatable) </del> <ins> return !equality.Semantic.DeepEqual(oldNode.Status.Allocatable, newNode.Status.Allocatable) </ins> } func nodeLabelsChanged(newNode *v1.Node, oldNode *v1.Node) bool { <del> return !reflect.DeepEqual(oldNode.GetLabels(), newNode.GetLabels()) </del> <ins> return !equality.Semantic.DeepEqual(oldNode.GetLabels(), newNode.GetLabels()) </ins> } func nodeTaintsChanged(newNode *v1.Node, oldNode *v1.Node) bool { <del> return !reflect.DeepEqual(newNode.Spec.Taints, oldNode.Spec.Taints) </del> <ins> return !equality.Semantic.DeepEqual(newNode.Spec.Taints, oldNode.Spec.Taints) </ins> } func nodeConditionsChanged(newNode *v1.Node, oldNode *v1.Node) bool {

see Can we predict whether the pod is schedulable before running into scheduling cycle via the node annotations, then we can leverage the preEnqueue extension point since you already have a external controller.
Thanks for the advice. But I'm afraid it would not be the best solution for us, because doing that outside the scheduling cycle would cause the custom plugins to malfunction in my scenario.
The concerns raised by make sense to me. In my understanding, most in-tree plugins will classify the event as an event.
No. When no fit Pod, UnschedulablePlugins will have plugin(s) rejected the Pod. (e.g., all nodes don't have enough capacity, UnschedulablePlugins has noderesourcefit. all nodes don't satisfy node affinity, UnschedulablePlugins has nodeaffinity) is a rare case like plugins other than PreFilter/Filter returns non-success. I don't think we should care about it.
But, this is a good point, I think we should refactor all plugins that register event with to have a specific Node update event. For example, . It should have with instead of . event with , that should be the last-resort when we don't know which kind of Node update will make Pods schedulable. (I guess no such case at least in the in-tree plugins.)
1) One possibility is Reserve Plugin, I the capability here 2) Another possibility is PreFilter, but not because failed by the plugin, but no fit node exists Have no idea whether we have other places, I guess no. But yes, this only impacts unschedulable pods has no failed plugins.
~I think this is for intended, like ~ ~1. 1) we have a plugin failed by nodeAffinity~ ~2. 2) we add a new node~ ~3. 3) we receive the node add event, but because node not ready, we failed by nodeTaint plugin as ~ 4) node become ready, we get the node update event, but dismatch with nodeAffinity registered event (if we registered with UpdateNodeLabel event)~ ~5. 5) pod will not be moved to the active or backoff queue~ updated below
weigh in for advices as whether this is a good idea to take nodeAnnotations as a new event.


Question: if nodeConditionsChanged(newNode, oldNode) { events = append(events, queue.NodeConditionChange) } <ins> if nodeAnnotationsChanged(newNode, oldNode) { events = append(events, queue.NodeAnnotationChange) } </ins> return events } func nodeAllocatableChanged(newNode *v1.Node, oldNode *v1.Node) bool { <del> return !reflect.DeepEqual(oldNode.Status.Allocatable, newNode.Status.Allocatable) </del> <ins> return !equality.Semantic.DeepEqual(oldNode.Status.Allocatable, newNode.Status.Allocatable) </ins> } func nodeLabelsChanged(newNode *v1.Node, oldNode *v1.Node) bool { <del> return !reflect.DeepEqual(oldNode.GetLabels(), newNode.GetLabels()) </del> <ins> return !equality.Semantic.DeepEqual(oldNode.GetLabels(), newNode.GetLabels()) </ins> } func nodeTaintsChanged(newNode *v1.Node, oldNode *v1.Node) bool { <del> return !reflect.DeepEqual(newNode.Spec.Taints, oldNode.Spec.Taints) </del> <ins> return !equality.Semantic.DeepEqual(newNode.Spec.Taints, oldNode.Spec.Taints) </ins> } func nodeConditionsChanged(newNode *v1.Node, oldNode *v1.Node) bool {

As I know, some plugins will read the embedded prometheus metrics in node annotations for scoring, but not for filtering. At least we should mitigate the impacts.
iv. node become ready, we get the node update event, but dismatch with nodeAffinity registered event (if we registered with UpdateNodeLabel event) (3) is not exact. When we receive the node add event, the scheduler retry to schedule the Pod because the nodeaffinity has Node add event. And it's rejected by the nodetaint plugin. (not node affinity) So, in (4), we get the node update event, it matches with node taint registered event, and the Pod is moved to active/backoffQ (node taint registers node update event. Yes I think we can change node taint's node update event to more specific one(s) though)
As I replied in your PR, we acknowledged that in , and yes let's have your patch. Well, that's a good point. So, when all nodes are filtered out by PreFilterResult, then no failedplugin can be returned. That's a problem. I think we should return the plugin name that returns PreFilterResult with not all Node names in this case (= the plugins that contributes to reducing the candidate Nodes via PreFilterResult). What do you think?
Sorry, the previous one is not right, a detailed one: a. we have a pod failed by nodeAffinity. pod will be placed into the unschedulablePods b. node autoscaler triggered, a new node , emitted a node add event c. called -d. node not ready right now and holds a notReady taint, then failed, pod still stays in the unschedulablePods e. node gets ready, emitted a node update event, but dismatch with the registered event(if we register with in nodeAffinity plugin) f. pod will stay in unschedulabelPods I guess this is the right flow. Anyway, glad to see it be more effect in the future.
If we have no performance problems, I think it's ok to take nodeAnnotations into consideration.
Wouldn't this be solved with proposal on ClusterEventWithHint? If so, I prefer we leave the existing code as-is.
whatever we do here, we wouldn't backport, so I think is the right move forward.
We put many stuff here.


Question: if nodeConditionsChanged(newNode, oldNode) { events = append(events, queue.NodeConditionChange) } <ins> if nodeAnnotationsChanged(newNode, oldNode) { events = append(events, queue.NodeAnnotationChange) } </ins> return events } func nodeAllocatableChanged(newNode *v1.Node, oldNode *v1.Node) bool { <del> return !reflect.DeepEqual(oldNode.Status.Allocatable, newNode.Status.Allocatable) </del> <ins> return !equality.Semantic.DeepEqual(oldNode.Status.Allocatable, newNode.Status.Allocatable) </ins> } func nodeLabelsChanged(newNode *v1.Node, oldNode *v1.Node) bool { <del> return !reflect.DeepEqual(oldNode.GetLabels(), newNode.GetLabels()) </del> <ins> return !equality.Semantic.DeepEqual(oldNode.GetLabels(), newNode.GetLabels()) </ins> } func nodeTaintsChanged(newNode *v1.Node, oldNode *v1.Node) bool { <del> return !reflect.DeepEqual(newNode.Spec.Taints, oldNode.Spec.Taints) </del> <ins> return !equality.Semantic.DeepEqual(newNode.Spec.Taints, oldNode.Spec.Taints) </ins> } func nodeConditionsChanged(newNode *v1.Node, oldNode *v1.Node) bool {

Let me summarize each: Born from from is right. But, that's the problem with preCheck. (Some preCheck depends on some in-tree plugins' logic and it affects badly to other plugins enqueueing ) The preCheck is removed after is completed. After preCheck is removed, will be the correct behavior. So, we can do the optimization that changes node to a specific event. will be fixed in: It's kind of rare case, but I feel we need to have some improvement like Edit: create another issue for this: I'd say (2) and (3) are kind of rare cases. So, the blocker here is (1). (And, (1) is blocked by the removal of preCheck, which is blocked by ) As my conclusion, it's safer to add after (1) is done so that we can make sure the performance won't be affected badly. what do you think?
Instead, can we just keep an event for "node updated", then individual plugins can decide whether a annotations change is worth requeuing or not?
Yes, (1) can be replaced with that idea. But, I'd say changing to a specific is slightly better because the scheduling queue could filter out some Node update events without executing the hint functions.
If rescheduling can be triggered with precise events, that would be great. I haven't revisit since last time, I'll take a look again.
Since we are missing an event for , can we add it? But we also have to modify existing events like so: Because change of such field is also an Update. And in future releases we can get rid of some fine-grained events.
Since we will add hintFuncs for all in-tree plugins, so I think it's ok to add nodeAnnotationsChanged event now, we can filter out unrelated events in hitFuncs. Any other concerns? cc Prefer to keep them as people can register fine-gained events instead a general event.
If by "now" you mean v1.30, then I'm ok with that.


Question: import ( "fmt" <del> "os" "path/filepath" </del> "regexp" "strconv" "strings"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: We have a private registry configured (where all images including the sandbox one), are hosted. The registry credentials are being pulled by one of k8s credentials providers () With Kubernetes 1.7 sandbox image failed to be pulled with the following error message: Looks like with this only creds from docker config are being evaluated on the sandbox image pull What you expected to happen: Sandbox image pull should be attempted using credentials returned by not just the basic docker credential provider, but other credentials providers registered in the system. How to reproduce it (as minimally and precisely as possible): 1) Register private repo credentials with one of the k8s credential providers (we've tried with rancher). Configure the to use the image from the the private repo. 2) Make sure you are not logged in to this private repo on the kubelet host 3) Try to create a pod. Sandbox image pull will fail with an error mentioned above. Anything else we need to know?: Non-sandbox images from the same private repo were able to download correctly using private repo credentials provider. Environment: Kubernetes version (use ): 1.7.2 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node Just tested the same use case against k8s 1.5.x, it worked fine there and sandbox image was pulled using the credentials passed by the credential provider, not from docker config file. So it might have regressed in 1.7.x (haven't tested it on 1.6, so can't tell for sure)
Confirmed a bug introduced in dockershim. Will file a PR to fix this.


Question: import ( "fmt" <del> "os" "path/filepath" </del> "regexp" "strconv" "strings"

Could you also verify whether fixed your problem?
tested against your commits cherry-picked on top of 1.8 beta, the fix seems to work
thanks.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: AllocatedResourcesStatus: []v1.ResourceStatus{expectedStatus}, }, } <ins> // Sort the resources for the expected status and actual status sortContainerStatuses(status.ContainerStatuses) sortContainerStatuses(expectedContainerStatuses) </ins> if !reflect.DeepEqual(status.ContainerStatuses, expectedContainerStatuses) { t.Errorf("UpdateAllocatedResourcesStatus failed, expected: %v, got: %v", expectedContainerStatuses, status.ContainerStatuses) } } <ins> // Helper function to sort ResourceHealth slices func sortResourceHealth(resources []v1.ResourceHealth) { sort.SliceStable(resources, func(i, j int) bool { return resources[i].ResourceID < resources[j].ResourceID }) } // Helper function to sort ContainerStatus slices func sortContainerStatuses(statuses []v1.ContainerStatus) { for i := range statuses { for j := range statuses[i].AllocatedResourcesStatus { sortResourceHealth(statuses[i].AllocatedResourcesStatus[j].Resources) } } } </ins>

TestUpdateAllocatedResourcesStatus Few days ago No response No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
I could reproduce this with in maybe 1/10 times.
cc I am unfamilar with this code so wanted to see if you have any ideas on this.
yes, I'll have a look ASAP
In the flake, the ResourceHealth array is with a different order. If this is acceptable, we should tune the test.
nice catch ! I think the best fix is ensure the ordering of statuses either in the production code or in the test code before to check Note: if we do this in the production code then we should go full throttle and offer a full API guarantee (docs, explicit tests) because users will depend on this
A quick fix is like if we want to ignore the order.


Question: go_test( name = "go_default_test", srcs = [ <ins> "azure_loadbalancer_test.go", </ins> "azure_test.go", "azure_util_test.go", ],

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area cloudprovider /sig azure What happened: I've created a service of type in a Kubernetes cluster created via the ACS Engine. The load balancer has been created correctly after some time and a random NodePort (at this time ) has been allocated. However, in the Azure Portal I'm seeing that the health probe of that load balancer is pointing to port which does not exist in the whole Kubernetes system at all. Because of that wrongly configured health probe the load balancer is not reachable (i/o timeout). What you expected to happen: The health probe would have been configured to NodePort . How to reproduce it (as minimally and precisely as possible): Unfortunately, it does not behave always like this but only occasionally. In some cases the health probe gets configured correctly, but in others not. I've used the following manifest: Anything else we need to know?: When updating the health probe manually to the correct NodePort in the Azure portal, the load balancer starts to work properly. Environment: Kubernetes version (use ): 1.7.5 Cloud provider or hardware configuration: Azure Install tools: ACS Engine Others: -
Thanks for reporting the problem. Could you share the service spec after it is created? e.g. by running . And may I ask whether there is a service just removed with same name before this one?
Sure, thanks, please find the manifest below [1]. In fact I'll have some other services with the same name, but in other namespaces. In the namespace I didn't have a service named before. [1] Service manifest after creation
Filed PR , which should fix the problem. Need cherry-picked to v1.7 branch after it is merged.
Thank you very much that was really fast!
has been merged. created cherry pick PR for release 1.7.


Question: } return allErrs } <ins> func validateNodeSelectorMutation(fldPath *field.Path, newNodeSelector, oldNodeSelector map[string]string) field.ErrorList { var allErrs field.ErrorList // Validate no existing node selectors were deleted or mutated. for k, v1 := range oldNodeSelector { if v2, ok := newNodeSelector[k]; !ok || v1 != v2 { allErrs = append(allErrs, field.Invalid(fldPath, newNodeSelector, "only additions to spec.nodeSelector are allowed (no mutations or deletions)")) return allErrs } } return allErrs } func validateNodeAffinityMutation(nodeAffinityPath *field.Path, newNodeAffinity, oldNodeAffinity *core.NodeAffinity) field.ErrorList { var allErrs field.ErrorList // If old node affinity was nil, anything can be set. if oldNodeAffinity == nil || oldNodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution == nil { return allErrs } oldTerms := oldNodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution.NodeSelectorTerms var newTerms []core.NodeSelectorTerm if newNodeAffinity != nil && newNodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution != nil { newTerms = newNodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution.NodeSelectorTerms } // If there are no old terms, we can set the new terms to anything. // If there are old terms, we cannot add any new ones. if len(oldTerms) > 0 && len(oldTerms) != len(newTerms) { return append(allErrs, field.Invalid(nodeAffinityPath.Child("requiredDuringSchedulingIgnoredDuringExecution").Child("nodeSelectorTerms"), newTerms, "no additions/deletions to non-empty NodeSelectorTerms list are allowed")) } // For requiredDuringSchedulingIgnoredDuringExecution, if old NodeSelectorTerms // was empty, anything can be set. If non-empty, only additions of NodeSelectorRequirements // to matchExpressions or fieldExpressions are allowed. for i := range oldTerms { if !validateNodeSelectorTermHasOnlyAdditions(newTerms[i], oldTerms[i]) { allErrs = append(allErrs, field.Invalid(nodeAffinityPath.Child("requiredDuringSchedulingIgnoredDuringExecution").Child("nodeSelectorTerms").Index(i), newTerms[i], "only additions are allowed (no mutations or deletions)")) } } return allErrs } func validateNodeSelectorTermHasOnlyAdditions(newTerm, oldTerm core.NodeSelectorTerm) bool { if len(oldTerm.MatchExpressions) == 0 && len(oldTerm.MatchFields) == 0 { if len(newTerm.MatchExpressions) > 0 || len(newTerm.MatchFields) > 0 { return false } } // Validate MatchExpressions only has additions (no deletions or mutations) if l := len(oldTerm.MatchExpressions); l > 0 { if len(newTerm.MatchExpressions) < l { return false } if !apiequality.Semantic.DeepEqual(newTerm.MatchExpressions[:l], oldTerm.MatchExpressions) { return false } } // Validate MatchFields only has additions (no deletions or mutations) if l := len(oldTerm.MatchFields); l > 0 { if len(newTerm.MatchFields) < l { return false } if !apiequality.Semantic.DeepEqual(newTerm.MatchFields[:l], oldTerm.MatchFields) { return false } } return true } </ins>

See for context. This issue is creating to track work on the API validation changes. Validation relaxation of certain fields in the pod spec to allow for mutable scheduling directives.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scheduling
Docs PR:


Question: return "", "", err } <del> scsiController = getSCSIController(vmDevices, vs.cfg.Disk.SCSIControllerType) </del> <ins> // Get VM device list _, vmDevices, _, err := getVirtualMachineDevices(ctx, vs.cfg, vs.client, vSphereInstance) if err != nil { glog.Errorf("cannot get vmDevices for VM err=%s", err) return "", "", fmt.Errorf("cannot get vmDevices for VM err=%s", err) } scsiControllersOfRequiredType := getSCSIControllersOfType(vmDevices, diskControllerType) scsiController := getAvailableSCSIController(scsiControllersOfRequiredType) </ins> if scsiController == nil { glog.Errorf("cannot find SCSI controller in VM") // attempt clean up of scsi controller

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): v1.4.x v1.5.x v1.6.x master Environment: vSphere based install with vSphere Cloud Provider enabled. Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): NA Kernel (e.g. ):NA Install tools:kubernetes-anywhere Others: What happened: vSphere Cloud Provider configures additional SCSI adapters to scale the number of volumes that can be attached to a VM (1 adapter = 16 disks). The code within the cloud provider does not handle this correctly. Additional disks attach successfully to the VM but when k8s verifies the attach status it fails leading to k8s retrying the attach that never succeeds. In addition to this when the SCSI adapter is created for the first time the code does not use it to attach the disk leading to a failed attach disk request to the cloud provider. What you expected to happen: Additional disks should be to the VM upto the max number supported. How to reproduce it (as minimally and precisely as possible): Create a single node cluster and create a statefulset with 16 replicas or have a pod with more than 16 disks. Anything else we need to know:
A fix for this is coming up..
cc
Huh, I wonder if something changed between 1.3 and newer?


Question: return "", "", err } <del> scsiController = getSCSIController(vmDevices, vs.cfg.Disk.SCSIControllerType) </del> <ins> // Get VM device list _, vmDevices, _, err := getVirtualMachineDevices(ctx, vs.cfg, vs.client, vSphereInstance) if err != nil { glog.Errorf("cannot get vmDevices for VM err=%s", err) return "", "", fmt.Errorf("cannot get vmDevices for VM err=%s", err) } scsiControllersOfRequiredType := getSCSIControllersOfType(vmDevices, diskControllerType) scsiController := getAvailableSCSIController(scsiControllersOfRequiredType) </ins> if scsiController == nil { glog.Errorf("cannot find SCSI controller in VM") // attempt clean up of scsi controller

We had tested up to 40-something volumes on 1.3. There was a fun discussion internally because it would always fail at SCSI ID 7 and I had to explain SCSI to some younger engineers :)


Question: // Build a detailed log of the denial. // Make the whole block conditional so we don't do a lot of string-building we won't use. <del> if glog.V(2) { </del> <ins> if glog.V(5) { </ins> var operation string if requestAttributes.IsResourceRequest() { b := &bytes.Buffer{}

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: the psp admission controller iterates through applicable psps for both the user and the service account. This leads to at least one RBAC DENY message for each pod. What you expected to happen: "RBAC DENY" should be a more meaningful message How to reproduce it (as minimally and precisely as possible): enable psp w/rbac
closing in favor of


Question: } // Check nodeTree after remove. The node should be removed from the nodeTree even if there are // still pods on it. <del> if cache.nodeTree.NumNodes != 0 || cache.nodeTree.Next() != "" { </del> <ins> if cache.nodeTree.NumNodes() != 0 || cache.nodeTree.Next() != "" { </ins> t.Errorf("unexpected cache.nodeTree after removing node: %v", node.Name) } }

Description: , part of the is exporting a field which allows to modify its value externally. This also means that this field is not thread-safe. Solution: Expose a method that returns this value in a thread-safe way and do not export the field. /kind bug /sig scheduling


Question: import ( "fmt" "log" <del> "net/http" "net/http/httptest" </del> "reflect" "runtime" "sync"

In 0.13.2, the apiserver has automatically exposed metrics about the process, like start time, memory usage, processor usage, etc. This stopped working sometime between then and when 0.14.0 was cut. They're still correctly exported from the kubelet, interestingly enough. An e2e or integration test is also needed to catch this if it happens again.


Question: maxUnavailable: 1, numUnavailable: 0, }, <ins> { name: "Two nodes with pods, MaxUnavailable in percents, pod terminating", Manager: func() *daemonSetsController { manager, _, _ := newTestController() addNodes(manager.nodeStore, 0, 2, nil) return manager }(), ds: func() *extensions.DaemonSet { ds := newDaemonSet("x") intStr := intstr.FromString("50%") ds.Spec.UpdateStrategy.RollingUpdate = &extensions.RollingUpdateDaemonSet{MaxUnavailable: &intStr} return ds }(), nodeToPods: func() map[string][]*v1.Pod { mapping := make(map[string][]*v1.Pod) pod0 := newPod("pod-0", "node-0", simpleDaemonSetLabel, nil) pod1 := newPod("pod-1", "node-1", simpleDaemonSetLabel, nil) now := metav1.Now() markPodReady(pod0) markPodReady(pod1) pod1.DeletionTimestamp = &now mapping["node-0"] = []*v1.Pod{pod0} mapping["node-1"] = []*v1.Pod{pod1} return mapping }(), maxUnavailable: 1, numUnavailable: 1, }, </ins> } for _, c := range cases {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The DaemonSet controller will schedule a pod on a node when the old pod is still in Terminating state. This breaks the expectation that only one pod will be running at a time per node. What you expected to happen: The DaemonSet controller should not let two pods running at the same time on the same node. How to reproduce it (as minimally and precisely as possible): Create a DaemonSet. Set terminationGracePeriodSeconds to be relatively long. Kill a pod. Watch a new pod start on the same node when the old pod is still terminating. Anything else we need to know?: Seems this behavior was introduced in Environment: Kubernetes version (use ): 1.7.2 Cloud provider or hardware configuration**: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps
/cc
cc
cc: I think the expected behavior is correct, the terminating pod could be performing an important step such as 'removing a lock' or 'cleanup' etc., we should allow that to gracefully die before creating a new one. IMHO we should revoke to old behavior, but why did we need this new behavior in the first place? That is not clear to me from the PR and Issue.
We don't currently guarantee at least or at most one scheduling for DaemonSet. Both behaviors are under consideration via issue and in the future work section of the . The reason for is to "Make sure that controllers are not adopting children that are marked for deletion" (). This is necessary for GC to work properly.
Well the behavior may be technically undefined, but can I request that in the future changes like this be mentioned in the release notes?
Yes, certainly.


Question: maxUnavailable: 1, numUnavailable: 0, }, <ins> { name: "Two nodes with pods, MaxUnavailable in percents, pod terminating", Manager: func() *daemonSetsController { manager, _, _ := newTestController() addNodes(manager.nodeStore, 0, 2, nil) return manager }(), ds: func() *extensions.DaemonSet { ds := newDaemonSet("x") intStr := intstr.FromString("50%") ds.Spec.UpdateStrategy.RollingUpdate = &extensions.RollingUpdateDaemonSet{MaxUnavailable: &intStr} return ds }(), nodeToPods: func() map[string][]*v1.Pod { mapping := make(map[string][]*v1.Pod) pod0 := newPod("pod-0", "node-0", simpleDaemonSetLabel, nil) pod1 := newPod("pod-1", "node-1", simpleDaemonSetLabel, nil) now := metav1.Now() markPodReady(pod0) markPodReady(pod1) pod1.DeletionTimestamp = &now mapping["node-0"] = []*v1.Pod{pod0} mapping["node-1"] = []*v1.Pod{pod1} return mapping }(), maxUnavailable: 1, numUnavailable: 1, }, </ins> } for _, c := range cases {

Also "The DaemonSet controller should not let two pods running at the same time on the same node." is not necessarily the same thing as respecting termination prior to launching a new Pod. The former implies a more strict requirement than the latter. I think we should consider at least attempting to wait for a terminating Pod to be deleted prior to creating another Pod on the Node for same DaemonSet as default behavior.
will something like work then?


Question: func (options *ServerRunOptions) Validate() []error { var errors []error <ins> if errs := options.GenericServerRunOptions.Validate(); len(errs) > 0 { errors = append(errors, errs...) } </ins> if errs := options.Etcd.Validate(); len(errs) > 0 { errors = append(errors, errs...) }

See I think it's the last file that has no validation in pkg . Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery


Question: return false, fmt.Errorf("Spec does not reference a RBD volume type") } <ins> func getVolumeAccessModes(spec *volume.Spec) ([]v1.PersistentVolumeAccessMode, error) { // Only PersistentVolumeSpec has AccessModes if spec.PersistentVolume != nil { if spec.PersistentVolume.Spec.RBD != nil { return spec.PersistentVolume.Spec.AccessModes, nil } else { return nil, fmt.Errorf("Spec does not reference a RBD volume type") } } return nil, nil } </ins> func parsePodSecret(pod *v1.Pod, secretName string, kubeClient clientset.Interface) (string, error) { secret, err := volutil.GetSecretForPod(pod, secretName, kubeClient) if err != nil {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When use ceph as storageclass, i set accessModes: ReadOnlyMany, and create pods using the claim, only one pod would be running and the others on different nodes would get stuck on ContainerCreating. But if the pods are all on same node, it works fine. What you expected to happen: Pods on different nodes can mount ceph rbd volume. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.9.2 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): 3.10.0-327.18.2.el7.x86_64 Kernel (e.g. ): CentOS Linux release 7.2.1511 (Core) Install tools: kubeadm Others:
/sig storage
/assign
, how would this work? For example, how do you put the initial contents of the RBD volume on it if it is in the ReadOnlyMany access mode? How would you use this feature? As a cheap way of sharing files across a Kubernetes cluster? I'm just interested. -ben england
Do we really want to support ? I have the same questions as mentioned above. If we set the PVC AccessMode to , then the field in will need to be set true ? If so, for the first time, in , we will mount the device without formatting, is that intended ? We need to format the device manually before we use it ? /cc FYI: also there is an issue related to this:
We have this same problem for GCE PD, which supports ReadOnlyMany. You need to mount the volume first as RWO, in order to format the filesystem and populate it with data (mounting a read only volume with no data on it seems pointless). Then you have to fully unmount/detach it, and then you can deploy pods using it as ROX. Basically, you can dynamically provision it in RWO to format + populate data. But afterwards, you need to manually create a different PV/PVC for ROX
continuation


Question: <ins> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Bring up a Kubernetes cluster. # Usage: #   wget -q -O - https://get.k8s.io | sh # or #   curl -sS https://get.k8s.io | sh # # Advanced options #  Set KUBERNETES_PROVIDER to choose between different providers: #  Google Compute Engine [default] #   * export KUBERNETES_PROVIDER=gce; wget -q -O - https://get.k8s.io | sh #  Amazon EC2 #   * export KUBERNETES_PROVIDER=aws; wget -q -O - https://get.k8s.io | sh #  Microsoft Azure #   * export KUBERNETES_PROVIDER=azure; wget -q -O - https://get.k8s.io | sh #  Vagrant (local virtual machines) #   * export KUBERNETES_PROVIDER=vagrant; wget -q -O - https://get.k8s.io | sh #  VMWare VSphere #   * export KUBERNETES_PROVIDER=vsphere; wget -q -O - https://get.k8s.io | sh #  Rackspace #   * export KUBERNETES_PROVIDER=rackspace; wget -q -O - https://get.k8s.io | sh # #  Set KUBERNETES_SKIP_DOWNLOAD to non-empty to skip downloading a release. #  Set KUBERNETES_SKIP_CONFIRM to skip the installation confirmation prompt. set -o errexit set -o nounset set -o pipefail function create-cluster { echo "Creating a kubernetes on ${KUBERNETES_PROVIDER:-gce}..." ( cd kubernetes ./cluster/kube-up.sh echo "Kubernetes binaries at ${PWD}/kubernetes/cluster/" echo "You may want to add this directory to your PATH in $HOME/.profile" echo "Installation successful!" ) } if [[ "${KUBERNETES_SKIP_DOWNLOAD-}" ]]; then create-cluster exit 0 fi release=v0.7.2 release_url=https://storage.googleapis.com/kubernetes-release/release/${release}/kubernetes.tar.gz uname=$(uname) if [[ "${uname}" == "Darwin" ]]; then platform="darwin" elif [[ "${uname}" == "Linux" ]]; then platform="linux" else echo "Unknown, unsupported platform: (${uname})." echo "Supported platforms: Linux, Darwin." echo "Bailing out." exit 2 fi machine=$(uname -m) if [[ "${machine}" == "x86_64" ]]; then arch="amd64" elif [[ "${machine}" == "i686" ]]; then arch="386" elif [[ "${machine}" == "arm*" ]]; then arch="arm" else echo "Unknown, unsupported architecture (${machine})." echo "Supported architectures x86_64, i686, arm*" echo "Bailing out." exit 3 fi file=kubernetes.tar.gz echo "Downloading kubernetes release ${release} to ${PWD}/kubernetes.tar.gz" if [[ -n "${KUBERNETES_SKIP_CONFIRM-}" ]]; then echo "Is this ok? [Y]/n" read confirm if [[ "$confirm" == "n" ]]; then echo "Aborting." exit 0 fi fi if [[ $(which wget) ]]; then wget -O ${file} ${release_url} elif [[ $(which curl) ]]; then curl -L -o ${file} ${release_url} else echo "Couldn't find curl or wget.  Bailing out." exit 1 fi echo "Unpacking kubernetes release ${release}" tar -xzf ${file} rm ${file} create-cluster </ins>

Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here: Additional Context:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
cc Do we want to keep this job and fix it, or do something different considering the debs/rpms efforts?
I think we should keep this job as long as we build packages for Google infra (those are using different specs then packages for OBS).
fixed it:


Question: k8sNodeRouteTag = "k8s-node-route" // AffinityTypeNone - no session affinity. <del> gceAffinityTypeNone = "None" </del> <ins> gceAffinityTypeNone = "NONE" </ins> // AffinityTypeClientIP - affinity based on Client IP. gceAffinityTypeClientIP = "CLIENT_IP" // AffinityTypeClientIPProto - affinity based on Client IP and port.

Hi, I have a problem with my GCE cluster about LoadBalancer type Services It's creating a lot of unused static external IP on the google network of my project the cluster is running kubernetes v1.1.2. Tried restarting nodes, tried destroy/recreate of the entire cluster.
Seeing exactly the same issue at GKE since a few hours. First on a v1.0.7 cluster which I later upgraded to v1.1.2 to fix the problem but the issue is still there. Double checked my yaml config files but no changes from our side. Existing loadbalancers do work as before.
I dealt with this last night and a workaround was to add to the spec
Thx :+1:
We'll get this cherry picked into the two release branches once it's been verified at HEAD.


Question: grains: node_ip: $MASTER_IP master_ip: $MASTER_IP <ins> network_mode: openvswitch </ins> etcd_servers: $MASTER_IP cloud_provider: vagrant roles:

The salt configs assume that anything under "os_family = RedHat" will use the ovs mechanisms that are only set up for vagrant. This means that we can't use Fedora on GCE.
I can clean this up today.


Question: } // Update updates the pet in the 'pet' pcb to match the pet in the 'expectedPet' pcb. <ins> // If the pod object of a pet which to be updated has been changed in server side, we // will get the actual value and set pet identity before retries. </ins> func (p *apiServerPetClient) Update(pet *pcb, expectedPet *pcb) (updateErr error) { <del> var getErr error </del> pc := podClient(p.c, pet.parent.Namespace) <del> pod, needsUpdate, err := copyPetID(pet, expectedPet) if err != nil || !needsUpdate { return err } glog.Infof("Resetting pet %v to match PetSet %v spec", pod.Name, pet.parent.Name) for i, p := 0, &pod; ; i++ { _, updateErr = pc.Update(p) </del> <ins> for i := 0; ; i++ { updatePod, needsUpdate, err := copyPetID(pet, expectedPet) if err != nil || !needsUpdate { return err } glog.Infof("Resetting pet %v/%v to match PetSet %v spec", pet.pod.Namespace, pet.pod.Name, pet.parent.Name) _, updateErr = pc.Update(&updatePod) </ins> if updateErr == nil || i >= updateRetries { return updateErr } <del> if p, getErr = pc.Get(pod.Name); getErr != nil { </del> <ins> getPod, getErr := pc.Get(updatePod.Name) if getErr != nil { </ins> return getErr } <ins> pet.pod = getPod </ins> } }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Consider the following code, When previous update fails, Pet Set Controller will Get a new Pod which may don't contain the Pet identity information(see ). If so, we can't 100% update Pod with a given pet identity information. I think it's a bug and we should do between update retries?
Fixed.


Question: if err := validateHost(); err != nil { return &validator{validateHostErr: err} } <del> appArmorFS, err := getAppArmorFS() if err != nil { return &validator{ validateHostErr: fmt.Errorf("error finding AppArmor FS: %v", err), } } return &validator{ appArmorFS: appArmorFS, } </del> <ins> return &validator{} </ins> } type validator struct { validateHostErr error <del> appArmorFS      string </del> } func (v *validator) Validate(pod *v1.Pod) error {

As mentioned the return value of is not being used any longer and the entire method can be removed. The usage of was removed in Now there is no need to have the function and store it's result at all. /sig node /kind cleanup /good-first-issue /help-wanted KEP xref:
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : As mentioned the return value of is not being used any longer and the entire method can be removed. /kind cleanup /good-first-issue /help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/assign
I am having issues submitting the pull request. I removed that portion of the code under however this is my first time contributing and would like some help submitting this new updated version..
/assign
/assign
/assign
/assign


Question: <ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package storage import ( "context" "fmt" "path/filepath" "time" v1 "k8s.io/api/core/v1" storagev1 "k8s.io/api/storage/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" clientset "k8s.io/client-go/kubernetes" "github.com/onsi/ginkgo" "github.com/onsi/gomega" "k8s.io/apimachinery/pkg/util/rand" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" "k8s.io/kubernetes/test/e2e/storage/testsuites" "k8s.io/kubernetes/test/e2e/storage/utils" ) var _ = utils.SIGDescribe("PersistentVolumes-expansion ", func() { f := framework.NewDefaultFramework("persistent-local-volumes-expansion") ginkgo.Context("loopback local block volume", func() { var ( config *localTestConfig scName string ) testVolType := BlockFsWithFormatLocalVolumeType var testVol *localTestVolume testMode := immediateMode ginkgo.BeforeEach(func() { nodes, err := e2enode.GetBoundedReadySchedulableNodes(f.ClientSet, maxNodes) framework.ExpectNoError(err) scName = fmt.Sprintf("%v-%v", testSCPrefix, f.Namespace.Name) // Choose a random node randomNode := &nodes.Items[rand.Intn(len(nodes.Items))] hostExec := utils.NewHostExec(f) ltrMgr := utils.NewLocalResourceManager("local-volume-test", hostExec, hostBase) config = &localTestConfig{ ns:           f.Namespace.Name, client:       f.ClientSet, timeouts:     f.Timeouts, nodes:        nodes.Items, randomNode:   randomNode, scName:       scName, discoveryDir: filepath.Join(hostBase, f.Namespace.Name), hostExec:     hostExec, ltrMgr:       ltrMgr, } setupExpandableLocalStorageClass(config, &testMode) testVols := setupLocalVolumesPVCsPVs(config, testVolType, config.randomNode, 1, testMode) testVol = testVols[0] }) ginkgo.AfterEach(func() { cleanupLocalVolumes(config, []*localTestVolume{testVol}) cleanupStorageClass(config) }) ginkgo.It("should support online expansion on node", func() { var ( pod1    *v1.Pod pod1Err error ) ginkgo.By("Creating pod1") pod1, pod1Err = createLocalPod(config, testVol, nil) framework.ExpectNoError(pod1Err) verifyLocalPod(config, testVol, pod1, config.randomNode.Name) // We expand the PVC while l.pod is using it for online expansion. ginkgo.By("Expanding current pvc") currentPvcSize := testVol.pvc.Spec.Resources.Requests[v1.ResourceStorage] newSize := currentPvcSize.DeepCopy() newSize.Add(resource.MustParse("10Mi")) framework.Logf("currentPvcSize %s, newSize %s", currentPvcSize.String(), newSize.String()) newPVC, err := testsuites.ExpandPVCSize(testVol.pvc, newSize, f.ClientSet) framework.ExpectNoError(err, "While updating pvc for more size") testVol.pvc = newPVC gomega.Expect(testVol.pvc).NotTo(gomega.BeNil()) pvcSize := testVol.pvc.Spec.Resources.Requests[v1.ResourceStorage] if pvcSize.Cmp(newSize) != 0 { framework.Failf("error updating pvc size %q", testVol.pvc.Name) } // Now update the underlying volume manually err = config.ltrMgr.ExpandBlockDevice(testVol.ltr, 10 /*number of 1M blocks to add*/) framework.ExpectNoError(err, "while expanding loopback device") // now update PV to matching size pv, err := UpdatePVSize(testVol.pv, newSize, f.ClientSet) framework.ExpectNoError(err, "while updating pv to more size") gomega.Expect(pv).NotTo(gomega.BeNil()) testVol.pv = pv ginkgo.By("Waiting for file system resize to finish") testVol.pvc, err = testsuites.WaitForFSResize(testVol.pvc, f.ClientSet) framework.ExpectNoError(err, "while waiting for fs resize to finish") pvcConditions := testVol.pvc.Status.Conditions framework.ExpectEqual(len(pvcConditions), 0, "pvc should not have conditions") }) }) }) func UpdatePVSize(pv *v1.PersistentVolume, size resource.Quantity, c clientset.Interface) (*v1.PersistentVolume, error) { pvName := pv.Name pvToUpdate := pv.DeepCopy() var lastError error waitErr := wait.PollImmediate(5*time.Second, csiResizeWaitPeriod, func() (bool, error) { var err error pvToUpdate, err = c.CoreV1().PersistentVolumes().Get(context.TODO(), pvName, metav1.GetOptions{}) if err != nil { return false, fmt.Errorf("error fetching pv %s: %v", pvName, err) } pvToUpdate.Spec.Capacity[v1.ResourceStorage] = size pvToUpdate, err = c.CoreV1().PersistentVolumes().Update(context.TODO(), pvToUpdate, metav1.UpdateOptions{}) if err != nil { framework.Logf("error updating PV %s: %v", pvName, err) lastError = err return false, nil } return true, nil }) if waitErr == wait.ErrWaitTimeout { return nil, fmt.Errorf("timed out attempting to update PV size. last update error: %v", lastError) } if waitErr != nil { return nil, fmt.Errorf("failed to expand PV size: %v", waitErr) } return pvToUpdate, nil } func setupExpandableLocalStorageClass(config *localTestConfig, mode *storagev1.VolumeBindingMode) { enableExpansion := true sc := &storagev1.StorageClass{ ObjectMeta: metav1.ObjectMeta{ Name: config.scName, }, Provisioner:          "kubernetes.io/no-provisioner", VolumeBindingMode:    mode, AllowVolumeExpansion: &enableExpansion, } _, err := config.client.StorageV1().StorageClasses().Create(context.TODO(), sc, metav1.CreateOptions{}) framework.ExpectNoError(err) } </ins>

There are use cases where users want to expand the local volume (may be managed by LVM or something similar), but even though they can edit underlying PV (and expand the underlying device) - the reported size in pvc.Status keeps reporting older value because in-tree local volumes are non-expandable. We could make this easier for users by - implementing interface for local volumes. This would allow following steps for local volume expansion: /admin edits the pvc - requesting new size. expands the underlying device. edits the PV and sets new size. will expand the filesystem and update This would at least get the user from impossible territory to possible. cc /sig storage
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hi & any news on this issue ? The related PR seems stalled.
merged. So this is fixed now.
Oh right ! I didn't looked close enough and thought it was only for directory based volumes. Thank you.


Question: apiVersion: apps/v1 kind: Deployment metadata: <del> name: event-exporter-v0.3.1 </del> <ins> name: event-exporter-v0.3.4 </ins> namespace: kube-system labels: k8s-app: event-exporter <del> version: v0.3.1 </del> <ins> version: v0.3.4 </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec:

the following images are using old client-go that would not work correctly starting 1.22+ if those images use in-cluster config(mounted service account token) to talk to kube-apiserver: if the above images use in-cluster config, they should use client-go 1.14+. xref: /assign /cc /sig instrumentation /priority important-soon
/triage accepted


Question: SecurityContext:               &v1.PodSecurityContext{}, TerminationGracePeriodSeconds: &period, SchedulerName:                 api.DefaultSchedulerName, <del> EnableServiceLinks:            &enableServiceLinks, </del> }, } tests := []struct {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug The change in a new field () to the Pod spec that is set to true by default. This is causing problems for DaemonSets in some situations. DaemonSets are using ControllerRevisions to keep track of previous configurations and each revision will contain among other things the Pod spec for the DaemonSet. If a DaemonSet is created with the enableServiceLinks property is set to true, the ControllerRevision will also contain this field. If the cluster is rolled back to a previous version of kubernetes, the DeamonSet struct does not have this field and so the ControllerRevision and the DaemonSet will no longer match. This leads to a new ControllerRevision being created and a rolling update of all pods in the DaemonSet. Would it be a better solution to rename the field to and enable service links whenever the field is not set? This would fix the problem with the DaemonSets unless the field is explicitly set before the rollback. This is causing the master upgrade tests to fail as reported in /sig apps /cc /cc /cc
I guess the change in podSpec and the defaulting will break all the workload resources as they are based on comparing the template's podSpec. ~Is the new value defaulted to true also for the DaemonSet's podSpec template? If not, controller might go hot loop like with mutating admission controllers.~ I don't like us making such changes to v1 podSpec. The other way around seems more sensible. /cc
Why is this a bad thing? We will continue to add fields to podSpec - the issue isn't the defaulting (this would happen regardless of which way it defaulted), it's that the controller can't really be version aware. The new value would default to true in the daemonset as well.
I'm not sure that a rollback requiring daemonsets to recreate is a bug. I think that's potentially "the best possible thing that we can do in that case".
/cc SIG Release blocking dashboard with upgrade test failures - https://k8s-
So the problem isn't the fact that a field has been . It the way in which the field was .


Question: SecurityContext:               &v1.PodSecurityContext{}, TerminationGracePeriodSeconds: &period, SchedulerName:                 api.DefaultSchedulerName, <del> EnableServiceLinks:            &enableServiceLinks, </del> }, } tests := []struct {

enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either ca So the problem isn't the fact that a field has been . It the way in which the field was . enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either case), you won't trigger the rolling restart on rollback or controller manager failoverse), you won't trigger the rolling restart on rollback or controller manager failover unless you explicitly opt out of the behavior. I don't see why the former behavior is more desirable than the latter.
For background/context, the choice between enable/disable was initially discussed here:
Are you saying that the issue is that default value is not the same as the Go "zero value" ? The concept of zero value is interesting but it forces APIs to bend into unnatural shapes to respect it. Since forever Kubernetes has explicitly said that we have version-specific default values which ARE NOT governed by zero values, but instead are governed by the API version. It is safe to assume that if a field is documented as having a default in a version, you can rely on that field being populated with that default, with a few exceptions.


Question: SecurityContext:               &v1.PodSecurityContext{}, TerminationGracePeriodSeconds: &period, SchedulerName:                 api.DefaultSchedulerName, <del> EnableServiceLinks:            &enableServiceLinks, </del> }, } tests := []struct {

We generally ask for boolean fields to be "in the affirmative direction", as in this case. Now, we could revisit this. It would be a significant change in direction, but it's not impossible. But I don't think that solves your problem in general -- if this field were optional string, with a default value of "Hello there", you would be in the same boat right?
So the underlying issue is that we are defaulting into the PodSpec which gets adopted into all of the PodTemplateSpecs of every object. That sort of defaults the users declared intent. Why not just default the value into the Pod itself . Is the intention really to default this in all Templates or to make it optional in the Template and default it in the Pod itself?
/milestone v1.13
I can't speak to the ideology of whether to default the field in all PodSpecs or just the Pod, but your suggestion makes sense to me. I'm not sure of the past reasoning for adding other defaulted fields, most recently SchedulerName in Jan. 2017 (), to all PodSpecs instead of just to the Pod object. Did we not have these same tests in place then? This would effectively be saying we should never add new defaulted fields to PodSpec. If this is the plan, I can create a PR for the fix and I'll add a big warning comment to SetDefaults_PodSpec.
As an example of where we don't default the template we have a CronJob that embeds JobTemplate, we never default that template, only an actual Job gets defaulting. I agree with that not defaulting any templates does make sense.
Agree to only default this in pods. Agree to adding the comment. Most important defaulting happens in pods to keep higher level pod specs simpler. We may need to raise the general pattern in a discussion with api review in general, since this is a change from previous strategy, but the previous strategy was not explicitly chosen (to my recollection at least)
Hi all do we have an agreed path forward for fixing this issue in 1.13? if so can we have an ETA for the fix to land? the release team would like for the tests to get passing in the blocking suites in an effort to unmask any other failures that might show up once these are fixed.


Question: SecurityContext:               &v1.PodSecurityContext{}, TerminationGracePeriodSeconds: &period, SchedulerName:                 api.DefaultSchedulerName, <del> EnableServiceLinks:            &enableServiceLinks, </del> }, } tests := []struct {

/assign
We have a PR with a fix that will hopefully merge today or tomorrow. I will keep an eye on the dashboard to make sure it fixes the failing daemonset tests.
Thanks Can we link the PR here plz. /cc
Reopening, because of the auto-close. Tests are not passing yet. /reopen
Reopening this issue. <detailsIn response to : Reopening, because of the auto-close. Tests are not passing yet. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
PR that should (hopefully) close this issue is here:
DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- if it doesn't pop up again - /close
Closing this issue. <detailsIn response to : DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "fmt" "math/rand" <ins> "net/http" "net/http/httptest" "testing" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/testapi" "github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/master" "github.com/GoogleCloudPlatform/kubernetes/plugin/pkg/admission/admit" </ins> "github.com/coreos/go-etcd/etcd" "github.com/golang/glog"


Question: var err error if backwardCompatibleInterface, ok := s.ResourceEncodingConfig.(CompatibilityResourceEncodingConfig); ok { <del> codecConfig.StorageVersion, err = backwardCompatibleInterface.BackwardCompatibileStorageEncodingFor(groupResource, example) </del> <ins> codecConfig.StorageVersion, err = backwardCompatibleInterface.BackwardCompatibileStorageEncodingFor(chosenStorageResource, example) </ins> if err != nil { return nil, err }

I have an extension apiserver, and I was trying to upgrading the dependency . In our code, we have a serval resources has its , and we expect the storage version is their . for example, we have a resource in both and group, and they are wired up by method, and expected encoded as to storage. However, this seems broken after upgrade to v1.31, so I did some invertgation by myself, and I find the storage version is changed by here: In old behavior honor , but now don't. storage version semantic of would be honored. there is not easy approach, need to create an extension api server and find out. No response <detailsClient Version: v1.30.1 Kustomize Version: v5.0.4-0.- Server Version: v1.31.0-beta.0.25+-dirty </details<detailsNONE </details<detailsKind Cluster run on Darwin Darwin ReficuldeMacBook- 23.0.0 Darwin Kernel Version 23.0.0: Fri Sep 15 14:41:34 PDT 2023; root:xnu-.1.13~1/RELEASEARM64_T8103 arm64 </details<detailskind </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
PTAL cc


Question: {CPU: 2000}, }, }, <ins> Host: "machine1", </ins> } cpuAndMemory := api.PodState{ Manifest: api.ContainerManifest{

When creating replication controller (e.g. frontend php-redis from guestbook example), scheduler gets stuck and never recovers. This is a race, I have to run several times to see the error (A script to delete controller and pods, then create new controller, leave enough time in between for k8s to react). I've created PR for an attempted fix.
I saw this too while following the guest book and creating the php/frontend step. In my case, I had 2 minions while the replication controller configuration had configured 3 replicas, so 2 pods got scheduled -each on another minion, the third pod had empty Host (saw that when doing 'list pods'), and the 3 of them were stuck in "waiting" status. I deleted the controller and the pods, fixed the configuration in json file to be 2 replicas and recreated the controller and everything worked. Does your fix take care of this use case? (too many replicas, too few minions/one of the pods is not scheduled)
This is different from your use case. If you have more replicas than minions, then the scheduler will get stuck for sure (with the error msg probably being 'failed to find a fit for pod'). AFAICT, it's not support yet.
If you remove the HostPorts, it should work for you. thanks for report.
, thanks for your replies. I understand why it isn't working, but I'd expect one of the 2 options to happen instead of the current situation: the creation of replication controller if there are less minions than replicas, AND there is a hostPort property. In other words - validation of the content during creation. least schedule all replicas that can get a minion (in my case 2 out of 3) and leave only one at "waiting". Is there an issue open about the use case I describe?
should be true today. is less possible than it seems like-- minions can come and go, and become over or under loaded in the time between creation and when the pod shows up at the scheduler. So even if we check up front, we still have to check again later, and sometimes the first check will pass but not the second.


Question: printFlags := JSONPathPrintFlags{ TemplateArgument: templateArg, } <ins> if !sort.StringsAreSorted(printFlags.AllowedFormats()) { t.Fatalf("allowed formats are not sorted") } </ins> p, err := printFlags.ToPrinter(tc.outputFormat) if tc.expectNoMatch {

/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, } fs := cmd.Flags() <del> namedFlagSets := opts.Flags() </del> verflag.AddFlags(namedFlagSets.FlagSet("global")) globalflag.AddGlobalFlags(namedFlagSets.FlagSet("global"), cmd.Name()) for _, f := range namedFlagSets.FlagSets {

ci-kubernetes-e2e-gci-gce-alpha-features kubetest.Up Jul 02 2021, 01:26:42 UTC+0200 /sig release
based on testgrid probably cc <img width="1203" alt="Screen Shot 2021-07-02 at 4 24 25 PM" src="https://user-"
/cc working on it
/assign
the scheduler fails to start with this error: it seems the feature gates from the command line does not take effect in the defaulting phrase, do you know how to fix it?
ah, found there is a cycle here: The scheduler options , but the default config's default values depend on the feature gates options. I think this disables other feature gate controlled logic in the defaulting too:
I tried to fix it by updating the component config after the CLI flags (including the feature gates) are parsed: PTAL
fixed the flake. is back green now.
thanks!


Question: <ins> amd64=alpine:3.6 arm=arm32v6/alpine:3.6 arm64=arm64v8/alpine:3.6 ppc64le=ppc64le/alpine:3.6 </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature This issue is for creating image as multiarch
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing


Question: } // podRequest returns the total resource request of a pod which is the <del> // max(sum of init container requests, sum of container requests) </del> <ins> // max(max of init container requests, sum of container requests) </ins> func podRequest(pod *v1.Pod, resourceName v1.ResourceName) resource.Quantity { containerValue := resource.Quantity{Format: resource.BinarySI} if resourceName == resourceDisk && !utilfeature.DefaultFeatureGate.Enabled(features.LocalStorageCapacityIsolation) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Kubelet constantly crashes trying to evict pods under memory pressure What you expected to happen: kubelet to be able to evict pods How to reproduce it (as minimally and precisely as possible): Fresh 1.9.1 Kubernetes, installed via kubeadm Here are relevant eviction parameters for kubelet Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"", GitTreeState:"archive", BuildDate:"2018-01-05T20:28:28Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-01-04T11:40:06Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: 8GiB of RAM OS (e.g. from /etc/os-release): Gentoo Kernel (e.g. ): Linux 4.9.34-gentoo SMP Sat Jan 13 23:06:16 MSK 2018 x86_64 Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz GenuineIntel GNU/Linux Install tools: Others:
contains the following code I think there is a bug in that it is iterating through InitContainers but getting resources of regular containers. I have a pod that has 3 initContainers. Looks like this could be a copy&paste error.
/sig node
indeed there's a mismatch between initContainers and spec containers, I'll take a look.
looks like we need to do the following and the problem was introduced in cc
- Indeed, I'll open a PR for this
reopening until it is cherrypicked to 1.9
It was cherry-picked in 1.9.3.
/close


Question: if ext != "" { name = name[:len(name)-len(ext)] } <del> if ext != ".json" { </del> <ins> if !(ext == ".json" || ext == ".yaml") { </ins> return nil } glog.Infof("Testing %s", path)

Which jobs are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /priority important-soon /cc /cc /milestone v1.18 /sig release
/assign
This is failing because a 1.18 release for CRI tools doesn't exist yet. I can tweak the test job arguments to skip this later.
(As a note, this job is important, but not ) /remove-priority critical-urgent /priority important-soon
Do we need to remove the job from dashboard?
-- No. I'll work on fixing the job later today.
I've started to investigate, but I haven't yet reproduced the new error locally: This is likely related to a recent refactor of the package that handles interpreting deb/rpm package versions. As is not currently in use, I'm moving this to the milestone. /milestone v1.19
the version issue is fixed via The test is now failing because it tries to download cri-tools v1.18.0: which I think is related to this code part:
Updating testgrid link here: Any updates here /retitle [Flaky Test] build-packages-debs (ci-release-build-packages-debs) /kind flaky-test
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Updating testgrid link here: /kind flaky-test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake
I was assuming that is working on this issue. I can jump in if you want and do not find the time to work on it. :)
/assign /unassign Sascha -- Can you take this? I thought I was going to have time to look at this, but it's been a while.
Sure, the reason for the failing test seems to have changed, I'm on it.
Fix is incoming in


Question: `ReadonlyRootfs:` + fmt.Sprintf("%v", this.ReadonlyRootfs) + `,`, `SupplementalGroups:` + fmt.Sprintf("%v", this.SupplementalGroups) + `,`, `ApparmorProfile:` + fmt.Sprintf("%v", this.ApparmorProfile) + `,`, <ins> `SeccompProfilePath:` + fmt.Sprintf("%v", this.SeccompProfilePath) + `,`, </ins> `}`, }, "") return s

CRI supports AppArmor, Seccomp and Sysctls via annotations , we should move them to typed and structured fields instead. (refer ) [x] Apparmor () [x] Sysctls () [x] Seccomp () cc
/cc
/sig node
Seccomp is addressed at .


Question: `ReadonlyRootfs:` + fmt.Sprintf("%v", this.ReadonlyRootfs) + `,`, `SupplementalGroups:` + fmt.Sprintf("%v", this.SupplementalGroups) + `,`, `ApparmorProfile:` + fmt.Sprintf("%v", this.ApparmorProfile) + `,`, <ins> `SeccompProfilePath:` + fmt.Sprintf("%v", this.SeccompProfilePath) + `,`, </ins> `}`, }, "") return s

Seccomp profile is passed to runtime by in CRI: is relative to node's local seccomp profile root, it is defined in kubelet. But runtime doesn't know it. We should pass full profile path in CRI. cc/ Will file a PR to fix this later.
Add support for Seccomp sounds like a new feature. The milestone should be reset to , correct?
not really. The current implementation supports seccomp already. If it's missing critical information to function properly, this would be a bug.
are we blocking releases for CRI in 1.5? i was not sure, so marking this as non-release-blocker and leaving 1.5 milestone. please correct if i am mistaken.
I don't get why this is a 1.5 targetted thing myself. Yes we want CRI to be in a good state in 1.5, but afaik we have zero intention of anyone actually using (at the very least because of the lack of exec auth regressing kubelet security). why should this be part of 1.5?
First, this is a small enough change that allows non-docker integrations to properly support seccomp. Second, if "we have zero intention of anyone actually using" is a valid reason, we should never fix any bugs in CRI. We do want people to try using it (perhaps not in production yet). However, given that seccomp is only in alpha, I am ok with dropping this and just continue the docker support for now.
Perhaps I misunderstood the meaning of code freeze, but I thought it was only bug-fixes regardless of size of change (or exceptions that have gone through some process) What I meant is we don't intend this to be used generally as part of the 1.5 release. Because of that, I don't think we should consider bugs in CRI bugs for the purpose of the 1.5 release freeze. My statement is just that we shouldn't violate code-freeze for this. We should totally fix this right after the 1.5 freeze window ends, and saying that I'm arguing we should never fix anything is disingenuous and taking it to an absurd extreme.
the only thing we don't seem to agree on is whether seccomp support was considered to be supported for CRI in 1.5.


Question: `ReadonlyRootfs:` + fmt.Sprintf("%v", this.ReadonlyRootfs) + `,`, `SupplementalGroups:` + fmt.Sprintf("%v", this.SupplementalGroups) + `,`, `ApparmorProfile:` + fmt.Sprintf("%v", this.ApparmorProfile) + `,`, <ins> `SeccompProfilePath:` + fmt.Sprintf("%v", this.SeccompProfilePath) + `,`, </ins> `}`, }, "") return s

My interpretation is that we assumed seccomp would be supported until we found out that it wouldn't for other runtimes due to the lack of profile path. That's the only reason why I considered this a bug. Of course we could say that this is not really an important feature in CRI, so we can drop it, but we can be more clear on that.
If there's a bug in a feature no one uses, does fixing it make a sound? I think what we're not agreeing on is the role of CRI in 1.5. I understood it as something that is not generally usable. We can't recommend a customer use it, even as an experimental feature, due to security aspects (if nothing else). Correct me if I'm wrong. It's a developer integration point, and developers integrating with it are doing so with the kubernetes master branch, not with some release. Those reasons together result in my opinion that CRI is still not really a customer/release feature, and thus bugs or features related to it should not be exceptions to the freeze window, but rather should targeted at master as soon as the freeze lifts. If this were already post-freeze, would we be cherry-picking this over to 1.5?
This is not true. cri-o is depending on this fix for apparmor support.
is it depending on the 1.5 release branch directly, or on master?
I think it's better to make this in 1.5, so cri-o could work with first release with experimental CRI. It helps to encourage people to use CRI and find features misses or bugs.
Modifying the user-specified annotation is a little bit unsettling. I think adding a new annotation explicitly for the path would be better. Or, we should just add an explicit field for this. I spoke to and separately. I still think this is a bug fix, but we may want to be more careful about the fix. Given the time constraint, let's just list this as a known issue and work on it for 1.6.
OK, let's move this to next release and think carefully about seccomp feature besides this: kubelet may just pass the profile as string via CRI; may assume the profile locate in a file in a known format and pass path to the runtime; or maybe some other solutions.


Question: `ReadonlyRootfs:` + fmt.Sprintf("%v", this.ReadonlyRootfs) + `,`, `SupplementalGroups:` + fmt.Sprintf("%v", this.SupplementalGroups) + `,`, `ApparmorProfile:` + fmt.Sprintf("%v", this.ApparmorProfile) + `,`, <ins> `SeccompProfilePath:` + fmt.Sprintf("%v", this.SeccompProfilePath) + `,`, </ins> `}`, }, "") return s

We need to clearly define it. define profile format clearly
Please document known issues for 1.5 in
(copied from ) And let's move here to discuss how to pass profiles to runtime. Related with .
This is issue is related with two level APIs: CRI, since we may also want to support profiles from ConfigMaps, I think we should pass full profile contents instead of path to runtime. issues is also related with seccomp api in kube-apiserver, it is based on annotations today. We should define seccomp profile format clearly (see ) and decide whether to add struct types to pod spec or just continue the annotations way. I think it's more clear to add specific struct types to pod spec.
Agreed - the path stuff is really a crutch (and implementation-specific) - I think is the answer here.
I don't disagree, but it's not strictly a requirement for supporting ConfigMaps (kubelet could create a file with the contents). An open question is whether to pass it as a raw string, or wait for and pass the structured data. The annotations way is deprecated (). I think the bigger question is where/how to put the data. E.g. structured data in the API, unstructured (string type) data in the API, reference to another object (e.g. ConfigMap). If the Kubelet needs to parse the spec, I think that's a good argument for a structured data type (either a standalone object or embedded in the pod API).
Yes, already opened to replace annotations with typed structs. I personally prefer a structured data type in both kubernetes API and CRI. WDYT?
A counter argument is that as the spec grows or new features are , we need to keep updating the Kubernetes API to maintain parity, or update the code in each CRI implementation to translate the Kubernetes spec. OTOH, if we use unstructured data we can use 3rd party libraries to parse out the data we need (assuming the spec evolves in a backwards compatible way).
What spec?
Whichever seccomp profile spec is being implemented by runtimes. E.g. if we decide to base the Kubernetes seccomp profile API off the OCI seccomp profile spec, and OCI adds some new features, at some point we'd probably want to add those features to Kubernetes.


Question: `ReadonlyRootfs:` + fmt.Sprintf("%v", this.ReadonlyRootfs) + `,`, `SupplementalGroups:` + fmt.Sprintf("%v", this.SupplementalGroups) + `,`, `ApparmorProfile:` + fmt.Sprintf("%v", this.ApparmorProfile) + `,`, <ins> `SeccompProfilePath:` + fmt.Sprintf("%v", this.SeccompProfilePath) + `,`, </ins> `}`, }, "") return s

It also means that if different runtimes support different feature sets, we'll be stuck either tracking only the common features, or specifying features that may be ignored by the runtime. This may be a non-issue if the seccomp feature set rarely changes.
Well, yes, but that applies to everything in the CRI, and is one of the tradeoffs we ma[d|k]e by introducing this abstraction in the first place.
My $0.02: Moving the existing logic/information for seccomp into a first-class field for CRI (as in ) is productive because of the known problems with the annotation approach. My initial proposal for seccomp support included a resource similar to what has proposed in kubernetes/community and we were unable to come to an agreement on the issue refers to here: I do not necessarily expect that we will be able to come to an agreement about the seccomp resource in time for 1.7, so I think we should proceed with and do backward compatible CRI changes later if we need to inject the content of the seccomp profile instead of the path. I think we would deeply regret embedding API surface to describe seccomp profiles in the Pod API :)


Question: By("Creating a directory under discovery path") framework.Logf("creating local volume under path %q", volumePath) mkdirCmd := fmt.Sprintf("mkdir %v -m 777", volumePath) <del> _, err := framework.NodeExec(node0.Name, mkdirCmd) </del> <ins> err := framework.IssueSSHCommand(mkdirCmd, framework.TestContext.Provider, node0) </ins> Expect(err).NotTo(HaveOccurred()) <ins> By("Waiting for a PersitentVolume to be created") </ins> oldPV, err := waitForLocalPersistentVolume(config.client, volumePath) Expect(err).NotTo(HaveOccurred())

8 tests failed, 2 jobs failed, 88 builds failed. Failure stats cover 1 day time range '3 Aug 2017 03:31 UTC' to '4 Aug 2017 03:31 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
This should have been fixed by /cc
The test panic is fixed, but now it's failing because it can't SSH into the node.
/assign


Question: pair{FROM, FROM}: true, } <del> pod := func(name, check string) *api.Pod { return &api.Pod{ </del> <ins> pod := func(name, check string, final bool) *api.Pod { p := &api.Pod{ </ins> ObjectMeta: api.ObjectMeta{ Name:   name, Labels: map[string]string{"check": check}, }, } <ins> if final { p.Labels["final"] = "true" } return p } deletePod := func(p *api.Pod) bool { return p.Labels["final"] == "true" </ins> } tests := []func(string){ func(name string) { name = "a-" + name <del> source.Add(pod(name, FROM)) source.Modify(pod(name, TO)) </del> <ins> source.Add(pod(name, FROM, false)) source.Modify(pod(name, TO, true)) </ins> }, func(name string) { name = "b-" + name <del> source.Add(pod(name, FROM)) source.ModifyDropWatch(pod(name, TO)) </del> <ins> source.Add(pod(name, FROM, false)) source.ModifyDropWatch(pod(name, TO, true)) </ins> }, func(name string) { name = "c-" + name <del> source.AddDropWatch(pod(name, FROM)) source.Modify(pod(name, ADD_MISSED)) source.Modify(pod(name, TO)) </del> <ins> source.AddDropWatch(pod(name, FROM, false)) source.Modify(pod(name, ADD_MISSED, false)) source.Modify(pod(name, TO, true)) </ins> }, func(name string) { name = "d-" + name <del> source.Add(pod(name, FROM)) </del> <ins> source.Add(pod(name, FROM, true)) </ins> }, }

I got this by running the following against commit : There have been previous attempts to fix this: It appears it's still an issue :cry: We're also seeing it in OpenShift (), although our code hasn't pulled in the fix from quite yet.
cc
What go version are you using? Previously this test has been catching legit bugs, so I'm not sure about the "flaky" moniker.
go 1.5.1. I noticed you had a comment in another issue or PR about this possibly being a bug in Go itself (or did I misread)?
I doubt it's a bug in go. (go1.5 had an unrelated bug, maybe that's what you're thinking of.) We haven't actually flipped 1.5 on so I may sit on this for a bit.
I instrumented this code, and this is being caused by duplicate deletions causing the wait group to decrement below 0. Are duplicate deletions a bug? Or should the wait group be guarded against that?
ping If duplicate deletes are not a bug, I can put together a quick PR to guard against that. If they are a bug, I could try to track it down sometime this week (even thought I'm not that familiar with this code) if no one else is going to look into it.
Duplicate deletes are a bug. (assuming that the test harness isn't actually issuing multiple deletes for the same item.)
That would be great if you could take this, since I won't have time.
So it would be an error in the test if was called multiple times for the same pod? Or should the framework be de-duping that?
IMO, that's a test error. ...If the test is doing that because the framework dup'd an update or an add, then it's a framework error. (adds and updates shouldn't be duped, either) , krousey wrote:
from what I gather, this test is just trying to delete after each pod reaches it's final intended state. This code is deleting on every event, including intermediate ones. In particular, this test case is occasionally causing duplicate deletes to be issued: Sometimes it catches the transition from FROM to ADDMISSED, and then gets ADDMISSED to TO. I think the fix is to tell the test to only issue the delete when the final state is TO.


Question: pair{FROM, FROM}: true, } <del> pod := func(name, check string) *api.Pod { return &api.Pod{ </del> <ins> pod := func(name, check string, final bool) *api.Pod { p := &api.Pod{ </ins> ObjectMeta: api.ObjectMeta{ Name:   name, Labels: map[string]string{"check": check}, }, } <ins> if final { p.Labels["final"] = "true" } return p } deletePod := func(p *api.Pod) bool { return p.Labels["final"] == "true" </ins> } tests := []func(string){ func(name string) { name = "a-" + name <del> source.Add(pod(name, FROM)) source.Modify(pod(name, TO)) </del> <ins> source.Add(pod(name, FROM, false)) source.Modify(pod(name, TO, true)) </ins> }, func(name string) { name = "b-" + name <del> source.Add(pod(name, FROM)) source.ModifyDropWatch(pod(name, TO)) </del> <ins> source.Add(pod(name, FROM, false)) source.ModifyDropWatch(pod(name, TO, true)) </ins> }, func(name string) { name = "c-" + name <del> source.AddDropWatch(pod(name, FROM)) source.Modify(pod(name, ADD_MISSED)) source.Modify(pod(name, TO)) </del> <ins> source.AddDropWatch(pod(name, FROM, false)) source.Modify(pod(name, ADD_MISSED, false)) source.Modify(pod(name, TO, true)) </ins> }, func(name string) { name = "d-" + name <del> source.Add(pod(name, FROM)) </del> <ins> source.Add(pod(name, FROM, true)) </ins> }, }

I'll do that and some stress testing to see if it goes away.
3589 runs so far, 10 failures and all 10 failures seem to confirm the above behavior.
Guarding against this: runs so far, 0 failures I will clean this up and send out a fix shortly.
Awesome, thanks! , krousey wrote:
Accidentally left the test case running. Good news:
It's probably safe to say that if there's remaining corner cases, your setup is not going to exercise them! :) , krousey wrote: