Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When trying to mount an Azure File mount it fails, it looks as though its putting a new line between the name of the Azure Storage Account and the Azure storage domain. There are no new line characters in the secret that holds the account name. The following is the secret: The following is the declaration within the section of the Kubernetes yaml: The following is the error that's shown in `kubectl describe po/pod_name: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS Kernel (e.g. ): 4.13.0-1011-azure Install tools: Others:
/sig storage
/sig azure
It seems to me that does decode to , so it seems to be behaving correctly. Maybe it's a good idea to trim the return char for and fields after ?
is specified by user, so trim the is not a good idea in the code. Another simple way is to use kubectl create secret to create azure-secret, the accountname and key won't require base64 encoded, so you won't mess up with let me know if you have any question, thanks.
Thanks !
Bumped into this issue today too on our cluster. Can confirm command worked. Is there an easy way to time the /n off the end of the echo for before using base64? Thanks.
yes it's
Of course it is! Doh!
i think would've made better sense to trim the azurestorageaccountname value when receiving it... anyways the storage account name wont have n in them right? I think that just made better developer experience...
is not allowed in azure storage account name
so if azure storage account name does not allow in the name.. then why not just trim the name instead of error... Because devs would just and that would already have the n in the resulting base64-encoded string... right?
sounds reasonable /reopen
Reopened this issue. <detailsIn response to : sounds reasonable /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
awesome. Thanks
so then this goes in to kube version 1.14.6 in AKS ? or how does this work?
It would only go into k8s v1.16.0 since it's not a critical bug, we don't want to do hot fix to the old release.
got it. thanks Andy.

op.result = result finished := time.Now() op.finished = &finished <del> close(op.notify) </del> <ins> op.notify <- true </ins> } // WaitFor waits for the specified duration, or until the operation finishes,


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When trying to mount an Azure File mount it fails, it looks as though its putting a new line between the name of the Azure Storage Account and the Azure storage domain. There are no new line characters in the secret that holds the account name. The following is the secret: The following is the declaration within the section of the Kubernetes yaml: The following is the error that's shown in `kubectl describe po/pod_name: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS Kernel (e.g. ): 4.13.0-1011-azure Install tools: Others:
/sig storage
/sig azure
It seems to me that does decode to , so it seems to be behaving correctly. Maybe it's a good idea to trim the return char for and fields after ?
is specified by user, so trim the is not a good idea in the code. Another simple way is to use kubectl create secret to create azure-secret, the accountname and key won't require base64 encoded, so you won't mess up with let me know if you have any question, thanks.
Thanks !
Bumped into this issue today too on our cluster. Can confirm command worked. Is there an easy way to time the /n off the end of the echo for before using base64? Thanks.
yes it's
Of course it is! Doh!
i think would've made better sense to trim the azurestorageaccountname value when receiving it... anyways the storage account name wont have n in them right? I think that just made better developer experience...
is not allowed in azure storage account name
so if azure storage account name does not allow in the name.. then why not just trim the name instead of error... Because devs would just and that would already have the n in the resulting base64-encoded string... right?
sounds reasonable /reopen
Reopened this issue. <detailsIn response to : sounds reasonable /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
awesome. Thanks
so then this goes in to kube version 1.14.6 in AKS ? or how does this work?
It would only go into k8s v1.16.0 since it's not a critical bug, we don't want to do hot fix to the old release.
got it. thanks Andy.

select { case <-time.After(timeout): case <-op.notify: <ins> // Re-send on this channel in case there are others // waiting for notification. op.notify <- true </ins> } }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When trying to mount an Azure File mount it fails, it looks as though its putting a new line between the name of the Azure Storage Account and the Azure storage domain. There are no new line characters in the secret that holds the account name. The following is the secret: The following is the declaration within the section of the Kubernetes yaml: The following is the error that's shown in `kubectl describe po/pod_name: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS Kernel (e.g. ): 4.13.0-1011-azure Install tools: Others:
/sig storage
/sig azure
It seems to me that does decode to , so it seems to be behaving correctly. Maybe it's a good idea to trim the return char for and fields after ?
is specified by user, so trim the is not a good idea in the code. Another simple way is to use kubectl create secret to create azure-secret, the accountname and key won't require base64 encoded, so you won't mess up with let me know if you have any question, thanks.
Thanks !
Bumped into this issue today too on our cluster. Can confirm command worked. Is there an easy way to time the /n off the end of the echo for before using base64? Thanks.
yes it's
Of course it is! Doh!
i think would've made better sense to trim the azurestorageaccountname value when receiving it... anyways the storage account name wont have n in them right? I think that just made better developer experience...
is not allowed in azure storage account name
so if azure storage account name does not allow in the name.. then why not just trim the name instead of error... Because devs would just and that would already have the n in the resulting base64-encoded string... right?
sounds reasonable /reopen
Reopened this issue. <detailsIn response to : sounds reasonable /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
awesome. Thanks
so then this goes in to kube version 1.14.6 in AKS ? or how does this work?
It would only go into k8s v1.16.0 since it's not a critical bug, we don't want to do hot fix to the old release.
got it. thanks Andy.

if accountName == "" || accountKey == "" { return "", "", fmt.Errorf("Invalid %v/%v, couldn't extract azurestorageaccountname or azurestorageaccountkey", nameSpace, secretName) } <ins> accountName = strings.TrimSpace(accountName) </ins> return accountName, accountKey, nil }


Question:
the said: However the are different, it does not allow to set spec and metadata, but other changes are not ignored. And updates to other parts are not validated.
/cc for some suggestions.
/assign
Made a pull request:
Is the documentation wrong or the implementation? I think there was a longer discussion about the correct behaviour in the PR, but haven't found it yet.
It strange that other parts rather than metadata and spec can be modified and they are not validated. So I think the implementation is wrong, at least.
We should follow the design doc, I think.
/sig api-machinery
/priority important-soon
I agree the current behaviour is wrong. Added some comments to Especially, worries me whether we really have to right /status behaviour. Not really related to the given issue. But as we are touching the /status behaviour, we should clarify that as well.
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 3 days, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details

"//vendor/k8s.io/client-go/discovery:go_default_library", ], ) <ins> go_test( name = "go_default_test", srcs = ["status_strategy_test.go"], embed = [":go_default_library"], deps = ["//vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured:go_default_library"], ) </ins>


Question:
the said: However the are different, it does not allow to set spec and metadata, but other changes are not ignored. And updates to other parts are not validated.
/cc for some suggestions.
/assign
Made a pull request:
Is the documentation wrong or the implementation? I think there was a longer discussion about the correct behaviour in the PR, but haven't found it yet.
It strange that other parts rather than metadata and spec can be modified and they are not validated. So I think the implementation is wrong, at least.
We should follow the design doc, I think.
/sig api-machinery
/priority important-soon
I agree the current behaviour is wrong. Added some comments to Especially, worries me whether we really have to right /status behaviour. Not really related to the given issue. But as we are touching the /status behaviour, we should clarify that as well.
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 3 days, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details

} func (a statusStrategy) PrepareForUpdate(ctx context.Context, obj, old runtime.Object) { <ins> // update is only allowed to set status </ins> newCustomResourceObject := obj.(*unstructured.Unstructured) <del> oldCustomResourceObject := old.(*unstructured.Unstructured) </del> newCustomResource := newCustomResourceObject.UnstructuredContent() <del> oldCustomResource := oldCustomResourceObject.UnstructuredContent() // update is not allowed to set spec and metadata _, ok1 := newCustomResource["spec"] _, ok2 := oldCustomResource["spec"] switch { case ok2: newCustomResource["spec"] = oldCustomResource["spec"] case ok1: delete(newCustomResource, "spec") } </del> <ins> status, ok := newCustomResource["status"] </ins> <del> newCustomResourceObject.SetAnnotations(oldCustomResourceObject.GetAnnotations()) newCustomResourceObject.SetFinalizers(oldCustomResourceObject.GetFinalizers()) newCustomResourceObject.SetGeneration(oldCustomResourceObject.GetGeneration()) newCustomResourceObject.SetLabels(oldCustomResourceObject.GetLabels()) newCustomResourceObject.SetOwnerReferences(oldCustomResourceObject.GetOwnerReferences()) newCustomResourceObject.SetSelfLink(oldCustomResourceObject.GetSelfLink()) </del> <ins> // copy old object into new object oldCustomResourceObject := old.(*unstructured.Unstructured) // overridding the resourceVersion in metadata is safe here, we have already checked that // new object and old object have the same resourceVersion. *newCustomResourceObject = *oldCustomResourceObject.DeepCopy() // set status newCustomResource = newCustomResourceObject.UnstructuredContent() if ok { newCustomResource["status"] = status } else { delete(newCustomResource, "status") } </ins> } // ValidateUpdate is the default update validation for an end user updating status.


Question:
the said: However the are different, it does not allow to set spec and metadata, but other changes are not ignored. And updates to other parts are not validated.
/cc for some suggestions.
/assign
Made a pull request:
Is the documentation wrong or the implementation? I think there was a longer discussion about the correct behaviour in the PR, but haven't found it yet.
It strange that other parts rather than metadata and spec can be modified and they are not validated. So I think the implementation is wrong, at least.
We should follow the design doc, I think.
/sig api-machinery
/priority important-soon
I agree the current behaviour is wrong. Added some comments to Especially, worries me whether we really have to right /status behaviour. Not really related to the given issue. But as we are touching the /status behaviour, we should clarify that as well.
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 3 days, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details

<ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package customresource import ( "context" "reflect" "testing" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" ) func TestPrepareForUpdate(t *testing.T) { strategy := statusStrategy{} tcs := []struct { old      *unstructured.Unstructured obj      *unstructured.Unstructured expected *unstructured.Unstructured }{ { // changes to spec are ignored old: &unstructured.Unstructured{ Object: map[string]interface{}{ "spec": "old", }, }, obj: &unstructured.Unstructured{ Object: map[string]interface{}{ "spec": "new", }, }, expected: &unstructured.Unstructured{ Object: map[string]interface{}{ "spec": "old", }, }, }, { // changes to other places are also ignored old: &unstructured.Unstructured{ Object: map[string]interface{}{ "spec": "old", }, }, obj: &unstructured.Unstructured{ Object: map[string]interface{}{ "new": "new", }, }, expected: &unstructured.Unstructured{ Object: map[string]interface{}{ "spec": "old", }, }, }, { // delete status old: &unstructured.Unstructured{ Object: map[string]interface{}{ "spec":   "old", "status": "old", }, }, obj: &unstructured.Unstructured{ Object: map[string]interface{}{ "spec": "old", }, }, expected: &unstructured.Unstructured{ Object: map[string]interface{}{ "spec": "old", }, }, }, { // update status old: &unstructured.Unstructured{ Object: map[string]interface{}{ "spec":   "old", "status": "old", }, }, obj: &unstructured.Unstructured{ Object: map[string]interface{}{ "status": "new", }, }, expected: &unstructured.Unstructured{ Object: map[string]interface{}{ "spec":   "old", "status": "new", }, }, }, { // update status and other parts old: &unstructured.Unstructured{ Object: map[string]interface{}{ "spec":   "old", "status": "old", }, }, obj: &unstructured.Unstructured{ Object: map[string]interface{}{ "spec":   "new", "new":    "new", "status": "new", }, }, expected: &unstructured.Unstructured{ Object: map[string]interface{}{ "spec":   "old", "status": "new", }, }, }, } for index, tc := range tcs { strategy.PrepareForUpdate(context.TODO(), tc.obj, tc.old) if !reflect.DeepEqual(tc.obj, tc.expected) { t.Errorf("test %d failed: expected: %v, got %v", index, tc.expected, tc.obj) } } } </ins>


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: On Windows Server with sandbox support the CNI gets called for all the containers within the pod including the sandbox container itself. What you expected to happen: The CNI should be called only for the sandbox container since they share the same network details. How to reproduce it (as minimally and precisely as possible): kubelet Pod at the parameters that get passed when the CNI is called Anything else we need to know?: This applies to Windows Server with sandbox support. The CNI should be called only for the infra container. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig windows /sig network /kind bug /assign /cc
Reopen this because of
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
has been merged now. /close
Closing this issue. <detailsIn response to : has been merged now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
has this issue been fixed? Is the CNI getting called only for the infra container? From what I see, the CNI still gets called for every single container from within a pod:
See comments It is required for windows containers.
The way I understand it, it's that the DNS issue will be fixed in Windows RS5 and there will not be required additional calls. The workaround at the moment is to call the CNI multiple times until the DNS issue gets fixed in Windows. As far as I know, that is the only reason why the additional calls to the CNI are required.
Is this true? Do we still need to invoke cni for each container in RS5? And even with new cni dns capabilities ()?
the PR that you linked it's unrelated to this issue. The CNI should be called for the sandbox pod which is the infra container. All the other containers created within the same pod are sharing all network details, excepting DNS. This is an issue in 1709 and 1803 where not everything gets replicated properly to the other containers. As a workaround for this, you need to call AttachEndpoint from HCS on the newly created container network interface only to replicate the DNS as well. You can also see more details about this issue here:
Can you re-open the issue in the meanwhile since this issue still exists?
I still think this isn't a issue, but a requirement for windows containers. Even after RS5, we should probably support old windows versions. WDYH
Why would the spec of the CNI should be different on Windows? On Linux you know that the sandbox ID that will get passed to the CNI is always the sandbox container. On Windows the sandbox ID won't be only that, it will also be an ID of a container from within the same pod. Does this sound like normal behavior to you? For now I agree, the workaround is to call the CNI for every single container because of the Windows issue with DNS. But this doesn't mean we have to stick with this for future Windows releases which will not require calling the CNI ADD multiple times. Also, this is a hack for now () which needs to be fixed. Please re-open the issue until we get feedback from others as well, thanks.
OK. Reopened the issue. Let's wait a while for feedbacks from other folks.
Apologize for the delay. Windows doenst have the namespace support native on the platform. we this support as part of RS5 Platform release. We are working on making runtime spec changes and CRI/ContainerD changes to use this model. here is a runtime spec PR that got merged. This is orthogonal to the DNS issue. Once we are done with the CRI/ContainerD changes then if u use CRI/containerD as runtime and if your OS version is RS5 and above then you don't need this second CNI add call. But as Pengfei said, if you are on a downlevel version or using dockershim as Runtime then you will have to make the second CNI call. So in my opinion this code will stay as workaround forever until we remove the support for downlevel OS. Hope this helps. I would recommend closing this issue.
thanks for the confirmation. In my opinion this should stay open until it gets fixed at least for future Windows releases, because the way it is at the moment, the CNI will still be called multiple times regardless of the Windows version. Why would you recommend closing this issue? Shouldn't this be fixed for futures Windows releases? The other calls will be useless when Windows gets proper namespace support. Also, I recommend watching this presentation about the CNI which had place at the KubeCon from May I don't see why we shouldn't have this open until the issue gets fixed for future releases and we stick with the official SPEC of the CNI.
and I had talked before about end of life of downlevel OS like 1803 and 1709. I think we should keep this open or open a new Issue to track updating the code so that the CNI is not called multiple times
This code would exists as far as we support 1709, 1803. The new workflow to use the windows namespace is getting implemented in containerd/cri. We can move kubernetes to use this new flow only for RS5 and above, if we plan to do it here as well. There isn't a good way to switch between 2 workflows of cni callflow wrt the Windows version hence we are not planning to change here till 1709 & 1803 exists, without adding some ugly code.
thanks for the explanation
I totally agree that this has to be fixed for future release and we are actively working on it. I was recommending to close this issue and open a feature to track this request. We should be able to raise a PR in another 2 weeks to support native network namespace support. This code will stay as long as we support down-level OS'es. I am fine either way. I am trying to close any open issues that is not a bug on Windows Networking(Platform).
/kind feature Although from a kubernetes standpoint, this isn't a new feature. It doesn't need a new KEP and won't change any API. It will just be a future Windows network code cleanup.
/kind cleanup
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
- can you clarify a few things so we know what's required to fix this? 's the minimum OS version needed? Windows Server 2019, or a future release? this be possible to fix for dockershim, or just CRI-containerD? Thanks!
What's the minimum OS version needed? Windows Server 2019, or a future release? [Dinesh]: Minimum OS version needed is 2019 Will this be possible to fix for dockershim, or just CRI-containerD? [Dinesh]: yes its possible to fix for Dockershim as well. but we are starting it with ContainerD. ContainerD is already done and available for testing.
excellent. thank you Dinesh!
ping - you have logs coming in email for this. Can you help us determine if it's needed for 1.14 vs 1.15?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Ping
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Status on this?

var IP string // TODO: Remove this when sandbox is available on windows <ins> // Currently windows supports both sandbox and non-sandbox cases. </ins> // This is a workaround for windows, where sandbox is not in use, and pod IP is determined through containers belonging to the Pod. <del> if IP = ds.determinePodIPBySandboxID(podSandboxID); IP == "" { </del> <ins> if IP = ds.determinePodIPBySandboxID(podSandboxID, r); IP == "" { </ins> IP = ds.getIP(podSandboxID, r) }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: On Windows Server with sandbox support the CNI gets called for all the containers within the pod including the sandbox container itself. What you expected to happen: The CNI should be called only for the sandbox container since they share the same network details. How to reproduce it (as minimally and precisely as possible): kubelet Pod at the parameters that get passed when the CNI is called Anything else we need to know?: This applies to Windows Server with sandbox support. The CNI should be called only for the infra container. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig windows /sig network /kind bug /assign /cc
Reopen this because of
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
has been merged now. /close
Closing this issue. <detailsIn response to : has been merged now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
has this issue been fixed? Is the CNI getting called only for the infra container? From what I see, the CNI still gets called for every single container from within a pod:
See comments It is required for windows containers.
The way I understand it, it's that the DNS issue will be fixed in Windows RS5 and there will not be required additional calls. The workaround at the moment is to call the CNI multiple times until the DNS issue gets fixed in Windows. As far as I know, that is the only reason why the additional calls to the CNI are required.
Is this true? Do we still need to invoke cni for each container in RS5? And even with new cni dns capabilities ()?
the PR that you linked it's unrelated to this issue. The CNI should be called for the sandbox pod which is the infra container. All the other containers created within the same pod are sharing all network details, excepting DNS. This is an issue in 1709 and 1803 where not everything gets replicated properly to the other containers. As a workaround for this, you need to call AttachEndpoint from HCS on the newly created container network interface only to replicate the DNS as well. You can also see more details about this issue here:
Can you re-open the issue in the meanwhile since this issue still exists?
I still think this isn't a issue, but a requirement for windows containers. Even after RS5, we should probably support old windows versions. WDYH
Why would the spec of the CNI should be different on Windows? On Linux you know that the sandbox ID that will get passed to the CNI is always the sandbox container. On Windows the sandbox ID won't be only that, it will also be an ID of a container from within the same pod. Does this sound like normal behavior to you? For now I agree, the workaround is to call the CNI for every single container because of the Windows issue with DNS. But this doesn't mean we have to stick with this for future Windows releases which will not require calling the CNI ADD multiple times. Also, this is a hack for now () which needs to be fixed. Please re-open the issue until we get feedback from others as well, thanks.
OK. Reopened the issue. Let's wait a while for feedbacks from other folks.
Apologize for the delay. Windows doenst have the namespace support native on the platform. we this support as part of RS5 Platform release. We are working on making runtime spec changes and CRI/ContainerD changes to use this model. here is a runtime spec PR that got merged. This is orthogonal to the DNS issue. Once we are done with the CRI/ContainerD changes then if u use CRI/containerD as runtime and if your OS version is RS5 and above then you don't need this second CNI add call. But as Pengfei said, if you are on a downlevel version or using dockershim as Runtime then you will have to make the second CNI call. So in my opinion this code will stay as workaround forever until we remove the support for downlevel OS. Hope this helps. I would recommend closing this issue.
thanks for the confirmation. In my opinion this should stay open until it gets fixed at least for future Windows releases, because the way it is at the moment, the CNI will still be called multiple times regardless of the Windows version. Why would you recommend closing this issue? Shouldn't this be fixed for futures Windows releases? The other calls will be useless when Windows gets proper namespace support. Also, I recommend watching this presentation about the CNI which had place at the KubeCon from May I don't see why we shouldn't have this open until the issue gets fixed for future releases and we stick with the official SPEC of the CNI.
and I had talked before about end of life of downlevel OS like 1803 and 1709. I think we should keep this open or open a new Issue to track updating the code so that the CNI is not called multiple times
This code would exists as far as we support 1709, 1803. The new workflow to use the windows namespace is getting implemented in containerd/cri. We can move kubernetes to use this new flow only for RS5 and above, if we plan to do it here as well. There isn't a good way to switch between 2 workflows of cni callflow wrt the Windows version hence we are not planning to change here till 1709 & 1803 exists, without adding some ugly code.
thanks for the explanation
I totally agree that this has to be fixed for future release and we are actively working on it. I was recommending to close this issue and open a feature to track this request. We should be able to raise a PR in another 2 weeks to support native network namespace support. This code will stay as long as we support down-level OS'es. I am fine either way. I am trying to close any open issues that is not a bug on Windows Networking(Platform).
/kind feature Although from a kubernetes standpoint, this isn't a new feature. It doesn't need a new KEP and won't change any API. It will just be a future Windows network code cleanup.
/kind cleanup
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
- can you clarify a few things so we know what's required to fix this? 's the minimum OS version needed? Windows Server 2019, or a future release? this be possible to fix for dockershim, or just CRI-containerD? Thanks!
What's the minimum OS version needed? Windows Server 2019, or a future release? [Dinesh]: Minimum OS version needed is 2019 Will this be possible to fix for dockershim, or just CRI-containerD? [Dinesh]: yes its possible to fix for Dockershim as well. but we are starting it with ContainerD. ContainerD is already done and available for testing.
excellent. thank you Dinesh!
ping - you have logs coming in email for this. Can you help us determine if it's needed for 1.14 vs 1.15?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Ping
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Status on this?

return nil } <del> func (ds *dockerService) determinePodIPBySandboxID(uid string) string { </del> <ins> func (ds *dockerService) determinePodIPBySandboxID(uid string, sandbox *dockertypes.ContainerJSON) string { </ins> return "" }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: On Windows Server with sandbox support the CNI gets called for all the containers within the pod including the sandbox container itself. What you expected to happen: The CNI should be called only for the sandbox container since they share the same network details. How to reproduce it (as minimally and precisely as possible): kubelet Pod at the parameters that get passed when the CNI is called Anything else we need to know?: This applies to Windows Server with sandbox support. The CNI should be called only for the infra container. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig windows /sig network /kind bug /assign /cc
Reopen this because of
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
has been merged now. /close
Closing this issue. <detailsIn response to : has been merged now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
has this issue been fixed? Is the CNI getting called only for the infra container? From what I see, the CNI still gets called for every single container from within a pod:
See comments It is required for windows containers.
The way I understand it, it's that the DNS issue will be fixed in Windows RS5 and there will not be required additional calls. The workaround at the moment is to call the CNI multiple times until the DNS issue gets fixed in Windows. As far as I know, that is the only reason why the additional calls to the CNI are required.
Is this true? Do we still need to invoke cni for each container in RS5? And even with new cni dns capabilities ()?
the PR that you linked it's unrelated to this issue. The CNI should be called for the sandbox pod which is the infra container. All the other containers created within the same pod are sharing all network details, excepting DNS. This is an issue in 1709 and 1803 where not everything gets replicated properly to the other containers. As a workaround for this, you need to call AttachEndpoint from HCS on the newly created container network interface only to replicate the DNS as well. You can also see more details about this issue here:
Can you re-open the issue in the meanwhile since this issue still exists?
I still think this isn't a issue, but a requirement for windows containers. Even after RS5, we should probably support old windows versions. WDYH
Why would the spec of the CNI should be different on Windows? On Linux you know that the sandbox ID that will get passed to the CNI is always the sandbox container. On Windows the sandbox ID won't be only that, it will also be an ID of a container from within the same pod. Does this sound like normal behavior to you? For now I agree, the workaround is to call the CNI for every single container because of the Windows issue with DNS. But this doesn't mean we have to stick with this for future Windows releases which will not require calling the CNI ADD multiple times. Also, this is a hack for now () which needs to be fixed. Please re-open the issue until we get feedback from others as well, thanks.
OK. Reopened the issue. Let's wait a while for feedbacks from other folks.
Apologize for the delay. Windows doenst have the namespace support native on the platform. we this support as part of RS5 Platform release. We are working on making runtime spec changes and CRI/ContainerD changes to use this model. here is a runtime spec PR that got merged. This is orthogonal to the DNS issue. Once we are done with the CRI/ContainerD changes then if u use CRI/containerD as runtime and if your OS version is RS5 and above then you don't need this second CNI add call. But as Pengfei said, if you are on a downlevel version or using dockershim as Runtime then you will have to make the second CNI call. So in my opinion this code will stay as workaround forever until we remove the support for downlevel OS. Hope this helps. I would recommend closing this issue.
thanks for the confirmation. In my opinion this should stay open until it gets fixed at least for future Windows releases, because the way it is at the moment, the CNI will still be called multiple times regardless of the Windows version. Why would you recommend closing this issue? Shouldn't this be fixed for futures Windows releases? The other calls will be useless when Windows gets proper namespace support. Also, I recommend watching this presentation about the CNI which had place at the KubeCon from May I don't see why we shouldn't have this open until the issue gets fixed for future releases and we stick with the official SPEC of the CNI.
and I had talked before about end of life of downlevel OS like 1803 and 1709. I think we should keep this open or open a new Issue to track updating the code so that the CNI is not called multiple times
This code would exists as far as we support 1709, 1803. The new workflow to use the windows namespace is getting implemented in containerd/cri. We can move kubernetes to use this new flow only for RS5 and above, if we plan to do it here as well. There isn't a good way to switch between 2 workflows of cni callflow wrt the Windows version hence we are not planning to change here till 1709 & 1803 exists, without adding some ugly code.
thanks for the explanation
I totally agree that this has to be fixed for future release and we are actively working on it. I was recommending to close this issue and open a feature to track this request. We should be able to raise a PR in another 2 weeks to support native network namespace support. This code will stay as long as we support down-level OS'es. I am fine either way. I am trying to close any open issues that is not a bug on Windows Networking(Platform).
/kind feature Although from a kubernetes standpoint, this isn't a new feature. It doesn't need a new KEP and won't change any API. It will just be a future Windows network code cleanup.
/kind cleanup
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
- can you clarify a few things so we know what's required to fix this? 's the minimum OS version needed? Windows Server 2019, or a future release? this be possible to fix for dockershim, or just CRI-containerD? Thanks!
What's the minimum OS version needed? Windows Server 2019, or a future release? [Dinesh]: Minimum OS version needed is 2019 Will this be possible to fix for dockershim, or just CRI-containerD? [Dinesh]: yes its possible to fix for Dockershim as well. but we are starting it with ContainerD. ContainerD is already done and available for testing.
excellent. thank you Dinesh!
ping - you have logs coming in email for this. Can you help us determine if it's needed for 1.14 vs 1.15?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Ping
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Status on this?

return nil } <del> func (ds *dockerService) determinePodIPBySandboxID(uid string) string { </del> <ins> func (ds *dockerService) determinePodIPBySandboxID(uid string, sandbox *dockertypes.ContainerJSON) string { </ins> glog.Warningf("determinePodIPBySandboxID is unsupported in this build") return "" }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: On Windows Server with sandbox support the CNI gets called for all the containers within the pod including the sandbox container itself. What you expected to happen: The CNI should be called only for the sandbox container since they share the same network details. How to reproduce it (as minimally and precisely as possible): kubelet Pod at the parameters that get passed when the CNI is called Anything else we need to know?: This applies to Windows Server with sandbox support. The CNI should be called only for the infra container. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig windows /sig network /kind bug /assign /cc
Reopen this because of
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
has been merged now. /close
Closing this issue. <detailsIn response to : has been merged now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
has this issue been fixed? Is the CNI getting called only for the infra container? From what I see, the CNI still gets called for every single container from within a pod:
See comments It is required for windows containers.
The way I understand it, it's that the DNS issue will be fixed in Windows RS5 and there will not be required additional calls. The workaround at the moment is to call the CNI multiple times until the DNS issue gets fixed in Windows. As far as I know, that is the only reason why the additional calls to the CNI are required.
Is this true? Do we still need to invoke cni for each container in RS5? And even with new cni dns capabilities ()?
the PR that you linked it's unrelated to this issue. The CNI should be called for the sandbox pod which is the infra container. All the other containers created within the same pod are sharing all network details, excepting DNS. This is an issue in 1709 and 1803 where not everything gets replicated properly to the other containers. As a workaround for this, you need to call AttachEndpoint from HCS on the newly created container network interface only to replicate the DNS as well. You can also see more details about this issue here:
Can you re-open the issue in the meanwhile since this issue still exists?
I still think this isn't a issue, but a requirement for windows containers. Even after RS5, we should probably support old windows versions. WDYH
Why would the spec of the CNI should be different on Windows? On Linux you know that the sandbox ID that will get passed to the CNI is always the sandbox container. On Windows the sandbox ID won't be only that, it will also be an ID of a container from within the same pod. Does this sound like normal behavior to you? For now I agree, the workaround is to call the CNI for every single container because of the Windows issue with DNS. But this doesn't mean we have to stick with this for future Windows releases which will not require calling the CNI ADD multiple times. Also, this is a hack for now () which needs to be fixed. Please re-open the issue until we get feedback from others as well, thanks.
OK. Reopened the issue. Let's wait a while for feedbacks from other folks.
Apologize for the delay. Windows doenst have the namespace support native on the platform. we this support as part of RS5 Platform release. We are working on making runtime spec changes and CRI/ContainerD changes to use this model. here is a runtime spec PR that got merged. This is orthogonal to the DNS issue. Once we are done with the CRI/ContainerD changes then if u use CRI/containerD as runtime and if your OS version is RS5 and above then you don't need this second CNI add call. But as Pengfei said, if you are on a downlevel version or using dockershim as Runtime then you will have to make the second CNI call. So in my opinion this code will stay as workaround forever until we remove the support for downlevel OS. Hope this helps. I would recommend closing this issue.
thanks for the confirmation. In my opinion this should stay open until it gets fixed at least for future Windows releases, because the way it is at the moment, the CNI will still be called multiple times regardless of the Windows version. Why would you recommend closing this issue? Shouldn't this be fixed for futures Windows releases? The other calls will be useless when Windows gets proper namespace support. Also, I recommend watching this presentation about the CNI which had place at the KubeCon from May I don't see why we shouldn't have this open until the issue gets fixed for future releases and we stick with the official SPEC of the CNI.
and I had talked before about end of life of downlevel OS like 1803 and 1709. I think we should keep this open or open a new Issue to track updating the code so that the CNI is not called multiple times
This code would exists as far as we support 1709, 1803. The new workflow to use the windows namespace is getting implemented in containerd/cri. We can move kubernetes to use this new flow only for RS5 and above, if we plan to do it here as well. There isn't a good way to switch between 2 workflows of cni callflow wrt the Windows version hence we are not planning to change here till 1709 & 1803 exists, without adding some ugly code.
thanks for the explanation
I totally agree that this has to be fixed for future release and we are actively working on it. I was recommending to close this issue and open a feature to track this request. We should be able to raise a PR in another 2 weeks to support native network namespace support. This code will stay as long as we support down-level OS'es. I am fine either way. I am trying to close any open issues that is not a bug on Windows Networking(Platform).
/kind feature Although from a kubernetes standpoint, this isn't a new feature. It doesn't need a new KEP and won't change any API. It will just be a future Windows network code cleanup.
/kind cleanup
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
- can you clarify a few things so we know what's required to fix this? 's the minimum OS version needed? Windows Server 2019, or a future release? this be possible to fix for dockershim, or just CRI-containerD? Thanks!
What's the minimum OS version needed? Windows Server 2019, or a future release? [Dinesh]: Minimum OS version needed is 2019 Will this be possible to fix for dockershim, or just CRI-containerD? [Dinesh]: yes its possible to fix for Dockershim as well. but we are starting it with ContainerD. ContainerD is already done and available for testing.
excellent. thank you Dinesh!
ping - you have logs coming in email for this. Can you help us determine if it's needed for 1.14 vs 1.15?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Ping
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Status on this?

return nil } <del> func (ds *dockerService) determinePodIPBySandboxID(sandboxID string) string { </del> <ins> func (ds *dockerService) determinePodIPBySandboxID(sandboxID string, sandbox *dockertypes.ContainerJSON) string { // Versions and feature support // ============================ // Windows version >= Windows Server, Version 1709, Supports both sandbox and non-sandbox case // Windows version == Windows Server 2016 Support only non-sandbox case // Windows version < Windows Server 2016 is Not Supported // Sandbox support in Windows mandates CNI Plugin. // Presence of CONTAINER_NETWORK flag is considered as non-Sandbox cases here // Hyper-V isolated containers are also considered as non-Sandbox cases // Todo: Add a kernel version check for more validation // Hyper-V only supports one container per Pod yet and the container will have a different // IP address from sandbox. Retrieve the IP from the containers as this is a non-Sandbox case. // TODO(feiskyer): remove this workaround after Hyper-V supports multiple containers per Pod. if networkMode := os.Getenv("CONTAINER_NETWORK"); networkMode == "" && sandbox.HostConfig.Isolation != kubeletapis.HypervIsolationValue { // Sandbox case, fetch the IP from the sandbox container. return ds.getIP(sandboxID, sandbox) } // Non-Sandbox case, fetch the IP from the containers within the Pod. </ins> opts := dockertypes.ContainerListOptions{ All:     true, Filters: dockerfilters.NewArgs(),


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: On Windows Server with sandbox support the CNI gets called for all the containers within the pod including the sandbox container itself. What you expected to happen: The CNI should be called only for the sandbox container since they share the same network details. How to reproduce it (as minimally and precisely as possible): kubelet Pod at the parameters that get passed when the CNI is called Anything else we need to know?: This applies to Windows Server with sandbox support. The CNI should be called only for the infra container. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig windows /sig network /kind bug /assign /cc
Reopen this because of
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
has been merged now. /close
Closing this issue. <detailsIn response to : has been merged now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
has this issue been fixed? Is the CNI getting called only for the infra container? From what I see, the CNI still gets called for every single container from within a pod:
See comments It is required for windows containers.
The way I understand it, it's that the DNS issue will be fixed in Windows RS5 and there will not be required additional calls. The workaround at the moment is to call the CNI multiple times until the DNS issue gets fixed in Windows. As far as I know, that is the only reason why the additional calls to the CNI are required.
Is this true? Do we still need to invoke cni for each container in RS5? And even with new cni dns capabilities ()?
the PR that you linked it's unrelated to this issue. The CNI should be called for the sandbox pod which is the infra container. All the other containers created within the same pod are sharing all network details, excepting DNS. This is an issue in 1709 and 1803 where not everything gets replicated properly to the other containers. As a workaround for this, you need to call AttachEndpoint from HCS on the newly created container network interface only to replicate the DNS as well. You can also see more details about this issue here:
Can you re-open the issue in the meanwhile since this issue still exists?
I still think this isn't a issue, but a requirement for windows containers. Even after RS5, we should probably support old windows versions. WDYH
Why would the spec of the CNI should be different on Windows? On Linux you know that the sandbox ID that will get passed to the CNI is always the sandbox container. On Windows the sandbox ID won't be only that, it will also be an ID of a container from within the same pod. Does this sound like normal behavior to you? For now I agree, the workaround is to call the CNI for every single container because of the Windows issue with DNS. But this doesn't mean we have to stick with this for future Windows releases which will not require calling the CNI ADD multiple times. Also, this is a hack for now () which needs to be fixed. Please re-open the issue until we get feedback from others as well, thanks.
OK. Reopened the issue. Let's wait a while for feedbacks from other folks.
Apologize for the delay. Windows doenst have the namespace support native on the platform. we this support as part of RS5 Platform release. We are working on making runtime spec changes and CRI/ContainerD changes to use this model. here is a runtime spec PR that got merged. This is orthogonal to the DNS issue. Once we are done with the CRI/ContainerD changes then if u use CRI/containerD as runtime and if your OS version is RS5 and above then you don't need this second CNI add call. But as Pengfei said, if you are on a downlevel version or using dockershim as Runtime then you will have to make the second CNI call. So in my opinion this code will stay as workaround forever until we remove the support for downlevel OS. Hope this helps. I would recommend closing this issue.
thanks for the confirmation. In my opinion this should stay open until it gets fixed at least for future Windows releases, because the way it is at the moment, the CNI will still be called multiple times regardless of the Windows version. Why would you recommend closing this issue? Shouldn't this be fixed for futures Windows releases? The other calls will be useless when Windows gets proper namespace support. Also, I recommend watching this presentation about the CNI which had place at the KubeCon from May I don't see why we shouldn't have this open until the issue gets fixed for future releases and we stick with the official SPEC of the CNI.
and I had talked before about end of life of downlevel OS like 1803 and 1709. I think we should keep this open or open a new Issue to track updating the code so that the CNI is not called multiple times
This code would exists as far as we support 1709, 1803. The new workflow to use the windows namespace is getting implemented in containerd/cri. We can move kubernetes to use this new flow only for RS5 and above, if we plan to do it here as well. There isn't a good way to switch between 2 workflows of cni callflow wrt the Windows version hence we are not planning to change here till 1709 & 1803 exists, without adding some ugly code.
thanks for the explanation
I totally agree that this has to be fixed for future release and we are actively working on it. I was recommending to close this issue and open a feature to track this request. We should be able to raise a PR in another 2 weeks to support native network namespace support. This code will stay as long as we support down-level OS'es. I am fine either way. I am trying to close any open issues that is not a bug on Windows Networking(Platform).
/kind feature Although from a kubernetes standpoint, this isn't a new feature. It doesn't need a new KEP and won't change any API. It will just be a future Windows network code cleanup.
/kind cleanup
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
- can you clarify a few things so we know what's required to fix this? 's the minimum OS version needed? Windows Server 2019, or a future release? this be possible to fix for dockershim, or just CRI-containerD? Thanks!
What's the minimum OS version needed? Windows Server 2019, or a future release? [Dinesh]: Minimum OS version needed is 2019 Will this be possible to fix for dockershim, or just CRI-containerD? [Dinesh]: yes its possible to fix for Dockershim as well. but we are starting it with ContainerD. ContainerD is already done and available for testing.
excellent. thank you Dinesh!
ping - you have logs coming in email for this. Can you help us determine if it's needed for 1.14 vs 1.15?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Ping
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Status on this?

continue } <del> // Versions and feature support // ============================ // Windows version == Windows Server, Version 1709,, Supports both sandbox and non-sandbox case // Windows version == Windows Server 2016   Support only non-sandbox case // Windows version < Windows Server 2016 is Not Supported // Sandbox support in Windows mandates CNI Plugin. // Presence of CONTAINER_NETWORK flag is considered as non-Sandbox cases here // Todo: Add a kernel version check for more validation if networkMode := os.Getenv("CONTAINER_NETWORK"); networkMode == "" { // On Windows, every container that is created in a Sandbox, needs to invoke CNI plugin again for adding the Network, // with the shared container name as NetNS info, // This is passed down to the platform to replicate some necessary information to the new container // // This place is chosen as a hack for now, since ds.getIP would end up calling CNI's addToNetwork // That is why addToNetwork is required to be idempotent // Instead of relying on this call, an explicit call to addToNetwork should be // done immediately after ContainerCreation, in case of Windows only. TBD Issue # to handle this if r.HostConfig.Isolation == kubeletapis.HypervIsolationValue { // Hyper-V only supports one container per Pod yet and the container will have a different // IP address from sandbox. Return the first non-sandbox container IP as POD IP. // TODO(feiskyer): remove this workaround after Hyper-V supports multiple containers per Pod. if containerIP := ds.getIP(c.ID, r); containerIP != "" { return containerIP } } else { // Do not return any IP, so that we would continue and get the IP of the Sandbox ds.getIP(sandboxID, r) } } else { // ds.getIP will call the CNI plugin to fetch the IP if containerIP := ds.getIP(c.ID, r); containerIP != "" { return containerIP } </del> <ins> if containerIP := ds.getIP(c.ID, r); containerIP != "" { return containerIP </ins> } }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: I am suffering from mysterious timeouts. See for the evidence from my latest investigation. It includes this line from the function in . The relevant line is this: I still do not know the type of the error in question or where it is coming from. It would be helpful if the log message included the type of the error object --- such as with or, if that does not include the type's package name, perhaps something that would really nail the type. What you expected to happen: I would expect more information about an unexpected type of error. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.3 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I think this timeout is from resourcequota admission. Did you enable admission plugin? will create a pr to make ResourceQuota return err.
thanks for the suggestion, I will respond in the original issue.
I appreciate removing a source of errors of unexpected type, but this issue is about what to do whenever an unexpected type of error does arrive. I suggest formatting it with instead of .
/reopen That PR did not really fix this issue.

// by REST storage - these typically indicate programmer // error by not using pkg/api/errors, or unexpected failure // cases. <del> runtime.HandleError(fmt.Errorf("apiserver received an error that is not an metav1.Status: %v", err)) </del> <ins> runtime.HandleError(fmt.Errorf("apiserver received an error that is not an metav1.Status: %#+v", err)) </ins> return &metav1.Status{ TypeMeta: metav1.TypeMeta{ Kind:       "Status",


Question:
/kind bug What happened: Wrote a that returned a faulty json patch which specified an our of range index (e.G. when there is only one container). This resulted in the apiserver crashing. Stacktrace: What you expected to happen: No crash /sig-api-machinery
/sig api-machinery
This maybe a bug in . I will follow this question.
This bug had been fixed in PR: you can checkout it.
Thanks for your comment but I don't think the linked PR fixed the issue, as it got merged almost a year ago and I am running Kube 1.10.1 which does contain that patch but still panics. Just from quickly looking at the code I the linked PR I believe there are two things wrong: The check if the index is valid only happens when a negative index was given should be
Once the PR of is merged I will update kubernetes.
/kind bug

}, { "ImportPath": "github.com/evanphx/json-patch", <del> "Rev": "ed7cfbae1fffc071f71e068df27bf4f0521402d8" </del> <ins> "Rev": "94e38aa1586e8a6c8a75770bddf5ff84c48a106b" </ins> }, { "ImportPath": "github.com/exponent-io/jsonpath",


Question:
/kind bug What happened: Wrote a that returned a faulty json patch which specified an our of range index (e.G. when there is only one container). This resulted in the apiserver crashing. Stacktrace: What you expected to happen: No crash /sig-api-machinery
/sig api-machinery
This maybe a bug in . I will follow this question.
This bug had been fixed in PR: you can checkout it.
Thanks for your comment but I don't think the linked PR fixed the issue, as it got merged almost a year ago and I am running Kube 1.10.1 which does contain that patch but still panics. Just from quickly looking at the code I the linked PR I believe there are two things wrong: The check if the index is valid only happens when a negative index was given should be
Once the PR of is merged I will update kubernetes.
/kind bug

}, { "ImportPath": "github.com/evanphx/json-patch", <del> "Rev": "ed7cfbae1fffc071f71e068df27bf4f0521402d8" </del> <ins> "Rev": "94e38aa1586e8a6c8a75770bddf5ff84c48a106b" </ins> }, { "ImportPath": "github.com/ghodss/yaml",


Question:
/kind bug What happened: Wrote a that returned a faulty json patch which specified an our of range index (e.G. when there is only one container). This resulted in the apiserver crashing. Stacktrace: What you expected to happen: No crash /sig-api-machinery
/sig api-machinery
This maybe a bug in . I will follow this question.
This bug had been fixed in PR: you can checkout it.
Thanks for your comment but I don't think the linked PR fixed the issue, as it got merged almost a year ago and I am running Kube 1.10.1 which does contain that patch but still panics. Just from quickly looking at the code I the linked PR I believe there are two things wrong: The check if the index is valid only happens when a negative index was given should be
Once the PR of is merged I will update kubernetes.
/kind bug

<del> ## JSON-Patch </del> <ins> # JSON-Patch `jsonpatch` is a library which provides functionallity for both applying [RFC6902 JSON patches](http://tools.ietf.org/html/rfc6902) against documents, as well as for calculating & applying [RFC7396 JSON merge patches](https://tools.ietf.org/html/rfc7396). </ins> <del> Provides the ability to modify and test a JSON according to a [RFC6902 JSON patch](http://tools.ietf.org/html/rfc6902) and [RFC7396 JSON Merge Patch](https://tools.ietf.org/html/rfc7396). </del> <ins> [![GoDoc](https://godoc.org/github.com/evanphx/json-patch?status.svg)](http://godoc.org/github.com/evanphx/json-patch) [![Build Status](https://travis-ci.org/evanphx/json-patch.svg?branch=master)](https://travis-ci.org/evanphx/json-patch) [![Report Card](https://goreportcard.com/badge/github.com/evanphx/json-patch)](https://goreportcard.com/report/github.com/evanphx/json-patch) </ins> <del> *Version*: **1.0** </del> <ins> # Get It! </ins> <del> [![GoDoc](https://godoc.org/github.com/evanphx/json-patch?status.svg)](http://godoc.org/github.com/evanphx/json-patch) </del> <ins> **Latest and greatest**: ```bash go get -u github.com/evanphx/json-patch ``` </ins> <del> [![Build Status](https://travis-ci.org/evanphx/json-patch.svg?branch=master)](https://travis-ci.org/evanphx/json-patch) </del> <ins> **Stable Versions**: * Version 3: `go get -u gopkg.in/evanphx/json-patch.v3` (previous versions below `v3` are unavailable) # Use It! * [Create and apply a merge patch](#create-and-apply-a-merge-patch) * [Create and apply a JSON Patch](#create-and-apply-a-json-patch) * [Comparing JSON documents](#comparing-json-documents) * [Combine merge patches](#combine-merge-patches) ## Create and apply a merge patch Given both an original JSON document and a modified JSON document, you can create a [Merge Patch](https://tools.ietf.org/html/rfc7396) document. It can describe the changes needed to convert from the original to the modified JSON document. Once you have a merge patch, you can apply it to other JSON documents using the `jsonpatch.MergePatch(document, patch)` function. ```go package main import ( "fmt" jsonpatch "github.com/evanphx/json-patch" ) func main() { // Let's create a merge patch from these two documents... original := []byte(`{"name": "John", "age": 24, "height": 3.21}`) target := []byte(`{"name": "Jane", "age": 24}`) patch, err := jsonpatch.CreateMergePatch(original, target) if err != nil { panic(err) } // Now lets apply the patch against a different JSON document... alternative := []byte(`{"name": "Tina", "age": 28, "height": 3.75}`) modifiedAlternative, err := jsonpatch.MergePatch(alternative, patch) fmt.Printf("patch document:   %sn", patch) fmt.Printf("updated alternative doc: %sn", modifiedAlternative) } ``` When ran, you get the following output: ```bash $ go run main.go patch document:   {"height":null,"name":"Jane"} updated tina doc: {"age":28,"name":"Jane"} ``` ## Create and apply a JSON Patch You can create patch objects using `DecodePatch([]byte)`, which can then be applied against JSON documents. The following is an example of creating a patch from two operations, and applying it against a JSON document. ```go package main import ( "fmt" jsonpatch "github.com/evanphx/json-patch" ) func main() { original := []byte(`{"name": "John", "age": 24, "height": 3.21}`) patchJSON := []byte(`[ {"op": "replace", "path": "/name", "value": "Jane"}, {"op": "remove", "path": "/height"} ]`) patch, err := jsonpatch.DecodePatch(patchJSON) if err != nil { panic(err) } modified, err := patch.Apply(original) if err != nil { panic(err) } fmt.Printf("Original document: %sn", original) fmt.Printf("Modified document: %sn", modified) } ``` When ran, you get the following output: ```bash $ go run main.go Original document: {"name": "John", "age": 24, "height": 3.21} Modified document: {"age":24,"name":"Jane"} ``` ## Comparing JSON documents Due to potential whitespace and ordering differences, one cannot simply compare JSON strings or byte-arrays directly. As such, you can instead use `jsonpatch.Equal(document1, document2)` to determine if two JSON documents are _structurally_ equal. This ignores whitespace differences, and key-value ordering. ```go package main import ( "fmt" jsonpatch "github.com/evanphx/json-patch" ) func main() { original := []byte(`{"name": "John", "age": 24, "height": 3.21}`) similar := []byte(` { "age": 24, "height": 3.21, "name": "John" } `) different := []byte(`{"name": "Jane", "age": 20, "height": 3.37}`) if jsonpatch.Equal(original, similar) { fmt.Println(`"original" is structurally equal to "similar"`) } if !jsonpatch.Equal(original, different) { fmt.Println(`"original" is _not_ structurally equal to "similar"`) } } ``` When ran, you get the following output: ```bash $ go run main.go "original" is structurally equal to "similar" "original" is _not_ structurally equal to "similar" ``` ## Combine merge patches Given two JSON merge patch documents, it is possible to combine them into a single merge patch which can describe both set of changes. The resulting merge patch can be used such that applying it results in a document structurally similar as merging each merge patch to the document in succession. ```go package main import ( "fmt" jsonpatch "github.com/evanphx/json-patch" ) func main() { original := []byte(`{"name": "John", "age": 24, "height": 3.21}`) nameAndHeight := []byte(`{"height":null,"name":"Jane"}`) ageAndEyes := []byte(`{"age":4.23,"eyes":"blue"}`) // Let's combine these merge patch documents... combinedPatch, err := jsonpatch.MergeMergePatches(nameAndHeight, ageAndEyes) if err != nil { panic(err) } // Apply each patch individual against the original document withoutCombinedPatch, err := jsonpatch.MergePatch(original, nameAndHeight) if err != nil { panic(err) } withoutCombinedPatch, err = jsonpatch.MergePatch(withoutCombinedPatch, ageAndEyes) if err != nil { panic(err) } // Apply the combined patch against the original document withCombinedPatch, err := jsonpatch.MergePatch(original, combinedPatch) if err != nil { panic(err) } // Do both result in the same thing? They should! if jsonpatch.Equal(withCombinedPatch, withoutCombinedPatch) { fmt.Println("Both JSON documents are structurally the same!") } fmt.Printf("combined merge patch: %s", combinedPatch) } ``` When ran, you get the following output: ```bash $ go run main.go Both JSON documents are structurally the same! combined merge patch: {"age":4.23,"eyes":"blue","height":null,"name":"Jane"} ``` # CLI for comparing JSON documents You can install the commandline program `json-patch`. This program can take multiple JSON patch documents as arguments, and fed a JSON document from `stdin`. It will apply the patch(es) against the document and output the modified doc. **patch.1.json** ```json [ {"op": "replace", "path": "/name", "value": "Jane"}, {"op": "remove", "path": "/height"} ] ``` **patch.2.json** ```json [ {"op": "add", "path": "/address", "value": "123 Main St"}, {"op": "replace", "path": "/age", "value": "21"} ] ``` </ins> <del> ### API Usage </del> <ins> **document.json** ```json { "name": "John", "age": 24, "height": 3.21 } ``` </ins> <del> * Given a `[]byte`, obtain a Patch object </del> <ins> You can then run: </ins> <del> `obj, err := jsonpatch.DecodePatch(patch)` </del> <ins> ```bash $ go install github.com/evanphx/json-patch/cmd/json-patch $ cat document.json | json-patch -p patch.1.json -p patch.2.json {"address":"123 Main St","age":"21","name":"Jane"} ``` </ins> <del> * Apply the patch and get a new document back </del> <ins> # Help It! Contributions are welcomed! Leave [an issue](https://github.com/evanphx/json-patch/issues) or [create a PR](https://github.com/evanphx/json-patch/compare). </ins> <del> `out, err := obj.Apply(doc)` * Create a JSON Merge Patch document based on two json documents (a to b): </del> <ins> Before creating a pull request, we'd ask that you make sure tests are passing and that you have added new tests when applicable. </ins> <del> `mergeDoc, err := jsonpatch.CreateMergePatch(a, b)` * Bonus API: compare documents for structural equality </del> <ins> Contributors can run tests using: </ins> <del> `jsonpatch.Equal(doca, docb)` </del> <ins> ```bash go test -cover ./... ``` Builds for pull requests are tested automatically using [TravisCI](https://travis-ci.org/evanphx/json-patch). </ins>


Question:
/kind bug What happened: Wrote a that returned a faulty json patch which specified an our of range index (e.G. when there is only one container). This resulted in the apiserver crashing. Stacktrace: What you expected to happen: No crash /sig-api-machinery
/sig api-machinery
This maybe a bug in . I will follow this question.
This bug had been fixed in PR: you can checkout it.
Thanks for your comment but I don't think the linked PR fixed the issue, as it got merged almost a year ago and I am running Kube 1.10.1 which does contain that patch but still panics. Just from quickly looking at the code I the linked PR I believe there are two things wrong: The check if the index is valid only happens when a negative index was given should be
Once the PR of is merged I will update kubernetes.
/kind bug

package jsonpatch import ( <ins> "bytes" </ins> "encoding/json" "fmt" "reflect"


Question:
/kind bug What happened: Wrote a that returned a faulty json patch which specified an our of range index (e.G. when there is only one container). This resulted in the apiserver crashing. Stacktrace: What you expected to happen: No crash /sig-api-machinery
/sig api-machinery
This maybe a bug in . I will follow this question.
This bug had been fixed in PR: you can checkout it.
Thanks for your comment but I don't think the linked PR fixed the issue, as it got merged almost a year ago and I am running Kube 1.10.1 which does contain that patch but still panics. Just from quickly looking at the code I the linked PR I believe there are two things wrong: The check if the index is valid only happens when a negative index was given should be
Once the PR of is merged I will update kubernetes.
/kind bug

var errBadJSONDoc = fmt.Errorf("Invalid JSON Document") var errBadJSONPatch = fmt.Errorf("Invalid JSON Patch") <ins> var errBadMergeTypes = fmt.Errorf("Mismatched JSON Documents") </ins> // MergeMergePatches merges two merge patches together, such that // applying this resulting merged merge patch to a document yields the same


Question:
/kind bug What happened: Wrote a that returned a faulty json patch which specified an our of range index (e.G. when there is only one container). This resulted in the apiserver crashing. Stacktrace: What you expected to happen: No crash /sig-api-machinery
/sig api-machinery
This maybe a bug in . I will follow this question.
This bug had been fixed in PR: you can checkout it.
Thanks for your comment but I don't think the linked PR fixed the issue, as it got merged almost a year ago and I am running Kube 1.10.1 which does contain that patch but still panics. Just from quickly looking at the code I the linked PR I believe there are two things wrong: The check if the index is valid only happens when a negative index was given should be
Once the PR of is merged I will update kubernetes.
/kind bug

return json.Marshal(doc) } <del> // CreateMergePatch creates a merge patch as specified in http://tools.ietf.org/html/draft-ietf-appsawg-json-merge-patch-07 // // 'a' is original, 'b' is the modified document. Both are to be given as json encoded content. // The function will return a mergeable json document with differences from a to b. // // An error will be returned if any of the two documents are invalid. func CreateMergePatch(a, b []byte) ([]byte, error) { aI := map[string]interface{}{} bI := map[string]interface{}{} err := json.Unmarshal(a, &aI) </del> <ins> // resemblesJSONArray indicates whether the byte-slice "appears" to be // a JSON array or not. // False-positives are possible, as this function does not check the internal // structure of the array. It only checks that the outer syntax is present and // correct. func resemblesJSONArray(input []byte) bool { input = bytes.TrimSpace(input) hasPrefix := bytes.HasPrefix(input, []byte("[")) hasSuffix := bytes.HasSuffix(input, []byte("]")) return hasPrefix && hasSuffix } // CreateMergePatch will return a merge patch document capable of converting // the original document(s) to the modified document(s). // The parameters can be bytes of either two JSON Documents, or two arrays of // JSON documents. // The merge patch returned follows the specification defined at http://tools.ietf.org/html/draft-ietf-appsawg-json-merge-patch-07 func CreateMergePatch(originalJSON, modifiedJSON []byte) ([]byte, error) { originalResemblesArray := resemblesJSONArray(originalJSON) modifiedResemblesArray := resemblesJSONArray(modifiedJSON) // Do both byte-slices seem like JSON arrays? if originalResemblesArray && modifiedResemblesArray { return createArrayMergePatch(originalJSON, modifiedJSON) } // Are both byte-slices are not arrays? Then they are likely JSON objects... if !originalResemblesArray && !modifiedResemblesArray { return createObjectMergePatch(originalJSON, modifiedJSON) } // None of the above? Then return an error because of mismatched types. return nil, errBadMergeTypes } // createObjectMergePatch will return a merge-patch document capable of // converting the original document to the modified document. func createObjectMergePatch(originalJSON, modifiedJSON []byte) ([]byte, error) { originalDoc := map[string]interface{}{} modifiedDoc := map[string]interface{}{} err := json.Unmarshal(originalJSON, &originalDoc) </ins> if err != nil { return nil, errBadJSONDoc } <del> err = json.Unmarshal(b, &bI) </del> <ins> err = json.Unmarshal(modifiedJSON, &modifiedDoc) </ins> if err != nil { return nil, errBadJSONDoc } <del> dest, err := getDiff(aI, bI) </del> <ins> dest, err := getDiff(originalDoc, modifiedDoc) </ins> if err != nil { return nil, err } return json.Marshal(dest) } <ins> // createArrayMergePatch will return an array of merge-patch documents capable // of converting the original document to the modified document for each // pair of JSON documents provided in the arrays. // Arrays of mismatched sizes will result in an error. func createArrayMergePatch(originalJSON, modifiedJSON []byte) ([]byte, error) { originalDocs := []json.RawMessage{} modifiedDocs := []json.RawMessage{} err := json.Unmarshal(originalJSON, &originalDocs) if err != nil { return nil, errBadJSONDoc } err = json.Unmarshal(modifiedJSON, &modifiedDocs) if err != nil { return nil, errBadJSONDoc } total := len(originalDocs) if len(modifiedDocs) != total { return nil, errBadJSONDoc } result := []json.RawMessage{} for i := 0; i < len(originalDocs); i++ { original := originalDocs[i] modified := modifiedDocs[i] patch, err := createObjectMergePatch(original, modified) if err != nil { return nil, err } result = append(result, json.RawMessage(patch)) } return json.Marshal(result) } </ins> // Returns true if the array matches (must be json types). // As is idiomatic for go, an empty array is not the same as a nil array. func matchesArray(a, b []interface{}) bool {


Question:
/kind bug What happened: Wrote a that returned a faulty json patch which specified an our of range index (e.G. when there is only one container). This resulted in the apiserver crashing. Stacktrace: What you expected to happen: No crash /sig-api-machinery
/sig api-machinery
This maybe a bug in . I will follow this question.
This bug had been fixed in PR: you can checkout it.
Thanks for your comment but I don't think the linked PR fixed the issue, as it got merged almost a year ago and I am running Kube 1.10.1 which does contain that patch but still panics. Just from quickly looking at the code I the linked PR I believe there are two things wrong: The check if the index is valid only happens when a negative index was given should be
Once the PR of is merged I will update kubernetes.
/kind bug

} idx = len(ary) - idx } <ins> if idx < 0 || idx >= len(ary) || idx > len(cur) { return fmt.Errorf("Unable to access invalid index: %d", idx) } </ins> copy(ary[0:idx], cur[0:idx]) ary[idx] = val copy(ary[idx+1:], cur[idx:])


Question:
/kind bug /priority critical-urgent What happened: In I'm removing all the reapers, unfortunately when invoking foreground deletion on a DaemonSet the delete operation hangs and controller indefinitely tries to manage a removed DS. What you expected to happen: A foreground deletion properly removes DaemonSet and all its dependents. How to reproduce it (as minimally and precisely as possible): Invoke a delete operation against kube API (without using command). Anything else we need to know?: This behavior makes it so that none of the consumers of the API is not able to remove the DS controller due to present in the DS reaper. A working fix that passed the tests with the aforementioned PR is in but I'm not 100% it's correct. This is an issue that needs fixing before 1.11 goes out. /assign /sig apps /cc
fyi
I filed to fix it. As mentioned in the PR, this is because the DaemonSet controller races with garbage collector when it tries to update DaemonSet status of the DaemonSet being deleted. Here's what happens when hangs: /something performs a foreground deletion on a DaemonSet finalizer and DeletionTimestamp are both set history objects (ControllerRevisions) and pods are being deleted by garbage collector; meanwhile, DaemonSet controller tries to update DaemonSet status. Updating DaemonSet status requires constructing DaemonSet history objects, to figure out current revision and which pods do/don't belong to current revision updating DaemonSet status, DaemonSet controller tries to create a DaemonSet history object that matches current DaemonSet spec collector then tries to delete that DaemonSet history object. And repeat. Because we can't make DaemonSet pods be deleted before DaemonSet history objects (DaemonSet history objects don't own DaemonSet pods!), we cannot reliably calculate DaemonSet status without history objects anyways. Therefore, we don't update DaemonSet status for DaemonSet being deleted. Note that the reason why the kubectl delete hack works is because it forces DaemonSet pods to be removed before history objects.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details

return fmt.Errorf("couldn't get key for object %#v: %v", ds, err) } <ins> // If the DaemonSet is being deleted (either by foreground deletion or // orphan deletion), we cannot be sure if the DaemonSet history objects // it owned still exist -- those history objects can either be deleted // or orphaned. Garbage collector doesn't guarantee that it will delete // DaemonSet pods before deleting DaemonSet history objects, because // DaemonSet history doesn't own DaemonSet pods. We cannot reliably // calculate the status of a DaemonSet being deleted. Therefore, return // here without updating status for the DaemonSet being deleted. if ds.DeletionTimestamp != nil { return nil } </ins> // Construct histories of the DaemonSet, and get the hash of current history cur, old, err := dsc.constructHistory(ds) if err != nil {


Question:
/kind bug /priority critical-urgent What happened: In I'm removing all the reapers, unfortunately when invoking foreground deletion on a DaemonSet the delete operation hangs and controller indefinitely tries to manage a removed DS. What you expected to happen: A foreground deletion properly removes DaemonSet and all its dependents. How to reproduce it (as minimally and precisely as possible): Invoke a delete operation against kube API (without using command). Anything else we need to know?: This behavior makes it so that none of the consumers of the API is not able to remove the DS controller due to present in the DS reaper. A working fix that passed the tests with the aforementioned PR is in but I'm not 100% it's correct. This is an issue that needs fixing before 1.11 goes out. /assign /sig apps /cc
fyi
I filed to fix it. As mentioned in the PR, this is because the DaemonSet controller races with garbage collector when it tries to update DaemonSet status of the DaemonSet being deleted. Here's what happens when hangs: /something performs a foreground deletion on a DaemonSet finalizer and DeletionTimestamp are both set history objects (ControllerRevisions) and pods are being deleted by garbage collector; meanwhile, DaemonSet controller tries to update DaemonSet status. Updating DaemonSet status requires constructing DaemonSet history objects, to figure out current revision and which pods do/don't belong to current revision updating DaemonSet status, DaemonSet controller tries to create a DaemonSet history object that matches current DaemonSet spec collector then tries to delete that DaemonSet history object. And repeat. Because we can't make DaemonSet pods be deleted before DaemonSet history objects (DaemonSet history objects don't own DaemonSet pods!), we cannot reliably calculate DaemonSet status without history objects anyways. Therefore, we don't update DaemonSet status for DaemonSet being deleted. Note that the reason why the kubectl delete hack works is because it forces DaemonSet pods to be removed before history objects.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details

} hash := cur.Labels[apps.DefaultDaemonSetUniqueLabelKey] <del> if ds.DeletionTimestamp != nil || !dsc.expectations.SatisfiedExpectations(dsKey) { </del> <ins> if !dsc.expectations.SatisfiedExpectations(dsKey) { </ins> // Only update status. return dsc.updateDaemonSetStatus(ds, hash) }


Question:
/kind bug /priority critical-urgent What happened: In I'm removing all the reapers, unfortunately when invoking foreground deletion on a DaemonSet the delete operation hangs and controller indefinitely tries to manage a removed DS. What you expected to happen: A foreground deletion properly removes DaemonSet and all its dependents. How to reproduce it (as minimally and precisely as possible): Invoke a delete operation against kube API (without using command). Anything else we need to know?: This behavior makes it so that none of the consumers of the API is not able to remove the DS controller due to present in the DS reaper. A working fix that passed the tests with the aforementioned PR is in but I'm not 100% it's correct. This is an issue that needs fixing before 1.11 goes out. /assign /sig apps /cc
fyi
I filed to fix it. As mentioned in the PR, this is because the DaemonSet controller races with garbage collector when it tries to update DaemonSet status of the DaemonSet being deleted. Here's what happens when hangs: /something performs a foreground deletion on a DaemonSet finalizer and DeletionTimestamp are both set history objects (ControllerRevisions) and pods are being deleted by garbage collector; meanwhile, DaemonSet controller tries to update DaemonSet status. Updating DaemonSet status requires constructing DaemonSet history objects, to figure out current revision and which pods do/don't belong to current revision updating DaemonSet status, DaemonSet controller tries to create a DaemonSet history object that matches current DaemonSet spec collector then tries to delete that DaemonSet history object. And repeat. Because we can't make DaemonSet pods be deleted before DaemonSet history objects (DaemonSet history objects don't own DaemonSet pods!), we cannot reliably calculate DaemonSet status without history objects anyways. Therefore, we don't update DaemonSet status for DaemonSet being deleted. Note that the reason why the kubectl delete hack works is because it forces DaemonSet pods to be removed before history objects.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details

"//vendor/github.com/renstrom/dedent:go_default_library", "//vendor/github.com/spf13/cobra:go_default_library", "//vendor/github.com/spf13/pflag:go_default_library", <del> "//vendor/k8s.io/api/apps/v1:go_default_library", </del> "//vendor/k8s.io/api/autoscaling/v1:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/api/policy/v1beta1:go_default_library",


Question:
/kind bug /priority critical-urgent What happened: In I'm removing all the reapers, unfortunately when invoking foreground deletion on a DaemonSet the delete operation hangs and controller indefinitely tries to manage a removed DS. What you expected to happen: A foreground deletion properly removes DaemonSet and all its dependents. How to reproduce it (as minimally and precisely as possible): Invoke a delete operation against kube API (without using command). Anything else we need to know?: This behavior makes it so that none of the consumers of the API is not able to remove the DS controller due to present in the DS reaper. A working fix that passed the tests with the aforementioned PR is in but I'm not 100% it's correct. This is an issue that needs fixing before 1.11 goes out. /assign /sig apps /cc
fyi
I filed to fix it. As mentioned in the PR, this is because the DaemonSet controller races with garbage collector when it tries to update DaemonSet status of the DaemonSet being deleted. Here's what happens when hangs: /something performs a foreground deletion on a DaemonSet finalizer and DeletionTimestamp are both set history objects (ControllerRevisions) and pods are being deleted by garbage collector; meanwhile, DaemonSet controller tries to update DaemonSet status. Updating DaemonSet status requires constructing DaemonSet history objects, to figure out current revision and which pods do/don't belong to current revision updating DaemonSet status, DaemonSet controller tries to create a DaemonSet history object that matches current DaemonSet spec collector then tries to delete that DaemonSet history object. And repeat. Because we can't make DaemonSet pods be deleted before DaemonSet history objects (DaemonSet history objects don't own DaemonSet pods!), we cannot reliably calculate DaemonSet status without history objects anyways. Therefore, we don't update DaemonSet status for DaemonSet being deleted. Note that the reason why the kubectl delete hack works is because it forces DaemonSet pods to be removed before history objects.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question:
/kind bug /priority critical-urgent What happened: In I'm removing all the reapers, unfortunately when invoking foreground deletion on a DaemonSet the delete operation hangs and controller indefinitely tries to manage a removed DS. What you expected to happen: A foreground deletion properly removes DaemonSet and all its dependents. How to reproduce it (as minimally and precisely as possible): Invoke a delete operation against kube API (without using command). Anything else we need to know?: This behavior makes it so that none of the consumers of the API is not able to remove the DS controller due to present in the DS reaper. A working fix that passed the tests with the aforementioned PR is in but I'm not 100% it's correct. This is an issue that needs fixing before 1.11 goes out. /assign /sig apps /cc
fyi
I filed to fix it. As mentioned in the PR, this is because the DaemonSet controller races with garbage collector when it tries to update DaemonSet status of the DaemonSet being deleted. Here's what happens when hangs: /something performs a foreground deletion on a DaemonSet finalizer and DeletionTimestamp are both set history objects (ControllerRevisions) and pods are being deleted by garbage collector; meanwhile, DaemonSet controller tries to update DaemonSet status. Updating DaemonSet status requires constructing DaemonSet history objects, to figure out current revision and which pods do/don't belong to current revision updating DaemonSet status, DaemonSet controller tries to create a DaemonSet history object that matches current DaemonSet spec collector then tries to delete that DaemonSet history object. And repeat. Because we can't make DaemonSet pods be deleted before DaemonSet history objects (DaemonSet history objects don't own DaemonSet pods!), we cannot reliably calculate DaemonSet status without history objects anyways. Therefore, we don't update DaemonSet status for DaemonSet being deleted. Note that the reason why the kubectl delete hack works is because it forces DaemonSet pods to be removed before history objects.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details

"github.com/golang/glog" "github.com/spf13/cobra" <del> appsv1 "k8s.io/api/apps/v1" </del> "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <del> "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/util/wait" </del> "k8s.io/client-go/dynamic" "k8s.io/kubernetes/pkg/kubectl/cmd/templates" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util"