Question:
bumped the vendored docker library and introduced a regression where pods with images that don't contain an explicit "" repo path result in failures. Results in the following error: Pod gets shows that the image has been pulled, though not by that name Changing the image to in the pod spec corrects the issue. I think the new code that is changing the behavior is somewhere in here is a new function brought in by the bump. If I checkout the commit before the bump, the problem does not occur. related:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
taking a look now
What docker version are you using?
Do you have any e2e tests that are failing? We run a lot of tests with image "busybox"
Didn't think about it before, but I guess 1.12 is the supported version for 1.8 eh? Let me try it on 1.12.
Hm, weird. We also have cluster e2e test coverage for 1.13. Many test are using directly from this function
I installed a completely fresh box with docker 1.12 and could still recreate. I have no explanation :confused: from kubelet built from upstream/release-1.8
This is one of our e2e test node log: You could find tons of . One thing I don't understand on your side is that, logs in our test are always: Why on your node it's ? Where does that image come from? What if you delete it?
does this mean that upgrades will fail if the local image names need to change?
Seems not. Based on the following code, it's fine to pull/inspect with new reference when we previously pulled with old reference. Could not reproduce this.
I could not reproduce with the code above either.
For any image which has no tag, kubelet will add a default tag , and default repo to it. So will change to and send to dockerd to pull. But dockerd knows and are the same thing, so should return right instead of not found. (as demo) Can you make sure your dockerd is the a version compatible with k8s 1.8?
Just ran into this. If you up the kubelet's log level to 4, you'll see a "not found" error from For example: If we look at the code in that helper function, we have this: Here is the result of inspecting the image on my system (only showing relevant fields): You'll notice that the RepoTag it has is for , not . I am guessing the fact that we're somewhere inserting "library" into the image's name is what's tripping this up.
If it wasn't clear, I believe dockershim is what's having the issue. Once the kubelet sends the protobuf message to the dockershim socket to pull the image, it goes from to to to and then into the helper function I listed above.
It appears to be this: to to to That's where the "library" is coming from.
Prior to the PR that bumped docker/distribution, used to have , which does not do any normalization (i.e. including "library").
OK, I see why when we bumped to the newer docker/distribution, we switch to using . It's because now requires that the input be in "canonical" format (in this case, has and , if applicable), which you get automatically by calling ...
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
could you give steps to reproduce this? We can't reproduce this and many of the node/cluster e2e tests pull images using unqualified name such as "buxybox"
What docker distribution are you using? I could not reproduce this myself. I don't understand how does the reference show up, we never pull image with it. Is this a problem specific to a docker distribution?
run the e2e tests on a centos/fedora/rhel image
Is there any signal in test-grid for centos/fedora/rhel? Would be nice to catch issues like this sooner than later.
run the e2e tests on a centos/fedora/rhel image there is not, and definitely should be, given that k8s releases rpms as part of each release. will be on the short list for the release retrospective.
I feel like this is a problem in docker distribution in centos/fedora/rhel. Docker should not return when we are using to pull image. Should we fix the corresponding docker distribution instead of blocking Kubernete release?
Since that would mean releasing a new version would regress existing users, we generally say that we'll do our best to tolerate it instead of releasing something known broken. We've done that with various versions of Docker, systemd, and iptables for years, and I'm sure we'll do our best to do that. I do agree that this looks like a bug in that docker distribution that should be fixed. I also agree that this should have been in the test grid and that the distro owner is responsible for ensuring the project has that signal.
Thanks! I agree that we should not break existing users. I'm fine with as long as we keep in mind we made this change, and clean it up in the future. I think it would be helpful if we could have test coverage in test-grid on more distros, so that we could have caught similar problems earlier.
Agree - I think anyone who causes a release slip is required to make sure they fix the problem so it can't happen in the future. And get better all around testing. On Sep 27, 2017, at 7:40 PM, Lantao Liu <: ! I agree that we should not break existing users. I'm fine with long as we keep in mind we made this change, and clean it up in the future. I think it would be helpful if we could have test coverage on more distros, so that we could have caught similar problems earlier. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
I have opened a bug for this
This has broken the CRI as well. Kubernetes should not normalize image names on behalf of the CRI runtime. It's up to implementors to decide how to handle an image name and since there's not a standard around Kube and Image names, I'd prefer not to break the CRI.
Are you saying it's currently broken, or the fix was ok? , Antonio Murdaca <:
The fix is ok but the original patch caused another bug
Which one?
caused a bug fixed by
Does this problem fixed?

Output: true, }, { <ins> Inspected: dockertypes.ImageInspect{ ID:       "sha256:9bbdf247c91345f0789c10f50a57e36a667af1189687ad1de88a6243d05a2227", RepoTags: []string{"docker.io/busybox:latest"}, }, Image:  "docker.io/library/busybox:latest", Output: true, }, { </ins> // RepoDigest match is is required Inspected: dockertypes.ImageInspect{ ID:          "",


Question:
(SEE UPDATE at ) Currently, if you're writing an initializer controller, you're most likely using it to initialize a and deploying the controller with . Therefore you're likely using the following trick to avoid the chicken and egg problem: This bypasses the initializers for the pods and lets your controller to be deployed even though your InitializerConfiguration is configured to capture all pods. Problem occurs when you apply an update to this Deployment with the same manifest containing empty pending list. You get error: I think this is a somewhat major UX issue. It requires users to deploy the initializer controller with the but later on when they need to update the controller with , they can't use the same manifest containing the empty list. /area api /area extensibility /sig api-machinery /kind bug /assign
/cc
We silently ignore some metadata updates, which is one option. Potentially we shouldn't require empty initializers pending, but simply empty initializers (which means - init directly), and only ignore that. Are there any examples of use of SMP for similar? I'm not sure this meets the bar for 1.8 but if we can find an example of something consistent already in the API I'd be willing to review.
I don't have any concrete examples, there's an example initializer at making use the trick, I use it myself, and I suspect many more initializers to come will use this, too. I'm not sure if this would work, but does setting provide a way to skip initialization (if possible at all)? If not, we might actually need a separate field to signal skipping initialization.
/cc
Currently the initializer plugin to on UPDATE, but that happens after it updating already initialized objects. We can simply change their order.
I'd like to amend the issue, it's more serious than I thought. (thanks for bringing it to my attention If you're deploying your Initializer with and if you have an initializerconfiguration for objects, it's impossible to deploy your initializer in the first place without deleting InitializerConfiguration first: is not inherited by the created on is not honored (this field is not propagated to the created Pod). I think (2) should be fixed somehow, otherwise there will be no way of writing an initializer for Pods and deploying the initializer itself using Deployment/ReplicaSet/RC without deleting the InitializerConfiguration first. Maybe what we need is an exclusion/exemption selector (or namespace(s)) as discussed in to allow deployment of Initializers themselves easier.
For 1, see I don't want to pursue that though, because i) we'll need to fix workload controllers one by one, and ii) we need to carefully consider permission issues. I think we should focus on adding the namespace selector.

} updated := accessor.GetInitializers() <ins> // controllers deployed with an empty initializers.pending have their initializers set to nil // but should be able to update without changing their manifest if updated != nil && len(updated.Pending) == 0 && updated.Result == nil { accessor.SetInitializers(nil) updated = nil } </ins> existingAccessor, err := meta.Accessor(a.GetOldObject()) if err != nil { // if the old object does not have an accessor, but the new one does, error out


Question:
(SEE UPDATE at ) Currently, if you're writing an initializer controller, you're most likely using it to initialize a and deploying the controller with . Therefore you're likely using the following trick to avoid the chicken and egg problem: This bypasses the initializers for the pods and lets your controller to be deployed even though your InitializerConfiguration is configured to capture all pods. Problem occurs when you apply an update to this Deployment with the same manifest containing empty pending list. You get error: I think this is a somewhat major UX issue. It requires users to deploy the initializer controller with the but later on when they need to update the controller with , they can't use the same manifest containing the empty list. /area api /area extensibility /sig api-machinery /kind bug /assign
/cc
We silently ignore some metadata updates, which is one option. Potentially we shouldn't require empty initializers pending, but simply empty initializers (which means - init directly), and only ignore that. Are there any examples of use of SMP for similar? I'm not sure this meets the bar for 1.8 but if we can find an example of something consistent already in the API I'd be willing to review.
I don't have any concrete examples, there's an example initializer at making use the trick, I use it myself, and I suspect many more initializers to come will use this, too. I'm not sure if this would work, but does setting provide a way to skip initialization (if possible at all)? If not, we might actually need a separate field to signal skipping initialization.
/cc
Currently the initializer plugin to on UPDATE, but that happens after it updating already initialized objects. We can simply change their order.
I'd like to amend the issue, it's more serious than I thought. (thanks for bringing it to my attention If you're deploying your Initializer with and if you have an initializerconfiguration for objects, it's impossible to deploy your initializer in the first place without deleting InitializerConfiguration first: is not inherited by the created on is not honored (this field is not propagated to the created Pod). I think (2) should be fixed somehow, otherwise there will be no way of writing an initializer for Pods and deploying the initializer itself using Deployment/ReplicaSet/RC without deleting the InitializerConfiguration first. Maybe what we need is an exclusion/exemption selector (or namespace(s)) as discussed in to allow deployment of Initializers themselves easier.
For 1, see I don't want to pursue that though, because i) we'll need to fix workload controllers one by one, and ii) we need to carefully consider permission issues. I think we should focus on adding the namespace selector.

glog.V(5).Infof("Modifying uninitialized resource %s", a.GetResource()) <del> if updated != nil && len(updated.Pending) == 0 && updated.Result == nil { accessor.SetInitializers(nil) } </del> // because we are called before validation, we need to ensure the update transition is valid. if errs := validation.ValidateInitializersUpdate(updated, existing, initializerFieldPath); len(errs) > 0 { return errors.NewInvalid(a.GetKind().GroupKind(), a.GetName(), errs)


Question:
(SEE UPDATE at ) Currently, if you're writing an initializer controller, you're most likely using it to initialize a and deploying the controller with . Therefore you're likely using the following trick to avoid the chicken and egg problem: This bypasses the initializers for the pods and lets your controller to be deployed even though your InitializerConfiguration is configured to capture all pods. Problem occurs when you apply an update to this Deployment with the same manifest containing empty pending list. You get error: I think this is a somewhat major UX issue. It requires users to deploy the initializer controller with the but later on when they need to update the controller with , they can't use the same manifest containing the empty list. /area api /area extensibility /sig api-machinery /kind bug /assign
/cc
We silently ignore some metadata updates, which is one option. Potentially we shouldn't require empty initializers pending, but simply empty initializers (which means - init directly), and only ignore that. Are there any examples of use of SMP for similar? I'm not sure this meets the bar for 1.8 but if we can find an example of something consistent already in the API I'd be willing to review.
I don't have any concrete examples, there's an example initializer at making use the trick, I use it myself, and I suspect many more initializers to come will use this, too. I'm not sure if this would work, but does setting provide a way to skip initialization (if possible at all)? If not, we might actually need a separate field to signal skipping initialization.
/cc
Currently the initializer plugin to on UPDATE, but that happens after it updating already initialized objects. We can simply change their order.
I'd like to amend the issue, it's more serious than I thought. (thanks for bringing it to my attention If you're deploying your Initializer with and if you have an initializerconfiguration for objects, it's impossible to deploy your initializer in the first place without deleting InitializerConfiguration first: is not inherited by the created on is not honored (this field is not propagated to the created Pod). I think (2) should be fixed somehow, otherwise there will be no way of writing an initializer for Pods and deploying the initializer itself using Deployment/ReplicaSet/RC without deleting the InitializerConfiguration first. Maybe what we need is an exclusion/exemption selector (or namespace(s)) as discussed in to allow deployment of Initializers themselves easier.
For 1, see I don't want to pursue that though, because i) we'll need to fix workload controllers one by one, and ii) we need to carefully consider permission issues. I think we should focus on adding the namespace selector.

newInitializers: &metav1.Initializers{Pending: []metav1.Initializer{{Name: "init.k8s.io"}}}, err:             "field is immutable once initialization has completed", }, <ins> { name:            "empty initializer list is treated as nil initializer", oldInitializers: nil, newInitializers: &metav1.Initializers{}, verifyUpdatedObj: func(obj runtime.Object) (bool, string) { accessor, err := meta.Accessor(obj) if err != nil { return false, "cannot get accessor" } if accessor.GetInitializers() != nil { return false, "expect nil initializers" } return true, "" }, err: "", }, </ins> } plugin := initializer{


Question:
/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

"flag" "fmt" "net" <ins> "sort" </ins> "strings" "github.com/golang/glog"


Question:
/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

// String is the method to format the flag's value, part of the flag.Value interface. func (c *cidrs) String() string { <del> return strings.Join(c.ipn.StringSlice(), ",") </del> <ins> s := c.ipn.StringSlice() sort.Strings(s) return strings.Join(s, ",") </ins> } // Set supports a value of CSV or the flag repeated multiple times


Question:
/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

for format := range jsonFormats { formats = append(formats, format) } <ins> sort.Strings(formats) </ins> return formats }


Question:
/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

printFlags := JSONPathPrintFlags{ TemplateArgument: templateArg, } <ins> if !sort.StringsAreSorted(printFlags.AllowedFormats()) { t.Fatalf("allowed formats are not sorted") } </ins> p, err := printFlags.ToPrinter(tc.outputFormat) if tc.expectNoMatch {


Question:
/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

TemplateArgument: &tc.templateArg, AllowMissingKeys: tc.allowMissingKeys, } <ins> if !sort.StringsAreSorted(printFlags.AllowedFormats()) { t.Fatalf("allowed formats are not sorted") } </ins> outputFormat := "jsonpath" p, err := printFlags.ToPrinter(outputFormat)


Question:
/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

import ( "fmt" "io/ioutil" <ins> "sort" </ins> "strings" "github.com/spf13/cobra"


Question:
/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

for format := range templateFormats { formats = append(formats, format) } <ins> sort.Strings(formats) </ins> return formats }


Question:
/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

"fmt" "io/ioutil" "os" <ins> "sort" </ins> "strings" "testing"


Question:
/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

printFlags := GoTemplatePrintFlags{ TemplateArgument: templateArg, } <ins> if !sort.StringsAreSorted(printFlags.AllowedFormats()) { t.Fatalf("allowed formats are not sorted") } </ins> p, err := printFlags.ToPrinter(tc.outputFormat) if tc.expectNoMatch {


Question:
/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

TemplateArgument: &tc.templateArg, AllowMissingKeys: tc.allowMissingKeys, } <ins> if !sort.StringsAreSorted(printFlags.AllowedFormats()) { t.Fatalf("allowed formats are not sorted") } </ins> outputFormat := "template" p, err := printFlags.ToPrinter(outputFormat)


Question:
/kind e2e-test-failure GCE serial and GCI GCE serial started failing last evening https://k8s- https://k8s-
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/testing: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
https://k8s- is still green, I would suspect something ~last Friday afternoon gets merged
Here is what I found, after a while, the node lost their authorization to connect to API server. I believe this has the same root cause as
I am going to revert offending pr () to see if it helps. EDIT: Looks like is in the process of reverting it:

ETCD_QUORUM_READ: $(yaml-quote ${ETCD_QUORUM_READ}) EOF fi <ins> if [ -n "${CLUSTER_SIGNING_DURATION:-}" ]; then cat >>$file <<EOF CLUSTER_SIGNING_DURATION: $(yaml-quote ${CLUSTER_SIGNING_DURATION}) EOF fi </ins> else # Node-only env vars.


Question:
/kind e2e-test-failure GCE serial and GCI GCE serial started failing last evening https://k8s- https://k8s-
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/testing: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
https://k8s- is still green, I would suspect something ~last Friday afternoon gets merged
Here is what I found, after a while, the node lost their authorization to connect to API server. I believe this has the same root cause as
I am going to revert offending pr () to see if it helps. EDIT: Looks like is in the process of reverting it:

# Optional: [Experiment Only] Run kube-proxy as a DaemonSet if set to true, run as static pods otherwise. KUBE_PROXY_DAEMONSET="${KUBE_PROXY_DAEMONSET:-false}" # true, false <ins> # Optional: duration of cluster signed certificates. CLUSTER_SIGNING_DURATION="${CLUSTER_SIGNING_DURATION:-}" </ins> # Optional: enable pod priority ENABLE_POD_PRIORITY="${ENABLE_POD_PRIORITY:-}" if [[ "${ENABLE_POD_PRIORITY}" == "true" ]]; then


Question:
/kind e2e-test-failure GCE serial and GCI GCE serial started failing last evening https://k8s- https://k8s-
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/testing: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
https://k8s- is still green, I would suspect something ~last Friday afternoon gets merged
Here is what I found, after a while, the node lost their authorization to connect to API server. I believe this has the same root cause as
I am going to revert offending pr () to see if it helps. EDIT: Looks like is in the process of reverting it:

if [[ -n "${VOLUME_PLUGIN_DIR:-}" ]]; then params+=" --flex-volume-plugin-dir=${VOLUME_PLUGIN_DIR}" fi <ins> if [[ -n "${CLUSTER_SIGNING_DURATION:-}" ]]; then params+=" --experimental-cluster-signing-duration=$CLUSTER_SIGNING_DURATION" fi </ins> local -r kube_rc_docker_tag=$(cat /home/kubernetes/kube-docker-files/kube-controller-manager.docker_tag) local container_env="" if [[ -n "${ENABLE_CACHE_MUTATION_DETECTOR:-}" ]]; then


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG /kind bug What happened: Attempting to run e2e test on GCE without cluster started fails with stack trace What you expected to happen: A useful error message to be printed How to reproduce it (as minimally and precisely as possible): Without a cluster started: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig testing /assign
This happens because gcloud returns an error when a resource isn't found and ginkgo- sets errexit

if [[ -z "${KUBE_MASTER_IP-}" ]]; then local master_address_name="${MASTER_NAME}-ip" echo "Looking for address '${master_address_name}'" >&2 <del> KUBE_MASTER_IP=$(gcloud compute addresses describe "${master_address_name}"  --project "${PROJECT}" --region "${REGION}" -q --format='value(address)') fi if [[ -z "${KUBE_MASTER_IP-}" ]]; then echo "Could not detect Kubernetes master node.  Make sure you've launched a cluster with 'kube-up.sh'" >&2 exit 1 </del> <ins> if ! KUBE_MASTER_IP=$(gcloud compute addresses describe "${master_address_name}"  --project "${PROJECT}" --region "${REGION}" -q --format='value(address)') ||  [[ -z "${KUBE_MASTER_IP-}" ]]; then echo "Could not detect Kubernetes master node.  Make sure you've launched a cluster with 'kube-up.sh'" >&2 exit 1 fi </ins> fi echo "Using master: $KUBE_MASTER (external IP: $KUBE_MASTER_IP)" >&2 }


Question:
It is possible to have multiple signals that point to the same type of resource, e.g., both SignalNodeFsAvailable and SignalAllocatableNodeFsAvailable refer to the same resource NodeFs. Change the map from map[v1.ResourceName]evictionapi.Signal to map[v1.ResourceName][]evictionapi.Signal
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/kind cleanup /priority important/soon taking a guess at labels to satisfy the milestone munger
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Looks like the PR for this is just awaiting merge

} // update our local observations based on the amount reported to have been reclaimed. // note: this is optimistic, other things could have been still consuming the pressured resource in the interim. <del> signal := resourceToSignal[resourceToReclaim] value, ok := observations[signal] if !ok { glog.Errorf("eviction manager: unable to find value associated with signal %v", signal) continue </del> <ins> for _, signal := range resourceClaimToSignal[resourceToReclaim] { value, ok := observations[signal] if !ok { glog.Errorf("eviction manager: unable to find value associated with signal %v", signal) continue } value.available.Add(*reclaimed) </ins> } <del> value.available.Add(*reclaimed) </del> // evaluate all current thresholds to see if with adjusted observations, we think we have met min reclaim goals if len(thresholdsMet(m.thresholdsMet, observations, true)) == 0 { return true


Question:
It is possible to have multiple signals that point to the same type of resource, e.g., both SignalNodeFsAvailable and SignalAllocatableNodeFsAvailable refer to the same resource NodeFs. Change the map from map[v1.ResourceName]evictionapi.Signal to map[v1.ResourceName][]evictionapi.Signal
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/kind cleanup /priority important/soon taking a guess at labels to satisfy the milestone munger
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Looks like the PR for this is just awaiting merge

} } } <ins> // TestAllocatableNodeFsPressure func TestAllocatableNodeFsPressure(t *testing.T) { podMaker := makePodWithDiskStats summaryStatsMaker := makeDiskStats podsToMake := []podToMake{ {name: "guaranteed-low", requests: newEphemeralStorageResourceList("200Mi", "100m", "1Gi"), limits: newEphemeralStorageResourceList("200Mi", "100m", "1Gi"), rootFsUsed: "200Mi"}, {name: "guaranteed-high", requests: newEphemeralStorageResourceList("800Mi", "100m", "1Gi"), limits: newEphemeralStorageResourceList("800Mi", "100m", "1Gi"), rootFsUsed: "800Mi"}, {name: "burstable-low", requests: newEphemeralStorageResourceList("300Mi", "100m", "100Mi"), limits: newEphemeralStorageResourceList("300Mi", "200m", "1Gi"), logsFsUsed: "300Mi"}, {name: "burstable-high", requests: newEphemeralStorageResourceList("800Mi", "100m", "100Mi"), limits: newEphemeralStorageResourceList("800Mi", "200m", "1Gi"), rootFsUsed: "800Mi"}, {name: "best-effort-low", requests: newEphemeralStorageResourceList("300Mi", "", ""), limits: newEphemeralStorageResourceList("300Mi", "", ""), logsFsUsed: "300Mi"}, {name: "best-effort-high", requests: newEphemeralStorageResourceList("800Mi", "", ""), limits: newEphemeralStorageResourceList("800Mi", "", ""), rootFsUsed: "800Mi"}, } pods := []*v1.Pod{} podStats := map[*v1.Pod]statsapi.PodStats{} for _, podToMake := range podsToMake { pod, podStat := podMaker(podToMake.name, podToMake.requests, podToMake.limits, podToMake.rootFsUsed, podToMake.logsFsUsed, podToMake.perLocalVolumeUsed) pods = append(pods, pod) podStats[pod] = podStat } podToEvict := pods[5] activePodsFunc := func() []*v1.Pod { return pods } fakeClock := clock.NewFakeClock(time.Now()) podKiller := &mockPodKiller{} diskInfoProvider := &mockDiskInfoProvider{dedicatedImageFs: false} capacityProvider := newMockCapacityProvider(newEphemeralStorageResourceList("6Gi", "1000m", "10Gi"), newEphemeralStorageResourceList("1Gi", "1000m", "10Gi")) diskGC := &mockDiskGC{imageBytesFreed: int64(0), err: nil} nodeRef := &v1.ObjectReference{Kind: "Node", Name: "test", UID: types.UID("test"), Namespace: ""} config := Config{ MaxPodGracePeriodSeconds: 5, PressureTransitionPeriod: time.Minute * 5, Thresholds: []evictionapi.Threshold{ { Signal:   evictionapi.SignalAllocatableNodeFsAvailable, Operator: evictionapi.OpLessThan, Value: evictionapi.ThresholdValue{ Quantity: quantityMustParse("1Ki"), }, }, }, } summaryProvider := &fakeSummaryProvider{result: summaryStatsMaker("3Gi", "6Gi", podStats)} manager := &managerImpl{ clock:           fakeClock, killPodFunc:     podKiller.killPodNow, imageGC:         diskGC, containerGC:     diskGC, config:          config, recorder:        &record.FakeRecorder{}, summaryProvider: summaryProvider, nodeRef:         nodeRef, nodeConditionsLastObservedAt: nodeConditionsObservedAt{}, thresholdsFirstObservedAt:    thresholdsObservedAt{}, } // create a best effort pod to test admission bestEffortPodToAdmit, _ := podMaker("best-admit", newEphemeralStorageResourceList("", "", ""), newEphemeralStorageResourceList("", "", ""), "0Gi", "", "") burstablePodToAdmit, _ := podMaker("burst-admit", newEphemeralStorageResourceList("1Gi", "", ""), newEphemeralStorageResourceList("1Gi", "", ""), "1Gi", "", "") // synchronize manager.synchronize(diskInfoProvider, activePodsFunc, capacityProvider) // we should not have disk pressure if manager.IsUnderDiskPressure() { t.Fatalf("Manager should not report disk pressure") } // try to admit our pods (they should succeed) expected := []bool{true, true} for i, pod := range []*v1.Pod{bestEffortPodToAdmit, burstablePodToAdmit} { if result := manager.Admit(&lifecycle.PodAdmitAttributes{Pod: pod}); expected[i] != result.Admit { t.Fatalf("Admit pod: %v, expected: %v, actual: %v", pod, expected[i], result.Admit) } } // induce disk pressure! fakeClock.Step(1 * time.Minute) pod, podStat := podMaker("guaranteed-high-2", newEphemeralStorageResourceList("2000Mi", "100m", "1Gi"), newEphemeralStorageResourceList("2000Mi", "100m", "1Gi"), "2000Mi", "", "") podStats[pod] = podStat pods = append(pods, pod) summaryProvider.result = summaryStatsMaker("6Gi", "6Gi", podStats) manager.synchronize(diskInfoProvider, activePodsFunc, capacityProvider) // we should have disk pressure if !manager.IsUnderDiskPressure() { t.Fatalf("Manager should report disk pressure") } // check the right pod was killed if podKiller.pod != podToEvict { t.Fatalf("Manager chose to kill pod: %v, but should have chosen %v", podKiller.pod, podToEvict.Name) } observedGracePeriod := *podKiller.gracePeriodOverride if observedGracePeriod != int64(0) { t.Fatalf("Manager chose to kill pod with incorrect grace period.  Expected: %d, actual: %d", 0, observedGracePeriod) } // reset state podKiller.pod = nil podKiller.gracePeriodOverride = nil // try to admit our pod (should fail) expected = []bool{false, false} for i, pod := range []*v1.Pod{bestEffortPodToAdmit, burstablePodToAdmit} { if result := manager.Admit(&lifecycle.PodAdmitAttributes{Pod: pod}); expected[i] != result.Admit { t.Fatalf("Admit pod: %v, expected: %v, actual: %v", pod, expected[i], result.Admit) } } // reduce disk pressure fakeClock.Step(1 * time.Minute) pods[5] = pods[len(pods)-1] pods = pods[:len(pods)-1] // we should have disk pressure (because transition period not yet met) if !manager.IsUnderDiskPressure() { t.Fatalf("Manager should report disk pressure") } // try to admit our pod (should fail) expected = []bool{false, false} for i, pod := range []*v1.Pod{bestEffortPodToAdmit, burstablePodToAdmit} { if result := manager.Admit(&lifecycle.PodAdmitAttributes{Pod: pod}); expected[i] != result.Admit { t.Fatalf("Admit pod: %v, expected: %v, actual: %v", pod, expected[i], result.Admit) } } // move the clock past transition period to ensure that we stop reporting pressure fakeClock.Step(5 * time.Minute) manager.synchronize(diskInfoProvider, activePodsFunc, capacityProvider) // we should not have disk pressure (because transition period met) if manager.IsUnderDiskPressure() { t.Fatalf("Manager should not report disk pressure") } // no pod should have been killed if podKiller.pod != nil { t.Fatalf("Manager chose to kill pod: %v when no pod should have been killed", podKiller.pod.Name) } // all pods should admit now expected = []bool{true, true} for i, pod := range []*v1.Pod{bestEffortPodToAdmit, burstablePodToAdmit} { if result := manager.Admit(&lifecycle.PodAdmitAttributes{Pod: pod}); expected[i] != result.Admit { t.Fatalf("Admit pod: %v, expected: %v, actual: %v", pod, expected[i], result.Admit) } } } func TestNodeReclaimForAllocatableFuncs(t *testing.T) { podMaker := makePodWithDiskStats summaryStatsMaker := makeDiskStats podsToMake := []podToMake{ {name: "guaranteed-low", requests: newEphemeralStorageResourceList("200Mi", "100m", "1Gi"), limits: newEphemeralStorageResourceList("200Mi", "100m", "1Gi"), rootFsUsed: "200Mi"}, {name: "guaranteed-high", requests: newEphemeralStorageResourceList("800Mi", "100m", "1Gi"), limits: newEphemeralStorageResourceList("800Mi", "100m", "1Gi"), rootFsUsed: "800Mi"}, {name: "burstable-low", requests: newEphemeralStorageResourceList("300Mi", "100m", "100Mi"), limits: newEphemeralStorageResourceList("300Mi", "200m", "1Gi"), logsFsUsed: "300Mi"}, {name: "burstable-high", requests: newEphemeralStorageResourceList("800Mi", "100m", "100Mi"), limits: newEphemeralStorageResourceList("800Mi", "200m", "1Gi"), rootFsUsed: "800Mi"}, {name: "best-effort-low", requests: newEphemeralStorageResourceList("300Mi", "", ""), limits: newEphemeralStorageResourceList("300Mi", "", ""), logsFsUsed: "300Mi"}, {name: "best-effort-high", requests: newEphemeralStorageResourceList("800Mi", "", ""), limits: newEphemeralStorageResourceList("800Mi", "", ""), rootFsUsed: "800Mi"}, } pods := []*v1.Pod{} podStats := map[*v1.Pod]statsapi.PodStats{} for _, podToMake := range podsToMake { pod, podStat := podMaker(podToMake.name, podToMake.requests, podToMake.limits, podToMake.rootFsUsed, podToMake.logsFsUsed, podToMake.perLocalVolumeUsed) pods = append(pods, pod) podStats[pod] = podStat } podToEvict := pods[5] activePodsFunc := func() []*v1.Pod { return pods } fakeClock := clock.NewFakeClock(time.Now()) podKiller := &mockPodKiller{} diskInfoProvider := &mockDiskInfoProvider{dedicatedImageFs: false} capacityProvider := newMockCapacityProvider(newEphemeralStorageResourceList("6Gi", "1000m", "10Gi"), newEphemeralStorageResourceList("1Gi", "1000m", "10Gi")) imageGcFree := resource.MustParse("800Mi") diskGC := &mockDiskGC{imageBytesFreed: imageGcFree.Value(), err: nil} nodeRef := &v1.ObjectReference{Kind: "Node", Name: "test", UID: types.UID("test"), Namespace: ""} config := Config{ MaxPodGracePeriodSeconds: 5, PressureTransitionPeriod: time.Minute * 5, Thresholds: []evictionapi.Threshold{ { Signal:   evictionapi.SignalAllocatableNodeFsAvailable, Operator: evictionapi.OpLessThan, Value: evictionapi.ThresholdValue{ Quantity: quantityMustParse("10Mi"), }, }, }, } summaryProvider := &fakeSummaryProvider{result: summaryStatsMaker("6Gi", "6Gi", podStats)} manager := &managerImpl{ clock:           fakeClock, killPodFunc:     podKiller.killPodNow, imageGC:         diskGC, containerGC:     diskGC, config:          config, recorder:        &record.FakeRecorder{}, summaryProvider: summaryProvider, nodeRef:         nodeRef, nodeConditionsLastObservedAt: nodeConditionsObservedAt{}, thresholdsFirstObservedAt:    thresholdsObservedAt{}, } // synchronize manager.synchronize(diskInfoProvider, activePodsFunc, capacityProvider) // we should not have disk pressure if manager.IsUnderDiskPressure() { t.Errorf("Manager should not report disk pressure") } // induce hard threshold fakeClock.Step(1 * time.Minute) pod, podStat := podMaker("guaranteed-high-2", newEphemeralStorageResourceList("2000Mi", "100m", "1Gi"), newEphemeralStorageResourceList("2000Mi", "100m", "1Gi"), "2000Mi", "", "") podStats[pod] = podStat pods = append(pods, pod) summaryProvider.result = summaryStatsMaker("6Gi", "6Gi", podStats) manager.synchronize(diskInfoProvider, activePodsFunc, capacityProvider) // we should have disk pressure if !manager.IsUnderDiskPressure() { t.Fatalf("Manager should report disk pressure since soft threshold was met") } // verify image gc was invoked if !diskGC.imageGCInvoked || !diskGC.containerGCInvoked { t.Fatalf("Manager should have invoked image gc") } // verify no pod was killed because image gc was sufficient if podKiller.pod == nil { t.Fatalf("Manager should have killed a pod, but not killed") } // check the right pod was killed if podKiller.pod != podToEvict { t.Fatalf("Manager chose to kill pod: %v, but should have chosen %v", podKiller.pod, podToEvict.Name) } observedGracePeriod := *podKiller.gracePeriodOverride if observedGracePeriod != int64(0) { t.Fatalf("Manager chose to kill pod with incorrect grace period.  Expected: %d, actual: %d", 0, observedGracePeriod) } // reset state diskGC.imageGCInvoked = false diskGC.containerGCInvoked = false podKiller.pod = nil podKiller.gracePeriodOverride = nil // reduce disk pressure fakeClock.Step(1 * time.Minute) pods[5] = pods[len(pods)-1] pods = pods[:len(pods)-1] // we should have disk pressure (because transition period not yet met) if !manager.IsUnderDiskPressure() { t.Fatalf("Manager should report disk pressure") } // move the clock past transition period to ensure that we stop reporting pressure fakeClock.Step(5 * time.Minute) manager.synchronize(diskInfoProvider, activePodsFunc, capacityProvider) // we should not have disk pressure (because transition period met) if manager.IsUnderDiskPressure() { t.Fatalf("Manager should not report disk pressure") } // no pod should have been killed if podKiller.pod != nil { t.Fatalf("Manager chose to kill pod: %v when no pod should have been killed", podKiller.pod.Name) } // no image gc should have occurred if diskGC.imageGCInvoked || diskGC.containerGCInvoked { t.Errorf("Manager chose to perform image gc when it was not neeed") } // no pod should have been killed if podKiller.pod != nil { t.Errorf("Manager chose to kill pod: %v when no pod should have been killed", podKiller.pod.Name) } } </ins>


Question:
It is possible to have multiple signals that point to the same type of resource, e.g., both SignalNodeFsAvailable and SignalAllocatableNodeFsAvailable refer to the same resource NodeFs. Change the map from map[v1.ResourceName]evictionapi.Signal to map[v1.ResourceName][]evictionapi.Signal
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/kind cleanup /priority important/soon taking a guess at labels to satisfy the milestone munger
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Looks like the PR for this is just awaiting merge

signalToNodeCondition map[evictionapi.Signal]v1.NodeConditionType // signalToResource maps a Signal to its associated Resource. signalToResource map[evictionapi.Signal]v1.ResourceName <del> // resourceToSignal maps a Resource to its associated Signal resourceToSignal map[v1.ResourceName]evictionapi.Signal </del> <ins> // resourceClaimToSignal maps a Resource that can be reclaimed to its associated Signal resourceClaimToSignal map[v1.ResourceName][]evictionapi.Signal </ins> ) func init() {


Question:
It is possible to have multiple signals that point to the same type of resource, e.g., both SignalNodeFsAvailable and SignalAllocatableNodeFsAvailable refer to the same resource NodeFs. Change the map from map[v1.ResourceName]evictionapi.Signal to map[v1.ResourceName][]evictionapi.Signal
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/kind cleanup /priority important/soon taking a guess at labels to satisfy the milestone munger
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Looks like the PR for this is just awaiting merge

signalToResource[evictionapi.SignalNodeFsAvailable] = resourceNodeFs signalToResource[evictionapi.SignalNodeFsInodesFree] = resourceNodeFsInodes <del> resourceToSignal = map[v1.ResourceName]evictionapi.Signal{} for key, value := range signalToResource { resourceToSignal[value] = key } // Hard-code here to make sure resourceNodeFs maps to evictionapi.SignalNodeFsAvailable // (TODO) resourceToSignal is a map from resource name to a list of signals resourceToSignal[resourceNodeFs] = evictionapi.SignalNodeFsAvailable </del> <ins> // maps resource to signals (the following resource could be reclaimed) resourceClaimToSignal = map[v1.ResourceName][]evictionapi.Signal{} resourceClaimToSignal[resourceNodeFs] = []evictionapi.Signal{evictionapi.SignalNodeFsAvailable} resourceClaimToSignal[resourceImageFs] = []evictionapi.Signal{evictionapi.SignalImageFsAvailable} resourceClaimToSignal[resourceNodeFsInodes] = []evictionapi.Signal{evictionapi.SignalNodeFsInodesFree} resourceClaimToSignal[resourceImageFsInodes] = []evictionapi.Signal{evictionapi.SignalImageFsInodesFree} </ins> } // validSignal returns true if the signal is supported.


Question:
It is possible to have multiple signals that point to the same type of resource, e.g., both SignalNodeFsAvailable and SignalAllocatableNodeFsAvailable refer to the same resource NodeFs. Change the map from map[v1.ResourceName]evictionapi.Signal to map[v1.ResourceName][]evictionapi.Signal
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/kind cleanup /priority important/soon taking a guess at labels to satisfy the milestone munger
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Looks like the PR for this is just awaiting merge

return res } <ins> func newEphemeralStorageResourceList(ephemeral, cpu, memory string) v1.ResourceList { res := v1.ResourceList{} if ephemeral != "" { res[v1.ResourceEphemeralStorage] = resource.MustParse(ephemeral) } if cpu != "" { res[v1.ResourceCPU] = resource.MustParse(cpu) } if memory != "" { res[v1.ResourceMemory] = resource.MustParse("1Mi") } return res } </ins> func newResourceRequirements(requests, limits v1.ResourceList) v1.ResourceRequirements { res := v1.ResourceRequirements{} res.Requests = requests


Question: Currently, always returns with 0 exit code no matter the test fails or not. We should fix it to return non-zero exit code for test failure.

ginkgo := filepath.Join(outputDir, "ginkgo") test := filepath.Join(outputDir, "e2e_node.test") <del> if *systemSpecName == "" { runCommand(ginkgo, *ginkgoFlags, test, "--", *testFlags) return </del> <ins> args := []string{*ginkgoFlags, test, "--", *testFlags} if *systemSpecName != "" { rootDir, err := builder.GetK8sRootDir() if err != nil { glog.Fatalf("Failed to get k8s root directory: %v", err) } systemSpecFile := filepath.Join(rootDir, systemSpecPath, *systemSpecName+".yaml") args = append(args, fmt.Sprintf("--system-spec-name=%s --system-spec-file=%s", *systemSpecName, systemSpecFile)) </ins> } <del> rootDir, err := builder.GetK8sRootDir() if err != nil { glog.Fatalf("Failed to get k8s root directory: %v", err) </del> <ins> if err := runCommand(ginkgo, args...); err != nil { glog.Exitf("Test failed: %v", err) </ins> } <del> systemSpecFile := filepath.Join(rootDir, systemSpecPath, *systemSpecName+".yaml") runCommand(ginkgo, *ginkgoFlags, test, "--", fmt.Sprintf("--system-spec-name=%s --system-spec-file=%s", *systemSpecName, systemSpecFile), *testFlags) </del> return }


Question:
When pod is evicted, pod is terminated. After this change , attach/detach controller should be able to detach the pod correctly so that new pod which uses the same volume will start up. We need to have an e2e test for this scenario.
what environment is good for this e2e test? GCE or GKE or AWS? I see we have which has func, Can you comment on why is insufficient to cover this issue?
does the pod need to be evicted? Or would be ok to delete the pod and then check that the PD is detached. Then create a new pod using the same PD and test that it attaches?
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.9 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ping ^^
sorry for the delay. Here we want to test the detach behavior of evicted pods. So delete a pod directly is different from it. When Pod is evicted, the containers will be killed and the pod status is terminated. But the pod api object is not deleted from API server. Before PR , such evicted pods will not trigger detach operation because the pod api object is not deleted from api server. So the purpose of the test is to make sure detach operation is triggered for such evicted pods.
Thanks Do you have a suggestion for how to force a pod to be evicted?
You could have the pod do some memory intensive application
ah, like my use of in the io-streaming pr :-)
- Here's another option to cause eviction: .

"//vendor/google.golang.org/api/googleapi:go_default_library", "//vendor/k8s.io/api/batch/v1:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", <ins> "//vendor/k8s.io/api/policy/v1beta1:go_default_library", </ins> "//vendor/k8s.io/api/rbac/v1beta1:go_default_library", "//vendor/k8s.io/api/storage/v1:go_default_library", "//vendor/k8s.io/api/storage/v1beta1:go_default_library",


Question:
When pod is evicted, pod is terminated. After this change , attach/detach controller should be able to detach the pod correctly so that new pod which uses the same volume will start up. We need to have an e2e test for this scenario.
what environment is good for this e2e test? GCE or GKE or AWS? I see we have which has func, Can you comment on why is insufficient to cover this issue?
does the pod need to be evicted? Or would be ok to delete the pod and then check that the PD is detached. Then create a new pod using the same PD and test that it attaches?
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.9 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ping ^^
sorry for the delay. Here we want to test the detach behavior of evicted pods. So delete a pod directly is different from it. When Pod is evicted, the containers will be killed and the pod status is terminated. But the pod api object is not deleted from API server. Before PR , such evicted pods will not trigger detach operation because the pod api object is not deleted from api server. So the purpose of the test is to make sure detach operation is triggered for such evicted pods.
Thanks Do you have a suggestion for how to force a pod to be evicted?
You could have the pod do some memory intensive application
ah, like my use of in the io-streaming pr :-)
- Here's another option to cause eviction: .

. "github.com/onsi/ginkgo" . "github.com/onsi/gomega" "k8s.io/api/core/v1" <ins> policy "k8s.io/api/policy/v1beta1" </ins> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/uuid" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> clientset "k8s.io/client-go/kubernetes" v1core "k8s.io/client-go/kubernetes/typed/core/v1" "k8s.io/kubernetes/pkg/api/testapi"


Question:
When pod is evicted, pod is terminated. After this change , attach/detach controller should be able to detach the pod correctly so that new pod which uses the same volume will start up. We need to have an e2e test for this scenario.
what environment is good for this e2e test? GCE or GKE or AWS? I see we have which has func, Can you comment on why is insufficient to cover this issue?
does the pod need to be evicted? Or would be ok to delete the pod and then check that the PD is detached. Then create a new pod using the same PD and test that it attaches?
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.9 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ping ^^
sorry for the delay. Here we want to test the detach behavior of evicted pods. So delete a pod directly is different from it. When Pod is evicted, the containers will be killed and the pod status is terminated. But the pod api object is not deleted from API server. Before PR , such evicted pods will not trigger detach operation because the pod api object is not deleted from api server. So the purpose of the test is to make sure detach operation is triggered for such evicted pods.
Thanks Do you have a suggestion for how to force a pod to be evicted?
You could have the pod do some memory intensive application
ah, like my use of in the io-streaming pr :-)
- Here's another option to cause eviction: .

gcePDDetachPollTime = 10 * time.Second nodeStatusTimeout   = 10 * time.Minute nodeStatusPollTime  = 1 * time.Second <ins> podEvictTimeout     = 2 * time.Minute </ins> maxReadRetry        = 3 minNodes            = 2 ) var _ = SIGDescribe("Pod Disks", func() { var ( <ins> ns         string cs         clientset.Interface </ins> podClient  v1core.PodInterface nodeClient v1core.NodeInterface host0Name  types.NodeName


Question:
When pod is evicted, pod is terminated. After this change , attach/detach controller should be able to detach the pod correctly so that new pod which uses the same volume will start up. We need to have an e2e test for this scenario.
what environment is good for this e2e test? GCE or GKE or AWS? I see we have which has func, Can you comment on why is insufficient to cover this issue?
does the pod need to be evicted? Or would be ok to delete the pod and then check that the PD is detached. Then create a new pod using the same PD and test that it attaches?
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.9 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ping ^^
sorry for the delay. Here we want to test the detach behavior of evicted pods. So delete a pod directly is different from it. When Pod is evicted, the containers will be killed and the pod status is terminated. But the pod api object is not deleted from API server. Before PR , such evicted pods will not trigger detach operation because the pod api object is not deleted from api server. So the purpose of the test is to make sure detach operation is triggered for such evicted pods.
Thanks Do you have a suggestion for how to force a pod to be evicted?
You could have the pod do some memory intensive application
ah, like my use of in the io-streaming pr :-)
- Here's another option to cause eviction: .

BeforeEach(func() { framework.SkipUnlessNodeCountIsAtLeast(minNodes) <ins> cs = f.ClientSet ns = f.Namespace.Name </ins> <del> podClient = f.ClientSet.Core().Pods(f.Namespace.Name) nodeClient = f.ClientSet.Core().Nodes() nodes = framework.GetReadySchedulableNodesOrDie(f.ClientSet) </del> <ins> podClient = cs.Core().Pods(ns) nodeClient = cs.Core().Nodes() nodes = framework.GetReadySchedulableNodesOrDie(cs) </ins> Expect(len(nodes.Items)).To(BeNumerically(">=", minNodes), fmt.Sprintf("Requires at least %d nodes", minNodes)) host0Name = types.NodeName(nodes.Items[0].ObjectMeta.Name) host1Name = types.NodeName(nodes.Items[1].ObjectMeta.Name)


Question:
When pod is evicted, pod is terminated. After this change , attach/detach controller should be able to detach the pod correctly so that new pod which uses the same volume will start up. We need to have an e2e test for this scenario.
what environment is good for this e2e test? GCE or GKE or AWS? I see we have which has func, Can you comment on why is insufficient to cover this issue?
does the pod need to be evicted? Or would be ok to delete the pod and then check that the PD is detached. Then create a new pod using the same PD and test that it attaches?
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.9 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ping ^^
sorry for the delay. Here we want to test the detach behavior of evicted pods. So delete a pod directly is different from it. When Pod is evicted, the containers will be killed and the pod status is terminated. But the pod api object is not deleted from API server. Before PR , such evicted pods will not trigger detach operation because the pod api object is not deleted from api server. So the purpose of the test is to make sure detach operation is triggered for such evicted pods.
Thanks Do you have a suggestion for how to force a pod to be evicted?
You could have the pod do some memory intensive application
ah, like my use of in the io-streaming pr :-)
- Here's another option to cause eviction: .

} }) <del> Context("detach from a disrupted node [Slow] [Disruptive]", func() { </del> <ins> Context("detach in a disrupted environment [Slow] [Disruptive]", func() { </ins> const ( deleteNode    = 1 // delete physical node deleteNodeObj = 2 // delete node's api object only <ins> evictPod      = 3 // evict host0Pod on node0 </ins> ) type testT struct { <del> descr  string // It description nodeOp int    // disruptive operation performed on target node </del> <ins> descr     string // It description disruptOp int    // disruptive operation performed on target node </ins> } tests := []testT{ { <del> descr:  "node is deleted", nodeOp: deleteNode, </del> <ins> descr:     "node is deleted", disruptOp: deleteNode, </ins> }, { <del> descr:  "node's API object is deleted", nodeOp: deleteNodeObj, </del> <ins> descr:     "node's API object is deleted", disruptOp: deleteNodeObj, }, { descr:     "pod is evicted", disruptOp: evictPod, </ins> }, } for _, t := range tests { <del> nodeOp := t.nodeOp </del> <ins> disruptOp := t.disruptOp </ins> It(fmt.Sprintf("when %s", t.descr), func() { framework.SkipUnlessProviderIs("gce") origNodeCnt := len(nodes.Items) // healhy nodes running kublet


Question:
When pod is evicted, pod is terminated. After this change , attach/detach controller should be able to detach the pod correctly so that new pod which uses the same volume will start up. We need to have an e2e test for this scenario.
what environment is good for this e2e test? GCE or GKE or AWS? I see we have which has func, Can you comment on why is insufficient to cover this issue?
does the pod need to be evicted? Or would be ok to delete the pod and then check that the PD is detached. Then create a new pod using the same PD and test that it attaches?
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.9 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ping ^^
sorry for the delay. Here we want to test the detach behavior of evicted pods. So delete a pod directly is different from it. When Pod is evicted, the containers will be killed and the pod status is terminated. But the pod api object is not deleted from API server. Before PR , such evicted pods will not trigger detach operation because the pod api object is not deleted from api server. So the purpose of the test is to make sure detach operation is triggered for such evicted pods.
Thanks Do you have a suggestion for how to force a pod to be evicted?
You could have the pod do some memory intensive application
ah, like my use of in the io-streaming pr :-)
- Here's another option to cause eviction: .

diskName, err := framework.CreatePDWithRetry() framework.ExpectNoError(err, "Error creating a pd") <del> targetNode := &nodes.Items[0] </del> <ins> targetNode := &nodes.Items[0] // for node delete ops </ins> host0Pod := testPDPod([]string{diskName}, host0Name, false, 1) containerName := "mycontainer" defer func() { By("defer: cleaning up PD-RW test env") framework.Logf("defer cleanup errors can usually be ignored") <del> if nodeOp == deleteNode { podClient.Delete(host0Pod.Name, metav1.NewDeleteOptions(0)) } </del> <ins> By("defer: delete host0Pod") podClient.Delete(host0Pod.Name, metav1.NewDeleteOptions(0)) By("defer: detach and delete PDs") </ins> detachAndDeletePDs(diskName, []types.NodeName{host0Name}) <del> if nodeOp == deleteNodeObj { targetNode.ObjectMeta.SetResourceVersion("0") // need to set the resource version or else the Create() fails _, err := nodeClient.Create(targetNode) framework.ExpectNoError(err, "defer: Unable to re-create the deleted node") </del> <ins> if disruptOp == deleteNode || disruptOp == deleteNodeObj { if disruptOp == deleteNodeObj { targetNode.ObjectMeta.SetResourceVersion("0") // need to set the resource version or else the Create() fails By("defer: re-create host0 node object") _, err := nodeClient.Create(targetNode) framework.ExpectNoError(err, fmt.Sprintf("defer: Unable to re-create the deleted node object %q", targetNode.Name)) } By("defer: verify the number of ready nodes") numNodes := countReadyNodes(cs, host0Name) // if this defer is reached due to an Expect then nested // Expects are lost, so use Failf here if numNodes != origNodeCnt { framework.Failf("defer: Requires current node count (%d) to return to original node count (%d)", numNodes, origNodeCnt) } </ins> } <del> numNodes := countReadyNodes(f.ClientSet, host0Name) Expect(numNodes).To(Equal(origNodeCnt), fmt.Sprintf("defer: Requires current node count (%d) to return to original node count (%d)", numNodes, origNodeCnt)) </del> }() By("creating host0Pod on node0") _, err = podClient.Create(host0Pod) framework.ExpectNoError(err, fmt.Sprintf("Failed to create host0Pod: %v", err)) <ins> By("waiting for host0Pod to be running") </ins> framework.ExpectNoError(f.WaitForPodRunningSlow(host0Pod.Name)) By("writing content to host0Pod")


Question:
When pod is evicted, pod is terminated. After this change , attach/detach controller should be able to detach the pod correctly so that new pod which uses the same volume will start up. We need to have an e2e test for this scenario.
what environment is good for this e2e test? GCE or GKE or AWS? I see we have which has func, Can you comment on why is insufficient to cover this issue?
does the pod need to be evicted? Or would be ok to delete the pod and then check that the PD is detached. Then create a new pod using the same PD and test that it attaches?
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.9 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ping ^^
sorry for the delay. Here we want to test the detach behavior of evicted pods. So delete a pod directly is different from it. When Pod is evicted, the containers will be killed and the pod status is terminated. But the pod api object is not deleted from API server. Before PR , such evicted pods will not trigger detach operation because the pod api object is not deleted from api server. So the purpose of the test is to make sure detach operation is triggered for such evicted pods.
Thanks Do you have a suggestion for how to force a pod to be evicted?
You could have the pod do some memory intensive application
ah, like my use of in the io-streaming pr :-)
- Here's another option to cause eviction: .

By("verifying PD is present in node0's VolumeInUse list") framework.ExpectNoError(waitForPDInVolumesInUse(nodeClient, diskName, host0Name, nodeStatusTimeout, true /* should exist*/)) <del> if nodeOp == deleteNode { </del> <ins> if disruptOp == deleteNode { </ins> By("getting gce instances") gceCloud, err := framework.GetGCECloud() framework.ExpectNoError(err, fmt.Sprintf("Unable to create gcloud client err=%v", err))


Question:
When pod is evicted, pod is terminated. After this change , attach/detach controller should be able to detach the pod correctly so that new pod which uses the same volume will start up. We need to have an e2e test for this scenario.
what environment is good for this e2e test? GCE or GKE or AWS? I see we have which has func, Can you comment on why is insufficient to cover this issue?
does the pod need to be evicted? Or would be ok to delete the pod and then check that the PD is detached. Then create a new pod using the same PD and test that it attaches?
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.9 milestone for more than 3 days: kind: Must specify exactly one of , or . <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
ping ^^
sorry for the delay. Here we want to test the detach behavior of evicted pods. So delete a pod directly is different from it. When Pod is evicted, the containers will be killed and the pod status is terminated. But the pod api object is not deleted from API server. Before PR , such evicted pods will not trigger detach operation because the pod api object is not deleted from api server. So the purpose of the test is to make sure detach operation is triggered for such evicted pods.
Thanks Do you have a suggestion for how to force a pod to be evicted?
You could have the pod do some memory intensive application
ah, like my use of in the io-streaming pr :-)
- Here's another option to cause eviction: .

By("deleting host0") resp, err := gceCloud.DeleteInstance(framework.TestContext.CloudConfig.ProjectID, framework.TestContext.CloudConfig.Zone, string(host0Name)) framework.ExpectNoError(err, fmt.Sprintf("Failed to delete host0Pod: err=%v response=%#v", err, resp)) <del> By("expecting host0 node to be recreated") numNodes := countReadyNodes(f.ClientSet, host0Name) </del> <ins> By("expecting host0 node to be re-created") numNodes := countReadyNodes(cs, host0Name) </ins> Expect(numNodes).To(Equal(origNodeCnt), fmt.Sprintf("Requires current node count (%d) to return to original node count (%d)", numNodes, origNodeCnt)) output, err = gceCloud.ListInstanceNames(framework.TestContext.CloudConfig.ProjectID, framework.TestContext.CloudConfig.Zone) framework.ExpectNoError(err, fmt.Sprintf("Unable to get list of node instances err=%v output=%s", err, output)) Expect(false, strings.Contains(string(output), string(host0Name))) <del> } else if nodeOp == deleteNodeObj { </del> <ins> } else if disruptOp == deleteNodeObj { </ins> By("deleting host0's node api object") framework.ExpectNoError(nodeClient.Delete(string(host0Name), metav1.NewDeleteOptions(0)), "Unable to delete host0's node object") By("deleting host0Pod") framework.ExpectNoError(podClient.Delete(host0Pod.Name, metav1.NewDeleteOptions(0)), "Unable to delete host0Pod") <ins> } else if disruptOp == evictPod { evictTarget := &policy.Eviction{ ObjectMeta: metav1.ObjectMeta{ Name:      host0Pod.Name, Namespace: ns, }, } By("evicting host0Pod") err = wait.PollImmediate(framework.Poll, podEvictTimeout, func() (bool, error) { err = cs.CoreV1().Pods(ns).Evict(evictTarget) if err != nil { return false, nil } else { return true, nil } }) Expect(err).NotTo(HaveOccurred(), fmt.Sprintf("failed to evict host0Pod after %v", podEvictTimeout)) </ins> } By("waiting for pd to detach from host0")


Question:
/kind bug What happened: When a cluster is provisioned using a containerised kubelet using the following images: Pods with PVCs are not created because kubelet cannot mount the disks. If you look at events, you can see that it's missing some binaries: This executable is . This problem can be resolved by mounting paths from the host into the rkt container, specifically and . Is this expected? mentioned in , that mounting operations are moving out from the main process into containers. Is this part of that effort? This caught me quite unexpected, and I didn't find any docs or changelog entries which indicated additional host mounts were necessary. If it's deliberate, I can close this until we perhaps document the breaking change. Is there a good place where we can document this? If it's not, I'll leave this open as a tracking issue. What you expected to happen: The hyperkube image to work without lots of host mounts. How to reproduce it (as minimally and precisely as possible): Run kubelet with rkt in one of the above listed images (see CoreOS' ). Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OpenStack OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: tectonic-installer
/sig storage /sig node
Seems the base hyperkube image was changed: done in /cc
The pkg/util/mount/mount_linux source defaults to ext4 if a fstype is not selected. It makes sense to restore the e2fsprogs package within the hyperkube image.
cc cc cc This seems like something that should be in the 1.8 milestone. Can someone who owns the hyperkube image comment on that?
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Containerization of mount is not going to happen until 1.9 at the earliest. So if hyperkube is broken, it should be fixed before ship.
"modprobe" also went missing in 1.8 hyperkube images... Both kubelet and kube-proxy need it.
Are you noticing any buggy behaviour on those components without it? Would you mind creating an issue and/or PR to restore that dep
Done: And done: :)

RUN echo CACHEBUST>/dev/null && clean-install  iptables  <ins> e2fsprogs  </ins> ebtables  ethtool  ca-certificates


Question:
/kind bug What happened: When a cluster is provisioned using a containerised kubelet using the following images: Pods with PVCs are not created because kubelet cannot mount the disks. If you look at events, you can see that it's missing some binaries: This executable is . This problem can be resolved by mounting paths from the host into the rkt container, specifically and . Is this expected? mentioned in , that mounting operations are moving out from the main process into containers. Is this part of that effort? This caught me quite unexpected, and I didn't find any docs or changelog entries which indicated additional host mounts were necessary. If it's deliberate, I can close this until we perhaps document the breaking change. Is there a good place where we can document this? If it's not, I'll leave this open as a tracking issue. What you expected to happen: The hyperkube image to work without lots of host mounts. How to reproduce it (as minimally and precisely as possible): Run kubelet with rkt in one of the above listed images (see CoreOS' ). Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OpenStack OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: tectonic-installer
/sig storage /sig node
Seems the base hyperkube image was changed: done in /cc
The pkg/util/mount/mount_linux source defaults to ext4 if a fstype is not selected. It makes sense to restore the e2fsprogs package within the hyperkube image.
cc cc cc This seems like something that should be in the 1.8 milestone. Can someone who owns the hyperkube image comment on that?
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Containerization of mount is not going to happen until 1.9 at the earliest. So if hyperkube is broken, it should be fixed before ship.
"modprobe" also went missing in 1.8 hyperkube images... Both kubelet and kube-proxy need it.
Are you noticing any buggy behaviour on those components without it? Would you mind creating an issue and/or PR to restore that dep
Done: And done: :)

REGISTRY?=gcr.io/google-containers IMAGE?=debian-hyperkube-base <del> TAG=0.2 </del> <ins> TAG=0.3 </ins> ARCH?=amd64 CACHEBUST?=1


Question:
/kind bug What happened: When a cluster is provisioned using a containerised kubelet using the following images: Pods with PVCs are not created because kubelet cannot mount the disks. If you look at events, you can see that it's missing some binaries: This executable is . This problem can be resolved by mounting paths from the host into the rkt container, specifically and . Is this expected? mentioned in , that mounting operations are moving out from the main process into containers. Is this part of that effort? This caught me quite unexpected, and I didn't find any docs or changelog entries which indicated additional host mounts were necessary. If it's deliberate, I can close this until we perhaps document the breaking change. Is there a good place where we can document this? If it's not, I'll leave this open as a tracking issue. What you expected to happen: The hyperkube image to work without lots of host mounts. How to reproduce it (as minimally and precisely as possible): Run kubelet with rkt in one of the above listed images (see CoreOS' ). Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OpenStack OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: tectonic-installer
/sig storage /sig node
Seems the base hyperkube image was changed: done in /cc
The pkg/util/mount/mount_linux source defaults to ext4 if a fstype is not selected. It makes sense to restore the e2fsprogs package within the hyperkube image.
cc cc cc This seems like something that should be in the 1.8 milestone. Can someone who owns the hyperkube image comment on that?
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Containerization of mount is not going to happen until 1.9 at the earliest. So if hyperkube is broken, it should be fixed before ship.
"modprobe" also went missing in 1.8 hyperkube images... Both kubelet and kube-proxy need it.
Are you noticing any buggy behaviour on those components without it? Would you mind creating an issue and/or PR to restore that dep
Done: And done: :)

ARCH?=amd64 HYPERKUBE_BIN?=_output/dockerized/bin/linux/$(ARCH)/hyperkube <del> BASEIMAGE=gcr.io/google-containers/debian-hyperkube-base-$(ARCH):0.2 </del> <ins> BASEIMAGE=gcr.io/google-containers/debian-hyperkube-base-$(ARCH):0.3 </ins> TEMP_DIR:=$(shell mktemp -d -t hyperkubeXXXXXX) all: build


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: my yaml: exec kubectl set resources command,just update first object: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind bug
/assign

} if len(o.Output) > 0 { <del> return o.PrintObject(o.Cmd, o.Local, o.Mapper, obj, o.Out) </del> <ins> if err := o.PrintObject(o.Cmd, o.Local, o.Mapper, obj, o.Out); err != nil { return err } continue </ins> } cmdutil.PrintSuccess(o.Mapper, o.ShortOutput, o.Out, info.Mapping.Resource, info.Name, false, "env updated")


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: my yaml: exec kubectl set resources command,just update first object: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind bug
/assign

t.Errorf("did not set env: %s", buf.String()) } } <ins> func TestSetMultiResourcesEnvLocal(t *testing.T) { f, tf, codec, ns := cmdtesting.NewAPIFactory() tf.Client = &fake.RESTClient{ GroupVersion:         legacyscheme.Registry.GroupOrDie(api.GroupName).GroupVersion, NegotiatedSerializer: ns, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { t.Fatalf("unexpected request: %s %#vn%#v", req.Method, req.URL, req) return nil, nil }), } tf.Namespace = "test" tf.ClientConfig = &restclient.Config{ContentConfig: restclient.ContentConfig{GroupVersion: &legacyscheme.Registry.GroupOrDie(api.GroupName).GroupVersion}} buf := bytes.NewBuffer([]byte{}) cmd := NewCmdEnv(f, os.Stdin, buf, buf) cmd.SetOutput(buf) cmd.Flags().Set("output", "name") cmd.Flags().Set("local", "true") mapper, typer := f.Object() tf.Printer = &printers.NamePrinter{Decoders: []runtime.Decoder{codec}, Typer: typer, Mapper: mapper} opts := EnvOptions{FilenameOptions: resource.FilenameOptions{ Filenames: []string{"../../../../test/fixtures/pkg/kubectl/cmd/set/multi-resource-yaml.yaml"}}, Out:   buf, Local: true} err := opts.Complete(f, cmd, []string{"env=prod"}) if err == nil { err = opts.RunEnv(f) } if err != nil { t.Fatalf("unexpected error: %v", err) } expectedOut := "replicationcontrollers/first-rcnreplicationcontrollers/second-rcn" if buf.String() != expectedOut { t.Errorf("expected out:n%snbut got:n%s", expectedOut, buf.String()) } } </ins>


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: my yaml: exec kubectl set resources command,just update first object: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind bug
/assign

} } } <ins> func TestSetMultiResourcesImageLocal(t *testing.T) { f, tf, codec, ns := cmdtesting.NewAPIFactory() tf.Client = &fake.RESTClient{ GroupVersion:         legacyscheme.Registry.GroupOrDie(api.GroupName).GroupVersion, NegotiatedSerializer: ns, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { t.Fatalf("unexpected request: %s %#vn%#v", req.Method, req.URL, req) return nil, nil }), } tf.Namespace = "test" tf.ClientConfig = &restclient.Config{ContentConfig: restclient.ContentConfig{GroupVersion: &legacyscheme.Registry.GroupOrDie(api.GroupName).GroupVersion}} buf := bytes.NewBuffer([]byte{}) cmd := NewCmdImage(f, buf, buf) cmd.SetOutput(buf) cmd.Flags().Set("output", "name") cmd.Flags().Set("local", "true") mapper, typer := f.Object() tf.Printer = &printers.NamePrinter{Decoders: []runtime.Decoder{codec}, Typer: typer, Mapper: mapper} opts := ImageOptions{FilenameOptions: resource.FilenameOptions{ Filenames: []string{"../../../../test/fixtures/pkg/kubectl/cmd/set/multi-resource-yaml.yaml"}}, Out:   buf, Local: true} err := opts.Complete(f, cmd, []string{"*=thingy"}) if err == nil { err = opts.Validate() } if err == nil { err = opts.Run() } if err != nil { t.Fatalf("unexpected error: %v", err) } expectedOut := "replicationcontrollers/first-rcnreplicationcontrollers/second-rcn" if buf.String() != expectedOut { t.Errorf("expected out:n%snbut got:n%s", expectedOut, buf.String()) } } </ins>


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: my yaml: exec kubectl set resources command,just update first object: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind bug
/assign

} if o.Local || cmdutil.GetDryRunFlag(o.Cmd) { <del> return o.PrintObject(o.Cmd, o.Local, o.Mapper, info.Object, o.Out) </del> <ins> if err := o.PrintObject(o.Cmd, o.Local, o.Mapper, info.Object, o.Out); err != nil { return err } continue </ins> } obj, err := resource.NewHelper(info.Client, info.Mapping).Patch(info.Namespace, info.Name, types.StrategicMergePatchType, patch.Patch)


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: my yaml: exec kubectl set resources command,just update first object: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind bug
/assign

t.Errorf("did not set resources: %s", buf.String()) } } <ins> func TestSetMultiResourcesLimitsLocal(t *testing.T) { f, tf, codec, ns := cmdtesting.NewAPIFactory() tf.Client = &fake.RESTClient{ GroupVersion:         legacyscheme.Registry.GroupOrDie(api.GroupName).GroupVersion, NegotiatedSerializer: ns, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { t.Fatalf("unexpected request: %s %#vn%#v", req.Method, req.URL, req) return nil, nil }), } tf.Namespace = "test" tf.ClientConfig = &restclient.Config{ContentConfig: restclient.ContentConfig{GroupVersion: &legacyscheme.Registry.GroupOrDie(api.GroupName).GroupVersion}} buf := bytes.NewBuffer([]byte{}) cmd := NewCmdResources(f, buf, buf) cmd.SetOutput(buf) cmd.Flags().Set("output", "name") cmd.Flags().Set("local", "true") mapper, typer := f.Object() tf.Printer = &printers.NamePrinter{Decoders: []runtime.Decoder{codec}, Typer: typer, Mapper: mapper} opts := ResourcesOptions{FilenameOptions: resource.FilenameOptions{ Filenames: []string{"../../../../test/fixtures/pkg/kubectl/cmd/set/multi-resource-yaml.yaml"}}, Out:               buf, Local:             true, Limits:            "cpu=200m,memory=512Mi", Requests:          "cpu=200m,memory=512Mi", ContainerSelector: "*"} err := opts.Complete(f, cmd, []string{}) if err == nil { err = opts.Validate() } if err == nil { err = opts.Run() } if err != nil { t.Fatalf("unexpected error: %v", err) } expectedOut := "replicationcontrollers/first-rcnreplicationcontrollers/second-rcn" if buf.String() != expectedOut { t.Errorf("expected out:n%snbut got:n%s", expectedOut, buf.String()) } } </ins>


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: my yaml: exec kubectl set resources command,just update first object: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind bug
/assign

} if o.Local || o.DryRun { <del> return o.PrintObject(o.Mapper, info.Object, o.Out) </del> <ins> if err := o.PrintObject(o.Mapper, info.Object, o.Out); err != nil { return err } continue </ins> } obj, err := resource.NewHelper(info.Client, info.Mapping).Patch(info.Namespace, info.Name, types.StrategicMergePatchType, patch.Patch)


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: my yaml: exec kubectl set resources command,just update first object: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind bug
/assign

<ins> apiVersion: v1 kind: ReplicationController metadata: name: first-rc spec: replicas: 1 selector: app: mock template: metadata: labels: app: mock spec: containers: - name: mock-container image: gcr.io/google-containers/pause:2.0 --- apiVersion: v1 kind: ReplicationController metadata: name: second-rc spec: replicas: 1 selector: app: mock template: metadata: labels: app: mock spec: containers: - name: mock-container image: gcr.io/google-containers/pause:2.0 </ins>  No newline at end of file