Question: The PUT and POST verbs on objects will ignore the "status" values. Otherwise, PUT expects the whole object to be specified. Therefore, if a field is omitted it is assumed that the client wants to clear that field's value. <del> Modification of just part of an object may be achieved by GETting the resource, modifying part of the spec, labels, or annotations, and then PUTting it back. See [concurrency control](#concurrency-control-and-consistency), below, regarding read-modify-write consistency when using this pattern. Some objects may expose alternative resource representations that allow mutation of the status, or performing custom actions on the object. </del> <ins> The PUT verb does not accept partial updates. Modification of just part of an object may be achieved by GETting the resource, modifying part of the spec, labels, or annotations, and then PUTting it back. See [concurrency control](#concurrency-control-and-consistency), below, regarding read-modify-write consistency when using this pattern. Some objects may expose alternative resource representations that allow mutation of the status, or performing custom actions on the object. </ins> All objects that represent a physical resource whose state may vary from the user's desired intent SHOULD have a "spec" and a "status".  Objects whose state cannot vary from the user's desired intent MAY have only "spec", and MAY rename "spec" to a more appropriate name.

sending the following json to update a service: "{"apiVersion":"v1beta1","port":6380,"containerPort":6379}" fails with 422 and message: "service "" is invalid: name: required value ''" It's not clear from the message what is exactly the problem. is the "name" field missing? isn't it called "id" at the moment? if yes, another issue () points on the fact that including the id fails the request with another error.
similar issue happens when creating a service: input (based on the guestbook example) : POST fails with :
Internal names are not converted to external during rendering of the status message. Name is the newer api version field name for what used to be called id
but the input json contains "id", so why does it fail in the first place?
I'm referring to the first comment, where update failed because you omitted the ID of the service you were updating
Update requires the full resource to be specified. GET the resource, modify the fields you want to change, and do a PUT.
Using for the field name conversion problem.
As mentioned, PUT should not accept partial updates - PATCH should, and the issue for that is . This is documented in : "Otherwise, PUT expects the whole object to be specified." Where else does it need to be documented?


Question: cases := map[string]struct { pod        *v1.Pod pvcs       []*testPVC <ins> pvs        []*testPV </ins> trigger    func(config *testConfig) shouldFail bool }{

Add test for "on PVC add", see Test for "on PVC update" is not needed, because it has already been tested in all volume scheduling tests.
/sig
docs:


Question: package gce import ( <del> "errors" </del> "fmt" "strings"

Is this a BUG REPORT or FEATURE REQUEST?: As part of the initiative for external cloud providers (), we GetZoneByProviderID and GetZoneByNodeName. See for more details. The previous PR did not actually implement those methods for existing cloud providers in . These methods should be implemented by all cloud providers in the future. cc
Could someone add the label and the label for this issue please?
/sig cluster-lifecycle
/area cloudprovider
/assign wlan0
cc We need the following label - help-wanted and add it to the 1.8(?) milestone cc to implement it for their respective clouds.
I'll try to have a look by the end of the week.
Ok, I will take care of the Rackspace and OpenStack.
Thank you !
- Is there any listing where to see the already implemented cloud provider ?
No providers have this yet. Current PRs: Openstack: Rackspace: Please contribute if you have something ready :)
Sorry for the delay guys :)
Have you implemented them for gce?
Intending to work on it tonight, but if you have something I can work on azure
Yes, I have prepared for GCE.
Azure .
Cool, than you all! Any other providers we should implement this for in time for v1.8?
I'll take a look tomorrow to see if there's something left
Let's see which providers are left before closing this...
- I guess this one should be closed ?
If you say that all providers have this now, thanks, great!


Question: // features that enable backwards compatibility but are scheduled to be removed // ... <del> HPAScaleToZero:         {Default: false, PreRelease: featuregate.Alpha}, LegacyNodeRoleBehavior: {Default: false, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.22 </del> <ins> HPAScaleToZero: {Default: false, PreRelease: featuregate.Alpha}, </ins> }

What is missing? Based on SIG call it's probably "tests" but I am not sure. Could use some investigation and followup - how do we get it to GA? /sig network /kind cleanup
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
This one seems not a good first one, I'll take this to myself and start bothering people in Slack to check what's missing, mostly as I've seen this is also related to cloud providers :)
I think this one originally. According to the KEP in This should have went GA in v.19 and be removed in v1.20. Given it's still marked beta, at the very least we should mark it GA in v1.20 and remove it in v1.21. anything I'm missing here?
yeap. Honest question, changing a Feature Gate to GA process is only changing the related line in and also putting a comment that this goes to GA? If so, probably this one can also be a good first issue and we can open a future issue for the removal in v1.21 once v1.20 ships. I think this will also need to be changed in the in v1.21 right? If is ok with that, I would like then to mark this as a good first issue to turn the feature GA in v1.20 :)
Well, taking a look into how secure is to move to GA: In the KEP there are some considerations about removing the IsMasterNode function, which seems to be already dealt in commit E2E tests needs to be also checked about usage of IsMasterNode(), which I could not find anymore and also seems to be removed from the same commit above. About the same Kep, it's directly related to other Feature Gate: ` So what I think that needs to be done: Promote LegacyNodeRoleBehavior to GA and Default it to false as proposed in the Kep -I'll open another issue just for this According to the LegacyNodeRoleBehavior should be removed in v1.19 but according to the Kep in v1.21, so let's stick with v1.21? ServiceNodeExclusion - Should be moved to GA, already defaulted to true. This issue is going to deal ONLY with this.


Question: // features that enable backwards compatibility but are scheduled to be removed // ... <del> HPAScaleToZero:         {Default: false, PreRelease: featuregate.Alpha}, LegacyNodeRoleBehavior: {Default: false, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.22 </del> <ins> HPAScaleToZero: {Default: false, PreRelease: featuregate.Alpha}, </ins> }

NodeDisruptionExclusion - Should be moved to GA, already defaulted to true. I'll open another issue for this I'm afraid of this having some sort of direct impact in cloud providers, do you want me to take this to the sig-cloudproviders meet? Otherwise can we move the above Feature Gates to GA, and schedule their removal for v1.21, wdyt?
Per deprecation lifecycle it seems that a feature needs 6 months or 2 releases (the longer one) in Beta before going to GA. So this actually seems to be the missing part before moving to GA. As the 3 FeatureGates are directed related, probably this issue should be: Move all of them to GA in v1.21 Default LegacyNodeRoleBehavior to false Sounds a good approach? I can keep this opened until we reach v1.21
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
/retitle Move Feature gate ServiceNodeExclusion to GA and LockToDefault
/triage accepted as this FG is scheduled to be moved to GA in v1.21, do you want to tackle on this?
fixed in /close
Closing this issue. <detailsIn response to : fixed in /close </details


Question: "//pkg/apis/core/v1/helper:go_default_library", "//pkg/apis/storage/v1/util:go_default_library", "//pkg/client/conditions:go_default_library", <ins> "//pkg/kubelet/events:go_default_library", </ins> "//pkg/kubelet/metrics:go_default_library", "//pkg/util/slice:go_default_library", "//pkg/volume/util:go_default_library",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: This feature is lacking coverage, especially for containerized kubelet environments.
/assign since I'm the owner of this feature.
And just fixes a bug for containerized kubelet when the mount file is an empty file.
Are we going to enable these tests in v1.11? We're getting code freeze soon.
There is a local containerized cluster pull and CI job that we can probably leverage. Right now it only runs conformance tests, we may be able to add new suites to enable other tests. Cc who can point us to the owner for the containerized testing effort
yes, seems to be holding up pretty well - https://k8s-,%20master%20(dev)
In I made pull-kubernetes-local-e2e-containerized to run also all test with string. I'm enabling some tests (subpath!) in , feel free to suggest more tests.
Ref
IIUC, the default host path type is . And when host path type is , we expect container runtime to create the host path if it doesn't exist (docker is doing that today). Is that true? If that is, we should revert part of , because container runtime should still create host path if it does not exist. /cc I tried to enforce , and got a lot of errors , e.g.:
Correct. This is mainly for backwards compatibility.
I see. If so, I'll remove the CRI definition "If the hostPath doesn't exist (e.g. hostPath volume), runtimes should report errors". Thanks!
SGTM. Refer to .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale will help fix this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: "//pkg/apis/core/v1/helper:go_default_library", "//pkg/apis/storage/v1/util:go_default_library", "//pkg/client/conditions:go_default_library", <ins> "//pkg/kubelet/events:go_default_library", </ins> "//pkg/kubelet/metrics:go_default_library", "//pkg/util/slice:go_default_library", "//pkg/volume/util:go_default_library",

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .


Question: "//pkg/apis/core/v1/helper:go_default_library", "//pkg/apis/storage/v1/util:go_default_library", "//pkg/client/conditions:go_default_library", <ins> "//pkg/kubelet/events:go_default_library", </ins> "//pkg/kubelet/metrics:go_default_library", "//pkg/util/slice:go_default_library", "//pkg/volume/util:go_default_library",

Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <ins> // FinalError constructs an error that indicates to a poll function that // polling can be stopped immediately because some permanent error has been // encountered that is not going to go away. // // TODO (@pohly): move this into framework once the refactoring from // https://github.com/kubernetes/kubernetes/pull/112043 allows it. Right now it // leads to circular dependencies. func FinalError(err error) error { return &FinalErr{Err: err} } type FinalErr struct { Err error } func (err *FinalErr) Error() string { if err.Err != nil { return fmt.Sprintf("final error: %s", err.Err.Error()) } return "final error, exact problem unknown" } func (err *FinalErr) Unwrap() error { return err.Err } // IsFinal checks whether the error was marked as final by wrapping some error // with FinalError. func IsFinal(err error) bool { var finalErr *FinalErr return errors.As(err, &finalErr) } </ins> // maybeTimeoutError returns a TimeoutError if err is a timeout. Otherwise, wrap err. // taskFormat and taskArgs should be the task being performed when the error occurred, // e.g. "waiting for pod to be running".

treats any error returned by the condition as terminal, but silently ignores any errors that aren't returned along with . We should change WaitForPodCondition to behave like . /sig testing /area e2e-test-framework
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Was there any reason why you instead of doing this right away? I fully agree with this issue, we should abort early. I just found the TODO because I have a failing test that ran longer than expected after a final pod failure and that made me check out why it didn't abort quicker :grin:


Question: executor := exec.New() args := []string{"trigger"} <del> cmd := executor.Command("/usr/bin/udevadm", args...) </del> <ins> cmd := executor.Command("udevadm", args...) </ins> _, err := cmd.CombinedOutput() if err != nil { glog.Errorf("error running udevadm trigger %vn", err)

I am running into a problem with the Cinder plugin when running Kubernetes inside Docker using the kube-deploy/docker-multinode project I modified the worker and master scripts of the latter project such that: --cloud-provider and --cloud-config options are to the hyperkube command I mount a host dir with the cloud_config file into the kubelet container (with the -v option) The cinder plugin of kubernetes (for attaching Openstack Cinder volumes) attaches a volume successfully and then tries to find the corresponding device. However the cinder plugin can't find this device by disk-id: The last log entry shows a failure to run . I tried to run this command from within a bash session inside the kubelet container and then got as error: How can I solve this? Thanks for any advice
CC
It basically tries to run to probe all attachments so sym link for the disk is created in . Is this the command you tried to run from command line?
Yes. However I now see that I misread the debuglogs that the full command was the following: . But this is wrong syntax obviously I launched Kubernetes version 1.2.6 in the dockerized version of hyperkube. I assume that the error lies somewhere in the fact that certain commands don't work properly when run inside a container. To further debug, I just had a look at the code from that reported the error (see below). First an executor is created using . Then this executor is asked to execute command . The implementation of the executor will try to execute the command in a child process (hence the fork)
I think I'm running into a related issue, or at least the same root cause. When using PV+PVC Cinder volumes the Kubelet fails to confirm that they are attached (even though I see the volume is attached in Horizon). The same error shows in the kubelet logs: However, on Ubuntu 16.04 at least, is located in a different place:
Adding a symlink to the correct location seems to have resolved the issue for me:


Question: - --memory={{ base_metrics_server_memory }} - --extra-memory={{ metrics_server_memory_per_node }}Mi - --threshold=5 <del> - --deployment=metrics-server-v0.3.0 </del> <ins> - --deployment=metrics-server-v0.3.1 </ins> - --container=metrics-server - --poll-period=300000 - --estimator=exponential

This started after was merged. The HPA test started being very flaky since then: https://k8s- The HPA test in containerd e2e become even more flaky since then with the same failure pattern: https://k8s-
I think this is probably /priority critical-urgent Release team hasn't looked close enough at this issue as we've been trying to clear other issues in front of it. If we crack those and this is unrelated root cause, then this one's going to be a release blocker.
It's going to be supremely hard to debug this unless we have logs from metrics-server and/or snapshots of the HPA over time. I'm trying to reproduce on a local cluster, but so far, the test passes consistently. did you manage to reproduce locally?
is there any way to grab logs of pods from a testing cluster?
you probably want to save it as part of the artifacts?
yeah, what's the process for that?
somewhere along
See for an attempt at debugging this.
can we provide the data is looking for? we're seeing this quite often.
ack, more logs would be helpful. Currently trying to get usable logs off the CI with the aforementioned PR, but I keep making minor mistakes in the log collection script :-/. It seems to repro every few runs though, so with any luck I'll see it again soon.
Got a WIP fix up: Need to do one more tweak before merge.
Re-running tests with the above PR, if they succeed I'll merge and get someone to do a build when the people capable of running builds are on next.
status check? I'm worried this is a 1.12 blocker and need to make a call by end of day today regarding delaying the release.
According to our testing, metrics-server 0.3.1 seems to have fixed it - thank you


Question: - '/event-exporter' # BEGIN_PROMETHEUS_TO_SD - name: prometheus-to-sd-exporter <del> image: gcr.io/google-containers/prometheus-to-sd:v0.2.1 </del> <ins> image: gcr.io/google-containers/prometheus-to-sd:v0.2.2 </ins> command: - /monitor <del> - --component=event_exporter </del> - --stackdriver-prefix={{ prometheus_to_sd_prefix }}/addons - --api-override={{ prometheus_to_sd_endpoint }} <del> - --whitelisted-metrics=stackdriver_sink_received_entry_count,stackdriver_sink_request_count,stackdriver_sink_successfully_sent_entry_count volumeMounts: - name: ssl-certs mountPath: /etc/ssl/certs </del> <ins> - --source=event_exported:http://localhost:80?whitelisted=stackdriver_sink_received_entry_count,stackdriver_sink_request_count,stackdriver_sink_successfully_sent_entry_count - --pod-id=$(POD_NAME) - --namespace-id=$(POD_NAMESPACE) env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace </ins> # END_PROMETHEUS_TO_SD terminationGracePeriodSeconds: 30 volumes:

Currently there are several, and some of them have problems with excessive logging could you please do this?
Any estimate on this fix? Asking because I'm seeing the issue referenced in
It was already fixed in head and in 1.8 long time ago. Fix was merged to 1.7 recently, so we are just waiting for new patch release.


Question: { genericapirequest.RequestInfo{ APIGroup:   "apps", <ins> APIVersion: "v1", Resource:   "daemonsets", }, map[string]string{"a": "b"}, map[string]string{"c": "d"}, field.ErrorList{ &field.Error{ Type:  field.ErrorTypeInvalid, Field: field.NewPath("spec").Child("selector").String(), BadValue: &metav1.LabelSelector{ MatchLabels:      map[string]string{"c": "d"}, MatchExpressions: []metav1.LabelSelectorRequirement{}, }, Detail: "field is immutable", }, }, }, { genericapirequest.RequestInfo{ APIGroup:   "apps", </ins> APIVersion: "v1beta2", Resource:   "daemonsets", },

/priority failing-test /priority critical-urgent /sig apps This test case has been failing for a while and affects a number of jobs: This is affecting multiple jobs on the testgrid dashboard, and prevents us from cutting 1.9.0-alpha.2 (kubernetes/sig-release). Is there work ongoing to bring this job back to green?
All test failures look similar: . ~Is it due to network issue instead of DaemonSet?~
This seems related to the new DaemonSet. The connection refused error is happening because kube-apiserver is crashing:
needs an update
Ah, well... I guess it's halfway working as intended. We put that there to make the tests fail if we forgot to update it when adding a new version. We had hoped it would fail during presubmit tests though. :(
Sending a fix now
I wonder why the test is not failing on submit queue tests. Do i need to fix deployment and replicaset?
The rest of the group is not merged, and the v1 PR takes the selector mutability into account.
Since this bug crashes apiserver I'd expect it to affect many other e2e tests.
Noted, we need to fix deployment and replicaset when the v1 PR is merged.
Not quite sure how this test isn't failing before but is now consistently failing. I'd expect the commit that triggers this to block the submit queue. The e2e test is still accessing DaemonSet through extensions/v1beta1 endpoints. From the above apiserver log it's hard to tell what updates DaemonSet via apps/v1 before the code panics. I tried to reproduce it locally, and found that garbage collector PUT apps/v1 DaemonSet:
When apps/v1 gets introduced, it may be ordered before apps/beta1 alphabetically. IIRC, there is a code change on the apps/v1 PR that changes the order of API versions such that apps/v1 is ranked before apps/v1beta2 and apps/v1beta2 before apps/v1beta1. Perhaps this problem can be fixed when the PR merges.
Since crashes apiserver and makes debugging harder, which defeats the purpose of using it, we should return error on unsupported versions instead.
The first run with this change is still in progress, but this test already passed: https://k8s- Thanks for the quick fix


Question: "os" "path" "reflect" <ins> "regexp" </ins> "sort" "strings" "sync"

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Unknown yet. Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated /sig scheduling /help
Can I give it a try?
Sure, feel free to take it. /assign ahmad-diaa /remove-help
I keep trying to get it fail locally by running the following command: but it never does, am I doing something wrong? Is there something else that I should be doing and would be more helpful?
Sometimes the flake is hard to reproduce in local env. You may have to read the error log of the flakes thoroughly to get some clues. BTW: you can also run the test multiple times using the directive:
After looking through the logs I came up with some information that might be useful As far as I understand, the reason for the flake is the of the medium priority pod not being cleared That is because the update request for removing returns the following error: So my current guess is that might be outdated in some cases As I am not really familiar with the logic, I would like to know whether or not those are valid points so that I could dig deeper. What do you think?
If that's the case, you can set here: You can raise a PR and manually try for ~20 times to see you can reproduce the flake.
found another issue tracking this too. it seems it has been flaky for a while.
Sorry, didn't mean to take this one for you. I worked the duplicate issue and submitted a PR. Feel free to review that PR, or submit your own if you have a different solution.
This sometimes happens. But we usually take the first PR unless it's clearly worked by someone. In terms of which style to resolve the Conflict error, I'm asking here:
Sorry, didn't mean to take this one for you. I worked the duplicate issue and submitted a PR. Feel free to review that PR, or submit your own if you have a different solution. No worries, I haven't started implementing yet and your PR seems to be following the correct approach as well according to this:


Question: "os" "path" "reflect" <ins> "regexp" </ins> "sort" "strings" "sync"

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: [TODO] Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig node These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.


Question: "os" "path" "reflect" <ins> "regexp" </ins> "sort" "strings" "sync"

Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func TestGetMaxDataDiskCount(t *testing.T) { tests := []struct { instanceType string <del> sizeList     *[]compute.VirtualMachineSize </del> expectResult int64 }{ { instanceType: "standard_d2_v2", <del> sizeList: &[]compute.VirtualMachineSize{ {Name: to.StringPtr("Standard_D2_V2"), MaxDataDiskCount: to.Int32Ptr(8)}, {Name: to.StringPtr("Standard_D3_V2"), MaxDataDiskCount: to.Int32Ptr(16)}, }, </del> expectResult: 8, }, { instanceType: "NOT_EXISTING", <del> sizeList: &[]compute.VirtualMachineSize{ {Name: to.StringPtr("Standard_D2_V2"), MaxDataDiskCount: to.Int32Ptr(8)}, }, </del> expectResult: defaultAzureVolumeLimit, }, { instanceType: "", <del> sizeList:     &[]compute.VirtualMachineSize{}, </del> expectResult: defaultAzureVolumeLimit, }, } for _, test := range tests { <del> result := getMaxDataDiskCount(test.instanceType, test.sizeList) </del> <ins> result := getMaxDataDiskCount(test.instanceType) </ins> assert.Equal(t, test.expectResult, result) } }

<!-- Please only use this template for submitting enhancement requests -- What would you like to be : related to Why is this needed: /assign /sig azure
here is the way about how to get MaxDataDiskCount: and then build kubelet, copy kubelet to agent node and then get MaxDataDiskCount data: There are 207 VM sizes totally.
Update: this command could get VM size list easily:


Question: # and kubelet export CLOUD_PROVIDER_FLAG="${CLOUD_PROVIDER_FLAG:-external}" <ins> # Don't run the node-ipam-controller on the KCM if cloud-provider external if [[ "${CLOUD_PROVIDER_FLAG}" ==  "external" ]]; then RUN_CONTROLLERS="${RUN_CONTROLLERS:-*,-node-ipam-controller}" fi </ins> # When ENABLE_AUTH_PROVIDER_GCP is set, following flags for out-of-tree credential provider for GCP # are presented to kubelet: # --image-credential-provider-config=${path-to-config}

Logs look like: originally mentioned in started a WIP PR to recreate, please see: /sig cloud-provider /sig testing cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
more comments on the original issue, it may be possible that error is a symton and the root cause is on the installation process
xref:


Question: state := meal.cfgCtlr.priorityLevelStates[pl.Name] if state == nil { labelValues := []string{pl.Name} <del> state = &priorityLevelState{reqsObsPair: meal.cfgCtlr.reqsObsPairGenerator.Generate(1, 1, labelValues), execSeatsObs: meal.cfgCtlr.execSeatsObsGenerator.Generate(1, 1, labelValues)} </del> <ins> state = &priorityLevelState{reqsObsPair: meal.cfgCtlr.reqsObsPairGenerator.Generate(1, 1, labelValues), execSeatsObs: meal.cfgCtlr.execSeatsObsGenerator.Generate(0, 1, labelValues)} </ins> } qsCompleter, err := queueSetCompleterForPL(meal.cfgCtlr.queueSetFactory, state.queues, pl, meal.cfgCtlr.requestWaitLimit, state.reqsObsPair, state.execSeatsObs) if err != nil {

While investigating another issue, I noticed that the histograms never get an Observation of zero. This makes no sense. See and related following remarks for some details. I expected the count of seats occupied by executing requests to be zero some times. Start an apiserver and scrape it; look at and see that the value is zero. No response <details</details<detailsnone </details<details</details<details</details<details</details<details</details


Question: nodeName := types.NodeName(a.plugin.host.GetHostName()) diskName := volumeSource.DiskName <del> var lun int32 if runtime.GOOS == "windows" { </del> <ins> lun := int32(-1) if runtime.GOOS != "windows" { // on Linux, usually devicePath is like "/dev/disk/azure/scsi1/lun2", get LUN directly lun, err = getDiskLUN(devicePath) if err != nil { klog.V(2).Infof("azureDisk - WaitForAttach: getDiskLUN(%s) failed with error: %v", devicePath, err) } } if lun < 0 { </ins> klog.V(2).Infof("azureDisk - WaitForAttach: begin to GetDiskLun by diskName(%s), DataDiskURI(%s), nodeName(%s), devicePath(%s)", diskName, volumeSource.DataDiskURI, nodeName, devicePath) lun, err = diskController.GetDiskLun(diskName, volumeSource.DataDiskURI, nodeName)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /assign


Question: }) framework.ExpectNoError(err, "creating apiservice %s with namespace %s", "v1alpha1.wardle.k8s.io", namespace) <del> // Wait for the extension apiserver to be up and healthy // kubectl get deployments -n <aggregated-api-namespace> && status == Running // NOTE: aggregated apis should generally be set up in there own namespace (<aggregated-api-namespace>). As the test framework // is setting up a new namespace, we are just using that. err = framework.WaitForDeploymentComplete(client, deployment) </del> err = wait.Poll(100*time.Millisecond, 30*time.Second, func() (bool, error) { request := restClient.Get().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders") request.SetHeader("Accept", "application/json")

https://k8s- https://k8s- https://k8s- https://k8s- https://k8s- ... A lot of them /cc
/assign
Still see this failure after is merged, although less frequently: https://k8s- https://k8s- Reopen.
Looks like the client doesn't differentiate between 500 and 503 here: It should. Will open a fix shortly
opened
fix merged, will monitor https://k8s-
Thanks! :)
haven't seen this in the 12 hours since the fix merged, was seeing it every 2-3 hours before. /close


Question: Containers: []v1.Container{ { Name:  "agnhost", <del> Image: imageutils.GetE2EImage(imageutils.Agnhost), </del> <ins> Image: AgnHostImage, </ins> Args:  args, }, },

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: Since when has it been failing: Ever Reason for failure: Vm on Azure with no public IP assigned aren't able to perform a command to the internet. Anything else we need to know: The test (see ) tries to test the internet connection for a pod by pinging 8.8.8.8. But azure does not provide ICMP support for VM without a public IP therefore this test cannot succeed.(see ) A fix would be to change the test method to a tcp request instead of a ping request (the test already performs a http get to in its BeforeSuite (see ))
/sig network /sig testing /sig azure
you may refer to (Pods don't have connectivity to external networks )
cc WDYT about the proposal to change this from an ICMP call? We probably need input from sig-architecture as well since this as well /sig architecture
/triage unresolved
An alternative is to issue a TCP/UDP request to 8.8.8.8 on port 53.
ICMP is dropped by Azure for security reasons. This is intended. Can use curl as a replacement. I will fix the corresponding test.
SGTM
Can you please review the related PR? I'll in the PR again. Thanks!


Question: done V=2 kube::log::info "Env for ${platform}: GOOS=${GOOS-} GOARCH=${GOARCH-} GOROOT=${GOROOT-} CGO_ENABLED=${CGO_ENABLED-} CC=${CC-}" <del> V=3 kube::log::info "Building binaries with GCFLAGS=${gogcflags} ASMFLAGS=${goasmflags} LDFLAGS=${goldflags}" </del> <ins> V=3 kube::log::info "Building binaries with GCFLAGS=${gogcflags} LDFLAGS=${goldflags}" </ins> local -a build_args if [[ "${#statics[@]}" != 0 ]]; then

While working on for , I found that our kubernetes1.24 package stopped to produce bit-identical results on 2 fresh builds. The first chunk is likely just the hash of the remainder, so can be ignored. The other chunks indicate that a random build temporary path is used for Sources should be able to reproduce bit-identical binaries e.g. for SLSA level4 On Debian or openSUSE do No response <detailsWARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-05-05T00:00:00Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v4.5.4 console $ cat /etc/os-release $ uname -a Linux bmwiedemann-factory 5.18.6-1-default SMP PREEMPTDYNAMIC Thu Jun 23 05:46:18 UTC 2022 () x8664 x8664 x86_64 GNU/Linux </details<detailsgo1.18 </details<details</details<details</details
/sig release /area build-release
can you please post a link to the full logs from the build?
any news on this? Haven't heard from (context: we worked to fix this a couple of times in )
Unfortunately not, is this something we can fix on the OBS side? If the go build timestamp () is the main issue here then we can approach a fix in our build pipeline.
It would be good if would not be embedded in the first place. It might be possible to replace it somehow in our builds, but all other people wanting reproducible builds, would need to replicate that.
full logs from the build: from kubernetes-1.24.3 built 2022-08-18
can you please try setting GOCACHE=off when you run the build? I am suspecting that the temp directory is being generated here: (or add an export in for that env var)
I to but still got these diffs.


Question: done V=2 kube::log::info "Env for ${platform}: GOOS=${GOOS-} GOARCH=${GOARCH-} GOROOT=${GOROOT-} CGO_ENABLED=${CGO_ENABLED-} CC=${CC-}" <del> V=3 kube::log::info "Building binaries with GCFLAGS=${gogcflags} ASMFLAGS=${goasmflags} LDFLAGS=${goldflags}" </del> <ins> V=3 kube::log::info "Building binaries with GCFLAGS=${gogcflags} LDFLAGS=${goldflags}" </ins> local -a build_args if [[ "${#statics[@]}" != 0 ]]; then

So i built golang from source, peppered it with changes to track down where this is getting created and found that it is coming from: I don't know yet how to fix it. will dig more when i can (travelling for the next few days)
Would it work to override the , I see that they use the : Ref: https://reproducible- Edit: Probably not, we may make it configurable in golang :thinking:
this is so tricky :( will dig more when i get back from my event this week
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned".


Question: done V=2 kube::log::info "Env for ${platform}: GOOS=${GOOS-} GOARCH=${GOARCH-} GOROOT=${GOROOT-} CGO_ENABLED=${CGO_ENABLED-} CC=${CC-}" <del> V=3 kube::log::info "Building binaries with GCFLAGS=${gogcflags} ASMFLAGS=${goasmflags} LDFLAGS=${goldflags}" </del> <ins> V=3 kube::log::info "Building binaries with GCFLAGS=${gogcflags} LDFLAGS=${goldflags}" </ins> local -a build_args if [[ "${#statics[@]}" != 0 ]]; then

<detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is still relevant even with 1.27. It might be worth revisiting this issue again. /reopen /triage accepted
Reopened this issue. <detailsIn response to : This is still relevant even with 1.27. It might be worth revisiting this issue again. /reopen /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
latest run -
I found that our package has the same issue, so it probably it is a toolchain issue in golang - and much faster to debug outside of the slow kubernetes build. =filed
AIUI, the conclusion by the golang people seems to be that commit is not the right approach and one should use with the go call instead. Not sure how that interacts with other parts of your build.
cc (please see referenced commit above!)
I see. I inherited the gcflags before so I plead innocence. It seems plausible to change it from to - I don't know when I will have time to look at that, though.


Question: done V=2 kube::log::info "Env for ${platform}: GOOS=${GOOS-} GOARCH=${GOARCH-} GOROOT=${GOROOT-} CGO_ENABLED=${CGO_ENABLED-} CC=${CC-}" <del> V=3 kube::log::info "Building binaries with GCFLAGS=${gogcflags} ASMFLAGS=${goasmflags} LDFLAGS=${goldflags}" </del> <ins> V=3 kube::log::info "Building binaries with GCFLAGS=${gogcflags} LDFLAGS=${goldflags}" </ins> local -a build_args if [[ "${#statics[@]}" != 0 ]]; then

My patch doesn't seem to fix the whole problem: thockin-glaptop4 kubernetes buildtrimpath /$ make kubelet +++ [1114 19:10:12] Building go targets for linux/amd64 (non-static) thockin-glaptop4 kubernetes buildtrimpath /$ md5sum output/bin/kubelet output/bin/kubelet thockin-glaptop4 kubernetes buildtrimpath /$ make kubelet +++ [1114 19:12:40] Building go targets for linux/amd64 (non-static) thockin-glaptop4 kubernetes buildtrimpath /$ md5sum output/bin/kubelet output/bin/kubelet thockin-glaptop4 kubernetes build_trimpath /$
diff --- a 2023-11-14 19:13:45. -0800 +++ b 2023-11-14 19:13:56. -0800 -1,4 +1,4 -38rZHdQckYF9EEzmndR/BAXOwzCcq0Mk1UHT62ha/QLiRgXP4oojWh3oWP9/TWG7KW90IL8vKt1UOEPR +LeLArvl7sgWAd55ROKN2/BAXOwzCcq0Mk1UHT62ha/QLiRgXP4oojWh3oWP9/cYb5ZGQwa6js077YiAvK /lib64/ld-linux-x86- AUATUSH []AA] -,8 +,8 go1.21.3 runtime/cgo: call frame too large -2023-11-15T03:12:40Z -2023-11-15T03:12:40Z +2023-11-15T03:13:47Z +2023-11-15T03:13:47Z pthreadcreate failed: %s bad stack bounds: lo=%p hi=%p expand 32-byte kexpand 32-byte k
That seems unrelated. I tested with and it made our openSUSE package fully reproducible. Your diff contains a timestamp - maybe would get rid of that. The top part of the diff is just the go build-id hash of the remainder and can be ignored.


Question: done V=2 kube::log::info "Env for ${platform}: GOOS=${GOOS-} GOARCH=${GOARCH-} GOROOT=${GOROOT-} CGO_ENABLED=${CGO_ENABLED-} CC=${CC-}" <del> V=3 kube::log::info "Building binaries with GCFLAGS=${gogcflags} ASMFLAGS=${goasmflags} LDFLAGS=${goldflags}" </del> <ins> V=3 kube::log::info "Building binaries with GCFLAGS=${gogcflags} LDFLAGS=${goldflags}" </ins> local -a build_args if [[ "${#statics[@]}" != 0 ]]; then

Thanks for confirming
Will there be backports to 1.24++ ? I found it applies cleanly to 1.26++ and works with some small tweaks in 1.24 and 1.25. If not, I'd submit the patch for our openSUSE packages.
1.24 and 1.25 reached EOL and we don't support these branches any longer. We can cherry-pick this to 1.26+ if folks ( agree
Cherry-picks: 1.29 1.28 1.27 1.26


Question: "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" <del> e2erc "k8s.io/kubernetes/test/e2e/framework/rc" </del> e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" testutils "k8s.io/kubernetes/test/utils" <del> imageutils "k8s.io/kubernetes/test/utils/image" </del> ) // Resource is a collection of compute resource.

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Kubernetes e2e suite.[sig-scheduling] SchedulerPriorities [Serial] Pod should avoid nodes that have avoidPod annotation June 19th 2021 () Testgrid: https://k8s- Prow history: Looks like pods are scheduled where they shouldn't be? : The tests passes in maybe 20% of the builds, but it looks random, so I don't treat it as a flake. At the moment there are also sig-storage tests consistently failing in this job: , They don't look related though. /sig scheduling /kind failing-test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
try to have a look.
try to have a look.
/cc
This is expected. The feature (which was alpha) is now disabled by default. Users can still enable it by switching to a v1beta1 scheduler component config. Let's remove the test.


Question: "command": [ "/bin/sh", "-c", <del> "/usr/local/bin/kube-apiserver {{params}} --allow-privileged={{pillar['allow_privileged']}} 1>>/var/log/kube-apiserver.log 2>&1" </del> <ins> "exec /usr/local/bin/kube-apiserver {{params}} --allow-privileged={{pillar['allow_privileged']}} 1>>/var/log/kube-apiserver.log 2>&1" </ins> ], {{container_env}} "livenessProbe": {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When kube-apiserver's livenessProbe suggests that pod is unhealthy, kubelet tries to kill its container with 30 seconds grace period. The container is killed ungracefully with SIGKILL when grace period passes. This happens each time. What you expected to happen: I expected that SIGTERM will be passed to kube-apiserver which will try to gracefully stop before 30 seconds pass (at least in some cases). How to reproduce it (as minimally and precisely as possible): Inject kube-apiserver failure (e.g. ) Watch kubelet logs for container's death You will see that container is killed after 30 seconds each time Anything else we need to know?: I believe the problem is that in kube-apiserver.manifest uses '/bin/sh -c CMD' as a PID1 process which doesn't pass SIGTERM signals to child process. See Environment: Kubernetes version (use ): v1.7.11
/sig node
Looks like without the pipe command will have to be removed which will make the logs hard to track. There is indeed some which makes kube-apiserver run in background and trapping SIGTERM then send a SIGTERM to apiserver's PID.
I think this is the case for all our components, not just kube-apiserver. - FYI
How about using 'exec'-based solution suggested here: In our case it means prepending 'exec': I did changed this in my cluster and the process seems to meet requirements: fd/1 and fd/2 are redirected to /var/log/kube- container is killed instantaneously If everyone is fine with such a solution, I can prepare PR adding 'exec ' in front of all kubernetes/cluster/saltbase/salt/kube-apiserver/*.manifest using '/bin/sh -c CMD' pattern.
Not sure if understand what you mean. Before there was no logs, so /bin/sh -c wasn't needed. -based solution seems to have all required properties. What is your concern about using it?
Umm. just ignore my previous comment. I just tried prepending and it works fine for me.


Question: "command": [ "/bin/sh", "-c", <del> "/usr/local/bin/kube-apiserver {{params}} --allow-privileged={{pillar['allow_privileged']}} 1>>/var/log/kube-apiserver.log 2>&1" </del> <ins> "exec /usr/local/bin/kube-apiserver {{params}} --allow-privileged={{pillar['allow_privileged']}} 1>>/var/log/kube-apiserver.log 2>&1" </ins> ], {{container_env}} "livenessProbe": {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 3 Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: All of the salt scripts bundled in kubernetes (for example, ) start up the component in a broken way. They use which means the shell will intercept all signals. So these daemons are not gracefully shutdown when the pod is stopped. This causes serious problems since when apiserver is killed abruptly it doesn't seem to close watches correctly. This was observed in for example. What you expected to happen: Running is incorrect. It should run the binary directly. Otherwise, it needs to perform an so the binary correctly replaces the shell script process. How to reproduce it (as minimally and precisely as possible): Run kube-apiserver inside a container with and try to stop it. You can't - eventually docker will force kill it. Anything else we need to know:
It looks like we changed from running the binaries directly to using in (from a few years ago to enable us to capture logs. or may recall why we had to use a shell since they reviewed that PR.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: "command": [ "/bin/sh", "-c", <del> "/usr/local/bin/kube-apiserver {{params}} --allow-privileged={{pillar['allow_privileged']}} 1>>/var/log/kube-apiserver.log 2>&1" </del> <ins> "exec /usr/local/bin/kube-apiserver {{params}} --allow-privileged={{pillar['allow_privileged']}} 1>>/var/log/kube-apiserver.log 2>&1" </ins> ], {{container_env}} "livenessProbe": {

Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: <ins> /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ // This is an empty file to avoid "build constraints exclude all Go files" error while running unit tests on non-linux platforms. package ipset </ins>

Tried running make test against local latest fork of kubernetes master Go version Go env Successfully able to run the test Try running make test on darwin os. No response <details</detailsLatest master <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Bro i think there might be problem with your setup , its not an issue .
test cases are working fine
Thanks for verifying, I will double check from my side, I hope you verified it on darwin os.
/sig release
Also seeing this when trying to run on mac


Question: func (o *imagePlugin) GetVolumeName(spec *volume.Spec) (string, error) { return o.spec.Name(), nil } func (o *imagePlugin) CanSupport(spec *volume.Spec) bool { <del> return spec.Volume.Image != nil </del> <ins> return spec != nil && spec.Volume != nil && spec.Volume.Image != nil </ins> } func (o *imagePlugin) NewMounter(spec *volume.Spec, podRef *v1.Pod, opts volume.VolumeOptions) (volume.Mounter, error) {

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
please see <img width="840" alt="image" src=""
looking at the failures could be between the following SHA(s):
the main commit there is can you please look into it?
shows the same as well. commit range is
Found the issue, proposing a fix in


Question: } return } <del> // test with cost budget exceeded errs, _ = celValidator.Validate(ctx, field.NewPath("root"), &s, tt.obj, tt.oldObj, 0) var found bool for _, err := range errs { if err.Type == field.ErrorTypeInvalid && strings.Contains(err.Error(), "validation failed due to running out of cost budget, no further validation rules will be run") { found = true } } if !found { t.Errorf("expect cost limit exceed err but did not find") } if len(errs) > 1 { t.Errorf("expect to return cost budget exceed err once") } // test with PerCallLimit exceeded found = false celValidator = NewValidator(&s, 0) if celValidator == nil { t.Fatal("expected non nil validator") } errs, _ = celValidator.Validate(ctx, field.NewPath("root"), &s, tt.obj, tt.oldObj, tt.costBudget) for _, err := range errs { if err.Type == field.ErrorTypeInvalid && strings.Contains(err.Error(), "no further validation rules will be run due to call cost exceeds limit for rule") { found = true break } } if !found { t.Errorf("expect PerCostLimit exceed err but did not find") } </del> }) } for rule, expectErrToContain := range tt.errors {

/kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
The flake started after was merged. /cc I run it locally and it costs about 190s~220s.
/sig api-machinery testing
should fix. Needs approval.
/triage accepted


Question: return result } <del> func createRC(ns, rsName string, replicas int32, rcPodLabels map[string]string, f *framework.Framework, resource *v1.ResourceRequirements) *v1.ReplicationController { rc := &v1.ReplicationController{ TypeMeta: metav1.TypeMeta{ Kind:       "ReplicationController", APIVersion: "v1", }, ObjectMeta: metav1.ObjectMeta{ Name: rsName, }, Spec: v1.ReplicationControllerSpec{ Replicas: &replicas, Template: &v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: rcPodLabels, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:      rsName, Image:     imageutils.GetPauseImageName(), Resources: *resource, }, }, }, }, }, } rc, err := f.ClientSet.CoreV1().ReplicationControllers(ns).Create(context.TODO(), rc, metav1.CreateOptions{}) framework.ExpectNoError(err) return rc } </del> func getRandomTaint() v1.Taint { return v1.Taint{ Key:    fmt.Sprintf("kubernetes.io/e2e-scheduling-priorities-%s", string(uuid.NewUUID()[:23])),


Question: "fieldmanager_test.go", "lastappliedmanager_test.go", "lastappliedupdater_test.go", <ins> "managedfieldsupdater_test.go", </ins> "skipnonapplied_test.go", ], data = [

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I'm trying to automate the usage of to check for changes that need to be applied to the cluster based on a static set of manifests. However, for some manifests, if I run the output contains differences on fields called and , after the manifest was applied to the cluster. I ran . After that, I ran The following output was produced (I copied just the first diff, as many more followed): What you expected to happen: No differences should be detected. How to reproduce it (as minimally and precisely as possible): Run Then run Anything else we need to know?: This does not happen to all manifests. If I try with the manifest no differences are detected. I also tried with manifests loaded from disk, and was able to reproduce the behavior with the following manifest: With the above manifest the following occurs: Empirically, the differences observed always seem to be some seconds apart between the two versions. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: baremetalOS (e.g: ):Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig CLI
Ref: The newest version of has a different result still has a problem when diffing :
Sorry, I was unaware that there was a specific repository on which to report kubectl related issues. If needed, I can recreate this issue on the kubernetes/kubectl repository. Do let me know if this is preferred.
/assign
Hey, thanks for the report About the first issue, applying "". It looks like this yaml is dynamically generated and different every time you get it: So I think this one is totally working as intended. We're showing a difference because there is an actual difference. The second one is actually a bug that we are supposed to have fixed in 1.18. Let me look at it.
Ah, I see. That does account for the differences in the date field. Most unfortunate for me, but definitely not a bug in kubernetes/kubelet. My apologies, I should have checked that first.


Question: "fieldmanager_test.go", "lastappliedmanager_test.go", "lastappliedupdater_test.go", <ins> "managedfieldsupdater_test.go", </ins> "skipnonapplied_test.go", ], data = [

So the only actual issue is with the changing time field. Thanks for looking into that. Do let me know if there is anything I can do to help.
Can you show me the object before you run that diff? I'm curious what the first "fieldManager" is and why it's different from "kubectl-client-side-apply". Also, did you create the object using ?
I figured out what the problem is in that second message: the cpu goes from an to a : Note that I'm using a non-server-side capable apiserver version, but I think the problem is the same: we can't detect that this object has been changed because cpu goes from to , or from to and we consider this as a change. I have no clue how to solve that "bug".
Hi The test you made is comparing the local yaml against the version the apiserver currently has, correct? That would be similar to running the diff with the option to run server side, which I think yields a suspiciously similar result: I also suspect that it is for this reason that if I run the apply command with the dry-run flag, we always get incoming changes to apply: For comparison sake, the same test with the server-metrics manifest, that does not produce any diff changes, does not detect any incoming changes (notice all of the items are marked with unchanged): I also have no idea how to fix this properly, but perhaps running a yaml formatter on both sources (left and right) before comparison could produce better results? A canonical representation of both inputs might be better suited for comparison. Perhaps such a behavior could be controlled with a new enabling CLI flag. Again, thanks for looking into this. Having a more reliable diff behavior would go a long way for us to detect configuration mismatches against our configuration management system.
Sure thing! I spawned from this bug report to cover the "failed to create typed patch object" problem. I will try to test with the newly minted 1.19 release later today, to see if any previously observed behavior changes.
I recreated the cluster as version 1.19.0, and here are the results of the previous tests: Ran as expected Ran as expected Ran showing unexpected diff results
Thanks for all the details that's actually really useful for me to understand what's going on.
Some more context.


Question: "fieldmanager_test.go", "lastappliedmanager_test.go", "lastappliedupdater_test.go", <ins> "managedfieldsupdater_test.go", </ins> "skipnonapplied_test.go", ], data = [

A Gatekeeper user reported behavior where always shows a diff in the timestamp for of its CRD objects -- it looks like non-CRDs are unaffected. This happens when re-applying the static manifest found at
Thanks, this is useful, we'll look into it!


Question: NodeAllocatableChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeAllocatable, Label: "NodeAllocatableChange"} // NodeLabelChange is the event when node label is changed. NodeLabelChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeLabel, Label: "NodeLabelChange"} <ins> // NodeAnnotationChange is the event when node annotation is changed. NodeAnnotationChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeAnnotation, Label: "NodeAnnotationChange"} </ins> // NodeTaintChange is the event when node taint is changed. NodeTaintChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeTaint, Label: "NodeTaintChange"} // NodeConditionChange is the event when node condition is changed.

Due to limited character space, I've stored additional information in annotations on nodes. My custom scheduling plugins read these annotations to schedule Pods. However, when a Pod fails to be scheduled on a node, I want it to be removed from the Unschedulable queue. This can be achieved by modifying the annotations of the node, allowing the Pod to be rescheduled. Unfortunately, the eventHandler of the scheduler only supports five types of node events, and it does not include the nodeAnnotationsChanged event. Consequently, I am currently unable to trigger rescheduling by changing the annotations of nodes. /sig scheduling Unschedulable pods will be rescheduled after annotations of nodes changed. None No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign
It's more like a new feature, not bug because it doesn't lead to any bad behavior on the upstream. /remove-kind bug /kind feature
My concern is node annotation is arbitrary, change a node annotation doesn't imply a pod is schedulable. But for other events, like node labels, node taints, they're the constraints we consider in scheduling. They're different, I'm afraid this will lead to invalid scheduling attempts. Who is responsible for change the annotations in you scenario?
This enhancement has no impact on the in-tree plugins / upstream scheduler, right? Only (custom) plugins, that want to reschedule Pods via the node annotation change, will register event to EventsToRegister.
I have a controller capable of modifying node annotations to indicate node's current availability. And my custom plugins need to be aware of these changes.
I guess we can still be impacted when scheduling pod failed because of no fit node rather than failed by the plugin.


Question: NodeAllocatableChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeAllocatable, Label: "NodeAllocatableChange"} // NodeLabelChange is the event when node label is changed. NodeLabelChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeLabel, Label: "NodeLabelChange"} <ins> // NodeAnnotationChange is the event when node annotation is changed. NodeAnnotationChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeAnnotation, Label: "NodeAnnotationChange"} </ins> // NodeTaintChange is the event when node taint is changed. NodeTaintChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeTaint, Label: "NodeTaintChange"} // NodeConditionChange is the event when node condition is changed.

see Can we predict whether the pod is schedulable before running into scheduling cycle via the node annotations, then we can leverage the preEnqueue extension point since you already have a external controller.
Thanks for the advice. But I'm afraid it would not be the best solution for us, because doing that outside the scheduling cycle would cause the custom plugins to malfunction in my scenario.
The concerns raised by make sense to me. In my understanding, most in-tree plugins will classify the event as an event.
No. When no fit Pod, UnschedulablePlugins will have plugin(s) rejected the Pod. (e.g., all nodes don't have enough capacity, UnschedulablePlugins has noderesourcefit. all nodes don't satisfy node affinity, UnschedulablePlugins has nodeaffinity) is a rare case like plugins other than PreFilter/Filter returns non-success. I don't think we should care about it.
But, this is a good point, I think we should refactor all plugins that register event with to have a specific Node update event. For example, . It should have with instead of . event with , that should be the last-resort when we don't know which kind of Node update will make Pods schedulable. (I guess no such case at least in the in-tree plugins.)
1) One possibility is Reserve Plugin, I the capability here 2) Another possibility is PreFilter, but not because failed by the plugin, but no fit node exists Have no idea whether we have other places, I guess no. But yes, this only impacts unschedulable pods has no failed plugins.
~I think this is for intended, like ~ ~1. 1) we have a plugin failed by nodeAffinity~ ~2. 2) we add a new node~ ~3. 3) we receive the node add event, but because node not ready, we failed by nodeTaint plugin as ~ 4) node become ready, we get the node update event, but dismatch with nodeAffinity registered event (if we registered with UpdateNodeLabel event)~ ~5. 5) pod will not be moved to the active or backoff queue~ updated below
weigh in for advices as whether this is a good idea to take nodeAnnotations as a new event.


Question: NodeAllocatableChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeAllocatable, Label: "NodeAllocatableChange"} // NodeLabelChange is the event when node label is changed. NodeLabelChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeLabel, Label: "NodeLabelChange"} <ins> // NodeAnnotationChange is the event when node annotation is changed. NodeAnnotationChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeAnnotation, Label: "NodeAnnotationChange"} </ins> // NodeTaintChange is the event when node taint is changed. NodeTaintChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeTaint, Label: "NodeTaintChange"} // NodeConditionChange is the event when node condition is changed.

As I know, some plugins will read the embedded prometheus metrics in node annotations for scoring, but not for filtering. At least we should mitigate the impacts.
iv. node become ready, we get the node update event, but dismatch with nodeAffinity registered event (if we registered with UpdateNodeLabel event) (3) is not exact. When we receive the node add event, the scheduler retry to schedule the Pod because the nodeaffinity has Node add event. And it's rejected by the nodetaint plugin. (not node affinity) So, in (4), we get the node update event, it matches with node taint registered event, and the Pod is moved to active/backoffQ (node taint registers node update event. Yes I think we can change node taint's node update event to more specific one(s) though)
As I replied in your PR, we acknowledged that in , and yes let's have your patch. Well, that's a good point. So, when all nodes are filtered out by PreFilterResult, then no failedplugin can be returned. That's a problem. I think we should return the plugin name that returns PreFilterResult with not all Node names in this case (= the plugins that contributes to reducing the candidate Nodes via PreFilterResult). What do you think?
Sorry, the previous one is not right, a detailed one: a. we have a pod failed by nodeAffinity. pod will be placed into the unschedulablePods b. node autoscaler triggered, a new node , emitted a node add event c. called -d. node not ready right now and holds a notReady taint, then failed, pod still stays in the unschedulablePods e. node gets ready, emitted a node update event, but dismatch with the registered event(if we register with in nodeAffinity plugin) f. pod will stay in unschedulabelPods I guess this is the right flow. Anyway, glad to see it be more effect in the future.
If we have no performance problems, I think it's ok to take nodeAnnotations into consideration.
Wouldn't this be solved with proposal on ClusterEventWithHint? If so, I prefer we leave the existing code as-is.
whatever we do here, we wouldn't backport, so I think is the right move forward.
We put many stuff here.


Question: NodeAllocatableChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeAllocatable, Label: "NodeAllocatableChange"} // NodeLabelChange is the event when node label is changed. NodeLabelChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeLabel, Label: "NodeLabelChange"} <ins> // NodeAnnotationChange is the event when node annotation is changed. NodeAnnotationChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeAnnotation, Label: "NodeAnnotationChange"} </ins> // NodeTaintChange is the event when node taint is changed. NodeTaintChange = framework.ClusterEvent{Resource: framework.Node, ActionType: framework.UpdateNodeTaint, Label: "NodeTaintChange"} // NodeConditionChange is the event when node condition is changed.

Let me summarize each: Born from from is right. But, that's the problem with preCheck. (Some preCheck depends on some in-tree plugins' logic and it affects badly to other plugins enqueueing ) The preCheck is removed after is completed. After preCheck is removed, will be the correct behavior. So, we can do the optimization that changes node to a specific event. will be fixed in: It's kind of rare case, but I feel we need to have some improvement like Edit: create another issue for this: I'd say (2) and (3) are kind of rare cases. So, the blocker here is (1). (And, (1) is blocked by the removal of preCheck, which is blocked by ) As my conclusion, it's safer to add after (1) is done so that we can make sure the performance won't be affected badly. what do you think?
Instead, can we just keep an event for "node updated", then individual plugins can decide whether a annotations change is worth requeuing or not?
Yes, (1) can be replaced with that idea. But, I'd say changing to a specific is slightly better because the scheduling queue could filter out some Node update events without executing the hint functions.
If rescheduling can be triggered with precise events, that would be great. I haven't revisit since last time, I'll take a look again.
Since we are missing an event for , can we add it? But we also have to modify existing events like so: Because change of such field is also an Update. And in future releases we can get rid of some fine-grained events.
Since we will add hintFuncs for all in-tree plugins, so I think it's ok to add nodeAnnotationsChanged event now, we can filter out unrelated events in hitFuncs. Any other concerns? cc Prefer to keep them as people can register fine-gained events instead a general event.
If by "now" you mean v1.30, then I'm ok with that.


Question: // Verify the correct network configuration was passed inputConfig := struct { RuntimeConfig struct { <del> Bandwidth    map[string]interface{}   `json:"bandwidth"` PortMappings []map[string]interface{} `json:"portMappings"` </del> <ins> PortMappings []map[string]interface{}   `json:"portMappings"` Bandwidth    map[string]interface{}     `json:"bandwidth"` IpRanges     [][]map[string]interface{} `json:"ipRanges"` </ins> } `json:"runtimeConfig"` }{} inputBytes, inerr := ioutil.ReadFile(inputFile)

/kind feature The CNI "capability arguments" parameter - currently used to pass port mappings - has a few new capabilities: bandwidth limiting and network address management. The dockershim's networking driver should use them. The network address is just passing the PodCIDR dynamically to the plugin, so it no longer has to retrieve it from the APIServer. Bandwidth limiting is already implemented in Kubenet. It is configured via annotations. There is no upgrade risk in providing these values; non-compliant plugins will simply ignore them. Administrators need to explicitly enable each capability individually in their CNI configuration file. To avoid skew between runtimes, the other runtimes should also be upgraded to pass these. We may also wish to look in to a way to add this more directly to the CRI, so we don't have this skew problem, but that's a different issue.
/sig network
/assign
fyi.
Looks like this was never done for CRI?
cc


Question: // CustomResourceDefinitionCondition contains details for the current condition of this pod. type CustomResourceDefinitionCondition struct { <del> // Type is the type of the condition. </del> <ins> // Type is the type of the condition. Types include Established, NamesAccepted and Terminating. </ins> Type CustomResourceDefinitionConditionType // Status is the status of the condition. // Can be True, False, Unknown.

What would you like to be : A comment in that includes the possible values of . I got the possible values from . The should contain the further explanation available in the linked code that explains what each condition type means. Why is this needed: Currently the output of is incomplete: It is important to include the values of because that's the value used to . It is not user-friendly to require users to guess the possible string values for condition or require them to read the code to deduce what the possible values are.
/sig cli
Please let me do it~
oops, I just notice your message. I already submitted a PR for this.
It's ok~


Question: return base } <del> // multiply by the appropriate exponential scale switch q.Format { case DecimalExponent, DecimalSI: return base * math.Pow10(exponent) default: // fast path for exponents that can fit in 64 bits if exponent > 0 && exponent < 7 { return base * float64(int64(1)<<(exponent*10)) } return base * math.Pow(2, float64(exponent*10)) } </del> <ins> return base * math.Pow10(exponent) </ins> } // AsInt64 returns a representation of the current value as an int64 if a fast conversion

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I found that for with returns a very small value. The value should be almost the same as the one of . My minimal : Result: While the top three lines () are good, the float values in the bottom three lines () might be broken. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/assign
/cc /triage accepted


Question: // Update implements Manager. func (f *managedFieldsUpdater) Update(liveObj, newObj runtime.Object, managed Managed, manager string) (runtime.Object, Managed, error) { self := "current-operation" <ins> formerSet := managed.Fields()[manager] </ins> object, managed, err := f.fieldManager.Update(liveObj, newObj, managed, self) if err != nil { return object, managed, err


Question: // Update implements Manager. func (f *managedFieldsUpdater) Update(liveObj, newObj runtime.Object, managed Managed, manager string) (runtime.Object, Managed, error) { self := "current-operation" <ins> formerSet := managed.Fields()[manager] </ins> object, managed, err := f.fieldManager.Update(liveObj, newObj, managed, self) if err != nil { return object, managed, err


Question: // Update implements Manager. func (f *managedFieldsUpdater) Update(liveObj, newObj runtime.Object, managed Managed, manager string) (runtime.Object, Managed, error) { self := "current-operation" <ins> formerSet := managed.Fields()[manager] </ins> object, managed, err := f.fieldManager.Update(liveObj, newObj, managed, self) if err != nil { return object, managed, err


Question: kube::log::usage_from_stdin() { local messages=() while read -r line; do <del> messages+=$line </del> <ins> messages+=("$line") </ins> done kube::log::usage "${messages[@]}"

Example: I have a fixing PR ready, will send it in a moment.
/sig docs /assign closing because linked pr fixed this
/close


Question: return err } <del> func TestApiExamples(t *testing.T) { expected := map[string]runtime.Object{ "controller":       &api.ReplicationController{}, "controller-list":  &api.ReplicationControllerList{}, "pod":              &api.Pod{}, "pod-list":         &api.PodList{}, "service":          &api.Service{}, "external-service": &api.Service{}, "service-list":     &api.ServiceList{}, </del> <ins> func TestExampleObjectSchemas(t *testing.T) { cases := map[string]map[string]runtime.Object{ "../api/examples": { "controller":       &api.ReplicationController{}, "controller-list":  &api.ReplicationControllerList{}, "pod":              &api.Pod{}, "pod-list":         &api.PodList{}, "service":          &api.Service{}, "external-service": &api.Service{}, "service-list":     &api.ServiceList{}, }, "../examples/guestbook": { "frontend-controller":    &api.ReplicationController{}, "redis-slave-controller": &api.ReplicationController{}, "redis-master":           &api.Pod{}, "frontend-service":       &api.Service{}, "redis-master-service":   &api.Service{}, "redis-slave-service":    &api.Service{}, }, "../examples/walkthrough": { "pod1": &api.Pod{}, "pod2": &api.Pod{}, }, </ins> } <del> tested := 0 err := walkJSONFiles("../api/examples", func(name, path string, data []byte) { expectedType, found := expected[name] if !found { t.Errorf("%s does not have a test case defined", path) return } tested += 1 if err := latest.Codec.DecodeInto(data, expectedType); err != nil { t.Errorf("%s did not decode correctly: %vn%s", path, err, string(data)) return </del> <ins> for path, expected := range cases { tested := 0 err := walkJSONFiles(path, func(name, path string, data []byte) { expectedType, found := expected[name] if !found { t.Errorf("%s does not have a test case defined", path) return } tested += 1 if err := latest.Codec.DecodeInto(data, expectedType); err != nil { t.Errorf("%s did not decode correctly: %vn%s", path, err, string(data)) return } if errors := validateObject(expectedType); len(errors) > 0 { t.Errorf("%s did not validate correctly: %v", path, errors) } }) if err != nil { t.Errorf("Expected no error, Got %v", err) </ins> } <del> if errors := validateObject(expectedType); len(errors) > 0 { t.Errorf("%s did not validate correctly: %v", path, errors) </del> <ins> if tested != len(expected) { t.Errorf("Expected %d examples, Got %d", len(expected), tested) </ins> } <del> }) if err != nil { t.Errorf("Expected no error, Got %v", err) } if tested != len(expected) { t.Errorf("Expected %d examples, Got %d", len(expected), tested) </del> } } <del> func TestExamples(t *testing.T) { expected := map[string]runtime.Object{ "frontend-controller":    &api.ReplicationController{}, "redis-slave-controller": &api.ReplicationController{}, "redis-master":           &api.Pod{}, "frontend-service":       &api.Service{}, "redis-master-service":   &api.Service{}, "redis-slave-service":    &api.Service{}, } tested := 0 err := walkJSONFiles("../examples/guestbook", func(name, path string, data []byte) { expectedType, found := expected[name] if !found { t.Errorf("%s does not have a test case defined", path) return } tested += 1 if err := latest.Codec.DecodeInto(data, expectedType); err != nil { t.Errorf("%s did not decode correctly: %vn%s", path, err, string(data)) return } if errors := validateObject(expectedType); len(errors) > 0 { t.Errorf("%s did not validate correctly: %v", path, errors) } }) if err != nil { t.Errorf("Expected no error, Got %v", err) } if tested != len(expected) { t.Errorf("Expected %d examples, Got %d", len(expected), tested) } } var jsonRegexp = regexp.MustCompile("(?ms)^```w*n({.+?})w*n^```") </del> <ins> var sampleRegexp = regexp.MustCompile("(?ms)^```(?:(?P<type>yaml)w*n(?P<content>.+?)|w*n(?P<content>{.+?}))w*n^```") var subsetRegexp = regexp.MustCompile("(?ms).{3}") </ins> func TestReadme(t *testing.T) { <del> path := "../README.md" data, err := ioutil.ReadFile(path) if err != nil { t.Fatalf("Unable to read file: %v", err) </del> <ins> paths := []string{ "../README.md", "../examples/walkthrough/README.md", </ins> } <del> match := jsonRegexp.FindStringSubmatch(string(data)) if match == nil { return } for _, json := range match[1:] { expectedType := &api.Pod{} if err := latest.Codec.DecodeInto([]byte(json), expectedType); err != nil { t.Errorf("%s did not decode correctly: %vn%s", path, err, string(data)) return } if errors := validateObject(expectedType); len(errors) > 0 { t.Errorf("%s did not validate correctly: %v", path, errors) } encoded, err := latest.Codec.Encode(expectedType) </del> <ins> for _, path := range paths { data, err := ioutil.ReadFile(path) </ins> if err != nil { <del> t.Errorf("Could not encode object: %v", err) </del> <ins> t.Errorf("Unable to read file %s: %v", path, err) </ins> continue } <del> t.Logf("Found pod %sn%s", json, encoded) </del> <ins> matches := sampleRegexp.FindAllStringSubmatch(string(data), -1) if matches == nil { continue } for _, match := range matches { var content, subtype string for i, name := range sampleRegexp.SubexpNames() { if name == "type" { subtype = match[i] } if name == "content" && match[i] != "" { content = match[i] } } if subtype == "yaml" && subsetRegexp.FindString(content) != "" { t.Logf("skipping (%s): n%s", subtype, content) continue } //t.Logf("testing (%s): n%s", subtype, content) expectedType := &api.Pod{} if err := latest.Codec.DecodeInto([]byte(content), expectedType); err != nil { t.Errorf("%s did not decode correctly: %vn%s", path, err, string(content)) continue } if errors := validateObject(expectedType); len(errors) > 0 { t.Errorf("%s did not validate correctly: %v", path, errors) } _, err := latest.Codec.Encode(expectedType) if err != nil { t.Errorf("Could not encode object: %v", err) continue } } </ins> } }

Which jobs are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /priority important-soon /cc /cc /milestone v1.18 /sig release
/assign
This is failing because a 1.18 release for CRI tools doesn't exist yet. I can tweak the test job arguments to skip this later.
(As a note, this job is important, but not ) /remove-priority critical-urgent /priority important-soon
Do we need to remove the job from dashboard?
-- No. I'll work on fixing the job later today.
I've started to investigate, but I haven't yet reproduced the new error locally: This is likely related to a recent refactor of the package that handles interpreting deb/rpm package versions. As is not currently in use, I'm moving this to the milestone. /milestone v1.19
the version issue is fixed via The test is now failing because it tries to download cri-tools v1.18.0: which I think is related to this code part:
Updating testgrid link here: Any updates here /retitle [Flaky Test] build-packages-debs (ci-release-build-packages-debs) /kind flaky-test
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Updating testgrid link here: /kind flaky-test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake
I was assuming that is working on this issue. I can jump in if you want and do not find the time to work on it. :)
/assign /unassign Sascha -- Can you take this? I thought I was going to have time to look at this, but it's been a while.
Sure, the reason for the failing test seems to have changed, I'm on it.
Fix is incoming in


Question: "//test/images/nettest:all-srcs", "//test/images/no-snat-test:all-srcs", "//test/images/no-snat-test-proxy:all-srcs", <ins> "//test/images/nonewprivs:all-srcs", </ins> "//test/images/port-forward-tester:all-srcs", "//test/images/porter:all-srcs", "//test/images/resource-consumer:all-srcs",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature This issue is for creating image as multiarch
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing


Question: // NodeAddresses is an implementation of Instances.NodeAddresses. func (gce *GCECloud) NodeAddresses(instance string) ([]api.NodeAddress, error) { <del> externalIP, err := gce.metadataAccess(EXTERNAL_IP_METADATA_URL) </del> <ins> externalIP, err := gce.getExternalIP(instance) </ins> if err != nil { <del> return nil, err </del> <ins> return nil, fmt.Errorf("couldn't get external IP for instance %s: %v", instance, err) </ins> } <del> nodeAddresses := []api.NodeAddress{{Type: api.NodeExternalIP, Address: externalIP}} if legacyHostAddress, err := gce.getLegacyHostAddress(instance); err == nil { nodeAddresses = append(nodeAddresses, *legacyHostAddress) } return nodeAddresses, nil </del> <ins> return []api.NodeAddress{ {Type: api.NodeExternalIP, Address: externalIP}, // TODO(mbforbes): Remove NodeLegacyHostIP once v1beta1 is removed. {Type: api.NodeLegacyHostIP, Address: externalIP}, }, nil </ins> } <del> func (gce *GCECloud) getLegacyHostAddress(instance string) (*api.NodeAddress, error) { </del> <ins> func (gce *GCECloud) getExternalIP(instance string) (string, error) { </ins> inst, err := gce.getInstanceByName(instance) if err != nil { <del> return nil, err </del> <ins> return "", err </ins> } ip := net.ParseIP(inst.NetworkInterfaces[0].AccessConfigs[0].NatIP) if ip == nil { <del> return nil, fmt.Errorf("invalid network IP: %s", inst.NetworkInterfaces[0].AccessConfigs[0].NatIP) </del> <ins> return "", fmt.Errorf("invalid network IP: %s", inst.NetworkInterfaces[0].AccessConfigs[0].NatIP) </ins> } <del> return &api.NodeAddress{Type: api.NodeLegacyHostIP, Address: ip.String()}, nil </del> <ins> return ip.String(), nil </ins> } // ExternalID returns the cloud provider ID of the specified instance.

When we populate the addresses of the nodes in we call NodeAddresses, which for the gce provider gets the external IP from the which as expected puts the IP address of the master into the external IP field. The Legacy IP field is still filled in by asking the cloud provider for the instance data, so it contains the public IP of the node. /cc
If we merge this should get fixed, because the external IPs will be calculated on the nodes rather than on the master.
Since we are going to delay adding until I can rebase on top of it seems like it's worth fixing this separately.
This is blocking so assigning to him.


Question: func ValidatePersistentVolumeUpdate(newPv, oldPv *api.PersistentVolume) field.ErrorList { allErrs := field.ErrorList{} allErrs = ValidatePersistentVolume(newPv) <ins> // PersistentVolumeSource should be immutable after creation. if !apiequality.Semantic.DeepEqual(newPv.Spec.PersistentVolumeSource, oldPv.Spec.PersistentVolumeSource) { allErrs = append(allErrs, field.Forbidden(field.NewPath("spec", "persistentvolumesource"), "is immutable after creation")) } </ins> newPv.Status = oldPv.Status return allErrs }

/kind bug Right now we do not validate PV updates so admin can update at any time. Kubernetes has no facility to reflect this update - it won't magically re-mount an existing volume to e.g. a different NFS server. IMO the PersistentVolumeSource to be read-only and set to stone once created.
/assign
I think it is a good idea to make the VolumeSource read-only but I am afraid ship has sailed in v1 and existing code/tooling that relies on the ability to modify PVs will break as a result of this change.
- do you have any advice on how we should proceed on fixing this issue?


Question: gsutil ls -lhr "${gcs_destination}" || return 1 } <del> function kube::release::gcs::publish_latest() { local latest_file_dst="gs://${KUBE_GCS_RELEASE_BUCKET}/${KUBE_GCS_LATEST_FILE}" mkdir -p "${RELEASE_STAGE}/upload" || return 1 echo ${KUBE_GCS_LATEST_CONTENTS} > "${RELEASE_STAGE}/upload/latest" || return 1 </del> <ins> # Publish a new ci version, (latest,) but only if the release files actually # exist on GCS. # # Globals: #   See callees # Arguments: #   None # Returns: #   Success function kube::release::gcs::publish_ci() { kube::release::gcs::verify_release_files || return 1 kube::release::gcs::publish 'ci/latest.txt' || return 1 } </ins> <del> gsutil -m cp "${RELEASE_STAGE}/upload/latest" "${latest_file_dst}" || return 1 </del> <ins> # Publish a new official version, (latest or stable,) but only if the release # files actually exist on GCS and the release we're dealing with is newer than # the contents in GCS. # # Globals: #   KUBE_GCS_PUBLISH_VERSION #   See callees # Arguments: #   release_kind: either 'latest' or 'stable' # Returns: #   Success function kube::release::gcs::publish_official() { local -r release_kind="${1-}" kube::release::gcs::verify_release_files || return 1 kube::release::parse_and_validate_release_version "${KUBE_GCS_PUBLISH_VERSION}" || return 1 local -r version_major="${BASH_REMATCH[1]}" local -r version_minor="${BASH_REMATCH[2]}" </ins> <del> if [[ ${KUBE_GCS_MAKE_PUBLIC} =~ ^[yY]$ ]]; then gsutil acl ch -R -g all:R "${latest_file_dst}" >/dev/null 2>&1 || return 1 </del> <ins> local publish_files if [[ "${release_kind}" == 'latest' ]]; then publish_files=(release/latest.txt release/latest-${version_major}.txt release/latest-${version_major}.${version_minor}.txt) elif [[ "${release_kind}" == 'stable' ]]; then publish_files=(release/stable.txt release/stable-${version_major}.txt release/stable-${version_major}.${version_minor}.txt) else kube::log::error "Wrong release_kind: must be 'latest' or 'stable'." return 1 </ins> fi <del> kube::log::status "gsutil cat ${latest_file_dst}:" gsutil cat ${latest_file_dst} || return 1 </del> <ins> for publish_file in ${publish_files[*]}; do # If there's a version that's above the one we're trying to release, don't # do anything, and just try the next one. kube::release::gcs::verify_release_gt "${publish_file}" || continue kube::release::gcs::publish "${publish_file}" || return 1 done </ins> } <del> # Publish a new latest.txt, but only if the release we're dealing with # is newer than the contents in GCS. function kube::release::gcs::publish_latest_official() { local -r new_version=${KUBE_GCS_LATEST_CONTENTS} local -r latest_file_dst="gs://${KUBE_GCS_RELEASE_BUCKET}/${KUBE_GCS_LATEST_FILE}" local -r version_regex="^v(0|[1-9][0-9]*).(0|[1-9][0-9]*).(0|[1-9][0-9]*)$" [[ ${new_version} =~ ${version_regex} ]] || { kube::log::error "publish_latest_official passed bogus value: '${new_version}'" </del> <ins> # Verify that the release files we expect actually exist. # # Globals: #   KUBE_GCS_RELEASE_BUCKET #   KUBE_GCS_RELEASE_PREFIX # Arguments: #   None # Returns: #   If release files exist function kube::release::gcs::verify_release_files() { local -r release_dir="gs://${KUBE_GCS_RELEASE_BUCKET}/${KUBE_GCS_RELEASE_PREFIX}" if ! gsutil ls "${release_dir}" >/dev/null 2>&1 ; then kube::log::error "Release files don't exist at '${release_dir}'" </ins> return 1 <del> } </del> <ins> fi } # Check if the new version is greater than the version currently published on # GCS. # # Globals: #   KUBE_GCS_PUBLISH_VERSION #   KUBE_GCS_RELEASE_BUCKET # Arguments: #   publish_file: the GCS location to look in # Returns: #   If new version is greater than the GCS version function kube::release::gcs::verify_release_gt() { local -r publish_file="${1-}" local -r new_version=${KUBE_GCS_PUBLISH_VERSION} local -r publish_file_dst="gs://${KUBE_GCS_RELEASE_BUCKET}/${publish_file}" kube::release::parse_and_validate_release_version "${new_version}" || return 1 </ins> local -r version_major="${BASH_REMATCH[1]}" local -r version_minor="${BASH_REMATCH[2]}" local -r version_patch="${BASH_REMATCH[3]}" local gcs_version <del> gcs_version=$(gsutil cat "${latest_file_dst}") [[ ${gcs_version} =~ ${version_regex} ]] || { kube::log::error "${latest_file_dst} contains invalid release version, can't compare: '${gcs_version}'" return 1 } </del> <ins> if gcs_version="$(gsutil cat "${publish_file_dst}")"; then kube::release::parse_and_validate_release_version "${gcs_version}" || { kube::log::error "${publish_file_dst} contains invalid release version, can't compare: '${gcs_version}'" return 1 } </ins> <del> local -r gcs_version_major="${BASH_REMATCH[1]}" local -r gcs_version_minor="${BASH_REMATCH[2]}" local -r gcs_version_patch="${BASH_REMATCH[3]}" local greater=true if [[ "${gcs_version_major}" -gt "${version_major}" ]]; then greater=false elif [[ "${gcs_version_major}" -lt "${version_major}" ]]; then : # fall out elif [[ "${gcs_version_minor}" -gt "${version_minor}" ]]; then greater=false elif [[ "${gcs_version_minor}" -lt "${version_minor}" ]]; then : # fall out elif [[ "${gcs_version_patch}" -ge "${version_patch}" ]]; then greater=false fi </del> <ins> local -r gcs_version_major="${BASH_REMATCH[1]}" local -r gcs_version_minor="${BASH_REMATCH[2]}" local -r gcs_version_patch="${BASH_REMATCH[3]}" local greater=true if [[ "${gcs_version_major}" -gt "${version_major}" ]]; then greater=false elif [[ "${gcs_version_major}" -lt "${version_major}" ]]; then : # fall out elif [[ "${gcs_version_minor}" -gt "${version_minor}" ]]; then greater=false elif [[ "${gcs_version_minor}" -lt "${version_minor}" ]]; then : # fall out elif [[ "${gcs_version_patch}" -ge "${version_patch}" ]]; then greater=false fi </ins> <del> if [[ "${greater}" != "true" ]]; then kube::log::status "${gcs_version} (latest on GCS) >= ${new_version} (just uploaded), not updating ${latest_file_dst}" </del> <ins> if [[ "${greater}" != "true" ]]; then kube::log::status "${new_version} (just uploaded) <= ${gcs_version} (latest on GCS), not updating ${publish_file_dst}" return 1 else kube::log::status "${new_version} (just uploaded) > ${gcs_version} (latest on GCS), updating ${publish_file_dst}" fi else  # gsutil cat failed; file does not exist kube::log::error "Release file '${publish_file_dst}' does not exist.  Continuing." </ins> return 0 fi <ins> } </ins> <del> kube::log::status "${new_version} (just uploaded) > ${gcs_version} (latest on GCS), updating ${latest_file_dst}" kube::release::gcs::publish_latest || return 1 </del> <ins> # Publish a release to GCS: upload a version file, if KUBE_GCS_MAKE_PUBLIC, # make it public, and verify the result. # # Globals: #   KUBE_GCS_RELEASE_BUCKET #   RELEASE_STAGE #   KUBE_GCS_PUBLISH_VERSION #   KUBE_GCS_MAKE_PUBLIC # Arguments: #   publish_file: the GCS location to look in # Returns: #   If new version is greater than the GCS version function kube::release::gcs::publish() { local -r publish_file="${1-}" local -r publish_file_dst="gs://${KUBE_GCS_RELEASE_BUCKET}/${publish_file}" mkdir -p "${RELEASE_STAGE}/upload" || return 1 echo "${KUBE_GCS_PUBLISH_VERSION}" > "${RELEASE_STAGE}/upload/latest" || return 1 gsutil -m cp "${RELEASE_STAGE}/upload/latest" "${publish_file_dst}" || return 1 local contents if [[ ${KUBE_GCS_MAKE_PUBLIC} =~ ^[yY]$ ]]; then gsutil acl ch -R -g all:R "${publish_file_dst}" >/dev/null 2>&1 || return 1 # If public, validate public link local -r public_link="https://storage.googleapis.com/${KUBE_GCS_RELEASE_BUCKET}/${publish_file}" kube::log::status "Validating uploaded version file at ${public_link}" contents="$(curl -s "${public_link}")" else # If not public, validate using gsutil kube::log::status "Validating uploaded version file at ${publish_file_dst}" contents="$(gsutil cat "${publish_file_dst}")" fi if [[ "${contents}" == "${KUBE_GCS_PUBLISH_VERSION}" ]]; then kube::log::status "Contents as expected: ${contents}" else kube::log::error "Expected contents of file to be ${KUBE_GCS_PUBLISH_VERSION}, but got ${contents}" return 1 fi </ins> }

(cc Despite doing very, very similar things, doesn't rely on . It should, or the logic should otherwise be consolidated. I'm going to maybe work on this to make easier.