Question: "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", "type": "string" }, <del> "listMeta": { </del> <ins> "metadata": { </ins> "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta", "description": "Standard list metadata" }

Listing s on v1.31.0 returns an object with a field instead of the standard field for (list) objects: Listing s on v1.30.3 behaves correctly: This is also propagated to the Swagger spec (which is how I noticed it, since I maintain the Rust bindings for the client API). v1.31.0: v1.30.3: As explained above. v1.31.0 with feature gate enabled and runtime config . With kind, this is: the API server request as in the above curl command in any way you like as long as it lets you see the raw JSON response. No response <details</details<detailskind 0.23.0 </details<details</details<details</details<details</details<details</details
/sig api-machinery
It seems there may be a discrepancy in the API response format between v1.30.3 and v1.31.0. The object is expected to follow standard Kubernetes API conventions, which typically use the field for list objects.
This deviation might cause compatibility issues for API clients that rely on this structure.
If this deviation from the expected API structure wasn't intentional, it should be aligned with the standard conventions.
The presence of instead of could disrupt client integrations, as you've encountered with the Swagger spec and Rust bindings.
cc
/triage accepted
/assign
this must be fixed to json:"metadata" protobuf:"bytes,1,opt,name=metadata"`` the serialization will not work with standard clients
/priority critical-urgent
Shoot, that's clearly a mistake.


Question: "persistent_volumes.go", "persistent_volumes-gce.go", "persistent_volumes-local.go", <ins> "persistent_volumes-volumemode.go", </ins> "pv_protection.go", "pvc_protection.go", "regional_pd.go",

It would better to have e2e for iSCSI persistent volume. /sig storage
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: <del> {% if grains['os_family'] == 'RedHat' %} </del> <ins> {% if grains.network_mode is defined and grains.network_mode == 'openvswitch' %} </ins> openvswitch: pkg:

The salt configs assume that anything under "os_family = RedHat" will use the ovs mechanisms that are only set up for vagrant. This means that we can't use Fedora on GCE.
I can clean this up today.


Question: func (f *Factory) NewCmdRollingUpdate(out io.Writer) *cobra.Command { cmd := &cobra.Command{ Use:   "rollingupdate <old-controller-name> -f <new-controller.json>", <del> Short: "Perform a rolling update of the given replicationController", Long: `Perform a rolling update of the given replicationController.", </del> <ins> Short: "Perform a rolling update of the given ReplicationController", Long: `Perform a rolling update of the given ReplicationController. </ins> Replaces named controller with new controller, updating one pod at a time to use the new PodTemplate. The new-controller.json must specify the same namespace as the

Manually updated docs have a tendency to drift from actual implementation. It shouldn't be too hard to add a script to generate the markdown from a . We could then run the script manually or as a commit hook when changes are made to pkg/kubectl.
A big chunk of that is cut/paste from the help output anyway.. Brendan On Jan 9, 2015 4:20 PM, "Jeff Lowdermilk" wrote:
Why are we including the help output? Seems like cobra should do that, or we should just tell the user to ask for help if they want it.
It's useful to see generated markdown of help output for the full command tree, e.g. if you want to link to the syntax on irc or something. Anyway, this was quite easy to do, I'll have a PR shortly.


Question: Expect(container.Create()).To(Succeed()) defer container.Delete() <del> By("check the pod phase") Eventually(container.GetPhase, retryTimeout, pollInterval).Should(Equal(testCase.phase)) Consistently(container.GetPhase, consistentCheckTimeout, pollInterval).Should(Equal(testCase.phase)) </del> <ins> // We need to check container state first. The default pod status is pending, If we check // pod phase first, and the expected pod phase is Pending, the container status may not // even show up when we check it. </ins> By("check the container state") <del> status, err := container.GetStatus() Expect(err).NotTo(HaveOccurred()) Expect(GetContainerState(status.State)).To(Equal(testCase.state)) </del> <ins> getState := func() (ContainerState, error) { status, err := container.GetStatus() if err != nil { return ContainerStateUnknown, err } return GetContainerState(status.State), nil } Eventually(getState, retryTimeout, pollInterval).Should(Equal(testCase.state)) Consistently(getState, consistentCheckTimeout, pollInterval).Should(Equal(testCase.state)) By("check the pod phase") Expect(container.GetPhase()).To(Equal(testCase.phase)) </ins> By("it should be possible to delete") Expect(container.Delete()).To(Succeed())

https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull from private registry without secret {E2eNode Suite}
Same with , the fix is under review
https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull from private registry without secret {E2eNode Suite}
https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull from private registry without secret {E2eNode Suite}
The fix has been merged , close this one. Feel free to reopen if this happens again.


Question: Expect(container.Create()).To(Succeed()) defer container.Delete() <del> By("check the pod phase") Eventually(container.GetPhase, retryTimeout, pollInterval).Should(Equal(testCase.phase)) Consistently(container.GetPhase, consistentCheckTimeout, pollInterval).Should(Equal(testCase.phase)) </del> <ins> // We need to check container state first. The default pod status is pending, If we check // pod phase first, and the expected pod phase is Pending, the container status may not // even show up when we check it. </ins> By("check the container state") <del> status, err := container.GetStatus() Expect(err).NotTo(HaveOccurred()) Expect(GetContainerState(status.State)).To(Equal(testCase.state)) </del> <ins> getState := func() (ContainerState, error) { status, err := container.GetStatus() if err != nil { return ContainerStateUnknown, err } return GetContainerState(status.State), nil } Eventually(getState, retryTimeout, pollInterval).Should(Equal(testCase.state)) Consistently(getState, consistentCheckTimeout, pollInterval).Should(Equal(testCase.state)) By("check the pod phase") Expect(container.GetPhase()).To(Equal(testCase.phase)) </ins> By("it should be possible to delete") Expect(container.Delete()).To(Succeed())

https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull non-existing image from {E2eNode Suite}
https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull non-existing image from {E2eNode Suite}
https://k8s- Failed: [] Container Runtime Conformance Test container runtime conformance blackbox test when running a container with a new image should not be able to pull non-existing image from {E2eNode Suite}
The fix has been merged , close this one. Feel free to reopen if this happens again.


Question: return allErrs } <ins> func isKubernetesDomainPrefix(key string) bool { domainPrefix := getDomainPrefix(key) if domainPrefix == "kubernetes.io" || strings.HasSuffix(domainPrefix, ".kubernetes.io") { return true } if domainPrefix == "k8s.io" || strings.HasSuffix(domainPrefix, ".k8s.io") { return true } return false } func getDomainPrefix(key string) string { if parts := strings.SplitN(key, "/", 2); len(parts) == 2 { return parts[0] } return "" } </ins> func usesEmailClaim(ast *celgo.Ast) bool { return hasSelectExp(ast.Expr(), "claims", "email") }

Disallow setting all and namespaced extra info in custom ways for now. /assign aramase enj /sig auth
/triage accepted


Question: if !found { return nil, errors.New("couldn't find container") } <del> return kl.statsFromContainerPath(fmt.Sprintf("/docker/%s", dockerContainer.ID), req) </del> <ins> return kl.statsFromContainerPath(cc, fmt.Sprintf("/docker/%s", dockerContainer.ID), req) </ins> } // GetRootInfo returns stats (from Cadvisor) of current machine (root container). func (kl *Kubelet) GetRootInfo(req *info.ContainerInfoRequest) (*info.ContainerInfo, error) { <del> return kl.statsFromContainerPath("/", req) </del> <ins> cc := kl.GetCadvisorClient() if cc == nil { return nil, fmt.Errorf("no cadvisor connection") } return kl.statsFromContainerPath(cc, "/", req) </ins> } func (kl *Kubelet) GetMachineInfo() (*info.MachineInfo, error) { <del> return kl.cadvisorClient.MachineInfo() </del> <ins> cc := kl.GetCadvisorClient() if cc == nil { return nil, fmt.Errorf("no cadvisor connection") } return cc.MachineInfo() </ins> } func (kl *Kubelet) healthy(podFullName string, currentState api.PodState, container api.Container, dockerContainer *docker.APIContainers) (health.Status, error) {

So today, cadvisor became wedged and just stopped paying attention to its socket. Not kubelet's fault. But when I would launch all I would get would be: It took me a bit of trolling through netstat and lsof to figure out that the problem was cadvisor being completely locked up. kill -9 cadvisor and everything started working well. I wonder if we should emit more messages during startup about what operation we are about to attempt, not just the results of operations, as I was blaming docker for a while....
kubelet shouldn't block on this. I'll take a look.
Any ideas what was locking cAdvisor? and was it just not responding to requests?
I don't have any idea why it was locked. If I find another one locked I'll hang onto it to debug. Journal had: obviously the 'kill' came from me. It would not respond to systemctl cadvisor stop. netstat looking at port 4194 showed me At which point I realized cadvisor wasn't handling its sockets right since so many long dead kubelet sockets hadn't been cleaned up, so I kill -9'd it.
I cut some interesting info from the cadvisor journal log (trying to cut hostnames) You can see the last message about failed to update stats for container ... happened Oct 08 01:06:01. It was then silent until I killed it Oct 08 19:20:13, even though I'm pretty sure dnf was still being launched every hour on the hour. So it was wedged for about 18 hours.
What kernel is this on? We've seen there is a kernel bug fixed in 3.12+ that sometimes hangs us forever on exit. We're not quite sure how it gets triggered...
16.3


Question: # Command kubectl get pods --all-namespaces --namespace=default <ins> ### Check --all-namespaces option shows namespaces # Create objects in multiple namespaces kubectl create "${kube_flags[@]}" namespace all-ns-test-1 kubectl create "${kube_flags[@]}" serviceaccount test -n all-ns-test-1 kubectl create "${kube_flags[@]}" namespace all-ns-test-2 kubectl create "${kube_flags[@]}" serviceaccount test -n all-ns-test-2 # Ensure listing across namespaces displays the namespace output_message=$(kubectl get serviceaccounts --all-namespaces "${kube_flags[@]}") kube::test::if_has_string "${output_message}" "all-ns-test-1" kube::test::if_has_string "${output_message}" "all-ns-test-2" # Clean up kubectl delete "${kube_flags[@]}" namespace all-ns-test-1 kubectl delete "${kube_flags[@]}" namespace all-ns-test-2 </ins> ### Clean up # Pre-condition: valid-pod exists kube::test::get_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" 'valid-pod:'

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: namespace disappears in the output of What you expected to happen: namespace exists How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: It is related to Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli


Question: import ( "os/exec" <ins> "path/filepath" </ins> "regexp" "time"


Question: } func copyFromPod(f cmdutil.Factory, cmd *cobra.Command, cmderr io.Writer, src, dest fileSpec) error { <del> if len(src.File) == 0 { </del> <ins> if len(src.File) == 0 || len(dest.File) == 0 { </ins> return errFileCannotBeEmpty }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: panics What you expected to happen: Syntax error, as it is not legit cp syntax. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Occurs in every version 1.9.x and 1.10.x, but 1.8.x is safe. is similar, but it has a fix candidate already going in, that is not fixing this, so opening a separate issue. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: VM OS (e.g. from /etc/os-release): Ubuntu 18.04 Kernel (e.g. ): /sig cli
It seems to be fixed in 1.11 by
It was merged three months ago, any idea why it isn't in 1.10 too?
It was merged on March 31 and 1.10 was released on March 27: Can confirm it is not present in 1.10: Maybe we can cherry-pick it?
I've opened
Fixed by /close
Closing this issue. <detailsIn response to : Fixed by Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: kube::log::info_from_stdin() { local messages=() while read -r line; do <del> messages+=$line </del> <ins> messages+=("$line") </ins> done kube::log::info "${messages[@]}"

Example: I have a fixing PR ready, will send it in a moment.
/sig docs /assign closing because linked pr fixed this
/close


Question: [vagrant@kubernetes-minion-1] $ sudo journalctl -r -u kubelet ``` <ins> ### Interacting with your Kubernetes cluster with Vagrant. With your Kubernetes cluster up, you can manage the nodes in your cluster with the regular Vagrant commands. </ins> To push updates to new Kubernetes code after making source changes: ``` vagrant provision ``` <del> To shutdown and then restart the cluster: </del> <ins> To stop and then restart the cluster: </ins> ``` vagrant halt vagrant up

Which jobs are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /priority important-soon /cc /cc /milestone v1.18 /sig release
/assign
This is failing because a 1.18 release for CRI tools doesn't exist yet. I can tweak the test job arguments to skip this later.
(As a note, this job is important, but not ) /remove-priority critical-urgent /priority important-soon
Do we need to remove the job from dashboard?
-- No. I'll work on fixing the job later today.
I've started to investigate, but I haven't yet reproduced the new error locally: This is likely related to a recent refactor of the package that handles interpreting deb/rpm package versions. As is not currently in use, I'm moving this to the milestone. /milestone v1.19
the version issue is fixed via The test is now failing because it tries to download cri-tools v1.18.0: which I think is related to this code part:
Updating testgrid link here: Any updates here /retitle [Flaky Test] build-packages-debs (ci-release-build-packages-debs) /kind flaky-test
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Updating testgrid link here: /kind flaky-test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake
I was assuming that is working on this issue. I can jump in if you want and do not find the time to work on it. :)
/assign /unassign Sascha -- Can you take this? I thought I was going to have time to look at this, but it's been a while.
Sure, the reason for the failing test seems to have changed, I'm on it.
Fix is incoming in


Question: for _, test := range storagePodsTests { t.Run(test.name, func(t *testing.T) { <ins> for k, v := range test.features { defer featuregatetesting.SetFeatureGateDuringTest(t, feature.DefaultFeatureGate, k, v)() } </ins> node := v1.Node{Status: v1.NodeStatus{Capacity: makeResources(10, 20, 32, 5, 20, 5).Capacity, Allocatable: makeAllocatableResources(10, 20, 32, 5, 20, 5)}} test.nodeInfo.SetNode(&node)

What happened: In a v1.18 cluster with the LocalStorageCapacityIsolation feature gate disabled setting an ephemeral-storage request or limit causes the pod to be unschedulable: If the ephemeral-storage request is removed the pod is scheduled. This behavior exists in v1.19 and a recent v1.20 build. What you expected to happen: In previous releases such pods are scheduled. I expect that any ephemeral-storage limit is not enforced. How to reproduce it (as minimally and precisely as possible): Disable the LocalStorageCapacityIsolation feature guide by adding to the kube-scheduler. Create a pod like this: Anything else we need to know?: The problem was originally seen after moving from IBM Cloud Red Hat OpenShift v4.4 to v4.5. Pods that worked in 4.4 (Kubernetes v1.17) remained in a Pending state in OpenShift v4.5 (Kubernetes 1.18). We discovered that the IBM Cloud OpenShift service has the LocalStorageCapacityIsolation feature-gate disabled while Red Hat has enabled the feature. We then replicated the behavior with a plain Kubernetes cluster. Environment: Kubernetes version (use ): v1.18.10, v1.19.3, v1.20.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-scheduling
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
I can confirm it's a bug. Will work on it.


Question: for _, test := range storagePodsTests { t.Run(test.name, func(t *testing.T) { <ins> for k, v := range test.features { defer featuregatetesting.SetFeatureGateDuringTest(t, feature.DefaultFeatureGate, k, v)() } </ins> node := v1.Node{Status: v1.NodeStatus{Capacity: makeResources(10, 20, 32, 5, 20, 5).Capacity, Allocatable: makeAllocatableResources(10, 20, 32, 5, 20, 5)}} test.nodeInfo.SetNode(&node)

/assign
Clarifying the recreate steps: I disabled the LocalStorageCapacityIsolation feature guide by adding to the kube-scheduler since this behavior seems to be within the scheduler.
Thanks for confirming!
This bug has been fixed in master branch, and will be backported to [1.18] and [1.19] soon. [1.18]: [1.19]:


Question: if filePath == m.checkpointFile() { continue } <del> stat, err := os.Stat(filePath) </del> <ins> // TODO: Until the bug - https://github.com/golang/go/issues/33357 is fixed, os.stat wouldn't return the // right mode(socket) on windows. Hence deleting the file, without checking whether // its a socket, on windows. stat, err := os.Lstat(filePath) </ins> if err != nil { klog.Errorf("Failed to stat file %s: %v", filePath, err) continue

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Flakes: Full failure: Which test(s) are failing: Since when has it been failing: Since merging Testgrid link: Reason for failure: Serial slow tests are failing with: and the parallel tests are flaking with: Anything else we need to know: I would not expect kubelet to crash if the node if the plugin is not present. /sig windows cc:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It seems to me that it is not related to a plugin being not present, but that it cannot open the endpoint where potential plugins could register to. In other words, I suspect this error happens independently from any plugins. If we can trust this message , then this would indicate that it tries to open that listening socket twice. This would mean that is called twice. If this is the case, the question would be: Why? It would be interesting to see if shows twice.
There is an issue on some Windows not updated. When you restart kubelet and the socket file So the workarround, is to remove before restart kubeblet
And why tests on the PR didn't trigger this error ?
Question: Is it legal that is called twice? It seems that the function is called from , which is analogous to the Linux implementation Calling twice looks fishy to me. It seems to me that this problem is not caused by, but only exposed by the device plugin change
You mean that kubelet is initialized twice on the same computer, causing this error the second time? How is this handled on Linux? If this is allowed we could simply delete if present before opening it again.
Are you sure it's call twice ? because on windows is not present but replace by you can see the first line comment and .


Question: if filePath == m.checkpointFile() { continue } <del> stat, err := os.Stat(filePath) </del> <ins> // TODO: Until the bug - https://github.com/golang/go/issues/33357 is fixed, os.stat wouldn't return the // right mode(socket) on windows. Hence deleting the file, without checking whether // its a socket, on windows. stat, err := os.Lstat(filePath) </ins> if err != nil { klog.Errorf("Failed to stat file %s: %v", filePath, err) continue

A simple test is: kubelet and check the socket file is still present kubelet you got the error, stop it socket file kubelet again, normally it's working Maybe the fix can be add in
I think your explanation is very plausible: That it is not called twice within a single kubelet initialization, but kubelet is initialized again for some reason on the same node. (It still means that it is called twice, but I didn't think about the possibility that kubelet is initialized twice.)
There are various ways of fixing it. But I would be interested in learning how this is handled under Linux. To choose the same path for Windows, too.
Thanks for your quick response and analysis :+1: It would be interesting to see if shows twice. You can find the kubelet logs (and many others) for the windows nodes in the "artifacts" tab on This set of tests runs configuration for GMSA tests which as part of the set up of the domain controller, causes a node resets. This explains why one of the nodes seems ok (it isn't randomly selected as part of the domain machines) and why the parallel tests don't cause a full failure (they don't cause node resets). Not all clusters use that script. This set of tests don't use them either (uses aks-engine which has a different start up script). IMHO this is a breaking change scince any cluster that has a node restart would fail to come back online. and kubelet should be able to restart (because of node restart or crash for another reason, etc.) It looks like there is code to clean up old sockets but it is not working on windows: Given there is clean up code there and it's a breaking change I think this this should be fixed in kubelet.
Oh yes, I missed that! (I should have looked only a few lines above the line causing the error.) It at least looks exactly like the code we need. The function is implemented in the same file: But why it is failing to work under Windows? Do you have an idea? Yes, sounds good.
Ok so on windows failed here
So this exemple reproduce the error. the failed only on socket file not on regular file ! Link to
It's bug on windows 1903 not present on windows 1809 The fix can be replace by
Very cool!


Question: if filePath == m.checkpointFile() { continue } <del> stat, err := os.Stat(filePath) </del> <ins> // TODO: Until the bug - https://github.com/golang/go/issues/33357 is fixed, os.stat wouldn't return the // right mode(socket) on windows. Hence deleting the file, without checking whether // its a socket, on windows. stat, err := os.Lstat(filePath) </ins> if err != nil { klog.Errorf("Failed to stat file %s: %v", filePath, err) continue

Just for interest, how did you find
I hate Windows ;-) and I googled
It's bug on windows 1903 golang/go (comment) not present on windows 1809 This set of tests are running on 2019 LTSC (also known as 1809). If this is only an issue on 1903 is there something else happening?
Sorry, I read to quickly, there is a change of behaviour between 1809 and 1903 about unix socket, If my understanding is good on 1903 unix socket is like regular file but not on 1809, so it's why os.Stat failed on 1809. but I tested on Windows 10 20H2 and the issue still here with
So in summary, the key point is: PR fixes this issue


Question: github.com/satori/go.uuid v1.2.0 // indirect github.com/stretchr/objx v0.1.1 // indirect github.com/stretchr/testify v1.2.2 <del> github.com/vmware/govmomi v0.20.0 </del> <ins> github.com/vmware/govmomi v0.20.1 </ins> golang.org/x/crypto v0.0.0-20181025213731-e84da0312774 golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a google.golang.org/api v0.0.0-20181220000619-583d854617af

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The vSphere cloud provider is unable to fetch the node failure domain labels when using VCP with SAML auth because of an authentication failure. For context, the SAML token auth is described here - SAML auth support for the zones api was introduced in this PR - , which was cherry-picked to 1.14.1 here - I've tested the credentials I'm using with govc to ensure that I'm able to get a token: The credentials also work fine when using VCP without the node labels feature. Relevant kubelet logs: What you expected to happen: VCP should add failure domain labels to the node object. How to reproduce it (as minimally and precisely as possible): Setup SAML auth for VCP with zone labeling enabled. Anything else we need to know?: There was a previous issue that was raised () but that was before SAML auth support for zones was implemented. This could potentially be an issue with govc handling of SAML tokens since even with the Administrator role assigned to the user, I'm still unable to list tags. The role and keys were generated with the following: Environment: Kubernetes version (use ): 14.1 Cloud provider or hardware configuration: vSphere OS (e.g: ): Linux & Windows
/sig vmware
Would you mind taking a look at this? I can provide more info if needed.
govc's command only works with the SOAP endpoint currently. The REST endpoint has its own session manager and requires a separate login. At the moment, govc's tags commands login/logout when each command is run and don't use a persisted session (which are saved to ~/.govmomi/sessions). However, you can use a token returned by to test SAML auth with the tags commands. For example: Can you give that try? There are some TODOs around this btw:
Those commands work for me. So that would imply the solution user, role, and permissions are set up correctly? Would you expect the to also work?


Question: github.com/satori/go.uuid v1.2.0 // indirect github.com/stretchr/objx v0.1.1 // indirect github.com/stretchr/testify v1.2.2 <del> github.com/vmware/govmomi v0.20.0 </del> <ins> github.com/vmware/govmomi v0.20.1 </ins> golang.org/x/crypto v0.0.0-20181025213731-e84da0312774 golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a google.golang.org/api v0.0.0-20181220000619-583d854617af

I seem to be getting a 404 when using with the flag - Does seem to work when specifying a tag name: Are there any other govc commands I should check / verify?
yes, the solution user setup looks right. Not sure what the issue is with . Can you share your vCenter build number ()? I'll try to reproduce the issue.
$ govc about -u vcdo Name: VMware vCenter Server Vendor: VMware, Inc. Version: 6.7.0 Build: OS type: linux-x64 API type: VirtualCenter API version: 6.7.2 Product ID: vpx UUID: -55ed-4737-9cc4-
I can reproduce this, but don't know what the issue is yet. The version of Go and vSphere have changed since , but if it was one of those was the cause, I'd expect SAML auth to fail with the govc tags commands too. All I have to go on for now is the vCenter log message below. I'll update when I have more info.
I have it working with the change below, but need to figure out a few things first: Why this is the case when run inside kubelet and not govc (note that the port change to 443 breaks govc) A good way to determine which port to use (depends on the answer to the above) If this is behavior is specific to 6.7U2 or something triggered by a change in K8s I suppose a quick fix would change to try port 443 if 80 fails, but would prefer to avoid that as the failure generates stack traces in the
/assign
This is obvious in retrospect, but the issue is the message signature needs to use the same port as the original request URL. However, in the govc case there was no port set in the request URL. The client defaults to port when connecting (based on the scheme), but does not include any port with the header. In the case, the default is on the server side. This can be reproduced with if includes the port , rather than just as we were using. The VCP tests that use token auth for tags lookup just don't validate the signature, so this wasn't a change related to 6.7U2.


Question: github.com/satori/go.uuid v1.2.0 // indirect github.com/stretchr/objx v0.1.1 // indirect github.com/stretchr/testify v1.2.2 <del> github.com/vmware/govmomi v0.20.0 </del> <ins> github.com/vmware/govmomi v0.20.1 </ins> golang.org/x/crypto v0.0.0-20181025213731-e84da0312774 golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a google.golang.org/api v0.0.0-20181220000619-583d854617af

I'll open a K8s PR to pull in the fix:
Updating k/vendor to govmomi master adds almost 6K lines for a 3 line fix. I may just create a govmomi branch off of v0.20 to reduce churn in the K8s release branches for this fix, may have to wait until tomorrow.
Really appreciate the insight into this issue. Can confirm that specifying the port (443) in the GOVC_URL reproduces the issue. Thank you!
Unfortunately, I'm seeing the same issue with v1.14.3: cloud_config Works fine when I use the username and password of a vsphere admin, but using cert and key still has authentication issues.
any related errors in vCenter's ?
Seeing a lot of this exception:
whoa! that stack trace is like punch in the gut (brings back memories!) :)
indeed!! apologies, I must have had the change for PR locally, but failed to include it as I had focused on the signature issue and updating vendor/


Question: location.Scheme = "https" rloc, err := r.serviceResolver.ResolveEndpoint(handlingInfo.serviceNamespace, handlingInfo.serviceName) if err != nil { <del> http.Error(w, fmt.Sprintf("missing route (%s)", err.Error()), http.StatusInternalServerError) </del> <ins> glog.Errorf("error resolving %s/%s: %v", handlingInfo.serviceNamespace, handlingInfo.serviceName, err) http.Error(w, "service unavailable", http.StatusServiceUnavailable) </ins> return } location.Host = rloc.Host

https://k8s- https://k8s- https://k8s- https://k8s- https://k8s- ... A lot of them /cc
/assign
Still see this failure after is merged, although less frequently: https://k8s- https://k8s- Reopen.
Looks like the client doesn't differentiate between 500 and 503 here: It should. Will open a fix shortly
opened
fix merged, will monitor https://k8s-
Thanks! :)
haven't seen this in the 12 hours since the fix merged, was seeing it every 2-3 hours before. /close


Question: t.Errorf("Expect an NotFound error, got %v", err) } } <ins> func TestPrefix(t *testing.T) { scheme, codecs := testScheme(t) codec := apitesting.TestCodec(codecs, examplev1.SchemeGroupVersion) server := etcdtesting.NewEtcdTestClientServer(t) defer server.Terminate(t) testcases := map[string]string{ "custom/prefix":     "/custom/prefix", "/custom//prefix//": "/custom/prefix", "/registry":         "/registry", } for configuredPrefix, effectivePrefix := range testcases { helper := newEtcdHelper(server.Client, scheme, codec, configuredPrefix) if helper.pathPrefix != effectivePrefix { t.Errorf("configured prefix of %s, expected effective prefix of %s, got %s", configuredPrefix, effectivePrefix, helper.pathPrefix) } } } </ins>

an API server with , populate with data, stop server etcd2 to etcd3 the API server with data is missing from the API. The etcd2 client takes the configured prefix and passes it to the etcd client as . The etcd3 client simply passes in the configured prefix. is a no-op on the default prefix (`/registry), but can modify custom prefixes that did not start with a slash. cc
I think we have to preserve the behavior from the etcd2 client setup. It's a no-op for the default prefix, but requiring a change to a custom prefix when switching from etcd2 to etcd3 is really unexpected, especially when the failure mode is that the apiserver starts up as if it's a brand new cluster (including creating default projects, kubernetes service, etc)
cc
/cc re: upgrade testing.
Throwing release blocker on here just so we don't lose it (given data loss potential)
- thank for debugging and fixing it.


Question: } } <ins> func podToEndpointAddress(pod *v1.Pod) *v1.EndpointAddress { return &v1.EndpointAddress{ IP:       pod.Status.PodIP, NodeName: &pod.Spec.NodeName, TargetRef: &v1.ObjectReference{ Kind:            "Pod", Namespace:       pod.ObjectMeta.Namespace, Name:            pod.ObjectMeta.Name, UID:             pod.ObjectMeta.UID, ResourceVersion: pod.ObjectMeta.ResourceVersion, }} } func podAddressChanged(oldPod, newPod *v1.Pod) bool { // Convert the pod to an EndpointAddress, clear inert fields, // and see if they are the same. newEndpointAddress := podToEndpointAddress(newPod) oldEndpointAddress := podToEndpointAddress(oldPod) // Ignore the ResourceVersion because it changes // with every pod update. This allows the comparison to // show equality if all other relevant fields match. newEndpointAddress.TargetRef.ResourceVersion = "" oldEndpointAddress.TargetRef.ResourceVersion = "" if reflect.DeepEqual(newEndpointAddress, oldEndpointAddress) { // The pod has not changed in any way that impacts the endpoints return false } return true } func determineNeededServiceUpdates(oldServices, services sets.String, podChanged bool) sets.String { if podChanged { // if the labels and pod changed, all services need to be updated services = services.Union(oldServices) } else { // if only the labels changed, services not common to // both the new and old service set (i.e the disjunctive union) // need to be updated services = services.Difference(oldServices).Union(oldServices.Difference(services)) } return services } </ins> // When a pod is updated, figure out what services it used to be a member of // and what services it will be a member of, and enqueue the union of these. // old and cur must be *v1.Pod types.

/kind bug What happened: On a large cluster with enough pods, endpoints take as long as 10 minutes to show up after their associated pods and service are created. We believe that this is because every pod change results in an update of the addresses within the endpoint since they refer to a pod via a whose changes every time a pod change is observed. All these non-update updates eventually overwhelm the endpoint controller's queue rate limit and the queue starts to grow. While an increase in the queue's rate limit can alleviate the problem, an admin can keep a much lower limit if the non-update updates are dropped rather than processed. What you expected to happen: When a pod related to an endpoint is updated, I would expect the endpoint to update only if there is a change in the pod that relates to the endpoint, such as a change in the pod's IP address. How to reproduce it (as minimally and precisely as possible): a pod and associated service: the endpoint for changes: the watch running, and in another session, edit a pod to add a label: Then add another label on the line after the label, such as that the watched endpoint changed despite there not being any meaningful change to the fields the endpoint cares about. Anything else we need to know?: The PR addresses this issue. Environment: Kubernetes version (use ):
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig scalability
what ever happened with the effort to move leases off of Endpoints?


Question: masqueradeValue := 1 << uint(masqueradeBit) masqueradeMark := fmt.Sprintf("%#08x/%#08x", masqueradeValue, masqueradeValue) <del> if nodeIP == nil { klog.Warningf("invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP") nodeIP = net.ParseIP("127.0.0.1") } </del> isIPv6 := utilnet.IsIPv6(nodeIP) klog.V(2).Infof("nodeIP: %v, isIPv6: %v", nodeIP, isIPv6)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using Kubernetes 1.16.1, I started kube-proxy in iptables mode as a static pod on a master node. The master node was intentionally (and permanently) not available in the API as a Node object. kube-proxy started up and terminated soon after because it failed to (which is supposed to function as the bound address) and eventually returned an error: What you expected to happen: I expected the same behavior as in Kubernetes 1.15 where failure to look up the Node in the API would result in . How to reproduce it (as minimally and precisely as possible): Configure kube-proxy with iptables mode on Kubernetes 1.16 without the hosting Node being available in the Kubernetes API (under the OS hostname). Anything else we need to know?: PR (Fix for kube-proxy to wait for some duration for the node to be defined) introduced an additional . I believe this is the part that is causing the change in behavior: in Kubernetes 1.15, the check would be omitted and cause to set the default bind address to 127.0.0.1 (when iptables mode is set; I haven't checked other modes). The PR's primary purpose seems to be a different one (retrying node IP retrieval from the API), so my understanding is that the check was not really needed; unless I'm misunderstanding things which I don't want to rule out at all. The quick workaround was to explicitly configure kube-proxy with . However, I'm not entirely sure if that's a good solution long-term compared to letting Kubernetes do The Right Thing, which is what I'd like to clarify through this issue. Environment: Kubernetes version (use ): v1.16.1 Cloud provider or hardware configuration: Kubernetes running on DigitalOcean VMs (droplets) OS (e.g: ): Debian GNU/Linux 9 (stretch) Kernel (e.g.


Question: masqueradeValue := 1 << uint(masqueradeBit) masqueradeMark := fmt.Sprintf("%#08x/%#08x", masqueradeValue, masqueradeValue) <del> if nodeIP == nil { klog.Warningf("invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP") nodeIP = net.ParseIP("127.0.0.1") } </del> isIPv6 := utilnet.IsIPv6(nodeIP) klog.V(2).Infof("nodeIP: %v, isIPv6: %v", nodeIP, isIPv6)

): Linux master-k8s-1-16-test1 4.19.0-0.bpo.6-amd64 SMP Debian 4.19.67-2+deb10u1~bpo9+1 (2019-09-30) x86_64 GNU/Linux /sig network /cc who were involved in the referenced PR.
I made a similar comment in , wondering if this is a breaking change in v1.16 that we should revert? Previously if is nil, kube-proxy defaults to but now kube-proxy always exits with error instead of defaulting to local host. I think a lot of external cloud providers relied on this behavior because they needed to talk to API server prior to node addresses being set by the provider. Thoughts /assign
Opened to preserve behavior before v1.16 where node IP defaults to 127.0.01 is none is found. PTAL


Question: } func (r realJobControl) DeleteJob(namespace string, name string) error { <del> return r.KubeClient.BatchV1().Jobs(namespace).Delete(name, nil) </del> <ins> background := metav1.DeletePropagationBackground return r.KubeClient.BatchV1().Jobs(namespace).Delete(name, &metav1.DeleteOptions{PropagationPolicy: &background}) </ins> } type fakeJobControl struct {

What happened: There seems to be a regression after 1.13 release. A Cronjob with default settings does not clean up completed pods. The job themselves are getting deleted as expected, but the pods are left behind. This seems to be related to What you expected to happen: Expected the pods for now non-existent child jobs of a cronjob to be cleaned up. This is the way it used to work prior to 1.13/ How to reproduce it (as minimally and precisely as possible): run on this manifest: Wait, more than 3 minutes, observe, that while only 3 successful jobs are present, the pods from now deleted jobs are left behind and never get deleted. Anything else we need to know?: pointed out the following relevant places in the code base: deleting a job orphans pods by default and the cronjob controller doesn't do anything different than the default when deleting Environment: Kubernetes version (use ): 1.13.0 Cloud provider or hardware configuration: ubuntu VM in vSphere OS (e.g. from /etc/os-release): 16.04.5 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-135-generic Install tools: kubeadm <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig apps
Should include a known issues in 1.13.0 as well /priority critical-urgent
Lemme see if I can easily reproduce this.
/milestone v1.13
Fixes are in (master) and (1.13).


Question: missingKeyTests := []jsonpathTest{ {"nonexistent field", "{.hello}", storeData, "", false}, <ins> {"nonexistent field 2", "before-{.hello}after", storeData, "before-after", false}, </ins> } testJSONPath(missingKeyTests, true, t)

no longer continues parsing an item when a key is encountered that has no value. This appears to be a behavior change introduced in Kubernetes version 1.22. will continue . Run the following commands against a cluster running CoreDNS pods in the namespace: Nothing. Kubernetes version (use ): v1.22.0 Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Debug data for Kubernetes version 1.21 cluster:
Debug data for Kubernetes version 1.22 cluster:
/sig cli
I reproduce the issue in my 1.23 environment. I'll try to fix it. /assign
PR works on it.
/unassign
to ensure I understand your comment correct, should fix this issue?
Yes, I've tested it. can resolve it. The bug is the line:
Yes, I've tested it. can resolve it. The bug is in this line:


Question: rand.Uint32()) } <ins> // TODO(vmarmol): This should probably return an error. </ins> // Unpacks a container name, returning the pod full name and container name we would have used to // construct the docker name. If the docker name isn't the one we created, we may return empty strings. func ParseDockerName(name string) (podFullName string, podUID types.UID, containerName string, hash uint64) {

Successfully ran containers with "onFailure" restartPolicy are restarting after purging oldest docker containers. Kubelet purges/removes oldest containers periodically from docker. After purging, we loose the exit code and history of previously ran containers and restart containers as if they were never run.
Thanks & I can help fix this. Let me know. Thanks
Thanks for reporting Just to make sure I understand the issue: You ran some containers with an "onFailure" restartPolicy. When the Kubelet purged old containers the container was restarted (unexpectedly).
Ah, after reading again I think the issue is that after the containers are purged, when the container exits successfully it get restarted although it shouldn't. Is that correct?
After reading through the code I found a bug in the implementation that is most likely causing this error. Should have a PR soon.
Yes. Correct. After initial container run, it should never be run again, since it already ran to completion. Will wait for your PR.
PR is out. This should fix the issue. Feel free to test if you can :)


Question: if err != nil { return false, err } <ins> if *ssGet.Status.ObservedGeneration < ss.Generation { return false, nil } </ins> if ssGet.Status.Replicas != expectedReplicas { Logf("Waiting for stateful set status to become %d, currently %d", expectedReplicas, ssGet.Status.Replicas) return false, nil

https://k8s- do you know the statefulset owners?
cc
I did not see this happening until recently.
The controller misses that the first pod transitioned to unready and proceeds to create the second pod. The test uses a client to hit the api server and will scale up the statefulset as soon as the broken pod is observed but the controller uses caches so it may observe the scaled up statefulset quicker than the updated pod.
I will update the test to ensure that we wait until the unreadiness is observed on StatefulSet.Status.Replicas field prior to the scaling operation
SGTM


Question: if requestInfo == nil { requestInfo = &request.RequestInfo{Verb: req.Method, Path: req.URL.Path} } <del> var g compbasemetrics.GaugeMetric </del> <ins> var g, e compbasemetrics.GaugeMetric </ins> scope := CleanScope(requestInfo) // We don't use verb from <requestInfo>, as this may be propagated from

apiserverlongrunninggauge and apiserverregisterwatchers are kinda redundant . We discussed this in our SIG instrumentation meeting and while it's clear that apiserverlongrunninggauge, the name does not conform to instrumentation guidelines (it was created before those guidelines existed). What we want to do is deprecate both metrics and create a new one which actually does conform to our naming conventions probably something like . has volunteered to undertake this. /cc /sig instrumentation
/assign
GitHub didn't allow me to assign the following users: yan-lgtm. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted
Hello! Have you still working on this issue? I am new to the community, and can I try it?
Hey I the deprecate in If you want to add the new metric, feel free to do so.
Can I add the new metric? If I can, can you give me some advice about the new metric? Thank you~
/assign


Question: msg, )) } <ins> // warn if labelSelector is empty which is no-match. if t.LabelSelector == nil { warnings = append(warnings, fmt.Sprintf("%s: a null labelSelector results in matching no pod", fieldPath.Child("spec", "topologySpreadConstraints").Index(i).Child("labelSelector"))) } </ins> } // use of deprecated annotations

Currently, when the Label Selector in the Pod Affinity is empty, no objects will be match. That means if it's in a required PodAffinity, the Pod will never be schedulable. And if it's in a required PodAntiAffinity, that PodAntiAffinity will be no-op. So, I'm thinking PodAffinity with empty Label Selector is very likely mistake. It'd be good if we could notice it for users. To prevent such a misconfiguration. Probably, we cannot filter out such a Pod creation in the validation for a backward compatibility. /assign /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm wondering if it has already discussed somewhere or not. Please just let me know if we have some decisions on it.
I am not sure, where do you propose to put the warning? If the pod is unschedulable, that will show up in the events and pod status.
Do you mean the place to implement this warning? I suspect ↓ is a suitable place:
A warning sounds useful, even if the pod is unschedulable, as it gives you a heads up at the time when you create the Pod. Whereas the Pod scheduling failure might be a bit harder to interpret.
ok, I thought you were referring to something that the scheduler will produce after attempting to schedule the pod. Yeah, if there is already a setup to print warnings on pod creation, it is certainly useful for this case.
Can you please also add it in topology spread?
Sure


Question: } } <del> func (c *cacheWatcher) nextBookmarkTime(now time.Time) (time.Time, bool) { // For now we return 2s before deadline (and maybe +infinity is now already passed this time) // but it gives us extensibility for the future(false when deadline is not set). </del> <ins> func (c *cacheWatcher) nextBookmarkTime(now time.Time, bookmarkFrequency time.Duration) (time.Time, bool) { // We try to send bookmarks: // (a) roughly every minute // (b) right before the watcher timeout - for now we simply set it 2s before //     the deadline // The former gives us periodicity if the watch breaks due to unexpected // conditions, the later ensures that on timeout the watcher is as close to // now as possible - this covers 99% of cases. heartbeatTime := now.Add(bookmarkFrequency) </ins> if c.deadline.IsZero() { <del> return c.deadline, false </del> <ins> // Timeout is set by our client libraries (e.g. reflector) as well as defaulted by // apiserver if properly configured. So this shoudln't happen in practice. return heartbeatTime, true } if pretimeoutTime := c.deadline.Add(-2 * time.Second); pretimeoutTime.Before(heartbeatTime) { heartbeatTime = pretimeoutTime } if heartbeatTime.Before(now) { return time.Time{}, false </ins> } <del> return c.deadline.Add(-2 * time.Second), true </del> <ins> return heartbeatTime, true </ins> } func getEventObject(object runtime.Object) runtime.Object {

Currently, (best-effort) watch bookmarks are sent just right before the given watch is supposed to timeout: In we're adjusting how the size of watch cache is determined and make it dynamic and more duration-oriented (rather than based on some magic constants) - we're targeting to always keep 5 minutes of history in watch cache. With that change, we should get back to our bookmark policy: change it to (additionally to right before timeout) also send bookmarks every N seconds (setting N to slightly less than expected watch cache history period, so slightly less than X minutes we will use there) scale-test this change (to see if it doesn't introduce non-negligible overhead) if so permanently introduce this change /sig scalability api-machinery
/cc
/cc
/cc
I'm reverting as it broke bookmarks. I'm going to fix and re-submit for review hopefully later this week.


Question: KubeProxyVersion:        version.Get().String(), }, Capacity: v1.ResourceList{ <del> v1.ResourceCPU:     *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:  *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourceStorage: *resource.NewQuantity(500*mb, resource.BinarySI), v1.ResourcePods:    *resource.NewQuantity(0, resource.DecimalSI), </del> <ins> v1.ResourceCPU:    *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(10E9, resource.BinarySI), v1.ResourcePods:   *resource.NewQuantity(0, resource.DecimalSI), </ins> }, Allocatable: v1.ResourceList{ <del> v1.ResourceCPU:     *resource.NewMilliQuantity(1800, resource.DecimalSI), v1.ResourceMemory:  *resource.NewQuantity(9900E6, resource.BinarySI), v1.ResourceStorage: *resource.NewQuantity(300*mb, resource.BinarySI), v1.ResourcePods:    *resource.NewQuantity(0, resource.DecimalSI), </del> <ins> v1.ResourceCPU:    *resource.NewMilliQuantity(1800, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(9900E6, resource.BinarySI), v1.ResourcePods:   *resource.NewQuantity(0, resource.DecimalSI), </ins> }, Addresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "127.0.0.1"},

Local storage is not expected to be available as a scheduling resource in v1.7. Kubelet is incorrectly exposing a resource named . This is incorrect because the feature is alpha and so it requires opt-in via feature flags and, the resource names are invalid. The expected resource names are listed - and . cc This is v1.7 blocker because we cannot expose a new resource in the API and rename it in a subsequent release.
CC I'll find someone on the storage team to address this.
/assign
GitHub didn't allow me to assign the following users: verult. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: cmdParent := cmd.Parent() if cmdParent != nil { <del> p.FullCmdName = cmdParent.CommandPath() </del> <ins> p.ParentCommandName = cmdParent.CommandPath() </ins> } <del> if len(p.FullCmdName) > 0 && cmdutil.IsSiblingCommandExists(cmd, "describe") { p.SuggestedCmdUsage = fmt.Sprintf("Use '%s describe %s -n %s' to see all of the containers in this pod.", p.FullCmdName, p.ResourceName, p.Namespace) </del> <ins> if len(p.ParentCommandName) > 0 && cmdutil.IsSiblingCommandExists(cmd, "describe") { p.EnableSuggestedCmdUsage = true </ins> } p.Config, err = f.ToRESTConfig()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The pod name is missing in suggested cmd usage when executing : What you expected to happen: The suggested cmd usage should be a complete cmd like the one got from How to reproduce it (as minimally and precisely as possible): Use to attach any pod/deployment/service, the PodName will be missing in the suggested cmd usage like above. Anything else we need to know?: It works fine for kubectl < v1.12 Environment: Kubernetes version (use ): v1.13.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli /assign


Question: kubeadmconstants.ClusterAdminsGroupAndClusterRoleBinding, kubeadmconstants.SuperAdminKubeConfigFileName) <del> if _, err := superAdminClient.RbacV1().ClusterRoleBindings().Create( </del> <ins> err = wait.PollUntilContextTimeout( </ins> ctx, <del> clusterRoleBinding, metav1.CreateOptions{}, ); err != nil { return nil, errors.Wrapf(err, "unable to create the %s ClusterRoleBinding", kubeadmconstants.ClusterAdminsGroupAndClusterRoleBinding) </del> <ins> retryInterval, retryTimeout, true, func(ctx context.Context) (bool, error) { if _, err := superAdminClient.RbacV1().ClusterRoleBindings().Create( ctx, clusterRoleBinding, metav1.CreateOptions{}, ); err != nil { lastError = err if apierrors.IsAlreadyExists(err) { // This should not happen, as the previous "create" call that uses // the admin.conf should have passed. Return the error. return true, err } // Retry on any other type of error. return false, nil } return true, nil }) if err != nil { return nil, errors.Wrapf(lastError, "unable to create the %s ClusterRoleBinding by using %s", kubeadmconstants.ClusterAdminsGroupAndClusterRoleBinding, kubeadmconstants.SuperAdminKubeConfigFileName) </ins> } // Once the CRB is in place, start using the admin.conf client.

master-informing: periodic-kubernetes-e2e-kind-kms ! No response /sig cluster-lifecycle
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.29
/assign but this test job is maintained by sig auth: /sig auth looks like the API server has decided to consistently fail on an unpolled CRB create API call from the kubeadm client. this did not happen on the kind presubmit jobs here: or kubeadm periodic jobs using the kinder tool: or my local tests. so what is different here and why are we getting informer errors? is this highlighting a bug in core k8s? cc best we can do in kubeadm is make that API call polled. FTR, the API call exactly prior to this failing API call is already polled.
Hello. I want to echo that the code freeze is starting (next week), and we want to ensure that each PR has a chance to be merged and issues can be addressed. Is this PR still targeting the 1.29 release?
FTR, the API call exactly prior to this failing API call is already polled. +1 to have the API call polled just as the API call with .
here is the kubeadm PR.
we are tracking a failing test in this ticket and we can proceed with fixing this after code freeze is in effect. i just submitted an associated fix PR here:
As the Alpha 3 release cut is approaching (2nd Nov) /priority critical-urgent
/assign
we have a couple of kubeadm passes here: of the presubmit variant of this job: pull-kubernetes-e2e-kind-kms this seems like a flake and not a failure, given the offending PR merged on the 27th and looking at the history of the periodic:
we are having the similar issue with i saw in the logs I am not sure what I should do, could you help me?


Question: kubeadmconstants.ClusterAdminsGroupAndClusterRoleBinding, kubeadmconstants.SuperAdminKubeConfigFileName) <del> if _, err := superAdminClient.RbacV1().ClusterRoleBindings().Create( </del> <ins> err = wait.PollUntilContextTimeout( </ins> ctx, <del> clusterRoleBinding, metav1.CreateOptions{}, ); err != nil { return nil, errors.Wrapf(err, "unable to create the %s ClusterRoleBinding", kubeadmconstants.ClusterAdminsGroupAndClusterRoleBinding) </del> <ins> retryInterval, retryTimeout, true, func(ctx context.Context) (bool, error) { if _, err := superAdminClient.RbacV1().ClusterRoleBindings().Create( ctx, clusterRoleBinding, metav1.CreateOptions{}, ); err != nil { lastError = err if apierrors.IsAlreadyExists(err) { // This should not happen, as the previous "create" call that uses // the admin.conf should have passed. Return the error. return true, err } // Retry on any other type of error. return false, nil } return true, nil }) if err != nil { return nil, errors.Wrapf(lastError, "unable to create the %s ClusterRoleBinding by using %s", kubeadmconstants.ClusterAdminsGroupAndClusterRoleBinding, kubeadmconstants.SuperAdminKubeConfigFileName) </ins> } // Once the CRB is in place, start using the admin.conf client.

thanks
Can you pls tell where is the the log you pasted above? I searched in this file: , and some other logs related with kubeadm but cannot find it. The original issue is expected to be fixed by: It's now continuous green now: But the log you pasted do point to the change we landed last week:
kubeadm introduced a new step in the "init" process. it creates an file but it needs RBAC for it. this RBAC is created in a rather complicated manner (with a fallback), but in your case it's failing at the first step where the API server is not reachable in 40 seconds. do you expect the API server to no be ready in 40 seconds? are you sure there are no other problems with the setup and it just ends up failing at this first API server step where RBAC is created? other jobs like CAPI (non-upgrade), kinder, kind are green with respect to this. i wonder if we can just stop returning this error and fallback to the super- "create" attempt later: this will allow for more time for certain setups to start - 40 + 40 seconds. ... it exactly to catch problems like the one is reporting - no early connectivity to the API server. but if this is now the "first thing" in kubeadm, it might have introduced a bit of a timeout regression for slower setups.
also note that the kubeadm CP provider in CAPI does not yet support this feature and the upgrade CAPI jobs are also failing:
actually, the feature seems change the order of "init": it waits for the apiserver + kubelet (, 40seconds for kubelet) and the RBAC create ( when the first client() creation happens) at the same time. i think what has to be changed here is: the parallel aspect of the apiserver + kubelet wait. it has been difficult to manage and confusing. delay the RBAC create to be after the apiserver + kubelet waiting is done. maybe something we can do for .29 is 2. but 1 is a bigger refactor.
PR with an attempt to improve the kubeadm wait logic: i do think CAPG has unrelated problems. the kubelet might be failing to start or starting but not creating the static pod for kube-apiserver.


Question: kubeadmconstants.ClusterAdminsGroupAndClusterRoleBinding, kubeadmconstants.SuperAdminKubeConfigFileName) <del> if _, err := superAdminClient.RbacV1().ClusterRoleBindings().Create( </del> <ins> err = wait.PollUntilContextTimeout( </ins> ctx, <del> clusterRoleBinding, metav1.CreateOptions{}, ); err != nil { return nil, errors.Wrapf(err, "unable to create the %s ClusterRoleBinding", kubeadmconstants.ClusterAdminsGroupAndClusterRoleBinding) </del> <ins> retryInterval, retryTimeout, true, func(ctx context.Context) (bool, error) { if _, err := superAdminClient.RbacV1().ClusterRoleBindings().Create( ctx, clusterRoleBinding, metav1.CreateOptions{}, ); err != nil { lastError = err if apierrors.IsAlreadyExists(err) { // This should not happen, as the previous "create" call that uses // the admin.conf should have passed. Return the error. return true, err } // Retry on any other type of error. return false, nil } return true, nil }) if err != nil { return nil, errors.Wrapf(lastError, "unable to create the %s ClusterRoleBinding by using %s", kubeadmconstants.ClusterAdminsGroupAndClusterRoleBinding, kubeadmconstants.SuperAdminKubeConfigFileName) </ins> } // Once the CRB is in place, start using the admin.conf client.

Had some discussion with Carlos. I didn't follow this issue here in full detail, but probably the CAPG issue is resolved by the PR that we merged in core CAPI and will be part of the CAPI rc next week
yes, seems to work now :) thank you all!!!


Question: "name": podNamePrefix, }, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Resources: v1.ResourceRequirements{ Requests: v1.ResourceList{ <del> v1.ResourceCPU: tc.reportedCPURequests[i], </del> <ins> v1.ResourceCPU: reportedCPURequest, </ins> }, }, },

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: We are starting to use the HPA (horizontal pod autoscaling) for one of our components. What is special about this particular component is that the warm up period is very long (sometimes even more than 30 minutes). During that time, the pod is in Running state but its containers are in Not Ready state. This pod is of course not able to receive any traffic so the metric we use in the HPA is ignoring those pods that are not Ready. The equation used by HPA to calculate the new number of replicas is as follows Our problem is that the equation used by HPA to calculate the new number of replicas is including in the "current replica count" the pods that are "Not Ready" so the value for "new replica count" is not correct. Since this warm up is so long we end up by having a lot of more instances of our services of what it should be necessary. They eventually get deleted once gradually the metrics start going down but that will take a lot of time and a big waste of resources without mentioning that with a changing traffic this HPA will never stabilize What you expected to happen: I would like to have a flag somewhere where I tell HPA to don't include in the "current replica count" the pods that are in "Not Ready" state How to reproduce it (as minimally and precisely as possible): Have a component that takes a lot of time to start and being able to receive traffic. Then define a metric that will ignore the pods that are Not Ready. Finally create a metric that will scale that component based on the mentioned metric Anything else we need to know?: Environment: Kubernetes version 1.7 Cloud provider or hardware configuration: Azure
/sig autoscaling
Hi - thanks for reaching out about this! It looks like this section of the , handles when pods aren't ready. I think you this behavior in , so I'm interested in your thoughts on whats best here. If we do end up adding a flag for this, I'd be happy to write the code :) Also, purely for my own curiosity, I'm curious what type of pod is taking 30 minutes to become ready. If your traffic pattern varies at a rate faster than 30 minutes, it may be difficult for auto-scaling to be helpful.
can you also share a bit more about your HPA?


Question: "name": podNamePrefix, }, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Resources: v1.ResourceRequirements{ Requests: v1.ResourceList{ <del> v1.ResourceCPU: tc.reportedCPURequests[i], </del> <ins> v1.ResourceCPU: reportedCPURequest, </ins> }, }, },

What kind of metric type is it? Based on the description I'm suspecting that it is an Object metric, am I right?
So answer about why our process takes that much time to start. Basically this is a server that in order to do what he needs to do, he needs to load a lot of data in memory (several GBs) and then perform some warm up that are pretty intensive. The idea is that he doesn't need to grab (almost) anything from outside to attend a request so it's latency is extremely low (in the order of millisecond) By the way. Because of this we have another problem which is that there is not an specific flag per HPA object that we could tell k8s to don't try to scale up again passed some time so the new pods have enough time to get ready before the next scale happens. I know that there is a cluster wide flag but we will need to have a different one by resource that we scale. Now to answer You pointed another interesting topic. The metric we use is % Utilization. We are able to calculate the level of utilization of our pods and we want to keep this metric in an specific value. About the flag. I suggested the flag so it doesn't change the existing behavior by default but I honestly think that the default behavior should be to don't include the NonReady pods as part of the calculation
do you think we can add this change to 1.10? it's critical bug/feature for us and we are happy to help with code/testing.
It's really late for 1.10, especially since we don't have any code here, and I'd like to know more about what's going on here, and especially see if it reproduces on Kubernetes 1.9 (we've had HPA controller bugs in the past). FWIW, currently unready pods are treated as consuming 0% of their utilization on scale up, and it was intended that this should prevent such issues. That is, the formula is actually: If the pods are marked as not ready, I find it strange that they would be affecting the scaling that much, unless your other pods are under real heavy load (i.e. I would assume that the unready pods would skew the scaling back down so that the number of pods just stayed constant).
I believe there is different code branch here.


Question: "name": podNamePrefix, }, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Resources: v1.ResourceRequirements{ Requests: v1.ResourceList{ <del> v1.ResourceCPU: tc.reportedCPURequests[i], </del> <ins> v1.ResourceCPU: reportedCPURequest, </ins> }, }, },

metric definition { "type": "Object", "object": { "target": { "kind": "Service", "name": "prometheus-exporter" }, "metricName": "PercentUtilization", "targetValue": "20" } } which should handle as and doesn't check Pod state.
I think proper handling of unready pods is a difficult problem that will require significant work. I suspect we may find out we need to start distinguishing pods that are booting up from pods that are 'broken' and it took us quite a while to solve analogous problem with nodes in Cluster Autoscaler (to be clear: I don't know if we will have to do that, that's just my intuition for now). That being said I agree the formula in GetObjectMetricReplicas will generally lead to scaling-up too much if the pod takes longer to initialize then backoff period in HPA (3 minutes). Assume it takes pod 4 minutes to boot up and we're currently running 2 pods. Metric value is 20 and target is 10. In this case: scales-up to 4 pods (20/10 utilization 2 pods). are starting up, they're not doing anything yet. 3 minutes later HPA scales-up to 8 pods (20/10 utilization 4 pods). If we only counted ready pods the scale-up in 3 would not occur, because the desired replicas would still be 4 (20/10 * 2). My intuition would be that this is the expected behavior of HPA. Also if there is no space in the cluster we will just keep scaling-up until we hit max replicas. The only case where current behavior would be more correct than filtering out unready pods in this formula is if someone uses a metric that is calculated based on total number of replicas. Do we advertise that as the right way to use HPA? If we add to Object metrics in 1.11 this usage pattern will stop making sense. To sum up - I think we will need to change the formula to only count ready pods at some point. I can see how current behavior could be a problem in any use-case except the one described in paragraph above. I'm inclined to agree with and provide some way to change the behavior in 1.10 (a flag or an annotation?), treating it as a bug fix.


Question: "name": podNamePrefix, }, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Resources: v1.ResourceRequirements{ Requests: v1.ResourceList{ <del> v1.ResourceCPU: tc.reportedCPURequests[i], </del> <ins> v1.ResourceCPU: reportedCPURequest, </ins> }, }, },

WDYT
Ah, yeah, I missed that it was object metrics before. I'm not entirely convinced we need a flag here -- it might just make sense to bring Object metrics in line with all the other metrics types. On the other hand, there's a subtlety here -- the other metric types don't quite ignore unready pods -- they just ignore the metrics from unready pods, which causes a heavy bias in the direction of not scaling, but an overwhelming spike can still cause scaling. That behavior is not as achievable using object metrics. I'm inclined to just ignore that subtlety, though, and say that object metrics should ignore unready pods entirely.
I think this is what we should do long term. I was worried about breaking someone who relies on targetValue + averaged metric to get effect similar to targetAverageValue and though about providing something for transitional period, but if you're ok with it I'd rather just change it as you say.
I think it's safe to change as-is -- that feature's still in beta, and it's not an API change, so it shouldn't just hard-break anyone. Plus, we've change stuff like this before (when we did the initial resource metrics change).
As said, we are more than ready to help with the fix/test if required. This since this is a major blocking for us in the implementation of the HPA in our side
is anyone actively working on implementing this? If not, I'm happy to. I think I have a fairly clear picture of the change we need to make and will likely have time early this week to get it done.
(Also, from the conversation above, to we want to tag this as a release blocker for 1.10)?
I probably won't be able to work on this before the end of this week. Happy to do a review if you jump in and implement it.
I don't think this qualifies as a release blocking bug (but I'm willing to have someone try and convince me otherwise ;-)). There doesn't seem to be any behavior here that's explicitly broken -- it's just sub-optimal in some use cases. I'd definitely like to address this in 1.11, though.
it's not release blocker, but to share our experience ;).


Question: "name": podNamePrefix, }, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Resources: v1.ResourceRequirements{ Requests: v1.ResourceList{ <del> v1.ResourceCPU: tc.reportedCPURequests[i], </del> <ins> v1.ResourceCPU: reportedCPURequest, </ins> }, }, },

without this feature we cannot use HPA as is and we need to figure out how to build/run fork (which can be challenging because everything inside controller-manager). also we are planning to upgrade to 1.10 quickly, but not sure for 1.11 (e.g. we are two versions behind now).
Sounds like a plan :) Will try and get this done tomorrow morning!
- I submitted a pr for this - would love your eyes on it so you can see if you think it meets your use case :)
/assign
GitHub didn't allow me to assign the following users: mattjmcnaughton. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I checked the code and it seems good to me. About the comment about scaling down if the readyPods<CurrentPods, we should be fine since our metric is already not including the pods that are not ready.
Is there any advance in this ticket? I'm wondering if this made it to k8s 1.10 or if we need to wait for k8s 1.11?
The PR is lgtm-ed and it will merge once the code freeze is lifted. Meaning it will go to 1.11. To get it into 1.10 you would need to get approval for milestone from (sig-autoscaling lead).


Question: deleteAllEtcdKeys() defer deleteAllEtcdKeys() <del> _, s := RunAMaster(t) </del> <ins> _, s := runAMaster(t) </ins> defer s.Close() ns := api.NamespaceDefault

In 0.13.2, the apiserver has automatically exposed metrics about the process, like start time, memory usage, processor usage, etc. This stopped working sometime between then and when 0.14.0 was cut. They're still correctly exported from the kubelet, interestingly enough. An e2e or integration test is also needed to catch this if it happens again.


Question: } } <ins> var knownAuthTypes = map[string]bool{ "bearer":    true, "basic":     true, "negotiate": true, } // maskValue masks credential content from authorization headers // See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Authorization func maskValue(key string, value string) string { if !strings.EqualFold(key, "Authorization") { return value } if len(value) == 0 { return "" } var authType string if i := strings.Index(value, " "); i > 0 { authType = value[0:i] } else { authType = value } if !knownAuthTypes[strings.ToLower(authType)] { return "<masked>" } if len(value) > len(authType)+1 { value = authType + " <masked>" } else { value = authType } return value } </ins> func (rt *debuggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) { reqInfo := newRequestInfo(req)

This issue was reported in the Description Kubernetes requires an authentication mechanism to enforce users’ privileges. One method of authentication, bearer tokens, are opaque strings used to associate a user with their having successfully authenticated previously. Any user with possession of this token may masquerade as the original user (the “bearer”) without further authentication. Within Kubernetes, the bearer token is captured within the hyperkube kube-apiserver system logs at high verbosity levels (--v 10). A malicious user with access to the system logs on such a system could masquerade as any user who has previously logged into the system. Exploit Scenario Alice logs into a Kubernetes cluster and is issued a Bearer token. The system logs her token. Eve, who has access to the logs but not the production Kubernetes cluster, replays Alice’s Bearer token, and can masquerade as Alice to the cluster. Recommendation Short term, remove the Bearer token from the log. Do not log any authentication credentials within the system, including tokens, private keys, or passwords that may be used to authenticate to the production Kubernetes cluster, regardless of the logging level. Long term, either implement policies that enforce code review to ensure that sensitive data is not exposed in logs, or implement logging filters that check for sensitive data and remove it prior to outputting the log. In either case, ensure that sensitive data cannot be trivially stored in logs. Anything else we need to know?: See for current status of all issues created from these findings. The vendor gave this issue an ID of TOB-K8S-001 and it was finding 6 of the report. The vendor considers this issue Medium Severity. To view the original finding, begin on page 31 of the Environment: Kubernetes version: 1.13.4
/sig auth /sig api-machinery
/remove-kind feature /kind bug
Should remote the "Authorization" header here?
for authorization headers, everything after the type should be masked (and potentially, only recognized auth types should be logged). Something like:
I think people would be unpleasantly surprised to see credentials in a logfile after turning up logging to debug an issue. /priority important-soon
Does either of you intend to createe PR to remove the "Authorization" header ?


Question: } } <ins> var knownAuthTypes = map[string]bool{ "bearer":    true, "basic":     true, "negotiate": true, } // maskValue masks credential content from authorization headers // See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Authorization func maskValue(key string, value string) string { if !strings.EqualFold(key, "Authorization") { return value } if len(value) == 0 { return "" } var authType string if i := strings.Index(value, " "); i > 0 { authType = value[0:i] } else { authType = value } if !knownAuthTypes[strings.ToLower(authType)] { return "<masked>" } if len(value) > len(authType)+1 { value = authType + " <masked>" } else { value = authType } return value } </ins> func (rt *debuggingRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) { reqInfo := newRequestInfo(req)

This issue seems CVE-worthy to me: loss of confidentiality (bearer tokens) + cross of privilege boundary (debug log level could be set by admin, then logs read later by low privileged user). My draft cvss would be: CVSS:3.0/AV:N/AC:H/PR:L/UI:N/S:U/C:H/I:N/A:N (5.3)
Two potential vectors: local user with access to system logs (local access, low privilege) (4.7) remote user with access to system logs (network access, high privilege): (4.4) CVE-2019- has been assigned to this issue.
If you want to, go ahead
- should this have the official CVE feed label ( retrospectively, I mean)?
probably? I have no idea how that label is supposed to get applied, but there appear to be several CVE issues that don't have it -
I think you're likely to know who to highlight this to. Sorry to mention you individually; please feel free to point out docs if there are any.
/label official-cve-feed thanks for the tag, the label now!
thanks for the useful filter. Currently in-scope issues are issues that were closed as fixed by code changes to k/k. So with that in mind, I have the labels to some of the closed issues too that conform to the scope defined here:
I would like to reply nearly four years later: There are valid cases to want to see the token, for example my current case, trying to debug token validation between the API server and other services.


Question: resources []string <del> namespace string names     []string </del> <ins> namespace    string allNamespace bool names        []string </ins> resourceTuples []resourceTuple

The desciption of --all-namespaces is listing the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with --namespace." But when I used the following command: error: namespace may not be empty when retrieving a resource by name no namespace set on resource pods "counter" The error messages are unclear, and the second message isn't labeled as an error even.
And why couldn't use --all-namespaces flag?
You're asking for a specific pod name (counter) in all namespaces, which is a nonsensical query. The pod name is not a grep-like filter, it's a specific name. Yeah, the errors could be better. Describe could work, but potentially generates a LOT of information and API calls.
But I found the following API is supported now. Get The result is: has the more information. Do we need to support the following kubectl command(using the above API) or just return clear error messages? get pods counter --all-namespaces
I'd be ok with this being implemented via the field selector.
The issue is still here :). What's our expected result? I'd like to contribute this.
, after check the code, the URL was built by ; do you mean we should update it; or avoid to use , instead, build url () directly.
After checking the code, it's better to add fieldSelector to , is created for fieldSelector. Will submit a PR based on .
getting an object with a particular name across all namespaces seems like something we shouldn't encourage... it breaks the assumption of namespace independence.
, so we'll only update message for this issue? for example:
Disagree. Need easy way to troubleshouting the problem "why pods is stuck in Pending status". It's tiring for every pod to find who created it and in which namespace. In the logs there is only the name of the pod.
Now, with field selectors we can do To further my understanding, I'd ask:- doesn't itself break independence? One can search by label across all namespaces, it feels funny for by name to be special.


Question: if errs := options.Audit.Validate(); len(errs) > 0 { errors = append(errors, errs...) } <ins> if errs := options.Admission.Validate(); len(errs) > 0 { errors = append(errors, errs...) } </ins> if errs := options.InsecureServing.Validate("insecure-port"); len(errs) > 0 { errors = append(errors, errs...) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: param is not validated in func Validate What you expected to happen: validate admission plugins, if not registered, return err to prevent going on until call . How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig auth
/kind bug
from the user's perspective, a reasonable error is returned


Question: klog.Errorf("iscsi: could not find transport name in iface %s", b.Iface) return "", fmt.Errorf("could not parse iface file for %s", b.Iface) } <ins> addr := tp if strings.HasPrefix(tp, "[") { // Delete [] from IP address, links in /dev/disk/by-path do not have it. addr = strings.NewReplacer("[", "", "]", "").Replace(tp) } </ins> if iscsiTransport == "tcp" { <del> devicePath = strings.Join([]string{"/dev/disk/by-path/ip", tp, "iscsi", b.Iqn, "lun", b.Lun}, "-") </del> <ins> devicePath = strings.Join([]string{"/dev/disk/by-path/ip", addr, "iscsi", b.Iqn, "lun", b.Lun}, "-") </ins> } else { <del> devicePath = strings.Join([]string{"/dev/disk/by-path/pci", "*", "ip", tp, "iscsi", b.Iqn, "lun", b.Lun}, "-") </del> <ins> devicePath = strings.Join([]string{"/dev/disk/by-path/pci", "*", "ip", addr, "iscsi", b.Iqn, "lun", b.Lun}, "-") </ins> } if exist := waitForPathToExist(&devicePath, deviceDiscoveryTimeout, iscsiTransport); !exist {

Kubelet was not able to attach + mount an iSCSI PV when it has IPv6 server address. Volume is mounted + attached. Set up iSCSI target that's reachable over IPv6 and create a PV for it. For example: Kubelet logs: Notice around the IP address. Various iSCSI files in /sys and /dev/disk/by-path do not have these square brackets and kubelet is then not able to find the iSCSI host in /sys. That's where comes from. No response <detailsA recent master: console NAME="Fedora Linux" VERSION="36 (Workstation Edition)" </details<detailshack/local-up- </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-soon
/assign


Question: defaultIntOrString := intstr.FromString("25%") differentIntOrString := intstr.FromInt(5) period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug The change in a new field () to the Pod spec that is set to true by default. This is causing problems for DaemonSets in some situations. DaemonSets are using ControllerRevisions to keep track of previous configurations and each revision will contain among other things the Pod spec for the DaemonSet. If a DaemonSet is created with the enableServiceLinks property is set to true, the ControllerRevision will also contain this field. If the cluster is rolled back to a previous version of kubernetes, the DeamonSet struct does not have this field and so the ControllerRevision and the DaemonSet will no longer match. This leads to a new ControllerRevision being created and a rolling update of all pods in the DaemonSet. Would it be a better solution to rename the field to and enable service links whenever the field is not set? This would fix the problem with the DaemonSets unless the field is explicitly set before the rollback. This is causing the master upgrade tests to fail as reported in /sig apps /cc /cc /cc
I guess the change in podSpec and the defaulting will break all the workload resources as they are based on comparing the template's podSpec. ~Is the new value defaulted to true also for the DaemonSet's podSpec template? If not, controller might go hot loop like with mutating admission controllers.~ I don't like us making such changes to v1 podSpec. The other way around seems more sensible. /cc
Why is this a bad thing? We will continue to add fields to podSpec - the issue isn't the defaulting (this would happen regardless of which way it defaulted), it's that the controller can't really be version aware. The new value would default to true in the daemonset as well.
I'm not sure that a rollback requiring daemonsets to recreate is a bug. I think that's potentially "the best possible thing that we can do in that case".
/cc SIG Release blocking dashboard with upgrade test failures - https://k8s-
So the problem isn't the fact that a field has been . It the way in which the field was .


Question: defaultIntOrString := intstr.FromString("25%") differentIntOrString := intstr.FromInt(5) period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either ca So the problem isn't the fact that a field has been . It the way in which the field was . enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either case), you won't trigger the rolling restart on rollback or controller manager failoverse), you won't trigger the rolling restart on rollback or controller manager failover unless you explicitly opt out of the behavior. I don't see why the former behavior is more desirable than the latter.
For background/context, the choice between enable/disable was initially discussed here:
Are you saying that the issue is that default value is not the same as the Go "zero value" ? The concept of zero value is interesting but it forces APIs to bend into unnatural shapes to respect it. Since forever Kubernetes has explicitly said that we have version-specific default values which ARE NOT governed by zero values, but instead are governed by the API version. It is safe to assume that if a field is documented as having a default in a version, you can rely on that field being populated with that default, with a few exceptions.


Question: defaultIntOrString := intstr.FromString("25%") differentIntOrString := intstr.FromInt(5) period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

We generally ask for boolean fields to be "in the affirmative direction", as in this case. Now, we could revisit this. It would be a significant change in direction, but it's not impossible. But I don't think that solves your problem in general -- if this field were optional string, with a default value of "Hello there", you would be in the same boat right?
So the underlying issue is that we are defaulting into the PodSpec which gets adopted into all of the PodTemplateSpecs of every object. That sort of defaults the users declared intent. Why not just default the value into the Pod itself . Is the intention really to default this in all Templates or to make it optional in the Template and default it in the Pod itself?
/milestone v1.13
I can't speak to the ideology of whether to default the field in all PodSpecs or just the Pod, but your suggestion makes sense to me. I'm not sure of the past reasoning for adding other defaulted fields, most recently SchedulerName in Jan. 2017 (), to all PodSpecs instead of just to the Pod object. Did we not have these same tests in place then? This would effectively be saying we should never add new defaulted fields to PodSpec. If this is the plan, I can create a PR for the fix and I'll add a big warning comment to SetDefaults_PodSpec.
As an example of where we don't default the template we have a CronJob that embeds JobTemplate, we never default that template, only an actual Job gets defaulting. I agree with that not defaulting any templates does make sense.
Agree to only default this in pods. Agree to adding the comment. Most important defaulting happens in pods to keep higher level pod specs simpler. We may need to raise the general pattern in a discussion with api review in general, since this is a change from previous strategy, but the previous strategy was not explicitly chosen (to my recollection at least)
Hi all do we have an agreed path forward for fixing this issue in 1.13? if so can we have an ETA for the fix to land? the release team would like for the tests to get passing in the blocking suites in an effort to unmask any other failures that might show up once these are fixed.


Question: defaultIntOrString := intstr.FromString("25%") differentIntOrString := intstr.FromInt(5) period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

/assign
We have a PR with a fix that will hopefully merge today or tomorrow. I will keep an eye on the dashboard to make sure it fixes the failing daemonset tests.
Thanks Can we link the PR here plz. /cc
Reopening, because of the auto-close. Tests are not passing yet. /reopen
Reopening this issue. <detailsIn response to : Reopening, because of the auto-close. Tests are not passing yet. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
PR that should (hopefully) close this issue is here:
DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- if it doesn't pop up again - /close
Closing this issue. <detailsIn response to : DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: dockernat "github.com/docker/go-connections/nat" "github.com/golang/glog" <ins> utilerrors "k8s.io/apimachinery/pkg/util/errors" </ins> "k8s.io/kubernetes/pkg/credentialprovider" runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/v1alpha1/runtime" <ins> "k8s.io/kubernetes/pkg/kubelet/dockershim/libdocker" </ins> "k8s.io/kubernetes/pkg/kubelet/types" "k8s.io/kubernetes/pkg/security/apparmor" <del> "k8s.io/kubernetes/pkg/kubelet/dockershim/libdocker" </del> <ins> "k8s.io/kubernetes/pkg/util/parsers" </ins> ) const (

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: We have a private registry configured (where all images including the sandbox one), are hosted. The registry credentials are being pulled by one of k8s credentials providers () With Kubernetes 1.7 sandbox image failed to be pulled with the following error message: Looks like with this only creds from docker config are being evaluated on the sandbox image pull What you expected to happen: Sandbox image pull should be attempted using credentials returned by not just the basic docker credential provider, but other credentials providers registered in the system. How to reproduce it (as minimally and precisely as possible): 1) Register private repo credentials with one of the k8s credential providers (we've tried with rancher). Configure the to use the image from the the private repo. 2) Make sure you are not logged in to this private repo on the kubelet host 3) Try to create a pod. Sandbox image pull will fail with an error mentioned above. Anything else we need to know?: Non-sandbox images from the same private repo were able to download correctly using private repo credentials provider. Environment: Kubernetes version (use ): 1.7.2 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node Just tested the same use case against k8s 1.5.x, it worked fine there and sandbox image was pulled using the credentials passed by the credential provider, not from docker config file. So it might have regressed in 1.7.x (haven't tested it on 1.6, so can't tell for sure)
Confirmed a bug introduced in dockershim. Will file a PR to fix this.