Question: ginkgo.By("Ensuring resource quota status captures configMap creation") usedResources = v1.ResourceList{} usedResources[v1.ResourceQuotas] = resource.MustParse(strconv.Itoa(c + 1)) <del> // we expect there to be two configmaps because each namespace will receive // a ca.crt configmap by default. // ref:https://github.com/kubernetes/kubernetes/pull/68812 </del> usedResources[v1.ResourceConfigMaps] = resource.MustParse(hardConfigMaps) err = waitForResourceQuota(f.ClientSet, f.Namespace.Name, quotaName, usedResources) framework.ExpectNoError(err)

I ran conformance tests on a cluster that had a controller enabled that syncs a configmap across all clusters. It failed: Appears to be due to a 2 configmap limit: These clusters should pass conformance. This is a very common pattern (even Kubernetes itself does it!). I didn't run it myself, but apparently just install Istio and run the conformance suite. See for details Kubernetes version (use ): v1.20.2
/assign
my guess is the CA configmap created by default in 1.20 exacerbated this issue. I agree the conformance test should make fewer assumptions about the cluster it is running in and should accomodate default configmaps. left a comment on the PR in flight for this. I'd like to see the change picked to 1.20 as well.
/triage accepted /priority important-soon


Question: containerTopologyScope = "container" // podTopologyScope specifies the TopologyManagerScope per pod. podTopologyScope = "pod" <ins> // noneTopologyScope specifies the TopologyManagerScope when topologyPolicyName is none. noneTopologyScope = "none" </ins> ) type podTopologyHints map[string]map[string]TopologyHint

Follow up from: See details there. When is , there is no need to discover the NUMA topology. /sig node /priority backlog /kind feature /help-wanted
/cc
/triage accepted
I would like to take this up also can I get more brief information about it? code review comments from asking that inside the NewManager() implementation, we need to skip NewPolicyOptions and NewNUMAInfo but when the NumaTopology comes kubelet issue also comes. I need some more information to start with this!
/cc
When there is no topology manger policy specified, NewManager should be a no-op. Just skip all checks.
Exactly. Just to add: we may need few iterations over the change to get a clean flow in the call.
/assign


Question: return } for _, pod := range podList.Items { <del> framework.Logf("Kubectl output:n%v", framework.RunKubectlOrDie("log", pod.Name, "--namespace=kube-system")) </del> <ins> framework.Logf("Kubectl output:n%v", framework.RunKubectlOrDie("log", pod.Name, "--namespace=kube-system", "--container=heapster")) </ins> } }

https://k8s- It seems that when it takes a while, it tries to collect debug info. And that collection is flawed because kubectl needs a container for log collection when the pod has more than one container.


Question: By(fmt.Sprintf("Initial restart count of pod %s is %d", podDescr.Name, initialRestartCount)) // Wait for at most 48 * 5 = 240s = 4 minutes until restartCount is incremented <del> pass := false </del> <ins> restarts := false </ins> for i := 0; i < 48; i++ { // Wait until restartCount is incremented. time.Sleep(5 * time.Second)

As far as I can tell, container liveness probes of type exec flat out don't work against clusters built at head. I had the kubelet always log the output of the exec'ed command, and it's always empty no matter what I exec, causing the probe to fail since it requires the output to be equal to the string "ok". If you want to test it, try adding a liveness probe of this form to any container: I also tried these lovely forms, to more closely match the liveness example under examples/: On the plus side, this may be an opportunistic time to switch over to
The reason that the e2e test for exec liveness probes hasn't caught this is that it only tests that containers are restarted when they fail liveness probes. It doesn't test that containers aren't restarted when their liveness checks should be succeeding. The e2e test's exec'ed commands also only ever produced empty output, rather than the expected "fail" output.
Is it possible to include this in a new pre-release, since it fixes a bug in health checking? I see it merged in master and I'm wondering what the policy is of releasing new code in a separate (0.16.*) release.


Question: old_storage_version=${test_data[4]} kube::log::status "Verifying ${resource}/${namespace}/${name} has storage version ${old_storage_version} in etcd" <del> curl -s http://${ETCD_HOST}:${ETCD_PORT}/v2/keys/registry/${resource}/${namespace}/${name} | grep ${old_storage_version} </del> <ins> curl -s http://${ETCD_HOST}:${ETCD_PORT}/v2/keys/${ETCD_PREFIX}/${resource}/${namespace}/${name} | grep ${old_storage_version} </ins> done killApiServer

removed this test. We need to continue testing this. This wasn't about testing upgrades, the prefix has nothing to do with upgrades. It's just there so people can share etcd. We maybe don't need to double everything but we do need to make sure that keeps working.
To be honest - I think that what we was testing before didn't give us completely any information. We didn't testing what information we are storing in etcd and where. That said, if we had a hardcoded prefix somewhere in our code, we wouldn't discover it at all, and those tests would still be passing. We should have a test that is directly checking etcd contents. I think it should be possible to write a simple unit test for it. Will try to put something together next week.


Question: func (runner *runner) connectToFirewallD() { bus, err := runner.dbus.SystemBus() if err != nil { <del> glog.V(1).Info("Could not connect to D-Bus system bus: %s", err) </del> <ins> glog.V(1).Infof("Could not connect to D-Bus system bus: %s", err) </ins> return }

I0921 20:36:33. ] Could not connect to D-Bus system bus: %sdial unix /var/run/dbus/systembussocket: connect: no such file or directory


Question: } // validateValueValidation checks the value validation in a structural schema. <del> func validateValueValidation(v *ValueValidation, skipAnyOf, skipFirstAllOfAnyOf bool, fldPath *field.Path) field.ErrorList { </del> <ins> func validateValueValidation(v *ValueValidation, skipAnyOf, skipFirstAllOfAnyOf bool, lvl level, fldPath *field.Path) field.ErrorList { </ins> if v == nil { return nil }

We allow CRD authors to write . Currently we don't restrict writing validation for , which means people can have . For example, people can have additive name constraints (require CR to be ) with the following schema: This brings us two questions: do we if "metadata" is included in the schema: simplest option is to publish standard metav1.ObjectMeta schema info, and use any CRD-specified "metadata" properties as additive validation (just like in-tree types do, publishing standard ObjectMeta schema info, and implementing type-specific name validation internally) alternatively, we can try merging user-specified metadata schema with the standard one. Technically people should only add extra validation to existing metadata properties (adding new fields to metadata won't work since they will be pruned by built-in validation first), but it's unclear to me how arbitrary people may want the metadata validation to be term question is whether to allow people specifying metadata validation at all, or just allow a subset (like name validation) in CRD v1 /sig api-machinery /area custom-resources cc
What is the difference between these two? I mean the end result is the same right? The only thing that can be is validation. Nothing else works unless you are considering extensions?
The former means we drop/hide user-specified metadata schema when we publish CRD OpenAPI. We still validate user-specified constraints on server side, but client-side validation won't see them. The latter enables client-side user-specified metadata validation.
If it's possible to merge with standard metadata schema info, or at the cost of not publishing the standard schema. My inclination would be to publish the complete standard schema for metadata/apiVersion/kind, which we know is complete and accurate, and use and user-specified schema for those fields as addition validations server side


Question: WaitForCacheSync: func() bool { return cache.WaitForCacheSync(c.StopEverything, c.scheduledPodsHasSynced) }, <del> NextPod: func() *v1.Pod { return c.getNextPod() }, </del> <ins> NextPod:         internalqueue.MakeNextPodFunc(c.podQueue), </ins> Error:           c.MakeDefaultErrorFunc(podBackoff, c.podQueue), StopEverything:  c.StopEverything, VolumeBinder:    c.volumeBinder,

PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. Main tracking issue: The method of is a simple wrapper around . It should be converted into a simple closure instead of being a method. i.e. we should have, (code is in ) /sig scheduling /kind cleanup /help
I'd like to take this task. /assign /remove-help
Thanks, go for it! :smile:


Question: } function dump_nodes() { <del> local node_names </del> <ins> local node_names=() </ins> if [[ -n "${1:-}" ]]; then echo "Dumping logs for nodes provided as args to dump_nodes() function" node_names=( "$@" )

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In https://k8s- failed due to . Afterwards crashed as follows: What you expected to happen: Graceful exit.
/assign


Question: package v1_test import ( <del> "encoding/json" </del> <ins> "fmt" </ins> "reflect" "testing"

What happened: kubelet doesn't seem to be able to manage containers started with CRI-O any more I'm testing kubelet + CRI-O on an Ubuntu 18.04 server installation, inside a qemu/kvm VM managed by Vagrant (see ), in order to be able to test k8s + KubeVirt + KataContainers, which are all compatible with CRI-O kubelet will obsess on starting new containers, which looks like and endless process, and the cluster never gets ready. For instance here are some syslogs : What you expected to happen: It used to work (see asciinema recording at ) a few days ago, but I fear some recent changes broke it. How to reproduce it (as minimally and precisely as possible): Testing with vagrant up --provider=libvirt on Linux with a clone of that commit: vagrant ssh into the VM and ps -edf f or tail -f /var/log/syslog would do... Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: qemu/kvm VM +OS (e.g: ): Ubuntu 18.04.2 LTS Kernel (e.g. ): Linux kubernetes-vagrant-01 4.15.0-48-generic -Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm as in Network plugin and version (if this is a network-related bug): Others:
/sig node
I can confirm this. If I start cri-o from a systemd service it fails with the above mentioned error message. Starting crio with without systemd works just fine. systemd file:
Thanks for your detailed report :) Just want to do some quick version confirming - you've been using k8s 1.14.1 both when it did and did not work? Any other version changes?
Not sure for before, when it used to work, but according to the asciinema recording mentioned above (), it seems the same. However, not sure for CRI-O, underlying OS stuff (systemd, kernel, etc.). Sorry.
I was looking for some information for days. I have exactly the same issue when I use podman + systemd on 18.04.2.


Question: package v1_test import ( <del> "encoding/json" </del> <ins> "fmt" </ins> "reflect" "testing"

It is the same for both 14.15 and 14.18 kernels. Podman without systemd runs fine. Podman with systemd fails at runc with the same error podman[]: open /dev/stderr: no such device or address podman[]: NAME: runc - Open Container Initiative runtime podman[]: runc is a command line client for running applications packaged according to ... podman[]: time="2019-05-11T13:04:59-07:00" level=error msg="open /dev/stderr: no such device or addressn" podman[]: open /dev/stderr: no such device or address
I've tried to reproduce it on 18.04.2 using Podman from but I am not able to reproduce it. Do you have any reproducer? How did you install Podman?
I used ceph unit files to run ceph in podman rather than docker. Systemd unit successfully starts docker but fails with podman. I can successfully run the same podman container run command from the command line. How were you trying to replicate this? Can you post your unit file?
Hi, I'am having the same problem with Kubernetes with CRI-O. After joining a second worker node it fails to start flannel. gives the following error: What I've tried: Installing previous version of Kubernetes (from to ) Installing previous version of crio (from to ) Building runc myself from their github page and selecting this in the Running it on Ubunut 18.04.2 Issue: (the one giuseppe linked)
I wonder if the issue is with systemd and runc. In both cases, crio and podman, I see runc help output which tells me that something is not being passed to runc that it needs.
I've only used so it is different than your test and you confirmed it works from the CLI for you as well. Could you share your service file so I test it?
I've tried systemd and cgroupfs but no luck so far.
so it seems that runc fails to re-open . That seems like a runc issue, as it doesn't really need to reopen it. To fix the failure, you could use something like:
Just an example.


Question: package v1_test import ( <del> "encoding/json" </del> <ins> "fmt" </ins> "reflect" "testing"

I use ceph-ansible to install cluster so it may need to install additional daemons like mon before running mgr but the error, in this case, should be different, [Unit] Description=Ceph Manager [Service] EnvironmentFile=-/etc/environment ExecStartPre=-/usr/bin/podman stop ceph-mgr-storage2n1-la ExecStartPre=-/usr/bin/podman rm ceph-mgr-storage2n1-la ExecStart=/usr/bin/podman run --rm --net=host  --memory=16G  --cpus=4  -v /var/lib/ceph:/var/lib/ceph:z  -v /etc/ceph:/etc/ceph:z  -v /var/run/ceph:/var/run/ceph:z  -v /etc/localtime:/etc/localtime:ro  -e CLUSTER=ceph  -e CEPHDAEMON=MGR  -e CONTAINERIMAGE=  {{ cephmgrdockerextraenv }}  --name=ceph-mgr-storage2n1-la  ExecStopPost=-/usr/bin/podman stop ceph-mgr-storage2n1-la Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi- delighted <:
BTW, I projectatomic ppa and used apt-get -y podman to install. My podman version is 1.3.0 Sasha Litvak — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
Corrected file.


Question: package v1_test import ( <del> "encoding/json" </del> <ins> "fmt" </ins> "reflect" "testing"

[Unit] Description=Ceph Manager [Service] EnvironmentFile=-/etc/environment ExecStartPre=-/usr/bin/podman stop ceph-mgr-storage2n1-la ExecStartPre=-/usr/bin/podman rm ceph-mgr-storage2n1-la ExecStart=/usr/bin/podman run --rm --net=host  --memory=16G  --cpus=4  -v /var/lib/ceph:/var/lib/ceph:z  -v /etc/ceph:/etc/ceph:z  -v /var/run/ceph:/var/run/ceph:z  -v /etc/localtime:/etc/localtime:ro  -e CLUSTER=ceph  -e CEPHDAEMON=MGR  -e CONTAINERIMAGE=  --name=ceph-mgr-storage2n1-la  ExecStopPost=-/usr/bin/podman stop ceph-mgr-storage2n1-la Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi- Giuseppe Scrivano <:
Would it have an effect system wide? Can I add it to a unit file or unit override instead? Giuseppe Scrivano <:
yes, I think it is better to add it only to the unit file
Why would runc work outside of systemd? And if it needs stderr, giving it Null in settings will redirect stderr to Null I presume? On Tue, May 14, 2019, 9:46 AM Giuseppe Scrivano <:
Running into same issues on Ubuntu 18.04.2 using cri-o-1.13 via PPA.
PR here: opencontainers/runc
Until it is avaialble with project atomic and such, can I use a null workaround? Giuseppe Scrivano <:
yes, but you will lose anything written to stderr. You could probably also use a file, like
What about actually mounting dev and using /dev/stderr ? On Tue, May 14, 2019, 3:11 PM Giuseppe Scrivano <:
is a symlink to . It is present, the issue here is that is a socket. The fix for runc got merged, could you use an updated runc?
I tried apt update/upgrade and restarting crio, but I'm still getting the issue.


Question: package v1_test import ( <del> "encoding/json" </del> <ins> "fmt" </ins> "reflect" "testing"

Could it be that crio's runc is not updated yet? Edit Build new runc from and it is working now. Think crio hasn't updated it yet with their repository.
I'm running this version from PPA repo for ubuntu and it fails: could you point to version/commit or whatever helps identify the fixed version/snapshot ? Thanks in advance.
at the moment it is only fixed upstream. You'd need to build runc manually
Is there a version of the package on ppa we can downgrade to? On Wed, May 15, 2019, 4:35 AM Giuseppe Scrivano <:
could we update the runc version in the PPA repo?
Thank you. I seem them built but nut pushed to ppa yet.
should be available now.. let me know..
cri-o-runc (1.0.0-rc6-1~dev~ubuntu18.04~ppa42) bionic; urgency=medium autobuilt -- Lokesh Mandvekar (Bot) <, 15 May 2019 16:54:40 +0000
Thank you, everyone, for a quick turn around. Quick testing showed that fix is working, but more testing is needed. Lokesh Mandvekar <:
I confirm my initial tests with seem to confirm the regression is fixed. Many thanks
I also confirm the fix working for podman. Olivier Berger <:


Question: package v1_test import ( <del> "encoding/json" </del> <ins> "fmt" </ins> "reflect" "testing"

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): initContainers imagePullPolicy initContainers required value Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: cluster/kube- Others: What happened: When trying to create either a ReplicaSet or Replication Controller whose Pod template contains an Init Container, the validation fails saying is a required value. What you expected to happen: The create should succeed and the Init Container should receive a default . How to reproduce it (as minimally and precisely as possible): Try to create either of the following: ReplicaSet Replication Controller Anything else do we need to know: If I add to the Init Container in the ReplicaSet or Replication Controller Pod template, the create succeeds. When I create an Init Container without in a StatefulSet Pod template, it succeeds. When I create an Init Container without in a bare Pod, it succeeds:
Looks like defaulting not being applied on server? That is probably due to removing defaulting from convert. Surprised we didn't catch this earlier
Some values still not being defaulted
, I'm going to create a PR if none works on it.
We decode from bytes, set defaults and then convert; but for init containers, we delay the decode from "decode phase" to "convert phase" at .


Question: package v1_test import ( <del> "encoding/json" </del> <ins> "fmt" </ins> "reflect" "testing"

The defaults has been handled for initContainer at . /cc I'm working to find a solution for that; if any suggestion, please let me know.
Would this bug affect the deletion of jobs with an init-container? I ran into this bug and adjusted my manifest to add the pull policy to the init-container. However, I now cannot delete said job claiming a field is immutable. Example:
, not sure now. Let me check it,
Same error here with version 1.5.1, 1.5.2 The Job "grafana-import-dashboards" is invalid: [0].imagePullPolicy: Required value With this setup: kubectl apply -f monitoring Not working even if I specify a pullpolicy in the yaml. If I roll back to 1.4.7, it is working. Thank you. Greg.
any updates on a PR to address this issue? Is anyone from able to pick this issue up otherwise?
, here's the latest status: there's is a PR to generate for each Object (), it works for Object (Pod, Job) now but List; I'm working to make it generating for List.
Why would you need List to work? No client side logic should be decoding lists except as a generic submission.
The List tests are failed in . There's no for List, so no code to decode init-container annotation, e.g. of should call 's recursively.
Thanks for the update
+1
Any update on this? I still getting this issue with the last version of 1.5.3 on GKE?
ping for the comments? should we merge the PR for 1.5?
, there's a PR () that handled that issue; but we're discussing not to merge it because initContainers had been moved to fields, and fields is the recommended way to use since 1.6. But I'd like to get & 's comments on 1.5.
This is still an issue in 1.5.3 when using annotations: Without the deployment would not be accepted
ping & :).
i dont think this is fixed in 1.5 I just encountered this bug on 1.5.3
, yes, the PR is NOT merged to any branch, so the issue is still here. I'll create a PR for 1.5.
Hi Anyone help this issue.


Question: package v1_test import ( <del> "encoding/json" </del> <ins> "fmt" </ins> "reflect" "testing"

#cat mongo- apiVersion: v1 kind: Pod metadata: labels: name: mongo-svc-a name: mongo-deployment spec: replicas: 1 template: metadata: labels: name: mongo spec: containers: image: mongo name: mongo ports: name: mongo containerPort: hostPort: volumeMounts: mountPath: "/data/db" name: mongopv volumes: name: mongopv persistentVolumeClaim: claimName: mongopvc When i want create mongo db container by using this file folowing error is occured. The Pod "mongo-deployment" is invalid:spec.containers: Required value
, it seems there're two here, please remove the first one and have a try.
Hi my kubectl binary is configured at ubuntu 16.04 machine.


Question: package v1_test import ( <del> "encoding/json" </del> <ins> "fmt" </ins> "reflect" "testing"

NAME LABELS STATUS 10.1 Ready,SchedulingDisabled 10.2 10.1.10.2 Ready 10.3 Ready apiVersion: v1 kind: Node metadata: creationTimestamp: 2017-03-03T14:41:52Z labels: 10.1.10.1 name: 10.1.10.1 resourceVersion: "" selfLink: /api/v1/nodes/10.1.10.1 uid: -001f-11e7-8e83- spec: externalID: 10.1.10.1 status: addresses: address: 10.1.10.1 type: LegacyHostIP address: 10.1.10.1 type: InternalIP capacity: cpu: "2" memory: 4041352Ki pods: "110" conditions: lastHeartbeatTime: 2017-04-05T18:58:42Z lastTransitionTime: 2017-03-03T14:41:52Z message: kubelet has sufficient disk space available reason: KubeletHasSufficientDisk status: "False" type: OutOfDisk lastHeartbeatTime: 2017-04-05T18:58:42Z lastTransitionTime: 2017-04-05T15:27:52Z message: kubelet is posting ready status reason: KubeletReady status: "True" type: Ready nodeInfo: bootID: -6124-4ac7-ab96- containerRuntimeVersion: docker://1.12.6 kernelVersion: 4.9.16-coreos-r1 kubeProxyVersion: v1.2.4+coreos.1 kubeletVersion: v1.2.4+coreos.1 machineID: osImage: Container Linux by CoreOS 1298.7.0 (Ladybug) systemUUID: C3FD9852-2DFB-21F4-CE07-2597A746D99D When i create pv for creation pod is shows pv is pending. Please give any suggestions.....
Is there solution of this problem with version 2.7. Still facing the same. what exactly expectation of helm upgrade --install relname chartsdirectory is is expected to work on first time installation in case with CI/CD Pipelines.


Question: configs[NginxNew] = Config{list.PromoterE2eRegistry, "nginx", "1.15-2"} configs[NodePerfNpbEp] = Config{list.PromoterE2eRegistry, "node-perf/npb-ep", "1.2"} configs[NodePerfNpbIs] = Config{list.PromoterE2eRegistry, "node-perf/npb-is", "1.2"} <del> configs[NodePerfTfWideDeep] = Config{list.PromoterE2eRegistry, "node-perf/tf-wide-deep", "1.2"} </del> <ins> configs[NodePerfTfWideDeep] = Config{list.PromoterE2eRegistry, "node-perf/tf-wide-deep", "1.3"} </ins> configs[Nonewprivs] = Config{list.PromoterE2eRegistry, "nonewprivs", "1.3"} configs[NonRoot] = Config{list.PromoterE2eRegistry, "nonroot", "1.2"} // Pause - when these values are updated, also update cmd/kubelet/app/options/container_runtime.go

E2eNode Suite.[sig-node] Node Performance Testing [Serial] [Slow] Run node performance testing with pre-defined workloads TensorFlow workload From the testgrid, it has been 1/2 flaky since the start of history (03/20/2022) and has been failing constantly (0/2) since 04/01/2022 No response /sig node
/triage accepted /priority important-soon
The failing test seems to be expecting GPU on machine and the image seems to be broken. I tried the image manually: Logs:
/assign as discussed at SIG Node CI meeting, can you please take a look? I see that pod is now running for couple minutes and then stops. I wonder if this is something environment related - maybe pod was evicted or something. I doesn't seem to be able to repro the failure of the pod locally
GitHub didn't allow me to assign the following users: xmcqueen. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is no longer a flake. It has failed every run since the start of June. Locally this test runs perfectly with a legit correct local run of the test suite code, doing the full tensorflow run of 300 epochs. The failure in the testgrid job seems to be fast (173 sec to build and fail, locally its 160 sec before it starts the tensorflow pod). I wonder if there's a blocked port or something from the data initializing python program. It needs access to: . Here's the command it runs: Is it feasible to add debugging code, like a pre-check for that http port? It would be good to at least force it to fail with a meaningful error message.
Note that HEAD is now at image version 1.2, and it is still failing per . The successful local runs of the test mentioned above used image version 1.2.


Question: configs[NginxNew] = Config{list.PromoterE2eRegistry, "nginx", "1.15-2"} configs[NodePerfNpbEp] = Config{list.PromoterE2eRegistry, "node-perf/npb-ep", "1.2"} configs[NodePerfNpbIs] = Config{list.PromoterE2eRegistry, "node-perf/npb-is", "1.2"} <del> configs[NodePerfTfWideDeep] = Config{list.PromoterE2eRegistry, "node-perf/tf-wide-deep", "1.2"} </del> <ins> configs[NodePerfTfWideDeep] = Config{list.PromoterE2eRegistry, "node-perf/tf-wide-deep", "1.3"} </ins> configs[Nonewprivs] = Config{list.PromoterE2eRegistry, "nonewprivs", "1.3"} configs[NonRoot] = Config{list.PromoterE2eRegistry, "nonroot", "1.2"} // Pause - when these values are updated, also update cmd/kubelet/app/options/container_runtime.go

Locally image version 1.1 does produce the cuda/GPU error shown above by :
More info is needed to find the RC for this test failure. I've put in a PR to get the test framework to preserve the container logs when the pod fails.
The pod logs show that it is timing out on an http connection, and it looks like it's the one where it tries to download the test data set for the tensorflow run. You can see it . Search down to . The data_download script in the image attempts to pull down a dataset from: . That url is live and healthy, so they might be blocking the gcloud.
This test depending on that external data doesn't look quite right :/
I'm discussing this test with sig-testing. I'll post an update later.
Talked to from sig-testing: he's contacting the remote to see if they are blocking us, because it looks like they are is pulling the script from a that is So I see three choices: upgrade this thing to a more recent edition delete it grab everything that's needed to run this and keep it in-house <- clear winner here what do you think: Maybe the folks over at sig-testing have a place to store the artifacts in-house? They are not massive (about 10MB see below) Size of data about 10MB: the tools and libs for the tensorflow run: the training and eval files for the testrun:
Here's a relevant issue in a related k8s repo about using/not using docker registries: Registries used in k/k CI should be on Kubernetes Community infra
looks good for the subject of putting up a new docker image containing the deps. This seems feasible and effective. I'd prefer it to deleting the test. I'll discuss it a bit more with a few others, but this seems like the way to go forward with this. TBS, I plan to take the above listed test requirements, and build a docker image containing all that is needed, and reset the test to pull the new docker image.
I've got some Dockerfile changes to fix this. I've tested it locally with no network and it does the full 300 epochs. PR coming soon.
Is it worth pushing up 182 files to get this green? It is about 6M of python, and 5M of a test dataset.


Question: configs[NginxNew] = Config{list.PromoterE2eRegistry, "nginx", "1.15-2"} configs[NodePerfNpbEp] = Config{list.PromoterE2eRegistry, "node-perf/npb-ep", "1.2"} configs[NodePerfNpbIs] = Config{list.PromoterE2eRegistry, "node-perf/npb-is", "1.2"} <del> configs[NodePerfTfWideDeep] = Config{list.PromoterE2eRegistry, "node-perf/tf-wide-deep", "1.2"} </del> <ins> configs[NodePerfTfWideDeep] = Config{list.PromoterE2eRegistry, "node-perf/tf-wide-deep", "1.3"} </ins> configs[Nonewprivs] = Config{list.PromoterE2eRegistry, "nonewprivs", "1.3"} configs[NonRoot] = Config{list.PromoterE2eRegistry, "nonroot", "1.2"} // Pause - when these values are updated, also update cmd/kubelet/app/options/container_runtime.go

Its all stashed away in the to get the docker image to contain the prereqs. It is selected subdirectories of the quite old archived version (v1.9.0) of the code from here: current version v2.7.2 and two small test data sets from here:
OK I got it down to 43 files, including the 2 datasets. That's the drivers and the datasets and the libs they need. It all gets packed into the docker image.
quick question - is there another repository that is under sig-node that can host this image?
To be clear, the test's docker image is currently in a repo, the : This PR adds artifacts to an existing docker image in the above repo.
I've cancelled the above PR. I'll try to get something else going.
There has been a decent amount of discussion in slack relating to handling the test's required artifacts. The solution that seems best is to get some practical version of the test requirements into locally owned and controlled storage, so the endpoints that provide access to the assets will at least be under internal control. This is not code, and is not under development. It is a static dataset, that is now stored in an offsite archive, and will not require maintenance. The datasets are published with the "": (see bottom right) "This allows for the sharing and adaptation of the datasets for any purpose, provided that the appropriate credit is given." Here are some links to the dataset and where its hosted: https://archive-
I'd like to get the dataset that holds the test data (about 650k) hosted in GCS where I can get it via http. With that change it is expected that the test will go green. The failures show it is failing when it tries to download the artifact from the uncontrolled remote endpoint. Somebody, please help me get this artifact into an internally controlled server. Thanks
Was the idea of downloading this only on image creation in dockerfile rejected?
Thanks for the reminder I did not fairly evaluate that proposal. I just jumped to my earlier proposal. Thinking about this proposal I see that it would mean downloading the test data for inclusion into the docker image, while the image's test driver tooling is built to pull down the dataset itself from a remote.


Question: configs[NginxNew] = Config{list.PromoterE2eRegistry, "nginx", "1.15-2"} configs[NodePerfNpbEp] = Config{list.PromoterE2eRegistry, "node-perf/npb-ep", "1.2"} configs[NodePerfNpbIs] = Config{list.PromoterE2eRegistry, "node-perf/npb-is", "1.2"} <del> configs[NodePerfTfWideDeep] = Config{list.PromoterE2eRegistry, "node-perf/tf-wide-deep", "1.2"} </del> <ins> configs[NodePerfTfWideDeep] = Config{list.PromoterE2eRegistry, "node-perf/tf-wide-deep", "1.3"} </ins> configs[Nonewprivs] = Config{list.PromoterE2eRegistry, "nonewprivs", "1.3"} configs[NonRoot] = Config{list.PromoterE2eRegistry, "nonroot", "1.2"} // Pause - when these values are updated, also update cmd/kubelet/app/options/container_runtime.go

That is a simple change to handle, while simply re-hosting the artifact on GCS would require only changing the endpoint. Also building the docker would probably still require network access to the artifact, which would probably require us to put it somewhere anyway. I assume, putting it into the docker image would best be done by downloading the test data artifact from somewhere. Would we then be storing the artifact twice - once in the docker image, and once in a locally controlled endpoint? If the docker build server has unrestricted access to the artifact, this proposal would be good. I don't have access to the servers to run tests. However, I could put up an RB for the Dockerfile to try it out. This is probably worth the effort. I'll get an PR going for trying to pull down the remote test-data artifact from the image build server, and bunde it into the docker image, and I'll link it to this ticket.
thank you. We do not change the image for a long time, so downloading it once would likely be enough. Once it is self-contained, it is even better for the continuous perf runs in tests.
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Currently the image builds for the first arch (amd64) but fails when it enters the cross compiling phase (arm64) and specifically when it tries to do apt installs from within the qemu-*-static emulator: The other images in node-perf fail to build in the same way. The jessie-dnsutils image build fails to build in the same way. I have tried pulling in newer base images up to buster, but it did not fix the problem. -Start focusing on the version of qemu. - qemu is version locked in the Makefile and has not changed in two years. There was an and the images were building at that time.


Question: configs[NginxNew] = Config{list.PromoterE2eRegistry, "nginx", "1.15-2"} configs[NodePerfNpbEp] = Config{list.PromoterE2eRegistry, "node-perf/npb-ep", "1.2"} configs[NodePerfNpbIs] = Config{list.PromoterE2eRegistry, "node-perf/npb-is", "1.2"} <del> configs[NodePerfTfWideDeep] = Config{list.PromoterE2eRegistry, "node-perf/tf-wide-deep", "1.2"} </del> <ins> configs[NodePerfTfWideDeep] = Config{list.PromoterE2eRegistry, "node-perf/tf-wide-deep", "1.3"} </ins> configs[Nonewprivs] = Config{list.PromoterE2eRegistry, "nonewprivs", "1.3"} configs[NonRoot] = Config{list.PromoterE2eRegistry, "nonroot", "1.2"} // Pause - when these values are updated, also update cmd/kubelet/app/options/container_runtime.go

Try to find clues over there.
I tried pinning many relevant versions of from but none helped. Here's a clue: This ugly hack gets it past the observed error:
Note the previous PR was missing an important change, but I do not see any feasible way that could explain the current problem with failing, so I have not felt any urgency to put up a PR for it. FTR I need to bump the VERSION. I did not bump the version in I thought maybe the CI would bump it automatically.
There's an image version in . Its still pulling the old version and failing. Bump it from 1.2 to 1.3. So one more commit is needed.
The new image is present and runs perfectly. Tested it on local linux and in google cloud: It ran to completion. Note the new image URL. Its NOT the same as the old one:
podspec references the image Here's the . tf-wide-deep's url is built The above has staging in it. . Its already built and pushed to staging. It must be promoted when this new version number goes out.
FTR: My "testing" above did not catch the current problem because I did not run the full e2e_test binary. I only tested the image via a . AND I should have read way back at the beginning.
Local run of : has to change too.
Local test using the published 1.3 image run the standard way was successful with the above: ran the test successfully. So promote the image per bump the image version in the e2e test manifest change the
After the image promotion is merged, there's STILL one more commit coming: bump the image version in the e2e test manifest change the
its promoted and is found at the expected url local containerd cleared out local test was rerun pulling from the new prod registry local test using that new registry passes
its fixed. the test is green again.


Question: } if inline := cmdutil.GetFlagString(cmd, "overrides"); len(inline) > 0 { <del> object, err = cmdutil.Merge(scheme.Codecs.LegacyCodec(scheme.Scheme.PrioritizedVersionsAllGroups()...), object, inline) </del> <ins> codec := runtime.NewCodec(scheme.DefaultJSONEncoder(), scheme.Codecs.UniversalDecoder(scheme.Scheme.PrioritizedVersionsAllGroups()...)) object, err = cmdutil.Merge(codec, object, inline) </ins> if err != nil { return err }

Hey Guys , I try to use the option --override with kubectl expose . First i did this and it work : Now i want to add another service with kubectl expose and this command does not work I have this error If i remove --overrides the command work . `
/sig cli
expose tries to convert version from "v1" to "_internal" which is not registered. This is the reason that the function doesn't works. == "" 316 if inline := cmdutil.GetFlagString(cmd, "overrides"); len(inline) 0 { 317 object, err = cmdutil.Merge(scheme.Codecs.LegacyCodec(scheme.Scheme.PrioritizedVersionsAllGroups()...), object, inline) 318 if err != nil { 319 return err 320 } 321 } === 156 // LegacyCodec encodes output to a given API versions, and decodes output into the internal form from 157 // any recognized source. The returned codec will always encode output to JSON. If a type is not 158 // found in the list of versions an error will be returned. 159 // 160 // This method is deprecated - clients and servers should negotiate a serializer by mime-type and 161 // invoke CodecForVersions. Callers that need only to read data should use UniversalDecoder(). 162 // 163 // TODO: make this call exist only in pkg/api, and initialize it with the set of default versions. 164 // All other callers will be forced to request a Codec directly. 165 func (f CodecFactory) LegacyCodec(version ) runtime.Codec { 166 return versioning.NewDefaultingCodecForScheme(f.scheme, f.legacySerializer, f.universal, schema.GroupVersions(version), runtime.InternalGroupVersioner) 167 }
kubectl help expose Expose a resource as a new Kubernetes service. Possible resources include (case insensitive): pod (po), service (svc), replicationcontroller (rc), deployment (deploy), replicaset (rs) In order to control the scope, I only register them as un-versioned type in the fix.


Question: "strings" "time" <ins> "encoding/json" </ins> appsv1 "k8s.io/api/apps/v1" "k8s.io/api/core/v1" storage "k8s.io/api/storage/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/apimachinery/pkg/util/strategicpatch" "k8s.io/apimachinery/pkg/util/wait" </ins> clientset "k8s.io/client-go/kubernetes" podutil "k8s.io/kubernetes/pkg/api/v1/pod" "k8s.io/kubernetes/pkg/kubelet/apis" kubeletapis "k8s.io/kubernetes/pkg/kubelet/apis" "k8s.io/kubernetes/pkg/volume/util" "k8s.io/kubernetes/test/e2e/framework" <del> "k8s.io/kubernetes/test/e2e/framework/providers/gce" </del> "k8s.io/kubernetes/test/e2e/storage/testsuites" "k8s.io/kubernetes/test/e2e/storage/utils" imageutils "k8s.io/kubernetes/test/utils/image"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Testgrid: https://k8s-
/assign
The test's assumption about test parameter is wrong. Previously it assumed the group name is the same across all zones, but they are different in regional clusters. Looking into using node taints as the permanent fix, as suggested by the TODO comment in the test.
Hm I think taint based evictions is still an alpha feature.
OK. I propose the following solution: Get the zone of Schedule the StatefulSet pod to that zone using node affinity; wait for the pod to run Remove the node affinity from StatefulSet Delete
Can it be simplified? Schedule the pod Figure out which zone that pod got scheduled in Delete the instance group of that zone
Unfortunately I don't know of a good way to look up the instance group given the zone, since there could be instance groups from multiple clusters in the same project
How about: Schedule the pod Figure out the zone Taint all the nodes in the zone Kill the pod
Taint with NoSchedule? yeah that sounds good


Question: return c.restConfig, nil } <ins> func (c *fakeAuthenticationInfoResolver) ClientConfigForService(serviceName, serviceNamespace string) (*rest.Config, error) { atomic.AddInt32(c.cachedCount, 1) return c.restConfig, nil } </ins> func newMatchEverythingRules() []registrationv1beta1.RuleWithOperations { return []registrationv1beta1.RuleWithOperations{{ Operations: []registrationv1beta1.OperationType{registrationv1beta1.OperationAll},

We are getting the following error when the apiserver connects the webhook on localhost (configured via URL). We should only use the SSL tunnel for the connections to nodes when the webhooks are running as services. /sig api-machinery /kind bug /assign
/close


Question: // Create a security group for the load balancer sgName := "k8s-elb-" + loadBalancerName sgDescription := fmt.Sprintf("Security group for Kubernetes ELB %s (%v)", loadBalancerName, serviceName) <del> securityGroupID, err = c.ensureSecurityGroup(sgName, sgDescription) </del> <ins> securityGroupID, err = c.ensureSecurityGroup(sgName, sgDescription, getLoadBalancerAdditionalTags(annotations)) </ins> if err != nil { glog.Errorf("Error creating load balancer security group: %q", err) return nil, err

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig aws What happened: In k8s 1.7, when creating a service of type LoadBalancer, K8S creates the ELB and a Security Group for it. For these resources, it adds a tag, constructed dynamically, containing the cluster id: "${cluster_id}" with the value "true" What you expected to happen: We'd like to have also a statically named tag , as it was in 1.5 ("KubernetesCluster") with the value of the cluster id. How to reproduce it (as minimally and precisely as possible): Running on AWS cloud provider, create a service of type LoadBalancer. Anything else we need to know?: The reason for what we expect is that we have tight security constraints that require us to fine tune the IAM policies for the Instance profiles that the controllers use. With a statically named tag we were able, in 1.5, to restrict the permissions in the IAM policy to act only on the k8s managed resources (ELBs, security groups). Here is the relevant part of the IAM policy: With a dynamically named tag we are not able to do this any more as AWS IAM doesn't support something like: We are sharing the Instance Profiles among different k8s clusters, hence the "" in the value of the tag. Environment: Kubernetes version (use ):Cloud provider or hardware configuration*: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: kube-aws Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : <!-- This form is for bug reports and feature requests ONLY! -- /sig aws In k8s 1.7, when creating a service of type LoadBalancer, K8S creates the ELB and a Security Group for it. For these resources, it adds a tag, constructed dynamically, containing the cluster id: "${cluster_id}" with the value "true" We'd like to have also a statically named tag , as it was in 1.5 ("KubernetesCluster") with the value of the cluster id. Running on AWS cloud provider, create a service of type LoadBalancer.


Question: // Create a security group for the load balancer sgName := "k8s-elb-" + loadBalancerName sgDescription := fmt.Sprintf("Security group for Kubernetes ELB %s (%v)", loadBalancerName, serviceName) <del> securityGroupID, err = c.ensureSecurityGroup(sgName, sgDescription) </del> <ins> securityGroupID, err = c.ensureSecurityGroup(sgName, sgDescription, getLoadBalancerAdditionalTags(annotations)) </ins> if err != nil { glog.Errorf("Error creating load balancer security group: %q", err) return nil, err

The reason for what we expect is that we have tight security constraints that require us to fine tune the IAM policies for the Instance profiles that the controllers use. With a statically named tag we were able, in 1.5, to restrict the permissions in the IAM policy to act only on the k8s managed resources (ELBs, security groups). Here is the relevant part of the IAM policy: Environment: Kubernetes version (use ):OS (e.g. from /etc/os-release):Others: </details
I believe they are moving away from the tag since there are many resources that can be shared by multiple clusters. could you use an annotation?
We removed the condition in roles in the end. Thank you for your suggestion.
That annotation affects the ELB but not any security groups that are created. can you please reopen this? You reported a real issue, and even if it no longer affects you, it needs to be tracked and fixed.
Indeed, this might be a feature that someone else would need.
In fact, this is a feature I need!
Food for thought: should we propagate the tags on the ELB to the SG, or should we introduce an annotation for the SG specifically?


Question: // NewKubeRegistry creates a new vanilla Registry without any Collectors // pre-registered. func NewKubeRegistry() KubeRegistry { <del> r := newKubeRegistry(version.Get()) </del> <ins> r := newKubeRegistry(BuildVersion()) </ins> return r }

What would you like to be : Once a metric be deprecated, we should update the unit test accordingly if there is any. We need to work out a way for a unit test freely select codebase version, and also should make sure the unit test won't fail in the subsequent release. Why is this needed: Deprecated also needs to be protected. /sig instrumentation /assign


Question: // Returns true if and only if changes were made // The security group must already exist func (c *Cloud) setSecurityGroupIngress(securityGroupID string, permissions IPPermissionSet) (bool, error) { <ins> // We do not want to make changes to the Global defined SG if securityGroupID == c.cfg.Global.ElbSecurityGroup { return false, nil } </ins> group, err := c.findSecurityGroup(securityGroupID) if err != nil { glog.Warningf("Error retrieving security group %q", err)

/kind bug What happened: When setting an elbSecurityGroup in for AWS, and deploying multiple services with differing ports, then the security group rules flap in AWS, causing services to become inaccessible (only one will be correctly configured at a time). What you expected to happen: I expect that the configured elbSecurityGroup's rules will not be touched at all. It must be pre-configured to permit access. Alternatively, all ports for all services are queried and managed together. How to reproduce it (as minimally and precisely as possible): See above. Anything else we need to know?: Most people are not running large clusters which may hit the AWS limits, so they won't use this option, nor see this issue. Environment: Kubernetes version (use ): 1.7.1 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 4.10.13-1-ARCH Install tools: Custom Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig aws
This is fixed by


Question: // Next, we compare the current list of endpoints with the list of master IP keys formatCorrect, ipCorrect, portsCorrect := checkEndpointSubsetFormatWithLease(e, masterIPs, endpointPorts, reconcilePorts) if formatCorrect && ipCorrect && portsCorrect { <del> return nil </del> <ins> return r.epAdapter.EnsureEndpointSliceFromEndpoints(corev1.NamespaceDefault, e) </ins> } if !formatCorrect {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: EndpointSlice for masters was not created after enabling EndpointSlice on a pre-existing cluster. Only those EndpointSlices managed by kube-controller-manager were created. What you expected to happen: EndpointSlice for masters should be created after enabling EndpointSlice on a pre-existing cluster, the same as the EndpointSlices managed by kube-controller-manager. How to reproduce it (as minimally and precisely as possible): kubeadm to create a cluster: the EndpointSlice feature by editing the kube-apiserver, kube-controller-manager, kube-proxy manifest kube-apiserver and kube-controller-manager is restarted, the EndpointSlice in namespace will be missing. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools:Network plugin and version (if this is a network-related bug): Others:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved /assign


Question: local stack_skip="${3:-0}" stack_skip=$((stack_skip + 1)) <del> # Always print the error message, regardless of KUBE_VERBOSE value kube::log::error "${message}" # Additional debug information printed if KUBE_VERBOSE is 4 or greater </del> if [[ ${KUBE_VERBOSE} -ge 4 ]]; then local source_file=${BASH_SOURCE[${stack_skip}]} local source_line=${BASH_LINENO[$((stack_skip - 1))]}

appears to have been trying to improve the logging in e2e test runs, but it seems to have affected much more than that, and now when a build fails, it logs so many useless messages that you have to scroll back to see why your compile actually failed. before: after: /assign cc (re )
GitHub didn't allow me to assign the following users: oxxenix. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : appears to have been trying to improve the logging in e2e test runs, but it seems to have affected much more than that, and now when a build fails, it logs so many useless messages that you have to scroll back to see why your compile actually failed. after: /assign cc (re ) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks for tracking this down. I agree that the additional error message is distracting and would like to see implemented differently so that it's not shown for manual invocations.
/sig release
do you have any tips or recommendations how to implement it differently? Thanks
It looks like that extra ouput is a from . I'd suggest to either revert or strip BASHCOMMAND from the error message if it's not that useful.
It looks like what you wanted in is the location and time stamp ("The update now displays both the specific location of the error in the code and a timestamp."), which is what we also get from the calltree after "make build". I would revert and then investigate why that shorter calltree is not shown after an E2E test failure. Keeping the calltree around as in Dan's "before" seems reasonable to me.


Question: local stack_skip="${3:-0}" stack_skip=$((stack_skip + 1)) <del> # Always print the error message, regardless of KUBE_VERBOSE value kube::log::error "${message}" # Additional debug information printed if KUBE_VERBOSE is 4 or greater </del> if [[ ${KUBE_VERBOSE} -ge 4 ]]; then local source_file=${BASH_SOURCE[${stack_skip}]} local source_line=${BASH_LINENO[$((stack_skip - 1))]}

It shows a little bit more location info with the change. Here is how failure looks like with the change (but without BASH_COMMAND: and without the change:
I mean, the call tree is pretty useless in the "compile error" case... if your problem is that the code doesn't compile, you don't need to be told what part of the build scripts it is that invokes . It's just that before it was a small amount of useless output that was easy to ignore, whereas now it's a lot of useless output, which completely pushes the useful output offscreen. But IMO, "no backtrace" would be nicer than "short backtrace"... (I realize there are probably other cases where the backtrace is useful, and I'm not saying we should get rid of it in those cases, whatever they are.)
I agree, stack trace for compilation errors is quite useless. Unfortunately a lot of targets 'make test-e2e-node' included, build binaries, so we can't easily show traceback in one case and don't show in another. Removing BASH_COMMAND from the output looks like a good trade-off to me, though. It has a little bit more info(sometimes usefull) about callstack and less verbose than currently. I'm also ok with reverting and investigating later.
/assign


Question: if err != nil { return nil, err } <ins> // Fall back to ARM API if the address is empty string. // TODO: this is a workaround because IMDS is not stable enough. // It should be removed after IMDS fixing the issue. if strings.TrimSpace(ipAddress.PrivateIP) == "" { return addressGetter(name) } // Use ip address got from instance metadata. </ins> addresses := []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: ipAddress.PrivateIP}, {Type: v1.NodeHostName, Address: string(name)},

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /sig azure What happened: The internal IP address is got from instance metadata. I suspect the instance metadata may not be stable, e.g. it has undocumented rate limits (refer ). Cross refer What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:


Question: // ValidateServiceStatusUpdate tests if required fields in the Service are set when updating status. func ValidateServiceStatusUpdate(service, oldService *core.Service) field.ErrorList { allErrs := ValidateObjectMetaUpdate(&service.ObjectMeta, &oldService.ObjectMeta, field.NewPath("metadata")) <del> allErrs = append(allErrs, ValidateLoadBalancerStatus(&service.Status.LoadBalancer, field.NewPath("status", "loadBalancer"))...) </del> <ins> allErrs = append(allErrs, ValidateLoadBalancerStatus(&service.Status.LoadBalancer, field.NewPath("status", "loadBalancer"), &service.Spec)...) </ins> return allErrs }

xref I think this may be one of the few cases where we can actually change the behavior in a "breaking" way. User creates service with , is empty Controller or user updates the status subresource, sets (not wiped or validated, should be) User updates in any way, is wiped It's clear that the is not meant to be set when is not "LoadBalancer". The question is: 1) Fail validation? 2) Silently discard the value? 3) Warn and then discard the value? 4) Just warn and leave it. Because we allowed it in the past, but it is so volatile, I lean towards 3. and - the people most familair with this type. Edit: PrepareForUpdate() runs before warnings. So: (1) is easy (2) is easy (4) is slightly less easy than (2) (3) is slightly less easy than (4)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
OMG change it to clusterIP as expected Status is wiped update status as ClusterIP ... and if I edit the service back to loadbalancer it keeps the status??
/kind bug
/retitle Service can be set when !=
We talked about this one at the SIG Network meeting today. suggested we use the feature gate for deprecation option to phase the current (problematic) functionality out.
/triage accept
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: testKubelet := newTestKubelet(t) kl := testKubelet.kubelet kl.nodeLister = testNodeLister{nodes: []api.Node{ <del> {ObjectMeta: api.ObjectMeta{Name: "testnode", Labels: map[string]string{"key": "B"}}}, </del> <ins> {ObjectMeta: api.ObjectMeta{Name: testKubeletHostname, Labels: map[string]string{"key": "B"}}}, </ins> }} testKubelet.fakeCadvisor.On("MachineInfo").Return(&cadvisorApi.MachineInfo{}, nil) testKubelet.fakeCadvisor.On("DockerImagesFsInfo").Return(cadvisorApiv2.FsInfo{}, nil)

Looks like some kubelet unit tests perform a lookupIp which ends up depending on the health of the local resolver. Specifically has this failing: Which leads to the test in: And we recently swapped out the node names from testnode to 127.0.0.1, so probably the culprit?
/ref
Interesting. I did swap out the node names in the unit test as part of to be 127.0.0.1, party because I thought that it wouldn't have any problems resolving during unit tests.
/cc


Question: testKubelet := newTestKubelet(t) kl := testKubelet.kubelet kl.nodeLister = testNodeLister{nodes: []api.Node{ <del> {ObjectMeta: api.ObjectMeta{Name: "testnode", Labels: map[string]string{"key": "B"}}}, </del> <ins> {ObjectMeta: api.ObjectMeta{Name: testKubeletHostname, Labels: map[string]string{"key": "B"}}}, </ins> }} testKubelet.fakeCadvisor.On("MachineInfo").Return(&cadvisorApi.MachineInfo{}, nil) testKubelet.fakeCadvisor.On("DockerImagesFsInfo").Return(cadvisorApiv2.FsInfo{}, nil)

cc
What is the environment where you are running the tests? They pass for me on my mac, a linux box, and shippable seems to be passing them as well.
I couldn 't reproduce the failure either.
I'm out right now; i'll try them when i get back. For the record, I run hack/test- Paul Morie wrote:
Ahem FTR This acronym is going to go quad-plat Paul Morie wrote:
I ran on both platforms and saw the tests pass.
I'm still seeing it. Won't have bandwidth to dig in personally until tomorrow.
seeing a similar failure after a git pull
I knew I wasn't crazy
environment: fedora 21, golang: 1.4, docker: 1.6
I think I see the problem. When I set the node name to in the test I was assuming it would parse as a valid ip address and not end up at the LookupIP call. But only the node name (and not the kubelet hostname) is being set (except in my new test where I ) so it's trying to do a lookup. Do you think you could test just changing the default hostname from "testnode" to "127.0.0.1" and see if that fixes it for you?
Trying now
I still get failures but no panic / threaddump.
gist coming
this looks like again?
Good that it isn't panicing any longer. I can dig into the test failures tomorrow (I think it should be relatively simple to resolve).
cool, thanks for looking
Passes in master now for me. Closing.
FYI, they will break again once goes in, and then we can merge my PR to see if that fixes it for you.
:+1:
is in -- are these tests breaking again?
I just rebased which I think will fix them.
They are breaking again.
is on it.


Question: } } <ins> func TestMinionRESTWithHealthCheck(t *testing.T) { minionRegistry := registrytest.NewMinionRegistry([]string{}, api.NodeResources{}) minionHealthRegistry := HealthyRegistry{ delegate: minionRegistry, client:   &notMinion{minion: "m1"}, } ms := NewREST(&minionHealthRegistry) ctx := api.NewContext() c, err := ms.Create(ctx, &api.Minion{ObjectMeta: api.ObjectMeta{Name: "m1"}}) if err != nil { t.Errorf("insert failed") } result := <-c if m, ok := result.Object.(*api.Minion); !ok || m.Name != "m1" { t.Errorf("insert return value was weird: %#v", result) } if _, err := ms.Get(ctx, "m1"); err == nil { t.Errorf("node is unhealthy, expect no result from apiserver") } } </ins> func contains(nodes *api.MinionList, nodeID string) bool { for _, node := range nodes.Items { if node.Name == nodeID {

apiserver reports creation failure if a node is not ready (unhealthy, or kubelet is not ready), but the api object is already stored in master. extracted from and A short term solution is to have health_registry check node health upon creation, not just on retrieving. Or ignore unhealthy node. A long term solution might fall into the interface revamp
Why can't apiserver accept an unhealthy minion.
What's the point of having an unhealthy minion that we'll never be able to use? One reason I can think of is that the minion can later becomes healthy and thus a valid node in k8s. But for now, as we don't have the ability to handle unhealthy node, delegate the responsibility to user makes more sense. However, I admit the title is misleading. What I really what to convey is that kubernetes currently allow registrying unhealthy node, by storing it in RESTStorage. But the create operation reports fail because the node can't pass health checking. I don't think this is the right behavior. We can either return success and do health check later, or return failure and do not store it. (i prefer the latter one for the above reason).
I tend to think that we should accept unhealthy minions - minions may go in and out of health, or go up and down at any time - so trying ensure health at registration time seems like a fruitless effort since we have to handle the error cases anyway.
I've no problem with accepting unhealthy minions, and I agree with your point. The real problem here is that the apiserver returns an inconsistent status. For example, if I have an unhealthy minion that I want to registry. From user's perspective, the registration fails, because apiserver returns an error (resource not found). So I retry, but apiserver returns failure again; this time, the error is resource already exists, no matter the minion is healthy or not.
- yes, we should remove the health check as part of the registration and just report success. agree there is inconsistency.
+1 On Nov 7, 2014 8:48 AM, "Derek Carr" wrote:
Seems like it's a clients responsibility to warn the user that a node might not be available prior to it being , but the the master's responsibility to assume that all properly specified nodes are reachable before making decisions regarding using a node.


Question: } } <ins> func TestMinionRESTWithHealthCheck(t *testing.T) { minionRegistry := registrytest.NewMinionRegistry([]string{}, api.NodeResources{}) minionHealthRegistry := HealthyRegistry{ delegate: minionRegistry, client:   &notMinion{minion: "m1"}, } ms := NewREST(&minionHealthRegistry) ctx := api.NewContext() c, err := ms.Create(ctx, &api.Minion{ObjectMeta: api.ObjectMeta{Name: "m1"}}) if err != nil { t.Errorf("insert failed") } result := <-c if m, ok := result.Object.(*api.Minion); !ok || m.Name != "m1" { t.Errorf("insert return value was weird: %#v", result) } if _, err := ms.Get(ctx, "m1"); err == nil { t.Errorf("node is unhealthy, expect no result from apiserver") } } </ins> func contains(nodes *api.MinionList, nodeID string) bool { for _, node := range nodes.Items { if node.Name == nodeID {

There's very little difference between an admin stopping a VM without clearing Kube and adding a node to the cluster prior to it being fully reachable (I think).
"clients responsibility" ? if we enable kubectl to create minions, it would be kubectl who should check if the minion is reachable and complain? I'm not sure what 'client' means. (but I agree 100%, don't report failure adding a minion just because it isn't reachable)
node controller plugin should also give warning


Question: defer backendServer.Close() defer func() { <del> if called != tc.ExpectCalled { </del> <ins> if called := atomic.LoadInt32(&timesCalled) > 0; called != tc.ExpectCalled { </ins> t.Errorf("%s: expected called=%v, got %v", tcName, tc.ExpectCalled, called) } }()

There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery


Question: #     sleep 4 #     i=$[$i+1] #   done <del> apiVersion: v1beta1 </del> <ins> apiVersion: v1beta3 </ins> kind: Pod <del> id: synthetic-logger-10lps-pod desiredState: manifest: version: v1beta1 id: synth-logger-10lps containers: - name: synth-lgr image: ubuntu:14.04 command: ["bash", "-c", "i="0"; while true; do echo -n "`hostname`: $i: "; date --rfc-3339 ns; sleep 0.1; i=$[$i+1]; done"] labels: name: synth-logging-source </del> <ins> metadata: creationTimestamp: null labels: name: synth-logging-source name: synthetic-logger-10lps-pod spec: containers: - args: - bash - -c - 'i="0"; while true; do echo -n "`hostname`: $i: "; date --rfc-3339 ns; sleep 0.1; i=$[$i+1]; done' image: ubuntu:14.04 name: synth-lgr </ins>

Hi , I followed the instructions in examples/logging-demo, and there was no "external IP" field for elasticsearch-logging or kibana-logging, as shown below: I start up my cluster via cluster/kube- Did I miss something or is it a bug? Thanks.
Correct. This demo does not create an ELB so there is no external IP address. Access is via service proxy at the master. Type: and you should see the URLs for Elasticsearch and Kibana.
please let us know whether: a. Satnam's comment resolves your issue b. the demo should be updated to prevent this misunderstanding c. the issue isn't resolved Thank you!
Yes, Satnam's comment resolve the issue. I updated the demo in .
Awesome, thanks for sending the patch! We'll discuss there. Always fantastic to have docs updated regularly by folks trying them.


Question: return err } <del> // GetDiskLun finds the lun on the host that the vhd is attached to, given a vhd's diskName and diskURI func (ss *scaleSet) GetDiskLun(diskName, diskURI string, nodeName types.NodeName) (int32, error) { </del> <ins> // GetDataDisks gets a list of data disks attached to the node. func (ss *scaleSet) GetDataDisks(nodeName types.NodeName) ([]compute.DataDisk, error) { </ins> _, _, vm, err := ss.getVmssVM(string(nodeName)) if err != nil { <del> return -1, err </del> <ins> return nil, err </ins> } <del> disks := *vm.StorageProfile.DataDisks for _, disk := range disks { if disk.Lun != nil && (disk.Name != nil && diskName != "" && *disk.Name == diskName) || (disk.Vhd != nil && disk.Vhd.URI != nil && diskURI != "" && *disk.Vhd.URI == diskURI) || (disk.ManagedDisk != nil && *disk.ManagedDisk.ID == diskURI) { // found the disk glog.V(4).Infof("azureDisk - find disk: lun %d name %q uri %q", *disk.Lun, diskName, diskURI) return *disk.Lun, nil } } return -1, fmt.Errorf("Cannot find Lun for disk %s", diskName) } // GetNextDiskLun searches all vhd attachment on the host and find unused lun // return -1 if all luns are used func (ss *scaleSet) GetNextDiskLun(nodeName types.NodeName) (int32, error) { _, _, vm, err := ss.getVmssVM(string(nodeName)) if err != nil { return -1, err } used := make([]bool, maxLUN) disks := *vm.StorageProfile.DataDisks for _, disk := range disks { if disk.Lun != nil { used[*disk.Lun] = true } } for k, v := range used { if !v { return int32(k), nil } } return -1, fmt.Errorf("All Luns are used") } // DisksAreAttached checks if a list of volumes are attached to the node with the specified NodeName func (ss *scaleSet) DisksAreAttached(diskNames []string, nodeName types.NodeName) (map[string]bool, error) { attached := make(map[string]bool) for _, diskName := range diskNames { attached[diskName] = false } _, _, vm, err := ss.getVmssVM(string(nodeName)) if err != nil { if err == cloudprovider.InstanceNotFound { // if host doesn't exist, no need to detach glog.Warningf("azureDisk - Cannot find node %q, DisksAreAttached will assume disks %v are not attached to it.", nodeName, diskNames) return attached, nil } return attached, err } disks := *vm.StorageProfile.DataDisks for _, disk := range disks { for _, diskName := range diskNames { if disk.Name != nil && diskName != "" && *disk.Name == diskName { attached[diskName] = true } } </del> <ins> if vm.StorageProfile.DataDisks == nil { return nil, nil </ins> } <del> return attached, nil </del> <ins> return *vm.StorageProfile.DataDisks, nil </ins> }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: VirtualMachineScaleSetVM and VirtualMachine are different data structures (because of different API versions), so the disk attach operation seems duplicate between vm and vmss. That means although disk operations are in same logic, some duplication are still required now. They should use same api version in the future and should be merged into same code base, e.g. move sharing logic to Refer What you expected to happen: vmss and vm disk operations should share same code logic. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I have checked with currently we can’t merge, e.g. following two structs are different since they using different sdk-for-azure versions. While in the end when sdk-for-azure version is upgraded, above two versions should be same and at that time, we should merge functions like ‘AttachDisk’, ‘DetachDiskByName’, 'GetDiskLun', GetNextDiskLun, DisksAreAttached


Question: if err != nil { return err } <ins> if len(rsrc.Versions) == 0 { return fmt.Errorf("ThirdPartyResource %s has no defined versions", rsrc.Name) } </ins> plural, _ := meta.KindToResource(unversioned.GroupVersionKind{ Group:   group, Version: rsrc.Versions[0].Name,

FYI. Currently the field of TPR is marked as which is wrong, actually if you try to create a TPR with empty versions, it will cause api server panic. Versions should be necessary field of TPR, we should validate this somehow.


Question: return unmountedVolumes } <ins> func (asw *actualStateOfWorld) SyncReconstructedVolume(volumeName v1.UniqueVolumeName, podName volumetypes.UniquePodName, outerVolumeSpecName string) { asw.Lock() defer asw.Unlock() if volumeObj, volumeExists := asw.attachedVolumes[volumeName]; volumeExists { if podObj, podExists := volumeObj.mountedPods[podName]; podExists { if podObj.outerVolumeSpecName != outerVolumeSpecName { podObj.outerVolumeSpecName = outerVolumeSpecName asw.attachedVolumes[volumeName].mountedPods[podName] = podObj } } } } </ins> func (asw *actualStateOfWorld) newAttachedVolume( attachedVolume *attachedVolume) AttachedVolume { return AttachedVolume{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --We have even encountered such case that pod continuously reports “Unable to attach or mount volumes for pod;...” when kubelet restarted. The reasons are listed in details as below: , as result of apiserver instantaneous unstablility, dswp get pvc‘s pv spec failed, then the pvc volume does not be to dsw. , syncStates() lists all the volume directories locally, and recontructs the volume from local volume, if the volume isn't in dsw, it will put the recontructed volume to asw, but now the volume.OuterVolumeSpecName isn't correct. chance, at this time, In , the fail to pvc volume is to dsw again normally in reconcile() -mountAttachVolumes(), it compares dsw and asw by volume's innerVolumename, it's all the same. but it could not sense the OuterVolumeSpecName dismatch. so it will not re-mount again to update asw cache , it checks pod's volume attach or mount success by comparing volume.OuterVolumeSpecName and , but this pv'c OuterVolumeSpecName isn't correct in asw. so pod continuously reports “Unable to attach or mount volumes for pod;...” reconcile() could sense the reconstruct volume's dismatch to the dsw, then re-mount again to make volume cache consistency Kubernetes version (use ): 1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
CC
I think this is a point that can be optimized. /cc
I will give a fix for this
Thanks for reporting the issue, this innerSpecName and outerSpecName are confusing and cause issues. Please ping me for review when you have a fix. Thanks!
Please don't triage your own issues immediately, give someone else a chance to review them :( /sig storage
okay, I know


Question: // DetachDisk detaches the disk specified by vmDiskPath func (vm *VirtualMachine) DetachDisk(ctx context.Context, vmDiskPath string) error { <del> vmDiskPath = RemoveStorageClusterORFolderNameFromVDiskPath(vmDiskPath) </del> device, err := vm.getVirtualDeviceByPath(ctx, vmDiskPath) if err != nil { klog.Errorf("Disk ID not found for VM: %q with diskPath: %q", vm.InventoryPath, vmDiskPath)

When user creates a static PV using folder name in path: on vSAN backed datastore then detach disk does not work because check fails - The code expects folder name to be UUID but it does not and hence it determines disk is not attached to the node and hence no detach is ever performed. /sig storage cc


Question: // satisfy the pod topology spread constraints. Hence it needs to preempt another low pod // to make the Pods spread like [<high>, <medium, low>]. runPausePod(f, mediumPodCfg) <del> e2epod.WaitForPodNotPending(cs, ns, mediumPodCfg.Name) </del> ginkgo.By("Verify there are 3 Pods left in this namespace") wantPods := sets.NewString("high", "medium", "low") <del> podList, err := cs.CoreV1().Pods(ns).List(context.TODO(), metav1.ListOptions{}) pods := podList.Items </del> <ins> var pods []v1.Pod // Wait until the number of pods stabilizes. Note that `medium` pod can get scheduled once the // second low priority pod is marked as terminating. // TODO: exact the wait.PollImmediate block to framework.WaitForNumberOfRunningPods. err := wait.PollImmediate(framework.Poll, framework.PollShortTimeout, func() (bool, error) { podList, err := cs.CoreV1().Pods(ns).List(context.TODO(), metav1.ListOptions{}) // ignore intermittent network error if err != nil { return false, nil } pods = podList.Items return len(pods) == 3, nil }) </ins> framework.ExpectNoError(err) <del> framework.ExpectEqual(len(pods), 3) </del> for _, pod := range pods { // Remove the ordinal index for low pod.

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: [sig-scheduling] SchedulerPreemption [Serial] PodTopologySpread Preemption validates proper pods are preempted Which test(s) are flaking: Testgrid link: Reason for failure: It's likely when the preemptor preempts low-priority pods, and gets running, the low-priority pods are still terminating. tries to wait for all low-priority pods to be totally deleted and do the final check. Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig scheduling /assign


Question: // applyPlatformSpecificDockerConfig applies platform-specific configurations to a dockertypes.ContainerCreateConfig struct. // The containerCleanupInfo struct it returns will be passed as is to performPlatformSpecificContainerCleanup <del> // after either: //   * the container creation has failed //   * the container has been successfully started //   * the container has been removed // whichever happens first. </del> <ins> // after either the container creation has failed or the container has been removed. </ins> func (ds *dockerService) applyPlatformSpecificDockerConfig(request *runtimeapi.CreateContainerRequest, createConfig *dockertypes.ContainerCreateConfig) (*containerCleanupInfo, error) { cleanupInfo := &containerCleanupInfo{}

What would you like to be : Periodically remove cleanupInfos for old containers from dockershim's map. One way of doing that would be to just forget about s older than say an hour, as they should always be short-lived. Why is this needed: introduced a new in-memory map for the dockershim, that could potentially (in pathological cases) cause memory leaks - for containers that use GMSA cred specs, get created successfully, but then never get started nor removed. This might be okay for the alpha release of the GMSA feature, but should be addressed in the beta. See
/assign wk8
/sig windows
/milestone v1.15
got moved out. let's move this issue with it as well /milestone v1.16
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
: yup we'd love to have this in 1.16, still waiting for kind reviewers on :) ...


Question: return } <ins> func isHostRunningCgroupV2(f *framework.Framework, host string) bool { result, err := e2essh.SSH("stat -fc %T /sys/fs/cgroup/", host, framework.TestContext.Provider) framework.ExpectNoError(err) framework.ExpectEqual(result.Code, 0) // 0x63677270 == CGROUP2_SUPER_MAGIC // https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html return strings.Contains(result.Stdout, "cgroup2") || strings.Contains(result.Stdout, "0x63677270") } </ins> func getNpdPodStat(f *framework.Framework, nodeName string) (cpuUsage, rss, workingSet float64) { summary, err := e2ekubelet.GetStatsSummary(f.ClientSet, nodeName) framework.ExpectNoError(err)

N/A [sig-node] NodeProblemDetector: N/A https://prow- Not cgroupv2 compatible when access cpu and memory data
/sig node
Hi, can i work on this ?
Sure. Thanks
/assign
/triage accepted
/priority important-soon
/cc
are you still working on this? Were you able to setup environment to repro the test failure? Let us know if you need any help. Thanks!
Sorry for the delay, I followed the wrong guide and ended up getting stuck. But now I have managed to set it up. Please give me few more days to work on it.
Thanks Feel free to reach out here or on slack if you need any help, thanks for working on this!
/retitle Make nodeproblemdetector test cgroupv2 compatible
Failing job:
Testgrid:


Question: } // Add volume to desired state of world <del> _, err = dswp.desiredStateOfWorld.AddPodToVolume( </del> <ins> uniqueVolumeName, err := dswp.desiredStateOfWorld.AddPodToVolume( </ins> uniquePodName, pod, volumeSpec, podVolume.Name, volumeGidValue) if err != nil { klog.ErrorS(err, "Failed to add volume to desiredStateOfWorld", "pod", klog.KObj(pod), "volumeName", podVolume.Name, "volumeSpecName", volumeSpec.Name())


Question: "io/ioutil" "os" "reflect" <ins> "strings" </ins> "testing" "github.com/imdario/mergo" <del> "k8s.io/kubernetes/pkg/client/restclient" </del> clientcmdapi "k8s.io/kubernetes/pkg/client/unversioned/clientcmd/api" )

In 1.1, if you specify "current-context: i-do-not-exist" in a kubeconfig, and there's no matching context by that name in the contexts, kubelet silently uses what I assume is either an empty or a default configuration. This can result in hilarity like wondering why your kubelet isn't using the TLS settings you provided, and spending several hours debugging the whole stack only to discover that you typo'd current-context :). It would be better if kubelet and other users of errored out on such a misconfiguration, to make the failure obvious and trivial.
cc
I think kubelet silently uses an empty configuration. Guess we should give a waring here.
Did you start to work on this?
Below is related code: can you confirm this? Besides if we want to log a waring in kubelet, we may need to get in kubelet.
cc
cc
I believe kubelet already uses . Adding a check in that current context exists if specified sounds like the right approach.
Are there any backwards compatibility implications here? would this break users who have existing configs (that fallback to default namespace)? , Jeff Lowdermilk wrote:
I think kubelet directly use not . Also already checks whether the user-specified current context exists, the problem is that if not exist, we just skip. We need to do something if the current context not exist, like log a warning.
Erroring on a specified and bad seems reasonable. If it is not specified and there's only one option, I don't think we should fail.
yeap, the case is that user specify the but not exist, in such case we should just fail.
If no one is working on this I can take it.
I think this is the root cause. Basically we should error on this not exist context. If you have extra bandwidth, please go ahead :)
This is related to - the defaulting is what allows us to continue. When that merges, only kubectl will have the "use localhost:8080" as defaulting.


Question: hosts := []string{} for _, n := range nodelist.Items { for _, addr := range n.Status.Addresses { <del> if addr.Type == addrType { </del> <ins> if addr.Type == addrType && addr.Address != "" { </ins> hosts = append(hosts, addr.Address) break }

Public IPs are expensive resources. Given that we're running scalability tests on 5k-node clusters, we are unnecessarily consuming 5k IPs which is a lot. We should switch our tests so that nodes have only private IP addresses. We are going to do that by configuring NAT so that nodes can talk to outside world (e.g. to download image from dockerhub). But that also requires changing some correctness tests which assume public IPs.
To summarize recent work: setting up private gce clusters with nat, feature is gated via the env test and dashboard with correctness tests on gce private cluster - https://k8s- 27 tests fail on fixing the tests - should fix majority of them
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
- is there anything missing to be done here?
No, we can close this one. It was done long time ago. Thanks for catching /close
Closing this issue. <detailsIn response to : No, we can close this one. It was done long time ago. Thanks for catching Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: setMaxElements:   42, expectedSetCost:  8, }, <ins> { name:             "enums with maxLength equals to the longest possible value", schemaGenerator:  genEnumWithRuleAndValues("self.contains('A')", "A", "B", "C", "LongValue"), expectedCalcCost: 2, setMaxElements:   1000, expectedSetCost:  401, }, </ins> } for _, testCase := range cases { t.Run(testCase.name, func(t *testing.T) {

pointed out to me that a CEL x-kubernetes-validations rule in a CRD was hitting the estimated cost limit when performing a simple "string has prefix" check on an en enum field. Putting a limit in the OpenAPI of the enum string field fixed the problem, but having to add a limit like this is counter-intuitive and unnecessary, since the allowed enum values already make the max allowed length obvious. The apiserver automatically computes the max length of an enum as the length of the longest allowed enum value. I believe we already refuse evaluate CEL rules on values that fail OpenAPI value validation, since we rely in the value validations to establish all length limits. I haven't tested it locally, but something like this should work (given enough nested arrays): As a workaround, one can manually set on the string enum, e.g.: 25+ All No response No response No response No response
/sig api-machinery /triage accepted /assign
/assign
I wasn't sure if we special-cased the maxLength / maxItems / maxProperties failures, or if all openapi validation had to pass before we started messing with CEL evaluation
it looks like we currently special-case specific openapi errors that short-circuit CEL validation, I'm not sure an enum validation failure would currently prevent evaluation:
The enum err is converted to () error which is not short-circuit here. But for defaulting case, all error would be short-circuit. I would suggest to add error into the blockingErr list to short-circuit CEL validation. Would this make sense?
yeah, that sounds good if we're going to rely on it to catch things that are longer than the maximum length enum value


Question: --allow "tcp:3389" & fi fi <ins> kube::util::wait-for-jobs || { code=$? echo -e "${color_red}Failed to create firewall rules.${color_norm}" >&2 exit $code } </ins> } function expand-default-subnetwork() {

Kubernetes e2e suite [It] [sig-network] Firewall rule should have correct firewall rules for e2e cluster See also a similar cluster with the same test failing: /kind failing-test /kind flake /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: WorkingDir string `gcfg:"working-dir"` // Soap round tripper count (retries = RoundTripper - 1) RoundTripperCount uint `gcfg:"soap-roundtrip-count"` <del> // Deprecated as the virtual machines will be automatically discovered. </del> <ins> // Is required on the controller-manager if it does not run on a VMware machine </ins> // VMUUID is the VM Instance UUID of virtual machine which can be retrieved from instanceUuid // property in VmConfigInfo, or also set as vc.uuid in VMX file. // If not set, will be fetched from the machine via sysfs (requires root)

Is this a BUG REPORT or FEATURE REQUEST?: What happened: Currently the vsphere cloud provider blocks the controller-manager from running if its running on a non-vmware host, because it reads and checks if its content starts with . What you expected to happen: The controller manager to start How to reproduce it (as minimally and precisely as possible): Start a on a non-vmware environment with the flags set Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig vmware
can you please share some more context- Why do you want vSphere cloud provider to work with non-vmware host (I believe you mean ESXi)? Where is this requirement coming from.
We are building a Kubernetes cluster management solution which does Kubernetes in Kubernetes and works by creating the control planes in one cluster, then creating worker instances on one of the supported platforms, among them vsphere. Currently the cluster hosting the control planes (We call it seed cluster) has to run on ESXi, otherwise the controller-manager crashes. Having the check in place for the Kubelets is completely fine and definitely makes sense, but for the controller-manager it should IMHO not be required.


Question: stdoutBuf := bytes.NewBuffer(nil) stderrBuf := bytes.NewBuffer(nil) w := newLogWriter(stdoutBuf, stderrBuf, &LogOptions{since: test.since, timestamp: test.timestamp, bytes: -1}) <del> err := w.write(msg) </del> <ins> err := w.write(msg, true) </ins> assert.NoError(t, err) assert.Equal(t, test.expectStdout, stdoutBuf.String()) assert.Equal(t, test.expectStderr, stderrBuf.String())

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Docker has a hardcoded 16K limit on log line length, which splits lines longer than that into multiple messages in the log file. in , however, doesn't account for this when is set to and prints the timestamp in front of every entry in the log file instead of every logical line. This leads to random timestamps appearing in the middle of log lines, potentially breaking the applications that parse these logs. What you expected to happen: inserts timestamps once per logical line, so original log data is preserved. Alternatively, allow API to request the logs in a sane format like JSON, which would allow merging the lines back and combining them with the timestamps more intelligently on the application side. How to reproduce it (as minimally and precisely as possible): Create : apiVersion: v1 kind: Pod metadata: name: log-lines spec: containers: name: main image: busybox command: ['sh', '-c', 'while sleep 1; do echo -n ; for i in $(seq 1 ); do echo -n A; done; echo ; done'] Start the pod with Fetch the logs with , or using the API () Observe random timestamps inserted in the middle of the log lines. Environment: Kubernetes version (use ):
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I will work on it. /sig cli /remove-sig node
The logs are being corrupted by the API endpoint in kubelet, so shouldn't it be in scope of SIG node?
The log is not corrupted by k8s. The same log could be got using "docker container logs -t CONTAINERID" I think you should post your requirement to Docker.


Question: stdoutBuf := bytes.NewBuffer(nil) stderrBuf := bytes.NewBuffer(nil) w := newLogWriter(stdoutBuf, stderrBuf, &LogOptions{since: test.since, timestamp: test.timestamp, bytes: -1}) <del> err := w.write(msg) </del> <ins> err := w.write(msg, true) </ins> assert.NoError(t, err) assert.Equal(t, test.expectStdout, stdoutBuf.String()) assert.Equal(t, test.expectStderr, stderrBuf.String())

Please refer following code Line 50 ~ 52 if config.Timestamps { logLine = append([]byte(msg.Timestamp.Format(jsonmessage.RFC3339NanoFixed)+" "), logLine...) }
kubelet doesn't rely on Docker when reading logs, it reads the files itself (see the code linked in the issue). I don't think Docker would change the forced line splitting (see for example), so this should be fixed in Kubernetes.
Your are right and kubelet read log file from host such as /var/log/pods/pods/NAMESPACEPOD-NAMExxx/....(JSON format) I will post a PR for it. /remove-sig cli /sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .


Question: stdoutBuf := bytes.NewBuffer(nil) stderrBuf := bytes.NewBuffer(nil) w := newLogWriter(stdoutBuf, stderrBuf, &LogOptions{since: test.since, timestamp: test.timestamp, bytes: -1}) <del> err := w.write(msg) </del> <ins> err := w.write(msg, true) </ins> assert.NoError(t, err) assert.Equal(t, test.expectStdout, stdoutBuf.String()) assert.Equal(t, test.expectStderr, stderrBuf.String())

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. are you or anybody on this thread are still interested to work on this? Please re-open if so. Meanwhile closing this ahead of bot doing it because the inactivity. /close
Closing this issue. <detailsIn response to : I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. Instructions for interacting with me using PR comments are available .