Question: * [kubectl](kubectl.md)	 - kubectl controls the Kubernetes cluster manager <del> ###### Auto generated by spf13/cobra on 5-Apr-2016 </del> <ins> ###### Auto generated by spf13/cobra on 3-Jun-2016 </ins> <!-- BEGIN MUNGE: GENERATED_ANALYTICS --> [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/user-guide/kubectl/kubectl_exec.md?pixel)]()

Need to finish so the client allows you to select an init container, and verify the kubelet works correctly.
(this is a nice intersection of new in 1.3, node and usability, if anyone wants to give it a shot. Pretty sure Clayton is busier than I am and I don't think I'll get to it)
Will be happy to review any changes - should be fairly straightforward.
Yes, init containers are "run to completion" always. , harryz wrote:
IIUC when the init containers of pod have not complete, the pod should be in status, which means will fail, if init containers run to completion, they will exit, still fail. I don't know what you mean exec should work on init containers?
the init container is still up and running even when the pod is in pending, we should be able to exec and debug why it's hung, for example.
add a straightforward exec implementation ptal.


Question: result.User = os.Getenv("USER") } <del> if bastion := os.Getenv("KUBE_SSH_BASTION"); len(bastion) > 0 { </del> <ins> if bastion := os.Getenv(sshBastionEnvKey); len(bastion) > 0 { </ins> stdout, stderr, code, err := runSSHCommandViaBastion(cmd, result.User, bastion, host, signer) result.Stdout = stdout result.Stderr = stderr

ci-kubernetes-e2e-gce-scale-correctness Kubernetes e2e suite: [sig-node] SSH should SSH to all nodes and run commands kubetest.Test Jul 10 14:01:51 UTC+0200 Kubernetes e2e suite: [sig-node] SSH should SSH to all nodes and run commands: more . Test /sig node
/cc
/triage accepted
Uhhh, this may help here
/milestone v1.22
/sig scalability FYI
I got annoyed that I couldn't click through to see the other logs, so now you can, ref: (spyglass doesn't render build- because it's 500MB, and the limit for rendering is 100MB)
/assign SIG Scalability chairs/TLs, please find someone to address or call this a non-release-blocking failure
Assigning to current scalability oncall (): /assign
This is sig-node issue - should be addres by SIG-node folks. /sig node
is being discussed as a fix for this and I'm not sure it's correct
is it sig scalability's intent that 5k node clusters should support SSH?
The SSH test in question passes in the "default" job: So the fact that this failure is unique to a scalability job means to me that scalability needs to own at least some part of this
I did a little bit of digging and ci-kubernetes-e2e-gce-scale-correctness uses option in kubetest, which according to seems to be setting KUBESSHBASTION env. So I believe this is main difference between our scalability job and "default".
Debugging the test in and providing a fix there. The crictl host PR () is doing something different.


Question: }) matchExpectations := ptrMatchAllFields(gstruct.Fields{ "Node": gstruct.MatchAllFields(gstruct.Fields{ <del> "NodeName":  Equal(framework.TestContext.NodeName), "StartTime": recent(maxStartAge), "SystemContainers": gstruct.MatchElements(summaryObjectID, gstruct.IgnoreExtras, gstruct.Elements{ "kubelet": sysContExpectations, "runtime": sysContExpectations, }), </del> <ins> "NodeName":         Equal(framework.TestContext.NodeName), "StartTime":        recent(maxStartAge), "SystemContainers": gstruct.MatchAllElements(summaryObjectID, systemContainers), </ins> "CPU": ptrMatchAllFields(gstruct.Fields{ "Time":                 recent(maxStatsAge), "UsageNanoCores":       bounded(100E3, 2E9),

Kubernetes version (use ): 6 Environment: GCE What happened: I created a cluster for Heapster testing purposes to debug failing test in It turned out that Summary API does not export system/misc stats in SystemContainers section (though there are kubelet and docker stats). Everything works as expected on nodes. See also https://k8s-
What node OS were you using? We don't create the system (misc) container on systemd, so it's expected that GCI nodes don't report those stats.
The default configuration for 1.4, which is: master: GCI nodes: Container VM
Are the nodes running 1.4.6 too? There was a bug with this in 1.4.5, but it should have been fixed in 1.4.6.
Yes, both are running in 1.4.6. If you want to reproduce from Heapster current head run . It will take some time and as a side effect you should have a cluster with the mentioned problem. This was catch by integration tests and I temporary disabled the check here
if you have spare cycles, can you look into this?
friendly ping
Looking at it now
I have found a couple things so far. On the containerVM nodes, cadvisor is exporting a container with "name": "/system" On the GCI Master, cadvisor is exporting a container with "name": "" The has this: "{% set system_container = "--system-cgroups=/system" -%}" (/system is the default misc container name) So the real question is: Should cadvisor export "/system" instead of "" on GCI? Or should we be configuring GCI to use "--system-"?
I thought we fix this very issue for 1.4 branch by a while back. Are you running into the same issue with 1.5?
I think I told you the wrong thing earlier. We don't support the misc (/system) cgroup on systemd systems (I can't remember why, but it was an explicit decision - or might know). The is something different, I believe a base systemd cgroup (slice). As for the misc container on ContainerVM nodes, we've been testing for the misc container in 1.5+ in the summary api test (https://k8s-) for a while, and I haven't seen this issue.


Question: }) matchExpectations := ptrMatchAllFields(gstruct.Fields{ "Node": gstruct.MatchAllFields(gstruct.Fields{ <del> "NodeName":  Equal(framework.TestContext.NodeName), "StartTime": recent(maxStartAge), "SystemContainers": gstruct.MatchElements(summaryObjectID, gstruct.IgnoreExtras, gstruct.Elements{ "kubelet": sysContExpectations, "runtime": sysContExpectations, }), </del> <ins> "NodeName":         Equal(framework.TestContext.NodeName), "StartTime":        recent(maxStartAge), "SystemContainers": gstruct.MatchAllElements(summaryObjectID, systemContainers), </ins> "CPU": ptrMatchAllFields(gstruct.Fields{ "Time":                 recent(maxStatsAge), "UsageNanoCores":       bounded(100E3, 2E9),

So if this is still reproducible, it's limited to 1.4. See this the bug was created for the post-fix.
Found the explanation here: The should use the flag to know the location that has the processes that are associated with , but it should not modify the cgroups of existing processes on the system during bootstrapping of the node. This is because is the on the host and it has not delegated authority to the to change how it manages .
IIUC this is intended behavior on systemd nodes. Based on the explanation provided, the kubelet doesnt have the authority to control "misc" processes on systemd.
+1. "misc" is just a bandaid for cvm. , David Ashpole <:
Ok, so I'm closing this issue.


Question: "errors" "testing" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/types" "k8s.io/kubernetes/pkg/volume" volumetest "k8s.io/kubernetes/pkg/volume/testing"

When user creates a static PV using folder name in path: on vSAN backed datastore then detach disk does not work because check fails - The code expects folder name to be UUID but it does not and hence it determines disk is not attached to the node and hence no detach is ever performed. /sig storage cc


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I created several Workloads which require up to ten ports for both UDP and TCP protocol. Now my logs get spammed with . What you expected to happen: should consider -combination instead of only . How to reproduce it (as minimally and precisely as possible): Deploy any service with a node-port with both protocols, e.g.: Anything else we need to know?: This was already acknowledged by here: However, as this thread addresses a different bug and seems kind of stuck, i'd like to suggest that this might be addressed in a new context. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Deployed via Rancher v2.4.3
/sig api-machinery
/remove-sig api-machinery /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
How did you get into this situation? The allocator should not have granted the same node port twice?
Oh right, that issue. The only real way out is to de-allocate the NodePorts and then re-allocate them, I think
As this happens for all of my ports in the said Production Application (the one where my log gets spammed) as well as complete fresh installation (where i verified the bug), i think there's no "way out" except fixing the bug. As said, for me it's necessary to assign both protocols to the same port. This might be a niche-use-case, but checking the it's actually very likely for ports to allow tcp AND udp. I'm not sure about the history of this, but it might be that issue was the origin, which was adressed in PR ("Allow services which use same port, different protocol to use the same nodePort for both"). But this PR doesn't contain the adjustment of the said , so still considers the behaviour as invalid, while it's actually allowed and supported.


/assign
Ahh, I thought you meant that you had accidentally been allocated the same nodePorts and I was very confused when you said "all of them". I have reproduced this issue, and I can see it by inspection. The obvious fix would be to collect ports split by protocol, but I think that's not right since we don't store it split. An even simpler fix would be for to build a . Then for each protocol, for each port, build that contains all the port numbers, but remove any duplicates ONLY IF they are duplicated across protocols. Duplicates in the same protocol should be retained (to report an error later). I don't have time to tackle this tonight (and write a test), but it seems like a good bug.
perhaps I can fix this problem /assign
things may be a little more complicated. Right now the is not aware of the protocol of the port so I guess simply rewriting function to return is not enough, the port allocator will still return on same node port even with different protocol. perhaps we need to refactor the to make it protocol-aware.
I don't think we need to do make the allocator be protocol aware. IIRC we allow you to use the same nodeport on different protocols ONLY within a single service. This means (I think) that we can do hyper-local deduplication as we collect the nodeports for each service. If we find the same nodeport number with different protocols, just remove the duplicates because it is allowed. I.e. a service that looks like: return - deduped. if ANY OTHER service tried to use it would be an error at the "rebuilt" allocator Likewise if a single Service claims to use the same nodeport number more than once on the same protocol, it should be an error: return - NOT deduped.
obviously, we want a great unit test here :P
ah got it! thanks for your detailed explanation!
Hi! I have filed a PR to resolve this issue, could you take a look?


Question: // Allow users to update labels and capacity oldMinion.Labels = minion.Labels oldMinion.Spec.Capacity = minion.Spec.Capacity <ins> // Clear status oldMinion.Status = minion.Status </ins> if !reflect.DeepEqual(oldMinion, minion) { glog.V(4).Infof("Update failed validation %#v vs %#v", oldMinion, minion)

Two problems: on node do not have namepace, but kubectl requires namespace. However, if explicitly set namespace to 'default', this will cause an error because we are updating not allowed field (only label update is allowed). is usually created by node controller, update node label with a file is tedious since user needs to create a file with id, resource.capacity, etc. can we make it easier?
It would make more sense to have a special global namespace than no namespace. We should make it easy in kubectl to update a single field or subset of fields, generally, for all objects. What this would do is GET, merge, and PUT.
I had a discussion with on IRC about namespace on node. We'd like to just use "" (empty string) as of now. As we are not supporting any namespace feature on node, it's less confusing if we can hide the concept for now. Thinking a little more, "" is the special global namespace. I'm going to update kubectl to not assume 'default' namespace on every resources. It works fine for pod, service, but not for node, external LB, etc.
There was some discussion about "" being (essentially) an alias to "default": is that dead? , Deyuan Deng wrote:
I've seen two versions of NamespaceALL ("") NamespaceAll is the default argument to specify on a context when you want to list or filter resources across An empty namespace is equivalent to the , but "default" is the canonical representation. Either means , or the definition conflicts - all namespace should include "default" namespace. Also, for things like Node, I'd really like a NamespaceNone.
- I think there is a confusion here because of a poor naming choice. is a constant intended to be used as a filter to LIST/WATCH across multiple namespaces. It is NOT intended to be the value that is set on a resource for persistence. The discussion was to update to which I think is still a good thing to do. I have held off doing that until after we embed the namespace as a path parameter in our URLs instead of a query parameter, which I plan to look at after the holidays. is a constant intended to be used as the namespace context of a request when it's not explicitly stated on a request URL.


Question: // Allow users to update labels and capacity oldMinion.Labels = minion.Labels oldMinion.Spec.Capacity = minion.Spec.Capacity <ins> // Clear status oldMinion.Status = minion.Status </ins> if !reflect.DeepEqual(oldMinion, minion) { glog.V(4).Infof("Update failed validation %#v vs %#v", oldMinion, minion)

Since create operations pull the namespace from the context during persistence, it is also the value intended to be persisted. This is referring to a request URL that does not specify a namespace. It is not referring to the persisted value in the database directly - although that is the value that gets pulled. I think a is a good thing to add AFTER we make the change to specify namespace as a path parameter instead of a query parameter, and we update/rename to something that clarifies that its a wild-card filter whose value is probably best represented as.
ok, sounds good. Thanks. Look forward to see api.NamespaceAll = "*" and api.NamespaceNone :)


Question: // EnvVarSource represents a source for the value of an EnvVar. // Only one of its fields may be set. type EnvVarSource struct { <del> // Selects a field of the pod: supports metadata.name, metadata.namespace, metadata.labels, metadata.annotations, </del> <ins> // Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['<KEY>']`, `metadata.annotations['<KEY>']`, </ins> // metadata.uid, spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs. // +optional FieldRef *ObjectFieldSelector

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I tried to create a pod with env variables referencing to annotations: What you expected to happen: Should Proceed. As in It stated is a valid option How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.2 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig docs
The API docs need to be updated to indicate the correct supported values. See for correct values
You are a lifesaver. Thank you!
/assign I can try to update the api documentation


Question: {% if grains['os_family'] == 'RedHat' -%} {% set daemon_args = "" -%} {% endif -%} <del> {% if grains.etcd_servers is defined -%} {% set etcd_servers = "-etcd_servers=http://" + grains.etcd_servers + ":4001" -%} </del> <ins> {% if grains.api_servers is defined -%} {% set api_servers = "-master=http://" + grains.api_servers + ":7080" -%} </ins> {% else -%} {% set ips = salt['mine.get']('roles:kubernetes-master', 'network.ip_addrs', 'grain').values() -%} <del> {% set etcd_servers = "-etcd_servers=http://" + ips[0][0] + ":4001" -%} </del> <ins> {% set api_servers = "-master=http://" + ips[0][0] + ":7080" -%} </ins> {% endif -%} <del> DAEMON_ARGS="{{daemon_args}} {{etcd_servers}}" </del> <ins> DAEMON_ARGS="{{daemon_args}} {{api_servers}}" </ins>

is sent as a fix. cc It's a test failure reported by the publishing robot.
The line it fails at:
It looks like a vendor copy vs gopath copy issue.
One possible fix is removing from the vendor/ directory. I'm testing locally.
Confirmed removing from the vendor/ directory fixed the test in robot.
We need to add the "issue filer" feature to the publishing robot, we can't rely on to report robot issues everytime :P I'm just not sure how long the publishing robot will be needed (i.e., when we will get rid of the staging/ directory) so not sure whether we should invest more time on the robot.


Question: func (s *gcepdSource) createVolume(f *framework.Framework) volInfo { var err error <del> framework.Logf("Creating GCE PD volume") s.diskName, err = framework.CreatePDWithRetry() framework.ExpectNoError(err, "Error creating PD") </del> <ins> framework.Logf("Creating GCE PD volume via dynamic provisioning") testCase := storageClassTest{ name:      "subpath", claimSize: "2G", } pvc := newClaim(testCase, f.Namespace.Name, "subpath") s.pvc, err = framework.CreatePVC(f.ClientSet, f.Namespace.Name, pvc) framework.ExpectNoError(err, "Error creating PVC") </ins> return volInfo{ source: &v1.VolumeSource{ <del> GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{PDName: s.diskName}, </del> <ins> PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: s.pvc.Name, }, </ins> }, } } func (s *gcepdSource) cleanupVolume(f *framework.Framework) { <del> if s.diskName != "" { err := framework.DeletePDWithRetry(s.diskName) framework.ExpectNoError(err, "Error deleting PD") </del> <ins> if s.pvc != nil { err := f.ClientSet.CoreV1().PersistentVolumeClaims(f.Namespace.Name).Delete(s.pvc.Name, nil) framework.ExpectNoError(err, "Error deleting PVC") </ins> } }

New tests for subpath consistently fail in ci-kubernetes-e2e-gce-multizone: https://k8s- /kind bug /sig storage
Is this a blocker for 1.10?
I would expect so, yes. /milestone v1.10
I'm on it /assign
The problem is likely due to inline GCE PD volumes not having enough zone information. Considering changing volumes to PVs instead.
fix was in progress in to change to a PV
/unassign
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: return fmt.Errorf("Error building CNI config: %v", err) } <ins> glog.V(3).Infof("Calling cni plugins to add container to network with cni runtime: %+v", rt) </ins> res, err := plugin.cniConfig.AddNetwork(plugin.netConfig, rt) if err != nil { return fmt.Errorf("Error adding container to network: %v", err)

Event for a pod: The cni plugins do exist at /opt/cni/bin. They cannot be executed and bash is complaining:
cc
I'm guessing this is a shared lib problem, since /usr/grte/v4/lib64/ld-linux-x86- doesn't exist on my gce vm. We should update docs to make it clear how someone would install manually (if they run into such problems again) and statically link (if the problem is indeed the shared lib) with something like Unsure how we'd support different GOOS (vend different tarballs vs checking in and building the plugin inline with our make scripts)
uggh. These binaries need to be ultra-static or else we need to containerize them.
Essentially doing would've prevented this bug :)
And see for the CNI_ARGS problems
also , re ... however, I'm not too sure if your comment was mean to be on this ticket.
cc
I have a working version. I uploaded my cni binary with bug fix on GCS. change source to change sourcehash to Welcome to give it a try.
Also see problem:


Question: utilruntime.HandleError(fmt.Errorf("unable to handle object in %T: %T", sched, obj)) return } <ins> klog.V(3).Infof("delete event for unscheduled pod %s/%s", pod.Namespace, pod.Name) </ins> if err := sched.SchedulingQueue.Delete(pod); err != nil { utilruntime.HandleError(fmt.Errorf("unable to dequeue %T: %v", obj, err)) }

This helps debugging scheduler's behavior, currently we don't have logs that indicate when the scheduler first saw a pod or a node. The log level should be v=3. /sig scheduling /cc
Do you mean add logging for callbacks in ?
yes, at least for add and delete events
Sure, I can also give this pr, before tomorrow
/assign
We actually have one v=2 log line for a Node being to the cache's tree: But it's probably good to keep this one.


Question: export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2 fi <ins> # For disabling elasticsearch ruby client sniffering feature. # Because, on k8s, sniffering feature sometimes causes failed to flush buffers error # due to between service name and ip address glitch. # And this should be needed for downstream helm chart configurations # for sniffer_class_name parameter. SIMPLE_SNIFFER=$( gem contents fluent-plugin-elasticsearch | grep elasticsearch_simple_sniffer.rb ) if [ -n "$SIMPLE_SNIFFER" ] && [ -f "$SIMPLE_SNIFFER" ] ; then FLUENTD_ARGS="$FLUENTD_ARGS -r $SIMPLE_SNIFFER" fi </ins> # Use exec to get the signal # A non-quoted string and add the comment to prevent shellcheck failures on this line. # See https://github.com/koalaman/shellcheck/wiki/SC2086

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Sometimes reported in fluent-plugin-elasticsearch repository for exhausted Elasticsearch cluster issue due to elasticsearch ruby client sniffering issue. This should be fixed with but k/k fluentd-elasticsearch image doesn't use this. Why is this needed: This change should be also applied in the downstream docker images and using helm charts such as
/sig instrumentation
/sig cloud-provider
/assign cosmo0920
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.


Question: export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libjemalloc.so.2 fi <ins> # For disabling elasticsearch ruby client sniffering feature. # Because, on k8s, sniffering feature sometimes causes failed to flush buffers error # due to between service name and ip address glitch. # And this should be needed for downstream helm chart configurations # for sniffer_class_name parameter. SIMPLE_SNIFFER=$( gem contents fluent-plugin-elasticsearch | grep elasticsearch_simple_sniffer.rb ) if [ -n "$SIMPLE_SNIFFER" ] && [ -f "$SIMPLE_SNIFFER" ] ; then FLUENTD_ARGS="$FLUENTD_ARGS -r $SIMPLE_SNIFFER" fi </ins> # Use exec to get the signal # A non-quoted string and add the comment to prevent shellcheck failures on this line. # See https://github.com/koalaman/shellcheck/wiki/SC2086

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "k8s.io/apimachinery/pkg/runtime/schema" runtimeserializer "k8s.io/apimachinery/pkg/runtime/serializer" utilruntime "k8s.io/apimachinery/pkg/util/runtime" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> discoveryendpoint "k8s.io/apiserver/pkg/endpoints/discovery/aggregated" genericfeatures "k8s.io/apiserver/pkg/features" utilfeature "k8s.io/apiserver/pkg/util/feature"

reported that conformance testing with the latest version of sonobuoy against a version 1.29.0-alpha.3 cluster fails on the support the 1.17 Sample API Server using the current Aggregator test: , related to I found the reason why the test is working fine in Kubernetes CI jobs but fails with sonobuoy is because the service account used by sonobuoy "sonobuoy-serviceaccount" doesn't have read permission of the non-resource url "/", which is used by the test to confirm the removal of group path: The permission of "sonobuoy-serviceaccount" is as below: There may be two ways to fix it: the RBAC rule of sonobuoy to include "get" of "/". However, could other test runners encounter this issue? Is there a doc listing the required permissions for running e2e tests? the e2e to skip checking the removal of the group path if the service account doesn't have permission to access the root path. Which way do we usually prefer? Running e2e with sonobuoy should succeed. a v1.29.0-alpha.3 cluster. No response <details</details<details</details<details</details<details</details<details</details<details</details
implemented this solution.
/sig api-machinery
thank you for opening this issue and working on a solution. I'm probably not the right person to decide on which solution is preferred. From my perspective, either solution is fine.
Note: Sonobuoy is not owned by any Kubernetes SIG. At most it is between SIG Architecture and the CNCF to choose whether to recommend it for running tests. SIG Testing owns the e2e image that sonobuoy can be used to schedule but not sonobuoy. IMHO this is a bug in running the tests via a pod, which is not how we develop the tests (they're run outside the cluster, by a cluster admin account typically). I would say 1) is the correct answer. xref: /sig architecture /sig testing /triage accepted cc
thanks for your comment. I guess 1) means non of the existing sonobuoy versions can pass K8s conformance test against a cluster 1.29.0-alpha3, as the RBAC rule is hardcoded.


Question: "k8s.io/apimachinery/pkg/runtime/schema" runtimeserializer "k8s.io/apimachinery/pkg/runtime/serializer" utilruntime "k8s.io/apimachinery/pkg/util/runtime" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> discoveryendpoint "k8s.io/apiserver/pkg/endpoints/discovery/aggregated" genericfeatures "k8s.io/apiserver/pkg/features" utilfeature "k8s.io/apiserver/pkg/util/feature"

I will file an issue to sonobuoy first.
question here is what are the set of client permissions required to run the conformance suite? perhaps a level of access or a more granular recommendation should be documented. what is your take on the root path requirement as approver for: EDIT: slack thread:
I agree, we already require e.g. privileged pods for a long time in conformance tests and we should make it clear that we expect a highly privileged client.
+1 for what is saying, that we should expect that the user running the tests has required permissions. These tests are not built for being run unprivileged.
i wanted to PR the admin note in our conformance docs under sig-arch, but then i saw this: so broke this guarantee. sounds like SIG Arch needs to comment.
This statement surprises me. Root discovery is one of the least-privileged permissions you can have on a cluster. For years this was even open for unauthorized users on the internet (if your api endpoint has been exposed). In other words, just add the permission. While thinking about it, it is surprising that the user in question does not have this access already. Don't we allow that for every authorized user?
In contrast, I hope that lots of privileged operations are tested by conformance today. If we don't test things like admission webhooks for example, we have much bigger problems. And admission webhooks are basically equivalent to cluster-admin.
that is outdated, we use hostNetwork pods one some of the tests IIRC
looks like we don't. before this ticket, i was under the impression that the conformance suite already requires admin and that all testers (like Sonobuoy) use admin level credentials. it is non-privileged (e.g., does not require root on nodes, access to raw network interfaces, or cluster admin permissions) looks like we may have to change that sentence completely.
Would have expected it here: The PR in question explicitly fixes the result returned by . If that is part of conformance (IMO it should), every conformance test needs access it obviously. Whether we require admin, or we add it manually, either would work.


Question: "k8s.io/apimachinery/pkg/runtime/schema" runtimeserializer "k8s.io/apimachinery/pkg/runtime/serializer" utilruntime "k8s.io/apimachinery/pkg/util/runtime" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> discoveryendpoint "k8s.io/apiserver/pkg/endpoints/discovery/aggregated" genericfeatures "k8s.io/apiserver/pkg/features" utilfeature "k8s.io/apiserver/pkg/util/feature"

adding to milestone and making release blocking ... we should not release 1.29 in a state that the most commonly used conformance tool will fail
IMO we should clarify the state and go fix the tool, but we should keep the test and we should NOT add skip. Conformance docs are not a release artifact, and sonobuoy release lifecycle is not a conformance artifact. If we can't get sonobuoy working in a reasonable time, we already know the tests can be run by scheduling the image (which is ours) directly with kubectl instead and we can document that in the upstream conformance docs. We need to fix the for this image: ,%20master%20(dev) And it should probably be to release-blocking in the future.
it looks like the e2e change made in was not consciously made part of conformance, it was just trying to functionally test the bugfix being made. For 1.29, I think we should revert the change to and make an integration test that validates the fix instead. We can revisit whether content of is part of conformance or not for 1.30
I opened to revert the e2e change and move the functional test to an integration test
Filed to try to get the image job passing again.
We probably need another issue to track 1.30 follow-up?
If no one has done it, I could create another issue and follow up with the items that have been discussed here: outdated privilege requirement in commonly used conformance tools like sonobuoy aware of the required priviledge to avoid similar situations in the future. permission of GET "/" to authorized users via the role for root discovery.


Question: }) It("should provide DNS for the cluster", func() { <del> if testContext.Provider == "vagrant" { </del> <ins> if providerIs("vagrant") { </ins> By("Skipping test which is broken for vagrant (See https://github.com/GoogleCloudPlatform/kubernetes/issues/3580)") return }

I noticed while running e2e tests locally that some tests were apparently being incorrectly skipped: I checked, and this is also true for our Jenkins runs. Currently it looks like and have this bug, though only the former actually skips the test, I think. The issue appears to be that the code inside the top-level in these files is being run before the testContext struct is filled out (and before is even called), and so testContext.Provider is empty. I'm not immediately sure what the proper fix is; I'm also worried that there might be other things broken by the fact that these blocks are called before initialization is complete. We also probably investigate if there is a better way to mark these tests as skipped; right now, there's no indication in the test report (, Jenkins test report, etc) that these tests were skipped.
I merged and which tried to disable these tests for non-{gke,gce} but apparently didn't follow-up and verify that they were still running on our platforms. /cc
Seems easy enough to fix. I'll have a PR out to fix this before Monday.


Question: }}, }, }, <ins> "long-port-name": { expectedErrors: 0, endpointSlice: &discovery.EndpointSlice{ ObjectMeta:  standardMeta, AddressType: addressTypePtr(discovery.AddressTypeIP), Ports: []discovery.EndpointPort{{ Name:     utilpointer.StringPtr(strings.Repeat("a", 63)), Protocol: protocolPtr(api.ProtocolTCP), }}, Endpoints: []discovery.Endpoint{{ Addresses: generateIPAddresses(1), }}, }, }, </ins> "empty-ports-and-endpoints": { expectedErrors: 0, endpointSlice: &discovery.EndpointSlice{

What happened: No endpoints created in the EndpointSlice for my service due to the port on the service being too long. Logs show: EndpointSlice created has no endpoint: What you expected to happen: Service that works with Endpoints should work with EndpointSlice. If this is intentional, it should be clearly documented How to reproduce it (as minimally and precisely as possible): Create a service with a long port name Environment: Kubernetes version (use ): 1.16 Cloud provider or hardware configuration: kind
/sig networking
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /sig networking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
For a rough measure of impact, Istio had 5 ports that were over this limit (and many more that were right at 15 characters), so I suspect it may not be terribly rare.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This needs to be resolved before endpointslices graduate from alpha
/assign
/remove-triage unresolved


Question: type makeLocalPodWith func(config *localTestConfig, volume *localTestVolume, nodeName string) *v1.Pod <del> func testPodWithNodeName(config *localTestConfig, testVolType localVolumeType, ep *eventPatterns, nodeName string, makeLocalPodFunc makeLocalPodWith, bindingMode storagev1.VolumeBindingMode) { </del> <ins> func testPodWithNodeConflict(config *localTestConfig, testVolType localVolumeType, nodeName string, makeLocalPodFunc makeLocalPodWith, bindingMode storagev1.VolumeBindingMode) { </ins> By(fmt.Sprintf("local-volume-type: %s", testVolType)) testVols := setupLocalVolumesPVCsPVs(config, testVolType, config.node0, 1, bindingMode) testVol := testVols[0]

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /assign What happened: The tests are failing again because the scheduler messages have changed yet again. I'm going to remove the check for the exact messages because this is at least the 3rd time this has happened. Instead depend on unit/integration tests to check for specific failures. Also there's no need for these tests to run across all types of local volumes because it's just testing the scheduler part.


Question: for _, epSvcPair := range connectionMap { if svcInfo, ok := proxier.serviceMap[epSvcPair.ServicePortName]; ok && svcInfo.GetProtocol() == v1.ProtocolUDP { endpointIP := utilproxy.IPPart(epSvcPair.Endpoint) <del> err := conntrack.ClearEntriesForNAT(proxier.exec, svcInfo.ClusterIPString(), endpointIP, v1.ProtocolUDP) </del> <ins> nodePort := svcInfo.GetNodePort() var err error if nodePort != 0 { err = conntrack.ClearEntriesForPortNAT(proxier.exec, endpointIP, nodePort, v1.ProtocolUDP) } else { err = conntrack.ClearEntriesForNAT(proxier.exec, svcInfo.ClusterIPString(), endpointIP, v1.ProtocolUDP) } </ins> if err != nil { klog.Errorf("Failed to delete %s endpoint connections, error: %v", epSvcPair.ServicePortName.String(), err) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT What happened: When deleting/recreating pods backing a UDP service, traffic from external senders stops flowing. What you expected to happen: Traffic would continue to flow to new pods. How to reproduce it (as minimally and precisely as possible): Based on reading of similar issues and commit messages it seems like this would be most noticable on high-volume UDP applications, statsd and syslog are good examples. a deployment and service for a UDP app. In my test case, this would be the container. an external node, create a small test script utilizing a statsd library such as pystatsd. The most important part is that the sender uses the same source-port for every packet sent. Ex: the sender. Install tcpdump into the pod and watch for incoming packets, example from the test case: the pod tcpdump in the newly created pod. Observe that no new matching packets are flowing. existing conntrack rules on all nodes in the cluster. You should see traffic start flowing to the pod again. example command for clearing conntrack rules on the nodes: Anything else we need to know?: I have tested this only on GKE-managed clusters that utilize google forwarding-rules for external LBs, but I believe the issue may be present on any platform where iptables mode for kube-proxy is used. Tested and verified on 1.8.6-gke.0 and 1.7.11-gke.1. Also tested with both Google Container-optimized OS and the Ubuntu-based node images. I have only test this on GKE It appears this was a known issue and should have been cleared up by: From my reading of , it seems as if the func should exec() to clear up UDP entries on service endpoint change, but the only place I see that func called is from syncProxyRules(). That func looks complex and possibly like it is only run occasionally and not on service/endpoint changes. Environment: Kubernetes version (use ): (Also tested on 1.7.11-gke.1) Cloud provider or hardware configuration: GKE-managed clusters, google cloud. OS (e.g.


Question: for _, epSvcPair := range connectionMap { if svcInfo, ok := proxier.serviceMap[epSvcPair.ServicePortName]; ok && svcInfo.GetProtocol() == v1.ProtocolUDP { endpointIP := utilproxy.IPPart(epSvcPair.Endpoint) <del> err := conntrack.ClearEntriesForNAT(proxier.exec, svcInfo.ClusterIPString(), endpointIP, v1.ProtocolUDP) </del> <ins> nodePort := svcInfo.GetNodePort() var err error if nodePort != 0 { err = conntrack.ClearEntriesForPortNAT(proxier.exec, endpointIP, nodePort, v1.ProtocolUDP) } else { err = conntrack.ClearEntriesForNAT(proxier.exec, svcInfo.ClusterIPString(), endpointIP, v1.ProtocolUDP) } </ins> if err != nil { klog.Errorf("Failed to delete %s endpoint connections, error: %v", epSvcPair.ServicePortName.String(), err) }

from /etc/os-release): tried both: and Kernel (e.g. ): and Install tools: Others:
/sig network
We also ran into this in kubernetes v1.7.6 using OpenShift 3.7:
~I believe this is the same as ~If so, it is fixed in 1.10: These are about HostPort, not NodePort, sorry for the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any other known workaround for this besides updating conntrack rules on the kube nodes? I tried deleting and recreating the service, but this didn't update the rules. When I restart the stream on the client side data flows correctly again (perhaps it's perceived as a new connection), but I have to restart the stream for each sender.
/remove-lifecycle stale
FWIW, this is not fixed in 1.10 - that fix was for HostPort, and this is still a bug for NodePort.
I am seeing this issue in 1.11. I am using LoadBalancer type.
has been working on a fix for this, FYI.
This issue is extremely impactful for telephony protocols that rely on static source ip and source port. Any update on a possible workaround? I was thinking a potential workaround might be to reduce the UDP timeout to something really short so the stale conntrack entries go away faster. Or maybe there is a way to flush conntrack on the event of pod deletion? Edit: Removed stupid idea of a workaround to disable conntrack for UDP. Can't disable conntrack for UDP because it's used for DNAT from node to container.
The workaround is to not use NodePorts. Other kinds of services are already correct.
not sure what you mean. I am seeing the issue with a LoadBalancer type so it does not affect NodePorts only. As a workaround I have created a small python script that watches the K8s API for pod deletions and clears the related conntrack entries when that happens. I can share it when I get to work on Monday if you want to try it out.


Question: for _, epSvcPair := range connectionMap { if svcInfo, ok := proxier.serviceMap[epSvcPair.ServicePortName]; ok && svcInfo.GetProtocol() == v1.ProtocolUDP { endpointIP := utilproxy.IPPart(epSvcPair.Endpoint) <del> err := conntrack.ClearEntriesForNAT(proxier.exec, svcInfo.ClusterIPString(), endpointIP, v1.ProtocolUDP) </del> <ins> nodePort := svcInfo.GetNodePort() var err error if nodePort != 0 { err = conntrack.ClearEntriesForPortNAT(proxier.exec, endpointIP, nodePort, v1.ProtocolUDP) } else { err = conntrack.ClearEntriesForNAT(proxier.exec, svcInfo.ClusterIPString(), endpointIP, v1.ProtocolUDP) } </ins> if err != nil { klog.Errorf("Failed to delete %s endpoint connections, error: %v", epSvcPair.ServicePortName.String(), err) }

I've my script . Note that it's not tested much so don't use it in a production environment. Also it cleans up all conntrack entries for a deleted pod, not only the UDP related ones.
We are also seeing this issue with type loadbalancer. We are on 1.11. Our use-case is the following: multiple devices are sending to an aggreagator machine. This machine forwards the UDP packages to a loadBalancer in gcp. When debugging the service, I noticed the following: The traffic is assigned to one pod, and the packages are not loadbalanced (client and server are the same cfr use-case) When redeploying a new cluster, Conntrack is cleared on termination signal of the old pod. (client receives connection refused) red/black and highlander deployment strategies have the same effect. How would I expect this to be working then? New cluster is . Conntrack is cleared Data is routed to NEW pods old pods are terminated Best Regards, Tim
I am seeing this exact same issue on our cluster which has k8s v1.14.5 . We have endpoint nginx pods backing a UDP service with nodeport. Suddenly one day we see traffic not hitting into nginx pod but reaching the node hosting the nginx pod. We could simulate the problem by restarting the nginx pods. I am assuming with this issue marked as closed; and having made its way into v1.14; we should not be seeing this. Would also like to point out; the one thing, in addition we have in the service is ExternalTrafficPolicy set to Local. Can anyone please guide/help on why we are seeing this in v.1.14.5 of our cluster.
Seeing this with k8s v1.16.5


Question: AttachDisk(isManagedDisk bool, diskName, diskURI string, nodeName types.NodeName, lun int32, cachingMode compute.CachingTypes) error // DetachDiskByName detaches a vhd from host. The vhd can be identified by diskName or diskURI. DetachDiskByName(diskName, diskURI string, nodeName types.NodeName) error <del> // GetDiskLun finds the lun on the host that the vhd is attached to, given a vhd's diskName and diskURI. GetDiskLun(diskName, diskURI string, nodeName types.NodeName) (int32, error) // GetNextDiskLun searches all vhd attachment on the host and find unused lun. Return -1 if all luns are used. GetNextDiskLun(nodeName types.NodeName) (int32, error) // DisksAreAttached checks if a list of volumes are attached to the node with the specified NodeName. DisksAreAttached(diskNames []string, nodeName types.NodeName) (map[string]bool, error) </del> <ins> // GetDataDisks gets a list of data disks attached to the node. GetDataDisks(nodeName types.NodeName) ([]compute.DataDisk, error) </ins> }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: VirtualMachineScaleSetVM and VirtualMachine are different data structures (because of different API versions), so the disk attach operation seems duplicate between vm and vmss. That means although disk operations are in same logic, some duplication are still required now. They should use same api version in the future and should be merged into same code base, e.g. move sharing logic to Refer What you expected to happen: vmss and vm disk operations should share same code logic. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I have checked with currently we can’t merge, e.g. following two structs are different since they using different sdk-for-azure versions. While in the end when sdk-for-azure version is upgraded, above two versions should be same and at that time, we should merge functions like ‘AttachDisk’, ‘DetachDiskByName’, 'GetDiskLun', GetNextDiskLun, DisksAreAttached


Question: } var clientBuilder controller.ControllerClientBuilder if s.UseServiceAccountCredentials { <del> if len(s.ServiceAccountKeyFile) > 0 { </del> <ins> if len(s.ServiceAccountKeyFile) == 0 { </ins> // It's possible another controller process is creating the tokens for us. // If one isn't, we'll timeout and exit when our client builder is unable to create the tokens. glog.Warningf("--use-service-account-credentials was specified without providing a --service-account-private-key-file")

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: During startup shows the following warning: However is set and refers to a valid X.509 private key file. All service accounts seem to be created despite the warning. How to reproduce it (as minimally and precisely as possible): start controller-manager with Anything else we need to know?: Environment: Kubernetes version (use ): v1.8.0 OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie)
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Thanks for the report. The condition for printing the warning is backwards, but has no effect on behavior. Fixed in
Still present in 1.8.3 , can it be ~backported~ cherry-picked to 1.8 branch please?
See for proposed cherry-pick


Question: <del> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package storage import ( "context" "fmt" "os/exec" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/kubernetes/test/e2e/framework" e2eoutput "k8s.io/kubernetes/test/e2e/framework/pod/output" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" "k8s.io/kubernetes/test/e2e/storage/utils" admissionapi "k8s.io/pod-security-admission/api" "github.com/onsi/ginkgo/v2" ) var _ = utils.SIGDescribe("GKE local SSD [Feature:GKELocalSSD]", func() { f := framework.NewDefaultFramework("localssd") f.NamespacePodSecurityEnforceLevel = admissionapi.LevelPrivileged ginkgo.BeforeEach(func() { e2eskipper.SkipUnlessProviderIs("gke") }) ginkgo.It("should write and read from node local SSD [Feature:GKELocalSSD]", func(ctx context.Context) { framework.Logf("Start local SSD test") createNodePoolWithLocalSsds("np-ssd") doTestWriteAndReadToLocalSsd(ctx, f) }) }) func createNodePoolWithLocalSsds(nodePoolName string) { framework.Logf("Create node pool: %s with local SSDs in cluster: %s ", nodePoolName, framework.TestContext.CloudConfig.Cluster) out, err := exec.Command("gcloud", "alpha", "container", "node-pools", "create", nodePoolName, fmt.Sprintf("--cluster=%s", framework.TestContext.CloudConfig.Cluster), "--local-ssd-count=1").CombinedOutput() if err != nil { framework.Failf("Failed to create node pool %s: Err: %vn%v", nodePoolName, err, string(out)) } framework.Logf("Successfully created node pool %s:n%v", nodePoolName, string(out)) } func doTestWriteAndReadToLocalSsd(ctx context.Context, f *framework.Framework) { var pod = testPodWithSsd("echo 'hello world' > /mnt/disks/ssd0/data  && sleep 1 && cat /mnt/disks/ssd0/data") var msg string var out = []string{"hello world"} e2eoutput.TestContainerOutput(ctx, f, msg, pod, 0, out) } func testPodWithSsd(command string) *v1.Pod { containerName := "test-container" volumeName := "test-ssd-volume" path := "/mnt/disks/ssd0" podName := "pod-" + string(uuid.NewUUID()) image := "ubuntu:14.04" return &v1.Pod{ TypeMeta: metav1.TypeMeta{ Kind:       "Pod", APIVersion: "v1", }, ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:    containerName, Image:   image, Command: []string{"/bin/sh"}, Args:    []string{"-c", command}, VolumeMounts: []v1.VolumeMount{ { Name:      volumeName, MountPath: path, }, }, }, }, RestartPolicy: v1.RestartPolicyNever, Volumes: []v1.Volume{ { Name: volumeName, VolumeSource: v1.VolumeSource{ HostPath: &v1.HostPathVolumeSource{ Path: path, }, }, }, }, NodeSelector: map[string]string{"cloud.google.com/gke-local-ssd": "true"}, }, } } </del>

Following up on , evaluate if the test for local SSD on GKE clusters is still necessary (it is also tested internally in Google). /assign
/sig storage
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle frozen
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This test should be removed as it's using a GKE feature that, while not deprecated, is discouraged in favor of newer APIs. Those APIs are well-tested internally in google, and I don't see value in additional toil upstream to maintain them.


Question: // Start the reconciler to fill ASW. stopChan, stoppedChan := make(chan struct{}), make(chan struct{}) go func() { <ins> defer close(stoppedChan) </ins> reconciler.Run(stopChan) <del> close(stoppedChan) </del> }() waitForMount(t, fakePlugin, volumeName, asw) // Stop the reconciler.

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-unit : (PR with no go code changes, should not have failed any unit tests) links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assigne
/assign
/assign tried to fix it in


Question: t.Errorf("Expected %v, got %v", e, a) } } <ins> func TestCacheParallelNoDeadlocksNoDoubleCalls(t *testing.T) { // Make 50 random keys keys := []string{} fuzz.New().NilChance(0).NumElements(50, 50).Fuzz(&keys) // Data structure for tracking when each key is accessed. type callTrack struct { sync.Mutex accessTimes []time.Time } calls := map[string]*callTrack{} for _, k := range keys { calls[k] = &callTrack{} } // This is called to fill the cache in the case of a cache miss // or cache entry expiration. We record the time. ff := func(key string) T { ct := calls[key] ct.Lock() ct.accessTimes = append(ct.accessTimes, time.Now()) ct.Unlock() // make sure that there is time for multiple requests to come in // for the same key before this returns. time.Sleep(time.Millisecond) return key } cacheDur := 10 * time.Millisecond c := NewTimeCache(RealClock{}, cacheDur, ff) // Spawn a bunch of goroutines, each of which sequentially requests // 500 random keys from the cache. runtime.GOMAXPROCS(16) var wg sync.WaitGroup for i := 0; i < 500; i++ { wg.Add(1) go func(seed int64) { r := rand.New(rand.NewSource(seed)) for i := 0; i < 500; i++ { c.Get(keys[r.Intn(len(keys))]) } wg.Done() }(rand.Int63()) } wg.Wait() // Since the cache should hold things for 10ms, no calls for a given key // should be more closely spaced than that. for k, ct := range calls { if len(ct.accessTimes) < 2 { continue } cur := ct.accessTimes[0] for i := 1; i < len(ct.accessTimes); i++ { next := ct.accessTimes[i] if next.Sub(cur) < cacheDur { t.Errorf("%v was called at %v and %v", k, cur, next) } cur = next } } } </ins>

Failed due to an apparently unrelated PR: https://travis-
Problem here is that I left a small chance of a double-call in my parallelized cache. Didn't realize this was being tested. Will fix.


Question: } } <ins> func TestWithMaxWaitRateLimiter(t *testing.T) { limiter := NewWithMaxWaitRateLimiter(DefaultControllerRateLimiter(), 500*time.Second) for i := 0; i < 100; i++ { if e, a := 5*time.Millisecond, limiter.When(i); e != a { t.Errorf("expected %v, got %v", e, a) } } for i := 100; i < 5100; i++ { if e, a := 500*time.Second, limiter.When(i); e < a { t.Errorf("expected %v, got %v", e, a) } } for i := 5100; i < 5200; i++ { if e, a := 500*time.Second, limiter.When(i); e != a { t.Errorf("expected %v, got %v", e, a) } } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Currently when using it adds as one of the limiters. This does not work as intended because there is no immediate sleep following the call to the limiter as seen in which does work correctly. The issue with the in its current usage is there is no forget nor a sleep immediately following the call so if is called thousands of times in a row, the timeout grows very quickly with no max. For example, if a clusters has 10k secrets and those are all enqueued using the the will end up with a 16 min wait before the tail end of objects can be enqueued and processed. I would expect some max backoff to take affect. This can be seen at
/sig api-machinery
/cc /triage accepted
it possible to set maxWaitingTime for BucketRateLimiter? when r.Limiter.Reserve().Delay() maxWaitingTime, the return of BucketRateLimiter When() is maxWaitingTime. defined is 1000ms / r.limit * MAXQPS(5000) this issue example (r.tokens = 100, r.limit = 10), if call When() times, the tail end of the object is enqueued waiting for 8.33 min. i'm not sure how to set the value of MAXQPS please help review it
The fix merged to close this issue does not fix the underlying issue with the DefaultControllerRateLimiter. The bucket is not being used correctly so this does not work as intended, please read my issue description again as to what is actually happening.
is another fix needed to solve this issue?
A token bucket rate limiter will only refill at the defined rate. If enough requeues are , it will take a long time for the refill rate to clear the queue. If a max wait is introduced, then requeues will be inserted into the queue faster than the refill rate of the token bucket rate limiter. A max wait will violate the token bucket rate limiter and we want token bucket rate limiting behavior.


Question: inferredHost = host } // Make a copy to avoid polluting the provided config <del> tlsConfigCopy, _ := utilnet.TLSClientConfig(transport) </del> <ins> tlsConfigCopy := utilnet.CloneTLSConfig(tlsConfig) </ins> tlsConfigCopy.ServerName = inferredHost tlsConfig = tlsConfigCopy }

When the master dials nodes, it is mutating the tlsConfig held by the master kubelet client, to add a server name. That means if two different nodes are dialed from a master started with , x509 verification will fail for the second one. Introduced in
this is a regression from 1.3 and breaks pod logs/exec/attach and node proxying on installations that use
We probably need to review that PR and look to see if any other deliberate copies were flagged incorrectly by govet.
Installations that depend on secure trust from externally provided kubelet server certs are the highest security rings. If we don't fix this we need to tell anyone using this flag not to upgrade until we release 1.4.1.
the rest of the changes in the govet cleanup PR looked fine to me. fix for this issue is in
Since we are telling folks they have to upgrade to kubectl 1.4.0 for rollingupdate to work it would definitely be a confusing message to say other folks should not upgrade to 1.4.0 at all.
If the fix is merged into master by this afternoon, lets discuss whether or not we should cherrypick it - e.g. risks involved, how we can manually qa to verify nothing else was broken, etc.
Risk involved: If you are using a secure cluster (defined as ensuring no one can MITM your nodes and masters) and you upgrade to 1.4.0 and it does not include this fix - you are broken.
What is the likelihood of interactions with other code? Is it possible this fix could introduce a breaking change to something else? Has the fix been manually verified? Are we confident that this will fully fixes the issue (vs other edge cases exist for secure clusters that this doesn't catch). Ideally I would like to see this cherrypicked into 1.4 this evening and then cut a new beta that includes it.
None. The single line that is currently broken was changed. The rest is a new function (with no side effects), and tests. No, it restores the original intent of the TLS config copy, and tests cover the cases we are using this with (secure, insecure, custom dialer) Yes. I swept for the same issue elsewhere and didn't find any issues. We test with a fully secured master-kubelet setup in Openshift and got a clean test run with this fix applied.
SGTM. Thanks for fixing this.


Question: migrated := getMigratedStatusBySpec(volumeToDetach.VolumeSpec) if err != nil { <del> // On failure, add volume back to ReportAsAttached list actualStateOfWorld.AddVolumeToReportAsAttached( logger, volumeToDetach.VolumeName, volumeToDetach.NodeName) </del> <ins> // On failure, mark the volume as uncertain. Attach() must succeed before adding the volume back // to node status as attached. uncertainError := actualStateOfWorld.MarkVolumeAsUncertain( logger, volumeToDetach.VolumeName, volumeToDetach.VolumeSpec, volumeToDetach.NodeName) if uncertainError != nil { klog.Errorf("DetachVolume.MarkVolumeAsUncertain failed to add the volume %q to actual state after detach error: %s", volumeToDetach.VolumeName, uncertainError) } </ins> eventErr, detailedErr := volumeToDetach.GenerateError("DetachVolume.Detach failed", err) return volumetypes.NewOperationContext(eventErr, detailedErr, migrated) }

After KCM restart, any Pod that reuses a volume that failed to detach gets stuck ContainerCreating with message: User "system:node:ip-10-0-243-" cannot get resource "volumeattachments" in API group "" at the cluster scope: no relationship found between node 'ip-10-0-243-' and this object See steps to reproduce below, it's quite a complex condition. It happens when a pod is restarted, a CSI driver "busy" (i.e. times out) + KCM restarted at a wrong time. Pods start just fine. a Pod that uses an attachable CSI volume. the Pod. Make sure it unmounts successfully, but any fails with a transient error, such as timeout. The external attacher continues to detach the volume at this point. kube-controller-manager (KCM) and start a new one. Wait for any leader election etc - it's important for the new KCM to populate A/D controller with no pod using the detaching volume. Let the A/D controller to observe detach error. a new Pod on the same node that uses the same volume. the CSI driver to detach the volume (i.e. stop injecting any transient errors). At this point: The external attacher succeeds detaching the volume, VolumeAttachment is deleted. The volume is marked as attached in node status (!!!) despite no VolumeAttachment exists. A/D controller does not even try to create a new VolumeAttachment, the volume is marked as attached in its Actual State of World (ASW). Kubelet tries to mount the volume, but WaitForAttach fails with: , basically forever. I created . The script expects that the cluster is created by , because it restarts KCM in the middle. It uses my hacked hostpath CSI driver to inject detach errors. Sources are at / the last commit(s) there. There are two issues that together lead to this situation: fails, , i.e. puts it back to . With the volume in , kubelet may try to mount it, while the external attacher tries to detach it. It is not as bad as it sounds, because kubelet can't read VolumeAttachment with the deletion timestamp and it will not call / of un-attached volume.


Question: migrated := getMigratedStatusBySpec(volumeToDetach.VolumeSpec) if err != nil { <del> // On failure, add volume back to ReportAsAttached list actualStateOfWorld.AddVolumeToReportAsAttached( logger, volumeToDetach.VolumeName, volumeToDetach.NodeName) </del> <ins> // On failure, mark the volume as uncertain. Attach() must succeed before adding the volume back // to node status as attached. uncertainError := actualStateOfWorld.MarkVolumeAsUncertain( logger, volumeToDetach.VolumeName, volumeToDetach.VolumeSpec, volumeToDetach.NodeName) if uncertainError != nil { klog.Errorf("DetachVolume.MarkVolumeAsUncertain failed to add the volume %q to actual state after detach error: %s", volumeToDetach.VolumeName, uncertainError) } </ins> eventErr, detailedErr := volumeToDetach.GenerateError("DetachVolume.Detach failed", err) return volumetypes.NewOperationContext(eventErr, detailedErr, migrated) }

KCM starts, all VolumeAttachments that don't have a Pod (that's good), but also (that's probably bad). When the newly started KCM re-tries to detach the volume and gets an error again, it marks the volume as fully attached (same as the previous point) and adds it to . Now the volume is both mounted + attached in ASW and A/D controller actually stops here and waits forever for kubelet to unmount the volume. <detailsToday's k/k master, </details<detailsNone </details<detailsN/A, it's KCM issue. </details<details, but seen in a downstream Kubernetes distro too. </details<details</details<details</details
/sig storage
/assign
/triage accepted
I have the same problem: Seems restart of Kubelet solves this problem. Kubernetes v1.25.13


Question: // ResourceConfiguration stores per resource configuration. type ResourceConfiguration struct { <del> // resources is a list of kubernetes resources which have to be encrypted. </del> <ins> // resources is a list of kubernetes resources which have to be encrypted. The resource names are derived from `resource` or `resource.group` of the group/version/resource. // eg: pandas.awesome.bears.example is a custom resource with 'group': awesome.bears.example, 'resource': pandas) </ins> Resources []string `json:"resources"` // providers is a list of transformers to be used for reading and writing the resources to disk. // eg: aesgcm, aescbc, secretbox, identity.

Update the godoc on the encryption config API on how to specify group/resources to be encrypted ref: godoc has the details na No response_ na na <details</details<details</details<details</details<details</details
/sig auth /triage accepted
/assign Cool-Coder174
Thanks for volunteering for this LMK if you need any help with it.
Just for clarification, I just need to add documentation on how to configure encryption on groups/resources?
You can update godoc to mention that custom resources can now be to the list of to be encrypted as well and how to add the of the resource to the list. e.g. from this test
/assign /unassign Release v1.26 deadlines are here so we need to get this completed ASAP.
/milestone v1.26
I apologize for the slow actions. I know time and contributions are valuable. May I request a walk through. This is my first time ever contributing and I really want to learn. I have been reading some of the code base on my spare time.
this feature is on the harder side of things to get started with. Reading the blog post and the associated documentation is probably the best starting point:
& what is the status on this issue? 1.26 is coming up! cc
/milestone v1.27


Question: return err } <del> // Record some output. outputSetLock.Lock() defer outputSetLock.Unlock() outputSet.Insert(key) </del> <ins> // Report this deletion. deletionCounter <- key </ins> } return nil },

From shippable, I noticed there are several failed cases, for example, E0413 17:34:38. ] watch of *api.Pod ended with: Stop requested --- FAIL: Example (0.61s) got: want: a-hello b-controller c-framework FAIL coverage: 85.9% of statements FAIL 7.652s cc/
another example:
ah-- this is a different thing. I will fix this, too.
I'm seeing this from an unrelated change
Yeah I think that's the one I expected to see.


Question: unversioned.GroupResource{Group: "", Resource: "endpoints"}:              "services/endpoints", unversioned.GroupResource{Group: "", Resource: "nodes"}:                  "minions", unversioned.GroupResource{Group: "", Resource: "services"}:               "services/specs", <ins> unversioned.GroupResource{Group: "extensions", Resource: "ingresses"}:    "ingress", </ins> } func (s *DefaultStorageFactory) ResourcePrefix(groupResource unversioned.GroupResource) string {

On a 1.3.6 master: But on a 1.4 master: Obviously this will cause upgrade issues. ideas? If not I will debug tomorrow.
defaults to using the resource name as the etcd prefix, but ingresses was not using the resource name as its prefix. fix in
reopening until cherry-pick merges
Yikes, good catch. This would have been a disaster. , Jordan Liggitt wrote:
merged to release-1.4


Question: annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: <del> affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: k8s-app operator: In values: ["kube-dns"] topologyKey: kubernetes.io/hostname </del> tolerations: - key: "CriticalAddonsOnly" operator: "Exists"

Our 5k-node cluster jobs recently started failing even before the e2e tests start. The reason is the step which checks that all kube-system pods are running is timing out: cc
From the scheduler logs I'm seeing that the pods are being scheduled quite slowly, getting worse with time: 2-3 pods/min looks quite bad. cc
Also finding such traces: Seems like most of the time is being spent in priority calculation.
From the pod spec, I'm seeing soft inter-pod anti-affinity at node-level topology: I'm almost sure this is the one causing the mischief. We've been having performance issues with anti-affinity traditionally :p
Ok... PR is the cause. It introduced anti-affinity to kube-dns. I think we should revert this change given known issues with anti-affinity. - Opinions?
FYI - it is taking 80m until the last pod is scheduled when I'm running 5k-node cluster locally.
That's quite unfortunate. It looks like we should revert the change until the anti-affinity change is scalable enough to be used more broadly, unless the scheduler folks have a target fix for this use case.
cc
We are very well aware of the performance issues in our affinity/anti-affinity functions, but I am not aware of a concrete solution at this point. This is partly due to the flexible design of affinity/anti-affinity concept. My humble opinion at this point is to drop some of the flexibility as they are rarely used, in order to streamline the algorithm and improve performance. Anyway, I think the proper solution to this problem needs more time than your requirements to address the issue in hand. So, you may want to change your configuration and drop the anti-affinity rule.
would it make sense to add a note to the anti-affinity user documentation to hint at performance problems at large scale, until this is fixed?
Re-opening this issue as this problem has struck again.


Question: annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: <del> affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: k8s-app operator: In values: ["kube-dns"] topologyKey: kubernetes.io/hostname </del> tolerations: - key: "CriticalAddonsOnly" operator: "Exists"

The following runs of our 5k-node scalability jobs failed to bring up the cluster on time: https://k8s- https://k8s- https://k8s- https://k8s- And the reason is kube-dns pods pending to be scheduled for a long time:
These failures started happening about 4 days ago and I'm pretty sure it's this one: Add self anti-affinity to kube-dns pods Sadly, we'll have to revert it again for now :( cc - Seems like the improvements to anti-affinity performance made under weren't enough?
We need to find other improvements for the predicate. If we cannot find major improvement opportunities, we may have to change its design to be more scalable at the cost of the reducing its flexibility.
- my personal opinion is that we need to change the design anyway. Having it work both ways (i.e. existing pods everywhere impacting placing of our pod) will always be painful (especially for things like cluster autoscaler)
I agree. The symetrical application of anti-affinity is problematic. I also think we need to change it.
Great. - do you agree with that too?
, I'm wondering after some affinity/anti-affinity optimization in scheduler, do you think if it's fine to test coreDNS with anti-affinity policy in large cluster with 5000 nodes?
We probably should try to see where we are - but this requires testing before enabling that.
I agree with - Could we first try checking with a benchmark integration test () as a proxy for real cluster test?
I'm wondering how many real nodes are required for testing?
Since the integration tests fake out the nodes, I think it's safe to crank it up to 5000 (that's the current largest integration test benchmark size) I'm also hoping to refactor our benchmarks soon, so let me know if you have any thoughts about them.


Question: tags = ["automanaged"], visibility = ["//visibility:public"], ) <ins> go_test( name = "go_default_test", srcs = ["scheme_test.go"], importpath = "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/scheme", library = ":go_default_library", deps = [ "//pkg/kubelet/apis/kubeletconfig/fuzzer:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/testing/roundtrip:go_default_library", ], ) </ins>

Solution: I will add one.
/cc
xref
Thanks for sending a PR so quickly!
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code slush, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: If this issue is not resolved or labeled as by Wed, Nov 22 it will be moved out of the v1.9 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: ) { if nodeInfo == nil || nodeInfo.Node() == nil { // This may happen during tests. <ins> metrics.EquivalenceCacheWrites.WithLabelValues("discarded_bad_node").Inc() </ins> return } // Skip update if NodeInfo is stale. if !cache.IsUpToDate(nodeInfo) { <ins> metrics.EquivalenceCacheWrites.WithLabelValues("discarded_stale").Inc() </ins> return }

/kind feature The equivalence cache feature is planned to move to beta in 1.11, and it would be useful to have some metrics attached to it. The equivalence cache should improve scheduling throughput, so we want to be able to correlate e.g. throughput with eCache hit rate. /sig scheduling /sig instrumentation
/assign misterikkit
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: output_message=$(! kubectl get pods a 2>&1 "${kube_flags[@]:?}") kube::test::if_has_string "${output_message}" 'pods "a" not found' <ins> kubectl delete pods a kubectl delete pods b # apply a kubectl apply --namespace nsb -l prune-group=true -f hack/testdata/prune/a.yaml "${kube_flags[@]:?}" # apply b with namespace kubectl apply --namespace nsb --prune -l prune-group=true -f hack/testdata/prune/b.yaml "${kube_flags[@]:?}" # check right pod exists kube::test::get_object_assert 'pods b' "{{${id_field:?}}}" 'b' # check wrong pod doesn't exist output_message=$(! kubectl get pods a 2>&1 "${kube_flags[@]:?}") kube::test::if_has_string "${output_message}" 'pods "a" not found' </ins> # cleanup kubectl delete pods b

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Actually, the parameter depends to all namespaces in the cluster. Filtering is only possible with labels. When running the should only depend on the objects defined in . Why is this needed: A typical use-case is having one repository and namespace per project. When working with declarative objects it should be possible to remove objects and get them pruned like having them described in the respository. This would also help when renaming objects within the repository - which actually needs a manual on the host for the old objects. This even could be a better solution for the problem described in
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
cc
/assign
GitHub didn't allow me to assign the following users: Liujingfang1. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have been aware of the issues in prune and we want to improve it. Recently we opened a for improving pruning in . With the proposed solution, the problem from current will be fixed.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: output_message=$(! kubectl get pods a 2>&1 "${kube_flags[@]:?}") kube::test::if_has_string "${output_message}" 'pods "a" not found' <ins> kubectl delete pods a kubectl delete pods b # apply a kubectl apply --namespace nsb -l prune-group=true -f hack/testdata/prune/a.yaml "${kube_flags[@]:?}" # apply b with namespace kubectl apply --namespace nsb --prune -l prune-group=true -f hack/testdata/prune/b.yaml "${kube_flags[@]:?}" # check right pod exists kube::test::get_object_assert 'pods b' "{{${id_field:?}}}" 'b' # check wrong pod doesn't exist output_message=$(! kubectl get pods a 2>&1 "${kube_flags[@]:?}") kube::test::if_has_string "${output_message}" 'pods "a" not found' </ins> # cleanup kubectl delete pods b

/lifecycle stale
/remove-lifecycle stale
I just deleted a bunch of things in another team's namespace (:blush:). I'd would love to have this feature. I had been using dry-run in another context, but then forgot to check it again after switching contexts.
Is there any workaround for this? I want to prune resources within a single namespace only.
We're still working on improving prune, the current mechanism is broken enough that we don't recommend people us it.
Also look like it's been fixed recently.
Can someone confirm if this is implemented? I tried using this feature with kubectl 1.18.2, which has , yet as with previous versions kubectl it returned an error about pruning a nonNamespaced object of kind Namespace.
I assume the problem is that "Namespace" is a non-namespaced resource, and therefore is not filtered by namespace, so gets deleted by "prune". As a workaround it's possible to manually pass a (whitelist of resources allowed to be deleted), and not include "Namespace" in it: The list of resources is taken from .
Would wrapping this loop with achieve the same result?


Question: // clusters: //   cow-cluster: //     LocationOfOrigin: "" <del> //     certificate-authority-data: REDACTED </del> <ins> //     certificate-authority-data: DATA+OMITTED </ins> //     server: http://cow.org:8080 // contexts: //   federal-context:

Is this a BUG REPORT or FEATURE REQUEST?: What happened: shows: The use of "REDACTED" for a CA here is misleading, since it's a public certificate that is not a secret. What you expected to happen: I expected to see the full CA in pem format (like when you cat ) or something like "SUPPRESSED" or "TRUNCATED" How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Nope Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g. from /etc/os-release):Kernel (e.g. ):
(for thoughts)
Reiterating the mentions to trigger a notification: <detailsIn response to : (for thoughts) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
use you can see the complete certificate authority data
Hey Thank you - I have tracking that . The point of this issue is that at best it's a bad UX; at worst it's a miseducation on pki. "REDACTED" isn't the right word here
If there are multiple clusters displaying certificates, the configuration will be too long, so this switch is turned off by default. But we can add an example to help use. thanks
Hey I don't personally view truncation the responsibility of the CLI... users can pipe things to a pager if they really care. That being said, I'm not concerned about the length. The issue is purely about the word choice of "REDACTED". The term means: The use of the term "REDACTED" in the output implies that a CA public key is remove for security when (as you've pointed out) it is removed for truncation. Truncation and redaction are fundamentally different things, and k8s is misusing the word redacted for this use case.
Not quite a bug, but I think you're right a better term could be used there as it is a public key, not a private key. /remove-kind bug
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: // clusters: //   cow-cluster: //     LocationOfOrigin: "" <del> //     certificate-authority-data: REDACTED </del> <ins> //     certificate-authority-data: DATA+OMITTED </ins> //     server: http://cow.org:8080 // contexts: //   federal-context:

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale I'm not sure what I need to do here. This was marked as a feature. Do you need something more from me?
We are waiting for someone to fix this.
/remove-good-first-issue I don't think this fits our definition of a :
So I have to keep replying to the comment bot every 30 days so it doesn't get auto-closed? That feels... odd
thanks for the education :) unfortunately other issues are higher priority and no one has stepped up to fix this. The autocloser is a form of bug bankruptcy that keeps our issue load in check. The contribex special interest group is definitely open to feedback on the bot.
The biggest hurdle in this is that Kubernetes is using some base64 trickery to display the string (basically setting the certificate data to the decoded bytes from ). We can change this to or but not (it has to be a valid base64 encoded string). I tried exploring multiple options for the text, but I couldn't find a suitable one. Given that we already print the user tokens and other credentials this is a bit weird. Should we redact those too? Is it okay if we print the CA certificate instead of truncating it?
I can't help but laugh at: I'm not wed to any particular name. I also have no problem displaying the entire certificate. The crux of the issue is that a CA public key is not secret or sensitive. It's a public key
I agree. The issue is with the constraints imposed by how this is implemented currently. I tried to come up with a suitable string (given the constraint) and couldn't find anything acceptable... The only suitable option that worked for me is . I personally prefer to print the CA certificate since we are already printing user tokens and client certs that are actual secrets.
I agree. Who do we need to get in agreement that printing the full CA is the best path forward?
the data is not omitted for security purposes, but to make the output more readable. see REDACTED is misleading... it should probably be "elided" or "..."
you can run to view full content
that's the crux of this issue - REDACTED is misleading.


Question: // clusters: //   cow-cluster: //     LocationOfOrigin: "" <del> //     certificate-authority-data: REDACTED </del> <ins> //     certificate-authority-data: DATA+OMITTED </ins> //     server: http://cow.org:8080 // contexts: //   federal-context:

However, pointed out that other "long" fields are still completely printed in plain text such as client certs, so I think we need to revisit the "readability" argument. What do you think?
they are not... client cert/key are elided in the same way for the same reason: something like "++data+omitted++" could be better If the intent of config view without --raw specified is to be readable (which it currently is), printing many extremely long fields isn't great
Hmm what about: I'm pretty sure that would give us a valid b64 string (8+12=20%4=0) and it would subtly hint that the data isn't a secret.
Only valid characters are . And adding a prefix from the certificate only complicates things. is a suitable usable option. I'll submit a PR to continue the discussion there.


Question: ginkgo.By("Creating a ResourceQuota") quotaName := "test-quota" resourceQuota := newTestResourceQuota(quotaName) <ins> resourceQuota.Spec.Hard[v1.ResourceConfigMaps] = resource.MustParse(hardConfigMaps) </ins> _, err = createResourceQuota(f.ClientSet, f.Namespace.Name, resourceQuota) framework.ExpectNoError(err)

I ran conformance tests on a cluster that had a controller enabled that syncs a configmap across all clusters. It failed: Appears to be due to a 2 configmap limit: These clusters should pass conformance. This is a very common pattern (even Kubernetes itself does it!). I didn't run it myself, but apparently just install Istio and run the conformance suite. See for details Kubernetes version (use ): v1.20.2
/assign
my guess is the CA configmap created by default in 1.20 exacerbated this issue. I agree the conformance test should make fewer assumptions about the cluster it is running in and should accomodate default configmaps. left a comment on the PR in flight for this. I'd like to see the change picked to 1.20 as well.
/triage accepted /priority important-soon


Question: } }) <del> ginkgo.It("should respect internalTrafficPolicy=Local Pod to Pod (hostNetwork: true) [Feature:ServiceInternalTrafficPolicy]", func() { </del> <ins> ginkgo.It("should respect internalTrafficPolicy=Local Pod and Node, to Pod (hostNetwork: true) [Feature:ServiceInternalTrafficPolicy]", func() { </ins> // windows kube-proxy does not support this feature yet // TODO: remove this skip when windows-based proxies implement internalTrafficPolicy e2eskipper.SkipIfNodeOSDistroIs("windows")

Failed here /retest Originally posted by in This fails because the pod tries to bind port 80, that is a very demanded port, but there are two tests doing that, that increase the chance to collide
/sig network
Close when merges
/triage accepted


Question: } /* <del> Release : v1.12 Testname: Security Context: readOnlyRootFilesystem=true. Description: when a container has configured readOnlyRootFilesystem to true, write operations are not allowed. This test is marked LinuxOnly since Windows does not support creating containers with read-only access. </del> <ins> Release : v1.15 Testname: Security Context, readOnlyRootFilesystem=true. Description: Container is configured to run with readOnlyRootFilesystem to true which will force containers to run with a read only root file system. Write operation MUST NOT be allowed and Pod MUST be in Failed state. At this moment we are not considering this test for Conformance due to use of SecurityContext. [LinuxOnly]: This test is marked as LinuxOnly since Windows does not support creating containers with read-only access. </ins> */ It("should run the container with readonly rootfs when readOnlyRootFilesystem=true [LinuxOnly] [NodeConformance]", func() { createAndWaitUserPod(true) }) /* <del> Release : v1.12 Testname: Security Context: readOnlyRootFilesystem=false. Description: when a container has configured readOnlyRootFilesystem to false, write operations are allowed. </del> <ins> Release : v1.15 Testname: Security Context, readOnlyRootFilesystem=false. Description: Container is configured to run with readOnlyRootFilesystem to false. Write operation MUST be allowed and Pod MUST be in Succeeded state. </ins> */ <del> It("should run the container with writable rootfs when readOnlyRootFilesystem=false [NodeConformance]", func() { </del> <ins> framework.ConformanceIt("should run the container with writable rootfs when readOnlyRootFilesystem=false [NodeConformance]", func() { </ins> createAndWaitUserPod(false) }) })

/kind cleanup /area conformance /area test /sig node /sig testing /sig architecture /assign /assign We have a number of tests that only have the tag that may make sense as tests, we should identify whether these tests cases meet (or could meet) the . We should open PR's to promote these if so. I would advise starting with the test/common cases
GitHub didn't allow me to assign the following users: mgdevstack, mkurylec. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /kind cleanup /area conformance /area test /sig node /sig testing /sig architecture /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The tests in test/e2enode don't actually get run as part of Conformance as near as I can tell. If we want to run them as part of conformance, they need to be migrated into test/e2e/common. Some of the tests in e2enode may not be able to be ported over because they assume a level access we don't have in a conformance environment.
regarding this testcase: [sig-storage] HostPath should support r/w [NodeConformance]: It uses hostPath and privileged=true, to read/write from /tmp folder. Which would you suggest is the best orientation to overcome the privilege mode setting and allow its promotion?
/assign
Regarding , the suite runs the test only if the kubelet configuration was required to have the QOS (cgroups-per-qos) enabled. This configuration for the kubelet is selected at the time the cluster is set up. Looking for where this might be configured found: where is set to true. For other settings it is also configured to true. where is set to true. BUT also found where it is mentioned that the flag is no longer available. Questions: would it be ok to promote this test to be conformance, even it can be skipped based on configuration of the cluster? And not supported in the containerized kubelet?
I would consider cgroups to be an implementation detail that Conformance shouldn't be verifying
It uses hostPath and privileged=true, to read/write from /tmp folder.


Question: } /* <del> Release : v1.12 Testname: Security Context: readOnlyRootFilesystem=true. Description: when a container has configured readOnlyRootFilesystem to true, write operations are not allowed. This test is marked LinuxOnly since Windows does not support creating containers with read-only access. </del> <ins> Release : v1.15 Testname: Security Context, readOnlyRootFilesystem=true. Description: Container is configured to run with readOnlyRootFilesystem to true which will force containers to run with a read only root file system. Write operation MUST NOT be allowed and Pod MUST be in Failed state. At this moment we are not considering this test for Conformance due to use of SecurityContext. [LinuxOnly]: This test is marked as LinuxOnly since Windows does not support creating containers with read-only access. </ins> */ It("should run the container with readonly rootfs when readOnlyRootFilesystem=true [LinuxOnly] [NodeConformance]", func() { createAndWaitUserPod(true) }) /* <del> Release : v1.12 Testname: Security Context: readOnlyRootFilesystem=false. Description: when a container has configured readOnlyRootFilesystem to false, write operations are allowed. </del> <ins> Release : v1.15 Testname: Security Context, readOnlyRootFilesystem=false. Description: Container is configured to run with readOnlyRootFilesystem to false. Write operation MUST be allowed and Pod MUST be in Succeeded state. </ins> */ <del> It("should run the container with writable rootfs when readOnlyRootFilesystem=false [NodeConformance]", func() { </del> <ins> framework.ConformanceIt("should run the container with writable rootfs when readOnlyRootFilesystem=false [NodeConformance]", func() { </ins> createAndWaitUserPod(false) }) })

Which would you suggest is the best orientation to overcome the privilege mode setting and allow its promotion? Michelle Au If my understanding is correct, on some platforms, writing to /tmp requires privileged. I don't see a way to to remove that requirement.
If my understanding is correct, on some platforms, writing to /tmp requires privileged. I don't see a way to to remove that requirement. Yeah writing to host doesn't seem like a thing we can expect of conformance tests. I wonder if it makes sense to tag this test with something to denote the requirement, ie: I see some tests in test/e2e_node use
The hostpath tests in already have a comment explaining this. Maybe the comment should be updated to be more clear that that is the reason why we can't add it to conformance.
That's /tmp on the host or in the container filesystem?
/tmp on the host
ping what else remains to be done before we close this out? I am guessing that we want to close , is there anything beyond that?
Once tests under (issue- ) and are promoted we are good to close this issue. Short update for reference- None of [NodeConformance] tests from "test/e2e_node" seems eligible for [Conformance] promotion at this moment due to node level dependencies. Remaining [NodeConformance] tests from "test/e2e" has following status - Waiting for to be closed and then below tests would be promoted to Conformance Needs to be Promoted Can not be promoted to Conformance as //Slow by design Can not be promoted due to Node level dependency
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen
/assign


Question: } func (zones Zones) List() ([]dnsprovider.Zone, error) { <ins> var zoneList []dnsprovider.Zone </ins> response, err := zones.impl.List(zones.project()).Do() if err != nil { <ins> if apiErr, ok := err.(*googleapi.Error); ok && apiErr.Code == 404 { return zoneList, nil } </ins> return nil, err } managedZones := response.ManagedZones() <del> zoneList := make([]dnsprovider.Zone, len(managedZones)) </del> <ins> zoneList = make([]dnsprovider.Zone, len(managedZones)) </ins> for i, zone := range managedZones { zoneList[i] = &Zone{zone, &zones} }

/kind bug What happened: walteraa [6:55 AM] Hi, The controller manager of the federation using Google cloud DNS is crashing, the log showed is that: the hyperkube command is that: I have created this in the Google DNS Someone knows why the DNS zone query is getting the 404 error?? :disappointed: What you expected to happen: It should have created the missing domain. How to reproduce it (as minimally and precisely as possible): As per above. Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.4 Cloud provider or hardware configuration**: OpenStack (irrelevant) OS (e.g. from /etc/os-release): irrelevant Kernel (e.g. ): irrelevant Install tools: irrelevant Others: irrelevant
The bug is related to this: The above code assumes that the provider code returns no error but an empty list if the domain does not exist, but instead the provider returns a 404. Need to fix the provider and/or the above code.
/sig federation
I am unlikely to have time to work on this soon, so if anyone else wants to, go right ahead.
/assign
An additional information: I have created a which is based on the credential JSON file created through instruction. After that, I have configured the Federation controller manager to mount this secret as a volume in the container and set the environment variable to the JSON file inside this volume. Before doing that, I was getting an authentication error so the google DNS provider couldn't be instantiated, and after doing the instructions that I mentioned before, this authentication error was solved but now I'm getting 404 error when Google DNS provider tries access the specified managed zone. The file that I have generated follow this structure: