The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the apiserver repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area apiserver /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the apimachinery repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the apiextensions-apiserver repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


This is the script that runs to capture event output during e2e runs: It's failing after the recent pflag merge:
cc
Or something. I might be completely off. But something broke it recently enough, and I know like his event log.
That command is actually invalid. is a flag on the command, not the root. Fix in


kubectl log stops working for me all the sudden: A little bit digging into the issue, I found the problem is only limited to /v1beta3: Query /v1beta1 Query through v1beta3
cc/
Does replacing "minions" by "nodes" work?
Cool, looks like it should be a simple kubectl fix, I will do it shortly.


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce ci-kubernetes-e2e-gci-gce Which test(s) are flaking: [sig-apps] CronJob should delete successful/failed finished jobs with limit of one job Testgrid link: Reason for failure: Anything else we need to know: this is the top flake for pull-kubernetes-e2e-gce: triage link: specific failures /sig apps /priority important-soon
this is the second flakiest test (failing on ~15% of PR runs)
I'll look into it. /assign
if there's not an obvious immediate fix, I would recommend: the test as to skip the test in merge-blocking job runs a second PR that reverts the change and adds debugging commits and do on that PR to flush out issues this issue open and in the current milestone to track resolution of the test before release (either by fixing the test, finding/fixing an underlying bug, or proving we have test coverage of the function via another method and deleting the test)
marked as in , will leave this issue open to track root causing, fixing, restoring the test before release
To follow up here, this was confirmed as root-caused, fixed and restored by


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After starting controller manager, get the following panic: /---------------------------------- I0712 08:54:30. 1 ] Deleting route 10.245.1.0/24 10.245.1.0/24 I0712 08:54:30. 1 ] Creating route for node ab-6l35lzekrg-0-7igjn3vr43tw-kube-minion-abhezrwnqucq 10.245.1.0/24 with hint -655c-11e7-9a48-, throttled 1.602µs I0712 08:54:30. 1 ] Creating route for node ab-hfvphu5tyl-0-623ksgxczeoh-kube-master-oevhfzd3xo7s 10.245.0.0/24 with hint -655c-11e7-9a48-, throttled 651ns I0712 08:54:31. 1 ] Created route for node ab-6l35lzekrg-0-7igjn3vr43tw-kube-minion-abhezrwnqucq 10.245.1.0/24 with hint -655c-11e7-9a48- after 349.278413ms I0712 08:54:32.


1 ] Created route for node ab-hfvphu5tyl-0-623ksgxczeoh-kube-master-oevhfzd3xo7s 10.245.0.0/24 with hint -655c-11e7-9a48- after 1.583805322s panic: runtime error: index out of range goroutine 1250 [running]: panic(0x3a3d4e0, 0xc42001a080) +0x1a1 (Routes).DeleteRoute(0xc4212e15a0, 0x40ff60e, 0xa, 0xc421bf22a0, 0x0, 0x0) +0x7d5 (RouteController).reconcile.func2(0xc4210bed70, 0xc421bf05e0, 0xc421bf22a0, 0xed0f7dac6, 0x3264f461, 0x6f143e0) +0x15a created by (RouteController).reconcile +0x6e6 /---------------------------------- What you expected to happen: Successed to delete route. How to reproduce it (as minimally and precisely as possible): verytime Anything else we need to know?: Environment: Kubernetes version (use ): v1.6.4 Cloud provider or hardware configuration*: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: cc /sig-openstack
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
cc
Reiterating the mentions to trigger a notification: <detailsIn response to : cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


HPA tests are currently failing in gke-large cluster suite: I did some debugging and the problem seems to be related to pod-nanny. Looking into logs from the last run: was first scheduled on the large node dedicated to it pod-nany updated heapster deployment then heapster was scheduled on the normal node (when it actually shouldn't fit) that point it was never updated by the pod nanny. that point, heapster started crashlooping (OOMing) [which in fact was expected in too small machine] So it's pretty clear that that pod-nanny changed the amount of resources to something smaller, which in fact it shouldn't.


What happened: The rbd plugin use "rbd info" error: The messages before and after json result is warning string and no effect to rbd info, but it is effect to parse the output. So the kubernetes will failed: What you expected to happen: Kubernete no failed and errors How to reproduce it (as minimally and precisely as possible): Remove the config file
/sig storage
We are blocked by this issue, can't resize any PVCs
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
That is another situation that cluster configuation is empty, aslo will return lots of warnning information: Also will trigger expand volume failed. But should we add -c=/dev/null in rbd info command?
Folks, the pull request that would have fixed this just got closed as it was rotting away. This is an actual problem in production, can we please fix this?
+1 we are still being plagued by this issue...
+1 {"name":"kubernetes-dynamic-pvc--147e-422c-9e67-","size":,"objects":2560,"order":22,"objectsize":,"blocknameprefix":"rbddata.","format":2,"features":["layering"],"flags":[]}, invalid character '-' after top-level value
I solved by exceuting this command
Wow...I was not expecting this to work ... but it did ;) Any insights on why is it working with just the presence of the file? This is very strange... Appreciate any comments...


What happened: Some of the Azure CSI translation checks in and needs to perform NPE checks for pointer based fields as pointed out in and What you expected to happen: NPE checks should be there How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.15 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
/priority critical-urgent


For example, kubernetes-ro service appears with empty selector (this service was created by the hack/local cluster) - is that ok? What meaning such service has if it has no entity to fwd traffic to?
kubernetes and kubernetes-ro are currently "virtual" - they are not backed by running pods, but by the master itself. , abonas wrote:
if that's internal impl, why is it exposed to user? let's say a client pulls all entities and builds a UI around it. then a service with no selector arrives, and it's internal to kube - how the client should know not to present it/touch it? (similar to the "net" container in pods)
It's exposed exactly FOR users - so they can find the apiserver. It has a corresponding Endpoints object, it gets a portal IP and a kube-proxy rule, and it gets service env vars and a DNS name. The fact that it is virtual is an implementation detail. That it exists is all users should need to know. , abonas wrote:
And FWIW, the master components may one day actually be pods... , Tim Hockin wrote:
thanks for the explanation. so any general rule how to distinguish kube pods/services from user defined?
Do we need to distinguish them? We could add labels, I guess, if it is really important to distinguish them.
sometimes users would want to see only the things they defined and not implementation details of the system.
One could argue this is why the users should be using labels. I'm not against defining labels for "special" stuff, but it's new territory, so we should enter with caution. , abonas wrote:
I think the right answer might be to put them in a special "cluster" or "system" namespace. , Tim Hockin wrote:
See If we put them in a "kubernetes" ns, the DNS name would be :) I'm not against segmenting them into a different namespace. We just need to decide on what that namespace is called, plumb through that configuration, decide on compatibility for a short while or not, and decide if that namespace is "special" or just "different" - i.e. should the skydns, heapster, etc pods go into this namespace? , Brendan Burns wrote:
As said, the services are there for applications that need to contact the apiserver.


Additionally, nil selectors can be used by users to import external services. Adding labels to the kubernetes services won't help. Users need to manage their objects using labels. /cc
Yeah, lots of consumers will expose "special" services. IN fact, in many clusters the "special" services will outweigh the unspecial ones. At this point, the namespace they are in is completely arbitrary, and I think because it is a name, it should be free for the end user to choose where they are surfaced.
Per the proposal here: , I think the namespace should continue to hold these services, and they require no special labels. I sense comment is made without thinking through what Kubernetes operators do to bootstrap a cluster. I think it's reasonable that namespace holds everything you need to run, and operators may or may not choose to segment their data in other namespaces. If they want to not see this information, its an exact reason for why they should use their own namespace. I don't think we need anything new here.
- I'm not clear on your meaning. Did you mean that we should make the "cluster admin" namespace a config param that everyone knows about and can be different cluster-to-cluster? Or did you mean that default is as arbitrary a name as any other? I'd be fine moving things out of default into "kube" or "kubernetes" or "cluster" or something, but I am less convinced that a flag is a good thing. , Clayton Coleman wrote:
- wherever we stick this stuff, it should be the namespace that is initially bootstrapped, which today is . I continue to believe that is as good a name as any other. I do not see a need to do more here.
I think as soon as you bring naming into the picture, making decisions based on names gets very tricky (what if I want my logging service to be everywhere?). If there is a magic namespace that exposes things to the rest of the cluster, we should ask whether there's only one magic namespace, or multiple? Are services supposed to cross namespaces? It's great that pods can find the master... but that's not the only thing they'll need to find in each namespace. And baking in special names (or customizable names) into your app code is fragile, because then your code only works inside Kubernetes.


I'm just trying to pump the brakes on special services prior to us designing them... :) ----- Original Message -----
oh, to be clear, I don't want "special" services - I am merely asking if moving non-user-defined objects (where that is defined very loosely) out of "default" is worthwhile. I tend to agree with Derek that it's not worth energy. , Clayton Coleman wrote:
agree with you that lots of consumers will expose "special" services We should not use the (namespace + k8s-service + skydns) mechanism for publishing the location of those special services. We should decouple the choice of namespace to run the pods of the service in from the well-known dns address of special services. Otherwise, it will be hard to move the pods of the service to another namespace, which people will definitely want to do as they grow. Instead, we need a place to publish well-known services which is not tied to the namespace of the pods of that service.
you open a whole new topic. I've noodled with the idea of a Name resource type. We would generate a Name for each Service as "service.namespace", but users could also add their own Name resources. , Eric Tune wrote:
I'd like to understand better your comment regarding nil selectors used to export external services - can you elaborate? give an example? my understanding is that service redirects traffic to pods based on labels. with nil selector service does nothing, so why is it a service in the first place? I'm pro decoupling special services/internal implementation services from user defined entities. whether by separate namespace, or by saying that the default namespace is reserved, or by any other solution. because otherwise internal implementation and user defined content are mixed and that's no good for a proper management. just tagging you so you could see this issue and the related discussions.
Services abstract backends - they don't have to abstract pods. Some examples: You want to have an external database cluster in production, but in test you use your own databases. You would have a service for both and use the environment variables, but the production service has a nil selector with its endpoints set to the database cluster. You want to have an empty service because you're reserving a name (that isn't accessible yet) but will be in the future You want to point your service to a service in another namespace or on another cluster.


You set the endpoints for your service to the publicIPs of the other service. ----- Original Message -----
, thanks. regarding the last point - I was under impression (based on some other discussion) that there would not be cross namespace interaction? that entities can can access only entities in the same namespace?
Cross-namespace is currently allowed and I think has to be allowed, though perhaps (in the fullness of time) that can be policy based rather than wide open. , abonas wrote:
please see this correspondence. my understanding is that there is isolation
Responded there. I think there's some less-than-crystal-clear words in play. , abonas wrote:
+1 to Eric's comment about ensuring there is a consistent manner for system services like kubernetes-ro, if that's not fixes for all clusters it's going to be a nightmare to write reusable containers for multiple different clusters. I don't care if we do that by using a constant namespace, or by giving system services two different names, but there needs to be at least one name that is constant. Regarding default vs. cluster namespace, I think the issue is one of clutter as a user is starting to understand k8s, you turn up a new cluster, launch a pod, or service, and it's hard to find your one object on the sea of system created pods and services. It's visual clutter that confuses people as they onboard. On Jan 23, 2015 3:01 PM, "Tim Hockin" wrote:
- can you drop me a line (my github name at google) - I want to ask you something, and you have no contact info :) , abonas wrote:
sorry for naive question but why would I care as an user that wants to run its existing workload on Kubernetes about kubernetes-ro ? If I write an application that uses kubernetes-ro then I'm locked into Kubernetes. I understand the need for kubernetes-ro being accessible by cluster level services such as monitoring, logging. I also can imagine that a power user might want to start its own cluster services that use kubernetes-ro. I don't see clearly why would majority of users care about kubernetes-ro being visible from their container.


I see a reason for the cluster services being visible and accessible for Kubernetes developer, but from an user perspective such services only make the system more complex: why do I see these services (pods,rc) ? are they related to my application ? should I care about them (especially when I use hosted Kubernetes) ? why does my container get all these environment variables for cluster services ?
I will update file to explain why one would have service without a selector.
Right now, every service goes everywhere for simplicity. Name spacing and filtering probably make sense in the long run, but it's not a high priority for v1.0 Brendan On Mar 4, 2015 5:02 AM, "Rafał Sokołowski" wrote:


In volume management code, VolumeSpec is firstly needed to find the volume plugin. After the volume plugin is found, VolumeSpec is again used to attach or mount or check volume is still attached or not. This information is reconstructed during controller failure. The assumptions used here are If pod still exists, VolumeSpec should be reconstructed when controller is restarted. However, as discover from this bug, it might not be true if PVC is deleted in the meantime because pod only has PVC information and requires to use PVC to get PV information. By fixing Issue , we could fix this problem.If pod is already deleted, VolumeSpec is not really needed for detaching. But before detaching finishes, syncState loop might check this volume's attachement status and will use VolumeSpec during checking. One safe guard we can put here is to fail FindAttachablePluginBySpec if spec is nil. This function is used by multiple places such as attachdetachcontroller, desiredstateofworld, and actualstateofworld, oeperation_generator etc. Instead of adding nil check many places, I feel FindAttachablePluginBySpec or even better FindPluginBySpec (is called by FindAttachablePluginBySpec and other places). Open an issue for this
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


recently merged, adding nodes as something we can put in bound object refs (yay!) Today while reading noderestriction code for a different reason, I realized we never extended noderestriction to block Kubelets from binding tokens to nodes other than themselves. Today, noderestriction does block Kubelets from binding tokens to Pods on other nodes, so perhaps it should do the same for bound node references. noderestriction to block it, probably. Read the code here: And note that we didn't modify in No response Not yet in any released version of K8s. N/A N/A N/A N/A N/A
/triage accepted
To summarize Slack discussion: TokenRequest can only bind to one object at a time. kubelet (node identity) is already required to bind to Pod by noderestriction. kube-apiserver automatically inserts the node information based on the Pod binding. kubelet does not set it. So this is technically still node bound. We should still add a comment in to avoid future confusion. There may be other reasons to add a check, like future proofing...


Seen in and in No hits on stack strace nothing evident on the git history of the test itself, but it seems , so most probable something in the device manager has changed? /sig node /kind flake
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/cc as this flakes a lot in


This was build from master earlier today.
cc This should be fixed before v1.4
This is critical
This needs to be tracked as a 1.4 blocker
ETA for resolution: This afternoon This is in the process of being cherrypicked


What would you like to be : The Event API is being promoted to v1 in and right now v1beta1 has higher priority than v1 in We should bump this in 1.20. Why is this needed: The new Event API is being promoted.
/sig instrumentation
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Two problems: on node do not have namepace, but kubectl requires namespace. However, if explicitly set namespace to 'default', this will cause an error because we are updating not allowed field (only label update is allowed). is usually created by node controller, update node label with a file is tedious since user needs to create a file with id, resource.capacity, etc. can we make it easier?
It would make more sense to have a special global namespace than no namespace. We should make it easy in kubectl to update a single field or subset of fields, generally, for all objects. What this would do is GET, merge, and PUT.
I had a discussion with on IRC about namespace on node. We'd like to just use "" (empty string) as of now. As we are not supporting any namespace feature on node, it's less confusing if we can hide the concept for now. Thinking a little more, "" is the special global namespace. I'm going to update kubectl to not assume 'default' namespace on every resources. It works fine for pod, service, but not for node, external LB, etc.
There was some discussion about "" being (essentially) an alias to "default": is that dead? , Deyuan Deng wrote:
I've seen two versions of NamespaceALL ("") NamespaceAll is the default argument to specify on a context when you want to list or filter resources across An empty namespace is equivalent to the , but "default" is the canonical representation. Either means , or the definition conflicts - all namespace should include "default" namespace. Also, for things like Node, I'd really like a NamespaceNone.
- I think there is a confusion here because of a poor naming choice. is a constant intended to be used as a filter to LIST/WATCH across multiple namespaces. It is NOT intended to be the value that is set on a resource for persistence. The discussion was to update to which I think is still a good thing to do. I have held off doing that until after we embed the namespace as a path parameter in our URLs instead of a query parameter, which I plan to look at after the holidays. is a constant intended to be used as the namespace context of a request when it's not explicitly stated on a request URL.


Since create operations pull the namespace from the context during persistence, it is also the value intended to be persisted. This is referring to a request URL that does not specify a namespace. It is not referring to the persisted value in the database directly - although that is the value that gets pulled. I think a is a good thing to add AFTER we make the change to specify namespace as a path parameter instead of a query parameter, and we update/rename to something that clarifies that its a wild-card filter whose value is probably best represented as.
ok, sounds good. Thanks. Look forward to see api.NamespaceAll = "*" and api.NamespaceNone :)


<!-- Please only use this template for submitting enhancement requests --What would you like to be : Sometimes reported in fluent-plugin-elasticsearch repository for exhausted Elasticsearch cluster issue due to elasticsearch ruby client sniffering issue. This should be fixed with but k/k fluentd-elasticsearch image doesn't use this. Why is this needed: This change should be also applied in the downstream docker images and using helm charts such as
/sig instrumentation
/sig cloud-provider
/assign cosmo0920
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.


This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: During startup shows the following warning: However is set and refers to a valid X.509 private key file. All service accounts seem to be created despite the warning. How to reproduce it (as minimally and precisely as possible): start controller-manager with Anything else we need to know?: Environment: Kubernetes version (use ): v1.8.0 OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie)
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Thanks for the report. The condition for printing the warning is backwards, but has no effect on behavior. Fixed in
Still present in 1.8.3 , can it be ~backported~ cherry-picked to 1.8 branch please?
See for proposed cherry-pick


Following up on , evaluate if the test for local SSD on GKE clusters is still necessary (it is also tested internally in Google). /assign
/sig storage
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle frozen
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This test should be removed as it's using a GKE feature that, while not deprecated, is discouraged in favor of newer APIs. Those APIs are well-tested internally in google, and I don't see value in additional toil upstream to maintain them.


On a 1.3.6 master: But on a 1.4 master: Obviously this will cause upgrade issues. ideas? If not I will debug tomorrow.
defaults to using the resource name as the etcd prefix, but ingresses was not using the resource name as its prefix. fix in
reopening until cherry-pick merges
Yikes, good catch. This would have been a disaster. , Jordan Liggitt wrote:
merged to release-1.4


Our 5k-node cluster jobs recently started failing even before the e2e tests start. The reason is the step which checks that all kube-system pods are running is timing out: cc
From the scheduler logs I'm seeing that the pods are being scheduled quite slowly, getting worse with time: 2-3 pods/min looks quite bad. cc
Also finding such traces: Seems like most of the time is being spent in priority calculation.
From the pod spec, I'm seeing soft inter-pod anti-affinity at node-level topology: I'm almost sure this is the one causing the mischief. We've been having performance issues with anti-affinity traditionally :p
Ok... PR is the cause. It introduced anti-affinity to kube-dns. I think we should revert this change given known issues with anti-affinity. - Opinions?
FYI - it is taking 80m until the last pod is scheduled when I'm running 5k-node cluster locally.
That's quite unfortunate. It looks like we should revert the change until the anti-affinity change is scalable enough to be used more broadly, unless the scheduler folks have a target fix for this use case.
cc
We are very well aware of the performance issues in our affinity/anti-affinity functions, but I am not aware of a concrete solution at this point. This is partly due to the flexible design of affinity/anti-affinity concept. My humble opinion at this point is to drop some of the flexibility as they are rarely used, in order to streamline the algorithm and improve performance. Anyway, I think the proper solution to this problem needs more time than your requirements to address the issue in hand. So, you may want to change your configuration and drop the anti-affinity rule.
would it make sense to add a note to the anti-affinity user documentation to hint at performance problems at large scale, until this is fixed?
Re-opening this issue as this problem has struck again.


The following runs of our 5k-node scalability jobs failed to bring up the cluster on time: https://k8s- https://k8s- https://k8s- https://k8s- And the reason is kube-dns pods pending to be scheduled for a long time:
These failures started happening about 4 days ago and I'm pretty sure it's this one: Add self anti-affinity to kube-dns pods Sadly, we'll have to revert it again for now :( cc - Seems like the improvements to anti-affinity performance made under weren't enough?
We need to find other improvements for the predicate. If we cannot find major improvement opportunities, we may have to change its design to be more scalable at the cost of the reducing its flexibility.
- my personal opinion is that we need to change the design anyway. Having it work both ways (i.e. existing pods everywhere impacting placing of our pod) will always be painful (especially for things like cluster autoscaler)
I agree. The symetrical application of anti-affinity is problematic. I also think we need to change it.
Great. - do you agree with that too?
, I'm wondering after some affinity/anti-affinity optimization in scheduler, do you think if it's fine to test coreDNS with anti-affinity policy in large cluster with 5000 nodes?
We probably should try to see where we are - but this requires testing before enabling that.
I agree with - Could we first try checking with a benchmark integration test () as a proxy for real cluster test?
I'm wondering how many real nodes are required for testing?
Since the integration tests fake out the nodes, I think it's safe to crank it up to 5000 (that's the current largest integration test benchmark size) I'm also hoping to refactor our benchmarks soon, so let me know if you have any thoughts about them.


! ! Which jobs are failing: Which test(s) are failing: Since when has it been failing: failures started between 3-4PM (EST), on 1/11/2019 the failures appear limited to containerd jobs (here's a triage search for note that only containerd jobs appear as failing): /sig node
the relevant commit range on master appears to be ... the relevent commit range in test-infra appears to be
cc
for test-infra: aws-janitor: clean up ELBs - definitely unrelated, only affects some of the AWS testing cleanup Update kubekins tag for eks jobs - definitely unrelated, affects the docker image in which some EKS testing runs only Update ubuntu image used for e2e node test - I think unrelated, appears to only affect the node e2e tests rest are merge commits
thanks for checking. it's also possible the commit ranges I linked to need to be expanded slightly... that was my first attempt at correlating merge timestamps with the triage board failures... it's possible I got timezone correlation wrong a few hours in one direction or the other.
ah, they probably do then. otherwise I'd say something else must have changed out of band (perhaps there's a pointer to a containerd version to use somewhere or something).
flakes or was something fixed offline (gce / gci / gke)? https://k8s- https://k8s- these turned (mostly) green without k/k and k/test-infra changes.
Huh. Disconcerting, but ok. /close
Closing this issue. <detailsIn response to : Huh. Disconcerting, but ok. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
All test failed because of not being able to pull image from : I guess this is a bug in , and they fixed it now.
nice catch. thanks
should the CSI images be in our regular repo?


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: I have a daemonsets with 250 replicas and maxUnavailable is 20%. When I try to update its image, the ds will scale down 50 old replicas and wait for SatisfiedExpectations, after that it will scale up 50 new replicas. But if a pod delete failed (node is notready, and pod stuck at terminating status), ds controller will never reach the expectation and just stuck here. What you expected to happen: When a pod stuck at terminating, ds controller should start new pods like deployment controller, instead of waiting for the delete batch finish and ready for next poll. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.1-8+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-25T13:04:04Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig apps /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
DaemonSet controller waits for old pods to terminate before creating new ones for best-effort at-most-one pod per node. This is a deliberate design decision.


In most cases, DaemonSets won't tolerate two pods scheduled on the same node at the same time; for example, it's common to communicate with DaemonSet pods via . I'm going to close this ticket as this works as designed. Please feel free to reopen the ticket if you think there are still issues to be resolved.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
why does the DaemonSet controller wait for all pods 50 to be done, before moving to the NEXT 50 ? I completely understand that we want only at most 1 pod per node. But if a pod deletion fails on the node, move on to the next pod as long as the total unavailable pods are less than maxUnavailable. This will allow us to make progress in the face of bad nodes or failed pod deletions due to failing tear down of bad volumes or other reasons.
/assign
thoughts on this one ?
Any progress on this issue? We also encountered this problem. There are thousands of nodes in our prod cluster and nodes often go down, which cause daemonset rolling update stucked, this problem really bother us.
PTAL
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Default azure file permission is not suitable for most users, dozens of users complain about this, set as is the best choice here, if users don't want , then they could use mountOptions to override and . What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign


I ran a deployment with replicas 1 and use to set container's subpath dir, which uses NFS as the storage backend, and create about dirs in container's subPath dir,when delete the pod, it spent much more time to end status, and when the replicas increase to 10 and do the rolling update, at the same time dir nums in containers increase to , it spent about 30 minutes to totally remove pod, which is intolerant. The pod is deleted fast. a pod using subPathExpr as the volume mount. dirs in the backend storage corresponds to the container volume mount the pod kubelet log We can find that it spent about 17s to clean the mount dir in kuelet. No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After some debugging, I inspected the pprof with flame graph when delete a pod, it shows that and cost much more time, here is the graph: ! Apparently the Walk func spent much time when doCleanPath, what we care is the top level dir,and this pr has some fast path to avoid much recursive time, but before gol1.16, the filepath.Walk is still less efficient, because when used in kubelet, it reads all the dir in the mount point before call the callback func in Walk even we don't use it after that, so it spends much time. And after go 1.16, filepath.WalkDir is introduced, which is more efficient than Walk, as the comment said in golang WalkDirFunc is called before reading a directory, so it can save much time when there are lots of dirs in the first level of mount points. So it's time wasting and meaningless when we use Walk so far because we traverse the dir but we didn't use it and return SkipDir err, the code is After I replaced it with , the took only about 41ms, which is a huge boost than 17s before.


When master opens up for , we need to add this back in: /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
note to self / 1.21 and 1.22 release notes team -- let's make sure the PR body on the original PR is copied over to the new PR. will think that the old release note and documentation has been covered in 1.21, but we will (I will update this comment with link) manually override it to not show up in 1.21 release notes.


Deployment is defined in a json file with several environment varibles. Deployment is deployed using "kubectl apply -f ". Pods get most of the environment variables defined, but not all. Specifically, the variables defined in the are named: { "name": "DBHOSTS", "value": "10.0.0.1" }, { "name": "SLACKENDPOINT", "value": "ERRORNOT" }, { "name": "LACKENDPOINT", "value": "ERRORNOT" }, { "name": "SLACK", "value": "ERRORNOT" }, { "name": "CKENDPOINT", "value": "ERRORNOT" }, { "name": "ACKENDPOINT", "value": "ERRORNOT" }, { "name": "KENDPOINT", "value": "ERRORNOT" }, { "name": "CASSANDRALOGGINGLEVEL", "value": "ERROR" }, { "name": "ENABLEGETIP", "value": "" } the environment variables that appear in a pod are: ENABLEGETIP CASSANDRALOGGINGLEVEL KENDPOINT ACKENDPOINT CKENDPOINT DBHOSTS That is, ENV variables named "SLACKENDPOINT", "LACKENDPOINT" and "SLACK" never get passed on to the pods.
It also won't accept "randomshit" as an environment varible. When I add that as an environment variable to an existing deployment and run "kubctl apply -f " nothing happens. This is on version v1.2.4 /
Ehh? Can you post a full but minimized yaml that I can run to reproduce it? , drashkov wrote:
We use JSON files, here is the full one. "SLACK_ENDPOINT" does not appear as an environment variable in the pods. Doing "kubectl apply -f " does not create new pods, and does not pass these variables to the pods. If I change the number of replicas and add a new pod, that pod will be created, but won't have that varible.
I am not able to reproduce on GKE at 1.2.4 when exec'ing into one of the Pods and printing the environment. This is the yaml I used - note the image is different: What are the symptoms you are seeing?


Can you provide yaml using standard public image along with the specific symptoms you are seeing?
did you create the deployment first with out the env vars, then add them via an apply?
Thanks for looking into it I managed to reproduce it with the yaml you posted. Here's what I did: the yaml you provided as '' the "SLACKENDPOINT" env var from that file and saved that as "" (no other changes) This creates the deployment with all env vars as expected This re-creates a new set of pods and they don't have the SLACKENDPOINT var, as expected. This does not re-create the pods, so the env var is not passed along. kubectl does not return errors.
cc cc
Thanks, I was able to reproduce the issue. It looks like the config with the ENV does get stored in the last-applied-configurations annotation: But does not get put into the pods:
After a few tries I found that if you add the env as the last one you'll be able to add it. Definitely a bug.
The generated three way merge patch wasn't correct (the annotation is good, but look at the spec part): "spec":{"template":{"spec":{"containers":[{"env":[{"name":"CASSANDRALOGGING_LEVEL","value":"ERROR"}],"name":"test-api"}]}}}}
It only recognizes the last env as diff
Opened a PR ptal.
Thanks everyone.


When a Pod admission fails due to node-pressure condition then the Pod's looks like this: "Pod The node had condition: [DiskPressure]." The "Pod The node..." reads awkward. A message like: "The node had condition: [DiskPressure].". a job with a pod on a single node cluster, when the node is under disk pressure (then it is tainted with ). The Pod gets stuck in the phase. the node by kubectl command line: Pod is scheduled but admission by Kubelet fails as the taint is re- shortly after its manual removal. Then, the status of the Pod contains the awkward message. No response <details</details<detailsReproducible on kind </details<details</details<details</details<details</details<details</details
/sig node /triage accepted /priority backlog
The first part () is returned here: . I wonder where the is being appended to the beginning. It's my first try to contribute, so I am a little bit lost.
I think this is a candidate place which might be responsible for the prefix (but haven't checked): I would suggest you to try to reproduce the issue locally first then experiment with modifying the code.
/assign
thanks, I've managed to reproduce and used Delve to make sure who's responsible. I opened the changing the message. If it's accepted it will change to .


ref: /sig scheduling /kind cleanup Part of: The unit test in the scheduling queue only assumes the case where the queueing hint is enabled. We should make another unit test to confirm works without QHint. should have been existing since before we introduced QHint. So, just bring the old one back should be fine.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
/assign


Kubernetes 1.3.0. During attach disk to VM, call to cleanUpController dereferenced a null pointer. Stack trace:
The seems points to a comment instead of a cleanUpCotroller function in v1.3.0. There might be some modifications later which make the line number inaccurate. Would you please double check which line it really should point to? Thanks.
that is correct, this was a build from v1.3.0 with changes related to this: therefore the line numbers do no line up with v1.3.0. In either case, the panic happened inside
Hi Davide, I saw your PR for this issue and I have another question. Why cleanUpController could be called with an empty newSCSIController? According to the context, an empty newSCSIController should not even be returned by CreateSCSIController rt? Thanks.
it doesn't look like it should happen, but it did and there wasn't any checks for nil pointers in our code or the Govmomi library.


seen in this inside a like loop we execute a function that return a bool and an error. If we want to fail fast we return the error directly and assert on the output of the wait function. If we want to retry we log the error and return false nil /kind flake /kind bug /sig testing /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc since I think you this test but most probable the code is inherited and is a copy paste thing, but besides that, the change is to remove those there
/assign


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: there is a disk resize scenario: In the beginning, it’s 3TiB with cachingMode=ReadOnly, and then disk detached, resized to 4TiB, then user find they could not attach that disk back to any agent node, I think that’s a bug we need to fix. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Testgrid: https://k8s- https://k8s- https://k8s- https://k8s- Sample failures: This started failing across multiple test jobs last night. The culprit appears to be PR over to you. /kind failing-test /priority critical-urgent /sig storage
/milestone v1.13
/assign
GitHub didn't allow me to assign the following users: mkimuram. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think should delete the PVC and wait for it to be deleted.
Thank you for your comment. I haven't yet catching up, but as for the PVC deletion code in , which is changed in the PR, PVC deletion is done inside for delete policy case or for retain policy case. Isn't it enough? In addition, could you let me know how I can check the matrix for the failure that we are focusing? I couldn't find the failure for provisioning tests themselves, there.
Is my understanding right that the purpose of is to check whether resource is leaking by checking the diff of resource between the test? If so, is it correct that the sample output above means that one pd is leaking, not the leak of pv or pvc itself? I found that in the test for retain policy is removed in this PR when it is moved to , because it is gce specific logic and looks unnecessary. Do you think that it could be a cause of this issue? If so, should deleting the backend storage be generally required before removing the dynamic provisioned PV with retain policy?


PriorityPidEvictionOrdering [Slow] [Serial] [Disruptive][NodeFeature:Eviction] when we run containers that should cause PIDPressure should eventually evict all of the correct pods /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node /assign /priority important-soon
fork-bomb-container-container exited with code 2. I tried to create a pod in my env of containerd: The pod can be running and it works as designed. I need to test it in cri-o later.
/cc
/triage accepted


ref:
identified one problem: firewall is not open for icmp in GCI setup. I opened it manually. Still does not work.
Here are the symptoms: node can ping each other one node, it can ping all pods on it, and vice versa. one node or pod, it can ping all cbr0 gateways(10.X.X.1). pod, it cannot ping but able to curl. It cannot ping any other pod, even on the same bridge, but able to curl.
My first thought would be somethings blockign ping. your options are iptables, firewall, icmp is disabled alltogether (sysctl or somewhere in /proc/sys/net/ there should be a setting). Try traceroute and see where it gets to.
I have a working version now. GCI setup icmp ACCEPT rules in INPUT chain of filter table. But not in OUTPUT and FORWARD chain.
In summary, 2 problems: is not opened for icmp is filtering icmp on FORWARD chain. ACCEPT on INPUT chain is not enough Will submit a fix
Great detective work, Is this a ping-specific problem, or are there going to be problems with other types of outgoing traffic from pods on GCI as well?
I do not see any problem for udp/tcp traffic so far. e2e tests already covers lots of them. Unfortunately, no e2e covers ping tests and GCI happens to filter it.
We should add a test in networking suite for ping.
Is this missing a milestone 1.3?


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: ! What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.20.1 Cloud provider or hardware configuration: OS (e.g: ): 7 Kernel (e.g. ): Linux zsm-test-2 3.10.0-1160.11.1.el7.x8664 SMP Fri Dec 18 16:34:56 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Do you know how to reproduce this problem?
Yes, can reproduce the problem stably, but the steps are a little complicated.
/triage accepted


When using a AWS EBS PV or on-line volume with a partition on a Nitro machine, the volume is available as NVMe device. Kubelet is not able to format & mount the partition: It seems to me that AWS EBS volume plugin does not support partitions at all. The partition is mounted. Using an in-line volume with this pod:
/sig storage /sig cloud-provider /area provider/aws
/triage accepted


Since 2 days ago, the test is becoming more and more flaky, and one of the examples is: https://k8s-
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
please triage this one. Currently I put it as the release blocker, and please feel free to remove that if you think it is not. Thanks!
The test deletes the node, and at the end, waits 5 minutes for the node to become ready, but it timed out. cc
The test logic hasn't changed after I last modified it, so the test deletes the node through gcloud, and the instance should be restarted automatically by GCE because it's part of an instance template. The likely possibility, as pointed out, is that another change made node discovery take longer, so extending the timeout would be enough. The very unlikely possibility is there's a problem discovering brand new instances.


The test case "executing a command with run and attach without stdin" is inherently racy. The test runs, and expected to see in the output. The case when the test passes: case 1: Kubernetes creates and starts the container without stdin; The container exits before attach runs; kubectl falls back to get container log, which contains . case 2: Kubernetes creates and starts the container without stdin; kubectl attach before "stdin closed" is printed; kubectl attach contains . The case when the test fails: Kubernetes creates and starts the container without stdin; The container prints "stdin closed", but hasn't exited yet; kubectl attach contains nothing. One example failure: Related issue: Can we modify this test not to check the output? Or use to check output instead? /cc
Uploaded to try to address
/reopen This test is still inherently racy, even after
Fix in (I hope!)
Moving into the current milestone as it was not fixed in 1.14
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now). Do we still target this issue to be fixed for 1.16?
Thanks . PR is available and awaiting review!
What is the PR in question? Is it targeting 1.16 or 1.17?
/reopen This test is having a lot of failures these days
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is right, it seems to be present in kind jobs now only
/assign let's see if we start to recover a bit of stability :smile:
The rate of failures decreased a lot since last changes Checking the logs of the failures it seems that the problem is that the pod has not finished before checking the output, as Lantao described The test fails Collecting events from namespace The framework logs the pods the kubelet thinks is on the node
note that it's still racy, but it's hopefully somewhat mitigated.
/reopen
Reopened this issue.


<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
any updates here?
nope, totally agree with the description, is racy by design, couldn't find any way or trick to avoid the randomness
EDIT: It's late and I'm actually not sure I'm remembering this correctly, take this comment with a heap of salt: Specific sub-tests were fixed previously I think, but this one sub-case: That one we can't use the "Wait for stdin input to signal we're attached" hack, so it's just racy and broken. We could remove that case maybe. I'm not sure who I'd want to weigh in on that, probably someone more active in node / CLI.
not super common atm but still happening. discussing
/unassign has a fix for this


[x] test approval issue : [x] test pr : 7 July 2022 [X] two weeks soak end date : 21 July 2022 [X] test promotion pr : According to this APIsnoop query, there is still some remaining ResourceQuota endpoints which are untested. The Status endpoints will be addressed in another test. listCoreV1ResourceQuotaForAllNamespacespatchCoreV1NamespacedResourceQuotadeleteCoreV1CollectionNamespacedResourceQuota Using a number of existing e2e test practices a new has been created for three ResourceQuota endpoints. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


What happened: I have a Kubernetes Cluster with 2 Windows Server 2019 Core VMs as worker nodes. I upgraded first the cluster, and then the first Windows node following the docs here: Every step ran without issue, but the kubelet service was not started. This was not immediately obvious, since "restart-service kubelet" does not propagate the error. I think "start-service kubelet" would be better advice. The logs said: After changing the value of cgroupDriver manually in from "systemd" to "cgroupfs", the kubelet started successfully. What you expected to happen: One of 2 things: Either the Windows upgrade docs telling me what I need to do to fix this, or the kubeadm not breaking the configuration for windows in the first place. How to reproduce it (as minimally and precisely as possible): a windows node to a Kubernetes v1.19 cluster via kubeadm the cluster via kubeadm the Windows node following the docs Anything else we need to know?: Environment: Kubernetes version (use ): 1.20.1 Cloud provider or hardware configuration: OS (e.g: ): Windows Server 2019 Core Kernel (e.g. ): Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others: /sig windows
this may cause by the default value of defaultCgroupDriver cgroupfs Would you provide the from Windows node?
/assign
kubeadm doesn't do anything about cgroup drivers on Windows because cgroups don't exist on Windows. ideally the kubelet checks about this should be skipped entirely in dockershim for non-Linux.
! However, windows docker return as cgroupfs.
probably 's output is the same structure on all OSes with being a default value. this field should not be checked on Windows at all.
/retitle kubelet: don't check for the Docker cgroup driver on Windows
adding since the PR is favorable. /triage accepted


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: call delete endpoint when service is not found. What you expected to happen: It is no need to do delete in controller. Because in service restStorage, endpoint is deleted when delete service. See How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/area controller-manager /sig apps
/kind bug


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kube-apiserver --help still shows this: It looks like the related feature can be turned off. However, dynamic volume provisioning Alpha has retried in 1.6. What you expected to happen: Remove from feature gate to avoid confusion. How to reproduce it (as minimally and precisely as possible): Run Anything else we need to know?: No Environment: Kubernetes version (use ): 1.7+ Cloud provider or hardware configuration**: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
/sig api-machinery
SIG API Machinery doesn't own the contents of the API. /sig storage


What would you like to be : Currently, does not have unit test coverage. Unless there's a strong reason not to, we'd like public facing functions to have unit test coverage. Why is this needed: There's a fairly small chance of GetAddressAndDialer being modified, and thus breaking, in the future without this test coverage. However, it should be relatively straightforward to add the test coverage, and could make a good first issue for a new contributor. There's also the benefit that it helps keep the test coverage in pkg/kubelet high :) And we never know, it might help catch a bug some day :) Extra note I think this ticket could be a great candidate for someone who is new to contributing to sig-node/k8s. I'd prefer to please give first dibs for picking this up to any folks who are currently underrepresented in the k8s community. Please reach out to on k8s Slack (or post directly in #sig-node and tag me) if you're interested, and I'll assign you the issue. I'm happy to mentor and be helpful in whatever way I can :)
/sig node
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Hi let's me solve it
Sounds like a plan! Can you please tag and myself for the review?
Hi, I've already created a pr which , please help me complete the review, thanks


hack/build-cross sets it without looking at the current value. And the value has to be an array, which is a bit awkward to use.
This could be improved indeed. Maybe something like and Haven't tested it though... But why would one want to change the value?
I just wanted to build one, so I did the "obvious" thing: hack/dev-build-and-` It failed for multiple reasons :) , Lucas Käldström wrote:
This issue has been a major downer for me. It's much harder right now than it ought to be to articulate the few platforms for which you'd like to build each of server binaries, client binaries, test binaries, etc. Every time I've tried to do this "the right way," I've spent hours looking at scripts and feeling frustrated that build options aren't documented better. Every time, I end up cutting my losses and resorting to temporary build script hacks instead, just to get the job done.
Doesn't fix this issue exactly, but I've found that you can easily build only if you set .
Taking a look at this today, I also thought that setting seems to be the most reasonable.
/sig release


Encountered this error while running on master: Should we set to by default? cc
Why is it empty? Would be better?
seems to expect $(whoami) based on the value of that variable for my machine. I can make that change if you would like.
Can you keep as an option if it is specified? returns a 6 digit numeric value on my machine (corporate provisioned) and that is incompatible with GCI images, as well as error-ing out during e2e provisioning.
that is a great idea, so should be defaulting to whoami in the case where it has not been defined.
OK, will file an PR to fix this.
Filed PR at . PTAL


Is this a BUG REPORT or FEATURE REQUEST? : BUG: With CRI enabled by default, kubelet reports the version of CRI instead of container runtime version: Kubernetes version (use ): Master and 1.6. What you expected to happen: Kubelet should report the version of container runtime. Will file a PR to fix this. cc


Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Testing bringing up k8s cluster using kubeadm from latest k8s code from master branch. kubelet is not coming up and below is the errors from syslog: upon looking at the code from where the error is happening, it is evident that the err checking is missing. What you expected to happen: deployment to succeed. How to reproduce it (as minimally and precisely as possible): Use the latest code from k8s and build a k8s release and bring up cluster using kubeadm Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration**: GCE OS (e.g. from /etc/os-release): 16.04 LTS (Xenial Xerus) Kernel (e.g. ): Linux e2e--master 4.4.0-21-generic -Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Others: /sig node /cc
PTAL


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-enabled-default Which test(s) are failing: CSI hostpath-v0 tests Since when has it been failing: 8/8 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: We need to update client-go to get the fix for selfLink, build new nfs provisioner images, and update them in the e2e test. The fix for client-go is being cherry-picked to 1.15 if you want to use a tagged release:
/assign
/assign
This implies rebase to 1.15.x in external-storage, which seem to be challenging (at least to me :-) )
14 cherry pick also in progress:
OK, I guess this line in the lib is the culprit: So I need to update lib then external-storage to pull it in
I don't think the lib itself needs to be updated. Only updating go-client to get the fix to is needed.
Right, lib doesn't dictate what version of client-go to use. Will wait for client-go release/commit
- thanks a lot for bumping the dependency. Did you have a chance to build the new container with it so that we could update it here:
yeah, opened . (Not sure how to trigger test, or we can just merge and see.)
You can manually create a PVC on an alpha cluster to test.


reported hitting this error: The instance had 4 security groups, and only one of them was tagged. However, the logic around picking the security group is incorrect.. if we get two non-tagged SGs before we see the first tagged SG, it throws an error. We'll want to cherry-pick this to 1.2
I think this one was a victim of flakiness in the auto-assigner when the author is the first-listed maintainer. Assigning to you, :-)


<!-- Please only use this template for submitting enhancement requests --What would you like to be : Report request throttling by the kubelet by default (i. e. V=3) Why is this needed: Request throttling can have serious consequences, but it is reported (in ) at V=4, so not by default. While attempting to probe limits of how many pods per node we could run, we hit a situation where the test pods would go Pending and eventually fail. After investigation, it turned out that the pod was being started by the kubelet, but that status update was not provided to the apiserver in a timely manner, resulting in the pod-to-pod network not coming up, so a pod attempting to connect would time out and fail. The delay was root caused to throttling by the kubelet; raising the KubeAPIBurst and KubeAPIQPS limits resolved the issue. The throttling was caused by fetching secrets for each pod, which was in its own namespace. The subjects of pods/node, namespaces/node, and throttling parameters are complex and I do not propose to solve them here (those are scalability issues that have to be approached carefully). However, being able to easily identify that throttling is taking place without having to restart the kubelet at higher verbosity would help administrators identify the possibility of this issue. My test attempted to create 500 pods across 3 nodes (OpenShift allows 250 pods/node). Running at V=5, I captured a histogram of the throttling delay in seconds. There were 7881 throttling events reported total over perhaps a 45 minute period, with a somewhat bimodal (in this case) distribution:
/sig node /sig scalability
Thanks for filing the pr to fix :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


The link in the following sentence is incorrect macOS Install Docker for Mac. See installation instructions . It redirects to instead of The link should get redirected to Navigate to Click on the hyperlink in the below section macOS Install Docker for Mac. See installation instructions . N/A N/A N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig contributor-experience


Hi, I have been trying to run api-server as an standalone service along with etcd. I am passing etcd-servers and cluster-ip-range as arguments, here is the error I am getting: etcd servers are up and running on that address. Any leads as to how to resolve this? TIA.
This is only happening with go1.8 version, not really sure what is causing issue while parsing the URI.
Caused by this change i believe - CC as 1.8beta1 is being worked on here -
Thanks
I'm getting this exact issue using Kubernetes 1.5.7 and Go 1.8.3. Reverting to Go 1.7 seems to resolve it. This happened after updating a Go 1.7-compiled 1.4.8 Kubernetes cluster to 1.5.7, not sure if that's a corner case. For reference, my stack trace:


cc
Yet another failure
There is already fix in flight
Ok. I see
This was closed 5 days ago but appears to not be fixed. Many PRs have referenced it as a flake since the fix went in.
- I referenced to it 2 after it was closed, but all of those failures were from runs before this was merged. So it's possible that it's already fixed.
When was the fix merged? I saw this a couple hours ago:
Oops, same test but the error is slightly different... Maybe this is a seperate flake
Yeah - it seems to be something different.
Also - I'm not claiming it's definitely fixed - I just said that we need to double check, because the two that I linked was from before merge.
I'll close this one for now. Please reopen the issue and cc us if the flake occurs again.


When working on I a strange issue. It seems that while auto-generating conversions it works fine, but during things like or in the presubmits what is generated is different. Looking into test results: it seems that the order in which the files are processed is different then. /kind bug /sig api-machinery
FTR: The log from above has some debug-logging and panics and the conversion-generator logs are those with
/assign
modifies conversion-gen, but doesn't change the generated files. Did you intentionally exclude those changes? I wonder if it's similar to I checked out , ran I was able to reproduce the error in If that's the case, a workaround to fix the auto-generating is to first remove the affected files and re-generate them locally.
This is supposed to be no-op :) no auto-generated files should change That's not the fix. The problem is with the test jobs themselves: for generating swagger doc, they actually generate different things than they generate when you call "make generated-files" My PR is only a trigger for the problem, it's not a root cause. The auto-generated files are correct, but as part of running swagger docs, we're in-process generating a different stuff.
It looks that packages were missing. has a shorter / list compared to the other checks (e.g. ) when I ran . In the case of , the component-base config package was missing, thus the FIXME code path got hit. The list of packages was derived from . It seems that the dir was dirty during . I'm exploring if we can clean up the dir like does.
It seems that this one isn't yet fully fixed. I have updated the based on the comments on it and a similar issue on that (which I think has a very similar root cause). However, now "govet" and "staticcheck" are failing: [Ironically all other tests are passing, so I think this is again connected to some garbage somewhere] - can you please help with this?
As previously, I can reproduce it by:
I'm fairly sure at this point that the inputdirs param here is too restrictive:
It seems that this commit might be addressing this:
Heh - I think it helped me locally, but it doesn't help for - I probably did something wrong. But I'm fairly sure it's something along those lines that we need to do.


- will you be able to help with it?
So yeah - I was completely wrong. Basically, running with with DBG it clearly is a problem with gen-conversion. And the problem is that we pass only limited input to it: I think we should always pass all inputs - but we should look a bit deeper into it.
OK - so this is the problematic part: We basically process only packages that are explicitly requested by the user and ignore ones that are imported. So in order to have the conversion func imported by the generator, the package where it lives has to be explicitly imported. So we should basically always pass all input packages to conversion generator.
I actually created a new one, as it seems to be a different thing. Closing in favor of
/triage accepted


/priority failing-test /priority important-soon /kind flake /sig-api-machinery this failure is currently one of the top flake and repros in both Presubmits and CI jobs. This test is also part of the Conformance suite, which makes the flake far more critical. Can one of you please help triage this issue.
/cc
/cc
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
flakes are continuing, hopefully logging gives us more info to work with
this test seems to flake quite a bit in the presubmits and its a conformance test on top of this. Can we please look into the new logs to see if we have more info to deflake this? Triage: Logs: https://k8s-
/cc
FWIW this test seems to pass consistently in our conformance dashboards - https://k8s-,%20master%20(dev). But seems to flake only in CI and Presubmits
Do those run at different parallelism levels? It could be load/timeout sensitive
I have also seen a general uptick in flakes in other e2e tests that could be sensitive to watch event latency, not just gc tests - for example, see the failures in https://k8s-
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hey, this test does not appear to be flaking anymore can we close this issue?
Closing based on non-response. /close
Closing this issue. <detailsIn response to : Closing based on non-response. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This was never actually fixed, looks like.


If you spec a /8 for your service Cluster IPs, the bitmask becomes 2 MB, which is too big for etcd. We should enforce that the prefix be /12 or smaller or something like that.
Yes


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): statusFailure Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 4 Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: Others What happened: In FlexVolume code: The statusFailure constant should have string literal "Failure", not "Failed". Because in the test code or documentation everywhere, only "Failure" string is used, not "Failed" for the flexvolume cmd output status string. So this is a typo bug. Here is the test: here is the documentation: here is the example: What you expected to happen: The statusFailure constant should have string literal "Failure", not "Failed" at this line: How to reproduce it (as minimally and precisely as possible): code inspect: Anything else do we need to know: nothing else


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: There will be error when kubelet restarts on a agent if pod in that agent has an azure disk mounted, the error would be like following: azureDisk - No t a mounting point for disk andykubewin175-dynamic-pvc--92c7-11e7-b86b- on varlibkubeletpods-92c7-11e7-b86b-volumespvc--92c7-11 e7-b86b- What you expected to happen: pod could still works well even if kubelet restarts on the agent How to reproduce it (as minimally and precisely as possible): consider following scenario: A pod mounting an azure disk in a k8s agent The kubelet is restarted in that k8s agent The pod could not start up, it always reports error as following: 4d 1m 3065 kubelet, 14777acs9000 Warning FailedMount MountVolume.SetUp failed for volume "pvc--92c7-11e7-b86b-" : azureDisk - No t a mounting point for disk andykubewin175-dynamic-pvc--92c7-11e7-b86b- on varlibkubeletpods-92c7-11e7-b86b-volumespvc--92c7-11 e7-b86b- 4d 1m 3157 kubelet, 14777acs9000 Warning FailedMount Error syncing pod Anything else we need to know?: Environment: Kubernetes version (use ): 1.7, 1.8 Cloud provider or hardware configuration**: azure OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list .


The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig azure
/assign andyzhangx


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Since when has it been failing: It has been consistently failing since 3/6, flaking a lot before that. Failure coincides with . Testgrid link: Reason for failure: cc /kind flake /kind failing-test /priority important-soon /milestone v1.14 /sig cluster-lifecycle
sig gcp may be of help here /sig gcp
/assign for routing from SCL side. These are google owned tests.
Looking into this now. Any other googlers also doing so feel free to ping me :-)
Looks like this bug is known with a fix in flight: cc
/assign
Thanks ! Will keep an eye on this for 5 consistent passes, will reopen if there's a problem.
/reopen Most gke jobs are able to create clusters now, however the serial jobs are still seeing failures
Reopened this issue. <detailsIn response to : /reopen Most gke jobs are able to create clusters now, however the serial jobs are still seeing failures Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Nevermind, it's because the serial job with the fix is still running. I've verified that the cluster creation succeeds in the latest job. /close
Closing this issue. <detailsIn response to : Nevermind, it's because the serial job with the fix is still running. I've verified that the cluster creation succeeds in the latest job. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area platform/gke <img width="1106" alt="Screen Shot 2019-03-16 at 12 00 52 PM" src="https://user-"Think we're done here


I have observed the DNS pod sometimes getting stuck. Need a health-check that tries to do a lookup or something like that.
also DNS should probably come up with N=2 by default
Why is this attached to 1.0 if it's P3? One of those is wrong.
Health checks I can believe. We're in the process of doing that for everything else -- Nothing else is HA, so I don't buy N=2 for DNS but not for other components.
Other components are somewhat harder to make HA. DNS is trivially HA-able. But not too important, it's available after the fact, too. , Brian Grant wrote:
so you mean a generic solution for health checking, or just you are adding health check for other service. In other words should I start adding health check for DNS?
I'd still like to see a health-check for DNS, but I am not sure what is appropriate, though. Maybe have kube2sky open an HTTP server and answer /healthz iff it can complete a non-existing DNS query? Better ideas? It seems the collected other fixes add up to a more stable DNS server? , Piotr Szczesniak wrote:
I assume this is about adding liveness probes to one or more of the containers in the dns pod: Is dig in the skydns image? We could use an exec probe to perform a DNS query.
duh, exec probe is obviously better :) But it doesn't need to get a name, it just needs to complete and not hang. so something like sh -c "alarm 5 nslookup foobar" maybe? , Brian Grant wrote:
leaving unassigned since I'm not working on it right now
are you currently working on this? If not, I'll grab it. Thanks! --brendan
I think that was looking into this today.
Yeah, I sat on it a little today due to not being able to get it to actually work, but I'm pretty sure it wasn't working because exec liveness probes as a whole are broken (). Sent PR to resolve this once exec probes are functional.


The needs updating to be in line with community agreed trajectory for the movement of cloud provider code out of k/k. For example the current statement: This statement is approximately 16 months old and implies there is still ambiguity, where there is now a much more clear plan which should be clearly communicated to those who might waste their time investing work in modifying something in this directory. The situation is one increasingly past maintenance mode, with out-of-tree implementations the goal in the 1.13 cycle followed by removal of in-tree provider code a few cycles later. See also discussion on slack: /kind cleanup /priority important-soon /sig cloud-provider /assign
Thanks for the reminder Tim, sorry this fell off my radar, will put together a PR today :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten Need to update the README again to link out to the enhancement issues per cloud provider.
PTAL :)
awesome
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Scheduler performance test does not work: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Run schedulerperf by following: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign
/sig scheduling


/kind failing-test Not sure if flake. Gubernator is showing a big spike in failures that started 06/14 around 10:00 ~UTC~ PT. This seems to be when merged. Failed on my PR: /sig testing
/sig api-machinery
/cc
filed I'm hesistant to bump it all the way back to 240, if you follow back the original PR / issue trail I ran a number of iterations of testing and it really shouldn't take 240s. four minutes is ludicrous for unit testing one package.
there are 39 failures in the last day in from timeout :/ /priority critical-urgent
Did any timed-out run produce any logs?
we capture the test logs but itself doesn't appear to have any.
e.g. at random (from triage above) ="raw build-" I'm not sure this code emits any when it fails.
Likely it's timing out on some step, but I would have expected some logging from something related -- I was just going to look at the last thing logged and figure that something after that must be the culprit. Does that log capture stderr and stdout of the test invocations?
it should, e.g. see a different kind of failure with logs: we're running with json output formatting then through to junit, and prow will capture stdout/stderr of whatever is running within the job as build-
reopening to keep an eye on this.
sigh: 180s
it's doing better (all recent runs passed, looks like 3 flakes due to this since the PR merged this afternoon) but we may still want to bump it again
We're at 100% pass rate the past 3 hours but CI is not very active right now. still just 3 fails. If we have a similar rate tomorrow morning I'd opt for another small bump (maybe 180s -190s) to cover remaining limited flakes still over going all the way back to 4 minutes.
If we knew which sub test was slow when it is slow, that would possibly narrow it down to a haystack small enough to look for needles in.
dug into it locally, follow up in:
Fairly confident this is now fixed after
closing as fixed by
confirmed
example:
that run happened at the fix merged at


The script encounters an error when executed on Mac OS X because the option is not supported. The error message appears as follows: I expected the script to execute successfully. Execute the command on macOS. No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig testing
/triage accepted
/priority backlog


it's impossible to GET replication controllers on v3. on v1 on the same server it works fine. example request: GET on http://localhost:8080/api/v1beta3/replicationControllers returns 404 http://localhost:8080/api/v1beta1/replicationControllers returns 200
v1beta3 moved to all lower case URL Would give that a try first Sent from my iPhone
looks like it's working. but it breaks backwards for clients who worked with v1. For instance, in my gem I can't now smoothly work with both versions because I need to treat the name of the resource differently. why don't at least redirect those who call replicationControllers to the lowercase too? Also, there's no way to know which resources are available and their names. A lot of REST api on their most top level at least publish the links to main resources, but if I call: http://localhost:8080/api/v1beta3/ it returns 404
Because we are actively trying to drop support for mixed case prior to 1.0, I did not continue supporting the old url paths. If you look at the go client, we have a "legacy" flag we use to control this and a few other behaviors.
well yes, but breaking backwards creates more work for the users... more "if this version ..." kind of stuff.
We got v1bet1/2 wrong and don't want to carry it forever. Stopping it early before v1 is our priority.
ok :( are the REST apis going through any kind of design review or code review?
v1beta1 is really an alpha quality API. It grew organically and did not go through any kind of design review, as evidenced by the many issues you've filed. We want to get rid of it ASAP. We intend v1beta3 to be the "release candidate" for the v1 API, which will be the stable API. As for knowing what exists, you can now browse or GET . Additionally, pull is in progress, and will return a list of valid paths to GET .
thanks. will check swagger for now


Already we're getting bugs from people who are missing some of the more subtle conversions.
Point them at the conversion tool: cmd/kube-version-
Priority? Team? Milestone?
Done: