But this is wrt addon changing cluster resources, which can potentially create race at arbitrary time after cluster start (though your fix may have gotten rid of this).
Ok.. Seems like I missed Wojtek's comment above.
So I just checked the ds-controller code and it seems like we're updating the ds status only if this condition is being violated: Which means one of those fields is probably mismatching. I'll try to add some logs and run the test at --v=4 to get more info.
anything is possible, but we do have tests ensuring this works properly (no-op updates do not actually result in an update+resourceVersion bump in etcd):
So I ran some 100-node experiments against with more debug logs and I think I now understand the reason for this. The controller-manager logs and mainly show the following difference: This logs show-up when the scaler is active. And these pod deletions are happening as part of rolling-upgrade of daemonset to newer version (the one having resources set). You can see they're getting deleted one-by-one, which also explains why it takes so long (and so are happening even 30 mins after the cluster creation). And the reason why they're deleted one-by-one is because ds-controller has a rolling-upgrade strategy where by default. Proof: And the increased mem-usage FMU is because of the rolling-upgrade performed by ds-controller (over the prolonged period) - as it's the only difference I observe from logs Further, the reason why the RV was increasing was because of few fields 'NumUnavailable', 'NumAvailable', etc fluctuating while the rolling-upgrade
I spoke offline with and suggested that we should increase maxUnavailable for the fluentd daemonset to a large enough value so that we're not bottlenecked by it. My reasoning is: fluentd is not some user workload which will have some service downtime in case there are many unavailable at once fluentds on nodes are independent of each other (i.e one fluentd shouldn't need to wait for another node's fluentd to come up) we should try to make the scaling happen as fast as possible - otherwise it may be too late before they cope-up with increased log-load in the cluster I'm going to make that change and test it against my PR (thanks for pointing that we can change maxUnavailable directly in ds config).


So increasing maxUnavailable indeed seems to help (ref: ): All the 101 replicas were deleted at once instead of one-by-one and happened pretty much within 2 mins of the original fluentd-ds seen by the controller: We should get that PR into 1.10.
Thanks for confirming this! Did increasing maxUnavailable also reduce c-m memory usage?
So the run was still over the threshold, but I was expecting it as I ran c-m with increased verbosity (which ). That said, I'll observe how the usages are once that PR gets in - and FMU it's still possible to have higher values due to the rolling-upgrade potentially happening during the test.
Is there anything else to do here or should this be closed now?
Let's close it once we're sure there are no more such failures. We may also need to bump the threshold. On Mon, Mar 26, 2018, 9:46 AM Kubernetes Submit Queue < :
So bumping maxUnavailable for fluentd rolling-upgrade in is causing couple of issues: Our latest 5k-node performance failed as part of BeforeSuite where we're checking that ~all kube-system pods are running or ready. The reason is because quite some fluentd pods were killed as part of the rolling-upgrade and we timed out before all those pods could come up Our 100-node performance runs due to high CPU/mem-usage of c-m observed during density test, in cases when the fluentd upgrade happens during the test - which is kind of expected So at this point, we might want to choose among one of the 2 options: Rollback , which comes at the cost of tradeoffs I explained in Increase our CPU/mem thresholds to eliminate those flakes in 100-node job and increase the "max-allowed-not-ready-pods" in kube-system ns to a higher value, so that our 5k-node test doesn't fail in BeforeSuite Thoughts?
TBH, I don't know why it was merged without waiting for opinions... I really vote for reverting that. We don't want this - this may mean making cluster "not fully usable" from user perspective.Which is not really what we want to test.
SGTM. Let's revert it and increase our thresholds for the time-being. The discussion about changing fluentd rolling-upgrades can continue to happen in the background.


[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 5 days, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
This issue is now fixed after my above PRs. We're having continuous for the job


In and , I started seeing these flakes. The job in question has not been very flaky before so this is odd. In and , I started seeing these flakes. The job "Job should not create pods when created in suspend state" is the one that is flaking. Triage says a few days ago. No response No response /sig apps
Did some digging and I did not see any changes that could account for an increase of flakiness in the job controller code. There was a change made to the e2e tests: The logs say: The code is pretty simple so I wonder if we are hitting a case where sometimes the suspend condition is not set and we fail. Maybe we could add a consistent call here?
/triage accepted /priority important-soon
/cc
/reopen /assign do you mind monitoring the CI state for 1 or 2 days to confirm your PR really fixed it ?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It makes sense to me.
cc I think this issue has been fixed. Can we close it?
/close Thanks for the fix. The test is not flaky now.
Closing this issue. <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


We're seeing an issue with large memory growth in NoExecuteTaintManager when pods with tolerations are rapidly created/deleted. The item to the workqueue contains the full pod objects received from the informer: That means those pod objects are retained until they are processed. Additionally, it looks like the processing loop never removed the keys from the workqueue when it was done. It also means that multiple updates for the same pod result in multiple items in the workqueue, rather than enqueuing the pod to be processed once, like other controllers do. Simplifying this controller to enqueue pod updates only by name/namespace, and operate on the latest pod obtained from the store (like other controllers do) would speed up the processing and relieve the memory burden of the workqueue. /sig scheduling /cc
/assign
Thanks for finding this Jordan - looking into your fixes.
the memory issue was fixed by /close


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: fsgroup is only intended for RWO volumes because it recursively changes the permissions of all the directories in the volume. This can cause problems for nfs-type volumes because multiple pods could access the same volume and have the permissions changed out from underneath. What you expected to happen: Determine whether or not the plugin supports fsgroup either via: access mode. This is still a bit ambiguous for nfs volume types, that can theoretically support rwo, especially since Kubernetes does not do any sort of access mode enforcement. a new csi capability
CC
Are you saying that CSI internal driver should check driver capability for RWO, if supported, only then apply value ? What would the be ?
Yes one possible way is to infer fsgroup is supported for all rwo volumes, which may not be true? Like a nfs volume technically supports rwo but not fsgroup. So the other option is to make it a new CSI capability. I'm not sure exactly sure how best to describe it though, maybe something about volume ownership management.
Will it work if is not set then don't chown the ownership? I would think that for shared file system types(nfs, glusterfs) - won't be set. For raw block storage too - I think we shouldn't have to do this.
what do you mean by ? Shouldnt have to solve this issue ?
I meant - which is obvious. I just meant that the entire mechanism of changing ownership of files should apply only to volume types with block storage filesystems. I am just wondering if we can get away with detecting fsType and not have to introduce new capability in CSI.
agreed, do not map well to current csi.Capabilities enums which makes it hard to determine RWO when mounting. Perhaps, instead of a new capability, we revisit the existing capabilities to see if there are any way RWO can be reliably inferred from returned capabilities.


Currently, this is what the code is doing which we knew needed to be revisited:
Users may not fully understand what fsgroup is and set it anyway, or a default psp with some fsgroup may be defined in which it will automatically be set on all pods. The first case we can attribute to user error, but the second case may not have a good workaround and could cause permission issues on rwx volumes. So I think we do need to solve this.
Yes, agreed this has to be solved. As you pointed out user error or improperly configured PSP can set value that does not match the correct attribute, causing permission issues. Here is what I think can be done without new CSI capabilities: , then do not apply fsgroup then: , apply fsgroup , do not apply fsgroup to void stated issue then: , apply fsgroup I think that may help deduce when to apply fsgroup.
If mount reference count <= 1, apply fsgroup If mount reference count 1, do not apply fsgroup to void stated issue A RWM volume (say NFS) can be mounted on multiple nodes. Do you have reliable count of mounts across all nodes?
In addition, that would be a change in semantics to how we support rwx volumes today. If we don't have a new CSI capability, then I think the best course of action is to only apply fsgroup for RWO volumes. Some questions remain: Do all RWO volumes support fsgroup? RWX is a superset of RWO. So if a user requests a RWO volume, it could still be satisfied by a RWX volume type. Can we detect this and still not apply the fsgroup even though the volume capability will say RWO?
If there's not a reliable way to enumerate node mounts (thinking about it, for CSI external drivers, the answer is NO). Theres is no reliable way to have k8s autocorrect a bad fsgroup/PV.AccessMode combo. So we would have to rely on the config provided by user/admin. I have come to similar conclusion as -- mainly, apply fsgroup only to volumes with access modes. The other k8s specified access modes, , cannot be reliably applied.


Even if we were to introduce additional plugin capabilities that the CO can query (from driver) to find out what modes can be applied, there would still be the possibility of user/admin mis-configuration. Probably not, Is this where you think an additional capability would help ( ) That is a good point. Can't think of a way to detect/guess RWX when RWO is specified, this without the driver providing that info. Can we adopt the rule until we decide what capabilities would look like ?
I will just try to summarize the conversation from Slack. We can safely assume that only applies to block storage type with file system on top of it. So for all - volumes that need to have called MUST have a file system() on them. Is that assumption correct? If it is - can we not use presence of field or we can query the local volume for fs type (like how we query selinux relabelling capability) and call only when - volume is RW and has a valid block storage file system type. Will this not solve our use case? EDIT: renamed to .
We'll need to make it clear that is only for block-based storage systems, and the field cannot be reused for some multi-protocol file server (ie, nfsv3, nfsv4, smb, etc). For that case, then the plugin will need to expose its own new storageclass parameter + volume attribute to specify protocol.
Can we not use to get fsType - and use that information for calling ? It may be problematic to rely entirely on present inside csi source.
the only problem with is that in CSI volume info is intentionally opaque to the CSI code inside k8s. All mount logic is delegated to the driver.
Instead of a new capability or trying to deduce how best to apply fsgroup, why not pass the to the driver and let it decide how to apply it (either as a attribute for the mount or top level request param)? Right now, all mount logic has been delegated to the external CSI driver except for applying permission. Thoughts ?
that is true but we can still inspect file system on the mounted path because by the time gets called volume is already mounted. This should be no different than how selinux support is determined. what you think?
is this approved for 1.12 milestone? Thank you for working on this,
still discussing how best to approach this.


It would be a 1.12.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Approved for 1.12
Based on previous post, this is a summary: If then is not applied (because it could be an indication of a nonblock fs, or error) Done is provided if fstype is invalid Done if fstype is valid and then apply fsgroup logic else ignore fsGroup Done
- any updates on ?
Waiting for a review and approval.
you have a outstanding comment on the PR. Can you please answer that?
How does this approach relate to inline CSI volumes? There's no PersistentVolume involved, and hence no way to specify an access mode. , but my driver ignores it (always using tmpfs). I considered having my CSI driver read the mounting pod's spec to see if it has a security context with "fsGroup" set, since that detail is not supplied to the driver, as . My goal is to have the volume mounted with the files on it readable by the container's user, but there's no way for my driver to know who that user will be. Am I supposed to just use 0444 for the ownership bits, to allow everyone to read the files?


returns It's because we recently change HorizontalPodAutoscalerSpec.MinReplicas from int to *int.


In our Tanzu Supervisor Cluster k8s setup, we observe that the "should verify that a failing subpath expansion can be modified during the lifecycle of a container" conformance test fails. The root-cause seems to be that the test is updating the pod annotations, thereby resulting in removing the other annotations present in the pod object. Removing the custom annotations that we use causes the test to fail during pod deletion. Should this test perhaps use Patch instead of Update so that the annotations are merged and not replaced? Thanks ginkgo.By("updating the pod") podClient.Update(pod.ObjectMeta.Name, func(pod *v1.Pod) { pod.ObjectMeta.Annotations = map[string]string{"mysubpath": "mypath"} })
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /sig conformance I personally agree that it should not remove existing annotations, but better ask the test owners
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig storage /sig conformance I personally agree that it should not remove existing annotations, but better ask the test owners Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
May I request the original author to weigh in on this? Thank you.
I dont have any issues with the code change to a merge but what would bother me is that conformance tests are being altered by outside influences which doesnt seem particularly right. I think this would be for the sig storage owners to decide
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale


WRT , container registry credentials should be included in a docker file instead. We deprecated for this reason in , but is more tricky, because the file it references is not static. We need to figure out what the alternative is for dynamically providing pull credentials to nodes. /cc
I think we need to enable Managed Service Identity (MSI) for all instances before deprecating . To do this, we also need to grant VM access to Resource Group in Azure IAM (e.g. config Azure AD in acs-engine). WDYT?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I think this is still active. Right, /remove-lifecycle stale
Thanks, this is still active
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
-- What's the status of this?
Sorry, no progress yet. Any ideas on this?
-- pinging again on this one.
We're not making any progress yet in this release. Still seeking the possible solutions for non-MSI cases, secret is one option for this. /milestone v1.15 Related and
None one is working on this. Let's move this out of 1.15 /milestone v1.16
thanks. This missed v1.15 feature timeline. LGTM of moving to v1.16.
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 5 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?


/milestone clear is preparing a KEP for this, but it's still not ready yet.
Will follow this KEP for implementation
This should be addressed by /milestone v1.18
Hello ! Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March, which is a few weeks from now. Is this issue still intended for 1.18? Thanks in advance!
C/O Code freeze for release 1.18 is starting in 5 March (less than 1 week from now) and we want to ensure that each issue has a chance to be finished on time. Is this still on track for this deadline?
/milestone v1.19 The work on the KEP hasn't started yet, so moving this to 1.19
Hello from Bug Triage team! Wanted to follow up on this issue with a friendly reminder that the code freeze for 1.19 is starting June 25th (about 4 weeks from now). As this issue is tagged for 1.19, is it still planned for this release?
/milestone v1.20
Hey Bug Triage here. Wanted to follow up on the status of this issue as we're approaching code freeze on 12.11.2020. This issue is tagged for 1.20, is it still planned for this release?
Hi Bug Triage here, seeing that there is not much recent activity here, and we are pretty late into code freeze, I am going to remove the milestone label. /milestone clear
/triage accepted /priority backlog I think we would be able to remove the flag once external kubelet credential provider promoted to GA
/kind deprecation
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available .


If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


https://k8s- It's quite flaky.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/assign
/sig networking
There are two failures in gci-gce-slow, so it's not specific to gke. https://k8s- Also in gce-slow: https://k8s-
/sig network
This is strange, seems like this test is trying to reach the test-netexec pod through servicePublicIP:serivcePort, but the request hit the gce default backend instead. cc
There is another type of failure: Seems to be the port collision between test service and fluentd daemonset, which changed to use few days ago ().
cc
Checked the kube-proxy logs and found this is not the case. Port was not allocated (although that could happen in theory) to any service during test:
This turned out to be an interesting corner case. It happens at the beginning after we set up the GCE network LoadBalancer for OnlyLocal service --- when the LB still forwarding traffic onto nodes that don't have endpoints on. For more context, when network LB forward traffic to nodes, packet arrives node with below src:dst: If no endpoints are on the node, and there happens to have a pod using hostPort with the same port number, packet will be DNAT to that pod IP instead. These are observations and I haven't traced down to the exact iptables entries. I'm changing service port in test to avoid collision for now. cc


At least on AWS, if the clock drifts 300 seconds (which it will), then we get the rather cryptic message: This is because AWS includes the timestamp in the request. Installing ntpd would stop this drifting from happening. If the drift does happen, the likely symptom is that EC2 calls would stop working: attach/detach volumes on the minions, instance sync and routing configuration on the master. I don't believe that GCE has the same problem.


From we can find that, hardcode "private" and "public", seems only support that network with such name. , it fetch the first ip. usually, it can include both v6 and v4 ip. what's the meaning to just fetch the first one ? is it v4 or v6 ?
ping and
Yes. Zoom out a level or two in the code, and you'll see we're trying to either a) setup an openstack loadbalancer to point to the minion node (a single address) using , or b) report to kubernetes all the addresses we know about the node and whether they are "internal" or "external" in . All we have at the starting point is the server name (from kubernetes), and we need to make up the rest of this information somehow. (I agree (a) should reuse the NodeAddresses() function from (b), but the latter is a newer function and the code hasn't yet been refactored in that way. This doesn't really change the substance of this discussion.) Re (1): The logic as currently implemented is: Assume any network named "private" is "internal" Assume any network named "public" is "external" Assume the AccessIPs (IPv4 and IPv6) are "external" Any other network addresses are ignored. I implemented the first two ("private" and "public") following the Rackspace public cloud precedent, and on the basis that users in other OpenStack clouds can use these names if they want to encourage an address other than the Access addresses. For the (a) use case, we just take the first address we find in order of the above; for (b) we report that full list. I preferred "private" to "public" since (again, at least on Rackspace public cloud), the traffic on "private" network is sometimes cheaper/free. I agree this isn't great, and please suggest something better if you have a use case in mind. Perhaps we should pull a list of network names that should be considered "internal" and "external" from the provider config? Perhaps I could assume that provider networks are public and non-provider networks are internal? I don't want to add too much configuration here if there's a good algorithm that will Just Work.


I think in practice it's probably ok to continue to prefer "internal" to "external" for loadbalancer setup, but I can contrive hypothetical cases that need more flexibility here too. Again, if you have a concrete need for such a thing, please file a bug so we have some idea of the flexibility required. Complicating all this is that Kubernetes never actually defined what an "internal" address was. just says "Generally the IP address of the node that is routable only within the cluster" which isn't a sufficiently strict definition. Looking at where Kubernetes actually uses it: just uses the first address reported (preferring InternalIPs to ExternalIPs) and the Kubernetes master configures an "SSH Tunnel Check" which just uses the first ExternalIP ... which kind of makes it pointless to ever even bother reporting anything beyond the first address. In both cases, it's assumed that this address is reachable by the Kubernetes master nodes, which makes the Internal vs External distinction a bit pointless. Re (2): is meant to return more than one IP, and the loadbalancer setup can only use a single IP for each node (or would unnecessarily track/healthceck multiple paths to the same backend for multiple addresses). If you're seeing single addresses returned by then you should file a bug. All the above treats IPv4 and IPv6 as equally important, if the OpenStack cloud has both, and the "first" address used for the loadbalancer will be an arbitrary choice between the two protocols (not quite true: will use AccessIPv4 before AccessIPv6). I think that's how it should be (the rest of the code should be protocol agnostic), but I wouldn't be surprised if there's some part of Kubernetes that doesn't support IPv6 properly. There certainly used to be lots of places that assumed IPv4 in Kubernetes, and GCE doesn't support IPv6 iirc so it gets less testing - if you've found something that breaks with IPv6, please file bugs. Now this bug: what would you like done here? Would you like the current situation to be documented further? Is the "private"/"public" naming thing too restrictive for your usecase? Something else?


Hi, if we talked about openstack, I expect it not bind to rackspace, although, the kubernetes called RK SDK, But from my understanding, seems RK itself SDK not bind network to "private" and "pubic". User can define public network with their names, like site-network, for example. or "testnet", And for private, it should be seem. For nova list, the addresses, it can have "OS-EXT-IPS:type":"fixed" or "OS-EXT-IPS:type":"floating" for floating, I think what you means "Public", for fixed, I think its what you means "Private". In other words, we'd better to find related information from other fields in nova or neutron infromation. Instead of bind to "private" and "public" name.
I this issue and wrote a patch that implements what suggests. I will send a PR soon
Glad to see some improvements here. And wish to use it if implemented


I am trying to bring up a kind cluster with containerd in order to run e2e test on mac + kind. Not sure whether this is a bug or I missed some prerequisites. Any help would be greatly appreciated :) building the image dra/node:latest works as expected:bringing up the kind cluster failed (seems that kubelet is unhealthy and healthchecked failed) below the control plane logs: below the logs of one worker: successfully bring up the kind cluster with containerd Tried on M2 macbook pro I followed NONE Darwin MacBook-Pro-de- 22.1.0 Darwin Kernel Version 22.1.0: Sun Oct 9 20:19:12 PDT 2022; root:xnu-8792.41.9~3/RELEASEARM64T6020 arm64 arm Darwin No response No response
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what type of mac you're using? for m1/m2 mac, you need to replace the linux-amd64 with linux-arm64(your log seems like using m1/m2 mac which same as the problem I've met before)
thanks I am going to try that
ping me if you can run e2e on mac. I setup the cluster but failed to run the e2e test..
can you please share the exact command you tried and the failure ?
If you use a newer version of kind you might not need to clobber the included containerd, not sure what version is required And yes, on M1/M2 you need to use arm binaries/images.
this is the command that I tried !


It worked for me just fine on M2 machine with this obvious fix for the build script: Here is a test output:
I'll submit a PR with the fix. Updating the documentation would be also nice to have. I hope and can do that.
DRA tests fail on kind 0.18.0. Which Containerd version is used there?
It was the latest 1.6.x at release time. 1.6 is their LTS branch before 2.0 comes which has some tricky changes for us to deal with (notably the CRI hosts config deprecation which will break all our local registry users :/)
Unfortunately 1.6 doesn't have CDI support, which DRA depends on. The version used in the build script is 1.6.0-830-g34d078e99, but it was probably built from containerd master after 1.6 was released.
thanks for providing the details. you could run the e2e as shown by and find the result


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When starting pods with AFS volumes, there is an initial delay of five seconds until the pod is transitioning from the "Scheduled" state. The reason for this is that currently the volume mounting happens inside a wait.Poll which will initially wait a specified interval (currently 5 seconds) before execution. Event log for comparison with and without AFS volume attached (seconds since event): What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign /priority important-soon /sig cloud-provider /area provider/azure
see


I an ELB to the Kibana cluster log viewer but it is not reported by which, aside from having a URL, should check each cluster service and if it has an external IP it should report that instead of the master proxy address.
cc


I've been pulling my hair out trying get kubectl talk to my local cluster. It seems that kubectl merges the local file with values pulled down from the cloud. I've done the following: The I just set is silently overridden. Questions: is this documented anywhere? should kubectl merge in my GCE cluster when I set my cloud-provider to something else? should kubectl override my previously set current-context?
workaround is to use flag to override the config file value.
but not a good UX long term
Someone else hit this as well: . Better fix is to have the local-up script set a context, which I will try to add today. As for the overwriting behavior, it's documented in , but based on what you describe, the intended load order is not working; local .kubeconfig should overwrite global. cc
That sounds infuriating. I'll take a look this morning.


After this () has been merged and EnpointSlice feature is now enabled by default, we noticed Kube-proxy will not work on Windows. Only message I can see in the logs is: /sig windows
/cc /cc /cc
How about separating the above check into two if statements so that the log can pinpoint which field is false ?
It seems long term, EndpointSlice support should be so that is kept in sync with other implementations.
cc
Thanks for reporting this! Windows proxiers were still intended to be reading from Endpoints, but apparently that's not working as intended. EndpointSlices are moving back to disabled by default for 1.17, so that should fix the failing tests for now. I'll work on an actual fix for this as well, but it may not make it in until after KubeCon. /assign /sig network /priority important-soon
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
/assign cc
GitHub didn't allow me to assign the following users: jocelynberrendonner. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
: Please let me know if you need any help on the fix! I'll be more than happy to review/help test the fix on Windows.


is sent as a fix. cc It's a test failure reported by the publishing robot.
The line it fails at:
It looks like a vendor copy vs gopath copy issue.
One possible fix is removing from the vendor/ directory. I'm testing locally.
Confirmed removing from the vendor/ directory fixed the test in robot.
We need to add the "issue filer" feature to the publishing robot, we can't rely on to report robot issues everytime :P I'm just not sure how long the publishing robot will be needed (i.e., when we will get rid of the staging/ directory) so not sure whether we should invest more time on the robot.


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: patched out the Size field (was PR ), it will cause issue again. See also and . What you expected to happen: PVC resizes of Cinder volumes successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/kind bug /sig storage /sig openstack
/assign


We have a new vanity url, , which redirects to the GCS release bucket - e.g. Right now, the https endpoint doesn't work, since isn't in the list of domains on the certificate. Once it works, we should switch all references over to the nicer vanity URL. x-ref
So what it required to make this issues actionable? Where to add this for the certificate?
the issue tracking that is
This needs to be triaged as a release-blocker or not for 1.5
not a release blocker.
Is it appropriate to move this to the next milestone or clear the 1.5 milestone? (and remove the non-release-blocker tag as well)
can this be closed?
I forgot to get to this in 1.6, but it'd still be good to do.
do you want this in v1.7? is there a sig to own this?
doesn't need to block 1.7, no.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/sig release


Our GKE scalability jobs (https://k8s-) are failing in the cluster teardown step (even though the e2e tests themselves passed). E.g failures: https://k8s- https://k8s- Seems like it's caused by some leaked network-related resources: /assign Krzysiek - Could you PTAL into this when you find time? It'll be great to have our GKE suite back to green. cc
I think this is related to an earlier bug we've seen (similar one for GCE): Basically service-controller is creating some cloud resources (like firewalls) for lb-type services which are not brought down properly when the k8s cluster is brought down.
Also another related issue - There should also be an internal bug for this somewhere. IMO we may be able to fix this in test-infra (without waiting for proper long-term fix on gke side).
I am adding clean-up method. This way all of the firewall rules will be cleaned up after tests. If this method has to remove any firewall rule, the teardown phase will return exception. It's not a fix for the bug - it only allows us to easy isolate tests leaking firewall rules.
Tests: should have session affinity work for LoadBalancer service with ESIPP on should be able to switch session affinity for LoadBalancer service with ESIPP on should have session affinity work for LoadBalancer service with ESIPP off should be able to switch session affinity for LoadBalancer service with ESIPP off leak resources related to the LoadBalancer. PR will fix this.


From After , Multi-AZ Clusters tests in 1.19 and 1.20 started timing out as createBalancedPodForNodes is extremely inefficient in scale: for each node (out of 5k), it lists all pods in a cluster serially (each list takes ~5s), which translates to ~7 hours. Optimizing this function by listing pods once and then in memory by Spec.NodeName /kind bug /priority critical-urgent /sig scheduling
/triage accepted
/assign
/reopen Looks that ComputeCPUMemFraction took only 12 minutes, but it stuck later, on It is scheduling one pod per each node: Which is waiting 5s to be ready: 5k nodes * 5s = ~7h
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Can you do the pod creations in parallel?
Yes, good idea. I'll update to do that


<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-kubernetes-e2e-ubuntu-gce [sig-storage] Subpath Container restart should verify that container can restart successfully after configmaps modified links to appreciated links to specific failures in spyglass appreciated Triage shows that this has been flaking quite frequently: It has flaked in 5 of the last 10 runs: /sig storage /priority important-soon <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Noticed some more flakes on this one
Yep, looks like this is still flaking quite frequently: Going to re-ping SIG Storage.
/assign
Test is flaking maybe half of the time. ~This test is failing in gce-ubuntu-master-default - not sure if it's failing in any other dashboards, nor whether this is run anywhere else.~ edit: This is failing on Ubuntu only. Not sure if this is limited to master branch. Fix to the original issue was in since 1.19. This is a subpath test which a configmap a container that mounts the configmap a liveness probe and waits for the container to restart. liveness probe and waits for the container to stop restarting. The test failed at the last step - the container never stopped restarting. Timeout is 2min. The liveness probe continuously checks whether a "probeFile" exists in (an emptyDir), and if it doesn't exist, whether the configmap file contains the modified value. The probeFile essentially acts as a condition which guards the probe from checking the configmap file. The test fails the liveness probe by removing the probeFile without updating the configmap just yet. It then fixes the liveness probe by updating the configmap to the modified value.


So if the container keeps restarting, the liveness probe could've been failing continuously within 2min, due to one of the following: (? not sure about this one) Command can't be executed (for example if container can't start) configmap file was never updated. Within 2min, the pod only restarted 5 times. The liveness probe's period is 2s, initial delay is 1s, and failure threshold is 1, so a restart due to liveness probe failure is triggered every 3s. The image is busybox. The command is just , so it's unlikely for the container to have crashed. There were no liveness probe unhealthy events after the test fixes the probe, and yet it continued to restart. No more events after probe is fixed.
Could we check if the configmap value was updated for sure to rule out that scenario ? I have not tried to run an end to end test locally :) so am not sure how to debug it
Will do! I ran the test in a GKE Ubuntu cluster running 1.19.6 and couldn't reproduce the failure. I'm now trying to run a cluster on GCE with Ubuntu nodes running k8s master, but haven't been successful after trying a number of techniques. If anyone happens to know how to do this let me know; I'll keep asking around otherwise.
I got a repro on a GKE cluster backed by COS. After the test failed the container ends up in Running state eventually, and the configmap file outputs the expected modified value. The fix is to increase the test timeout. PR coming up.


/kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
/sig cli
There may have two issues to fix: doesn't support run with assertion must check whether schema is nil first
/assign
/triage accepted


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If you create a service that reference a Basic IP in a standard load balancer cluster, the service will stuck at "Pending" state for days without detailed error from Azure response. When you describe the service, it only has following: What you expected to happen: In this case Azure responded 400 with detailed error message that the referenced IP is a basic IP and cannot be used on standard LB. I would expect the detailed error message show up as events for easier diagnostic. How to reproduce it (as minimally and precisely as possible): Create a service that reference a Basic IP in a standard load balancer cluster Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /area provider/azure /sig cloud-provider
/area cloudprovider


It looks like there was a change that first appeared in Kubernetes 1.29, where in 1.29 the flag on kubelet is no longer being used to populate the Node's Addresses.Hostname value. Here is the relevant output from a for a 1.28 cluster with : And here's the output from a 1.29 cluster with the same configuration: Here is the full list of parameters in case that is helpful: This is from an IBM Cloud Kubernetes Service 1.28 and 1.29 cluster. We do not think there was any cloud-configuration-manager or any other related change that would cause this difference. Is it possible it is a side effect of ? I expected the behavior to be consistent between Kubernetes releases, since I didn't find any reference to this change in Create an IBM Cloud Kubernetes Service cluster at 1.28 and 1.29 and see the difference No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yeah, it seems likely my change broke this, it should keep working as you say, let me take a look /assign
/sig network /sig cloud-provider
/assign /unassign Thanks
FYI it was an unintended side effect of:
Do the sig cloud provider folks need to help here? We did make external cloud providers the default as of 1.29, which may have uncovered this. Let us know!
No, it's just a dumb kubelet startup bug. Well, not dumb, but easy to fix.
It seems the problem is that once kubelet sets the hostname the cloud controller manager does not override it so maybe the right behavior is only override it if is actually overridden ?
what problem?


we are discussing if the fix should be backported or not It seems you are relaying on the with type Hostname and this not clearly documented We also find out recently that the behavior of this address will be overriden if the external cloud provider configures it if the external cloud provider returns the hostname , then this problem will be solved without need to backport
(Well, to clarify my comments there: it's not well-defined between different clouds and node linux distros. If you know what cloud and distro you are on, then the behavior might be consistent and usable, but that still makes for a lousy kubernetes API.) Anyway, the question here is: was this actually breaking anything major, or did you just happen to notice this and think "that's weird"? (Also, if anyone else was broken by this bug, feel free to chime in; trying to figure out what to do about backporting this, because the original fix for this bug ended up introducing another bug, and the fix for that_ bug is maybe not backport-able...)
So far I have only heard from a single customer (who was using IBM Cloud Kubernetes Service) who had something in their cluster relying on cluster node's Address.Hostname value, and so when they updated to 1.29 they were broken. I believe they just modified their code to get this "Hostname" value (which was really the node IP) from somewhere else. So, to answer the question, it is somewhere between the "I noticed it and thought 'that's weird'" option and the "it broke something major" option. We have this documented in our 1.29 release notes for IBM Cloud Kubernetes Service, so other customers will have fair warning before updating. So at this point I don't consider fixing this in 1.29 as essential.


Kubernetes version (use ): kubernetes/master Environment: na What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): see above Anything else we need to know: nope, have a fix.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list


Kubernetes version (use ): 6 Environment: GCE What happened: I created a cluster for Heapster testing purposes to debug failing test in It turned out that Summary API does not export system/misc stats in SystemContainers section (though there are kubelet and docker stats). Everything works as expected on nodes. See also https://k8s-
What node OS were you using? We don't create the system (misc) container on systemd, so it's expected that GCI nodes don't report those stats.
The default configuration for 1.4, which is: master: GCI nodes: Container VM
Are the nodes running 1.4.6 too? There was a bug with this in 1.4.5, but it should have been fixed in 1.4.6.
Yes, both are running in 1.4.6. If you want to reproduce from Heapster current head run . It will take some time and as a side effect you should have a cluster with the mentioned problem. This was catch by integration tests and I temporary disabled the check here
if you have spare cycles, can you look into this?
friendly ping
Looking at it now
I have found a couple things so far. On the containerVM nodes, cadvisor is exporting a container with "name": "/system" On the GCI Master, cadvisor is exporting a container with "name": "" The has this: "{% set system_container = "--system-cgroups=/system" -%}" (/system is the default misc container name) So the real question is: Should cadvisor export "/system" instead of "" on GCI? Or should we be configuring GCI to use "--system-"?
I thought we fix this very issue for 1.4 branch by a while back. Are you running into the same issue with 1.5?
I think I told you the wrong thing earlier. We don't support the misc (/system) cgroup on systemd systems (I can't remember why, but it was an explicit decision - or might know). The is something different, I believe a base systemd cgroup (slice). As for the misc container on ContainerVM nodes, we've been testing for the misc container in 1.5+ in the summary api test (https://k8s-) for a while, and I haven't seen this issue.


So if this is still reproducible, it's limited to 1.4. See this the bug was created for the post-fix.
Found the explanation here: The should use the flag to know the location that has the processes that are associated with , but it should not modify the cgroups of existing processes on the system during bootstrapping of the node. This is because is the on the host and it has not delegated authority to the to change how it manages .
IIUC this is intended behavior on systemd nodes. Based on the explanation provided, the kubelet doesnt have the authority to control "misc" processes on systemd.
+1. "misc" is just a bandaid for cvm. , David Ashpole <:
Ok, so I'm closing this issue.


The last 4 runs of https://k8s- have failed on the and steps with : https://k8s- https://k8s- https://k8s- https://k8s- The GCE version of this test suite appears to be fine. This is blocking the submit queue. CC
Is there a way to see what does /cluster/gce/list- output in the runs... exit status 1 is not very informative
Here is a non-gke failure: https://k8s-
Given that the last thing output from the listing of resource is: I would bet that the defined is failing to find a match from (see )
Hummm I ran it locally against but did not get the same failure
you might be more familiar with list-
This started happening on the 16th and became more regular on the 18th-- do you think the new janitor code might be causing this? More details here:
is giving me exit status 1, let me find which command it is.
Yeah, and have figured it out. Because of the janitor, we no longer have any default firewall rules in these projects (which might be a bug), and so now the is failing. I think in the distant past always printed out a header, even if nothing was found, though it hasn't done that in a long time. Fix for is to the on L52. We may also want to decide if we should be deleting the default firewalls.
We'll want to cherrypick that to the old branches, too.
hummm I'll let it skip default-firewall
Keeping open until we've verified fix


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: cAdvisor has a option to only collect docker related cgroups. Kubelet stats api now supports allocatable cgroup, /kubepods, for the runtime cgroup and the kubelet cgroup. But with settings, Kubelet stats api can not query pod related cgroups metrics. FYI: What you expected to happen: should be compatible with kubelet stats api.
/cc
/sig node /sig instrumentation
add to cadvisor in A comma-separated list of cgroup path prefix that needs to be collected even docker-only is specified bump cadvisor dep for Kubernetes setting to
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or .


/close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


/kind failing-test <!-- If this is a flake, please add: /kind flake --/kind flake <!-- Please assign a SIG using: /sig SIG-NAME --/sig testing apps
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you PTAL, it seems this test was in It doesn't seem to have a high rate of failure , but seems flakes are legit
/assign
side note: it's really hard to make it fail in a fast machine :) I had to temporarily add a sleep in the reconciler.
have you tried Jordan's instructions compiling and running with ? for integration you should have an external etcd instance
I haven't. Good to know for next time :)


Update the godoc on the encryption config API on how to specify group/resources to be encrypted ref: godoc has the details na No response_ na na <details</details<details</details<details</details<details</details
/sig auth /triage accepted
/assign Cool-Coder174
Thanks for volunteering for this LMK if you need any help with it.
Just for clarification, I just need to add documentation on how to configure encryption on groups/resources?
You can update godoc to mention that custom resources can now be to the list of to be encrypted as well and how to add the of the resource to the list. e.g. from this test
/assign /unassign Release v1.26 deadlines are here so we need to get this completed ASAP.
/milestone v1.26
I apologize for the slow actions. I know time and contributions are valuable. May I request a walk through. This is my first time ever contributing and I really want to learn. I have been reading some of the code base on my spare time.
this feature is on the harder side of things to get started with. Reading the blog post and the associated documentation is probably the best starting point:
& what is the status on this issue? 1.26 is coming up! cc
/milestone v1.27


tests are failing atm on GCI https://k8s- because VOLUMEPLUGINDIR=/home/kubernetes/flexvolume set in takes precedence over KUBELETTESTVOLUMEPLUGINDIR="/etc/srv/kubernetes/kubelet-plugins/volume/exec". As far as I can tell /home/kubernetes/flexvolume is not readable yet by containerized kubelet so the test needs to keep using "/etc/srv/kubernetes/kubelet-plugins/volume/exec" and the bash needs to be shuffled around to solve this. cc
/sig storage
This is more an acknowledgment of the issue than anything, can't write a PR until later
cc /kind e2e-test-failure
this is currently blocking alpha.3 release
we are just waiting for approval
I think they’re still failing, not sure what the release process is, but these tests started running only yesterday and so failure indicates something wrong with the test themselves, they shouldn’t block release. (They succeed on gce cluster with Debian nodes but fail on gci)
well, let's see if we can get the kinks worked out before the beta release on 9/27 or we will need to disable it from release-blocking
/assign /reopen
I0824 17:49:28. 8562 ] operationExecutor.VerifyControllerAttachedVolume started for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-e2e-tests-flexvolume-wf48v/-88f4-11e7-a3f9--flex-volume-0") pod "flex-client" (UID: "-88f4-11e7-a3f9-") edit: VerifyControllerAttachedVolume must succeed since the plugin is non-attachalbe and no error is reported. But the voluem is absent from node.volumesInUse I0824 17:44:06. 4 ] AttachVolume.Attach succeeded for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-j8962/flex-volume-0") from node "bootstrap-e2e-minion-group-9913" W0824 17:44:06.


4 plugin-] flexVolume driver k8s/dummy-attachable-e2e-tests-flexvolume-j8962: using default GetVolumeName for volume 0x1c5df60 I0824 17:43:59. 3728 ] operationExecutor.VerifyControllerAttachedVolume started for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-j8962/flex-volume-0") pod "flex-client" (UID: "-88f3-11e7-a3f9-") E0824 17:43:59. 3728 ] Operation for ""flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-j8962/flex-volume-0"" failed. No retries permitted until 2017-08-24 17:44:00. +0000 UTC (durationBeforeRetry 500ms). Error: Volume has not been to the list of VolumesInUse in the node's volume status for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-j8962/flex-volume-0") pod "flex-client" (UID: "-88f3-11e7-a3f9-") Seems there is something wrong with VerifyControllerAttachedVolume, still at a loss as to why it's showing up in GCI but not Debian edit: get node -o yaml shows: yet it errors because the volume hasn't been to volumesInUse??? Aug 24 20:31:42 mawong-e2e-minion-group-npjh kubelet[9431]: E0824 20:31:42. 9431 ] Operation for ""flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-z8xc4/flex-volume-0"" failed. No retries permitted until 2017-08-24 20:31:44. +0000 UTC (durationBeforeRetry 2s).


Error: Volume has not been to the list of VolumesInUse in the node's volume status for volume "flex-volume-0" (UniqueName: "flexvolume-k8s/dummy-attachable-e2e-tests-flexvolume-z8xc4/flex-volume-0") pod "flex-client" (UID: "-890b-11e7-9392-")
Never mind the GCI vs Debian issue, our test fix would have worked but in between and , something else changed and is causing both GCI and Debian tests to fail.
might be the cause, so we might be looking at legitimate failures caught by the test at this point (yay?) and not faulty tests. investigating...


It seems that exec probes must emit "ok" to be considered success, rather than return 0. If this is what we really want, we should document it somewhere.
This is (was?) a limitation of docker exec, which didn't return the code. Brendan On Jan 28, 2015 8:38 AM, "Tim Hockin" wrote:
The doc says it looks at exit status, so we need to fix one or the other.
for kubelet
for exec
could you take a look at this? Feel free to re-assign it back to me.
FWIW the current behavior isn't SO bad, and it makes clear a failure in the thing that probed vs a failed probe. Just saying, we don't NECESSARILY need to change it. On Jan 28, 2015 5:26 PM, "Dawn Chen" wrote:
With docker 1.5 we can go back to using the exitCode instead of "ok" string. OOM kills will also have special exitCodes in docker 1.5. , Tim Hockin wrote:
ok, I'm going to change the docs for now, we can change back when we standardize on 1.5...


Nodes Network when a minion node becomes unreachable [replication controller] recreates pods scheduled on the unreachable minion node AND allows scheduling of pods on a minion after it rejoins the cluster(Failed 17 times in the last 30 runs. Stability: 43 %)Failed with 0 retries
for triage
I'll look into this.
I talked to about this and he had a pretty good guess in about two seconds about what the problem is. ReplicationController sync method does this: but DaemonSet controller doesn't. (I guess when DaemonSet was copy-pasted from ReplicationController, this was accidentally removed.) Pod reflector and syncer are running in separate goroutines, and if pod reflector doesn't run then syncer will think there are no pods on the node.
To be clear: this is about the test Controller Manager should not create/delete replicas across restart which is the one that is actually failing.
I'm confused you're saying that this is about , but you're updating . I thought they are about two different tests...
Yes sorry, I put my comment in the wrong issue. Reassigning this one to you, and I'll copy my comment into the other one.
First failure: We check if all namespace are deleted. If other tests failed to delete their namespace this test will fail. I'm going to delete this logic as we already check if we have successfully deleted our namespace. See also for some more details.
Second failure: We disable network by running the following command: I checked and it seems that it results in ing master to hang. I suspect that Kubelet hangs on updating node status and depending when it breaks the connection (by timeout) it will fit in test timeouts or not.
What is the default timeout for connections using client libraries to apiserver? I'm checking the code, but can't find it.
I'm not sure if we set one, but we should. I guess I would expect to see that in the config, w/ default set by this function: , Filip Grzadkowski wrote:
So: let's add a timeout, and revert .
I think that adding a timeout and reverting are orthogonal. I we want to simulate network problems with node-master communication then using REJECT is better anyway. At the same time adding timeout makes sense for many other reasons.


OK then, let's add a timeout and test both DROP and REJECT-- no sort of network problem should hang a component.
I think it didn't hang. It just that the kubelet was retrying too slow and sometimes it fit test timeout, sometime it didn't.
Could we reduce the timeout to make it always fit the test timeout? , Filip Grzadkowski
I guess we can. I'll try tomorrow.
It seems we have a new type of failure: It seems that kubelet on a machine where network is blocked doesn't notice new pod. After reading logs I assume that kubelet doesn't notice that the watch is broken and doesn't try to relist everything. It notices new pod only after watch timeout.
Ugh, I hope we don't have to start sending watch heartbeats.
It seems that all recent failures are with the following error: which means the reason is here: - can you please take a look or delegate?
I did some more debugging of it. I was looking into this failure: There is something strange happening here: run performTemporaryNetworkFailure: succesfully block the load from kubelet: the succesfully wait for node to become not ready: some time nodecontroller successfully removes the pod: replication controller creates a new pod: here is something I don't undertand - because of some reason, the test doesn't observe that the pod actually disappeared. - any thoughts on it?
BTW - this is exactly the same situation in:
OK - I think this is related to default 30s deletion grace period.
Yes - it seems that this is exactly the case. I will send out PR fixing this out for review.
I'm reopening it for tracking. If there won't be any failures within next few days - I will move the test back from flaky and close this issue.
Thanks for fixing, .
It seems that the main issue is fixed. However, there was one failure in the last 30 runs: in which one pod simply didn't started (and it was "preparation" for the test, not the test itself).
During next 2 days there was exactly one more failure of this test: it is exactly the same as the previous one. However, those are duplicates of So I'm going to move this test out of flaky suite.


There were two patches merged recently in the kernel that will significantly improve the IPVS proxier: The first patch fixes an issue where a packet is dropped whenever a stale conntrack entry is reused. This caused a 1s delay whenever the stale conntrack entry was used. This was the original reason for setting . More details in The second patch fixes an issue when a source port is reused from a given client and the corresponding destination was deleted. This is easily reproducible when a single client sends a lot traffic to a small set of destinations and one of the destination servers are removed. More details in Based on these two patches, we should start setting (default) starting on Linux v5.9 when those two patches make it to the main line. The new sysctls for the IPVS proxier would be: We should probably wait for v5.9 before making any changes to kube-proxy, this is just a tracking issue so we don't forget.
/sig network /area ipvs
Should probably hold on the PR until v5.9 is released later this year. cc
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Yes it makes sense to wait until a kernel with the patch exists ;) I think we can also avoid touching it and follow the approach "only touch the sysctl we absolutely need, otherwise trust the admin to know what they need"
I install linux-5.9-rc2 in centos7. set /proc/sys/net/ipv4/vs/connreusemode = 1 i have three service: service-a ---service-b --service-c in k8s. service-c return "helloworld" I use ab to benchmark service-a. [root ~]# ab -n -c 1000 http://172.17.2.66:/ This is ApacheBench, Version 2.3 <Copyright 1996 Adam Twiss, Zeus Technology Ltd, Licensed to The Apache Software Foundation, Benchmarking 172.17.2.66 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed requests Finished requests Server Software: Server Hostname: 172.


17.2.66 Server Port: Document Path: / Document Length: 11 bytes Concurrency Level: 1000 Time taken for tests: 4.128 seconds Complete requests: Failed requests: 0 Write errors: 0 Total transferred: bytes HTML transferred: bytes Requests per second: 2422.76 [#/sec] (mean) Time per request: 412.753 [ms] (mean) Time per request: 0.413 [ms] (mean, across all concurrent requests) Transfer rate: 302.84 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 16 71.5 4 1004 Processing: 4 216 92.9 202 2340 Waiting: 4 215 92.7 202 2340 Total: 4 232 121.5 215 2341 Percentage of the requests served within a certain time (ms) 50% 215 66% 253 75% 280 80% 299 90% 348 95% 412 98% 490 99% 556 100% 2341 (longest request) [root ~]# ab -n -c 1000 http://172.17.2.66:/ This is ApacheBench, Version 2.3 <Copyright 1996 Adam Twiss, Zeus Technology Ltd, Licensed to The Apache Software Foundation, Benchmarking 172.17.2.66 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed requests Finished requests Server Software: Server Hostname: 172.17.2.66 Server Port: Document Path: / Document Length: 11 bytes Concurrency Level: 1000 Time taken for tests: 7.071 seconds Complete requests: Failed requests: 0 Write errors: 0 Total transferred: bytes HTML transferred: bytes Requests per second: 1414.21 [#/sec] (mean) Time per request: 707.110 [ms] (mean) Time per request: 0.707 [ms] (mean, across all concurrent requests) Transfer rate: 176.


78 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 9 66.2 2 1019 Processing: 10 661 575.2 372 3603 Waiting: 1 659 573.7 372 3603 Total: 10 670 584.6 380 3607 Percentage of the requests served within a certain time (ms) 50% 380 66% 742 75% 1008 80% 1183 90% 1535 95% 1784 98% 2376 99% 2479 100% 3607 (longest request) [root ~]# ab -n -c 1000 http://172.17.2.66:/ This is ApacheBench, Version 2.3 <Copyright 1996 Adam Twiss, Zeus Technology Ltd, Licensed to The Apache Software Foundation, Benchmarking 172.17.2.66 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed requests Finished requests Server Software: Server Hostname: 172.17.2.66 Server Port: Document Path: / Document Length: 11 bytes Concurrency Level: 1000 Time taken for tests: 13.580 seconds Complete requests: Failed requests: 0 Write errors: 0 Total transferred: bytes HTML transferred: bytes Requests per second: 736.40 [#/sec] (mean) Time per request: 1357.958 [ms] (mean) Time per request: 1.358 [ms] (mean, across all concurrent requests) Transfer rate: 92.05 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 4 6.6 1 49 Processing: 11 1333 491.9 1307 3185 Waiting: 1 1330 491.9 1305 3185 Total: 17 1337 491.


9 1311 3191 Percentage of the requests served within a certain time (ms) 50% 1311 66% 1565 75% 1679 80% 1748 90% 1911 95% 2129 98% 2384 99% 2650 100% 3191 (longest request) [root ~]# ab -n -c 1000 http://172.17.2.66:/ This is ApacheBench, Version 2.3 <Copyright 1996 Adam Twiss, Zeus Technology Ltd, Licensed to The Apache Software Foundation, Benchmarking 172.17.2.66 (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed requests Finished requests Server Software: Server Hostname: 172.17.2.66 Server Port: Document Path: / Document Length: 11 bytes Concurrency Level: 1000 Time taken for tests: 23.720 seconds Complete requests: Failed requests: 936 (Connect: 0, Receive: 0, Length: 936, Exceptions: 0) Write errors: 0 Total transferred: bytes HTML transferred: bytes Requests per second: 421.58 [#/sec] (mean) Time per request: 2372.009 [ms] (mean) Time per request: 2.372 [ms] (mean, across all concurrent requests) Transfer rate: 47.77 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 5 7.5 1 49 Processing: 8 2345 3364.8 333 Waiting: 0 1744 3030.9 277 Total: 8 2350 3364.8 340 Percentage of the requests served within a certain time (ms) 50% 340 66% 1434 75% 3390 80% 5490 90% 8669 95% 9233 98% 99% 100% (longest request) the last time tcp connections of service-b :
/assign Sorry, but I do not understand what are you trying to say.


: (
/cc
Will this be implemented in 1.21?
/area kube-proxy
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --command fails because of I think the link is from k/k's Am I right?? kubetest command succeeds With installed, OR Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release /sig testing
/assign
Need cherry-picks for 1.21, 1.20 to address this. is updated but downloaded downloads and executes its /reopen
Reopened this issue. <detailsIn response to : Need cherry-picks for 1.21, 1.20 to address this. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
this was cherry picked AFAICT, thank you!


The function in accepts an parameter which it describes as a way to "controller specific flags to be included in the complete set of controller manager flags". However, when the usage string is shown (either via or when parsing options fails), these additional options are not included in the usage string. This was identified via code inspection while trying to figure out what options did in . For e.g. , the output of should include a section corresponding to . This section should in turn list all of . a cloud provider executable that includes additional options, such as the additional options are not shown in the output No response_ n/a (all cloud providers are affected) n/a n/a n/a n/a
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider


<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? static manifest Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Bare metal and reproduced in Vagrant OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: custom scratch install and reproduced with kubeadm Others: Calico for networking. Used (Also able to reproduce without calico) What happened: Modified one of the static manifest files in . The kubelet killed the running pod and did NOT create a new pod. (waited 30 minutes, still no pod) What you expected to happen: Kubelet should have killed the pod and started a new one. How to reproduce it (as minimally and precisely as possible): Started with single node, vanilla Ubuntu Xenial, installed kubernetes via kubeadm, nothing special. Modify , for example change the logging level or even just whitespace at the end of the file. Observe the kubelet kill the controller manager pod, and does not restart it. Anything else we need to know: If I remove the file from the directory, and then replace the file, kubelet will start it up. Kubelet logs from the period where I updated the file:
Did you modify the yaml file using an editor (e.g., vim)? If the editor creates a temporary file in the same directory, kubelet can get confused and thinking that's a new pod (with the same name & namespace). When the temporary file gets removed, kubelet'd inadvertently delete the pod from its cache. Since kubelet relies on inotify for changes, if it doesn't get notified for any new changes from the original yaml file, it'd forever ignore the pod. There is an issue open to add periodic listing in addition to inotify (introduced in 1.5). This would make sure kubelet eventually recover from the situation. I can't find the issue, but there is another one for inotify: . Another fix is to make sure kubelet ignore such known temporary files (it already ignores files starting with ). In general, don't edit the file directly.


Copy the file to another place, edit, and copy it back to the manifests directory.
I'm seeing a similar problem except I never edited the file, might be fit for a separate issue
Yes I was using vim, and this explanation makes sense. Would be nice if it was smart about it. Would a PR to ignore the vim temp file be acceptable
I think it's reasonable. I doubt if anyone uses those file names on purpose.
The dot ignore should have covered the vim temp file too, but it doesn't work because the filename passed in is the absolute filename including the path. So the first character is not a dot. I'll try to submit a PR fix.
PR submitted
fixed?
How to work around? I coudn't start api server when using kubeadm to deploy k8s. Thanks!


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Not able to change the default storage class. What you expected to happen: Default storage class to change. How to reproduce it (as minimally and precisely as possible): But after few mins. Anything else we need to know?: Tried to remove the annotation but no luck. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: minikube OS (e.g. from /etc/os-release): Buildroot 2018.05 Kernel (e.g. ): 4.16.14 Install tools: minikube Others:
/assign
Those labels are not set on the issue: <detailsIn response to : /remove-sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
The cause is that kube-addon-manager is reconciling the standard storage class.
Those labels are not set on the issue: <detailsIn response to : /remove-sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for looking into it. Sorry for native question, I want to keep only ceph as a default storage class how can I do that?
hi I'm not really sure about the actual command to fix this. Basically, you could log into the VM and do the same modification as the PR above does, then restart minikube.
Has the fix made it into minikube yet? I'm still seeing the issue in 0.31.0
Looking at PR , the fix was made in kubernetes core, not minikube. The error still exists in minikube:


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: kubectl top nodes didn't report one of my nodes. I have a cluster of 8 and only 7 printed.


<pre:::: fang 10:04:02 (cgthayer-pchat) 0 platform; kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-charles-test-cluster-default-pool--jkrj 311m 33% 2470Mi 93% gke-charles-test-cluster-default-pool--krsh 154m 16% 1763Mi 66% gke-charles-test-cluster-default-pool--vhgq 265m 28% 1954Mi 73% gke-charles-test-cluster-default-pool--x3bs 186m 19% 1990Mi 75% gke-charles-test-cluster-default-pool--073f 270m 28% 1729Mi 65% gke-charles-test-cluster-default-pool--09zf 403m 42% 2698Mi 102% gke-charles-test-cluster-default-pool--p7h1 191m 20% 1582Mi 59% :::: fang 10:04:07 (cgthayer-pchat) 0 platform; kubectl get nodes NAME STATUS ROLES AGE VERSION gke-charles-test-cluster-default-pool--073f Ready <none7d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--09zf Ready <none7d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--cl65 Ready <none6d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--jkrj Ready <none8d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--krsh Ready <none8d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--p7h1 Ready <none7d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--vhgq Ready <none8d v1.7.12-gke.1 gke-charles-test-cluster-default-pool--x3bs Ready <none8d v1.7.12-gke.1 <preWhat you expected to happen: I expected all nodes to be printed.


Since the missing node might have bad data I might expect it to have the same number of columns with "-" values and a note in a new 6th column saying something like "unavailabe" "unresponsive" "stale-data" etc. e.g. <pregke-charles-test-cluster-default-pool--p7h1 - - - - unreachable </preHow to reproduce it (as minimally and precisely as possible): Not sure. Perhaps run a pod where you can intentionally trigger it to use too much cpu or memory. Anything else we need to know?: Environment: Kubernetes version (use ): kubectl version Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-12-21T06:34:11Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.8-gke.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-16T18:26:58Z", GoVersion:"go1.8.3b4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind feature
I raise a PR to print when the metrics is missing. As for the 6th column you mentioned, I think it could be debatable for its necessity. :)


See Jenkins job kubernetes-upgrade-1.0-1.1-gke-step3-e2e-old's last few runs. The relevant line is . Note that this is running e2e tests off of the branch. It's possible that this is also caused by , but it's not clear to me.
This is also failing on kubernetes-upgrade-1.0-1.1-gke-step5-e2e-old, where we're running tests against a cluster that's been fully upgraded, (master and nodes,) from 1.0 to 1.1.
Status update: this may be caused by , so that issue should be prioritized, but it's not totally clear if that will fix this problem.
said he would take a look and update the issue with any findings. It may be a bit premature to look into it before is resolved, but at least we can try to figure out how to manually repro this now, so that we will know whether fixes this issue.
I'm unable to find logs from failed runs, which makes everything slightly harder. I'll try to reproduce the problem on GCE cluster, but it may not work. FYI: most of was caused by waiting for skydns to come back up (which doesn't happen: ). I see that there's a difference in number of system pods between version 1.1 and 1.0 - there's only 5 of them in 1.1 and 8 of them in 1.0. I'm wondering if this may cause problems in 'old' version of test which we're running there. I'm going to try reproduce it on GCE (as this is the setup I understand better) and if I'll fail I'll try again on GKE.
AFAICT there's a problem with the test, not the upgrade itself. The test from 1.1 branch fails on 1.0 cluster.
It's caused by . Can we please stop renaming things? AFAICT the problem is with test only, everything should be working 'on production'. In v1.0 scheduler Events reasons start with a small letter (e.g. failedScheduling) and v1.0 test is looking for those kind of events to occur. In v1.1 we changed Event reasons to start with a capital letter (e.g.


FailedScheduling) and modified test accordingly. Effects are obvious. We can: revert patch it to v1.0 merge and patch a test fix which adds handling both versions () I'm for the last option, but whatever we do (except reverting) will break backward compatibility for everyone depending/counting/filtering/whatnot on scheduler Event reasons
thanks for your work on this. From on : Furthermore, in my (rookie) opinion, seems like it should be considered a bug fix rather than an API change. Thus I think makes good sense.
Thanks a lot for debugging this, and I agree we should go with your PR. I talked to Brian and he said we don't consider changes to values of fields in events to be breaking API changes. Unfortunately people do seem to depend on these in various places. One solution is what I proposed in so the compiler could prevent these incompatibilities (at least for components and tests that are compiled together).
+1 to creating a file containing constants for the event reasons that are currently in use.
I'll prepare a cherry pick.
Reopening until cherry pick is merged.
Cherry pick is .
The test mentioned in the first entry in this thread is still failing but not due to the error message related to this issue. Since the cherrypick has been merged, can we close this issue, and followup with additional debugging in ?
Works for me. Thanks to again for your work on this.
I actually still see failures of the same kind in (master 1.1, nodes 1.0) and (master 1.1, nodes 1.1). I assume this is because we haven't cherrypicked test fix into . So this is now a duplicate of .


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: kube-controller-manager should not panic. How to reproduce it (as minimally and precisely as possible): Kubernetes 1.12.6, , and Anything else we need to know?: It's running and , and logs says: Don't see any notable errors in . Is that because it's running the old ? Basically, is expecting , but somehow it's getting which I assume Environment: Kubernetes version (use ): v1.12.6 Cloud provider or hardware configuration: AWS EC2 OS (e.g: ): Amazon Linux 2 Kernel (e.g. ): 4.14.109-99.92.amzn2.x86_64
Could be related to , but this happens with all the patches from the issue /sig scalability /cc
/sig autoscaling /assign can you triage please?
/sig-remove scalabilty
/sig remove scalability
/sig scalability remove
/remove-sig scalability
API returned v1.Status instead of v1beta2.MetricValueList. We should check what is the type of the object
/sig instrumention
/assign
/sig instrumentation
seems promising. Is it missing anything?
I've root-caused the issue. Explained here -
Yep, should address the issue.


<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: should function for node-pod communication: udp Since when has it been failing: Unknown. Testgrid link: The test is currently excluded. Reason for failure: By default, cmd is used to execute commands in Windows containers, but it has some issues executing the command (no output): echo 'hostName' | nc -w 1 -u <targetIP<targetPortBut it has no issues running (output is correct): echo hostName | nc -w 1 -u <targetIP<targetPortThis causes the test "should function for node-pod communication: udp" to fail when using Windows Containers. Anything else we need to know: /sig testing /sig windows


<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: In some files, i found that there are many self-contained function, whose feature is as same as some util function. Obviously, they are redundant. What you expected to happen: Use util function to replace self-contained function How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
similar comment to Can you edit the description in to say instead of ? Thanks!
Thanks. Done.


TrimLeft returns a slice of the string s with all leading Unicode code points contained in cutset removed. To remove a prefix, use TrimPrefix instead. /sig release /kind bug
/assign


<!-- Please only use this template for submitting enhancement requests --What would you like to be : Update the help text for kubelet's flag to explicitly mention that it has no effect over the endpoint. Why is this needed: The help text for kubelet's flag describes it as "Enable cAdvisor json and endpoints." which would seem to include , however this is not the case as that endpoint is required by kube-metrics-server and exposes stats from cadvisor or CRI (depending on the stat provider selected internally). Related Issues: /sig node /sig instrumentation /kind documentation
/kind documentation /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /kind documentation /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign This should be updated in as well as (unless there is some automated process that updates the website from the options in code that I'm not aware of).
To summarize a conversation in slack, it appears there is a documented procedure for updating the website found at The procedure is long, so it is tempting to just copy in the small amount of text that was changed from into a website PR rather than pulling down so many moving parts. In any case, we likely should keep this issue open until both the change to and an associated website change are merged.
can you link to the Slack convo?
Can do - The conversation ended with a question whether there is a bug that would prevent an automated update of the documentation or if we need to submit a PR now that the fix has merged. I searched for an issue but didn't find a match. So we can either try to find someone on sig-docs who might know if there is an issue if it should be updated automagically, or just fix the doc if the website didn't get updated (and I just checked the master branch and it has not been updated yet).
Just in case that is the right answer, I created the PR.
I think the website gets autogenerated upon point releases but that'd be a good thing to clarify with the website team. I'll leave a comment asking as much.


There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
probably caused by
Could be a clash between these two?
cc
I still see the issue with local-cluster on the latest master: F0225 17:06:49. ] error: v1alpha1.KubeProxyConfiguration: FeatureGates: ReadMapCB: expect { or n, but found ", error found in byte of ......
looks like that's another issue, can you please file a new bug? with more logs? this one was about none of the hyperkube based processes starting up.
will do as it is also broke container kube-proxy as well. Same type of error.


The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the sample-controller repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the sample-apiserver repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the kube-aggregator repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the code-generator repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area code-generation /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery


The purpose of the file for each Kubernetes repository is to provide a list of people who can assist the Kubernetes in the event that a security issue related to the repository is discovered or disclosed. As described in the file, those on the list should agree to our . Please update the  file for the client-go repository. After finding people who are willing to work in this capacity, you should add them to the list, then remove PSC members (except any PSC member will be working as a security contact for this repository). The list is GitHub usernames, optionally followed by an email address. If no email address is listed, the PSC will use the email address found on git commits made by the listed user. The file may already have people listed who are secuirty contacts. In that case, simply remove any PSC members who aren't also security contacts for the repo. See for more information /area security /committee product-security /kind cleanup /lifecycle frozen /priority important-soon /sig api-machinery