Question: "--serialize-image-pulls", "false", "--pod-manifest-path", manifestPath, "--file-check-frequency", "10s", // Check file frequently so tests won't wait too long <ins> "--docker-disable-shared-pid=false", </ins> // Assign a fixed CIDR to the node because there is no node controller. // // Note: this MUST be in sync with with the IP in

cc/
gluster/gluster-centos is a container image that runs systemd so that it can run ntpd, crond, gssproxy, glusterd & sshd inside a single container. This isn't how Kubernetes is intended to be used and I don't know how much we should go out of our way to support it (this is a question for & seems to be a good write up of the challenges of running systemd inside of docker. It also prevents Kubernetes from doing process management like: reporting state of processes signaling individual processes for graceful restart ingesting and display daemon output (to stdout/err) Ideally we could change this many-processes-in-a-container to be many-containers-in-a-pod, something like this: This doesn't need a privileged container and doesn't run sshd or ntpd, neither of which are needed in Kubernetes. This also doesn't run crond, but that may actually be needed by gluster. I've never used gluster. (I used flags so I didn't have to investigate how to make these processes log to stdout) If you must run systemd in a container, adding this to the container config will get systemd to run: This tells systemd to run in system mode even though it's not PID 1 and disables its chroot detection (which is unrelated to PID 1 but checked when invoked as "systemd", I guess).
afaict, running in a container is well recommended ( and ) at certain scenarios and a common practice. iic, there are enough containers who make use of as its . If shared namespace which introduced in 1.7 breaks this, I believe its a backward compatibility break :(. I would invite here to share his thoughts as well. :)
Those articles are about how to work around shortcomings in vanilla docker that Kubernetes solves natively: resource sharing via pods, containing via SELinux, reaping orphaned zombies, etc. My earlier question was along the lines of how much effort should go into supporting use cases that are not using Kubernetes as intended. (I'm actually asking, not trying to make a point.) Does Kubernetes commit to continuing support for every container image that's ever run on Kubernetes? For how long? Indefinitely? or can we change it over the course of multiple releases? Anyway, this change is compatible with running an init system in a container.


Question: "--serialize-image-pulls", "false", "--pod-manifest-path", manifestPath, "--file-check-frequency", "10s", // Check file frequently so tests won't wait too long <ins> "--docker-disable-shared-pid=false", </ins> // Assign a fixed CIDR to the node because there is no node controller. // // Note: this MUST be in sync with with the IP in

The problem is with images that assume they will only ever have , which has so far been the case with docker due to a technical limitation, but it was never the case with other runtimes such as rkt, hyperv, etc. Brainstorming other options: shared pid namespace and require users of systemd to set an environment variable in their Kubernetes config (or port the multi-process container to a multi-container pod). shared pid namespace and enable it in a future release where we have the same problem. a solution where pid namespace is configurable per-pod, but not supported indefinitely. It would probably be implemented using annotations, and we will have to modify the Container Runtime Interface to support it. Over the course of multiple releases the default behavior would change but could be reverted based on the presence of an annotation. In a future release, users will have to change an annotation in their config to get the old behavior (rather than the environment variable). In a future future release, the annotation will stop working and users will have to set the environment variable. to supporting per-pod shared vs isolated pid namespaces indefinitely and adding it to the Kubernetes API. It doesn't make sense in some runtimes, so we probably can't require all runtimes to implement it, so it'll likely only see use with Docker. Last time we discussed it in SIG Node, and didn't want to support isolated pid namespaces indefinitely.
Yes running systemd in a container is a critical need for us. There are lots of situations where it is and will be used. Biggest use case if for people simply moving current workloads that run in a VM into a container. I have a service that runs fun under systemd in a VM. If I can simply take the init script and move it whole sale into a container, why wouldn't I want to do this. It's simple and does require a great deal of knowledge by the end user. service containers. Not every project can easily be moved into microservices. But they probably can run inside of containers. If we drop support for systemd running inside of containers, then we loose a big group of users. Getting access to journald information inside of the container. Applications that write to syslog and directly to the journal. Currently in container world without systemd, this information is dropped on the floor.


Question: "--serialize-image-pulls", "false", "--pod-manifest-path", manifestPath, "--file-check-frequency", "10s", // Check file frequently so tests won't wait too long <ins> "--docker-disable-shared-pid=false", </ins> // Assign a fixed CIDR to the node because there is no node controller. // // Note: this MUST be in sync with with the IP in

if I run systemd and journald, I can actually retrieve and use this information.
If the intension is to force all containers inside of a POD to run with the same PID Namespace? I think this is a mistake since it eliminates another really cool use case. I would love to be able to run two containers inside of a pod, and INIT container with escallated privileges and a locked down container, with no privileges. I want to make sure the locked down container can not see the processes of the privileged container. An example of this, might be a container that loads a kernel module that is needed by the locked down container. Another use case would be a buildah container. The INIT container mounts up a COW file system onto a directory on a location that the locked down container can write, The locked down container writes out the data for his image, when the container completes, the INIT container commits the data to the repo. I believe allowing different containers inside of a POD to run with different security and isolated PIDs should not be removed.
none of the things you've mentioned are removed by this change, save using docker to hide other processes in the pod, and I'm not sure why that's a requirement. Is anyone currently using the "privileged sidecar" pattern you describe or is it hypothetical? It just occurred to me that an interesting compromise here might be to only enable shared pid for multi-container pods. This would enable pod semantics for PID when it would make a difference while drop-in docker monoliths keep the PID 1 behavior they're expecting in docker. I'd have to double check, but I'm pretty sure that behavior could be implemented entirely in the dockershim with no API or CRI changes.
Hypothetical, but I am pushing it for a solution for problems I hear from users fairly frequently, where then need big privileges at start and stop of the pod, but not for the time is is available to the network.
ok, thanks. ftr, I think we should continue to support this pattern as well, and I don't think a shared PID namespace prohibits it. I will bring up this issue at Tuesday's sig node meetup if you'd like to join us. Does the config I mentioned in fix the issue with systemd?


Question: "--serialize-image-pulls", "false", "--pod-manifest-path", manifestPath, "--file-check-frequency", "10s", // Check file frequently so tests won't wait too long <ins> "--docker-disable-shared-pid=false", </ins> // Assign a fixed CIDR to the node because there is no node controller. // // Note: this MUST be in sync with with the IP in

Copying from a comment made elsewhere:
Update from SIG Node Meetup: Everyone wants to support these use cases, but we're not yet sure the requirements we want to support medium to long term. is going to sync up with and then we'll figure out what we want to support in a future release. For 1.8 the current plan is to leave docker shared pid namespace enabled by default. Cluster administrators can disable it per-node with the --docker-disable-shared-pid flag. I'll see about whether we should add something back to the release notes.
This seems somewhat backwards. Wouldn't it make more sense to go back to supporting systemd use cases by default NOW (by making the shared PID namespace behavior optional or something) until a final decision is reached and can be advertised to current users?
absolutely ! thank you very much
I think that is the best solution is to make the PID Namespace sharing optional.
- optional at what scope - cluster, node, pod. let's sync up before next sig-node and we can revisit if need be before 1.8 goes out. The main question right now is if pid namespace sharing is opt-in or opt-out. We need to weigh pro/cons of both.
I would expect it at the POD, since pods would have different requirements, not sure why you would want to have this at the cluster or node level.
yes it works :)
awesome, thanks! I'd like to better understand the requirements of people objecting to this change ( I think there's a bit of confusion about what has changed, so let's start with some facts: This container image runs on Kubernetes without changes to the image, but it requires (non-obvious) pod configuration. is nothing inherently incompatible between a shared pid namespace and a process manager like systemd. systemd and other init systems work with pid != 1, but may require special invocation. in this issue is specific to Kubernetes. That is, this container image wouldn't work with vanilla docker PID namespace sharing, either. change does not affect securityContext I can think of a few things that are objectionable about the change, but let's state them as requirements. For example, are these requirements for anyone? Kubernetes must run all containers published on docker hub with no additional pod configuration Kubernetes must run all systemd containers, regardless of how they are built, with zero additional pod configuration.


Question: "--serialize-image-pulls", "false", "--pod-manifest-path", manifestPath, "--file-check-frequency", "10s", // Check file frequently so tests won't wait too long <ins> "--docker-disable-shared-pid=false", </ins> // Assign a fixed CIDR to the node because there is no node controller. // // Note: this MUST be in sync with with the IP in

Kubernetes must run this specific container image with zero additional configuration Kubernetes must officially support isolated PID namespaces between containers in a single pod across all runtimes (docker, rkt, hyperv, etc). Configuration changes are ok, but the flag to systemd cannot be used because the documentation says its not useful. I want to use systemd as intended. Rather than having Kubernetes work around this container image, should we patch the Dockerfile to make it compatible with even the native docker shared-pid, unrelated to Kubernetes?
Configuration changes are ok, but the --system flag to systemd cannot be used because the documentation says its not useful. I want to use systemd as intended. We can ask the systemd guys why they recommend never using the --system flag, but I would guess one of the reasons is that processes showing up that were not created in the ancestry of systemd, could cause confusion. I would also guess that running more then one container with systemd --system would not be supported. Kubernetes must officially support isolated PID namespaces between containers in a single pod. This is required so that you can run different containers with different security constraints in the same pod at the same time. Your last comment makes the seem like it is the default, when actually it is the exception. almost no containers run with shared-pid normally. I guess in a side car container, it makes sense to support this scenario, but forcing this as the ONLY way to run containers is limiting.
This is the part I don't understand. Why? What does PID visibility have to do with security constraints? What am I missing? nor do they run with a shared network namespace, but we impose that on docker containers because that's the pod model.
An update from the last SIG Node: we'll disable this by default in 1.8 and revisit for 1.9 I'll use to track that.
if I have a privileged container process in the same pid namespace as a pid in a non priv container, then the container process in the non priv can attack the priv process. Having them in separate pid namespaces gives you better security. Bottom line, you have a potential change where containers that expect to be run as PID 1, whether legitimately or not, could break when you make this change. Kubernetes in my opinion should make it easy to opt out of this behavior.


Question: "--serialize-image-pulls", "false", "--pod-manifest-path", manifestPath, "--file-check-frequency", "10s", // Check file frequently so tests won't wait too long <ins> "--docker-disable-shared-pid=false", </ins> // Assign a fixed CIDR to the node because there is no node controller. // // Note: this MUST be in sync with with the IP in

/priority critical-urgent
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details
Update: is pending review and should be to the v1.8 Milestone to resolve this issue.


Question: "description": "Endpoint represents a single logical "backend" implementing a service.", "properties": { "addresses": { <del> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "items": { "default": "", "type": "string"

An can contain up to 1000 , which can each contain up to 100 . Or at least, that's what the docs say, but actually, kube-proxy (or more specifically, ) only looks at the first Address in each Endpoint: (This is compatible with how works, since it would only generate multi-Address Endpoints if a pod had multiple of the same address family, which is not allowed. But it's not compatible with manually-generated EndpointSlices.) Either all are used, or else the docs state that only is used and the field is an array solely for historical reasons.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /priority important-longterm /cc
All addresses in the slice must be of the same type, do you mean creating an Slice with different address types in the Addresses field?
I'm not sure if/how kube-proxy should handle an endpoint with more than 1 address. I think the goal of kube-proxy is to give each endpoint an equivalent probability of being selected. To extend that logic here, I think we'd need to take the probability assigned to an endpoint and divide it by the number of addresses attached to that endpoint. So if an endpoint has a 50% probability of being selected, and it provides 50 addresses, each should end up with a 1% probability of being selected. Of course with the way we need to do descending probabilities, the logic may get a bit complicated. This isn't as clear as it should be, but I don't think kube-proxy or any consumer of the EndpointSlice API needs to commit to supporting every value specified by that API. For example, many may filter by address type, Service, or some other means. I think it's also valid for a consumer to say they only support a single address per family per endpoint. So all of that to say, I'd support either documenting that kube-proxy only selects the first address in an endpoint, or updating kube-proxy to support more than one address per endpoint.


Question: "description": "Endpoint represents a single logical "backend" implementing a service.", "properties": { "addresses": { <del> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "items": { "default": "", "type": "string"

Given the number of changes currently in flight for kube-proxy, documentation seems like a safer initial approach to me.
/assign
can I have this editing only the first address(Address[0]) in Endpoint.Address to be used. The use of an array in this field is based on historical reasons. Isn't that right
thanks for volunteering to help with this one! I know has some big kube-proxy changes/fixes in flight right now, so it would be good to be sure you don't conflict with him. Otherwise, as long as isn't already working on this, I think you're free to have it.
I don't think we agreed that the array is for historical reasons only. We aren't currently using it, but we aren't ready to say that we won't use it in the future at this point. So, the docs should warn that kube-proxy currently ignores additional addresses, but it shouldn't imply that it's actually wrong to have additional addresses I'm not
thanks i will work on this
It was for a reason, but I think that reason was mixed-family slices (before we decided to break it up into single-family slices). What would it mean to have multiple addresses here? Are they assumed to be equivalent? Or do they have different context (like a split-horizon)? If the latter, there isn't any metadata here (i.e. it's not a struct like to allow new fields). I am inclined to believe it was meant as equivalent, and as such it's probably OK to document that assumption clearly. That said, I don't see how anyone could use multiple addresses in a meaningful way - they are either assumed equivalent or they lack enough metadata to distinguish their differences. ?? If we do document the equivalence, it should be generically worded. E.g. "These addresses are assumed to be equivalent and some consumers may choose to ignore all but the first value."
if it was meant as equivalent, why put multiple same addresses into Addresses? I don't understand what the purpose of this is.
The first version of the API had both IP families in a single slice, so it made sense to have multiple addresses. We changed that, though we didn't immediately remove the old support. So my best guess is that we should have converted this to a singular value when we did that, but we missed it.


Question: "description": "Endpoint represents a single logical "backend" implementing a service.", "properties": { "addresses": { <del> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "items": { "default": "", "type": "string"

On Wed, Nov 24, 2021, 5:45 PM Cyclinder
so we should change the API doc? Or It's just that documenting it will only use the first address in the , like you said above
I'd like some of the others to comment, but it's this a US holiday, so it might be a few days..
So maybe we should just declare that the array is essentially a bug, and only the first address is used? Maybe something should warn if you ever set additional addresses? (FTR, the context that I noticed this in was that I was adding unit tests, and was creating EndpointSlices with a single Endpoint and multiple Addresses, which doesn't work; you need multiple Endpoints with a single Address each.)
I think the simplest and most compat approach is to say "these are all assumed to be fungible and clients may choose to only use the first element". Thoughts?
I think the simplest and most compat approach is to say "these are all assumed to be fungible and clients may choose to only use the first element". Big +1 to that approach. I don't think it's necessarily invalid to set multiple addresses here, there may actually be some use cases for it, but adding this kind of statement should help clear things up significantly.
hmm as we push for multi homed pod this will make a lot of sense. There might also be a chance for community to provide a drop-in replacement for proxy that does much more than we do now. I think we should update the docs and don't modify anything for now.
Note that this API doesn't include per-IP metadata, so a multi-homed pod would be challenging with this mechanic. I think it's maybe time to reconsider multi-net more holistically. On Mon, Dec 13, 2021, 10:24 AM Khaled Henidak (Kal) <
:thinking:
If they're fungible, then "clients may choose to use only one of the elements" (or "only a subset"?). No reason to call out the first one specifically.
That's fair On Tue, Dec 14, 2021, 6:09 AM Dan Winship
I would like to add to this issue that it would makes sense to have kube-proxy reduce the subset of endpoints in the endpointslice based on the label of the node and the & attributes of the endpoint.


Question: "description": "Endpoint represents a single logical "backend" implementing a service.", "properties": { "addresses": { <del> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "description": "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "items": { "default": "", "type": "string"

Taking only into consideration the first endpoint which is in the same zone of the node or has no zone dependency and is ready for serving. & Would it make sense to reopen this issue?
I don't follow entirely what you are suggesting, can you elaborate?
kube-proxy should understand which zone a node is in and prefer endpoints in the same zone.
but that is unrelated to this issue, topology is considered See KEPs related, topology-aware-routing was replaced by topology-aware-hints
thank you for clarifying. It seems ambiguous to me to have a zone attribute and a hint attribute. I fail to implement the hinting on a service in v1.23.8, but this is urelated to this issue.
heh, don't worry, this topic is complex and we went through different solutions, you have more historical context on the KEPs I linked above and this thread in the mailing list


Question: // on port 443 // If the Service is nil, that means the handling for the API groupversion is handled locally on this server. // The call will simply delegate to the normal handler chain to be fulfilled. <ins> // +optional </ins> Service *ServiceReference // Group is the API group name this server hosts Group string

/sig api-machinery /kind bug /cc this makes any updates on a "Local" apiservice failing by kubectl (unless you disable the client-side validation). a quick repro: (edit whatever a local apiservice) a label "foo: bar" will fail on the schema validation # "" was not valid: # * : Invalid value: "The edited file failed validation": ValidationError(): missing required field "service" in client-side v1.15.0 server-side v1.15.2 referencing the published openapi v2 for apiservice#spec below:
/label help-wanted
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can on this if no one else is currently working on it.
sure, feel free to tag me to request reviewing..
awesome, I will /assign
are you still working on this one?


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package kubelet import ( "testing" "time" "github.com/golang/groupcache/lru" ) func TestTimeCache(t *testing.T) { cache := &timeCache{cache: lru.New(2)} if a, ok := cache.Get("123"); ok { t.Errorf("expected cache miss, got %v, %v", a, ok) } now := time.Now() soon := now.Add(time.Minute) cache.Add("now", now) cache.Add("soon", soon) if a, ok := cache.Get("now"); !ok || !a.Equal(now) { t.Errorf("expected cache hit matching %v, got %v, %v", now, a, ok) } if a, ok := cache.Get("soon"); !ok || !a.Equal(soon) { t.Errorf("expected cache hit matching %v, got %v, %v", soon, a, ok) } then := now.Add(-time.Minute) cache.Add("then", then) if a, ok := cache.Get("now"); ok { t.Errorf("expected cache miss from oldest evicted value, got %v, %v", a, ok) } if a, ok := cache.Get("soon"); !ok || !a.Equal(soon) { t.Errorf("expected cache hit matching %v, got %v, %v", soon, a, ok) } if a, ok := cache.Get("then"); !ok || !a.Equal(then) { t.Errorf("expected cache hit matching %v, got %v, %v", then, a, ok) } } </ins>

These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions


Question: func (kl *Kubelet) generateAPIPodStatus(pod *v1.Pod, podStatus *kubecontainer.PodStatus) v1.PodStatus { glog.V(3).Infof("Generating status for %q", format.Pod(pod)) <ins> s := kl.convertStatusToAPIStatus(pod, podStatus) </ins> // check if an internal module has requested the pod is evicted. for _, podSyncHandler := range kl.PodSyncHandlers { if result := podSyncHandler.ShouldEvict(pod); result.Evict { <del> return v1.PodStatus{ Phase:   v1.PodFailed, Reason:  result.Reason, Message: result.Message, } </del> <ins> s.Phase = v1.PodFailed s.Reason = result.Reason s.Message = result.Message return *s </ins> } } <del> s := kl.convertStatusToAPIStatus(pod, podStatus) </del> // Assume info is ready to process spec := &pod.Spec allStatus := append(append([]v1.ContainerStatus{}, s.ContainerStatuses...), s.InitContainerStatuses...)

Was running a pod with an activeDeadlineSeconds on a 1.10 cluster. The pod reached the deadline, but the pod status did not contain the latest state of the pod containers (pod containersStatus was empty). The status only had the following fields: The statuses were there immediately prior to the deadline being reached. This broke API clients that were looking at container status.
Looks like the status is just overwritten here if returns true
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: close(stopCh) } <ins> // TestEventSeriesWithEventSinkImplRace verifies that when Events are emitted to // an EventSink consecutively there is no data race.  This test is meant to be // run with the `-race` option. func TestEventSeriesWithEventSinkImplRace(t *testing.T) { kubeClient := fake.NewSimpleClientset() eventSink := &EventSinkImpl{Interface: kubeClient.EventsV1()} eventBroadcaster := NewBroadcaster(eventSink) stopCh := make(chan struct{}) eventBroadcaster.StartRecordingToSink(stopCh) recorder := eventBroadcaster.NewRecorder(scheme.Scheme, "test") recorder.Eventf(&v1.ObjectReference{}, nil, v1.EventTypeNormal, "reason", "action", "", "") recorder.Eventf(&v1.ObjectReference{}, nil, v1.EventTypeNormal, "reason", "action", "", "") err := wait.PollImmediate(100*time.Millisecond, 5*time.Second, func() (done bool, err error) { events, err := kubeClient.EventsV1().Events(metav1.NamespaceDefault).List(context.TODO(), metav1.ListOptions{}) if err != nil { return false, err } if len(events.Items) != 1 { return false, nil } if events.Items[0].Series == nil { return false, nil } return true, nil }) if err != nil { t.Fatal("expected that 2 identical Eventf calls would result in the creation of an Event with a Serie") } } </ins> func validateEvent(messagePrefix string, expectedUpdate bool, actualEvent *eventsv1.Event, expectedEvent *eventsv1.Event, t *testing.T) { recvEvent := *actualEvent

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I wrote a test creating multiple isomorphic Events with an "". and stumbled upon a data race. The data race happens in the function. It seems that the Event Series field can be read and written at the same time by two different goroutines. . This seems to happen in only one scenario: (1) a singleton Event is created (2) an isomorphic Event is created (1) lock addition to cache (2) wait for the lock to be released (1) unlock (2) lock + the Series (1) the freshly created Series (1) singleton Event is now considered as a series That being said, this seems to lead to some data inconsistencies as the singleton Event would be created with an unexpected Series. <details<summaryData race</summary|================== WARNING: DATA RACE Write at 0x00c0001237a0 by goroutine 23: (eventBroadcasterImpl).recordToSink.func1.1() +0x536 (eventBroadcasterImpl).recordToSink.func1() +0x84 Previous read at 0x00c0001237a0 by goroutine 24: (Event).DeepCopyInto() +0x64 (Event).DeepCopy() +0x56 (Event).DeepCopyObject() +0x33 () +0xe8 (CreateActionImpl).DeepCopy() <autogenerated:1 +0xb7 (Fake).Invokes() +0x103 (FakeEvents).Create() +0x214 (EventSinkImpl).Create() +0x195 () +0x32c (eventBroadcasterImpl).attemptRecording() +0xe4 (eventBroadcasterImpl).recordToSink.func1() +0x9d Goroutine 23 (running) created at: (eventBroadcasterImpl).recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (eventBroadcasterImpl).StartEventWatcher.func2() +0x91 Goroutine 24 (running) created at: (eventBroadcasterImpl).


Question: close(stopCh) } <ins> // TestEventSeriesWithEventSinkImplRace verifies that when Events are emitted to // an EventSink consecutively there is no data race.  This test is meant to be // run with the `-race` option. func TestEventSeriesWithEventSinkImplRace(t *testing.T) { kubeClient := fake.NewSimpleClientset() eventSink := &EventSinkImpl{Interface: kubeClient.EventsV1()} eventBroadcaster := NewBroadcaster(eventSink) stopCh := make(chan struct{}) eventBroadcaster.StartRecordingToSink(stopCh) recorder := eventBroadcaster.NewRecorder(scheme.Scheme, "test") recorder.Eventf(&v1.ObjectReference{}, nil, v1.EventTypeNormal, "reason", "action", "", "") recorder.Eventf(&v1.ObjectReference{}, nil, v1.EventTypeNormal, "reason", "action", "", "") err := wait.PollImmediate(100*time.Millisecond, 5*time.Second, func() (done bool, err error) { events, err := kubeClient.EventsV1().Events(metav1.NamespaceDefault).List(context.TODO(), metav1.ListOptions{}) if err != nil { return false, err } if len(events.Items) != 1 { return false, nil } if events.Items[0].Series == nil { return false, nil } return true, nil }) if err != nil { t.Fatal("expected that 2 identical Eventf calls would result in the creation of an Event with a Serie") } } </ins> func validateEvent(messagePrefix string, expectedUpdate bool, actualEvent *eventsv1.Event, expectedEvent *eventsv1.Event, t *testing.T) { recvEvent := *actualEvent

recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (eventBroadcasterImpl).StartEventWatcher.func2() +0x91 |================== |================== WARNING: DATA RACE Read at 0x00c0001236f0 by goroutine 23: (Event).DeepCopyInto() +0x64 (Event).DeepCopy() +0x72 () +0x4d () +0x98 (eventBroadcasterImpl).attemptRecording() +0xe4 (eventBroadcasterImpl).recordToSink.func1() +0x9d Previous write at 0x00c0001236f0 by goroutine 24: () +0x2e4 (eventBroadcasterImpl).attemptRecording() +0xe4 (eventBroadcasterImpl).recordToSink.func1() +0x9d Goroutine 23 (running) created at: (eventBroadcasterImpl).recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (eventBroadcasterImpl).StartEventWatcher.func2() +0x91 Goroutine 24 (running) created at: (eventBroadcasterImpl).recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (*eventBroadcasterImpl).StartEventWatcher.func2() +0x91 |================== </detailsWhat you expected to happen: No data race. How to reproduce it (as minimally and precisely as possible): You can run the following code snippet alongside any data race detector. I personally used . Anything else we need to know?: Environment: v0.25.0
/sig api-machinery
/assign Would you mind taking a look to this one? thanks!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity.


Question: close(stopCh) } <ins> // TestEventSeriesWithEventSinkImplRace verifies that when Events are emitted to // an EventSink consecutively there is no data race.  This test is meant to be // run with the `-race` option. func TestEventSeriesWithEventSinkImplRace(t *testing.T) { kubeClient := fake.NewSimpleClientset() eventSink := &EventSinkImpl{Interface: kubeClient.EventsV1()} eventBroadcaster := NewBroadcaster(eventSink) stopCh := make(chan struct{}) eventBroadcaster.StartRecordingToSink(stopCh) recorder := eventBroadcaster.NewRecorder(scheme.Scheme, "test") recorder.Eventf(&v1.ObjectReference{}, nil, v1.EventTypeNormal, "reason", "action", "", "") recorder.Eventf(&v1.ObjectReference{}, nil, v1.EventTypeNormal, "reason", "action", "", "") err := wait.PollImmediate(100*time.Millisecond, 5*time.Second, func() (done bool, err error) { events, err := kubeClient.EventsV1().Events(metav1.NamespaceDefault).List(context.TODO(), metav1.ListOptions{}) if err != nil { return false, err } if len(events.Items) != 1 { return false, nil } if events.Items[0].Series == nil { return false, nil } return true, nil }) if err != nil { t.Fatal("expected that 2 identical Eventf calls would result in the creation of an Event with a Serie") } } </ins> func validateEvent(messagePrefix string, expectedUpdate bool, actualEvent *eventsv1.Event, expectedEvent *eventsv1.Event, t *testing.T) { recvEvent := *actualEvent

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We might also want to look into that again for /reopen /assign /unassign
/remove-lifecycle rotten
Reopened this issue. <detailsIn response to : We might also want to look into that again for /unassign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/cc
I was able to reproduce this issue with client-go v0.25.0. I updated the reproducer to be compatible with the latest version of the client and attached the data race trace.
are you gong to submit a fix for this one? thanks. /triage accepted
Yes, I am planning to submit a fix for this issue.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: close(stopCh) } <ins> // TestEventSeriesWithEventSinkImplRace verifies that when Events are emitted to // an EventSink consecutively there is no data race.  This test is meant to be // run with the `-race` option. func TestEventSeriesWithEventSinkImplRace(t *testing.T) { kubeClient := fake.NewSimpleClientset() eventSink := &EventSinkImpl{Interface: kubeClient.EventsV1()} eventBroadcaster := NewBroadcaster(eventSink) stopCh := make(chan struct{}) eventBroadcaster.StartRecordingToSink(stopCh) recorder := eventBroadcaster.NewRecorder(scheme.Scheme, "test") recorder.Eventf(&v1.ObjectReference{}, nil, v1.EventTypeNormal, "reason", "action", "", "") recorder.Eventf(&v1.ObjectReference{}, nil, v1.EventTypeNormal, "reason", "action", "", "") err := wait.PollImmediate(100*time.Millisecond, 5*time.Second, func() (done bool, err error) { events, err := kubeClient.EventsV1().Events(metav1.NamespaceDefault).List(context.TODO(), metav1.ListOptions{}) if err != nil { return false, err } if len(events.Items) != 1 { return false, nil } if events.Items[0].Series == nil { return false, nil } return true, nil }) if err != nil { t.Fatal("expected that 2 identical Eventf calls would result in the creation of an Event with a Serie") } } </ins> func validateEvent(messagePrefix string, expectedUpdate bool, actualEvent *eventsv1.Event, expectedEvent *eventsv1.Event, t *testing.T) { recvEvent := *actualEvent

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
A PR is out to fix this issue: /remove-lifecycle stale


Question: // Check iptables chain and rules epIpt := netlinktest.ExpectedIptablesChain{ string(kubeServicesChain): {{ <del> JumpChain: string(KubeNodePortChain), MatchSet: kubeNodePortSetTCP, </del> <ins> JumpChain: string(KubeNodePortChain), MatchSet: "", </ins> }}, string(KubeNodePortChain): {{ JumpChain: "RETURN", MatchSet: kubeNodePortLocalSetTCP, }, { <del> JumpChain: string(KubeMarkMasqChain), MatchSet: "", </del> <ins> JumpChain: string(KubeMarkMasqChain), MatchSet: kubeNodePortSetTCP, </ins> }}, } checkIptables(t, ipt, epIpt)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /assign I'll make the pull request tomorrow. What happened: start kube-proxy in ipvs mode, create some NodePort services. Then look at the iptables rules. match-set of KUBE-NODE-PORT-TCP is inadequate to be qualified as NodePort traffic. "--dst-type LOCAL" is needed. What you expected to happen: since we have also KUBE-NODE-PORT-UDP, another way would be like this: How to reproduce it (as minimally and precisely as possible): start kube-proxy in ipvs mode, create some NodePort services. Then look at the iptables rules. Anything else we need to know?: Environment: Kubernetes version (use ): commit Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS Linux release 7.2.1511 (Core) Kernel (e.g. ): Linux VM34centos 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
/sig network /area kube-proxy /area ipvs /assign
/cc i remember this bug fixed, that's right?
, Thanks, seems like necessary. But you have any use case that ipvs proxier will have bugs in current mach rules?
It's already fixed by ? Can we close this issue now?
I don't see how fixed this issue. How about this scenario: k8s cluster A uses node port for service A k8s cluster B uses node port for service B pod (or host) in cluster A tries to access node port on k8s cluster B local node port rule matches the traffic and NATs it to the local node port, pod (or host) cannot reach cluster B's node port. Also, the remote node port could be replaced by any non-k8s host that happens to use port .
Then I think can solve this problem.


Question: import ( "context" <del> "encoding/json" </del> "fmt" "math" "sync"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-serial (gce-cos-master-serial) Kubernetes e2e suite.[sig-scheduling] SchedulerPriorities [Serial] Pod should avoid nodes that have avoidPod annotation June 19th 2021 () Testgrid: https://k8s- Prow history: Looks like pods are scheduled where they shouldn't be? : The tests passes in maybe 20% of the builds, but it looks random, so I don't treat it as a flake. At the moment there are also sig-storage tests consistently failing in this job: , They don't look related though. /sig scheduling /kind failing-test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
try to have a look.
try to have a look.
/cc
This is expected. The feature (which was alpha) is now disabled by default. Users can still enable it by switching to a v1beta1 scheduler component config. Let's remove the test.


Question: operator: Exists - key: {{ .MasterTaintKey }} effect: NoSchedule <ins> nodeSelector: beta.kubernetes.io/os: linux </ins> containers: - name: coredns image: {{ .ImageRepository }}/coredns:{{ .Version }}

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In a windows cluster, containers can get scheduled on Windows nodes where they will never work. Warning Failed 54m (x3 over 54m) kubelet, 35934k8s9000 Failed to pull image "coredns/coredns:1.2.2": rpc error: code = Unknown desc = no matching manifest for windows/amd64 in the manifest list entries What you expected to happen: CoreDNS starts and runs on Linux master nodes. How to reproduce it (as minimally and precisely as possible): Build a Windows Cluster. Note: I first encountered this issue when deploying with ACS-Engine: but it occurs to me that it should be mirrored here, so that the relevant node selectors can be included in the canonical add-on directory. /sig Windows
I suppose this is fine if there are always linux nodes in the cluster. I don't know how Kubernetes or docker manifests work for windows, but CoreDNS does build and run on windows: That is not included in the manifest, is it possible to get that working? I guess you would probably also have to change the way the mounted config file is referred to in windows? Or is it this change good enough?
For now, IMO yes. what do you think? I'm assuming, perhaps incorrectly, that not all core k8s components support windows yet. do you know the current state of Windows support for core k8s components? Is CoreDNS the only core component whose image doesnt support windows?
Have not looked into it in detail yet, but one complication would be configuration. AFAIK Windows systems don't have an ... and the default CoreDNS configuration makes specific reference to the file (to get upstream servers).
Yeah, good point. I was thinking about the -conf /etc/coredns/Corefile which I think we pass to coredns IIRC. But is a bigger issue.
While it is possible to run Windows coreDNS pods, this is still undocumented and not officially supported yet. Officially, only kubelet & kube-proxy are in beta on Windows and scheduled to GA with 1.13 (assuming it meets the criteria and community votes of course). I think for the time being it should be fine to limit scheduling to Linux only to avoid pods being stuck but we should design in a way to keep the option open of undoing this in the future. what do you think?


Question: // namespace used for all tests, do not change this const testNamespace = "dryrunnamespace" <ins> func DryRunCreateWithGenerateNameTest(t *testing.T, rsc dynamic.ResourceInterface, obj *unstructured.Unstructured, gvResource schema.GroupVersionResource) { // special resources with dots in the name cannot use generateName if strings.Contains(obj.GetName(), ".") { return } // Create a new object with generateName to ensure we don't taint the original object gnObj := obj.DeepCopy() gnObj.SetGenerateName(obj.GetName() + "-") gnObj.SetName("") DryRunCreateTest(t, rsc, gnObj, gvResource) } </ins> func DryRunCreateTest(t *testing.T, rsc dynamic.ResourceInterface, obj *unstructured.Unstructured, gvResource schema.GroupVersionResource) { createdObj, err := rsc.Create(context.TODO(), obj, metav1.CreateOptions{DryRun: []string{metav1.DryRunAll}}) if err != nil { <del> t.Fatalf("failed to dry-run create stub for %s: %#v", gvResource, err) </del> <ins> t.Fatalf("failed to dry-run create stub for %s: %#v: %v", gvResource, err, obj) </ins> } if obj.GroupVersionKind() != createdObj.GroupVersionKind() { t.Fatalf("created object doesn't have the same gvk as original object: got %v, expected %v", createdObj.GroupVersionKind(), obj.GroupVersionKind()) } <ins> if createdObj.GetUID() != "" { t.Fatalf("created object shouldn't have a uid: %v", createdObj) } if createdObj.GetResourceVersion() != "" { t.Fatalf("created object shouldn't have a resource version: %v", createdObj) } if obj.GetGenerateName() != "" && createdObj.GetName() != "" { t.Fatalf("created object's name should be an empty string if using GenerateName: %v", createdObj) } </ins> <del> if _, err := rsc.Get(context.TODO(), obj.GetName(), metav1.GetOptions{}); !apierrors.IsNotFound(err) { t.Fatalf("object shouldn't exist: %v", err) </del> <ins> // we won't have a generated name here, so we won't check for this case if obj.GetGenerateName() == "" { if _, err := rsc.Get(context.TODO(), obj.GetName(), metav1.GetOptions{}); !apierrors.IsNotFound(err) { t.Fatalf("object shouldn't exist: %v, %v", obj, err) } </ins> } }

Running kubectl create with dry-run returns unwanted uid metadata field, as well as a generated name when using generateName. This is inconsistent with the expectations as defined in which states I expected uid, and a generated name to be empty when using the dry-run create option. Expected metadata: Actual metadata: No response <detailsI've tested this from 1.19.0 through 1.23.0 </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
reopening, is reverted in to make sure requests with both name+generateName are handled correctly
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


Question: // namespace used for all tests, do not change this const testNamespace = "dryrunnamespace" <ins> func DryRunCreateWithGenerateNameTest(t *testing.T, rsc dynamic.ResourceInterface, obj *unstructured.Unstructured, gvResource schema.GroupVersionResource) { // special resources with dots in the name cannot use generateName if strings.Contains(obj.GetName(), ".") { return } // Create a new object with generateName to ensure we don't taint the original object gnObj := obj.DeepCopy() gnObj.SetGenerateName(obj.GetName() + "-") gnObj.SetName("") DryRunCreateTest(t, rsc, gnObj, gvResource) } </ins> func DryRunCreateTest(t *testing.T, rsc dynamic.ResourceInterface, obj *unstructured.Unstructured, gvResource schema.GroupVersionResource) { createdObj, err := rsc.Create(context.TODO(), obj, metav1.CreateOptions{DryRun: []string{metav1.DryRunAll}}) if err != nil { <del> t.Fatalf("failed to dry-run create stub for %s: %#v", gvResource, err) </del> <ins> t.Fatalf("failed to dry-run create stub for %s: %#v: %v", gvResource, err, obj) </ins> } if obj.GroupVersionKind() != createdObj.GroupVersionKind() { t.Fatalf("created object doesn't have the same gvk as original object: got %v, expected %v", createdObj.GroupVersionKind(), obj.GroupVersionKind()) } <ins> if createdObj.GetUID() != "" { t.Fatalf("created object shouldn't have a uid: %v", createdObj) } if createdObj.GetResourceVersion() != "" { t.Fatalf("created object shouldn't have a resource version: %v", createdObj) } if obj.GetGenerateName() != "" && createdObj.GetName() != "" { t.Fatalf("created object's name should be an empty string if using GenerateName: %v", createdObj) } </ins> <del> if _, err := rsc.Get(context.TODO(), obj.GetName(), metav1.GetOptions{}); !apierrors.IsNotFound(err) { t.Fatalf("object shouldn't exist: %v", err) </del> <ins> // we won't have a generated name here, so we won't check for this case if obj.GetGenerateName() == "" { if _, err := rsc.Get(context.TODO(), obj.GetName(), metav1.GetOptions{}); !apierrors.IsNotFound(err) { t.Fatalf("object shouldn't exist: %v, %v", obj, err) } </ins> } }

/close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: for _, test := range providers { project, zone, instance, err := splitProviderID(test.providerID) if (err != nil) != test.fail { <del> t.Errorf("Expected to failt=%t, with pattern %v", test.fail, test) </del> <ins> t.Errorf("Expected to fail=%t, with pattern %v", test.fail, test) </ins> } if test.fail {

Is this a BUG REPORT or FEATURE REQUEST?: As part of the initiative for external cloud providers (), we GetZoneByProviderID and GetZoneByNodeName. See for more details. The previous PR did not actually implement those methods for existing cloud providers in . These methods should be implemented by all cloud providers in the future. cc
Could someone add the label and the label for this issue please?
/sig cluster-lifecycle
/area cloudprovider
/assign wlan0
cc We need the following label - help-wanted and add it to the 1.8(?) milestone cc to implement it for their respective clouds.
I'll try to have a look by the end of the week.
Ok, I will take care of the Rackspace and OpenStack.
Thank you !
- Is there any listing where to see the already implemented cloud provider ?
No providers have this yet. Current PRs: Openstack: Rackspace: Please contribute if you have something ready :)
Sorry for the delay guys :)
Have you implemented them for gce?
Intending to work on it tonight, but if you have something I can work on azure
Yes, I have prepared for GCE.
Azure .
Cool, than you all! Any other providers we should implement this for in time for v1.8?
I'll take a look tomorrow to see if there's something left
Let's see which providers are left before closing this...
- I guess this one should be closed ?
If you say that all providers have this now, thanks, great!


Question: <del> // +build !linux </del> <ins> // +build windows </ins> /* Copyright 2014 Google Inc. All rights reserved.

This whole set of tests (if it can't be run) should be omitted from the test execution, otherwise folks running on Mac can't verify everything else is clean in a sane fashion. Passive aggressive assign to Tim, I may get around to fixing this before he does.
Last I checked Travis + Go didn't work. We should check again, and enable OS X builds in Travis if it's possible.
You could argue that if the unsupported mount () is being called in a test, there is a problem anyway - it has to run as root. Will investigate. On Jan 25, 2015 9:09 AM, "Brendan Burns" wrote:
Without a Mac to test on, I feel like this would be better fixed by someone who uses Mac... Probably: just rename to I just discovered "implicit build constraints", sigh. Go is crossing from user-friendly to sort of absurd, IMO.


Question: componentbaseconfig "k8s.io/component-base/config" "k8s.io/component-base/configz" "k8s.io/component-base/logs" <ins> metricsfeatures "k8s.io/component-base/metrics/features" </ins> "k8s.io/component-base/metrics/legacyregistry" <ins> "k8s.io/component-base/metrics/prometheus/slis" </ins> "k8s.io/component-base/version" "k8s.io/component-base/version/verflag" "k8s.io/klog/v2"

As per, , wire up metrics/slis for kube-proxy
/assign
/sig network /sig instrumentation
This sounds OK, but we need to translate it into real work-to-do - volunteers? /triage accepted
is tackling this one :)


Question: if err != nil { return nil, err } <ins> ams, err := getVolumeAccessModes(spec) if err != nil { return nil, err } </ins> return &rbdMounter{ rbd:          newRBD(podUID, spec.Name(), img, pool, ro, plugin, manager),

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When use ceph as storageclass, i set accessModes: ReadOnlyMany, and create pods using the claim, only one pod would be running and the others on different nodes would get stuck on ContainerCreating. But if the pods are all on same node, it works fine. What you expected to happen: Pods on different nodes can mount ceph rbd volume. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.9.2 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): 3.10.0-327.18.2.el7.x86_64 Kernel (e.g. ): CentOS Linux release 7.2.1511 (Core) Install tools: kubeadm Others:
/sig storage
/assign
, how would this work? For example, how do you put the initial contents of the RBD volume on it if it is in the ReadOnlyMany access mode? How would you use this feature? As a cheap way of sharing files across a Kubernetes cluster? I'm just interested. -ben england
Do we really want to support ? I have the same questions as mentioned above. If we set the PVC AccessMode to , then the field in will need to be set true ? If so, for the first time, in , we will mount the device without formatting, is that intended ? We need to format the device manually before we use it ? /cc FYI: also there is an issue related to this:
We have this same problem for GCE PD, which supports ReadOnlyMany. You need to mount the volume first as RWO, in order to format the filesystem and populate it with data (mounting a read only volume with no data on it seems pointless). Then you have to fully unmount/detach it, and then you can deploy pods using it as ROX. Basically, you can dynamically provision it in RWO to format + populate data. But afterwards, you need to manually create a different PV/PVC for ROX
continuation


Question: func isKeyNotFoundError(err error) bool { statusErr, _ := err.(*errors.StatusError) <del> // At the moment the server is returning 500 instead of a more specific // error. When changing this remember that it should be backward compatible // with old api servers that may be still returning 500. if statusErr != nil && statusErr.Status().Code == 500 { </del> <ins> if statusErr != nil && statusErr.Status().Code == http.StatusNotFound { </ins> return true } return false }

I keep seeing these server rejected event errors every now and then. My guess is it is trying to update an event that was expired in etcd. However I don't expect an error, since there is specific logic in to handle that use case. Any ideas why this may happening? I am using kubernetes 1.2. Here is an example, from a host getting to a cluster:
Found the problem - the recordEvent logic in was expecting the wrong status code. I've submitted a PR for it. This could be reproduced by: 1 - Update load balancers by remove a node, then adding it back 2 - Remove all events manually from etcd for that namespace 3 - Remove node and add it back again - this results in server rejected event error


Question: } } <ins> func TestLBSecurityGroupsAnnotation(t *testing.T) { awsServices := newMockedFakeAWSServices(TestClusterId) c, _ := newAWSCloud(CloudConfig{}, awsServices) sg1 := map[string]string{ServiceAnnotationLoadBalancerSecurityGroups: "sg-000001"} sg2 := map[string]string{ServiceAnnotationLoadBalancerSecurityGroups: "sg-000002"} sg3 := map[string]string{ServiceAnnotationLoadBalancerSecurityGroups: "sg-000001, sg-000002"} tests := []struct { name string annotations map[string]string expectedSGs []string }{ {"SG specified", sg1, []string{sg1[ServiceAnnotationLoadBalancerSecurityGroups]}}, {"Multiple SGs specified", sg3, []string{sg1[ServiceAnnotationLoadBalancerSecurityGroups], sg2[ServiceAnnotationLoadBalancerSecurityGroups]}}, } awsServices.ec2.(*MockedFakeEC2).expectDescribeSecurityGroups(TestClusterId, "k8s-elb-aid", "cluster.test") for _, test := range tests { t.Run(test.name, func(t *testing.T) { serviceName := types.NamespacedName{Namespace: "default", Name: "myservice"} sgList, err := c.buildELBSecurityGroupList(serviceName, "aid", test.annotations) assert.NoError(t, err, "buildELBSecurityGroupList failed") assert.True(t, sets.NewString(test.expectedSGs...).Equal(sets.NewString(sgList...)), "Security Groups expected=%q , returned=%q", test.expectedSGs, sgList) }) } } </ins> // Test that we can add a load balancer tag func TestAddLoadBalancerTags(t *testing.T) { loadBalancerName := "test-elb"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: We run Kuberentes on AWS. We are trying out service type. We use Terraform heavily for managing infrastructure on AWS. As a result we have several shared security groups with "perfect" rules. One of them is supposed to be attached to ELBs so our own IPs, partners, etc are whitelisted. Right now, it seems there isn't a way for Kubernetes to use the security group as is. There are several alternatives: Pass my existing security group as annotation. However, Kubernetes still tries to create one first, or use a global security group. If I don't provide any IP range, Kubernetes will whitelist . This seems insecure. Provide my own IP ranges as either annotation or . Then Kubernetes will create a security group for this ELB with the given IPs. This means I need to duplicate my IPs in all microservices. Provide my own IP ranges as either annotation or , and use a global security group. Then Kubernetes will modify the global SG with the given IPs. This seems problematic when lots of microservices share one SG but some want different IPs whitelisted. I want to entertain the idea below: Allow users to sepcify another security group to or object, which Kubernetes will take it and attach it to ELBs as is. If such security group is provided, skip other IP whitelisting steps. What you expected to happen: I can give Kubernetes existing security group(s). Kuberentes just attaches the security group(s) to the managed ELBs without modifying the rules. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: It may already be possible to achieve this, and I'm not aware of. Or there are other concerns so that Kubernetes didn't choose to implement this feature. Environment: Kubernetes version (use ): v1.7.1 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig aws
PR may help with this. . .
This problem could be really critical if one has the Kubernetes dashboard installed.


Question: } } <ins> func TestLBSecurityGroupsAnnotation(t *testing.T) { awsServices := newMockedFakeAWSServices(TestClusterId) c, _ := newAWSCloud(CloudConfig{}, awsServices) sg1 := map[string]string{ServiceAnnotationLoadBalancerSecurityGroups: "sg-000001"} sg2 := map[string]string{ServiceAnnotationLoadBalancerSecurityGroups: "sg-000002"} sg3 := map[string]string{ServiceAnnotationLoadBalancerSecurityGroups: "sg-000001, sg-000002"} tests := []struct { name string annotations map[string]string expectedSGs []string }{ {"SG specified", sg1, []string{sg1[ServiceAnnotationLoadBalancerSecurityGroups]}}, {"Multiple SGs specified", sg3, []string{sg1[ServiceAnnotationLoadBalancerSecurityGroups], sg2[ServiceAnnotationLoadBalancerSecurityGroups]}}, } awsServices.ec2.(*MockedFakeEC2).expectDescribeSecurityGroups(TestClusterId, "k8s-elb-aid", "cluster.test") for _, test := range tests { t.Run(test.name, func(t *testing.T) { serviceName := types.NamespacedName{Namespace: "default", Name: "myservice"} sgList, err := c.buildELBSecurityGroupList(serviceName, "aid", test.annotations) assert.NoError(t, err, "buildELBSecurityGroupList failed") assert.True(t, sets.NewString(test.expectedSGs...).Equal(sets.NewString(sgList...)), "Security Groups expected=%q , returned=%q", test.expectedSGs, sgList) }) } } </ins> // Test that we can add a load balancer tag func TestAddLoadBalancerTags(t *testing.T) { loadBalancerName := "test-elb"

If for some reasons the service recreates AWS ELB, the related security group will be recreated as well with the "0.0.0.0/0" input/output masks and then botnets could find the ELB and . JFYI, in our case in the one of clusters with K8s 1.5, the deployment and replication controller were installed from the dashboard using this vector of attack. Then special pods were created and root volume mounted inside. The system crontab was modified by processes in these pods and the crypto miner appeared in all K8s nodes on the system level. After this another pods were installed and the miner was executed inside the Kubernetes pods in parallel with the system level.
just had some downtime on a production cluster because of this. i really need it too ;)
+1
As for me, it isn't a feature, it's a kind of bug.
Some comments on which may be relevant - please comment if an alternative annotation that did replace the default security-groups (as opposed to adding to them) would/wouldn't meet your use-case:
For me, I don't add any annotation, I just want to change its security group rule. After it will create a aws elb, and a specificed secruity group with 0.0.0.0, I change 0.0.0.0 to However, the security group rule will change back to 0.0.0.0 after some minutes. Is this by design? Is there any way for me keep my changes, thanks.
I guess this is part of how the controller works. If you look at , the PR fixes this problem by allowing you to specify the behaviour you want with an annotation. For the way Kubernetes works, the ELB is owned by Kubernetes and you should never be forced to modify the resource manually, thing that ultimately will not work.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale This is still relevant cause is not merged yet.
This is a good change.


Question: } } <ins> func TestLBSecurityGroupsAnnotation(t *testing.T) { awsServices := newMockedFakeAWSServices(TestClusterId) c, _ := newAWSCloud(CloudConfig{}, awsServices) sg1 := map[string]string{ServiceAnnotationLoadBalancerSecurityGroups: "sg-000001"} sg2 := map[string]string{ServiceAnnotationLoadBalancerSecurityGroups: "sg-000002"} sg3 := map[string]string{ServiceAnnotationLoadBalancerSecurityGroups: "sg-000001, sg-000002"} tests := []struct { name string annotations map[string]string expectedSGs []string }{ {"SG specified", sg1, []string{sg1[ServiceAnnotationLoadBalancerSecurityGroups]}}, {"Multiple SGs specified", sg3, []string{sg1[ServiceAnnotationLoadBalancerSecurityGroups], sg2[ServiceAnnotationLoadBalancerSecurityGroups]}}, } awsServices.ec2.(*MockedFakeEC2).expectDescribeSecurityGroups(TestClusterId, "k8s-elb-aid", "cluster.test") for _, test := range tests { t.Run(test.name, func(t *testing.T) { serviceName := types.NamespacedName{Namespace: "default", Name: "myservice"} sgList, err := c.buildELBSecurityGroupList(serviceName, "aid", test.annotations) assert.NoError(t, err, "buildELBSecurityGroupList failed") assert.True(t, sets.NewString(test.expectedSGs...).Equal(sets.NewString(sgList...)), "Security Groups expected=%q , returned=%q", test.expectedSGs, sgList) }) } } </ins> // Test that we can add a load balancer tag func TestAddLoadBalancerTags(t *testing.T) { loadBalancerName := "test-elb"

The situation pre-1.13.0 is somewhat surprising because explicitly setting to an empty array like so: has no effect, and the Security Group Ingress is still .


Question: // second in a bucket, and pop up them once at the timeout. To be more specific, // if you set fire time at X, you can get the bookmark within (X-1,X+1) period. type watcherBookmarkTimeBuckets struct { <del> lock            sync.Mutex watchersBuckets map[int64][]*cacheWatcher startBucketID   int64 clock           clock.Clock </del> <ins> lock              sync.Mutex watchersBuckets   map[int64][]*cacheWatcher startBucketID     int64 clock             clock.Clock bookmarkFrequency time.Duration </ins> } <del> func newTimeBucketWatchers(clock clock.Clock) *watcherBookmarkTimeBuckets { </del> <ins> func newTimeBucketWatchers(clock clock.Clock, bookmarkFrequency time.Duration) *watcherBookmarkTimeBuckets { </ins> return &watcherBookmarkTimeBuckets{ <del> watchersBuckets: make(map[int64][]*cacheWatcher), startBucketID:   clock.Now().Unix(), clock:           clock, </del> <ins> watchersBuckets:   make(map[int64][]*cacheWatcher), startBucketID:     clock.Now().Unix(), clock:             clock, bookmarkFrequency: bookmarkFrequency, </ins> } } // adds a watcher to the bucket, if the deadline is before the start, it will be // added to the first one. func (t *watcherBookmarkTimeBuckets) addWatcher(w *cacheWatcher) bool { <del> nextTime, ok := w.nextBookmarkTime(t.clock.Now()) </del> <ins> nextTime, ok := w.nextBookmarkTime(t.clock.Now(), t.bookmarkFrequency) </ins> if !ok { return false }

Currently, (best-effort) watch bookmarks are sent just right before the given watch is supposed to timeout: In we're adjusting how the size of watch cache is determined and make it dynamic and more duration-oriented (rather than based on some magic constants) - we're targeting to always keep 5 minutes of history in watch cache. With that change, we should get back to our bookmark policy: change it to (additionally to right before timeout) also send bookmarks every N seconds (setting N to slightly less than expected watch cache history period, so slightly less than X minutes we will use there) scale-test this change (to see if it doesn't introduce non-negligible overhead) if so permanently introduce this change /sig scalability api-machinery
/cc
/cc
/cc
I'm reverting as it broke bookmarks. I'm going to fix and re-submit for review hopefully later this week.


Question: apiVersion: extensions/v1beta1 kind: Deployment metadata: <del> name: metrics-server-v0.3.0 </del> <ins> name: metrics-server-v0.3.1 </ins> namespace: kube-system labels: k8s-app: metrics-server kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile <del> version: v0.3.0 </del> <ins> version: v0.3.1 </ins> spec: selector: matchLabels: k8s-app: metrics-server <del> version: v0.3.0 </del> <ins> version: v0.3.1 </ins> template: metadata: name: metrics-server labels: k8s-app: metrics-server <del> version: v0.3.0 </del> <ins> version: v0.3.1 </ins> annotations: scheduler.alpha.kubernetes.io/critical-pod: '' seccomp.security.alpha.kubernetes.io/pod: 'docker/default'

This started after was merged. The HPA test started being very flaky since then: https://k8s- The HPA test in containerd e2e become even more flaky since then with the same failure pattern: https://k8s-
I think this is probably /priority critical-urgent Release team hasn't looked close enough at this issue as we've been trying to clear other issues in front of it. If we crack those and this is unrelated root cause, then this one's going to be a release blocker.
It's going to be supremely hard to debug this unless we have logs from metrics-server and/or snapshots of the HPA over time. I'm trying to reproduce on a local cluster, but so far, the test passes consistently. did you manage to reproduce locally?
is there any way to grab logs of pods from a testing cluster?
you probably want to save it as part of the artifacts?
yeah, what's the process for that?
somewhere along
See for an attempt at debugging this.
can we provide the data is looking for? we're seeing this quite often.
ack, more logs would be helpful. Currently trying to get usable logs off the CI with the aforementioned PR, but I keep making minor mistakes in the log collection script :-/. It seems to repro every few runs though, so with any luck I'll see it again soon.
Got a WIP fix up: Need to do one more tweak before merge.
Re-running tests with the above PR, if they succeed I'll merge and get someone to do a build when the people capable of running builds are on next.
status check? I'm worried this is a 1.12 blocker and need to make a call by end of day today regarding delaying the release.
According to our testing, metrics-server 0.3.1 seems to have fixed it - thank you


Question: }) } <ins> testDone := make(chan struct{}) defer close(testDone) </ins> ts := httptest.NewServer( withDeadline( WithTimeoutForNonLongRunningRequests(

Similar to and easily reproducible. The handler occasionally completes and sends a response before timing out. /kind failing-test /kind flake /sig api-machinery
/assign
/assign Thank you very much for fixing these tests.
/triage accepted


Question: Count:            1, LastObservedTime: metav1.MicroTime{Time: clock.Now()}, } <del> return isomorphicEvent </del> <ins> // Make a copy of the Event to make sure that recording it // doesn't mess with the object stored in cache. return isomorphicEvent.DeepCopy() </ins> } e.eventCache[eventKey] = eventCopy <del> return eventCopy </del> <ins> // Make a copy of the Event to make sure that recording it doesn't // mess with the object stored in cache. return eventCopy.DeepCopy() </ins> }() if evToRecord != nil { <del> recordedEvent := e.attemptRecording(evToRecord) if recordedEvent != nil { recordedEventKey := getKey(recordedEvent) e.mu.Lock() defer e.mu.Unlock() e.eventCache[recordedEventKey] = recordedEvent } </del> <ins> // TODO: Add a metric counting the number of recording attempts e.attemptRecording(evToRecord) // We don't want the new recorded Event to be reflected in the // client's cache because server-side mutations could mess with the // aggregation mechanism used by the client. </ins> } }() }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I wrote a test creating multiple isomorphic Events with an "". and stumbled upon a data race. The data race happens in the function. It seems that the Event Series field can be read and written at the same time by two different goroutines. . This seems to happen in only one scenario: (1) a singleton Event is created (2) an isomorphic Event is created (1) lock addition to cache (2) wait for the lock to be released (1) unlock (2) lock + the Series (1) the freshly created Series (1) singleton Event is now considered as a series That being said, this seems to lead to some data inconsistencies as the singleton Event would be created with an unexpected Series. <details<summaryData race</summary|================== WARNING: DATA RACE Write at 0x00c0001237a0 by goroutine 23: (eventBroadcasterImpl).recordToSink.func1.1() +0x536 (eventBroadcasterImpl).recordToSink.func1() +0x84 Previous read at 0x00c0001237a0 by goroutine 24: (Event).DeepCopyInto() +0x64 (Event).DeepCopy() +0x56 (Event).DeepCopyObject() +0x33 () +0xe8 (CreateActionImpl).DeepCopy() <autogenerated:1 +0xb7 (Fake).Invokes() +0x103 (FakeEvents).Create() +0x214 (EventSinkImpl).Create() +0x195 () +0x32c (eventBroadcasterImpl).attemptRecording() +0xe4 (eventBroadcasterImpl).recordToSink.func1() +0x9d Goroutine 23 (running) created at: (eventBroadcasterImpl).recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (eventBroadcasterImpl).StartEventWatcher.func2() +0x91 Goroutine 24 (running) created at: (eventBroadcasterImpl).


Question: Count:            1, LastObservedTime: metav1.MicroTime{Time: clock.Now()}, } <del> return isomorphicEvent </del> <ins> // Make a copy of the Event to make sure that recording it // doesn't mess with the object stored in cache. return isomorphicEvent.DeepCopy() </ins> } e.eventCache[eventKey] = eventCopy <del> return eventCopy </del> <ins> // Make a copy of the Event to make sure that recording it doesn't // mess with the object stored in cache. return eventCopy.DeepCopy() </ins> }() if evToRecord != nil { <del> recordedEvent := e.attemptRecording(evToRecord) if recordedEvent != nil { recordedEventKey := getKey(recordedEvent) e.mu.Lock() defer e.mu.Unlock() e.eventCache[recordedEventKey] = recordedEvent } </del> <ins> // TODO: Add a metric counting the number of recording attempts e.attemptRecording(evToRecord) // We don't want the new recorded Event to be reflected in the // client's cache because server-side mutations could mess with the // aggregation mechanism used by the client. </ins> } }() }

recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (eventBroadcasterImpl).StartEventWatcher.func2() +0x91 |================== |================== WARNING: DATA RACE Read at 0x00c0001236f0 by goroutine 23: (Event).DeepCopyInto() +0x64 (Event).DeepCopy() +0x72 () +0x4d () +0x98 (eventBroadcasterImpl).attemptRecording() +0xe4 (eventBroadcasterImpl).recordToSink.func1() +0x9d Previous write at 0x00c0001236f0 by goroutine 24: () +0x2e4 (eventBroadcasterImpl).attemptRecording() +0xe4 (eventBroadcasterImpl).recordToSink.func1() +0x9d Goroutine 23 (running) created at: (eventBroadcasterImpl).recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (eventBroadcasterImpl).StartEventWatcher.func2() +0x91 Goroutine 24 (running) created at: (eventBroadcasterImpl).recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (*eventBroadcasterImpl).StartEventWatcher.func2() +0x91 |================== </detailsWhat you expected to happen: No data race. How to reproduce it (as minimally and precisely as possible): You can run the following code snippet alongside any data race detector. I personally used . Anything else we need to know?: Environment: v0.25.0
/sig api-machinery
/assign Would you mind taking a look to this one? thanks!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity.


Question: Count:            1, LastObservedTime: metav1.MicroTime{Time: clock.Now()}, } <del> return isomorphicEvent </del> <ins> // Make a copy of the Event to make sure that recording it // doesn't mess with the object stored in cache. return isomorphicEvent.DeepCopy() </ins> } e.eventCache[eventKey] = eventCopy <del> return eventCopy </del> <ins> // Make a copy of the Event to make sure that recording it doesn't // mess with the object stored in cache. return eventCopy.DeepCopy() </ins> }() if evToRecord != nil { <del> recordedEvent := e.attemptRecording(evToRecord) if recordedEvent != nil { recordedEventKey := getKey(recordedEvent) e.mu.Lock() defer e.mu.Unlock() e.eventCache[recordedEventKey] = recordedEvent } </del> <ins> // TODO: Add a metric counting the number of recording attempts e.attemptRecording(evToRecord) // We don't want the new recorded Event to be reflected in the // client's cache because server-side mutations could mess with the // aggregation mechanism used by the client. </ins> } }() }

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We might also want to look into that again for /reopen /assign /unassign
/remove-lifecycle rotten
Reopened this issue. <detailsIn response to : We might also want to look into that again for /unassign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/cc
I was able to reproduce this issue with client-go v0.25.0. I updated the reproducer to be compatible with the latest version of the client and attached the data race trace.
are you gong to submit a fix for this one? thanks. /triage accepted
Yes, I am planning to submit a fix for this issue.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: Count:            1, LastObservedTime: metav1.MicroTime{Time: clock.Now()}, } <del> return isomorphicEvent </del> <ins> // Make a copy of the Event to make sure that recording it // doesn't mess with the object stored in cache. return isomorphicEvent.DeepCopy() </ins> } e.eventCache[eventKey] = eventCopy <del> return eventCopy </del> <ins> // Make a copy of the Event to make sure that recording it doesn't // mess with the object stored in cache. return eventCopy.DeepCopy() </ins> }() if evToRecord != nil { <del> recordedEvent := e.attemptRecording(evToRecord) if recordedEvent != nil { recordedEventKey := getKey(recordedEvent) e.mu.Lock() defer e.mu.Unlock() e.eventCache[recordedEventKey] = recordedEvent } </del> <ins> // TODO: Add a metric counting the number of recording attempts e.attemptRecording(evToRecord) // We don't want the new recorded Event to be reflected in the // client's cache because server-side mutations could mess with the // aggregation mechanism used by the client. </ins> } }() }

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
A PR is out to fix this issue: /remove-lifecycle stale


Question: // ValidateKubeletFlags validates Kubelet's configuration flags and returns an error if they are invalid. func ValidateKubeletFlags(f *KubeletFlags) error { unknownLabels := sets.NewString() <del> for k := range f.NodeLabels { </del> <ins> invalidLabelErrs := make(map[string][]string) for k, v := range f.NodeLabels { </ins> if isKubernetesLabel(k) && !kubeletapis.IsKubeletLabel(k) { unknownLabels.Insert(k) } <ins> if errs := validation.IsQualifiedName(k); len(errs) > 0 { invalidLabelErrs[k] = append(invalidLabelErrs[k], errs...) } if errs := validation.IsValidLabelValue(v); len(errs) > 0 { invalidLabelErrs[v] = append(invalidLabelErrs[v], errs...) } </ins> } if len(unknownLabels) > 0 { return fmt.Errorf("unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels: %vn--node-labels in the 'kubernetes.io' namespace must begin with an allowed prefix (%s) or be in the specifically allowed set (%s)", unknownLabels.List(), strings.Join(kubeletapis.KubeletLabelNamespaces(), ", "), strings.Join(kubeletapis.KubeletLabels(), ", ")) } <ins> if len(invalidLabelErrs) > 0 { labelErrs := []string{} for k, v := range invalidLabelErrs { labelErrs = append(labelErrs, fmt.Sprintf("'%s' - %s", k, strings.Join(v, ", "))) } return fmt.Errorf("invalid node labels: %s", strings.Join(labelErrs, "; ")) } </ins> if f.SeccompDefault && !utilfeature.DefaultFeatureGate.Enabled(features.SeccompDefault) { return fmt.Errorf("the SeccompDefault feature gate must be enabled in order to use the --seccomp-default flag")

kubelet won't return error when invalid node labels are passed to flag, and node can't take part in a cluster. kubelet should return errors when invalid node labels are passed. You pass flag to kubelet (value is invalid), node doesn't take part in a cluster. ref. No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig node
/triage accepted
this should not be a breaking change as invalid labels make the kubelet unusable anyway. So better not even start the kubelet. /assign since there is a PR already
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: t.Errorf("Failed to update node in schedulercache:n got: %+v nexpected: %+v", got, expected) } // Check nodeTree after update <del> if cache.nodeTree.NumNodes != 1 || cache.nodeTree.Next() != node.Name { </del> <ins> if cache.nodeTree.NumNodes() != 1 || cache.nodeTree.Next() != node.Name { </ins> t.Errorf("unexpected cache.nodeTree after updating node: %v", node.Name) }

Description: , part of the is exporting a field which allows to modify its value externally. This also means that this field is not thread-safe. Solution: Expose a method that returns this value in a thread-safe way and do not export the field. /kind bug /sig scheduling


Question: return "", err } klog.V(2).Infof("azureDisk - WaitForAttach: GetDiskLun succeeded, got lun(%v)", lun) <del> } else { lun, err = getDiskLUN(devicePath) if err != nil { return "", err } </del> } exec := a.plugin.host.GetExec(a.plugin.GetPluginName())

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /assign


Question: // back to the queue multiple times before it's successfully scheduled. // It shouldn't be updated once initialized. It's used to record the e2e scheduling // latency for a pod. <del> InitialAttemptTimestamp time.Time </del> <ins> InitialAttemptTimestamp *time.Time </ins> // If a Pod failed in a scheduling cycle, record the plugin names it failed by. UnschedulablePlugins sets.Set[string] // Whether the Pod is scheduling gated (by PreEnqueuePlugins) or not.

is recording the time that a Pod is gated. We use the timestamp when the scheduler inserts the pod into the queue: The period of time when a Pod fails PreEnque (like being gated) shouldn't be accounted in the podschedulingdurationseconds. Create a Pod with scheduling gates. Wait some time before removing the gate. Observe the metric No response <details</details<detailsAny </details<details</details<details</details<details</details<details</details
/sig scheduling
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
would you want to take a stab on this issue?
Yup, I can give it a shot. /assign helayoty
, as we discussed, we can address this issue with different solutions: the pod timestamp to the current time , so the gated time is substituted. timestamp . Use a pointer to the InitialAttemptTimestamp and interpret as the newest timestamp so the pod will go to the back of the queue. I'll go ahead and implement solution no. 3, wdyt?
sgtm. Any concerns ?
I think it should be option 1 + 3, and pInfo.Timestamp should be pInfo.InitialAttemptTimestamp, right? as records the time based on :
I think we should revisit this issue, in terms of its assumption: So is it reasonable to always exclude time spent in PreEnqueue from ? Note that PreEnqueue != SchedulingGates, is one of the PreEnqueue's usecases.


Question: // back to the queue multiple times before it's successfully scheduled. // It shouldn't be updated once initialized. It's used to record the e2e scheduling // latency for a pod. <del> InitialAttemptTimestamp time.Time </del> <ins> InitialAttemptTimestamp *time.Time </ins> // If a Pod failed in a scheduling cycle, record the plugin names it failed by. UnschedulablePlugins sets.Set[string] // Whether the Pod is scheduling gated (by PreEnqueuePlugins) or not.

In other words, some user of PreEnqueue may want to include the time spent in PreEnqueue.
is a signal that cluster providers can use to build SLOs. The time spent before passing PreEnqueue is not under control of kube-scheduler, so considering it would not allow to build an SLO. Are you aware of other users of PreEnqueue?
True, but comparing to state that metric X can or cannot be used to build SLO (which is hard IMO), I'm in favor of giving suggestion on SLI/SLO's semantic definition, for example: the official SLI definition of has a footnote clearly explaining its definition: So if a pod is gated, it's not "immediately schedulable", and hence doesn't fall into the definition of podstartup_latency.
A viable solution might be providing an extra label like to explicitly exclude the time spent on preenqueue.
That is true, but there is no way today to exclude those pods from the metric, or separate the time. Maybe we need an extra metric, but I think the new metric could be the one that includes the time guarded in PreEnqueue. It should probably be part of the KEP. The old metric is already stable, so we can't add labels. And it might already be in use for SLOs.
cc
I think we should be working on that. And treat it as part of GA requirement for PodSchedulingReadiness.
An edge(maybe not edge) case is a pod's scheduling attempts is 0 but the is a long time, so prefer not to account for the waiting time when gated. For users not using PreEnqueue, they will not be impacted, but for users using PreEnqueue, it's a break change for it increases the duration time. And we can add a new metric like to measure how long a pod stays in a scheduler.
I thought of "attempts", it's just I'm not sure it's a good idea to rely on "attempts" to define a SLI/SLO, b/c "attempts" is obtained from the in-memory object and in a HA config env, it's only updated by the primary scheduler instance. In other words, the value is inaccurate when the lease owner switches.
attempt=0 is good enough for most scenarios
do we have a conclusion?


Question: // back to the queue multiple times before it's successfully scheduled. // It shouldn't be updated once initialized. It's used to record the e2e scheduling // latency for a pod. <del> InitialAttemptTimestamp time.Time </del> <ins> InitialAttemptTimestamp *time.Time </ins> // If a Pod failed in a scheduling cycle, record the plugin names it failed by. UnschedulablePlugins sets.Set[string] // Whether the Pod is scheduling gated (by PreEnqueuePlugins) or not.

Should we move on with the solution or remove this issue and the linked PR from the current release?
I would like to move forward with this proposal. ?
To move it forward, I think we should add new metric(s) to explicitly tell it's including or excluding the time spent on PreEnqueue. Given is in stable level: We will need a new metric like "podschedulingexcludingpreenqueuedurationseconds".
28 Bug Triage Shadow here. Just checking-in, if this is still on track for the 1.28 release?
We don't have a conclusion yet, but as a bug, it should qualify for changes after the freezes.


Question: defer ss.vmssVMCache.Delete(key) klog.V(2).Infof("azureDisk - update(%s): vm(%s) - detach disk(%s)", nodeResourceGroup, nodeName, diskName) <del> _, err = ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, vm) </del> <ins> _, err = ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, newVM) </ins> if err != nil { klog.Errorf("azureDisk - detach disk(%s) from %s failed, err: %v", diskName, nodeName, err) } else {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When attach/detach azure disk in vmss, it updates whole VM info, including: networkProfile, storageProfile, while this would lead to race condition when there is also network related setting changing in k8s cloud provider in the mean time. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig azure /assign