Question: // Two different versions of the same pod will always have different RVs. return } <ins> podChanged := podAddressChanged(oldPod, newPod) // Check if the pod labels have changed, indicating a possibe // change in the service membership labelsChanged := false if !reflect.DeepEqual(newPod.Labels, oldPod.Labels) || !hostNameAndDomainAreEqual(newPod, oldPod) { labelsChanged = true } // If both the pod and labels are unchanged, no update is needed if !podChanged && !labelsChanged { return } </ins> services, err := e.getPodServiceMemberships(newPod) if err != nil { utilruntime.HandleError(fmt.Errorf("Unable to get pod %v/%v's service memberships: %v", newPod.Namespace, newPod.Name, err)) return } <del> // Only need to get the old services if the labels changed. if !reflect.DeepEqual(newPod.Labels, oldPod.Labels) || !hostNameAndDomainAreEqual(newPod, oldPod) { </del> <ins> if labelsChanged { </ins> oldServices, err := e.getPodServiceMemberships(oldPod) if err != nil { utilruntime.HandleError(fmt.Errorf("Unable to get pod %v/%v's service memberships: %v", oldPod.Namespace, oldPod.Name, err)) return } <del> services = services.Union(oldServices) </del> <ins> services = determineNeededServiceUpdates(oldServices, services, podChanged) </ins> } for key := range services { e.queue.Add(key) }

/kind bug What happened: On a large cluster with enough pods, endpoints take as long as 10 minutes to show up after their associated pods and service are created. We believe that this is because every pod change results in an update of the addresses within the endpoint since they refer to a pod via a whose changes every time a pod change is observed. All these non-update updates eventually overwhelm the endpoint controller's queue rate limit and the queue starts to grow. While an increase in the queue's rate limit can alleviate the problem, an admin can keep a much lower limit if the non-update updates are dropped rather than processed. What you expected to happen: When a pod related to an endpoint is updated, I would expect the endpoint to update only if there is a change in the pod that relates to the endpoint, such as a change in the pod's IP address. How to reproduce it (as minimally and precisely as possible): a pod and associated service: the endpoint for changes: the watch running, and in another session, edit a pod to add a label: Then add another label on the line after the label, such as that the watched endpoint changed despite there not being any meaningful change to the fields the endpoint cares about. Anything else we need to know?: The PR addresses this issue. Environment: Kubernetes version (use ):
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig scalability
what ever happened with the effort to move leases off of Endpoints?


Question: configs[CheckMetadataConcealment] = Config{e2eRegistry, "metadata-concealment", "1.2"} configs[CudaVectorAdd] = Config{e2eRegistry, "cuda-vector-add", "1.0"} configs[CudaVectorAdd2] = Config{e2eRegistry, "cuda-vector-add", "2.0"} <del> configs[DebianIptables] = Config{buildImageRegistry, "debian-iptables", "v12.1.2"} </del> <ins> configs[DebianIptables] = Config{buildImageRegistry, "debian-iptables", "buster-v1.3.0"} </ins> configs[EchoServer] = Config{e2eRegistry, "echoserver", "2.2"} configs[Etcd] = Config{gcRegistry, "etcd", "3.4.13-0"} configs[GlusterDynamicProvisioner] = Config{dockerGluster, "glusterdynamic-provisioner", "v1.0"}

What would you like to be : Have debian-iptables with iptables 1.8.5+ Why is this needed: To fix (nodelocaldns + iptables-nft) More informations Found a bug in iptables-nft, fixed in iptables 1.8.5, and now available in buster-backports https://alioth-
/sig api-machinery
ping as you did the last 2 updates
Yep, this is on my list. Will take of it next week. /sig release network /area release-eng /milestone v1.20 /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Opened to bump the debian-iptables images in kubernetes/kubernetes. /remove-triage unresolved
Here are the digests for the new image ():
Please reopen if you run into any issues with the new images. /close
Closing this issue. <detailsIn response to : Please reopen if you run into any issues with the new images. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ####################### # kubectl local proxy # ####################### <del> # This next block disabled until non-racy solution to # port allocation can be found (#19999). # #  # Make sure the UI can be proxied #  start-proxy #  check-curl-proxy-code /ui 301 #  check-curl-proxy-code /metrics 200 #  check-curl-proxy-code /api/ui 404 #  if [[ -n "${version}" ]]; then #    check-curl-proxy-code /api/${version}/namespaces 200 #  fi #  check-curl-proxy-code /static/ 200 #  stop-proxy # #  # Make sure the in-development api is accessible by default #  start-proxy #  check-curl-proxy-code /apis 200 #  check-curl-proxy-code /apis/extensions/ 200 #  stop-proxy # #  # Custom paths let you see everything. #  start-proxy /custom #  check-curl-proxy-code /custom/ui 301 #  check-curl-proxy-code /custom/metrics 200 #  if [[ -n "${version}" ]]; then #    check-curl-proxy-code /custom/api/${version}/namespaces 200 #  fi #  stop-proxy </del> <ins> # Make sure the UI can be proxied start-proxy check-curl-proxy-code /ui 301 check-curl-proxy-code /metrics 200 check-curl-proxy-code /api/ui 404 if [[ -n "${version}" ]]; then check-curl-proxy-code /api/${version}/namespaces 200 fi check-curl-proxy-code /static/ 200 stop-proxy # Make sure the in-development api is accessible by default start-proxy check-curl-proxy-code /apis 200 check-curl-proxy-code /apis/extensions/ 200 stop-proxy # Custom paths let you see everything. start-proxy /custom check-curl-proxy-code /custom/ui 301 check-curl-proxy-code /custom/metrics 200 if [[ -n "${version}" ]]; then check-curl-proxy-code /custom/api/${version}/namespaces 200 fi stop-proxy </ins> ########################### # POD creation / deletion #

I disabled a block of code in . The code was flaky (). Suggest one of the following solutions: this type of test coverage to an e2e, or VM or docker based test, where there are no port conflicts. kubectl proxy command to have a mode where it will pick a port for you, trying until it finds one. the test loop to allow failure of kubectl proxy command when it exits with a certain error code that means it could not get a port, and retries until it gets one.
Fixed in ?
No, not fixed:
We still need to reenable this code. I was unaware that it'd been disabled.
I made a reversion PR.


Question: if na, ok := nt.tree[zone]; ok { for _, nodeName := range na.nodes { if nodeName == n.Name { <del> klog.Warningf("node %v already exist in the NodeTree", n.Name) </del> <ins> klog.Warningf("node %q already exist in the NodeTree", n.Name) </ins> return } }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) Increase the log level of info logs to "v=2" when the scheduler or a node from its cache. 2) Increase the log level of info logs to "v=2" when the scheduler can't schedule a pod , , and add a log message for 2) report memory/cpu status of . Why is this needed: With cluster autoscaler now being used in production by some vendors, we want to have more visibility to events that could be related to the auto scaler to help debug customer issues. /sig scheduling /priority important-soon /assign
GitHub didn't allow me to assign the following users: congliu01. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : <!-- Please only use this template for submitting enhancement requests --1) Increase the log level of info logs to "v=2" when the scheduler or a node from its cache. 2) report memory/cpu status of . Why is this needed: With cluster autoscaler now being used in production by some vendors, we want to have more visibility to events that could be related to the auto scaler to help debug customer issues. /priority important-soon /assign </details
will consider this assigned to you.
/assign


Question: result += stat.RestartCount } } <ins> if *job.Spec.BackoffLimit == 0 { return result > 0 } </ins> return result >= *job.Spec.BackoffLimit }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: If you have a JobSpec and set with in 1.9.x, then the job will start a pod and never restart it regardless of success of failure. In 1.10.x, a job spec with and , k8s does not start any pods at all. We had to set in order to get the pod to run. The wording in the docs suggest that should only take effect when the pod is finished/or failed. It shouldn't cap starting the initial pod. How to reproduce it (as minimally and precisely as possible): Run the following job spec in 1.9.x and 1.10.x: Anything else we need to know?: The docs links this existing ticket saying is ineffective with , but the ticket talks about restarting too many times, not never starting a pod. Given this issue was slightly different, I originally did not think it was a backoffPolicy bug. Environment: Kubernetes version (use ): Tested on 1.9.7 and 1.10.6 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Fedora 28
/sig apps
We've been hit by this too, in v1.10.6. It seems to me that with the current behavior, a value of doesn't really make any semantic sense. I think the problem comes from the function in : called here: if is , then will end with .


Question: switch { case storage.IsNotFound(err): return errors.NewNotFound(qualifiedResource, "") <del> case storage.IsUnreachable(err): </del> <ins> case storage.IsUnreachable(err), storage.IsRequestTimeout(err): </ins> return errors.NewServerTimeout(qualifiedResource, "list", 2) // TODO: make configurable or handled at a higher level case storage.IsInternalError(err): return errors.NewInternalError(err)

It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1.


Question: switch { case storage.IsNotFound(err): return errors.NewNotFound(qualifiedResource, "") <del> case storage.IsUnreachable(err): </del> <ins> case storage.IsUnreachable(err), storage.IsRequestTimeout(err): </ins> return errors.NewServerTimeout(qualifiedResource, "list", 2) // TODO: make configurable or handled at a higher level case storage.IsInternalError(err): return errors.NewInternalError(err)

We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: tree      map[string]*nodeArray // a map from zone (region-zone) to an array of nodes in the zone. zones     []string              // a list of all the zones in the tree (keys) zoneIndex int <del> NumNodes  int </del> <ins> numNodes  int </ins> mu        sync.RWMutex }

Description: , part of the is exporting a field which allows to modify its value externally. This also means that this field is not thread-safe. Solution: Expose a method that returns this value in a thread-safe way and do not export the field. /kind bug /sig scheduling


Question: podInPendingStateTimeout  = time.Second * 60 // how long to wait for a pod to stay in pending state ) <del> var _ = ginkgo.Describe("[sig-node] DRA [Feature:DynamicResourceAllocation]", func() { </del> <ins> var _ = ginkgo.Describe("[sig-node] DRA [Feature:DynamicResourceAllocation][NodeAlphaFeature:DynamicResourceAllocation]", func() { </ins> f := framework.NewDefaultFramework("dra-node") f.NamespacePodSecurityEnforceLevel = admissionapi.LevelBaseline

sig-node : DRA related tests are failing and 2 other DRA related tests. Both jobs started failing on 7th June As per the logs DRA plugin is not getting registered to kubelet Jobs started failing after merging As per , CDI is not available out of the box for containerd which may be the reason for failure of the jobs. /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/cc
Looks like I shouldn't have labeled DRA Node tests as NodeFeature. However, re-labeling it NodeAlphaFeature would not work either - would start to fail. I'm going to remove . This should help.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Not fixed yet, pending


Question: command: - /bin/bash - -c <del> - /opt/kube-addons.sh 1>>/var/log/kube-addon-manager.log 2>&1 </del> <ins> - exec /opt/kube-addons.sh 1>>/var/log/kube-addon-manager.log 2>&1 </ins> resources: requests: cpu: 5m

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When kube-apiserver's livenessProbe suggests that pod is unhealthy, kubelet tries to kill its container with 30 seconds grace period. The container is killed ungracefully with SIGKILL when grace period passes. This happens each time. What you expected to happen: I expected that SIGTERM will be passed to kube-apiserver which will try to gracefully stop before 30 seconds pass (at least in some cases). How to reproduce it (as minimally and precisely as possible): Inject kube-apiserver failure (e.g. ) Watch kubelet logs for container's death You will see that container is killed after 30 seconds each time Anything else we need to know?: I believe the problem is that in kube-apiserver.manifest uses '/bin/sh -c CMD' as a PID1 process which doesn't pass SIGTERM signals to child process. See Environment: Kubernetes version (use ): v1.7.11
/sig node
Looks like without the pipe command will have to be removed which will make the logs hard to track. There is indeed some which makes kube-apiserver run in background and trapping SIGTERM then send a SIGTERM to apiserver's PID.
I think this is the case for all our components, not just kube-apiserver. - FYI
How about using 'exec'-based solution suggested here: In our case it means prepending 'exec': I did changed this in my cluster and the process seems to meet requirements: fd/1 and fd/2 are redirected to /var/log/kube- container is killed instantaneously If everyone is fine with such a solution, I can prepare PR adding 'exec ' in front of all kubernetes/cluster/saltbase/salt/kube-apiserver/*.manifest using '/bin/sh -c CMD' pattern.
Not sure if understand what you mean. Before there was no logs, so /bin/sh -c wasn't needed. -based solution seems to have all required properties. What is your concern about using it?
Umm. just ignore my previous comment. I just tried prepending and it works fine for me.


Question: command: - /bin/bash - -c <del> - /opt/kube-addons.sh 1>>/var/log/kube-addon-manager.log 2>&1 </del> <ins> - exec /opt/kube-addons.sh 1>>/var/log/kube-addon-manager.log 2>&1 </ins> resources: requests: cpu: 5m

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 3 Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: All of the salt scripts bundled in kubernetes (for example, ) start up the component in a broken way. They use which means the shell will intercept all signals. So these daemons are not gracefully shutdown when the pod is stopped. This causes serious problems since when apiserver is killed abruptly it doesn't seem to close watches correctly. This was observed in for example. What you expected to happen: Running is incorrect. It should run the binary directly. Otherwise, it needs to perform an so the binary correctly replaces the shell script process. How to reproduce it (as minimally and precisely as possible): Run kube-apiserver inside a container with and try to stop it. You can't - eventually docker will force kill it. Anything else we need to know:
It looks like we changed from running the binaries directly to using in (from a few years ago to enable us to capture logs. or may recall why we had to use a shell since they reviewed that PR.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: command: - /bin/bash - -c <del> - /opt/kube-addons.sh 1>>/var/log/kube-addon-manager.log 2>&1 </del> <ins> - exec /opt/kube-addons.sh 1>>/var/log/kube-addon-manager.log 2>&1 </ins> resources: requests: cpu: 5m

Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: name                           string hostnameOverride               bool nodeIP                         net.IP <ins> secondaryNodeIP                net.IP </ins> cloudProviderType              cloudProviderType nodeAddresses                  []v1.NodeAddress expectedAddresses              []v1.NodeAddress

When adding to kubelet and to the OpenStack external cloud provider, which is based on kubernetes v1.27.1, the node annotation only shows one address and consequently the node addresses also only displays one address. The kubelet logs shows that node-ip contains dual-stack addresses. The annotation on the node would contain both IPv4 and IPv6 addresses and the node addresses would list them. Deploy a cluster with to kubelet and external cloud provider. Also make sure the external cloud provider is based on Kubernetes 1.27. Kubelet logs: one of CCMs Pods: one of the nodes annotation: <details</details<details</details<details</details<details</details<details</details<details</details
/sig network /sig cloud-provider /cc
/assign
/triage accepted


Question: { Image:   image.GetE2EImage(), Name:    "computer-name-test", <del> Command: []string{"cmd.exe", "/K", "IF", "NOT", "%COMPUTERNAME%", "==", targetNode.Name, "(", "exit", "-1", ")"}, </del> <ins> Command: []string{"powershell.exe", "-Command", command}, </ins> }, }, RestartPolicy: v1.RestartPolicyNever,

Windows GCE jobs: [sig-windows] [Feature:WindowsHostProcessContainers] [MinimumKubeletVersion:1.22] HostProcess containers should run as a process on the host/node [sig-windows] [Feature:WindowsHostProcessContainers] [MinimumKubeletVersion:1.22] HostProcess containers should support various volume mount types Since the jobs were reintroduced. Windows COMPUTERNAMEs are typically limited to 15 characters [1], while the kubelet nodes spawned in the GCE test jobs are much longer (e.g.: `). The names might be truncated, resulting in the test failure. [1] No response /sig windows /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
dupe of /close
Closing this issue. <detailsIn response to : dupe of /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: { Image:   image.GetE2EImage(), Name:    "computer-name-test", <del> Command: []string{"cmd.exe", "/K", "IF", "NOT", "%COMPUTERNAME%", "==", targetNode.Name, "(", "exit", "-1", ")"}, </del> <ins> Command: []string{"powershell.exe", "-Command", command}, </ins> }, }, RestartPolicy: v1.RestartPolicyNever,

I want to run the e2e test but it failed. Reason: Node name and hostname is longer than 15 characters. But the ComputerName only used the first 15 characters. So they don't equal and the test failed. The e2etest should pass regardless of the hostname length. Deploy a Windows with long nodename (longer than 15 characters). Then, run e2e test. No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig testing
Can you show the error message?
STEP: Destroying namespace "host-process-test-windows-2574" for this suite. • Failure [6.334 seconds] [sig-windows] [Feature:WindowsHostProcessContainers] [Excluded:WindowsDocker] [MinimumKubeletVersion:1.22] HostProcess containers should run as a process on the host/node [It] Nov 21 22:12:49.003: Expected <v1.PodPhase: Failed to equal <v1.PodPhase: Succeeded {"msg":"FAILED [sig-windows] [Feature:WindowsHostProcessContainers] [Excluded:WindowsDocker] [MinimumKubeletVersion:1.22] HostProcess containers should run as a process on the host/node","total":1,"completed":0,"skipped":1147,"failed":1,"failures":["[sig-windows] [Feature:WindowsHostProcessContainers] [Excluded:WindowsDocker] [MinimumKubeletVersion:1.22] HostProcess containers should run as a process on the host/node"]}
/sig windows does this hostname error ring a bell?
We discussed this a bit on slack, the test is using %COMPUTERNAME% to validate we are running on the host. This value is truncated to 15 chars on windows due to netbios. We could use the hostname, which isn't truncated, with a fallback to computer name for this validation to make the test validate more scenarios. Important to note that some scenarios might not work if in older environments that use netbios but for our test purposes this will work. Long discussion on netbios here:
We'll be working on this as part of a broader windows operational readiness work. Thanks wenli for digging into it.


Question: { Image:   image.GetE2EImage(), Name:    "computer-name-test", <del> Command: []string{"cmd.exe", "/K", "IF", "NOT", "%COMPUTERNAME%", "==", targetNode.Name, "(", "exit", "-1", ")"}, </del> <ins> Command: []string{"powershell.exe", "-Command", command}, </ins> }, }, RestartPolicy: v1.RestartPolicyNever,

/assign
Just for broader context... the reason wenli discovered this is, bc we heavily test cluster API over here :), where the dynamically generated host name appends machine-set + cluster characters, so the machines have names like this: to the machines initial name, and it's hardcoded. We want to try to improve this upstream , and have a proposal to do this in flight also:
/sig windows /remove-sig testing
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.


Question: { Image:   image.GetE2EImage(), Name:    "computer-name-test", <del> Command: []string{"cmd.exe", "/K", "IF", "NOT", "%COMPUTERNAME%", "==", targetNode.Name, "(", "exit", "-1", ")"}, </del> <ins> Command: []string{"powershell.exe", "-Command", command}, </ins> }, }, RestartPolicy: v1.RestartPolicyNever,

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen being fixed by
Reopened this issue. <detailsIn response to : /reopen being fixed by Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.


Question: { Image:   image.GetE2EImage(), Name:    "computer-name-test", <del> Command: []string{"cmd.exe", "/K", "IF", "NOT", "%COMPUTERNAME%", "==", targetNode.Name, "(", "exit", "-1", ")"}, </del> <ins> Command: []string{"powershell.exe", "-Command", command}, </ins> }, }, RestartPolicy: v1.RestartPolicyNever,

This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: FSType:       "ext4", }, }, <ins> AccessModes: []v1.PersistentVolumeAccessMode{v1.ReadOnlyMany}, </ins> }, }, false), root: tmpDir,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When use ceph as storageclass, i set accessModes: ReadOnlyMany, and create pods using the claim, only one pod would be running and the others on different nodes would get stuck on ContainerCreating. But if the pods are all on same node, it works fine. What you expected to happen: Pods on different nodes can mount ceph rbd volume. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.9.2 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): 3.10.0-327.18.2.el7.x86_64 Kernel (e.g. ): CentOS Linux release 7.2.1511 (Core) Install tools: kubeadm Others:
/sig storage
/assign
, how would this work? For example, how do you put the initial contents of the RBD volume on it if it is in the ReadOnlyMany access mode? How would you use this feature? As a cheap way of sharing files across a Kubernetes cluster? I'm just interested. -ben england
Do we really want to support ? I have the same questions as mentioned above. If we set the PVC AccessMode to , then the field in will need to be set true ? If so, for the first time, in , we will mount the device without formatting, is that intended ? We need to format the device manually before we use it ? /cc FYI: also there is an issue related to this:
We have this same problem for GCE PD, which supports ReadOnlyMany. You need to mount the volume first as RWO, in order to format the filesystem and populate it with data (mounting a read only volume with no data on it seems pointless). Then you have to fully unmount/detach it, and then you can deploy pods using it as ROX. Basically, you can dynamically provision it in RWO to format + populate data. But afterwards, you need to manually create a different PV/PVC for ROX
continuation


Question: fs.MarkDeprecated("resource-container", "This feature will be removed in a later release.") fs.StringVar(&o.config.ClientConnection.Kubeconfig, "kubeconfig", o.config.ClientConnection.Kubeconfig, "Path to kubeconfig file with authorization information (the master location is set by the master flag).") fs.Var(utilflag.PortRangeVar{Val: &o.config.PortRange}, "proxy-port-range", "Range of host ports (beginPort-endPort, single port or beginPort+offset, inclusive) that may be consumed in order to proxy service traffic. If (unspecified, 0, or 0-0) then ports will be randomly chosen.") <del> fs.StringVar(&o.config.HostnameOverride, "hostname-override", o.config.HostnameOverride, "If non-empty, will use this string as identification instead of the actual hostname.") </del> <ins> fs.StringVar(&o.hostnameOverride, "hostname-override", o.hostnameOverride, "If non-empty, will use this string as identification instead of the actual hostname.") </ins> fs.Var(&o.config.Mode, "proxy-mode", "Which proxy mode to use: 'userspace' (older) or 'iptables' (faster) or 'ipvs' (experimental). If blank, use the best-available proxy (currently iptables).  If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy.") fs.Int32Var(o.config.IPTables.MasqueradeBit, "iptables-masquerade-bit", utilpointer.Int32PtrDerefOr(o.config.IPTables.MasqueradeBit, 14), "If using the pure iptables proxy, the bit of the fwmark space to mark packets requiring SNAT with.  Must be within the range [0, 31].") fs.DurationVar(&o.config.IPTables.SyncPeriod.Duration, "iptables-sync-period", o.config.IPTables.SyncPeriod.Duration, "The maximum interval of how often iptables rules are refreshed (e.g. '5s', '1m', '2h22m').  Must be greater than 0.")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: kube-proxy came up cleanly but mis-configured iptable rules due to a mis-match in the hostname and node name within the cluster triggering a default behavior. In this case the hostname returned the fqdn of the host and the cluster only had the hostname. This resulted in the following warning messages when kube-proxy started. Which then presents itself as pods not being able to access clusterip ip addresses with no route errors. What you expected to happen: Would it be better to fail to in this scenario rather than try to fall back to some default? I spent a good amount of time trying to track down the Error messages that were a result of this before finally working back to taking a closer at the Warning message. How to reproduce it (as minimally and precisely as possible): Start the kube-proxy with set to an invalid hostname should recreate this issue. Anything else we need to know?: While I think we can chalk up this issue to mis-configuration the purpose of opening this issue is really to have a conversation about the default behavior. I've seen several other issues filed that appear to be related to this behavior: , , . Would throwing an error and exiting non-zero be a better default action when the nodeIP is nil? What use case(s) does a default of support? If there must be a default behavior here maybe the IP address on the interface associated with the default gateway is a better default than ? Maybe the issue isn't in the proxier code but the kube-proxy itself should error when it fails to get node info rather than a warning? Environment: I'm not sure how much of this is important to this but here's what my environment was when I ran into this. Kubernetes version: 1.8.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): Centos 7 and Ubuntu 16.04
This is similar (though not the same symptom) to for which was supposed to fix it but was closed. One of the objections was up til now, kube-proxy has had no requirement that the hostname = nodename or even that the hostname be defined.


Question: fs.MarkDeprecated("resource-container", "This feature will be removed in a later release.") fs.StringVar(&o.config.ClientConnection.Kubeconfig, "kubeconfig", o.config.ClientConnection.Kubeconfig, "Path to kubeconfig file with authorization information (the master location is set by the master flag).") fs.Var(utilflag.PortRangeVar{Val: &o.config.PortRange}, "proxy-port-range", "Range of host ports (beginPort-endPort, single port or beginPort+offset, inclusive) that may be consumed in order to proxy service traffic. If (unspecified, 0, or 0-0) then ports will be randomly chosen.") <del> fs.StringVar(&o.config.HostnameOverride, "hostname-override", o.config.HostnameOverride, "If non-empty, will use this string as identification instead of the actual hostname.") </del> <ins> fs.StringVar(&o.hostnameOverride, "hostname-override", o.hostnameOverride, "If non-empty, will use this string as identification instead of the actual hostname.") </ins> fs.Var(&o.config.Mode, "proxy-mode", "Which proxy mode to use: 'userspace' (older) or 'iptables' (faster) or 'ipvs' (experimental). If blank, use the best-available proxy (currently iptables).  If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy.") fs.Int32Var(o.config.IPTables.MasqueradeBit, "iptables-masquerade-bit", utilpointer.Int32PtrDerefOr(o.config.IPTables.MasqueradeBit, 14), "If using the pure iptables proxy, the bit of the fwmark space to mark packets requiring SNAT with.  Must be within the range [0, 31].") fs.DurationVar(&o.config.IPTables.SyncPeriod.Duration, "iptables-sync-period", o.config.IPTables.SyncPeriod.Duration, "The maximum interval of how often iptables rules are refreshed (e.g. '5s', '1m', '2h22m').  Must be greater than 0.")

Making kube-proxy exit on failure to retrieve the node object might cause regressions in some legitimate use-cases. But I have no idea what those are. ?
I have a hard time imagining what the legit cases might be. I am confused, though - I only see this field being used in one very specific place, that would not have the effect you are claiming.
I can't see the error message from log? If kube-proxy failed to create kube-ipvs0 dummy interface, it will log something like "Failed to create dummy interface: kube-ipvs0, error:...".
This breaks Services of type and with . This is problematic for AWS environments where the kubelet is configured to override the hostname (AWS instances default to the short name and for the cloud provider integrations to work this needs to be overridden to the private fqdn of the host).
Any chance we could get this prioritized as a bug fix for v1.11?
cc
After putting a bit more thought into this, would it be possible to standardize on an environment variable that could be set through the downward api to override the default behavior of treating the hostname as the node name in the proxy? That would simplify the configuration on the kubeadm side (no need to override the command with the downward api, only setting the appropriate env variable), and would also provide a more accurate way to retrieve the actual node name for kube-proxy when deployed as a daemonset without the need to interact with the kubelet directly. I would suggest that we could use for this purpose. I'd also be happy to take a stab at implementing the changes needed for the v1.12 cycle.


Question: fs.MarkDeprecated("resource-container", "This feature will be removed in a later release.") fs.StringVar(&o.config.ClientConnection.Kubeconfig, "kubeconfig", o.config.ClientConnection.Kubeconfig, "Path to kubeconfig file with authorization information (the master location is set by the master flag).") fs.Var(utilflag.PortRangeVar{Val: &o.config.PortRange}, "proxy-port-range", "Range of host ports (beginPort-endPort, single port or beginPort+offset, inclusive) that may be consumed in order to proxy service traffic. If (unspecified, 0, or 0-0) then ports will be randomly chosen.") <del> fs.StringVar(&o.config.HostnameOverride, "hostname-override", o.config.HostnameOverride, "If non-empty, will use this string as identification instead of the actual hostname.") </del> <ins> fs.StringVar(&o.hostnameOverride, "hostname-override", o.hostnameOverride, "If non-empty, will use this string as identification instead of the actual hostname.") </ins> fs.Var(&o.config.Mode, "proxy-mode", "Which proxy mode to use: 'userspace' (older) or 'iptables' (faster) or 'ipvs' (experimental). If blank, use the best-available proxy (currently iptables).  If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy.") fs.Int32Var(o.config.IPTables.MasqueradeBit, "iptables-masquerade-bit", utilpointer.Int32PtrDerefOr(o.config.IPTables.MasqueradeBit, 14), "If using the pure iptables proxy, the bit of the fwmark space to mark packets requiring SNAT with.  Must be within the range [0, 31].") fs.DurationVar(&o.config.IPTables.SyncPeriod.Duration, "iptables-sync-period", o.config.IPTables.SyncPeriod.Duration, "The maximum interval of how often iptables rules are refreshed (e.g. '5s', '1m', '2h22m').  Must be greater than 0.")

Pods not getting scheduled on the new node joined to the cluster using kubespray The node joined to the cluster but new pods schedueld on the nodes are failing with error "get https://10.233.0.1 : dial tcp 10.233.0.1: 443 connection refused" Network plugin - calico kube-proxy running in ipvs mode new worker node, can connect to master server ip address over 6443 new worker node, cannot access cluster ip 10.233.0.1 over port 443 , nodelocaldns and kube-proxy pods are in running state on new node -proxy logs show error "invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP" Unable to write event: 'Post https://127.0.0.1:6443/api/v1/namespaces/default/events dial tcp 127.0.0.1:6443 connection refused" -ln output do not show any services addr show output do not show kube-ipvs0 and calico interfaces Can someone please help. Thanks


Question: driverName: driverName, } if shouldTimeout { <del> timeout := plugin.PluginClientTimeout + time.Millisecond </del> <ins> timeout := plugin.PluginClientTimeout + 10*time.Millisecond </ins> fakeDRADriverGRPCServer.timeout = &timeout }

This flakiness is reproducible on x86 and ppc64le architectures. https://prow.ppc64le- in /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/sig node /assign


Question: ] + select({ "@io_bazel_rules_go//go/platform:android": [ "docker_image_unsupported.go", <ins> "docker_sandbox_others.go", </ins> "docker_stats_unsupported.go", "helpers_unsupported.go", ], "@io_bazel_rules_go//go/platform:darwin": [ "docker_image_unsupported.go", <ins> "docker_sandbox_others.go", </ins> "docker_stats_unsupported.go", "helpers_unsupported.go", ], "@io_bazel_rules_go//go/platform:dragonfly": [ "docker_image_unsupported.go", <ins> "docker_sandbox_others.go", </ins> "docker_stats_unsupported.go", "helpers_unsupported.go", ], "@io_bazel_rules_go//go/platform:freebsd": [ "docker_image_unsupported.go", <ins> "docker_sandbox_others.go", </ins> "docker_stats_unsupported.go", "helpers_unsupported.go", ], "@io_bazel_rules_go//go/platform:linux": [ "docker_image_linux.go", <ins> "docker_sandbox_others.go", </ins> "docker_stats_linux.go", "helpers_linux.go", ], "@io_bazel_rules_go//go/platform:nacl": [ "docker_image_unsupported.go", <ins> "docker_sandbox_others.go", </ins> "docker_stats_unsupported.go", "helpers_unsupported.go", ], "@io_bazel_rules_go//go/platform:netbsd": [ "docker_image_unsupported.go", <ins> "docker_sandbox_others.go", </ins> "docker_stats_unsupported.go", "helpers_unsupported.go", ], "@io_bazel_rules_go//go/platform:openbsd": [ "docker_image_unsupported.go", <ins> "docker_sandbox_others.go", </ins> "docker_stats_unsupported.go", "helpers_unsupported.go", ], "@io_bazel_rules_go//go/platform:plan9": [ "docker_image_unsupported.go", <ins> "docker_sandbox_others.go", </ins> "docker_stats_unsupported.go", "helpers_unsupported.go", ], "@io_bazel_rules_go//go/platform:solaris": [ "docker_image_unsupported.go", <ins> "docker_sandbox_others.go", </ins> "docker_stats_unsupported.go", "helpers_unsupported.go", ], "@io_bazel_rules_go//go/platform:windows": [ "docker_image_windows.go", <ins> "docker_sandbox_windows.go", </ins> "docker_stats_windows.go", "helpers_windows.go", ],


Question: if !v.Served { continue } <del> // Defaults are not pruned here, but before being served. </del> spec, err := builder.BuildOpenAPIV2(crd, v.Name, builder.Options{V2: true}) <ins> // Defaults must be pruned here for CRDs to cleanly merge with the static // spec that already has defaults pruned spec.Definitions = handler.PruneDefaults(spec.Definitions) </ins> if err != nil { return nil, false, err }

Noticed while reproducing with client-side validation: Steps to reproduce: published openapi only contains expected metadata definitions: a CRD, e.g. simple- at published openapi: where the _v2 definition is referenced: For some reason, the same server publishing openapi for built-in types and custom resource definitions thinks the openapi definition differs. Reproduced against at least the following branches: master release-1.23 release-1.22 release-1.21 In addition to fixing this to make coherently refer to ObjectMeta definitions from built-in and custom types, we need a test for this scenario to ensure it doesn't creep back in.
/sig api-machinery /priority important-longterm
cc - this is the sort of thing I was remembering which prompted the question about merging openapi internally () This needs to be resolved and test coverage improved here ahead of making v2-facing openapi changes
is there a rule that we can validate and enforce here? i.e. are there cases where it's normal to have two metadata? Probably in the case where an extension apiserver has a different version of kubernetes that has a slightly different metadata. I wish we could catch somehow that this doesn't happen.
In a server with no aggregated API servers, we should never have multiple definitions for ObjectMeta (CRDs and built-in types should reference the same ObjectMeta)
/triage accepted
/cc
This is fixed by on master. For 1.24, 1.23, 1.22 and 1.21, cherry picks have been created: 1.24: 1.23: 1.22: 1.21:
also need it on 1.24?
Updated the list above.


Question: cpu: "30m" # BEGIN_PROMETHEUS_TO_SD - name: prometheus-to-sd-exporter <del> image: k8s.gcr.io/prometheus-to-sd:v0.5.0 </del> <ins> image: gke.gcr.io/prometheus-to-sd:v0.11.1-gke.1 </ins> # Request and limit resources to get guaranteed QoS. resources: requests:

the following images are using old client-go that would not work correctly starting 1.22+ if those images use in-cluster config(mounted service account token) to talk to kube-apiserver: if the above images use in-cluster config, they should use client-go 1.14+. xref: /assign /cc /sig instrumentation /priority important-soon
/triage accepted


Question: tests := []struct { new  core.Pod old  core.Pod <ins> opts PodValidationOptions </ins> err  string test string }{

See for context. This issue is creating to track work on the API validation changes. Validation relaxation of certain fields in the pod spec to allow for mutable scheduling directives.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scheduling
Docs PR:


Question: } } } <ins> func TestSetNominatedNodeName(t *testing.T) { tests := []struct { name                     string currentNominatedNodeName string newNominatedNodeName     string expectedPatchRequests    int expectedPatchData        string }{ { name:                     "Should make patch request to set node name", currentNominatedNodeName: "", newNominatedNodeName:     "node1", expectedPatchRequests:    1, expectedPatchData:        `{"status":{"nominatedNodeName":"node1"}}`, }, { name:                     "Should make patch request to clear node name", currentNominatedNodeName: "node1", newNominatedNodeName:     "", expectedPatchRequests:    1, expectedPatchData:        `{"status":{"nominatedNodeName":null}}`, }, { name:                     "Should not make patch request if nominated node is already set to the specified value", currentNominatedNodeName: "node1", newNominatedNodeName:     "node1", expectedPatchRequests:    0, }, { name:                     "Should not make patch request if nominated node is already cleared", currentNominatedNodeName: "", newNominatedNodeName:     "", expectedPatchRequests:    0, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { actualPatchRequests := 0 var actualPatchData string cs := &clientsetfake.Clientset{} cs.AddReactor("patch", "pods", func(action clienttesting.Action) (bool, runtime.Object, error) { actualPatchRequests++ patch := action.(clienttesting.PatchAction) actualPatchData = string(patch.GetPatch()) // For this test, we don't care about the result of the patched pod, just that we got the expected // patch request, so just returning &v1.Pod{} here is OK because scheduler doesn't use the response. return true, &v1.Pod{}, nil }) pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Status:     v1.PodStatus{NominatedNodeName: test.currentNominatedNodeName}, } preemptor := &podPreemptorImpl{Client: cs} if err := preemptor.setNominatedNodeName(pod, test.newNominatedNodeName); err != nil { t.Fatalf("Error calling setNominatedNodeName: %v", err) } if actualPatchRequests != test.expectedPatchRequests { t.Fatalf("Actual patch requests (%d) dos not equal expected patch requests (%d)", actualPatchRequests, test.expectedPatchRequests) } if test.expectedPatchRequests > 0 && actualPatchData != test.expectedPatchData { t.Fatalf("Patch data mismatch: Actual was %v, but expected %v", actualPatchData, test.expectedPatchData) } }) } } func TestUpdatePodCondition(t *testing.T) { tests := []struct { name                     string currentPodConditions     []v1.PodCondition newPodCondition          *v1.PodCondition expectedPatchRequests    int expectedPatchDataPattern string }{ { name:                 "Should make patch request to add pod condition when there are none currently", currentPodConditions: []v1.PodCondition{}, newPodCondition: &v1.PodCondition{ Type:               "newType", Status:             "newStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 1, 1, 1, 1, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","lastTransitionTime":".*","message":"newMessage","reason":"newReason","status":"newStatus","type":"newType"}]}}`, }, { name: "Should make patch request to add a new pod condition when there is already one with another type", currentPodConditions: []v1.PodCondition{ { Type:               "someOtherType", Status:             "someOtherTypeStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 11, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 10, 0, 0, 0, 0, time.UTC)), Reason:             "someOtherTypeReason", Message:            "someOtherTypeMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "newType", Status:             "newStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 1, 1, 1, 1, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"$setElementOrder/conditions":[{"type":"someOtherType"},{"type":"newType"}],"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","lastTransitionTime":".*","message":"newMessage","reason":"newReason","status":"newStatus","type":"newType"}]}}`, }, { name: "Should make patch request to update an existing pod condition", currentPodConditions: []v1.PodCondition{ { Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "currentType", Status:             "newStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 1, 1, 1, 1, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"$setElementOrder/conditions":[{"type":"currentType"}],"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","lastTransitionTime":".*","message":"newMessage","reason":"newReason","status":"newStatus","type":"currentType"}]}}`, }, { name: "Should make patch request to update an existing pod condition, but the transition time should remain unchanged because the status is the same", currentPodConditions: []v1.PodCondition{ { Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"$setElementOrder/conditions":[{"type":"currentType"}],"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","message":"newMessage","reason":"newReason","type":"currentType"}]}}`, }, { name: "Should not make patch request if pod condition already exists and is identical", currentPodConditions: []v1.PodCondition{ { Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, expectedPatchRequests: 0, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { actualPatchRequests := 0 var actualPatchData string cs := &clientsetfake.Clientset{} cs.AddReactor("patch", "pods", func(action clienttesting.Action) (bool, runtime.Object, error) { actualPatchRequests++ patch := action.(clienttesting.PatchAction) actualPatchData = string(patch.GetPatch()) // For this test, we don't care about the result of the patched pod, just that we got the expected // patch request, so just returning &v1.Pod{} here is OK because scheduler doesn't use the response. return true, &v1.Pod{}, nil }) pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Status:     v1.PodStatus{Conditions: test.currentPodConditions}, } updater := &podConditionUpdaterImpl{Client: cs} if err := updater.update(pod, test.newPodCondition); err != nil { t.Fatalf("Error calling update: %v", err) } if actualPatchRequests != test.expectedPatchRequests { t.Fatalf("Actual patch requests (%d) dos not equal expected patch requests (%d)", actualPatchRequests, test.expectedPatchRequests) } regex, err := regexp.Compile(test.expectedPatchDataPattern) if err != nil { t.Fatalf("Error compiling regexp for %v: %v", test.expectedPatchDataPattern, err) } if test.expectedPatchRequests > 0 && !regex.MatchString(actualPatchData) { t.Fatalf("Patch data mismatch: Actual was %v, but expected to match regexp %v", actualPatchData, test.expectedPatchDataPattern) } }) } } </ins>

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Unknown yet. Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated /sig scheduling /help
Can I give it a try?
Sure, feel free to take it. /assign ahmad-diaa /remove-help
I keep trying to get it fail locally by running the following command: but it never does, am I doing something wrong? Is there something else that I should be doing and would be more helpful?
Sometimes the flake is hard to reproduce in local env. You may have to read the error log of the flakes thoroughly to get some clues. BTW: you can also run the test multiple times using the directive:
After looking through the logs I came up with some information that might be useful As far as I understand, the reason for the flake is the of the medium priority pod not being cleared That is because the update request for removing returns the following error: So my current guess is that might be outdated in some cases As I am not really familiar with the logic, I would like to know whether or not those are valid points so that I could dig deeper. What do you think?
If that's the case, you can set here: You can raise a PR and manually try for ~20 times to see you can reproduce the flake.
found another issue tracking this too. it seems it has been flaky for a while.
Sorry, didn't mean to take this one for you. I worked the duplicate issue and submitted a PR. Feel free to review that PR, or submit your own if you have a different solution.
This sometimes happens. But we usually take the first PR unless it's clearly worked by someone. In terms of which style to resolve the Conflict error, I'm asking here:
Sorry, didn't mean to take this one for you. I worked the duplicate issue and submitted a PR. Feel free to review that PR, or submit your own if you have a different solution. No worries, I haven't started implementing yet and your PR seems to be following the correct approach as well according to this:


Question: } } } <ins> func TestSetNominatedNodeName(t *testing.T) { tests := []struct { name                     string currentNominatedNodeName string newNominatedNodeName     string expectedPatchRequests    int expectedPatchData        string }{ { name:                     "Should make patch request to set node name", currentNominatedNodeName: "", newNominatedNodeName:     "node1", expectedPatchRequests:    1, expectedPatchData:        `{"status":{"nominatedNodeName":"node1"}}`, }, { name:                     "Should make patch request to clear node name", currentNominatedNodeName: "node1", newNominatedNodeName:     "", expectedPatchRequests:    1, expectedPatchData:        `{"status":{"nominatedNodeName":null}}`, }, { name:                     "Should not make patch request if nominated node is already set to the specified value", currentNominatedNodeName: "node1", newNominatedNodeName:     "node1", expectedPatchRequests:    0, }, { name:                     "Should not make patch request if nominated node is already cleared", currentNominatedNodeName: "", newNominatedNodeName:     "", expectedPatchRequests:    0, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { actualPatchRequests := 0 var actualPatchData string cs := &clientsetfake.Clientset{} cs.AddReactor("patch", "pods", func(action clienttesting.Action) (bool, runtime.Object, error) { actualPatchRequests++ patch := action.(clienttesting.PatchAction) actualPatchData = string(patch.GetPatch()) // For this test, we don't care about the result of the patched pod, just that we got the expected // patch request, so just returning &v1.Pod{} here is OK because scheduler doesn't use the response. return true, &v1.Pod{}, nil }) pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Status:     v1.PodStatus{NominatedNodeName: test.currentNominatedNodeName}, } preemptor := &podPreemptorImpl{Client: cs} if err := preemptor.setNominatedNodeName(pod, test.newNominatedNodeName); err != nil { t.Fatalf("Error calling setNominatedNodeName: %v", err) } if actualPatchRequests != test.expectedPatchRequests { t.Fatalf("Actual patch requests (%d) dos not equal expected patch requests (%d)", actualPatchRequests, test.expectedPatchRequests) } if test.expectedPatchRequests > 0 && actualPatchData != test.expectedPatchData { t.Fatalf("Patch data mismatch: Actual was %v, but expected %v", actualPatchData, test.expectedPatchData) } }) } } func TestUpdatePodCondition(t *testing.T) { tests := []struct { name                     string currentPodConditions     []v1.PodCondition newPodCondition          *v1.PodCondition expectedPatchRequests    int expectedPatchDataPattern string }{ { name:                 "Should make patch request to add pod condition when there are none currently", currentPodConditions: []v1.PodCondition{}, newPodCondition: &v1.PodCondition{ Type:               "newType", Status:             "newStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 1, 1, 1, 1, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","lastTransitionTime":".*","message":"newMessage","reason":"newReason","status":"newStatus","type":"newType"}]}}`, }, { name: "Should make patch request to add a new pod condition when there is already one with another type", currentPodConditions: []v1.PodCondition{ { Type:               "someOtherType", Status:             "someOtherTypeStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 11, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 10, 0, 0, 0, 0, time.UTC)), Reason:             "someOtherTypeReason", Message:            "someOtherTypeMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "newType", Status:             "newStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 1, 1, 1, 1, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"$setElementOrder/conditions":[{"type":"someOtherType"},{"type":"newType"}],"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","lastTransitionTime":".*","message":"newMessage","reason":"newReason","status":"newStatus","type":"newType"}]}}`, }, { name: "Should make patch request to update an existing pod condition", currentPodConditions: []v1.PodCondition{ { Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "currentType", Status:             "newStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 1, 1, 1, 1, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"$setElementOrder/conditions":[{"type":"currentType"}],"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","lastTransitionTime":".*","message":"newMessage","reason":"newReason","status":"newStatus","type":"currentType"}]}}`, }, { name: "Should make patch request to update an existing pod condition, but the transition time should remain unchanged because the status is the same", currentPodConditions: []v1.PodCondition{ { Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"$setElementOrder/conditions":[{"type":"currentType"}],"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","message":"newMessage","reason":"newReason","type":"currentType"}]}}`, }, { name: "Should not make patch request if pod condition already exists and is identical", currentPodConditions: []v1.PodCondition{ { Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, expectedPatchRequests: 0, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { actualPatchRequests := 0 var actualPatchData string cs := &clientsetfake.Clientset{} cs.AddReactor("patch", "pods", func(action clienttesting.Action) (bool, runtime.Object, error) { actualPatchRequests++ patch := action.(clienttesting.PatchAction) actualPatchData = string(patch.GetPatch()) // For this test, we don't care about the result of the patched pod, just that we got the expected // patch request, so just returning &v1.Pod{} here is OK because scheduler doesn't use the response. return true, &v1.Pod{}, nil }) pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Status:     v1.PodStatus{Conditions: test.currentPodConditions}, } updater := &podConditionUpdaterImpl{Client: cs} if err := updater.update(pod, test.newPodCondition); err != nil { t.Fatalf("Error calling update: %v", err) } if actualPatchRequests != test.expectedPatchRequests { t.Fatalf("Actual patch requests (%d) dos not equal expected patch requests (%d)", actualPatchRequests, test.expectedPatchRequests) } regex, err := regexp.Compile(test.expectedPatchDataPattern) if err != nil { t.Fatalf("Error compiling regexp for %v: %v", test.expectedPatchDataPattern, err) } if test.expectedPatchRequests > 0 && !regex.MatchString(actualPatchData) { t.Fatalf("Patch data mismatch: Actual was %v, but expected to match regexp %v", actualPatchData, test.expectedPatchDataPattern) } }) } } </ins>

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: [TODO] Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig node These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.


Question: } } } <ins> func TestSetNominatedNodeName(t *testing.T) { tests := []struct { name                     string currentNominatedNodeName string newNominatedNodeName     string expectedPatchRequests    int expectedPatchData        string }{ { name:                     "Should make patch request to set node name", currentNominatedNodeName: "", newNominatedNodeName:     "node1", expectedPatchRequests:    1, expectedPatchData:        `{"status":{"nominatedNodeName":"node1"}}`, }, { name:                     "Should make patch request to clear node name", currentNominatedNodeName: "node1", newNominatedNodeName:     "", expectedPatchRequests:    1, expectedPatchData:        `{"status":{"nominatedNodeName":null}}`, }, { name:                     "Should not make patch request if nominated node is already set to the specified value", currentNominatedNodeName: "node1", newNominatedNodeName:     "node1", expectedPatchRequests:    0, }, { name:                     "Should not make patch request if nominated node is already cleared", currentNominatedNodeName: "", newNominatedNodeName:     "", expectedPatchRequests:    0, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { actualPatchRequests := 0 var actualPatchData string cs := &clientsetfake.Clientset{} cs.AddReactor("patch", "pods", func(action clienttesting.Action) (bool, runtime.Object, error) { actualPatchRequests++ patch := action.(clienttesting.PatchAction) actualPatchData = string(patch.GetPatch()) // For this test, we don't care about the result of the patched pod, just that we got the expected // patch request, so just returning &v1.Pod{} here is OK because scheduler doesn't use the response. return true, &v1.Pod{}, nil }) pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Status:     v1.PodStatus{NominatedNodeName: test.currentNominatedNodeName}, } preemptor := &podPreemptorImpl{Client: cs} if err := preemptor.setNominatedNodeName(pod, test.newNominatedNodeName); err != nil { t.Fatalf("Error calling setNominatedNodeName: %v", err) } if actualPatchRequests != test.expectedPatchRequests { t.Fatalf("Actual patch requests (%d) dos not equal expected patch requests (%d)", actualPatchRequests, test.expectedPatchRequests) } if test.expectedPatchRequests > 0 && actualPatchData != test.expectedPatchData { t.Fatalf("Patch data mismatch: Actual was %v, but expected %v", actualPatchData, test.expectedPatchData) } }) } } func TestUpdatePodCondition(t *testing.T) { tests := []struct { name                     string currentPodConditions     []v1.PodCondition newPodCondition          *v1.PodCondition expectedPatchRequests    int expectedPatchDataPattern string }{ { name:                 "Should make patch request to add pod condition when there are none currently", currentPodConditions: []v1.PodCondition{}, newPodCondition: &v1.PodCondition{ Type:               "newType", Status:             "newStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 1, 1, 1, 1, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","lastTransitionTime":".*","message":"newMessage","reason":"newReason","status":"newStatus","type":"newType"}]}}`, }, { name: "Should make patch request to add a new pod condition when there is already one with another type", currentPodConditions: []v1.PodCondition{ { Type:               "someOtherType", Status:             "someOtherTypeStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 11, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 10, 0, 0, 0, 0, time.UTC)), Reason:             "someOtherTypeReason", Message:            "someOtherTypeMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "newType", Status:             "newStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 1, 1, 1, 1, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"$setElementOrder/conditions":[{"type":"someOtherType"},{"type":"newType"}],"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","lastTransitionTime":".*","message":"newMessage","reason":"newReason","status":"newStatus","type":"newType"}]}}`, }, { name: "Should make patch request to update an existing pod condition", currentPodConditions: []v1.PodCondition{ { Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "currentType", Status:             "newStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 1, 1, 1, 1, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"$setElementOrder/conditions":[{"type":"currentType"}],"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","lastTransitionTime":".*","message":"newMessage","reason":"newReason","status":"newStatus","type":"currentType"}]}}`, }, { name: "Should make patch request to update an existing pod condition, but the transition time should remain unchanged because the status is the same", currentPodConditions: []v1.PodCondition{ { Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 1, 1, 1, 1, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "newReason", Message:            "newMessage", }, expectedPatchRequests:    1, expectedPatchDataPattern: `{"status":{"$setElementOrder/conditions":[{"type":"currentType"}],"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","message":"newMessage","reason":"newReason","type":"currentType"}]}}`, }, { name: "Should not make patch request if pod condition already exists and is identical", currentPodConditions: []v1.PodCondition{ { Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, }, newPodCondition: &v1.PodCondition{ Type:               "currentType", Status:             "currentStatus", LastProbeTime:      metav1.NewTime(time.Date(2020, 5, 13, 0, 0, 0, 0, time.UTC)), LastTransitionTime: metav1.NewTime(time.Date(2020, 5, 12, 0, 0, 0, 0, time.UTC)), Reason:             "currentReason", Message:            "currentMessage", }, expectedPatchRequests: 0, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { actualPatchRequests := 0 var actualPatchData string cs := &clientsetfake.Clientset{} cs.AddReactor("patch", "pods", func(action clienttesting.Action) (bool, runtime.Object, error) { actualPatchRequests++ patch := action.(clienttesting.PatchAction) actualPatchData = string(patch.GetPatch()) // For this test, we don't care about the result of the patched pod, just that we got the expected // patch request, so just returning &v1.Pod{} here is OK because scheduler doesn't use the response. return true, &v1.Pod{}, nil }) pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Status:     v1.PodStatus{Conditions: test.currentPodConditions}, } updater := &podConditionUpdaterImpl{Client: cs} if err := updater.update(pod, test.newPodCondition); err != nil { t.Fatalf("Error calling update: %v", err) } if actualPatchRequests != test.expectedPatchRequests { t.Fatalf("Actual patch requests (%d) dos not equal expected patch requests (%d)", actualPatchRequests, test.expectedPatchRequests) } regex, err := regexp.Compile(test.expectedPatchDataPattern) if err != nil { t.Fatalf("Error compiling regexp for %v: %v", test.expectedPatchDataPattern, err) } if test.expectedPatchRequests > 0 && !regex.MatchString(actualPatchData) { t.Fatalf("Patch data mismatch: Actual was %v, but expected to match regexp %v", actualPatchData, test.expectedPatchDataPattern) } }) } } </ins>

Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func(client kubeclientset.Interface, obj pkgruntime.Object) error { ingress := obj.(*extensionsv1beta1.Ingress) glog.V(4).Infof("Attempting to delete Ingress: %v", ingress) <del> err := client.Extensions().Ingresses(ingress.Namespace).Delete(ingress.Name, &metav1.DeleteOptions{}) </del> <ins> orphanDependents := false err := client.Extensions().Ingresses(ingress.Namespace).Delete(ingress.Name, &metav1.DeleteOptions{OrphanDependents: &orphanDependents}) </ins> return err })

Problem: When users sends a cascading deletion request to federation i.e they set orphanDependents=false, then federation should also set orphanDependents=false, when it sends the request to underlying clusters.
Fixed by


Question: // if the secret is not nil, compare it if expectedCall.expectedSecret != nil { <del> if !reflect.DeepEqual(expectedCall.expectedSecret, c.Request.Secret) { return allCalls, i, fmt.Errorf("Unexpected secret: expected %v, got %v", expectedCall.expectedSecret, c.Request.Secret) </del> <ins> if !reflect.DeepEqual(expectedCall.expectedSecret, c.Request.Secrets) { return allCalls, i, fmt.Errorf("Unexpected secret: expected %v, got %v", expectedCall.expectedSecret, c.Request.Secrets) </ins> } }

Seen in pull-kubernetes-e2e-gce-cos-alpha-features Specific job failure
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from the error message I don't know if it refers to the sig-storage error or to the sig-auth one
/sig storage
/kind failing-test
/assign From : ... I am not sure if it's related:
is the culprit. cannot be used inside a . This only gets caught when the test actually runs, otherwise Ginkgo never sees the incorrect .
I don’t understand why sometimes it can get the pv by namebut sometimes it failed
but then is a sig-auth test , not storage , sorry /sig auth this seems that was recently in
/remove sig-storage
/assign /priority important-soon
/unassign
is still failed. seems not fixing the whole problem. /reopen feel free to close if the ci is green.
Reopened this issue. <detailsIn response to : is still failed. /reopen feel free to close if the ci is green. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the failure now looks like it is just a sig storage issue:
cc
/assign looks like the new test is still an issue
I think this line is the problem: When the test creates the object, it may take time for the PV to be provisioned and to be bound to the PVC. So you should do another before this call so you can get the latest PVC object with the bound PV.
I opened to get PVC again before using .
Only do PersistentVolumeClaims.Get() is not enough, whether we need to wait for sometimes to wait it bound? because sometimes the testcase is successful and sometime is failed.
If the Pod was able to run, then the PVC has to be bound.
so I can't understand why sometimes the testcase is successful and sometimes is failed.


Question: // if the secret is not nil, compare it if expectedCall.expectedSecret != nil { <del> if !reflect.DeepEqual(expectedCall.expectedSecret, c.Request.Secret) { return allCalls, i, fmt.Errorf("Unexpected secret: expected %v, got %v", expectedCall.expectedSecret, c.Request.Secret) </del> <ins> if !reflect.DeepEqual(expectedCall.expectedSecret, c.Request.Secrets) { return allCalls, i, fmt.Errorf("Unexpected secret: expected %v, got %v", expectedCall.expectedSecret, c.Request.Secrets) </ins> } }

By default, the DelayBinding is false, it will wait for PVC bound during creation. This is weird.
It waits for the claim to be bound, but the test case still needs to refresh its pvc object.
Because of the refresh is async, so after the pvc was bound, we need to get the pvc again, isn't it?


Question: // case 1: node1:SG1  node2:SG2  return SG1,SG2 // case 2: node1:SG1,SG2  node2:SG3,SG4  return SG1,SG3 // case 3: node1:SG1,SG2  node2:SG2,SG3  return SG1,SG2 <del> securityGroupName := srv.SecurityGroups[0]["name"] nodeSecurityGroupIDs.Insert(securityGroupName.(string)) </del> <ins> secGroupNames.Insert(srv.SecurityGroups[0]["name"].(string)) </ins> } <del> return nodeSecurityGroupIDs.List(), nil </del> <ins> secGroupIDs := make([]string, secGroupNames.Len()) for i, name := range secGroupNames.List() { secGroupID, err := groups.IDFromName(network, name) if err != nil { return []string{}, err } secGroupIDs[i] = secGroupID } return secGroupIDs, nil </ins> } // isSecurityGroupNotFound return true while 'err' is object of gophercloud.ErrResourceNotFound

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kubelet fails when Kubernetes master gets started using cloud-provider openstack und kubeadm init command. says: What you expected to happen: kubelet starts and manages the loadbalancer and its security group. How to reproduce it (as minimally and precisely as possible): Setup an instance on Openstack. Place the following cloud-config in And : As well as : Run: For further information: +-----------------+
--------------------------+ +-----------------+
--------------------------+ +-----------------+
--------------------------+ Environment: Kubernetes version (use ): 1.9.1 Cloud provider or hardware configuration: Openstack OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS Kernel (e.g. ): x86_64 Install tools: kubeadm
/sig cloud-provider
Looks like auto-probing for for (s), please see [1] and [2]. Apologies, the deprecation should have been handled better. [1] [2]
If I understand you correctly, this cloud-config would be enough: If I try this, no Loadbalancer gets created at all and the service which requests type LoadBalancer says: The service I use there is the nginx helm chart ().
Can you please comment? /assign
/sig openstack
From that, '' is your service name and '-202f-4d44-a538-' is your subnet id, can you ensure '-202f-4d44-a538-' exist? Is it "-c60c-48cb-a146-"? I suspect that failed to autoprobe subnet. Can you give me more logs from kube controller manager
Actually, -202f-4d44-a538- is a network, not a subnet. The subnet has the ID -c1d5-4cc1-8b98-
And here the relevant logs from kube-controller-manager,first: And then it keeps repeating:
Oh, I see. Oops, should get 'sub network id', not 'network id'. I will fix it soon.
I tried the fix in 1.9.2, but it does not work. I tried multiple setting combinations of providing and/or/neither , but none works. The Service does not get a LoadBalancer IP assigned.


Question: // case 1: node1:SG1  node2:SG2  return SG1,SG2 // case 2: node1:SG1,SG2  node2:SG3,SG4  return SG1,SG3 // case 3: node1:SG1,SG2  node2:SG2,SG3  return SG1,SG2 <del> securityGroupName := srv.SecurityGroups[0]["name"] nodeSecurityGroupIDs.Insert(securityGroupName.(string)) </del> <ins> secGroupNames.Insert(srv.SecurityGroups[0]["name"].(string)) </ins> } <del> return nodeSecurityGroupIDs.List(), nil </del> <ins> secGroupIDs := make([]string, secGroupNames.Len()) for i, name := range secGroupNames.List() { secGroupID, err := groups.IDFromName(network, name) if err != nil { return []string{}, err } secGroupIDs[i] = secGroupID } return secGroupIDs, nil </ins> } // isSecurityGroupNotFound return true while 'err' is object of gophercloud.ErrResourceNotFound

Here are the logs and settings for the config version neither providing nor , since it seemed that's the way you recommended to me: cat /etc/kubernetes/pki/cloud-config kubectl get nodes journalctl -u kubelet.service kubectl describe svc ingress-nginx-ingress-controller Can you please re-open the issue? Do you need any other information?
Is it ok when you disable manage-security-group(manage-security-group=false)?
And please review .
/reopen
Yes, when I disable "manage-security-group" it assigns a LoadBalancer correctly. It also works, as soon as I run . I tried to check , but since I don't know go, I don't really understand what you are doing there and how it relates to my issue.
I check gophercloud.ErrResourceNotFound is not a interface, so the '' will return error and excute "" which is 'error occurred finding security group: ...'.
Okay, so you "just" print a better error message, right? But my question is, why does the error occur at all? In my understanding the kubelet should create a security-group on itself, if configured manage-security-groups=true and do not provide a security-group. However I also tried to define a valid security group id on my own, but in that case I again find these logs in kubelet (same as described first):
Okay, so you "just" print a better error message, right? But my question is, why does the error occur at all? In my understanding the kubelet should create a security-group on itself, if configured manage-security-groups=true and do not provide a security-group. No, not 'just print a better error message', openstack cloud provider can ensure security group for LB. Firstly, openstack cloud provider check whether security group does not exist, if not exist, it will create security group for LB. The "gophercloud.ErrResourceNotFound" means that security group not exist.
Now OpenStack cloud provider has not "node-security-group" variable. You should not specify it.
Please check this note:
Well, to me it does not look like the removal of any resources is the problem, rather that they do not get created at all. I can offer you a direct debugging session using Google Hangout or so, maybe that way it is easier to detect the real problem?
Have you see this log?


Question: // case 1: node1:SG1  node2:SG2  return SG1,SG2 // case 2: node1:SG1,SG2  node2:SG3,SG4  return SG1,SG3 // case 3: node1:SG1,SG2  node2:SG2,SG3  return SG1,SG2 <del> securityGroupName := srv.SecurityGroups[0]["name"] nodeSecurityGroupIDs.Insert(securityGroupName.(string)) </del> <ins> secGroupNames.Insert(srv.SecurityGroups[0]["name"].(string)) </ins> } <del> return nodeSecurityGroupIDs.List(), nil </del> <ins> secGroupIDs := make([]string, secGroupNames.Len()) for i, name := range secGroupNames.List() { secGroupID, err := groups.IDFromName(network, name) if err != nil { return []string{}, err } secGroupIDs[i] = secGroupID } return secGroupIDs, nil </ins> } // isSecurityGroupNotFound return true while 'err' is object of gophercloud.ErrResourceNotFound

It means the security group we need to create does not exist, we should create it, not raise error, right? And I get 'non-interface type ErrResourceNotFound on left' error, 'err.(*gophercloud.ErrResourceNotFound)' is incorrect.
Yes, exactly. I also tried to fix the error manually, so first I created the sg-group . After that kubelet complained about empty pool-targets. After fixing that, it complained about invalid endpoint which pointed to the nginx server, don't know how to fix that one yet. I just want to point out: All that errors only occur if I activate manageSecurityGroups, otherwise the kubelet is able to assign a LoadBalancer to the IngressController.
Hi, how is going? I've deployed k8s 1.9.5 on OpenStack with kubespray and I'm struggling with this issue. If I do not set the LBAAS is created but it's not assigned to any Security Group. So, the service is not reachable from the outside world. I can fix it by hand creating the Security Group and assign it to the VIP. If I do set kubelet fails to start. According to it's a valid parameter... (?) And finally, if I do as in his last comment, the LBAAS remains in a PENDING state forever... Well, I wonder if there is any workaround to deal with this... I presume you're working on the fix, aren't you? Any information will be appreciate. Thanks
I'm hitting the same issue as yours with openstack loadbalancer with 1.9.4. Anyone know if fixes this issue? If so, when will it be part of a release?
Any updates on this?
I just tried to use this feature in kubernetes 1.10.2 and it still seems to be broken. The security groups do not get created and the service creation is stuck at pending without an ip address. I used this configuration and i also tried all the alternatives listed here. Either nothing gets created or the controller-manager crashes if i add node-security-group.
It's an issue with as well. The behaviour is as described by
Bump
fixes this for us.
So, If I do not enable manage-security-groups, This should work?


Question: // case 1: node1:SG1  node2:SG2  return SG1,SG2 // case 2: node1:SG1,SG2  node2:SG3,SG4  return SG1,SG3 // case 3: node1:SG1,SG2  node2:SG2,SG3  return SG1,SG2 <del> securityGroupName := srv.SecurityGroups[0]["name"] nodeSecurityGroupIDs.Insert(securityGroupName.(string)) </del> <ins> secGroupNames.Insert(srv.SecurityGroups[0]["name"].(string)) </ins> } <del> return nodeSecurityGroupIDs.List(), nil </del> <ins> secGroupIDs := make([]string, secGroupNames.Len()) for i, name := range secGroupNames.List() { secGroupID, err := groups.IDFromName(network, name) if err != nil { return []string{}, err } secGroupIDs[i] = secGroupID } return secGroupIDs, nil </ins> } // isSecurityGroupNotFound return true while 'err' is object of gophercloud.ErrResourceNotFound

I am getting the following error: Error creating load balancer (will retry): failed to ensure load balancer for service default/lbs-test: error creating LB floatingip {FloatingNetworkID:YYYY FloatingIP: PortID:XXXX FixedIP: TenantID:}: Resource not found
Yes. Don't specify and at all in config. The LoadBalancer(s) will by default apply the security rules in the security-group .
You need to specify to automatically add / alter security groups. I get the following error after I've upgraded to 1.9.10:
I ran into the same issue. It seems to be related to how k8's openstack driver deals with adding floating ips to the LB. For now I set the lb to internal in the service metadata then manually associate a floating ip . This could also be a queens magnum issue not providing all required cloud config info. I am not sure yet.


Question: return err } if !notMnt { <del> return nil </del> <ins> // testing original mount point, make sure the mount link is valid if _, err := ioutil.ReadDir(dir); err == nil { glog.V(4).Infof("azureFile - already mounted to target %s", dir) return nil } // mount link is invalid, now unmount and remount later glog.Warningf("azureFile - ReadDir %s failed with %v, unmount this directory", dir, err) if err := b.mounter.Unmount(dir); err != nil { glog.Errorf("azureFile - Unmount directory %s failed with %v", dir, err) return err } notMnt = true </ins> } var accountKey, accountName string if accountName, accountKey, err = b.util.GetAzureCredentials(b.plugin.host, b.secretNamespace, b.secretName); err != nil { return err

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: azure file plugin on Windows does not work after node restart, this is due to powershell cmdlet has lost account name/key after reboot, we should remove the invalid link and do the mount again after kubelet restart. What you expected to happen: azure file plugin on Windows should work after node restart How to reproduce it (as minimally and precisely as possible): has created a pod with azure file mount on Windows agent node could get following error: Anything else we need to know?: Environment: Kubernetes version (use ): v1.9, v1.10 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign /sig windows /sig azure
Added to v1.10 milestone since it is a critical bug fix. cc


Question: package flocker import ( <ins> "io/ioutil" "os" </ins> "testing" flockerClient "github.com/ClusterHQ/flocker-go"

Unit tests on our CI have apparently been failing for 2 weeks and no one noticed (oops!)... The PR that seems to have caused the failures is . The failing test is - . Full Test Log: I'm not sure why this is failing on our CI and not on shippable. Has anyone else seen something similar? That test only seems to fail on the second run through with . Our unit tests run in a container built specifically for testing k8s-mesos: The Dockerfile for the env: Is anyone else able to repro with the above env or their own?
/cc
that test was around a week ago, it's doesn't seem possible to me that it was failing since 2 weeks ago. Anyway, we have few duplications of this issue: this one And the PR where is working on: It's a little bit difficult to me to follow all the conversations in all the differente places about the same bug, could it be possible to unificate some? Since you provided the environment I will try and take a look to reproduce it locally. Thanks!
I think I've got a working patch in
Closing in favor of &


Question: package flocker import ( <ins> "io/ioutil" "os" </ins> "testing" flockerClient "github.com/ClusterHQ/flocker-go"

reports the following failure after syncing with latest code: --- FAIL: TestSetUpAtInternal (0.00s) Error Trace: Error: Not equal: "expected-to-be-set-properly" (expected) != "" (actual) FAIL FAIL 0.067s cc/
CC
Taking a look. What do you mean with failed sometimes? Is it a random failure?
Can you please provide more information. This test seems to be consistently green in our CI runs.
The test is failing on this assertion: but as you can see on the previous line I am checking that there was no error on that call. So the function is returning for the error checking but it's not setting . Only place where it can happen is here: Is the not been clean between runs?
When I said "the only place where it can happen" it was because the other s are being returned after a check. I think that there is no other way to get there.
PS: I run the test suite 100 times and I didn't get any error: (That branch is based on the branch)
can you test to see if the patch in fixes things for you?
See for reproducible test environment and failure logs.
Based on the solution in it sounds like this test creates files the first time through which causes the tests to fail on the second time through.