Question: srcs = [ "factory.go", "plugins.go", <del> "signal.go", "signal_windows.go", </del> ], importpath = "k8s.io/kubernetes/pkg/scheduler/factory", visibility = ["//visibility:public"],


Question: if [[ -n "${move_kubelet}" ]]; then # Move /var/lib/kubelet to e.g. /mnt # (the backing for empty-dir volumes can use a lot of space!) <ins> # (As with /var/lib/docker, only if it is a directory; skip if symlink) </ins> if [[ -d /var/lib/kubelet ]]; then mv /var/lib/kubelet ${move_kubelet}/ fi mkdir -p ${move_kubelet}/kubelet <del> ln -s ${move_kubelet}/kubelet /var/lib/kubelet </del> <ins> # Create symlink for /var/lib/kubelet, unless it is already a symlink if [[ ! -e /var/lib/kubelet ]]; then ln -s ${move_kubelet}/kubelet /var/lib/kubelet fi </ins> KUBELET_ROOT="${move_kubelet}/kubelet" fi

On AWS we (will) now run our bootstrap script on every boot. The function to initialize ephemeral volumes doesn't tolerate being re-run, and we probably shouldn't re-run it anyway for speed. This does preserve the contents, which may be a good thing to save user's data (it doesn't belong on ephemeral drives, but...). On the other hand, if the ephemeral drives are corrupted, we lose the opportunity to reformat them and recover. Arguably, if the ephemeral storage is corrupted the right thing to do is to terminate the instance entirely.


Question: HostConfig: hc, } <del> // Apply linux-specific options. if err := ds.applySandboxLinuxOptions(hc, c.GetLinux(), createConfig, image, securityOptSeparator); err != nil { </del> <ins> // Apply platform-specific options. if err := ds.applySandboxPlatformOptions(hc, c, createConfig, image, securityOptSeparator); err != nil { </ins> return nil, err }


Question: allErrs = append(allErrs, field.Invalid(fldPath.Child("searches"), dnsConfig.Searches, "must not have more than 256 characters (including spaces) in the search list")) } for i, search := range dnsConfig.Searches { <ins> // it is fine to have a trailing dot if strings.HasSuffix(search, ".") { search = search[0 : len(search)-1] } </ins> allErrs = append(allErrs, ValidateDNS1123Subdomain(search, fldPath.Child("searches").Index(i))...) } // Validate options.

What happened: When I create a pod with specified dnsConfig, the validation says my spec is invalid. And the error is like below: What you expected to happen: The spec should be accepted as it is totally fine to have trailing period. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.3
Despite it's possible to have domains with trailing periods in the field of the , I agree with about this topic: FQDN and domains are different things. Maybe enforcing this in kubernetes is not a bad idea since the resolver seems to ignore the trailing dots.
let’s say you have name server A, and the serach domain at client has , when you try to resolve name abc, the client will send to name server A. Now what happened if nameserver A can not resolve , it is going to try to append its own searches, for example, name server has its searches as , then it tries to resolve (Depends on the ndots options, this may be allowed). Instead, if you have the trailing period in searches, the second try won’t happen.
what I'm trying to point out is that my understanding is that the and doesn't matter if you add a trailing dot or not to the domain used in the field, thus server A in your example is always going to receive a query to an FQDN and not going to do a recursive search. If you do a test in your system you can capture the traffic and check that the query is always the same with or without a traling dot. Anyway, adding domains with dots is allowed so you are right, only wanted to go deeper into this topic and shared my thoughts :)


Question: git  glusterfs-client  iptables  <ins> ipset  </ins> jq  kmod  openssh-client

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Upgraded to Kuberenetes 1.9 and now IPVS mode in kube-proxy stopped working due to What you expected to happen: Normal startup as it's been working flawless in 1.8.5 How to reproduce it (as minimally and precisely as possible): Use kube-proxy with IPVS service mode on 1.9.0 hyperkube image from GCR Anything else we need to know?: If i exec into the running kube-proxy pod and do then restart the container kube-proxy starts in ipvs mode fine Environment: Kubernetes version (use ):Cloud provider or hardware configuration: On-prem OS (e.g. from /etc/os-release): CoreOS Container Linux by CoreOS stable (1576.4.0) Kernel (e.g. ): 4.13.16-coreos-r2 Install tools: Own Others:
/sig network
According to jell in #sig-network these images are to be considered experimental and not used in production and might be missing binarys? , could someone shed some light on this
To be more precise: hyperkube binary can hide some options of underlying applications (e.g. of fix for that: ) and whole docker image of hyperkube - as this issue shows, can miss some useful binaries. That's why I named it as a tool to do experiments, while for real deployments probably a better choice would be to use underlying applications directly using own docker image, carefully tuned.
Is this connected to and by chance the changes hasn't propagated down to the public images yet?
image is based on (which is based on ) (and ) has a separate image based on But as has included support, IMO also should be extended by command, after this line.
Seems this issue already got fixed? Can close now?
Where do you get the info it's fixed, all my kube-proxy instances is still failbacking to iptables due to missing ipset that was present in 1.8.5
It was fixed by merging So, as it's in master, the fix will land in next release.
Ok, so it's fixed in the next version. I'll just switch away from Hyperkube then. Thanks
FYI: I've rebuilt hyperkube image for CoreOS with installed. Just for testing purposes. Everything works for me with it.


Question: git  glusterfs-client  iptables  <ins> ipset  </ins> jq  kmod  openssh-client

Waiting for the next release. :relaxed:
So does have ipset. So please use that. However does not have ipset (see below). The commit above was merged well before we shipped v1.9.0, But hyperkube image does not use the debian-iptables, it uses debian-hyperkube-base image if you see Anyone want to log a PR for hyperkube image to have ipset?
Can we re-open this, only fixes this on debian, not amd64!
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
will need to reopen, then :)


Question: }{ { name:     "no error, resp w/ cert, no counter increase", <del> checkers: []deprecatedCertificateAttributeChecker{&testNegativeChecker{counterRaiser{testCounter, ""}}}, </del> <ins> checkers: []deprecatedCertificateAttributeChecker{&testNegativeChecker{counterRaiser{testCounter, "", ""}}}, </ins> resp:     httpResponseWithCert(), }, { name:     "no error, resp w/o cert, no counter increase", <del> checkers: []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, ""}}}, </del> <ins> checkers: []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, "", ""}}}, </ins> resp:     httpResponseNoCert(), }, { name:            "no error, resp w/ cert, counter increase", <del> checkers:        []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, ""}}}, </del> <ins> checkers:        []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, "", ""}}}, </ins> resp:            httpResponseWithCert(), counterIncrease: true, }, { name:     "unrelated error, no resp, no counter increase", <del> checkers: []deprecatedCertificateAttributeChecker{&testNegativeChecker{counterRaiser{testCounter, ""}}}, </del> <ins> checkers: []deprecatedCertificateAttributeChecker{&testNegativeChecker{counterRaiser{testCounter, "", ""}}}, </ins> err:      fmt.Errorf("error"), }, { name:            "related error, no resp,  counter increase", <del> checkers:        []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, ""}}}, </del> <ins> checkers:        []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, "", ""}}}, </ins> err:             fmt.Errorf("error"), counterIncrease: true, },

go1.18 dropped support for certificates signed with the SHA-1 hash function - While this does improve security, and publicly trusted Certificate Authorities no longer issue such certificates, it is possible that user-provided certificates issued by private Certificate Authorities and used in webhooks or aggregated API servers do still use this. Similar to removal of support for certificates without subjectAltNames in go1.17, we need to: [x] add metrics when webhook/conversion/aggregated certificates with this hash are encountered - [x] document use of and the metric from 1 as a mitigation for clusters which still need to identify and replace use of such certificates - to release note and known issues [x] add audit annotation when SHA-1 cert is encountered - [x] update docs at and - /sig api-machinery instrumentation docs /priority important-soon /milestone v1.24 /kind bug documentation Upgraded 1.23 clusters to continue working as-is Configure an admission webhook, conversion webhook, or aggregated server with a serving certificate signed with the SHA-1 hash function. Upgrade to HEAD of 1.24. The certificate will no longer work by default No response 24 HEAD (go1.18-based) n/a n/a No response No response No response
it looks like NewMissingSANRoundTripperWrapperConstructor already has all the plumbing, and instead of missing SANs we could check the SignatureAlgorithm and output a distinct metric
had a not-yet-tested WIP branch at with a sketch of what it might look like
/assign who offered to pick this up
GitHub didn't allow me to assign the following users: stlaz. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign who offered to pick this up Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Indeed, please, you should trust Jordan :slightlysmilingface:
/assign
/triage accepted
These restrictions are on the GitHub side, this message is posted when it receives a specific error from the GitHub API while attempting to assign It is an annoying GitHub limitation, and one that seemingly does not exist in the web client vs the API.


Question: }{ { name:     "no error, resp w/ cert, no counter increase", <del> checkers: []deprecatedCertificateAttributeChecker{&testNegativeChecker{counterRaiser{testCounter, ""}}}, </del> <ins> checkers: []deprecatedCertificateAttributeChecker{&testNegativeChecker{counterRaiser{testCounter, "", ""}}}, </ins> resp:     httpResponseWithCert(), }, { name:     "no error, resp w/o cert, no counter increase", <del> checkers: []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, ""}}}, </del> <ins> checkers: []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, "", ""}}}, </ins> resp:     httpResponseNoCert(), }, { name:            "no error, resp w/ cert, counter increase", <del> checkers:        []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, ""}}}, </del> <ins> checkers:        []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, "", ""}}}, </ins> resp:            httpResponseWithCert(), counterIncrease: true, }, { name:     "unrelated error, no resp, no counter increase", <del> checkers: []deprecatedCertificateAttributeChecker{&testNegativeChecker{counterRaiser{testCounter, ""}}}, </del> <ins> checkers: []deprecatedCertificateAttributeChecker{&testNegativeChecker{counterRaiser{testCounter, "", ""}}}, </ins> err:      fmt.Errorf("error"), }, { name:            "related error, no resp,  counter increase", <del> checkers:        []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, ""}}}, </del> <ins> checkers:        []deprecatedCertificateAttributeChecker{&testPositiveChecker{counterRaiser{testCounter, "", ""}}}, </ins> err:             fmt.Errorf("error"), counterIncrease: true, },

adds the metrics. the release note on that issue describes the go change, links to go docs, describes use of the option, the new metrics, and the kube-apiserver logs what would you expect kubeadm to do with this? GODEBUG off by default and let users override GODEBUG on their own (is this possible)? GODEBUG by default on all clusters? GODEBUG by default only on upgrade? The risk of 1 is that a user upgrades without reading release notes and breaks a cluster that has SHA-1 certs in webhooks, etc The risk of 2 is that we propagate accepting insecure certificates longer, even in new clusters that we know don't have any pre-existing bad certs The risk of 3 is that it makes an upgraded 1.23 → 1.24 cluster and a new 1.24 cluster inconsistent
the majority of kubeadm users use kubeadm managed CA and certificates. there are less users that use external CA to sign their own certificates. sha1 attacks surfaced more than a before kubeadm's creation and Git hash collusion made the problem more known around 2017 - that was at a time when kubeadm did not support external CA. given these facts, an estimate on my side is that there are currently no kubeadm users that use sha1 certificates, but i would not exclude the possibility entirely. if such users exists, they are on they own and my vote goes for option 1, until they migrate their PKI to something secure. if the same users log tickets for us, our response would be "please use GODEBUG and migrate ASAP". all users should be able to see the notes in IMO, the PR needs an "action required" to appear on top of the release notes.
that's helpful, thanks agreed. weird, I set , but the label didn't get . the label manually.
and there's a way for them to set that envvar in kube-apiserver manifests when using kubeadm?
There's been a lot of intermittent github webhook delivery issues in the past few weeks or so, slash commands are going to be somewhat unreliable :/
they could use kubeadm patches to override the YAML manifest defaults - i.e. the contents of v1.Container -Env:


Question: if err != nil { return nil, err } <ins> name := accessor.GetName() if len(name) == 0 { return nil, fmt.Errorf("name is required") } </ins> outBytes, err := runtime.Encode(unstructured.UnstructuredJSONScheme, obj) if err != nil { return nil, err

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Providing an empty to the dynamic client's method results in a panic. What you expected to happen: An error should be returned on empty . How to reproduce it (as minimally and precisely as possible): Call with Anything else we need to know?: The expects a nonempty to be provided, but does not validate this. If an empty name is provided, this method will panic because it expects an , but the response will actually be an . Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: client-go 10.0.0
/sig api-machinery


Question: var _ streamProtocolHandler = &streamProtocolV2{} func (e *streamProtocolV2) stream(conn httpstream.Connection) error { <ins> var ( err                                                  error errorStream, remoteStdin, remoteStdout, remoteStderr httpstream.Stream ) </ins> headers := http.Header{} <ins> // set up all the streams first </ins> // set up error stream errorChan := make(chan error) headers.Set(api.StreamType, api.StreamTypeError) <del> errorStream, err := conn.CreateStream(headers) </del> <ins> errorStream, err = conn.CreateStream(headers) </ins> if err != nil { return err } <ins> // set up stdin stream if e.stdin != nil { headers.Set(api.StreamType, api.StreamTypeStdin) remoteStdin, err = conn.CreateStream(headers) if err != nil { return err } } // set up stdout stream if e.stdout != nil { headers.Set(api.StreamType, api.StreamTypeStdout) remoteStdout, err = conn.CreateStream(headers) if err != nil { return err } } // set up stderr stream if e.stderr != nil && !e.tty { headers.Set(api.StreamType, api.StreamTypeStderr) remoteStderr, err = conn.CreateStream(headers) if err != nil { return err } } // now that all the streams have been created, proceed with reading & copying // always read from errorStream </ins> go func() { message, err := ioutil.ReadAll(errorStream) switch {

We create the stdin stream and immediately start copying local stdin to the container. We do the same for stdout and stderr (create stream, start copying immediately). In the kubelet, we wait to receive all the streams before we start the exec and copy between the streams and the container. The spdy connection frame handling code has 5 frame workers, each with a queue of up to 50 frames. In the event that we try to push more than 50 frames to a single queue before all streams are ready, the 51st attempt to push a frame blocks until there is space available in the queue. But there won't be any space until the server starts reading from the streams, which won't happen until all the streams have been received, so we're stuck at this point. The fix is to create all the streams first, and only start copying after all the streams have been created and the server is ready to copy between the streams and the container. Originally reported here:
FYI
This seems pretty high priority.
It's an easy fix. has already tested the patch for OpenShift and it works. Are we willing to pull it in before is resolved, as that will take longer to put in place?
Here's the patch, for reference:
5 * 50 = 250 frames before you get stuck? Or did I misunderstand?
Oh, I guess all the frames go to the same worker for some reason?
anyway, I'm fine with taking this and cherry picking it. Order doesn't matter; someone has to rebase either way, right?
frames are assigned to each queue based on stream id, so if you sent 50 frames to the stdin stream, that queue fills up and the loop that processes incoming frames is blocked until something starts reading from the stream. Working on a PR now.
Gotcha, thanks. , Andy Goldstein wrote:
an update: I spent some time trying to create an e2e that would fail more often than not without this fix, but I haven't had any luck so far (I have it stream a copy of the kubectl binary from the local system to the pod). The issue that originally uncovered the problem was from running exec on a Windows system, where it reproduces consistently, but I haven't seen it fail much on Linux. I will keep trying...


Question: } By("Removing the test directory") <del> removeCmd := fmt.Sprintf("rm -r %s", volume.containerDir) _, err := framework.NodeExec(config.node0.Name, removeCmd) </del> <ins> removeCmd := fmt.Sprintf("rm -r %s", volume.hostDir) err := framework.IssueSSHCommand(removeCmd, framework.TestContext.Provider, config.node0) </ins> Expect(err).NotTo(HaveOccurred()) }

8 tests failed, 2 jobs failed, 88 builds failed. Failure stats cover 1 day time range '3 Aug 2017 03:31 UTC' to '4 Aug 2017 03:31 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
This should have been fixed by /cc
The test panic is fixed, but now it's failing because it can't SSH into the node.
/assign


Question: return newTaints } <del> // ensureNodeExistsByProviderIDOrInstanceID first checks if the instance exists by the provider id and then by calling instance id with node name func ensureNodeExistsByProviderIDOrInstanceID(instances cloudprovider.Instances, node *v1.Node) (bool, error) { exists, err := instances.InstanceExistsByProviderID(context.TODO(), node.Spec.ProviderID) if err != nil { providerIDErr := err _, err = instances.InstanceID(context.TODO(), types.NodeName(node.Name)) //TODO(anupn): Changing the check as InstanceID does not return error if err == nil { return false, nil </del> <ins> // ensureNodeExistsByProviderID checks if the instance exists by the provider id, // If provider id in spec is empty it calls instanceId with node name to get provider id func ensureNodeExistsByProviderID(instances cloudprovider.Instances, node *v1.Node) (bool, error) { providerID := node.Spec.ProviderID if providerID == "" { var err error providerID, err = instances.InstanceID(context.TODO(), types.NodeName(node.Name)) if err != nil { return false, err </ins> } <del> return false, fmt.Errorf("InstanceExistsByProviderID: Error fetching by providerID: %v Error fetching by NodeName: %v", providerIDErr, err) </del> <ins> if providerID == "" { glog.Warningf("Cannot find valid providerID for node name %q, assuming non existence", node.Name) return false, nil } </ins> } <del> return exists, nil </del> <ins> return instances.InstanceExistsByProviderID(context.TODO(), providerID) </ins> } func getNodeAddressesByProviderIDOrName(instances cloudprovider.Instances, node *v1.Node) ([]v1.NodeAddress, error) {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The change here removes exiternal id, but the call to instance id was not made used at all, it will always return false. At this time no node was able to register What you expected to happen: ProviderId was used in check How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment:
similar to are you using --cloud-provider in your controller manager? are you also setting it on your kubelet?
I'm using acs-engine to deploy the cluster. Config is as following: I've proposed a fix. PTAL
/sig cluster-lifecycle


Question: rbac.NewRule("get", "list", "watch").Groups(extensionsGroup).Resources("replicasets").RuleOrDie(), rbac.NewRule("get", "list", "watch").Groups(legacyGroup).Resources("replicationcontrollers").RuleOrDie(), rbac.NewRule("get", "list", "watch").Groups(policyGroup).Resources("poddisruptionbudgets").RuleOrDie(), <ins> rbac.NewRule("get", "list", "watch").Groups(appsGroup).Resources("statefulsets").RuleOrDie(), </ins> rbac.NewRule("update").Groups(policyGroup).Resources("poddisruptionbudgets/status").RuleOrDie(), eventsRule(), },

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No. What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail* the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: The is constantly flooded with the following error once per second: What you expected to happen: Log not flooded with error. How to reproduce it (as minimally and precisely as possible): Start a cluster at the above version. Check the on master. Anything else do we need to know: I didn't create any StatefulSets or PDBs. I only created some RCs.
Attn
cc This looks related to
Looks like an authorization issue. It doesn't seem to happen on a local cluster. From APIServer logs:


Question: originalNodeSecurityGroupIDs := lbaas.opts.NodeSecurityGroupIDs var err error <del> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, nodes) </del> <ins> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, lbaas.network, nodes) </ins> if err != nil { return fmt.Errorf("failed to find node-security-group for loadbalancer service %s/%s: %v", apiService.Namespace, apiService.Name, err) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kubelet fails when Kubernetes master gets started using cloud-provider openstack und kubeadm init command. says: What you expected to happen: kubelet starts and manages the loadbalancer and its security group. How to reproduce it (as minimally and precisely as possible): Setup an instance on Openstack. Place the following cloud-config in And : As well as : Run: For further information: +-----------------+
--------------------------+ +-----------------+
--------------------------+ +-----------------+
--------------------------+ Environment: Kubernetes version (use ): 1.9.1 Cloud provider or hardware configuration: Openstack OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS Kernel (e.g. ): x86_64 Install tools: kubeadm
/sig cloud-provider
Looks like auto-probing for for (s), please see [1] and [2]. Apologies, the deprecation should have been handled better. [1] [2]
If I understand you correctly, this cloud-config would be enough: If I try this, no Loadbalancer gets created at all and the service which requests type LoadBalancer says: The service I use there is the nginx helm chart ().
Can you please comment? /assign
/sig openstack
From that, '' is your service name and '-202f-4d44-a538-' is your subnet id, can you ensure '-202f-4d44-a538-' exist? Is it "-c60c-48cb-a146-"? I suspect that failed to autoprobe subnet. Can you give me more logs from kube controller manager
Actually, -202f-4d44-a538- is a network, not a subnet. The subnet has the ID -c1d5-4cc1-8b98-
And here the relevant logs from kube-controller-manager,first: And then it keeps repeating:
Oh, I see. Oops, should get 'sub network id', not 'network id'. I will fix it soon.
I tried the fix in 1.9.2, but it does not work. I tried multiple setting combinations of providing and/or/neither , but none works. The Service does not get a LoadBalancer IP assigned.


Question: originalNodeSecurityGroupIDs := lbaas.opts.NodeSecurityGroupIDs var err error <del> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, nodes) </del> <ins> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, lbaas.network, nodes) </ins> if err != nil { return fmt.Errorf("failed to find node-security-group for loadbalancer service %s/%s: %v", apiService.Namespace, apiService.Name, err) }

Here are the logs and settings for the config version neither providing nor , since it seemed that's the way you recommended to me: cat /etc/kubernetes/pki/cloud-config kubectl get nodes journalctl -u kubelet.service kubectl describe svc ingress-nginx-ingress-controller Can you please re-open the issue? Do you need any other information?
Is it ok when you disable manage-security-group(manage-security-group=false)?
And please review .
/reopen
Yes, when I disable "manage-security-group" it assigns a LoadBalancer correctly. It also works, as soon as I run . I tried to check , but since I don't know go, I don't really understand what you are doing there and how it relates to my issue.
I check gophercloud.ErrResourceNotFound is not a interface, so the '' will return error and excute "" which is 'error occurred finding security group: ...'.
Okay, so you "just" print a better error message, right? But my question is, why does the error occur at all? In my understanding the kubelet should create a security-group on itself, if configured manage-security-groups=true and do not provide a security-group. However I also tried to define a valid security group id on my own, but in that case I again find these logs in kubelet (same as described first):
Okay, so you "just" print a better error message, right? But my question is, why does the error occur at all? In my understanding the kubelet should create a security-group on itself, if configured manage-security-groups=true and do not provide a security-group. No, not 'just print a better error message', openstack cloud provider can ensure security group for LB. Firstly, openstack cloud provider check whether security group does not exist, if not exist, it will create security group for LB. The "gophercloud.ErrResourceNotFound" means that security group not exist.
Now OpenStack cloud provider has not "node-security-group" variable. You should not specify it.
Please check this note:
Well, to me it does not look like the removal of any resources is the problem, rather that they do not get created at all. I can offer you a direct debugging session using Google Hangout or so, maybe that way it is easier to detect the real problem?
Have you see this log?


Question: originalNodeSecurityGroupIDs := lbaas.opts.NodeSecurityGroupIDs var err error <del> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, nodes) </del> <ins> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, lbaas.network, nodes) </ins> if err != nil { return fmt.Errorf("failed to find node-security-group for loadbalancer service %s/%s: %v", apiService.Namespace, apiService.Name, err) }

It means the security group we need to create does not exist, we should create it, not raise error, right? And I get 'non-interface type ErrResourceNotFound on left' error, 'err.(*gophercloud.ErrResourceNotFound)' is incorrect.
Yes, exactly. I also tried to fix the error manually, so first I created the sg-group . After that kubelet complained about empty pool-targets. After fixing that, it complained about invalid endpoint which pointed to the nginx server, don't know how to fix that one yet. I just want to point out: All that errors only occur if I activate manageSecurityGroups, otherwise the kubelet is able to assign a LoadBalancer to the IngressController.
Hi, how is going? I've deployed k8s 1.9.5 on OpenStack with kubespray and I'm struggling with this issue. If I do not set the LBAAS is created but it's not assigned to any Security Group. So, the service is not reachable from the outside world. I can fix it by hand creating the Security Group and assign it to the VIP. If I do set kubelet fails to start. According to it's a valid parameter... (?) And finally, if I do as in his last comment, the LBAAS remains in a PENDING state forever... Well, I wonder if there is any workaround to deal with this... I presume you're working on the fix, aren't you? Any information will be appreciate. Thanks
I'm hitting the same issue as yours with openstack loadbalancer with 1.9.4. Anyone know if fixes this issue? If so, when will it be part of a release?
Any updates on this?
I just tried to use this feature in kubernetes 1.10.2 and it still seems to be broken. The security groups do not get created and the service creation is stuck at pending without an ip address. I used this configuration and i also tried all the alternatives listed here. Either nothing gets created or the controller-manager crashes if i add node-security-group.
It's an issue with as well. The behaviour is as described by
Bump
fixes this for us.
So, If I do not enable manage-security-groups, This should work?


Question: originalNodeSecurityGroupIDs := lbaas.opts.NodeSecurityGroupIDs var err error <del> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, nodes) </del> <ins> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, lbaas.network, nodes) </ins> if err != nil { return fmt.Errorf("failed to find node-security-group for loadbalancer service %s/%s: %v", apiService.Namespace, apiService.Name, err) }

I am getting the following error: Error creating load balancer (will retry): failed to ensure load balancer for service default/lbs-test: error creating LB floatingip {FloatingNetworkID:YYYY FloatingIP: PortID:XXXX FixedIP: TenantID:}: Resource not found
Yes. Don't specify and at all in config. The LoadBalancer(s) will by default apply the security rules in the security-group .
You need to specify to automatically add / alter security groups. I get the following error after I've upgraded to 1.9.10:
I ran into the same issue. It seems to be related to how k8's openstack driver deals with adding floating ips to the LB. For now I set the lb to internal in the service metadata then manually associate a floating ip . This could also be a queens magnum issue not providing all required cloud config info. I am not sure yet.


Question: // We want the IPs to end up in order by interface (in particular, we want eth0's // IPs first), but macs isn't necessarily sorted in that order so we have to // explicitly order by device-number (device-number == the "0" in "eth0"). <del> macIPs := make(map[int]string) </del> <ins> var macIDs []string macDevNum := make(map[string]int) </ins> for _, macID := range strings.Split(macs, "n") { if macID == "" { continue

What happened: does not handle "holes" in interface device-number correctly, dropping later interfaces. Create instance with eth0, eth1, eth2 ENIs as usual. Remove (eg: by selectively deleting pods) eth1. Only the addresses on eth0 are reported by NodeAddresses. What you expected to happen: Addresses on eth0 and eth2 should be reported.
/sig cloud-provider These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.


Question: "strings" "time" <del> "github.com/onsi/ginkgo/v2" </del> appsv1 "k8s.io/api/apps/v1" v1 "k8s.io/api/core/v1" <ins> storagev1 "k8s.io/api/storage/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <ins> types "k8s.io/apimachinery/pkg/types" </ins> utilerrors "k8s.io/apimachinery/pkg/util/errors" <ins> "k8s.io/apimachinery/pkg/util/uuid" </ins> clientset "k8s.io/client-go/kubernetes" <ins> "k8s.io/client-go/util/retry" </ins> "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2epv "k8s.io/kubernetes/test/e2e/framework/pv"

[X] test approval issue : [X] test pr : [x] two weeks soak end date : 11 Sept 2023 [x] test promotion pr : [x] remove endpoint from According to following APIsnoop query, there are two PersistentVolume and three PersistentVolumeClaim endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 13 endpoints. The e2e logs for this test are listed below. This query shows the following PV and PVC endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 13 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/cc
/reopen For tracking the rest of the steps to Promotion
Reopened this issue. <detailsIn response to : /reopen For tracking the rest of the steps to Promotion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: BOOTSTRAP_KUBECONFIG_FILE: $(yaml-quote "${WINDOWS_BOOTSTRAP_KUBECONFIG_FILE}") KUBEPROXY_KUBECONFIG_FILE: $(yaml-quote "${WINDOWS_KUBEPROXY_KUBECONFIG_FILE}") WINDOWS_INFRA_CONTAINER: $(yaml-quote "${WINDOWS_INFRA_CONTAINER}") <ins> WINDOWS_ENABLE_PIGZ: $(yaml-quote "${WINDOWS_ENABLE_PIGZ}") </ins> EOF }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : pigz for GCE Windows Why is this needed: Pigz improves the image layer extraction performance for multi-core systems. Pigz is usually preinstalled on Linux. This enhancement is scoped specifically for the GKE Windows node scripts. For Context:
/assign me
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
GitHub didn't allow me to assign the following users: me. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign me Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows /assign jeremyje
s/GKE/GCE/? as in kube-up?
Replaced but I don't understand the kube-up part.
is how one creates a GCE cluster from this repo. Otherwise we'd be talking about something that isn't part of kubernetes.


Question: } <ins> func TestValidatePersistentVolumeSourceUpdate(t *testing.T) { validVolume := testVolume("foo", "", api.PersistentVolumeSpec{ Capacity: api.ResourceList{ api.ResourceName(api.ResourceStorage): resource.MustParse("1G"), }, AccessModes: []api.PersistentVolumeAccessMode{api.ReadWriteOnce}, PersistentVolumeSource: api.PersistentVolumeSource{ HostPath: &api.HostPathVolumeSource{ Path: "/foo", Type: newHostPathType(string(api.HostPathDirectory)), }, }, StorageClassName: "valid", }) validPvSourceNoUpdate := validVolume.DeepCopy() invalidPvSourceUpdateType := validVolume.DeepCopy() invalidPvSourceUpdateType.Spec.PersistentVolumeSource = api.PersistentVolumeSource{ FlexVolume: &api.FlexVolumeSource{ Driver: "kubernetes.io/blue", FSType: "ext4", }, } invalidPvSourceUpdateDeep := validVolume.DeepCopy() invalidPvSourceUpdateDeep.Spec.PersistentVolumeSource = api.PersistentVolumeSource{ HostPath: &api.HostPathVolumeSource{ Path: "/updated", Type: newHostPathType(string(api.HostPathDirectory)), }, } scenarios := map[string]struct { isExpectedFailure bool oldVolume         *api.PersistentVolume newVolume         *api.PersistentVolume }{ "condition-no-update": { isExpectedFailure: false, oldVolume:         validVolume, newVolume:         validPvSourceNoUpdate, }, "condition-update-source-type": { isExpectedFailure: true, oldVolume:         validVolume, newVolume:         invalidPvSourceUpdateType, }, "condition-update-source-deep": { isExpectedFailure: true, oldVolume:         validVolume, newVolume:         invalidPvSourceUpdateDeep, }, } for name, scenario := range scenarios { errs := ValidatePersistentVolumeUpdate(scenario.newVolume, scenario.oldVolume) if len(errs) == 0 && scenario.isExpectedFailure { t.Errorf("Unexpected success for scenario: %s", name) } if len(errs) > 0 && !scenario.isExpectedFailure { t.Errorf("Unexpected failure for scenario: %s - %+v", name, errs) } } } </ins> func TestValidateLocalVolumes(t *testing.T) { scenarios := map[string]struct { isExpectedFailure bool

/kind bug Right now we do not validate PV updates so admin can update at any time. Kubernetes has no facility to reflect this update - it won't magically re-mount an existing volume to e.g. a different NFS server. IMO the PersistentVolumeSource to be read-only and set to stone once created.
/assign
I think it is a good idea to make the VolumeSource read-only but I am afraid ship has sailed in v1 and existing code/tooling that relies on the ability to modify PVs will break as a result of this change.
- do you have any advice on how we should proceed on fixing this issue?


Question: func TestSetDefaultDaemonSetSpec(t *testing.T) { defaultLabels := map[string]string{"foo": "bar"} period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug The change in a new field () to the Pod spec that is set to true by default. This is causing problems for DaemonSets in some situations. DaemonSets are using ControllerRevisions to keep track of previous configurations and each revision will contain among other things the Pod spec for the DaemonSet. If a DaemonSet is created with the enableServiceLinks property is set to true, the ControllerRevision will also contain this field. If the cluster is rolled back to a previous version of kubernetes, the DeamonSet struct does not have this field and so the ControllerRevision and the DaemonSet will no longer match. This leads to a new ControllerRevision being created and a rolling update of all pods in the DaemonSet. Would it be a better solution to rename the field to and enable service links whenever the field is not set? This would fix the problem with the DaemonSets unless the field is explicitly set before the rollback. This is causing the master upgrade tests to fail as reported in /sig apps /cc /cc /cc
I guess the change in podSpec and the defaulting will break all the workload resources as they are based on comparing the template's podSpec. ~Is the new value defaulted to true also for the DaemonSet's podSpec template? If not, controller might go hot loop like with mutating admission controllers.~ I don't like us making such changes to v1 podSpec. The other way around seems more sensible. /cc
Why is this a bad thing? We will continue to add fields to podSpec - the issue isn't the defaulting (this would happen regardless of which way it defaulted), it's that the controller can't really be version aware. The new value would default to true in the daemonset as well.
I'm not sure that a rollback requiring daemonsets to recreate is a bug. I think that's potentially "the best possible thing that we can do in that case".
/cc SIG Release blocking dashboard with upgrade test failures - https://k8s-
So the problem isn't the fact that a field has been . It the way in which the field was .


Question: func TestSetDefaultDaemonSetSpec(t *testing.T) { defaultLabels := map[string]string{"foo": "bar"} period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either ca So the problem isn't the fact that a field has been . It the way in which the field was . enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either case), you won't trigger the rolling restart on rollback or controller manager failoverse), you won't trigger the rolling restart on rollback or controller manager failover unless you explicitly opt out of the behavior. I don't see why the former behavior is more desirable than the latter.
For background/context, the choice between enable/disable was initially discussed here:
Are you saying that the issue is that default value is not the same as the Go "zero value" ? The concept of zero value is interesting but it forces APIs to bend into unnatural shapes to respect it. Since forever Kubernetes has explicitly said that we have version-specific default values which ARE NOT governed by zero values, but instead are governed by the API version. It is safe to assume that if a field is documented as having a default in a version, you can rely on that field being populated with that default, with a few exceptions.


Question: func TestSetDefaultDaemonSetSpec(t *testing.T) { defaultLabels := map[string]string{"foo": "bar"} period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

We generally ask for boolean fields to be "in the affirmative direction", as in this case. Now, we could revisit this. It would be a significant change in direction, but it's not impossible. But I don't think that solves your problem in general -- if this field were optional string, with a default value of "Hello there", you would be in the same boat right?
So the underlying issue is that we are defaulting into the PodSpec which gets adopted into all of the PodTemplateSpecs of every object. That sort of defaults the users declared intent. Why not just default the value into the Pod itself . Is the intention really to default this in all Templates or to make it optional in the Template and default it in the Pod itself?
/milestone v1.13
I can't speak to the ideology of whether to default the field in all PodSpecs or just the Pod, but your suggestion makes sense to me. I'm not sure of the past reasoning for adding other defaulted fields, most recently SchedulerName in Jan. 2017 (), to all PodSpecs instead of just to the Pod object. Did we not have these same tests in place then? This would effectively be saying we should never add new defaulted fields to PodSpec. If this is the plan, I can create a PR for the fix and I'll add a big warning comment to SetDefaults_PodSpec.
As an example of where we don't default the template we have a CronJob that embeds JobTemplate, we never default that template, only an actual Job gets defaulting. I agree with that not defaulting any templates does make sense.
Agree to only default this in pods. Agree to adding the comment. Most important defaulting happens in pods to keep higher level pod specs simpler. We may need to raise the general pattern in a discussion with api review in general, since this is a change from previous strategy, but the previous strategy was not explicitly chosen (to my recollection at least)
Hi all do we have an agreed path forward for fixing this issue in 1.13? if so can we have an ETA for the fix to land? the release team would like for the tests to get passing in the blocking suites in an effort to unmask any other failures that might show up once these are fixed.


Question: func TestSetDefaultDaemonSetSpec(t *testing.T) { defaultLabels := map[string]string{"foo": "bar"} period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

/assign
We have a PR with a fix that will hopefully merge today or tomorrow. I will keep an eye on the dashboard to make sure it fixes the failing daemonset tests.
Thanks Can we link the PR here plz. /cc
Reopening, because of the auto-close. Tests are not passing yet. /reopen
Reopening this issue. <detailsIn response to : Reopening, because of the auto-close. Tests are not passing yet. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
PR that should (hopefully) close this issue is here:
DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- if it doesn't pop up again - /close
Closing this issue. <detailsIn response to : DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return done, nil }) <del> if pollErr == wait.ErrWaitTimeout { </del> <ins> if pollErr != nil { </ins> t.Fatalf("failed to get the expected healthz status of OK for check: %s, error: %v, debug inner error: %v", checkName, pollErr, restErr) } }

{Failed === RUN TestKMSv2Healthz --- FAIL: TestKMSv2Healthz (167.38s) also shown in " TestKMSv2Healthz": 1, Not sure No response No response /sig auth
/assign /triage accepted Use for poll interval Change the poll timeout to 5m Update the poll error check (check if not nil)


Question: // Find out delta size expansionSize := (newSize.Value() - oldSize.Value()) <del> expansionSizeGB := int(volume.RoundUpSize(expansionSize, 1024*1024*1024)) </del> <ins> expansionSizeGB := int(volume.RoundUpSize(expansionSize, 1000*1000*1000)) </ins> // Make volume expansion request volumeExpandReq := &gapi.VolumeExpandRequest{Size: expansionSizeGB}

Glusterfs+Heketi use GB units for expanding volume. Using GiB incorrectly leads to expanding the volume to smaller size than user requested. /sig storage cc


Question: return utilerrors.NewAggregate(allErrs) } <del> // updateTaints updates taints of obj func (o TaintOptions) updateTaints(obj runtime.Object) error { </del> <ins> // updateTaints applies a taint option(o) to a node in cluster after computing the net effect of operation(i.e. does it result in an overwrite?), it reports back the end result in a way that user can easily interpret. func (o TaintOptions) updateTaints(obj runtime.Object) (string, error) { node, ok := obj.(*v1.Node) if !ok { return "", fmt.Errorf("unexpected type %T, expected Node", obj) } </ins> if !o.overwrite { <del> if err := validateNoTaintOverwrites(obj, o.taintsToAdd); err != nil { return err </del> <ins> if err := validateNoTaintOverwrites(node, o.taintsToAdd); err != nil { return "", err </ins> } } <del> newTaints, err := reorganizeTaints(obj, o.overwrite, o.taintsToAdd, o.taintsToRemove) </del> <ins> operation, newTaints, err := reorganizeTaints(node, o.overwrite, o.taintsToAdd, o.taintsToRemove) </ins> if err != nil { <del> return err } node, ok := obj.(*v1.Node) if !ok { return fmt.Errorf("unexpected type %T, expected Node", obj) </del> <ins> return "", err </ins> } node.Spec.Taints = newTaints <del> return nil </del> <ins> return operation, nil </ins> }

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.3.1185+-dirty Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.3.1185+-dirty" What happened: Kubectl is always reporting as node <XYZis tainted irrespective of operation when used with local cluster. taint nodes XYZ dedicated- node "XYZ" tainted What you expected to happen: It should have shown exact operation like tainted, untainted modified and overwritten but because of it is always returning tainted. How to reproduce it (as minimally and precisely as possible): taint nodes XYZ dedicated-
cc/
Regarding Yes I agree we should provide more clearer output/feedback. But providing something that makes output more confusing than existing is something would be even more concerning. So as I stated , it would be good to discuss/decide how the output/feedback should like for various operations for one and multi nodes scenarios.
I think having 1 return values is an exponential improvement on what we have now, even if they aren't perfect for all cases. Rather then a disproportionately long debate over a relatively simple improvement.... Could you guys suggest something simple that you're confident it will be easy to agree on, and we can implement it as a second pass ? we are happy to implement anything with n 1 return values :).
I think we'd be happy to consider any reasonable PR. I'm not sure why was closed; I didn't look at it though so I don't know if it solves the problem.
(hit close accidentally)
, I am not sure why was closed. But I raised a PR around 15 days ago() to address this.


Question: return oauth2.NewClient(oauth2.NoContext, tokenSource), nil } <ins> func (manager *GCEServiceManager) CreateDisk( project string, zone string, disk *compute.Disk) (*compute.Operation, error) { return manager.gce.service.Disks.Insert(project, zone, disk).Do() } func (manager *GCEServiceManager) GetDisk( project string, zone string, diskName string) (*compute.Disk, error) { return manager.gce.service.Disks.Get(project, zone, diskName).Do() } func (manager *GCEServiceManager) DeleteDisk( project string, zone string, diskName string) (*compute.Operation, error) { return manager.gce.service.Disks.Delete(project, zone, diskName).Do() } func (manager *GCEServiceManager) WaitForZoneOp(op *compute.Operation, zone string, mc *metricContext) error { return manager.gce.waitForZoneOp(op, zone, mc) } </ins>

Unit tests are needed for the GCE cloud provider disk interface.
Assigned to
/assign
GitHub didn't allow me to assign the following users: verult. Note that only can be assigned.. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details


Question: go_test( name = "go_default_test", <del> srcs = ["gce_test.go"], </del> <ins> srcs = [ "gce_disks_test.go", "gce_test.go", ], </ins> library = ":go_default_library", tags = ["automanaged"], <ins> deps = [ "//pkg/cloudprovider:go_default_library", "//vendor/google.golang.org/api/compute/v1:go_default_library", "//vendor/google.golang.org/api/googleapi:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", ], </ins> ) filegroup(


Question: - mountPath: /registration name: registration-dir - name: hostpath <del> image: quay.io/k8scsi/hostpathplugin:v1.2.0-rc3 </del> <ins> image: quay.io/k8scsi/hostpathplugin:v1.2.0-rc4 </ins> args: - "--v=5" - "--endpoint=$(CSI_ENDPOINT)"

What happened: The vresion of the hostpathplugin specified in the kubernetes e2e tests had a bug in cloning causing storageclass compare checks to always fail. As a result e2e tests could not pass for cloning in kubernetes. What you expected to happen: The version referencedin e2e tests should have a working hostpathplugin image to enable e2e testing of clone functinoality. How to reproduce it (as minimally and precisely as possible): Attempt to run the e2e cloning test: without modifying the hostpathplugin image version, test will fail due to storageClass mismatch even when set properly in the test. Anything else we need to know?: Fix in hostpathplugin is available in hostpathplugin:v1.2.0-rc4 There's also a required update to the provisioner side-car which has been submitted here:
/sig storage /assign


Question: AllowExpandedDNSConfig:                            utilfeature.DefaultFeatureGate.Enabled(features.ExpandedDNSConfig) || haveSameExpandedDNSConfig(podSpec, oldPodSpec), AllowInvalidLabelValueInSelector:                  false, AllowInvalidTopologySpreadConstraintLabelSelector: false, <ins> AllowMutableNodeSelectorAndNodeAffinity:           utilfeature.DefaultFeatureGate.Enabled(features.PodSchedulingReadiness), </ins> } if oldPodSpec != nil {

See for context. This issue is creating to track work on the API validation changes. Validation relaxation of certain fields in the pod spec to allow for mutable scheduling directives.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scheduling
Docs PR:


Question: "pod", klog.KObj(ps.pod)) continue } <del> if now.After(*ps.deadline) { </del> <ins> if cache.ttl != 0 && now.After(*ps.deadline) { </ins> klog.InfoS("Pod expired", "pod", klog.KObj(ps.pod)) if err := cache.removePod(ps.pod); err != nil { klog.ErrorS(err, "ExpirePod failed", "pod", klog.KObj(ps.pod))

The scheduler has a for the bind operation to succeed; if we don't get a response within 30s, the in-memory assignment of pod to node in the . A race condition will happen in the follow case: 1 is assigned to a node, scheduler cache is updated with the assignment, bind operation issued to apiserver. the apiserver is under huge pressure, bind takes more than 30s, scheduler expires the cached pod-to-node assignment. eventually succeeds, but because the apiserver is under huge pressure, the pod update with the node name takes a long time to propagate to the scheduler. the pod update took a long time to propagate and the cache entry expired, the scheduler is not aware that the assignment actually happened, and so it had no problem assigning a second pod to the same node that would otherwise not fit if the scheduler was aware that the first pod was eventually assigned to the node. On the scheduler side, what we need to do is make the 30s longer for large clusters, and ideally adaptable to cluster state. /sig scheduling No race conditions. Create a large scale cluster. No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
IIRC, the 30s timeout only starts running after we receive a 200 response for the binding. But yes, if it takes longer than 30s to receive the update, we would invalidate the cache. I don't know if there is particular reason to invalidate the cache at all. I suppose it was meant to guard against missing Pod deletion events. We need to improve that regardless. I would be in favor of dropping the timeout altogether.
ok, so the deadline is effectively for receiving the update. Perhaps making it longer, like 15min, is good first step before completely removing it just so we can easily revert in case something else goes wrong.
I sent to increase the timeout to 15min.


Question: "pod", klog.KObj(ps.pod)) continue } <del> if now.After(*ps.deadline) { </del> <ins> if cache.ttl != 0 && now.After(*ps.deadline) { </ins> klog.InfoS("Pod expired", "pod", klog.KObj(ps.pod)) if err := cache.removePod(ps.pod); err != nil { klog.ErrorS(err, "ExpirePod failed", "pod", klog.KObj(ps.pod))

We should backport this.
We should double check the history, but I think that in the past we were: asynchronously sending bind request updating the cache before sending the request and then the timeout was actually protecting us from the binding request actually not succeeding. So I agree with Aldo - that we should double check if this timeout is at all needed currently.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
We are probably good to remove this timeout altogether at this point. /remove-lifecycle stale
/good-first-issue
I see this issue was labeled as "good first issue" and I would like to start working on it!


Question: "pod", klog.KObj(ps.pod)) continue } <del> if now.After(*ps.deadline) { </del> <ins> if cache.ttl != 0 && now.After(*ps.deadline) { </ins> klog.InfoS("Pod expired", "pod", klog.KObj(ps.pod)) if err := cache.removePod(ps.pod); err != nil { klog.ErrorS(err, "ExpirePod failed", "pod", klog.KObj(ps.pod))

I would like to take this task
Plz assign to yourself to notify that you"re working on this.
/assign
/assign
done. thanks!
/assign
Could you please guide for this Issue are you suggesting we should not check cache before assigning the pod and need other way around?
FTR, I'm considering assigned to this issue, as they were the first to get assigned. You can see that in we increased the timeout. We want to get rid of the timeout. Then, we can just get rid of the logic that uses the timeout to remove some pods from the cache.
/reopen We still need to remove the code in 1.27
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/unassign
one more release before we can remove this code.
do you have plan to cherry pick for previous versions, e.g 1.23, 1.24?
did you mean 1.13 and 1.14?
oh, sorry my bad, I meant for 1.23 and 1.24
It is fixed in 1.23 , so it would be fixed in 1.24 too.
my understanding is the previous fix to increase the duration only reduce the chance of issue, but not completely avoid it, that's why the other 2 fixes and are needed, right?
Hi, is this issue still available?
yes, but since the likelihood that an issue happens is very low, I don't think it's worth the risk of cherry-picking at this point.
According to , we can remove the TTL code entirely in 1.27 I'll give the chance to to confirm if they can still do this. In the meantime: /unassign /unassign /remove-help
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.


Question: "pod", klog.KObj(ps.pod)) continue } <del> if now.After(*ps.deadline) { </del> <ins> if cache.ttl != 0 && now.After(*ps.deadline) { </ins> klog.InfoS("Pod expired", "pod", klog.KObj(ps.pod)) if err := cache.removePod(ps.pod); err != nil { klog.ErrorS(err, "ExpirePod failed", "pod", klog.KObj(ps.pod))

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale /unassign as we haven't heard from them.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
I want to help fix this /assign
Thanks. What is left is to remove the timeout logic altogether.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned".


Question: "pod", klog.KObj(ps.pod)) continue } <del> if now.After(*ps.deadline) { </del> <ins> if cache.ttl != 0 && now.After(*ps.deadline) { </ins> klog.InfoS("Pod expired", "pod", klog.KObj(ps.pod)) if err := cache.removePod(ps.pod); err != nil { klog.ErrorS(err, "ExpirePod failed", "pod", klog.KObj(ps.pod))

<detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen remove the expire logic
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen remove the expire logic Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think we can leave this closed. Entirely removing the timeouts has proven to be close to impossible without introducing regressions. We might need a few more iterations. Maybe we can reconsider when scheduling hints graduates to Stable.


Question: "create_clusterrole.go", "create_clusterrolebinding.go", "create_configmap.go", <ins> "create_cronjob.go", </ins> "create_deployment.go", "create_job.go", "create_namespace.go",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Why is this needed: Because of is not recommended to create cronjob (with flags) anymore. But there is no imperative command available for to create cronjob. /sig CLI <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/assign
GitHub didn't allow me to assign the following users: Pingan2017. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i wanna work on this. give me several days
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: func mustBeHealthy(t kubeapiservertesting.Logger, checkName, wantBodyContains string, clientConfig *rest.Config, excludes ...string) { t.Helper() var restErr error <del> pollErr := wait.PollImmediate(2*time.Second, 2*time.Minute, func() (bool, error) { </del> <ins> pollErr := wait.PollImmediate(1*time.Second, 5*time.Minute, func() (bool, error) { </ins> body, ok, err := getHealthz(checkName, clientConfig, excludes...) restErr = err if err != nil {


Question:
const unversionedWarningPre = `   <img src="http://kubernetes.io/img/warning.png" alt="WARNING"   <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"  width="25" height="25">  <img src="http://kubernetes.io/img/warning.png" alt="WARNING"   <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"  width="25" height="25">  <img src="http://kubernetes.io/img/warning.png" alt="WARNING"   <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"  width="25" height="25">  <img src="http://kubernetes.io/img/warning.png" alt="WARNING"   <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"  width="25" height="25">  <img src="http://kubernetes.io/img/warning.png" alt="WARNING"   <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"  width="25" height="25">
PLEASE NOTE: This document applies to the HEAD of the source tree

This includes but effects multiple docs.
This issue was also raised on the repository, see:
What's needed to fix this? Are the broken headers being generated automatically somewhere?
- The warning section is pre-pended to the md docs from the following script The fixes this issue, updating this script and all the docs... However, there is an open question as to whether "old" docs should be removed from this repo, which has prevented this PR from being merged.


Question: return portsToEndpoints } <ins> // ContainsIPv4Loopback returns true if the input is empty or one of the CIDR contains an IPv4 loopback address. func ContainsIPv4Loopback(cidrStrings []string) bool { if len(cidrStrings) == 0 { return true } // RFC 5735 127.0.0.0/8 - This block is assigned for use as the Internet host loopback address ipv4LoopbackStart := netutils.ParseIPSloppy("127.0.0.0") for _, cidr := range cidrStrings { if IsZeroCIDR(cidr) { return true } ip, ipnet, err := netutils.ParseCIDRSloppy(cidr) if err != nil { continue } if netutils.IsIPv6CIDR(ipnet) { continue } if ip.IsLoopback() { return true } if ipnet.Contains(ipv4LoopbackStart) { return true } } return false } </ins> // IsZeroCIDR checks whether the input CIDR string is either // the IPv4 or IPv6 zero CIDR func IsZeroCIDR(cidr string) bool {

Just similar to , for service with type nodeport , iptable proxy should add IsLoopback check for nodeAddress. Line 1104-1106 check nodeAddress is localhost address or not for ipvs mode proxy but for iptable mode proxy , when deal with nodePort type service, there is no similar check for nodeAddress since the use of localhost addresses for nodeports is the only know reason to require (), Just for the purpose of security ,this requirement shall be removed for proxy-mode=itable. Start a cluster with proxy-mode=iptable and some services Do; iptable -Ln | grep 127.0.0.1 just as says : ipv6+iptables , ::1 is not work for Nodeport all supporting proxy-mode=iptable any any /sig network /kind cleanup /area iptable
/sig network
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /area iptable Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind cleanup
maybe I'm missing something, but this was discussed in this issue and iptables needs to use localhost for nodeports, if we remove that we can break users. ipvs is different and it never needed loopback for nodeports
set sysctlRouteLocalnet when running in IPtable mode may still face a risk from CVE-2020-8558, so why not disable it in the future version?
because there are users that depend on that behavior, I can't remember the details , the referenced issue has them
I support finding a way to get to the end-state of not supporting nodeports on localhost, but I don't know the path.
/assign
it seems that localhost;nodeport is also used in blogs and forums the only tool we have for these situations is to add a feature gate and rollout the change in several releases, users can always set the sysctl value by themselves to recover previous behavior. should I create a small KEP to deprecate the behavior of setting the TTL? since is inconsistent with the other kube-proxy modes, there is a security risk and the workaround is quite simple, it seems a reasonable decision
I don't think we have a clear path to changing a defsult behavior, even one that is ill-advised.


Question: return portsToEndpoints } <ins> // ContainsIPv4Loopback returns true if the input is empty or one of the CIDR contains an IPv4 loopback address. func ContainsIPv4Loopback(cidrStrings []string) bool { if len(cidrStrings) == 0 { return true } // RFC 5735 127.0.0.0/8 - This block is assigned for use as the Internet host loopback address ipv4LoopbackStart := netutils.ParseIPSloppy("127.0.0.0") for _, cidr := range cidrStrings { if IsZeroCIDR(cidr) { return true } ip, ipnet, err := netutils.ParseCIDRSloppy(cidr) if err != nil { continue } if netutils.IsIPv6CIDR(ipnet) { continue } if ip.IsLoopback() { return true } if ipnet.Contains(ipv4LoopbackStart) { return true } } return false } </ins> // IsZeroCIDR checks whether the input CIDR string is either // the IPv4 or IPv6 zero CIDR func IsZeroCIDR(cidr string) bool {

As much as I'd like it to be the default, I think the only way to get there is to add a flag with compatible default behavior and encourage people to turn it on. e.g. add a kube-proxy config like defaulting to true, and encourage people/tools to set it to false.
I've found a better way ;)
/triage accepted


Question: return r, pods, svc } <ins> func JSONToYAMLOrDie(in []byte) []byte { data, err := yaml.JSONToYAML(in) if err != nil { panic(err) } return data } func streamYAMLTestData() (io.Reader, *api.PodList, *api.ServiceList) { pods, svc := testData() r, w := io.Pipe() go func() { defer w.Close() w.Write(JSONToYAMLOrDie([]byte(runtime.EncodeOrDie(latest.Codec, pods)))) w.Write([]byte("n---n")) w.Write(JSONToYAMLOrDie([]byte(runtime.EncodeOrDie(latest.Codec, svc)))) }() return r, pods, svc } </ins> type testVisitor struct { InjectErr error Infos     []*Info

If I stuff into a file and use that after -f, it works fine. cc
What's on stdin?
Bug, I'm only accepting json because I naively reused watch partitioner and we don't have test cases. Will fix tonight.
If that's the problem, why did it work when I gave it that file not over stdin?
Different code path. Stdin supports streaming json a la watch (multiple json objects in a stream), so you can do Unfortunately I missed yaml input and didn't use a segmenting yaml decoder (returning a stream of yaml objects).
Ah! Thanks for explanation. I think it would be good for us to ask for e2e runs before merging kubectl changes, since there's not quite enough coverage in test- yet.
Yeah, I'm slacking on that for these changes.
Fix incoming.
Fix provided. More tests to test-cmd... :)


Question: return defaultPlugins } <ins> type pluginIndex struct { index  int plugin v1beta2.Plugin } </ins> func mergePluginSet(defaultPluginSet, customPluginSet v1beta2.PluginSet) v1beta2.PluginSet { disabledPlugins := sets.NewString() <ins> enabledCustomPlugins := make(map[string]pluginIndex) </ins> for _, disabledPlugin := range customPluginSet.Disabled { disabledPlugins.Insert(disabledPlugin.Name) } <ins> for index, enabledPlugin := range customPluginSet.Enabled { enabledCustomPlugins[enabledPlugin.Name] = pluginIndex{index, enabledPlugin} } </ins> var enabledPlugins []v1beta2.Plugin if !disabledPlugins.Has("*") { for _, defaultEnabledPlugin := range defaultPluginSet.Enabled { if disabledPlugins.Has(defaultEnabledPlugin.Name) { continue } <ins> // The default plugin is explicitly re-configured, update the default plugin accordingly. if customPlugin, ok := enabledCustomPlugins[defaultEnabledPlugin.Name]; ok { klog.InfoS("Defaut plugin is explicitly re-configured and is overriding.", "plugin", defaultEnabledPlugin.Name) // update the default plugin in place to preserve order defaultEnabledPlugin = customPlugin.plugin // kick the plugin from the enabled list of custom plugins customPluginSet.Enabled = append(customPluginSet.Enabled[:customPlugin.index], customPluginSet.Enabled[customPlugin.index+1:]...) } </ins> enabledPlugins = append(enabledPlugins, defaultEnabledPlugin) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Start the scheduler with a scheduler config file which has a custom configuration for a specific plugin, the scheduler will fail to start and throw below exception, kube- error message, custom plugin config should take precedence over default plugin config. start a scheduler with a config file. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /assign
Does disabling the plugin and re-enabling the same plugin with a different weight work as a workaround?
This is WAI. You have to explicitly disable the default configuration.
I think so.
I not sure I agree with this, particularly, for someone who doesn't know details of code IMPL, e.g. an operator, Why bother to disable the plugin first and then re-enable the same plugin? there is no pointer to show they are different in a config file, as a developer, we know the reason behind the scenes, can we assume operator also know what that means to define a schedule config? By disabling the plugin and re-enable it means the operator has the knowledge this plugin is a default enabled plugin, should the operator check the source code to confirm this plugin's configuration? When this happens, given a different config for a specific plugin, should this deserve a corruption? I think scheduler should more robust to tolerate this.
We can improve the docs to include the default plugins. It's better for the API to be more predictable (always do the same thing). One thing to think about is order: if we do what your PR proposes, we are altering the order in which plugins are run, not just modifying the "weight" of the plugin.


Question: return defaultPlugins } <ins> type pluginIndex struct { index  int plugin v1beta2.Plugin } </ins> func mergePluginSet(defaultPluginSet, customPluginSet v1beta2.PluginSet) v1beta2.PluginSet { disabledPlugins := sets.NewString() <ins> enabledCustomPlugins := make(map[string]pluginIndex) </ins> for _, disabledPlugin := range customPluginSet.Disabled { disabledPlugins.Insert(disabledPlugin.Name) } <ins> for index, enabledPlugin := range customPluginSet.Enabled { enabledCustomPlugins[enabledPlugin.Name] = pluginIndex{index, enabledPlugin} } </ins> var enabledPlugins []v1beta2.Plugin if !disabledPlugins.Has("*") { for _, defaultEnabledPlugin := range defaultPluginSet.Enabled { if disabledPlugins.Has(defaultEnabledPlugin.Name) { continue } <ins> // The default plugin is explicitly re-configured, update the default plugin accordingly. if customPlugin, ok := enabledCustomPlugins[defaultEnabledPlugin.Name]; ok { klog.InfoS("Defaut plugin is explicitly re-configured and is overriding.", "plugin", defaultEnabledPlugin.Name) // update the default plugin in place to preserve order defaultEnabledPlugin = customPlugin.plugin // kick the plugin from the enabled list of custom plugins customPluginSet.Enabled = append(customPluginSet.Enabled[:customPlugin.index], customPluginSet.Enabled[customPlugin.index+1:]...) } </ins> enabledPlugins = append(enabledPlugins, defaultEnabledPlugin) } }

I'm leaning towards closing this ?
It's the key point I'm concerned about: commented here So let me ask a question: for a given input like: What's the expected output? {DefaultPlugin1, 1}, {DefaultPlugin2, 200}, {DefaultPlugin3, 3}, OR {DefaultPlugin1, 1}, {DefaultPlugin3, 3}, {DefaultPlugin2, 200}
Today, it is 2.
So in other words, if a user wants to customize a plugin's weight and keep its order, the only way is to use and give a full list of plugins? If so, that's not a good UX IMO.
Well, order doesn't matter for Score, so not really a problem.
All right, so the only case users could disable PluginXYZ while also enabling PluginXYZ is Score plugins, I'm fine with the order issue. Back to this issue, my 2 cents is to change the semantics of to a more enforced manner (proposed in this issue) in a new API version - v1beta2 or GA. As IMO it's a real UX improvement. WDYT?
I'm not convinced it's a UX improvement. If the user "enables" a filter plugin that was a default (say, by mistake), then its order changes. At least first disabling the plugin makes it more explicit. cc who has been working on decisions based on CC version
I guess it depends on the implementation. What is the full proposal?
/remove-kind bug /kind feature
That's a fair point, esp. if we adjust default plugins from version to version. I think a good UX is to throw an error/warning to indicate that a default Filter plugin is being re-enabled - this's useful no matter which semantic employes.
I'd like to: keep current logic to warn users that "PluginXYZ" is already registered (by default), if it's a non-Score plugin apply the "overridden" semantics to directive, for Score plugins only btw: let's get a consensus first and better adopt it in new CC version.
I propose (only for new version): Fail when plugin is already registered in the custom configuration, and only warning if the plugin was already part of defaults. Override any weight, preserving order I think that keeps everything consistent.


Question: return defaultPlugins } <ins> type pluginIndex struct { index  int plugin v1beta2.Plugin } </ins> func mergePluginSet(defaultPluginSet, customPluginSet v1beta2.PluginSet) v1beta2.PluginSet { disabledPlugins := sets.NewString() <ins> enabledCustomPlugins := make(map[string]pluginIndex) </ins> for _, disabledPlugin := range customPluginSet.Disabled { disabledPlugins.Insert(disabledPlugin.Name) } <ins> for index, enabledPlugin := range customPluginSet.Enabled { enabledCustomPlugins[enabledPlugin.Name] = pluginIndex{index, enabledPlugin} } </ins> var enabledPlugins []v1beta2.Plugin if !disabledPlugins.Has("*") { for _, defaultEnabledPlugin := range defaultPluginSet.Enabled { if disabledPlugins.Has(defaultEnabledPlugin.Name) { continue } <ins> // The default plugin is explicitly re-configured, update the default plugin accordingly. if customPlugin, ok := enabledCustomPlugins[defaultEnabledPlugin.Name]; ok { klog.InfoS("Defaut plugin is explicitly re-configured and is overriding.", "plugin", defaultEnabledPlugin.Name) // update the default plugin in place to preserve order defaultEnabledPlugin = customPlugin.plugin // kick the plugin from the enabled list of custom plugins customPluginSet.Enabled = append(customPluginSet.Enabled[:customPlugin.index], customPluginSet.Enabled[customPlugin.index+1:]...) } </ins> enabledPlugins = append(enabledPlugins, defaultEnabledPlugin) } }

I think this is what the last iteration of the PR is doing, but I only skimmed through it.
SGTM. To confirm, by "Fail when plugin is already registered in the custom configuration", this's only possible when users implement an out-of-tree plugin with a name identical to a default one, right?
I'm just referring to failing when the user repeats the same value twice. I think we can't prevent what you say.
This might be the reason why the order of the plugins should be reserved.
Unless it's score plugin like this issue, I don't know if there is any point to enable the default plugin in a config file. I think there is no need to distinguish score plugin with others, this will make the life easier, correct me?
Yeah, overall, is following this ideas, except that PR doesn't handle the repeated configuration, because if there is repeated value after the default plugin is overridden, it will hit the error below,
thanks for those valuable suggestion! So, we are still with beta1, and you suggest we can make some change in the new version, beta2? Is there any plan when this could happen? v1.22?
You are correct, there is no point, but it's good to have consistency. If it's allowed for Score, it should be allowed for other extension points. I agree Yes, that is the plan. It was 1.21, but we fall behind due to newer plans for deprecating plugins
let's say there are 3 default non-Score plugins A, B and C (in that order). I want to move B to the end of the list but I want to custom enable C too (for whatever reason; since we want to consistent with score plugins, let's allow this for other extension points too as said). if a non-Score plugin is custom disabled AND enabled, we should move it to the end of the list but if it's simply just enabled, then we should preserve order. if it's a score plugin, disabling and re-enabling has the same semantics as just simply enabling; they both will only change the weight. (idk if this has already been proposed; if it has been, consider this a vote of support :))
Are you saying that "disabled and enabled" should be treated the same as just "enabled"? That is, always move to the end of the list?


Question: return defaultPlugins } <ins> type pluginIndex struct { index  int plugin v1beta2.Plugin } </ins> func mergePluginSet(defaultPluginSet, customPluginSet v1beta2.PluginSet) v1beta2.PluginSet { disabledPlugins := sets.NewString() <ins> enabledCustomPlugins := make(map[string]pluginIndex) </ins> for _, disabledPlugin := range customPluginSet.Disabled { disabledPlugins.Insert(disabledPlugin.Name) } <ins> for index, enabledPlugin := range customPluginSet.Enabled { enabledCustomPlugins[enabledPlugin.Name] = pluginIndex{index, enabledPlugin} } </ins> var enabledPlugins []v1beta2.Plugin if !disabledPlugins.Has("*") { for _, defaultEnabledPlugin := range defaultPluginSet.Enabled { if disabledPlugins.Has(defaultEnabledPlugin.Name) { continue } <ins> // The default plugin is explicitly re-configured, update the default plugin accordingly. if customPlugin, ok := enabledCustomPlugins[defaultEnabledPlugin.Name]; ok { klog.InfoS("Defaut plugin is explicitly re-configured and is overriding.", "plugin", defaultEnabledPlugin.Name) // update the default plugin in place to preserve order defaultEnabledPlugin = customPlugin.plugin // kick the plugin from the enabled list of custom plugins customPluginSet.Enabled = append(customPluginSet.Enabled[:customPlugin.index], customPluginSet.Enabled[customPlugin.index+1:]...) } </ins> enabledPlugins = append(enabledPlugins, defaultEnabledPlugin) } }

No, just "enabled" would NOT move things to the end of the list (it will preserve order), but "disabled and enabled" would move to the end of the list.
ok, that sounds good
Another problem here is if we don't want to overridden anything but just config a plugin which is accidently a default plugin, something like below, scheduler will fail as well, I think we need to De-dup here instead of just a warning here. I guess this PR should already cover this, I will go back to check it again, at least one more testcase is needed to cover this case. keep current logic to warn users that "PluginXYZ" is already registered (by default), if it's a non-Score plugin
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle stale let me revive the PR later.
/remove-lifecycle stale


Question: podClient = f.PodClient() }) <del> It("should invoke init containers on a RestartNever pod", func() { </del> <ins> /* Release: v1.12 Testname: init-container-starts-app-restartnever-pod Description: Ensure that all InitContainers are started and all containers in pod are voluntarily terminated with exit status 0, and the system is not going to restart any of these containers when Pod has restart policy as RestartNever. */ framework.ConformanceIt("should invoke init containers on a RestartNever pod", func() { </ins> By("creating the pod") name := "pod-init-" + string(uuid.NewUUID()) value := strconv.Itoa(time.Now().Nanosecond())

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Evaluating e2e tests based on available options which are covering important aspects with user scenarios and eligible to promote for conformance. What you expected to happen: These 4 e2e tests evaluate different and based on . [NodeConformance] should invoke init containers on a RestartNever pod [NodeConformance] should invoke init containers on a RestartAlways pod [NodeConformance] should not start app containers if init containers fail on a RestartAlways pod [NodeConformance] should not start app containers and fail the pod if init containers fail on a RestartNever pod Anything else we need to know?: comment, Are these tests good to promote for conformance or do we need to look on other aspects as well ? are observed api endpoint hits during tests execution. (API coverage improvement) - InitContainer [NodeConformance] should invoke init containers on a RestartNever pod InitContainer [NodeConformance] should invoke init containers on a RestartAlways pod InitContainer [NodeConformance] should not start app containers if init containers fail on a RestartAlways pod InitContainer [NodeConformance] should not start app containers and fail the pod if init containers fail on a RestartNever pod /area conformance cc
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Go ahead and open a PR to ask this question. Yes == PR is merged, no == PR is closed (or amended to doc the reason these tests can't be promoted)


Question: m.updateAllocatedDevices(m.activePods()) allocatedDevicesUpdated = true } <del> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed) </del> <ins> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed, devicesToReuse[resource]) </ins> if err != nil { return err }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area hw-accelerators What happened: In the context of the device plugin, when using init containers in your pod spec, the current behavior will not reuse devices between init containers and regular containers. This is problematic as Pod scheduling is currently wrong: , the current predicate algorithm rightly assumes the maximum number of devices in use at one time is . However, the pod predicate ran on the node is the sum of devices used by regular and init containers. Meaning that your pod can bounce back between the scheduler and a node infinitely. Also given that many of the extended resources are typically more expensive that the built-in ones like CPU, Memory and Storage this bug has very real financial drawbacks in terms of resource usage (e.g: using 4 GPUs on a pod that only uses 2 GPUs at a time). Furthermore, this unexpected behavior was caught during implementation (but we did not catch the scheduling implications) and decision was taken to ignore it for the first implementation. We need to document it for 1.9 What you expected to happen: Devices can be reused by Init Containers Correct resource accounting In terms of the current behavior of the Allocate call to the device plugin, I'd expect the allocate call to be done before every Init containers start and once at the end of the last Init container (for regular containers). /assign
Yep, I realized the inconsistency between the predicate behavior of scheduler and that of node too. Having said that, considering we treat a pod as a whole when predicating currently, it's kind of difficult to reuse the runtime informations between init containers and regular containers. The real crux of the matter might be whether the scenario of requiring extended resources in init containers is common, if so, the issue should be addressed ASAP.
If I remembered correctly, the current design to improve DP includes a reconciliation in admit phase of Pod, it may be able to fix Allocate call for
can you point me to said design ? I'm not sure that the Allocate call for init containers is planned to be fixed as described.
Can you guys please point to the code where is happening. Looks to me that same logic is executed on both the scheduler and the node to calculate a pod's resource request and which is  On the node, function call tree looks like this: Is it possible to add few more words in explanation?


Question: m.updateAllocatedDevices(m.activePods()) allocatedDevicesUpdated = true } <del> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed) </del> <ins> allocDevices, err := m.devicesToAllocate(podUID, contName, resource, needed, devicesToReuse[resource]) </ins> if err != nil { return err }

PR that the mentioned change: Code in question is here:
Above looks incorrect description. Predicate that run on the scheduler and on the node is same i.e max(init, regular). Pod admission will pass at the node. Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'.
Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'. Absolutely correct, sorry to make you confused, I think Renaud name it mainly bacause the allocate func is supposed to be called during the predicate phase, that is, passed in as a part of PredicateAdmitHandler.
The original idea was onced listed in as a sub-task: Now it's removed (I don't know why ...), while I believe the idea was: device resources allocation during pod admit phase. This comment is also related if needed during reconcile in 1 and cache these devices info cached devices later during container create phase
Handle device resources allocation during pod admit phase. This comment (comment) is related Call Allocate() if needed during reconcile in 1 and cache these devices info Use cached devices later during container create phase Current code behaviors exactly as you described, but to address the issue, the key point might be to decide which devices of init containers could be reused by which regular containers, and generate cache accordingly in step2; when creating containers, we simply fetch runtime information from cache.
Thanks a lot for filing this issue, I think for 1.9, we probably want to list this as a known issue and may advise people to not request gpu resources in init containers. We can target to fix this problem in 1.10.
Agreed We don't have a clear solution for this tough. It's pretty clear to me that device resources allocation during pod admit phase is not the right model when init containers comes into the picture. GPUs are pretty simple example. On of the things we plan to do is scrub the GPU memory during the Allocate phase. If GPUs are re-used between initContainers and regular containers the current model doesn't fit.