Question: // attached to ELB created by a service. List always consist of at least // 1 member which is an SG created for this service or a SG from the Global config. // Extra groups can be specified via annotation, as can extra tags for any <del> // new groups. </del> <ins> // new groups. The annotation "ServiceAnnotationLoadBalancerSecurityGroups" allows for // setting the security groups specified. </ins> func (c *Cloud) buildELBSecurityGroupList(serviceName types.NamespacedName, loadBalancerName string, annotations map[string]string) ([]string, error) { var err error var securityGroupID string

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: We run Kuberentes on AWS. We are trying out service type. We use Terraform heavily for managing infrastructure on AWS. As a result we have several shared security groups with "perfect" rules. One of them is supposed to be attached to ELBs so our own IPs, partners, etc are whitelisted. Right now, it seems there isn't a way for Kubernetes to use the security group as is. There are several alternatives: Pass my existing security group as annotation. However, Kubernetes still tries to create one first, or use a global security group. If I don't provide any IP range, Kubernetes will whitelist . This seems insecure. Provide my own IP ranges as either annotation or . Then Kubernetes will create a security group for this ELB with the given IPs. This means I need to duplicate my IPs in all microservices. Provide my own IP ranges as either annotation or , and use a global security group. Then Kubernetes will modify the global SG with the given IPs. This seems problematic when lots of microservices share one SG but some want different IPs whitelisted. I want to entertain the idea below: Allow users to sepcify another security group to or object, which Kubernetes will take it and attach it to ELBs as is. If such security group is provided, skip other IP whitelisting steps. What you expected to happen: I can give Kubernetes existing security group(s). Kuberentes just attaches the security group(s) to the managed ELBs without modifying the rules. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: It may already be possible to achieve this, and I'm not aware of. Or there are other concerns so that Kubernetes didn't choose to implement this feature. Environment: Kubernetes version (use ): v1.7.1 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig aws
PR may help with this. . .
This problem could be really critical if one has the Kubernetes dashboard installed.


Question: // attached to ELB created by a service. List always consist of at least // 1 member which is an SG created for this service or a SG from the Global config. // Extra groups can be specified via annotation, as can extra tags for any <del> // new groups. </del> <ins> // new groups. The annotation "ServiceAnnotationLoadBalancerSecurityGroups" allows for // setting the security groups specified. </ins> func (c *Cloud) buildELBSecurityGroupList(serviceName types.NamespacedName, loadBalancerName string, annotations map[string]string) ([]string, error) { var err error var securityGroupID string

If for some reasons the service recreates AWS ELB, the related security group will be recreated as well with the "0.0.0.0/0" input/output masks and then botnets could find the ELB and . JFYI, in our case in the one of clusters with K8s 1.5, the deployment and replication controller were installed from the dashboard using this vector of attack. Then special pods were created and root volume mounted inside. The system crontab was modified by processes in these pods and the crypto miner appeared in all K8s nodes on the system level. After this another pods were installed and the miner was executed inside the Kubernetes pods in parallel with the system level.
just had some downtime on a production cluster because of this. i really need it too ;)
+1
As for me, it isn't a feature, it's a kind of bug.
Some comments on which may be relevant - please comment if an alternative annotation that did replace the default security-groups (as opposed to adding to them) would/wouldn't meet your use-case:
For me, I don't add any annotation, I just want to change its security group rule. After it will create a aws elb, and a specificed secruity group with 0.0.0.0, I change 0.0.0.0 to However, the security group rule will change back to 0.0.0.0 after some minutes. Is this by design? Is there any way for me keep my changes, thanks.
I guess this is part of how the controller works. If you look at , the PR fixes this problem by allowing you to specify the behaviour you want with an annotation. For the way Kubernetes works, the ELB is owned by Kubernetes and you should never be forced to modify the resource manually, thing that ultimately will not work.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale This is still relevant cause is not merged yet.
This is a good change.


Question: // attached to ELB created by a service. List always consist of at least // 1 member which is an SG created for this service or a SG from the Global config. // Extra groups can be specified via annotation, as can extra tags for any <del> // new groups. </del> <ins> // new groups. The annotation "ServiceAnnotationLoadBalancerSecurityGroups" allows for // setting the security groups specified. </ins> func (c *Cloud) buildELBSecurityGroupList(serviceName types.NamespacedName, loadBalancerName string, annotations map[string]string) ([]string, error) { var err error var securityGroupID string

The situation pre-1.13.0 is somewhat surprising because explicitly setting to an empty array like so: has no effect, and the Security Group Ingress is still .


Question: package sysctl import ( <ins> "fmt" goruntime "runtime" </ins> "k8s.io/apimachinery/pkg/util/version" "k8s.io/klog/v2" "k8s.io/kubernetes/pkg/proxy/ipvs" ) <ins> // refer to https://github.com/torvalds/linux/commit/122ff243f5f104194750ecbc76d5946dd1eec934. </ins> const ipLocalReservedPortsMinNamespacedKernelVersion = "3.16" var safeSysctls = []string{

E0320 18:35:17. 4656 ] "Failed to get kernel version." err="error reading osrelease file "/proc/sys/kernel/osrelease": open /proc/sys/kernel/osrelease: The system cannot find the path specified." handle windows node correctly. Run kubelet v1.27 beta+ on windows. This is an example v1.27.0-beta* N/A windows <details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node windows The issue came from
/assign
This Issue/PR has been marked as release blocker. Please have the fix merged by eod today otherwise the feature has to be reverted. For the Issue/PR owner: please have a revert/feature disable PR open today in case the fix failed to get in. Thank you! See the -- Kubernetes 1.27 release manager


Question: name: socket-dir - name: csi-provisioner <del> image: registry.k8s.io/sig-storage/csi-provisioner:v3.3.0 </del> <ins> image: registry.k8s.io/sig-storage/csi-provisioner:v3.4.0 </ins> args: - -v=5 - --csi-address=/csi/csi.sock

Seen in pull-kubernetes-e2e-gce-cos-alpha-features Specific job failure
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from the error message I don't know if it refers to the sig-storage error or to the sig-auth one
/sig storage
/kind failing-test
/assign From : ... I am not sure if it's related:
is the culprit. cannot be used inside a . This only gets caught when the test actually runs, otherwise Ginkgo never sees the incorrect .
I don’t understand why sometimes it can get the pv by namebut sometimes it failed
but then is a sig-auth test , not storage , sorry /sig auth this seems that was recently in
/remove sig-storage
/assign /priority important-soon
/unassign
is still failed. seems not fixing the whole problem. /reopen feel free to close if the ci is green.
Reopened this issue. <detailsIn response to : is still failed. /reopen feel free to close if the ci is green. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the failure now looks like it is just a sig storage issue:
cc
/assign looks like the new test is still an issue
I think this line is the problem: When the test creates the object, it may take time for the PV to be provisioned and to be bound to the PVC. So you should do another before this call so you can get the latest PVC object with the bound PV.
I opened to get PVC again before using .
Only do PersistentVolumeClaims.Get() is not enough, whether we need to wait for sometimes to wait it bound? because sometimes the testcase is successful and sometime is failed.
If the Pod was able to run, then the PVC has to be bound.
so I can't understand why sometimes the testcase is successful and sometimes is failed.


Question: name: socket-dir - name: csi-provisioner <del> image: registry.k8s.io/sig-storage/csi-provisioner:v3.3.0 </del> <ins> image: registry.k8s.io/sig-storage/csi-provisioner:v3.4.0 </ins> args: - -v=5 - --csi-address=/csi/csi.sock

By default, the DelayBinding is false, it will wait for PVC bound during creation. This is weird.
It waits for the claim to be bound, but the test case still needs to refresh its pvc object.
Because of the refresh is async, so after the pvc was bound, we need to get the pvc again, isn't it?


Question: defer cancel() go aggregatedManager.Run(testCtx.Done()) <del> require.True(t, waitForEmptyQueue(testCtx.Done(), aggregatedManager)) </del> <ins> require.True(t, waitForQueueComplete(testCtx.Done(), aggregatedManager)) </ins> // At this point external services have synced. Check if discovery document // includes the legacy resources

pull-kubernetes-unit sometimes fails with data race. TestDirty I think the test was 3 months ago with the flaking state. There are two issues. First, updating and reading a bool-type variable is not thread-safe. and there is also another problem checking the state before completely syncing it. No response /sig api-machinery
/assign
/triage accepted


Question: defer ss.vmssVMCache.Delete(key) klog.V(2).Infof("azureDisk - update(%s): vm(%s) - attach disk(%s)", nodeResourceGroup, nodeName, diskName) <del> _, err = ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, vm) </del> <ins> _, err = ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, newVM) </ins> if err != nil { detail := err.Error() if strings.Contains(detail, errLeaseFailed) || strings.Contains(detail, errDiskBlobNotFound) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When attach/detach azure disk in vmss, it updates whole VM info, including: networkProfile, storageProfile, while this would lead to race condition when there is also network related setting changing in k8s cloud provider in the mean time. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig azure /assign


Question: package util import ( <del> "bufio" </del> "crypto/rand" "fmt" <ins> "math/big" </ins> "regexp" "strings"

This issue was reported in the Impact Attackers may determine Bootstrap Token values based on timing-based side channels. This may in turn allow them to establish further attacks against a Kubernetes cluster, including joining nodes to an existing cluster. Feasibility of this attack is contingent on a number of factors including but not limited to the underlying hardware platform running the Kubernetes system, and the deployment landscape. Description Kubernetes Bootstrap Tokens are bearer token credentials employed to authenticate requests against the API server, when creating new clusters or joining new nodes to an existing cluster. NCC Group identified a number of functions that handle the Bootstrap Token, and whose implementations are not constant-time. The function , in package , generates Bootstrap Token secret values; it does so by sampling alphanumeric characters from a table, based on a random secret index. The access time of a table element, and resulting cache memory contents, may vary with the access index, which may assist attackers with inferring partial or full information about a token value. Recommendation When generating a secret token value, ensure that the selection process from the alphanumeric table is done in constant-time. The problem is akin to base64 encoding and decoding in constant-time, as implemented by the BearSSL TLS library. Given a byte in the 0..35 range, which we want to map to a digit range (ASCII values 48 to 57) and lower case character range (ASCII values 97 to 122), the constant-time selection process can be achieved as follows: . In order to increase the level of assurance that the modulo operation is performed in constant-time on various platforms, and despite changes to compilers, consider modeling the modulo operation as an explicit multiply and shift operation. The 1991 Granlund- Montgomery paper explains this technique. Avoid performing regular expression operations on secret data. Consider performing constant-time comparison of values instead. Component kube-apiserver Location Function in file Function in file Anything else we need to know? See umbrella issue for current status of all issues created from these findings. The vendor gave this issue an ID of NCC-E003660-TTV and it was finding 15 of the report under the kube-apiserver section.


Question: package util import ( <del> "bufio" </del> "crypto/rand" "fmt" <ins> "math/big" </ins> "regexp" "strings"

The vendor considers this issue Informational Risk, High Impact, Undetermined Exploitability To view the original finding, begin on page 42 of the Test Environment Kubernetes 1.24.3
/sig security
/sig cluster-lifecycle
/assign
PR that attempts to fix this:
/triage accepted poked #sig-security at slack for review ^


Question: }) // takes close to 140s to complete, so doesn't need to be marked [SLOW] <del> // TODO (aramase) remove phase 2 tag once phase 2 of dual stack is merged </del> ginkgo.It("should be able to reach pod on ipv4 and ipv6 ip", func() { serverDeploymentName := "dualstack-server" clientDeploymentName := "dualstack-client"

The test ""should have ipv4 and ipv6 node podCIDRs" depends on a feature that is optional (one of the ipam controllers in the controller manager) We have discussed this previously, but since this is actually affecting downstream CIs I propose to remove it form the e2e framework. For cloud providers, it can be moved to the own external cloud provider code, since the policy is to not host more tests with dependencies. For the feature itself, it can easily be replaced by a unit or an integration test. ref:
we have discussed it before, but now is affecting downstream CIs and is really a controller-manager test. /cc /cc for sig-testing visibility /sig network
Removing the test sounds reasonable as it's specific to cloud providers. I'll review if we have enough unit tests for this and open a PR to remove it from e2e.
We can add more if they don't exist, as a condition to remove the e2e
/assign
/triage accepted
It's not specific to cloud providers. It's specific to different network plugins; some require it and some don't.
I think that the comment is related to the e2e framework skipper, that only skips by cloud provider, but yeah, it is about CNIs


Question: v1.ResourceName("my.org"): resource.MustParse("9m"), }, }, <ins> skipLimitValueCheck: true, </ins> }, } for _, tc := range errorCase { t.Run(tc.name, func(t *testing.T) { <del> if errs := ValidateResourceRequirements(&tc.requirements, field.NewPath("resources")); len(errs) == 0 { </del> <ins> errs := ValidateResourceRequirements(&tc.requirements, field.NewPath("resources")) if len(errs) == 0 { </ins> t.Errorf("expected error") } <ins> validateNamesAndValuesInDescription(t, tc.requirements.Limits, errs, tc.skipLimitValueCheck, "limit") validateNamesAndValuesInDescription(t, tc.requirements.Requests, errs, tc.skipRequestValueCheck, "request") </ins> }) } } <ins> func validateNamesAndValuesInDescription(t *testing.T, r v1.ResourceList, errs field.ErrorList, skipValueTest bool, rl string) { for name, value := range r { containsName := false containsValue := false for _, e := range errs { if strings.Contains(e.Error(), name.String()) { containsName = true } if strings.Contains(e.Error(), value.String()) { containsValue = true } } if !containsName { t.Errorf("error must contain %s name", rl) } if !containsValue && !skipValueTest { t.Errorf("error must contain %s value", rl) } } } </ins> func TestValidateContainerResourceName(t *testing.T) { successCase := []struct { name         string

Created limit range object with this parameters then tried to create this pod got message: Error from server (Invalid): error when creating "pod-to-be-": Pod "nginx-to-be-removed-4" is invalid: spec.containers[0].resources.requests: Invalid value: "700m": must be less than or equal to cpu limit Pod should be created with Requests & Limits of cpu equal to 700m (since it is in range of min/max cpu limitrange object) create limitrange object and pod as in description No response <details</details<detailson premise/AKS </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
You have a default CPU limit of 10m, and no explicit CPU limit on the container, but a request larger than the limit. The request must always be less than or equal to the limit. Your pod needs to also set a higher limit, or the limit range needs to not set a default limit.
/remove-kind bug /kind support
/cc
but this is not obvious behavior since there is min & max set in limit range. This is limits (not min/max) that should be overwritten by pod. limits should be set to requests in this case imho
limit range limits what the user can set, it doesn't perform any mutations other than the defaulting and doesn't claim to. Min/max enforce that pods cannot set higher / lower. The default ensures that if the pod doesn't set a limit it will have one. personally I would remove the default limit, especially at 10m, this will aggressively limit pods. it would be, if the pod set a limit that would be a substantial breaking change, I don't think we could do that short of a v2 API and I'm not sure it would be correct behavior even then. we could make it opt-in, but then, why not just set them to match on the pod to begin with?


Question: Now we will increase the number of replicas from two to four: $ cd kubernetes <del> $ cluster/cloudcfg.sh resize dataController 4 </del> <ins> $ cluster/kubecfg.sh resize dataController 4 </ins> If you go back to the [demo website](http://localhost:8001/static/index.html) you should eventually see four boxes, one for each pod.

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Overall Since when has it been failing: Nov 20 between 8:41 PST and 12:43 PST (it's been "failing" before that as well, but now none of the tests are running at all) Testgrid link: Reason for failure: Looks like  failed (), so is not building the env properly Slack discussion: Anything else we need to know*: /sig cluster-lifecycle /area provider/gcp
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fixed here:
/assign


Question: return nil, false, err } <ins> if secret.DeletionTimestamp != nil { tokenErrorf(secret, "is deleted and awaiting removal") return nil, false, nil } </ins> if string(secret.Type) != string(bootstrapapi.SecretTypeBootstrapToken) || secret.Data == nil { tokenErrorf(secret, "has invalid type, expected %s.", bootstrapapi.SecretTypeBootstrapToken) return nil, false, nil

Resources can be deleted, but still exist in the API before they are reaped. They should not be valid during this time.
I'll add this to my pile. Thanks for filing :+1: FYI
FYI as well


Question: export KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. source "${KUBE_ROOT}/hack/lib/init.sh" <ins> git config http.https://gopkg.in.followRedirects true </ins> go get -u gopkg.in/mikedanese/gazel.v13/gazel "${GOPATH}/bin/gazel" -root="$(kube::realpath ${KUBE_ROOT})"

When I run it locally: Small test case to induce failure: I am using git version 2.11.0, if it makes any difference.
Also applies to
This is caused by the most recent git upgrade and affects all of our deps but gazel is the one we pull the most. how do we mitigate this?
We could possibly set in verify/update-gazel, but that feels a bit iffy because it has to be global. Possibly print out the command to fix it in case the go get fails?
Source code for is here . There doesn't seem to be an open issue for this.
I filed
says
Sorry for the trouble. We'll change a bit to ensure the manual configuration is not necessary.
Was this fixed in
No need to change the configuration anymore. We've changed itself to support the new git quirks.
thanks for fixing this!


Question: KUBE_PANIC_WATCH_DECODE_ERROR="${KUBE_PANIC_WATCH_DECODE_ERROR:-true}" export KUBE_PANIC_WATCH_DECODE_ERROR <del> # Handle case where OS has sha#sum commands, instead of shasum. if which shasum >/dev/null 2>&1; then SHA1SUM="shasum -a1" elif which sha1sum >/dev/null 2>&1; then SHA1SUM="sha1sum" else echo "Failed to find shasum or sha1sum utility." >&2 exit 1 fi </del> kube::test::find_dirs() { ( cd "${KUBE_ROOT}"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: unit tests with coverage succeed How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looking at : It is expecting group version string.
fixed in , clean up from


Question: namespaceOption := fmt.Sprintf("--namespace=%s", namespace) return framework.RunKubectl(namespace, append([]string{"exec", namespaceOption}, args...)...) } <ins> func getGmsaDomainIP(f *framework.Framework, podName string) string { output, _ := runKubectlExecInNamespace(f.Namespace.Name, podName, "powershell.exe", "--", "nslookup", gmsaDomain) re := regexp.MustCompile(`(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}`) idx := strings.Index(output, gmsaDomain) submatchall := re.FindAllString(output[idx:], -1) if len(submatchall) < 1 { framework.Logf("fail to get the ip of the gmsa domain") return "" } return submatchall[0] } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --There was a regression in containerd that required a fix for gmsa working with UNC paths. See the fix at We should have a test that validates this functionality. See for set up and place to add this test. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted /priority important-long-term /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --#### What happened: There was a regression in containerd that required a fix for gmsa working with UNC paths. See the fix at We should have a test that validates this functionality. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted /priority important-long-term /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : There was a regression in containerd that required a fix for gmsa working with UNC paths. See the fix at We should have a test that validates this functionality. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g.


Question: namespaceOption := fmt.Sprintf("--namespace=%s", namespace) return framework.RunKubectl(namespace, append([]string{"exec", namespaceOption}, args...)...) } <ins> func getGmsaDomainIP(f *framework.Framework, podName string) string { output, _ := runKubectlExecInNamespace(f.Namespace.Name, podName, "powershell.exe", "--", "nslookup", gmsaDomain) re := regexp.MustCompile(`(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}`) idx := strings.Index(output, gmsaDomain) submatchall := re.FindAllString(output[idx:], -1) if len(submatchall) < 1 { framework.Logf("fail to get the ip of the gmsa domain") return "" } return submatchall[0] } </ins>

): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted /priority important-long-term /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-longterm
/assign
if you have questions feel free to reachout on slack in sig-windows
Will do, thanks!
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


Question: namespaceOption := fmt.Sprintf("--namespace=%s", namespace) return framework.RunKubectl(namespace, append([]string{"exec", namespaceOption}, args...)...) } <ins> func getGmsaDomainIP(f *framework.Framework, podName string) string { output, _ := runKubectlExecInNamespace(f.Namespace.Name, podName, "powershell.exe", "--", "nslookup", gmsaDomain) re := regexp.MustCompile(`(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}`) idx := strings.Index(output, gmsaDomain) submatchall := re.FindAllString(output[idx:], -1) if len(submatchall) < 1 { framework.Logf("fail to get the ip of the gmsa domain") return "" } return submatchall[0] } </ins>

/lifecycle rotten [1]:
/remove-lifecycle rotten
/remove-lifecycle rotten
/assign
I have thought about this a bit and think it might be easiest to just expose a share on the domain controller. This does push more requirements into the configuration that needs to happen outside of the tests but it will make the tests a lot cleaner IMO.
/assign
GitHub didn't allow me to assign the following users: iXinqi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds", "type": "string" }, <del> "listMeta": { </del> <ins> "metadata": { </ins> "allOf": [ { "$ref": "#/components/schemas/io.k8s.apimachinery.pkg.apis.meta.v1.ListMeta"

Listing s on v1.31.0 returns an object with a field instead of the standard field for (list) objects: Listing s on v1.30.3 behaves correctly: This is also propagated to the Swagger spec (which is how I noticed it, since I maintain the Rust bindings for the client API). v1.31.0: v1.30.3: As explained above. v1.31.0 with feature gate enabled and runtime config . With kind, this is: the API server request as in the above curl command in any way you like as long as it lets you see the raw JSON response. No response <details</details<detailskind 0.23.0 </details<details</details<details</details<details</details<details</details
/sig api-machinery
It seems there may be a discrepancy in the API response format between v1.30.3 and v1.31.0. The object is expected to follow standard Kubernetes API conventions, which typically use the field for list objects.
This deviation might cause compatibility issues for API clients that rely on this structure.
If this deviation from the expected API structure wasn't intentional, it should be aligned with the standard conventions.
The presence of instead of could disrupt client integrations, as you've encountered with the Swagger spec and Rust bindings.
cc
/triage accepted
/assign
this must be fixed to json:"metadata" protobuf:"bytes,1,opt,name=metadata"`` the serialization will not work with standard clients
/priority critical-urgent
Shoot, that's clearly a mistake.


Question: if [[ -n "${KUBE_USER:-}" ]]; then local -r abac_policy_json="${src_dir}/abac-authz-policy.jsonl" remove-salt-config-comments "${abac_policy_json}" <del> sed -i -e "s@{{kube_user}}@${KUBE_USER}@g" "${abac_policy_json}" </del> <ins> sed -i -e "s/{{kube_user}}/${KUBE_USER}/g" "${abac_policy_json}" </ins> cp "${abac_policy_json}" /etc/srv/kubernetes/ fi

under kubetest.Up Nov 22 2023, last 7 runs failed No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
APIServer failed for
Yes it is failing for me too!
/sig testing
the generator config source must include the env to handle the "external" provider deprecation: /sig cloud-provider /area provider/gcp
/remove-sig testing /sig release
all other jobs seem green in the 1.29 blocking/informing, post that change. i will try sending a PR for this, but you need to find a non-USA approver if you want it merged before Monday. /assign
never mind, the generator is giving me errors: /unassign
/cc


Question: break } // If this is a client we know already, reuse the connection and goroutine. <del> svrConn, err := udp.getBackendConn(activeClients, cliAddr, proxier, service, myInfo.timeout) </del> <ins> svrConn, err := udp.getBackendConn(myInfo.activeClients, cliAddr, proxier, service, myInfo.timeout) </ins> if err != nil { continue }

Depends on UDP clients have a timeout (so we don't leak connections forever), but the test does not cover this. To do this, we probably need to track active connections, and plumb that all the way down to the goroutines spawned by the ProxyLoop routines. This is probably something we want to do anyway to be able to monitor.


Question: cmd := exec.CommandContext(ctx, filepath.Join(e.pluginBinDir, e.name), e.args...) cmd.Stdout, cmd.Stderr, cmd.Stdin = stdout, stderr, stdin <del> cmd.Env = []string{} for _, envVar := range e.envVars { cmd.Env = append(cmd.Env, fmt.Sprintf("%s=%s", envVar.Name, envVar.Value)) </del> <ins> var configEnvVars []string for _, v := range e.envVars { configEnvVars = append(configEnvVars, fmt.Sprintf("%s=%s", v.Name, v.Value)) </ins> } <ins> // Append current system environment variables, to the ones configured in the // credential provider file. Failing to do so may result in unsuccessful execution // of the provider binary, see https://github.com/kubernetes/kubernetes/issues/102750 // also, this behaviour is inline with Credential Provider Config spec cmd.Env = mergeEnvVars(e.environ(), configEnvVars) </ins> err = cmd.Run() if ctx.Err() != nil { return nil, fmt.Errorf("error execing credential provider plugin %s for image %s: %w", e.name, image, ctx.Err())

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I am attempting to pull an image from ECR using the new kubelet image credential provider feature which is an alpha features in Kubernetes v1.20. My CredentialProviderConfig file includes the environment variable AWSPROFILE which resides in the default () directory. The profile references a which allows me to execute a binary that returns a set of temporary AWS credentials. When I try running a pod with a container from ECR, I see the following errors in the kubelet logs: However, when I type the following into the command line on the kubelet: I get the following response: It doesn't appear the environment variables in my Config are being passed to the Kubelet when it executes the ecr-credential-provider binary. Below is Config file: I expect for the Kubelet to pass the environment variables in the Config file when it executes the ecr-credential-provider binary. I am using KinD to test this feature. Here is my config: No Kubernetes version (use ): 1.20 Cloud provider or hardware configuration: None OS (e.g: ): Ubuntu 20.10 Kernel (e.g. ): 5.10 Install tools: KinD Network plugin and version (if this is a network-related bug): Others: cc
/sig Node
/assign
from the code it appears that env variables are passed to the plugin from kubelet logs the plugin is only returning exit status 1, without any details about the error!!
Thanks Yes, I saw that. doesn't work. I wrote the following code to test this: If I uncomment , which imports the environment variables from the OS, it works.
Hello need some clarification. Is the intent of the above code to convey that isn't passing env vars correctly to the spawned process? Below code works as expected for me Also, could you tell if is a proprietary binary or a open source code something similar to because I want to use it in troubleshooting this issue.
Yes it is not passing the Envs to the spawned process.


Question: cmd := exec.CommandContext(ctx, filepath.Join(e.pluginBinDir, e.name), e.args...) cmd.Stdout, cmd.Stderr, cmd.Stdin = stdout, stderr, stdin <del> cmd.Env = []string{} for _, envVar := range e.envVars { cmd.Env = append(cmd.Env, fmt.Sprintf("%s=%s", envVar.Name, envVar.Value)) </del> <ins> var configEnvVars []string for _, v := range e.envVars { configEnvVars = append(configEnvVars, fmt.Sprintf("%s=%s", v.Name, v.Value)) </ins> } <ins> // Append current system environment variables, to the ones configured in the // credential provider file. Failing to do so may result in unsuccessful execution // of the provider binary, see https://github.com/kubernetes/kubernetes/issues/102750 // also, this behaviour is inline with Credential Provider Config spec cmd.Env = mergeEnvVars(e.environ(), configEnvVars) </ins> err = cmd.Run() if ctx.Err() != nil { return nil, fmt.Errorf("error execing credential provider plugin %s for image %s: %w", e.name, image, ctx.Err())

The code for ecr-credential-provider is in the AWS cloud controller repository,
Quick update, I had setup two test beds to reproduce the above issue, but both of them gave expected results and the issue was not reproducible. Below are my test setups: ECR Repository Test Bed 1 Execute from code and pass as environment variable Code Observed Result - Below credentials were generated Expected Result - Credentials should be generated Test Bed 2 Use as the credential provider within the kubelet, I had setup a Provider Config Expected result - Pods should be created based off custom image Observed result - Pods were created based off custom image Though, my test bed runs on kubernetes version , I could see that nothing significant has changed in since version Could you give me any other minor but important detail which you feel you could had missed out earlier which could help me in replicating the issue? Or could you double check if provider exists in your file?
i am using credential_process, in my profile. I am also using KinD for testing, but I don't think that should matter. I don't have a credentials file. I only have a config file.
This is the binary I'm calling with credential_process,
can you check kubelet logs for , It might be possible that is falling back to in-tree aws credential provider.
on my local setup credentials are being fetched. I'll setup another test bed with the Cognito provider which is using.
It is fetching credentials because it is falling back to other options available, It might be fetching from other credential provider if failed from external one. To confirm this you have to check kubelet logs.
In my env I just have one provider and there are no fallbacks. Though I was keeping an eye on kubelet logs while performing this activity and I saw no such logs. The creds were fetched in one shot via the profile which I had specified. And as I had mentioned earlier I don't see any issues in passing env vars to credential process. Also, I did try negative cases as well by passing invalid profile names and it did fail and pods weren't pulled from ECR
kubelet registers intree credential providers while startup, you need not to explicitly add any!! And I recently tried GCP credential provider, It does fall back to intree GCP credential provider after getting failed response from the one i set up.
Sure, makes sense.


Question: cmd := exec.CommandContext(ctx, filepath.Join(e.pluginBinDir, e.name), e.args...) cmd.Stdout, cmd.Stderr, cmd.Stdin = stdout, stderr, stdin <del> cmd.Env = []string{} for _, envVar := range e.envVars { cmd.Env = append(cmd.Env, fmt.Sprintf("%s=%s", envVar.Name, envVar.Value)) </del> <ins> var configEnvVars []string for _, v := range e.envVars { configEnvVars = append(configEnvVars, fmt.Sprintf("%s=%s", v.Name, v.Value)) </ins> } <ins> // Append current system environment variables, to the ones configured in the // credential provider file. Failing to do so may result in unsuccessful execution // of the provider binary, see https://github.com/kubernetes/kubernetes/issues/102750 // also, this behaviour is inline with Credential Provider Config spec cmd.Env = mergeEnvVars(e.environ(), configEnvVars) </ins> err = cmd.Run() if ctx.Err() != nil { return nil, fmt.Errorf("error execing credential provider plugin %s for image %s: %w", e.name, image, ctx.Err())

Though as I mentioned earlier I was keeping an eye on the kubelet logs and I didn't see the logs you had mentioned
i see in the logs when the kubelet starts, before attempting to run a pod that pulls an image from ECR.
Had a light bulb moment and I was able to reproduce this issue, it might not be applicable in your case but it could be worth trying out. In my local setup I ran the cluster as root and failed with error: E0614 18:19:27. ] Failed getting credential from external registry credential provider: error execing credential provider plugin ecr-credential-provider for image exit status 1 I have my aws credentials configure under my user () at path and since there is no similar configuration for the process failed. Since, I have access to code locally, I couple of log statements to print out current env vars and I could see that env var was set to :) Would it be possible for you to check is running as what user and is there any dissonance which can prohibit the user from successfully retrieving the credentials?
I am getting the error you mentioned, So far as I can tell, the kubelet is running as root and the file is in the directory. Again, I'm using KinD for testing. I'm not sure how KinD runs the kubelet. Below is the output from : The Go code I wrote to test this fails when using cmd.Env; it's only when I import the OS environment variables that it works. This may because I am using in my shared config file. credentialprocess is a mechanism to call another binary, e.g. , to get a set of temporary credentials from Cognito.
In order to simplify the sprawl and to pinpoint the offending component,could you remove the credential_process and Cognito and just try fetching creds via plain AWS credentials profile?
sure, however, supporting credential process is critical for my use case. I'm using Cognito as a gating mechanism to ECR. Only licensed users will be allowed to pull images from ECR. These users will not have IAM credentials (access key/secret key). They will authenticate to AWS using Cognito.
The following configuration works when I specify access key/secret key:
When I switch to a profile with credential process I get the following error:
isn't it the issue of ecr-credential-provider?


Question: cmd := exec.CommandContext(ctx, filepath.Join(e.pluginBinDir, e.name), e.args...) cmd.Stdout, cmd.Stderr, cmd.Stdin = stdout, stderr, stdin <del> cmd.Env = []string{} for _, envVar := range e.envVars { cmd.Env = append(cmd.Env, fmt.Sprintf("%s=%s", envVar.Name, envVar.Value)) </del> <ins> var configEnvVars []string for _, v := range e.envVars { configEnvVars = append(configEnvVars, fmt.Sprintf("%s=%s", v.Name, v.Value)) </ins> } <ins> // Append current system environment variables, to the ones configured in the // credential provider file. Failing to do so may result in unsuccessful execution // of the provider binary, see https://github.com/kubernetes/kubernetes/issues/102750 // also, this behaviour is inline with Credential Provider Config spec cmd.Env = mergeEnvVars(e.environ(), configEnvVars) </ins> err = cmd.Run() if ctx.Err() != nil { return nil, fmt.Errorf("error execing credential provider plugin %s for image %s: %w", e.name, image, ctx.Err())

This is the job of ecr-credential-provider to fetch credentials and return result in expected format to kubelet.
why does it work when I run it from the command line? Why does it work when I import envs from the OS?
you might have saved env on command line, same goes for OS.
That doesn't make sense to me. If I run it returns a properly formatted response. Same goes for the Go code I wrote. Does calling another binary, like credentialprocess, confuse the kubelet?
I am not familiar with aws credential process. Kubelet talks to with exec based plugin mechanism, is ecr-credential-provider calling the credential process you are talking about?
I have already validated that env vars are being passed to and there is no issue over there. If that wouldn't be the case then AWS credentials generation process via wouldn't work for you. I had debugged the code of kubelet myself and validated it + I wrote some logs in to print out the env vars and it was logging the passed vars. IMHO, now what remains to be seen here is why it fails with Cognito. I am going to create a test bed tomorrow replicating the exact setup which you have in the mean time if it's possible for you could you create a Linux VM and replicate your setup inside it? My intention is to see whether the Cognito based credential process goes through or not in a clean slate environment if it does then something on your local setup is interfering with passed env vars Thoughts?
Here's the execution flow ---- Does the above paints a fairly accurate picture? If not then feel free to correct :)
I included my KinD configuration in the beginning of this thread. Can you use that The binary has to be built from source, (or I can send it to you along with a temp username/password). There is an accompanying blog post for the binary at The flow is as describes above. The binary returns a json blob to the calling application that looks like this: During my testing (without the kubelet) I am able to get the proper response for the kubelet when referencing the AWS profile using or by setting to . However, when I try to pass the AWS profile in using exec.Env("AWS_PROFILE=default") I get and no further information. I have a feeling that doesn't properly chain or pass information back to the calling application.


Question: cmd := exec.CommandContext(ctx, filepath.Join(e.pluginBinDir, e.name), e.args...) cmd.Stdout, cmd.Stderr, cmd.Stdin = stdout, stderr, stdin <del> cmd.Env = []string{} for _, envVar := range e.envVars { cmd.Env = append(cmd.Env, fmt.Sprintf("%s=%s", envVar.Name, envVar.Value)) </del> <ins> var configEnvVars []string for _, v := range e.envVars { configEnvVars = append(configEnvVars, fmt.Sprintf("%s=%s", v.Name, v.Value)) </ins> } <ins> // Append current system environment variables, to the ones configured in the // credential provider file. Failing to do so may result in unsuccessful execution // of the provider binary, see https://github.com/kubernetes/kubernetes/issues/102750 // also, this behaviour is inline with Credential Provider Config spec cmd.Env = mergeEnvVars(e.environ(), configEnvVars) </ins> err = cmd.Run() if ctx.Err() != nil { return nil, fmt.Errorf("error execing credential provider plugin %s for image %s: %w", e.name, image, ctx.Err())

Let me replicate your setup in my local env and see if I can reproduce this issue. Also, could you tell what your AWS file look like? Is it something like this You can use placeholder values for sensitive information but correct the binary path if it's not
yes, that looks right. Ideally the cognito binary is located in a directory in the shell's PATH but it also works when you specify the full path in the config file like you've done. Are you planning to setup cognito in AWS? You'll need a cognito user pool and configure federation.
Yes, I am planning to setup Cognito Identity pool in my AWS Account.
So I replicated your setup on my local machine with slight change, instead of using AWS Cognito for generating credentials via I am calling a binary written by me which logs env vars to a specific path and prints out credentials in the format as prescribed by AWS. Below was my initial config file After I ran local cluster with this setup I noticed couple of log lines like below: I did some basic validation of perms + I even moved the binary to couple of well know locations but nothing worked Finally I modified my vars section of my deployment spec like below and it worked The logs which I had put in my custom credential provider printed following env vars validating the fact that, env vars are being passed In your logs do you see a log line something like above? If not then could you pass full contents of from start up until you create a Pod?
thanks for doing this. I never tried adding the PATH to the list of environment variables. I honestly didn't think it was necessary because I was referencing the full path to the binary in credential_process. I also never saw an error about not being able to find the executable in $PATH. I'll re-test after adding PATH as an ENV and let you know the results.
Adding the PATH did not work for me. I'm happy to send you the cognito binary with a username/password that you can use for testing. I have only tested this with KinD, but I don't think that is the issue here since it fails with exit code when when I try using .
^^ is my credential provider config
Thanks for the logs. I have cloned your code in my repo and I'll substitute my Cognito creds.


Question: cmd := exec.CommandContext(ctx, filepath.Join(e.pluginBinDir, e.name), e.args...) cmd.Stdout, cmd.Stderr, cmd.Stdin = stdout, stderr, stdin <del> cmd.Env = []string{} for _, envVar := range e.envVars { cmd.Env = append(cmd.Env, fmt.Sprintf("%s=%s", envVar.Name, envVar.Value)) </del> <ins> var configEnvVars []string for _, v := range e.envVars { configEnvVars = append(configEnvVars, fmt.Sprintf("%s=%s", v.Name, v.Value)) </ins> } <ins> // Append current system environment variables, to the ones configured in the // credential provider file. Failing to do so may result in unsuccessful execution // of the provider binary, see https://github.com/kubernetes/kubernetes/issues/102750 // also, this behaviour is inline with Credential Provider Config spec cmd.Env = mergeEnvVars(e.environ(), configEnvVars) </ins> err = cmd.Run() if ctx.Err() != nil { return nil, fmt.Errorf("error execing credential provider plugin %s for image %s: %w", e.name, image, ctx.Err())

If it doesn't work out then I'll reach out to you for creds
After several rounds of troubleshooting I was able to get working in my KinD environment. My credential loading setup is <-<-and below is my configuration of credential process Initially, I got the same error as yours, then I couple of custom log lines to print env vars in in directory and I could verify that the env vars specified in the credentials spec were indeed being passed. What I noticed in my KinD setup is that AWS SDK was logging an error because it couldn't find any "providers", though I could verify in the container that file and were indeed present. I altered my env vars section of the as below and it worked for me Apparently, what I noticed is is being passed ONLY the env vars which are present in the spec and everything else was missing! Or at least this is the behaviour in my local cluster setup + KinD cluster. Can you please add and other env vars necessary for the correct execution of AWS SDK and see if it works for you? If this doesn't work, then I'll send you the modified version of the which puts some additional logs to troubleshoot this issue further.
That did it! Thanks for helping me troubleshoot this. I wonder if this is unique to KinD or whether we would see this behavior in a "normal" cluster (i'll test it later). I will tell the developers who wrote the ecr-credential-provider to include an argument to increase the verbosity or to output logs to a file so this is easier to troubleshoot in the future.
What I have observed is, is only passed env vars which are in credential spec and the spec clearly says that the vars in spec are appended to existing env vars and this looks like an issue which needs deeper investigation. Could you please re-open this issue?
sure thing.
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
To me this looks like an issue with the kubelet and I have observed this behaviour with as well as my local cluster. What are your thoughts on this? If this looks like a bug then can you accept the triage? I would fix this.


Question: cmd := exec.CommandContext(ctx, filepath.Join(e.pluginBinDir, e.name), e.args...) cmd.Stdout, cmd.Stderr, cmd.Stdin = stdout, stderr, stdin <del> cmd.Env = []string{} for _, envVar := range e.envVars { cmd.Env = append(cmd.Env, fmt.Sprintf("%s=%s", envVar.Name, envVar.Value)) </del> <ins> var configEnvVars []string for _, v := range e.envVars { configEnvVars = append(configEnvVars, fmt.Sprintf("%s=%s", v.Name, v.Value)) </ins> } <ins> // Append current system environment variables, to the ones configured in the // credential provider file. Failing to do so may result in unsuccessful execution // of the provider binary, see https://github.com/kubernetes/kubernetes/issues/102750 // also, this behaviour is inline with Credential Provider Config spec cmd.Env = mergeEnvVars(e.environ(), configEnvVars) </ins> err = cmd.Run() if ctx.Err() != nil { return nil, fmt.Errorf("error execing credential provider plugin %s for image %s: %w", e.name, image, ctx.Err())

Thanks for looking into the issue, you are right the spec does says that. Feel free to fix it /triage accepted
Thanks!!!
/area kubelet
: I am facing the similar issue on my bare metal cluster This is my config Can you help. It works when I try to execute the credentials binary from local but doesn't work in the pod. Can you pls guide me
I've been fighting this for days - did you ever figure it out?


Question: if err != nil { return createResp.ID, fmt.Errorf("failed to start sandbox container for pod %q: %v", config.Metadata.Name, err) } <ins> // Rewrite resolv.conf file generated by docker. // NOTE: cluster dns settings aren't passed anymore to docker api in all cases, // not only for pods with host network: the resolver conf will be overwritten // after sandbox creation to override docker's behaviour. This resolv.conf // file is shared by all containers of the same pod, and needs to be modified // only once per pod. if dnsConfig := config.GetDnsConfig(); dnsConfig != nil { containerInfo, err := ds.client.InspectContainer(createResp.ID) if err != nil { return createResp.ID, fmt.Errorf("failed to inspect sandbox container for pod %q: %v", config.Metadata.Name, err) } if err := dockertools.RewriteResolvFile(containerInfo.ResolvConfPath, dnsConfig.Servers, dnsConfig.Searches, len(dnsConfig.Options) > 0); err != nil { return createResp.ID, fmt.Errorf("rewrite resolf.conf faield for pod %q: %v", config.Metadata.Name, err) } } // Do not invoke network plugins if in hostNetwork mode. </ins> if nsOptions := config.GetLinux().GetSecurityContext().GetNamespaceOptions(); nsOptions != nil && nsOptions.HostNetwork { return createResp.ID, nil }

Kubernetes version (use ): HEAD (~v1.6.0-beta.4) Environment: any What happened: I want to use ClusterFirstWithHostNet which was implemented in , but only dockertools (legacy/non-CRI) was updated to support this feature. The CRI dockershim should be modified to support this rewriting the file of Pods with HostNetwork=true as well. This is a bug in v1.6, I hope it can be prioritized to be fixed in v1.6.1, since it's blocking me from using CRI. Since CRI is enabled by default, it's expected that it should have feature parity with the earlier implementation.
Related
- I guess I didn't realize these crossed paths. Do you have time to take a look?
Will take a look
Thanks! Ping me directly on this.
Filed for supporting it in CRI.


Question: <ins> reviewers: - stevekuznetsov - deads2k - sttts approvers: - stevekuznetsov - deads2k - sttts </ins>

uses everywhere, directly and via client-go. On the other hand, we have in :
any idea why the import checker does not catch this?
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals


Question: <!-- BEGIN MUNGE: GENERATED_TOC --> - [Compute Resources](#compute-resources) <del> - [Container and Pod Resource Limits](#container-and-pod-resource-limits) - [How Pods with Resource Limits are Scheduled](#how-pods-with-resource-limits-are-scheduled) </del> <ins> - [Resource Requests and Limits of Pod and Container](#resource-requests-and-limits-of-pod-and-container) - [How Pods with Resource Requests are Scheduled](#how-pods-with-resource-requests-are-scheduled) </ins> - [How Pods with Resource Limits are Run](#how-pods-with-resource-limits-are-run) - [Monitoring Compute Resource Usage](#monitoring-compute-resource-usage) - [Troubleshooting](#troubleshooting)

did which addresses some of them, but turns up a lot more.
Alright, I will do that.


Question: // Grabber provides functions which grab metrics from components type Grabber struct { <del> client                    clientset.Interface externalClient            clientset.Interface grabFromAPIServer         bool grabFromControllerManager bool grabFromKubelets          bool grabFromScheduler         bool grabFromClusterAutoscaler bool masterName                string registeredMaster          bool </del> <ins> client                            clientset.Interface externalClient                    clientset.Interface grabFromAPIServer                 bool grabFromControllerManager         bool grabFromKubelets                  bool grabFromScheduler                 bool grabFromClusterAutoscaler         bool masterName                        string registeredMaster                  bool waitForControllerManagerReadyOnce sync.Once </ins> } // NewMetricsGrabber returns new metrics which are initialized.

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce ci-kubernetes-e2e-gci-gce Which test(s) are flaking: [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly] [sig-storage] In-tree Volumes [Driver: emptydir] [Testpattern: Inline-volume (default fs)] subPath should support file as subpath [LinuxOnly] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-link] [Testpattern: Pre-provisioned PV (default fs)] volumes should allow exec of files on the volume [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: blockfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing single file [LinuxOnly] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-bindmounted] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Inline-volume (default fs)] subPath should be able to unmount after the subpath directory is deleted [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Inline-volume (default fs)] subPath should support non-existent path [sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should support existing directories when readOnly specified in the volumeSource [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)]


Question: // Grabber provides functions which grab metrics from components type Grabber struct { <del> client                    clientset.Interface externalClient            clientset.Interface grabFromAPIServer         bool grabFromControllerManager bool grabFromKubelets          bool grabFromScheduler         bool grabFromClusterAutoscaler bool masterName                string registeredMaster          bool </del> <ins> client                            clientset.Interface externalClient                    clientset.Interface grabFromAPIServer                 bool grabFromControllerManager         bool grabFromKubelets                  bool grabFromScheduler                 bool grabFromClusterAutoscaler         bool masterName                        string registeredMaster                  bool waitForControllerManagerReadyOnce sync.Once </ins> } // NewMetricsGrabber returns new metrics which are initialized.

subPath should be able to unmount after the subpath directory is deleted [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-link-bindmounted] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing single file [LinuxOnly] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (ext3)] volumes should allow exec of files on the volume [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (ext3)] volumes should store data Testgrid link: Reason for failure: Failure fetching metrics: Anything else we need to know: This appears to have restarted abruptly around 2/11-2/12 /sig storage /priority important-soon
xref I think we have a general failure to start containers reliably. Almost every instance of these test failures contained the pod status error of "OCI runtime start failed: container process is already dead: unknown"
after sweeping a bunch of the in-tree test failures, almost all seemed to have as a root cause, triggered by very short-lived containers used to check file content or permissions. The only test I saw failing that didn't seem to fit that was: [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory examples: I'd recommend repurposing this issue to address that specifically and leave the OCI issue for the short-lived container issue
Here's a PR for context: So, I think this PR is another example of flakes. Here are some links to previous runs: and same message: This prow: has the above message and a few more, all A lot of in-tree volume issues, but there were some persistent volume ones too.


Question: // Grabber provides functions which grab metrics from components type Grabber struct { <del> client                    clientset.Interface externalClient            clientset.Interface grabFromAPIServer         bool grabFromControllerManager bool grabFromKubelets          bool grabFromScheduler         bool grabFromClusterAutoscaler bool masterName                string registeredMaster          bool </del> <ins> client                            clientset.Interface externalClient                    clientset.Interface grabFromAPIServer                 bool grabFromControllerManager         bool grabFromKubelets                  bool grabFromScheduler                 bool grabFromClusterAutoscaler         bool masterName                        string registeredMaster                  bool waitForControllerManagerReadyOnce sync.Once </ins> } // NewMetricsGrabber returns new metrics which are initialized.

But, that showed up once so far from what I've seen.
The "error getting c-m metrics" failures appear to have just restarted:
/priority critical-urgent bumping priority since we're seeing significant numbers of failures that just started in the last two days
this is failing many test runs per day, do you have updated status on this?
I haven't had chance to fully debug this yet. looked into this little bit and he suspected that these tests started to flake same time around was merged. I should be able to take a closer look tomorrow.
This occurrence fails because it's trying to get the metrics from the kube-controller-manager pod but it is not ready until one minute later if we grep in the kube-apiserver for the controller pod name we can observe that after the pod is up it starts to get 200 instead of 404
keeping open until shows this is resolved
seen again in
this time the container is running but still it can't find it
the errors comes from the request to the apiserver on How can we check that the pod is found and that the apiserver found it?
the first successful get of /metrics logged by the controller manager in that run is at: We're also not checking the error in the once.Do to make sure the pod was running, and once running, we should probably then also check the metrics path on the pod. Maybe something like this:
still happening: message logging missed using the correct errors, so the messages aren't helpful:
what's this stack trace in the apiserver? is this related? a 503 shouldn't cause that, right?
again it seems is not really waiting because it fails just 200 ms after the INFO msg, I think is because it checks if the pod is running, but the message is that the pod doesn't really exist. The fact that there are 9 tests failing with the same error and we are using has to be related to the way Ginkgo use parallel.
It turns out that the method used to wait for the controller manager pod fails if the pod doesn't exist
Ah, so first we should wait for the pod to exist, then be ready, then return metrics
This start to look promising
yes, looks good. thanks for all the work /close
Closing this issue. <detailsIn response to : yes, looks good.


Question: // Grabber provides functions which grab metrics from components type Grabber struct { <del> client                    clientset.Interface externalClient            clientset.Interface grabFromAPIServer         bool grabFromControllerManager bool grabFromKubelets          bool grabFromScheduler         bool grabFromClusterAutoscaler bool masterName                string registeredMaster          bool </del> <ins> client                            clientset.Interface externalClient                    clientset.Interface grabFromAPIServer                 bool grabFromControllerManager         bool grabFromKubelets                  bool grabFromScheduler                 bool grabFromClusterAutoscaler         bool masterName                        string registeredMaster                  bool waitForControllerManagerReadyOnce sync.Once </ins> } // NewMetricsGrabber returns new metrics which are initialized.

thanks for all the work Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Fixed by


Question: return nil } <del> func getAppArmorFS() (string, error) { mountsFile, err := os.Open("/proc/mounts") if err != nil { return "", fmt.Errorf("could not open /proc/mounts: %v", err) } defer mountsFile.Close() scanner := bufio.NewScanner(mountsFile) for scanner.Scan() { fields := strings.Fields(scanner.Text()) if len(fields) < 3 { // Unknown line format; skip it. continue } if fields[2] == "securityfs" { appArmorFS := path.Join(fields[1], "apparmor") if ok, err := utilpath.Exists(utilpath.CheckFollowSymlink, appArmorFS); !ok { msg := fmt.Sprintf("path %s does not exist", appArmorFS) if err != nil { return "", fmt.Errorf("%s: %v", msg, err) } return "", errors.New(msg) } return appArmorFS, nil } } if err := scanner.Err(); err != nil { return "", fmt.Errorf("error scanning mounts: %v", err) } return "", errors.New("securityfs not found") } </del>

As mentioned the return value of is not being used any longer and the entire method can be removed. The usage of was removed in Now there is no need to have the function and store it's result at all. /sig node /kind cleanup /good-first-issue /help-wanted KEP xref:
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : As mentioned the return value of is not being used any longer and the entire method can be removed. /kind cleanup /good-first-issue /help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/assign
I am having issues submitting the pull request. I removed that portion of the code under however this is my first time contributing and would like some help submitting this new updated version..
/assign
/assign
/assign
/assign


Question: For Kubernetes 1.0, we strongly recommend running the following set of admission control plug-ins (order matters): ``` <del> --admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota </del> <ins> --admission-control=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota </ins> ```

We should have a single NamespaceLifecycle plugin that enforces the Namespace rules now that all providers are off NamespaceAutoProvision. We may also want to make this no longer a user choice to configure and hard-wire the server to always run this check first to simplify configuration errors. For an example


Question: // AUTO-GENERATED FUNCTIONS START HERE. DO NOT EDIT. var map_Endpoint = map[string]string{ "":           "Endpoint represents a single logical "backend" implementing a service.", <del> "addresses":  "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "addresses":  "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "conditions": "conditions contains information about the current status of the endpoint.", "hostname":   "hostname of this endpoint. This field may be used by consumers of endpoints to distinguish endpoints from each other (e.g. in DNS names). Multiple endpoints which use the same hostname should be considered fungible (e.g. multiple A values in DNS). Must be lowercase and pass DNS Label (RFC 1123) validation.", "targetRef":  "targetRef is a reference to a Kubernetes object that represents this endpoint.",

An can contain up to 1000 , which can each contain up to 100 . Or at least, that's what the docs say, but actually, kube-proxy (or more specifically, ) only looks at the first Address in each Endpoint: (This is compatible with how works, since it would only generate multi-Address Endpoints if a pod had multiple of the same address family, which is not allowed. But it's not compatible with manually-generated EndpointSlices.) Either all are used, or else the docs state that only is used and the field is an array solely for historical reasons.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /priority important-longterm /cc
All addresses in the slice must be of the same type, do you mean creating an Slice with different address types in the Addresses field?
I'm not sure if/how kube-proxy should handle an endpoint with more than 1 address. I think the goal of kube-proxy is to give each endpoint an equivalent probability of being selected. To extend that logic here, I think we'd need to take the probability assigned to an endpoint and divide it by the number of addresses attached to that endpoint. So if an endpoint has a 50% probability of being selected, and it provides 50 addresses, each should end up with a 1% probability of being selected. Of course with the way we need to do descending probabilities, the logic may get a bit complicated. This isn't as clear as it should be, but I don't think kube-proxy or any consumer of the EndpointSlice API needs to commit to supporting every value specified by that API. For example, many may filter by address type, Service, or some other means. I think it's also valid for a consumer to say they only support a single address per family per endpoint. So all of that to say, I'd support either documenting that kube-proxy only selects the first address in an endpoint, or updating kube-proxy to support more than one address per endpoint.


Question: // AUTO-GENERATED FUNCTIONS START HERE. DO NOT EDIT. var map_Endpoint = map[string]string{ "":           "Endpoint represents a single logical "backend" implementing a service.", <del> "addresses":  "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "addresses":  "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "conditions": "conditions contains information about the current status of the endpoint.", "hostname":   "hostname of this endpoint. This field may be used by consumers of endpoints to distinguish endpoints from each other (e.g. in DNS names). Multiple endpoints which use the same hostname should be considered fungible (e.g. multiple A values in DNS). Must be lowercase and pass DNS Label (RFC 1123) validation.", "targetRef":  "targetRef is a reference to a Kubernetes object that represents this endpoint.",

Given the number of changes currently in flight for kube-proxy, documentation seems like a safer initial approach to me.
/assign
can I have this editing only the first address(Address[0]) in Endpoint.Address to be used. The use of an array in this field is based on historical reasons. Isn't that right
thanks for volunteering to help with this one! I know has some big kube-proxy changes/fixes in flight right now, so it would be good to be sure you don't conflict with him. Otherwise, as long as isn't already working on this, I think you're free to have it.
I don't think we agreed that the array is for historical reasons only. We aren't currently using it, but we aren't ready to say that we won't use it in the future at this point. So, the docs should warn that kube-proxy currently ignores additional addresses, but it shouldn't imply that it's actually wrong to have additional addresses I'm not
thanks i will work on this
It was for a reason, but I think that reason was mixed-family slices (before we decided to break it up into single-family slices). What would it mean to have multiple addresses here? Are they assumed to be equivalent? Or do they have different context (like a split-horizon)? If the latter, there isn't any metadata here (i.e. it's not a struct like to allow new fields). I am inclined to believe it was meant as equivalent, and as such it's probably OK to document that assumption clearly. That said, I don't see how anyone could use multiple addresses in a meaningful way - they are either assumed equivalent or they lack enough metadata to distinguish their differences. ?? If we do document the equivalence, it should be generically worded. E.g. "These addresses are assumed to be equivalent and some consumers may choose to ignore all but the first value."
if it was meant as equivalent, why put multiple same addresses into Addresses? I don't understand what the purpose of this is.
The first version of the API had both IP families in a single slice, so it made sense to have multiple addresses. We changed that, though we didn't immediately remove the old support. So my best guess is that we should have converted this to a singular value when we did that, but we missed it.


Question: // AUTO-GENERATED FUNCTIONS START HERE. DO NOT EDIT. var map_Endpoint = map[string]string{ "":           "Endpoint represents a single logical "backend" implementing a service.", <del> "addresses":  "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "addresses":  "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "conditions": "conditions contains information about the current status of the endpoint.", "hostname":   "hostname of this endpoint. This field may be used by consumers of endpoints to distinguish endpoints from each other (e.g. in DNS names). Multiple endpoints which use the same hostname should be considered fungible (e.g. multiple A values in DNS). Must be lowercase and pass DNS Label (RFC 1123) validation.", "targetRef":  "targetRef is a reference to a Kubernetes object that represents this endpoint.",

On Wed, Nov 24, 2021, 5:45 PM Cyclinder
so we should change the API doc? Or It's just that documenting it will only use the first address in the , like you said above
I'd like some of the others to comment, but it's this a US holiday, so it might be a few days..
So maybe we should just declare that the array is essentially a bug, and only the first address is used? Maybe something should warn if you ever set additional addresses? (FTR, the context that I noticed this in was that I was adding unit tests, and was creating EndpointSlices with a single Endpoint and multiple Addresses, which doesn't work; you need multiple Endpoints with a single Address each.)
I think the simplest and most compat approach is to say "these are all assumed to be fungible and clients may choose to only use the first element". Thoughts?
I think the simplest and most compat approach is to say "these are all assumed to be fungible and clients may choose to only use the first element". Big +1 to that approach. I don't think it's necessarily invalid to set multiple addresses here, there may actually be some use cases for it, but adding this kind of statement should help clear things up significantly.
hmm as we push for multi homed pod this will make a lot of sense. There might also be a chance for community to provide a drop-in replacement for proxy that does much more than we do now. I think we should update the docs and don't modify anything for now.
Note that this API doesn't include per-IP metadata, so a multi-homed pod would be challenging with this mechanic. I think it's maybe time to reconsider multi-net more holistically. On Mon, Dec 13, 2021, 10:24 AM Khaled Henidak (Kal) <
:thinking:
If they're fungible, then "clients may choose to use only one of the elements" (or "only a subset"?). No reason to call out the first one specifically.
That's fair On Tue, Dec 14, 2021, 6:09 AM Dan Winship
I would like to add to this issue that it would makes sense to have kube-proxy reduce the subset of endpoints in the endpointslice based on the label of the node and the & attributes of the endpoint.


Question: // AUTO-GENERATED FUNCTIONS START HERE. DO NOT EDIT. var map_Endpoint = map[string]string{ "":           "Endpoint represents a single logical "backend" implementing a service.", <del> "addresses":  "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.", </del> <ins> "addresses":  "addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267", </ins> "conditions": "conditions contains information about the current status of the endpoint.", "hostname":   "hostname of this endpoint. This field may be used by consumers of endpoints to distinguish endpoints from each other (e.g. in DNS names). Multiple endpoints which use the same hostname should be considered fungible (e.g. multiple A values in DNS). Must be lowercase and pass DNS Label (RFC 1123) validation.", "targetRef":  "targetRef is a reference to a Kubernetes object that represents this endpoint.",

Taking only into consideration the first endpoint which is in the same zone of the node or has no zone dependency and is ready for serving. & Would it make sense to reopen this issue?
I don't follow entirely what you are suggesting, can you elaborate?
kube-proxy should understand which zone a node is in and prefer endpoints in the same zone.
but that is unrelated to this issue, topology is considered See KEPs related, topology-aware-routing was replaced by topology-aware-hints
thank you for clarifying. It seems ambiguous to me to have a zone attribute and a hint attribute. I fail to implement the hinting on a service in v1.23.8, but this is urelated to this issue.
heh, don't worry, this topic is complex and we went through different solutions, you have more historical context on the KEPs I linked above and this thread in the mailing list


Question: rt.CapabilityArgs["bandwidth"] = bandwidthParam } <ins> // Set the PodCIDR rt.CapabilityArgs["ipRanges"] = [][]cniIpRange{{{Subnet: plugin.podCidr}}} </ins> return rt, nil }

/kind feature The CNI "capability arguments" parameter - currently used to pass port mappings - has a few new capabilities: bandwidth limiting and network address management. The dockershim's networking driver should use them. The network address is just passing the PodCIDR dynamically to the plugin, so it no longer has to retrieve it from the APIServer. Bandwidth limiting is already implemented in Kubenet. It is configured via annotations. There is no upgrade risk in providing these values; non-compliant plugins will simply ignore them. Administrators need to explicitly enable each capability individually in their CNI configuration file. To avoid skew between runtimes, the other runtimes should also be upgraded to pass these. We may also wish to look in to a way to add this more directly to the CRI, so we don't have this skew problem, but that's a different issue.
/sig network
/assign
fyi.
Looks like this was never done for CRI?
cc


Question: source "${KUBE_ROOT}/hack/lib/init.sh" if [[ ! -f "${KUBE_ROOT}/vendor/BUILD" ]]; then <del> echo "${KUBE_ROOT}/vendor/BUILD does not exist." echo echo "Run ./hack/update-bazel.sh" </del> <ins> echo "${KUBE_ROOT}/vendor/BUILD does not exist." >&2 echo >&2 echo "Run ./hack/update-bazel.sh" >&2 </ins> exit 1 fi

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: When running on my commits locally I got the following error: Affected PRs:
/sig testing
/cc
There's likely something up with your local environment. These jobs are running correctly in CI, although you need to look at the build log for output


Question: # Clean up kubectl delete namespace my-namespace <del> ############## </del> <ins> ###################### </ins> # Pods in Namespaces # <del> ############## </del> <ins> ###################### </ins> ### Create a new namespace # Pre-condition: the other namespace does not exist

The desciption of --all-namespaces is listing the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with --namespace." But when I used the following command: error: namespace may not be empty when retrieving a resource by name no namespace set on resource pods "counter" The error messages are unclear, and the second message isn't labeled as an error even.
And why couldn't use --all-namespaces flag?
You're asking for a specific pod name (counter) in all namespaces, which is a nonsensical query. The pod name is not a grep-like filter, it's a specific name. Yeah, the errors could be better. Describe could work, but potentially generates a LOT of information and API calls.
But I found the following API is supported now. Get The result is: has the more information. Do we need to support the following kubectl command(using the above API) or just return clear error messages? get pods counter --all-namespaces
I'd be ok with this being implemented via the field selector.
The issue is still here :). What's our expected result? I'd like to contribute this.
, after check the code, the URL was built by ; do you mean we should update it; or avoid to use , instead, build url () directly.
After checking the code, it's better to add fieldSelector to , is created for fieldSelector. Will submit a PR based on .
getting an object with a particular name across all namespaces seems like something we shouldn't encourage... it breaks the assumption of namespace independence.
, so we'll only update message for this issue? for example:
Disagree. Need easy way to troubleshouting the problem "why pods is stuck in Pending status". It's tiring for every pod to find who created it and in which namespace. In the logs there is only the name of the pod.
Now, with field selectors we can do To further my understanding, I'd ask:- doesn't itself break independence? One can search by label across all namespaces, it feels funny for by name to be special.


Question: checkNode := func(i int) { // We check the nodes starting from where we left off in the previous scheduling cycle, // this is to make sure all nodes have the same chance of being examined across pods. <del> nodeInfo := g.nodeInfoSnapshot.NodeInfoList[(g.lastProcessedNodeIndex+i)%allNodes] </del> <ins> nodeInfo := g.nodeInfoSnapshot.NodeInfoList[(g.nextStartNodeIndex+i)%allNodes] </ins> fits, failedPredicates, status, err := g.podFitsOnNode( ctx, state,

What would you like to be : A unit test that verifies the evaluation order of the nodes in the scheduler. Why is this needed: The scheduler evaluates the nodes in a specific order until it finds a pre-defined minimum number of feasible nodes. To give all nodes a fair chance of being evaluated, in each scheduling cycle, the scheduler starts evaluating from where it left off in the previous cycle. This is captured by the parameter in  We need to add a unit test that captures regressions related to this behavior to avoid bugs like /sig scheduling /priority important-soon /help
/assign
/remove-help


Question: if *confDoc { // Note: this assumes that you're running from the root of the kube src repo <del> header, err := ioutil.ReadFile("test/conformance/cf_header.md") if err == nil { fmt.Printf("%snn", header) </del> <ins> templ, err := template.ParseFiles("test/conformance/cf_header.md") if err != nil { fmt.Printf("Error reading the Header file information: %snn", err) } data := struct { Version string }{ Version: *version, </ins> } <ins> templ.Execute(os.Stdout, data) </ins> } totalConfTests = 0

What happened: The Conformance Document currently has a static header that is displayed from a file. The release information in the header should reflect proper release number so that the document reflect the current state. What you expected to happen: Should show proper current release information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
in reaction to comments is still hardcoded to mention v1.9 at the top, could we parameterize this?
/sig testing /area conformance /release-note-none
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
/priority backlog
This seems to have been addressed at Should we just close this?
No, specifically did not address this issue, this should stay open. See


Question: func newAzureAuthProvider(_ string, cfg map[string]string, persister restclient.AuthProviderConfigPersister) (restclient.AuthProvider, error) { // deprecated in v1.22, remove in v1.25 <del> // this should be updated to use klog.Warningf in v1.24 to more actively warn consumers </del> warnOnce.Do(func() { <del> klog.V(1).Infof(`WARNING: the azure auth plugin is deprecated in v1.22+, unavailable in v1.25+; use https://github.com/Azure/kubelogin instead. </del> <ins> klog.Warningf(`WARNING: the azure auth plugin is deprecated in v1.22+, unavailable in v1.25+; use https://github.com/Azure/kubelogin instead. </ins> To learn more, consult https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins`) })

Replace the azure auth plugin deprecation with a in v1.24 xref: /assign /milestone v1.24 /triage accepted /sig auth
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten /lifecycle frozen
/assign
/priority important-soon /kind cleanup deprecation


Question: "os" "testing" <del> "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-03-01/compute" "github.com/Azure/go-autorest/autorest/to" </del> "github.com/stretchr/testify/assert" "k8s.io/api/core/v1"

<!-- Please only use this template for submitting enhancement requests -- What would you like to be : related to Why is this needed: /assign /sig azure
here is the way about how to get MaxDataDiskCount: and then build kubelet, copy kubelet to agent node and then get MaxDataDiskCount data: There are 207 VM sizes totally.
Update: this command could get VM size list easily:


Question: shouldContinueRunning: true, }, { <ins> predicateName: "ErrPodNotMatchHostName", </ins> ds: &extensions.DaemonSet{ Spec: extensions.DaemonSetSpec{ Selector: &metav1.LabelSelector{MatchLabels: simpleDaemonSetLabel},

Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement /sig apps Description: is a key function of DaemonSet; it's better to have more unit test case for it, e.g. node status, pod on the node. There's several test cases to the DaemonSet's finial decision, which are kind of integration test to me; prefer to provide more unit test case for this key function.
GitHub didn't allow me to assign the following users: mathspanda. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
do you have any specific cases or shall I figure out something? I can take a look at this issue if you don't mind.
, , thanks very much for our effort on this enhancement; as you known, DaemonSet will use scheduler's predicates to decide which node can place pods; there's several predicates errors , the purpose of this issue is to trigger every error in unit test for code coverage.
Sure, thanks. I'll cover these predicates.
Also I've found interesting issue when I wanted to add test for predicate. It looks like is not handled or maybe I did something wrong, I couldn't find check. According following comments it should fail because AWS EBS forbids any two pods mounting the same volume ID : Test case: Any thoughts? is it correct behavior?