Question: const sysctlRouteLocalnet = "net/ipv4/conf/all/route_localnet" const sysctlBridgeCallIPTables = "net/bridge/bridge-nf-call-iptables" const sysctlVSConnTrack = "net/ipv4/vs/conntrack" <ins> const sysctlConnReuse = "net/ipv4/vs/conn_reuse_mode" </ins> const sysctlForward = "net/ipv4/ip_forward" const sysctlArpIgnore = "net/ipv4/conf/all/arp_ignore" const sysctlArpAnnounce = "net/ipv4/conf/all/arp_announce"

The benchmark is basically stress testing the new connection handling of IPVS. But I'm not sure we should be optimizing for such a thing, when in reality connections will be much longer lived (like with gRPC, or http/1.1+). It's a trade off between improved load balancing (connreusemode=1 or 2) and connection creation latency (connreusemode=0).
I have a php program use unirest to request other sevice in the cluster. It not use keepalive.
I would prefer to improve the latency number and I think the scenario where one client send lots of requests at the same time is not a common case and mostly happen in test environment.
It might be possible to improve this in the kernel too. You can see the change here: bypasses the reuse checks and expiring logic for new connections.
We set in IPVS proxier in case other people don't know what kernel parameters they should tune. You are familiar with IPVS and linux kernel, but I think many users don't have enough context.
, I think this was because the the container already deleted, but kube-proxy need time to flush the rules, iptables mode should also have this problem.
Yes, I will test preStop recently, may solve the problem
Here the likely reason for the slow throughput is described: As setting to causes , is any of the options listed possible to do? I guess only option 2 might be worth looking at as conntrack is probably needed for the remaining iptables components.
Hello everyone We are very fortunate to tell you that this bug has been fixed by us and has been verified to work very well. The patch is being submitted to the Linux kernel community. You can also apply this patch to your own kernel, and then only need to set (default) and (default). As the sysctl switch is newly . You can adapt the kube-proxy by judging whether there is , if so, it means that the current kernel is the version that fixed this bug.


Question: const sysctlRouteLocalnet = "net/ipv4/conf/all/route_localnet" const sysctlBridgeCallIPTables = "net/bridge/bridge-nf-call-iptables" const sysctlVSConnTrack = "net/ipv4/vs/conntrack" <ins> const sysctlConnReuse = "net/ipv4/vs/conn_reuse_mode" </ins> const sysctlForward = "net/ipv4/ip_forward" const sysctlArpIgnore = "net/ipv4/conf/all/arp_ignore" const sysctlArpAnnounce = "net/ipv4/conf/all/arp_announce"

That Can solve the following problems: update, IPVS keeps scheduling traffic to the destroyed Pod IPVS traffic scheduling after scaled up or rolling update IPVS low throughput issue second connection delay in masque low throughputÂ Bench can fill up ipvs service proxy in seconds 1s latency in when upgrading from on RHEL 8.1 -proxy ipvs connreuse_mode setting causes errors with high load from single client Thank you. By Yang Yuxi TencentCloudContainerTeam
Following-up on comment above for posterity. The patch mentioned above didn't make it to the kernel but there are two recently merged patches worth highlighting. One of them fixes the 1 second delay issue and the other fixes dropped packets when stale connection entries in the IPVS table are used: 1) 2)


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package azure import ( "fmt" "testing" "github.com/Azure/azure-sdk-for-go/arm/network" "github.com/Azure/go-autorest/autorest/to" "github.com/stretchr/testify/assert" ) func TestFindProbe(t *testing.T) { tests := []struct { msg           string existingProbe []network.Probe curProbe      network.Probe expected      bool }{ { msg:      "empty existing probes should return false", expected: false, }, { msg: "probe names match while ports unmatch should return false", existingProbe: []network.Probe{ { Name: to.StringPtr("httpProbe"), ProbePropertiesFormat: &network.ProbePropertiesFormat{ Port: to.Int32Ptr(1), }, }, }, curProbe: network.Probe{ Name: to.StringPtr("httpProbe"), ProbePropertiesFormat: &network.ProbePropertiesFormat{ Port: to.Int32Ptr(2), }, }, expected: false, }, { msg: "probe ports match while names unmatch should return false", existingProbe: []network.Probe{ { Name: to.StringPtr("probe1"), ProbePropertiesFormat: &network.ProbePropertiesFormat{ Port: to.Int32Ptr(1), }, }, }, curProbe: network.Probe{ Name: to.StringPtr("probe2"), ProbePropertiesFormat: &network.ProbePropertiesFormat{ Port: to.Int32Ptr(1), }, }, expected: false, }, { msg: "both probe ports and names match should return true", existingProbe: []network.Probe{ { Name: to.StringPtr("matchName"), ProbePropertiesFormat: &network.ProbePropertiesFormat{ Port: to.Int32Ptr(1), }, }, }, curProbe: network.Probe{ Name: to.StringPtr("matchName"), ProbePropertiesFormat: &network.ProbePropertiesFormat{ Port: to.Int32Ptr(1), }, }, expected: true, }, } for i, test := range tests { findResult := findProbe(test.existingProbe, test.curProbe) assert.Equal(t, test.expected, findResult, fmt.Sprintf("TestCase[%d]: %s", i, test.msg)) } } </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area cloudprovider /sig azure What happened: I've created a service of type in a Kubernetes cluster created via the ACS Engine. The load balancer has been created correctly after some time and a random NodePort (at this time ) has been allocated. However, in the Azure Portal I'm seeing that the health probe of that load balancer is pointing to port which does not exist in the whole Kubernetes system at all. Because of that wrongly configured health probe the load balancer is not reachable (i/o timeout). What you expected to happen: The health probe would have been configured to NodePort . How to reproduce it (as minimally and precisely as possible): Unfortunately, it does not behave always like this but only occasionally. In some cases the health probe gets configured correctly, but in others not. I've used the following manifest: Anything else we need to know?: When updating the health probe manually to the correct NodePort in the Azure portal, the load balancer starts to work properly. Environment: Kubernetes version (use ): 1.7.5 Cloud provider or hardware configuration: Azure Install tools: ACS Engine Others: -
Thanks for reporting the problem. Could you share the service spec after it is created? e.g. by running . And may I ask whether there is a service just removed with same name before this one?
Sure, thanks, please find the manifest below [1]. In fact I'll have some other services with the same name, but in other namespaces. In the namespace I didn't have a service named before. [1] Service manifest after creation
Filed PR , which should fix the problem. Need cherry-picked to v1.7 branch after it is merged.
Thank you very much that was really fast!
has been merged. created cherry pick PR for release 1.7.


Question: func BindContextFlags(contextInfo *clientcmdapi.Context, flags *pflag.FlagSet, flagNames ContextOverrideFlags) { flagNames.ClusterName.BindStringFlag(flags, &contextInfo.Cluster) flagNames.AuthInfoName.BindStringFlag(flags, &contextInfo.AuthInfo) <del> flagNames.Namespace.BindStringFlag(flags, &contextInfo.Namespace) </del> <ins> flagNames.Namespace.BindTransformingStringFlag(flags, &contextInfo.Namespace, RemoveNamespacesPrefix) } // RemoveNamespacesPrefix is a transformer that strips "ns/", "namespace/" and "namespaces/" prefixes case-insensitively func RemoveNamespacesPrefix(value string) (string, error) { for _, prefix := range []string{"namespaces/", "namespace/", "ns/"} { if len(value) > len(prefix) && strings.EqualFold(value[0:len(prefix)], prefix) { value = value[len(prefix):] break } } return value, nil </ins> }

Because always return names with a prefix like , it cannot be directly used to iterate over all namespaces to execute other commands. I've compiled from source: Reviewing some scripts, I came across a usage of like this: That line fetches namespace names to later run in a loop: That seemed to be a perfect use case for the flag, but it can't be used without extra hacks to get rid of the prefix (FWIW, is invalid). Changing the current output of might not be possible or desirable, for this could impact existing users that rely on the output format. If I'm wrong about the above limitation, than it would make sense to output just the name when and there is a single resource type, while using the ideas introduced in to show the resource type only when there is more than one type in the output, or when is explicitly set. Before: After:
could you help me ping the appropriate team to triage this? Thanks!
Please re-post your question to . We are trying to consolidate the channels to which questions for help/support are posted so that we can improve our efficiency in responding to your requests, and to make it easier for you to find answers to frequently asked questions and how to address common use cases. We regularly see messages posted in multiple forums, with the full response thread only in one place or, worse, spread across multiple forums. Also, the large volume of support issues on github is making it difficult for us to use issues to identify real bugs. The Kubernetes team scans stackoverflow on a regular basis, and will try to ensure your questions don't go unanswered. Before posting a new question, please search stackoverflow for answers to similar questions, and also familiarize yourself with: Again, thanks for using Kubernetes. The Kubernetes Team
perhaps I didn't do a good job at explaining the problem, but this is not a question of how to do something, but a report on a problem with the CLI design.
Sorry about that :-/. I'll try to route to the proper team.
update: console examples to the description.
the -o format is designed for scripting, the use case is just different from yours. You should be able to paste the output of into . Does do what you want? If this is a common use case we could create another -o flag value.


Question: func BindContextFlags(contextInfo *clientcmdapi.Context, flags *pflag.FlagSet, flagNames ContextOverrideFlags) { flagNames.ClusterName.BindStringFlag(flags, &contextInfo.Cluster) flagNames.AuthInfoName.BindStringFlag(flags, &contextInfo.AuthInfo) <del> flagNames.Namespace.BindStringFlag(flags, &contextInfo.Namespace) </del> <ins> flagNames.Namespace.BindTransformingStringFlag(flags, &contextInfo.Namespace, RemoveNamespacesPrefix) } // RemoveNamespacesPrefix is a transformer that strips "ns/", "namespace/" and "namespaces/" prefixes case-insensitively func RemoveNamespacesPrefix(value string) (string, error) { for _, prefix := range []string{"namespaces/", "namespace/", "ns/"} { if len(value) > len(prefix) && strings.EqualFold(value[0:len(prefix)], prefix) { value = value[len(prefix):] break } } return value, nil </ins> }

the use cases I've observed doing code reviews: over all namespaces to get certain resources from that namespace: over all namespaces, then iterate of over resource types that produce logs and fetch logs one by one: Above we can use for listing pods because later we'll pass it to that accept the syntax. We cannot do the same for namespaces, because doesn't like the format.
Alternatively we could be kubectl recognize as valid
I'll take a look on it if it's not taken by anyone yet. CC
Have you started this?
Only investigated due to being out of office since Thursday last week. I can work on it in upcoming week, but I'll understand if it's piority issue and you'll reassign it.
Ready :-)
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func getRandomClusterZone(c clientset.Interface) string { zones, err := e2enode.GetClusterZones(c) <ins> zone := "" </ins> framework.ExpectNoError(err) <del> framework.ExpectNotEqual(len(zones), 0) zonesList := zones.UnsortedList() return zonesList[rand.Intn(zones.Len())] </del> <ins> if len(zones) != 0 { zonesList := zones.UnsortedList() zone = zonesList[rand.Intn(zones.Len())] } return zone </ins> }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --If the provide is , run the below storage testcase, Hit below exception, At first glance, it looks like the return value is not correct, but looking into how the testcases was wrote, the root case should be that the testcase below doesn't support the provide, SSD PD on GCE/GKE gp2 EBS on AWS We should provide the correct error message which will be useful for trouble shooting instead. Kubernetes version (use ): All Cloud provider or hardware configuration: local OS (e.g: ): All Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing /sig storage /assign
NOTE: the "local" provider is specifically hack/local-up- any other cluster that is not one of "gce" etc. should use the default provider / "skeleton" provider (I know the naming is bad, but it's sort of an API now)
/remove-sig testing


Question: package storage import ( <ins> "fmt" "time" </ins> . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" "k8s.io/api/core/v1" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> clientset "k8s.io/client-go/kubernetes" <ins> kubeletmetrics "k8s.io/kubernetes/pkg/kubelet/metrics" </ins> "k8s.io/kubernetes/test/e2e/framework" "k8s.io/kubernetes/test/e2e/framework/metrics" )

In 1.8, we a new feature to expose PVC information through summary API so user can check the volume belongs to which PVC. We should add e2e test in under test/e2enode.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
GitHub didn't allow me to assign the following users: vkamra. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Can we please move this out of 1.8 milestone?
- I'm looking into adding this but yes we should move this out of 1.8. I don't have permissions to do that though.
ack Can we please move this out of 1.8?
Sorry am not sure why was this removed from 1.8 milestone. Are we not aiming to add the tests in 1.8? Or Are we disabling the feature or are we fine shipping it without any e2e tests in 1.8?
The test is in . I think we can keep this feature in 1.8


Question: "context" "net/http" "net/http/httptest" <ins> "sort" </ins> "strconv" "strings" "testing"

! /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
It should be fixed by
/sig api-machinery
/triage accepted


Question: return &missingSANChecker{ counterRaiser: counterRaiser{ counter: counter, <ins> id:      "missing-san", </ins> reason:  "relies on a legacy Common Name field instead of the SAN extension for subject validation", }, }

go1.18 dropped support for certificates signed with the SHA-1 hash function - While this does improve security, and publicly trusted Certificate Authorities no longer issue such certificates, it is possible that user-provided certificates issued by private Certificate Authorities and used in webhooks or aggregated API servers do still use this. Similar to removal of support for certificates without subjectAltNames in go1.17, we need to: [x] add metrics when webhook/conversion/aggregated certificates with this hash are encountered - [x] document use of and the metric from 1 as a mitigation for clusters which still need to identify and replace use of such certificates - to release note and known issues [x] add audit annotation when SHA-1 cert is encountered - [x] update docs at and - /sig api-machinery instrumentation docs /priority important-soon /milestone v1.24 /kind bug documentation Upgraded 1.23 clusters to continue working as-is Configure an admission webhook, conversion webhook, or aggregated server with a serving certificate signed with the SHA-1 hash function. Upgrade to HEAD of 1.24. The certificate will no longer work by default No response 24 HEAD (go1.18-based) n/a n/a No response No response No response
it looks like NewMissingSANRoundTripperWrapperConstructor already has all the plumbing, and instead of missing SANs we could check the SignatureAlgorithm and output a distinct metric
had a not-yet-tested WIP branch at with a sketch of what it might look like
/assign who offered to pick this up
GitHub didn't allow me to assign the following users: stlaz. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign who offered to pick this up Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Indeed, please, you should trust Jordan :slightlysmilingface:
/assign
/triage accepted
These restrictions are on the GitHub side, this message is posted when it receives a specific error from the GitHub API while attempting to assign It is an annoying GitHub limitation, and one that seemingly does not exist in the web client vs the API.


Question: return &missingSANChecker{ counterRaiser: counterRaiser{ counter: counter, <ins> id:      "missing-san", </ins> reason:  "relies on a legacy Common Name field instead of the SAN extension for subject validation", }, }

adds the metrics. the release note on that issue describes the go change, links to go docs, describes use of the option, the new metrics, and the kube-apiserver logs what would you expect kubeadm to do with this? GODEBUG off by default and let users override GODEBUG on their own (is this possible)? GODEBUG by default on all clusters? GODEBUG by default only on upgrade? The risk of 1 is that a user upgrades without reading release notes and breaks a cluster that has SHA-1 certs in webhooks, etc The risk of 2 is that we propagate accepting insecure certificates longer, even in new clusters that we know don't have any pre-existing bad certs The risk of 3 is that it makes an upgraded 1.23 → 1.24 cluster and a new 1.24 cluster inconsistent
the majority of kubeadm users use kubeadm managed CA and certificates. there are less users that use external CA to sign their own certificates. sha1 attacks surfaced more than a before kubeadm's creation and Git hash collusion made the problem more known around 2017 - that was at a time when kubeadm did not support external CA. given these facts, an estimate on my side is that there are currently no kubeadm users that use sha1 certificates, but i would not exclude the possibility entirely. if such users exists, they are on they own and my vote goes for option 1, until they migrate their PKI to something secure. if the same users log tickets for us, our response would be "please use GODEBUG and migrate ASAP". all users should be able to see the notes in IMO, the PR needs an "action required" to appear on top of the release notes.
that's helpful, thanks agreed. weird, I set , but the label didn't get . the label manually.
and there's a way for them to set that envvar in kube-apiserver manifests when using kubeadm?
There's been a lot of intermittent github webhook delivery issues in the past few weeks or so, slash commands are going to be somewhat unreliable :/
they could use kubeadm patches to override the YAML manifest defaults - i.e. the contents of v1.Container -Env:


Question: klog.V(4).Info(log("attacher.Attach finished OK with VolumeAttachment object [%s]", attachID)) <del> // TODO(71164): In 1.15, return empty devicePath return attachID, nil </del> <ins> // Don't return attachID as a devicePath. We can reconstruct the attachID using getAttachmentName() return "", nil </ins> } func (c *csiAttacher) WaitForAttach(spec *volume.Spec, _ string, pod *v1.Pod, timeout time.Duration) (string, error) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Followup to In 1.15, we can merge this commit: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: func (adc *attachDetachController) nodeAdd(obj interface{}) { node, ok := obj.(*v1.Node) <ins> // TODO: investigate if nodeName is empty then if we can return // kubernetes/kubernetes/issues/37777 if node == nil || !ok { return } nodeName := types.NodeName(node.Name) adc.nodeUpdate(nil, obj) // kubernetes/kubernetes/issues/37586 // This is to workaround the case when a node add causes to wipe out // the attached volumes field. This function ensures that we sync with // the actual status. adc.actualStateOfWorld.SetNodeStatusUpdateNeeded(nodeName) } func (adc *attachDetachController) nodeUpdate(oldObj, newObj interface{}) { node, ok := newObj.(*v1.Node) // TODO: investigate if nodeName is empty then if we can return </ins> if node == nil || !ok { return }

Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: $ kubectl describe pods test-ipj93 Name: test-ipj93 Namespace: default Node: kubernetes-minion-group-yxra/10.128.0.3 Start Time: Mon, 28 Nov 2016 14:39:28 -0800 Labels: <noneStatus: Pending IP: Controllers: <noneContainers: busybox: Container ID: Image: busybox Image ID: Port: Command: sh -c while true; do echo (hostname)-(cat /proc/stat cut -f 4 -d" "); sleep 30; done QoS Tier: cpu: Burstable memory: BestEffort Requests: cpu: 100m State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test ReadOnly: false default-token-kjoqg: Type: Secret (a volume populated by a Secret) SecretName: default-token-kjoqg Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 57m 57m 1 {default-scheduler } Normal Scheduled Successfully assigned test-ipj93 to kubernetes-minion-group-yxra 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulling pulling image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulled Successfully pulled image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Created Created container with docker id ; Security:[seccomp=unconfined] 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Started Started container with docker id 47m 47m 6 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.


Question: func (adc *attachDetachController) nodeAdd(obj interface{}) { node, ok := obj.(*v1.Node) <ins> // TODO: investigate if nodeName is empty then if we can return // kubernetes/kubernetes/issues/37777 if node == nil || !ok { return } nodeName := types.NodeName(node.Name) adc.nodeUpdate(nil, obj) // kubernetes/kubernetes/issues/37586 // This is to workaround the case when a node add causes to wipe out // the attached volumes field. This function ensures that we sync with // the actual status. adc.actualStateOfWorld.SetNodeStatusUpdateNeeded(nodeName) } func (adc *attachDetachController) nodeUpdate(oldObj, newObj interface{}) { node, ok := newObj.(*v1.Node) // TODO: investigate if nodeName is empty then if we can return </ins> if node == nil || !ok { return }

SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig.


Question: func (adc *attachDetachController) nodeAdd(obj interface{}) { node, ok := obj.(*v1.Node) <ins> // TODO: investigate if nodeName is empty then if we can return // kubernetes/kubernetes/issues/37777 if node == nil || !ok { return } nodeName := types.NodeName(node.Name) adc.nodeUpdate(nil, obj) // kubernetes/kubernetes/issues/37586 // This is to workaround the case when a node add causes to wipe out // the attached volumes field. This function ensures that we sync with // the actual status. adc.actualStateOfWorld.SetNodeStatusUpdateNeeded(nodeName) } func (adc *attachDetachController) nodeUpdate(oldObj, newObj interface{}) { node, ok := newObj.(*v1.Node) // TODO: investigate if nodeName is empty then if we can return </ins> if node == nil || !ok { return }

This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes


Question: SchedulerName:                  config.SchedulerName, Client:                         client, InformerFactory:                informers.NewSharedInformerFactory(client, 0), <del> PodInformer:                    factory.NewPodInformer(client, 0, config.SchedulerName), </del> <ins> PodInformer:                    factory.NewPodInformer(client, 0), </ins> AlgorithmSource:                config.AlgorithmSource, HardPodAffinitySymmetricWeight: config.HardPodAffinitySymmetricWeight, EventClient:                    eventClient,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Scheduler's pod informer has a selector that . This is incorrect. Scheduler must watch all the pods of the cluster, including those scheduled by other schedulers in order to update its cache of the state of the cluster. Otherwise, it will miss important events and the amount of available resources, pods on nodes, ... which are needed for scheduler to work correctly, will not be updated properly. /sig scheduling /assign
cc
For updating it cache, shouldn't the default scheduler rely on node info/status? Also its not very clear, what information the default scheduler needs from other pods (not to be scheduled by it) that it makes decision on? Just trying to understand the issue more clearly.
Scheduler watches informer events to update its cache. Node status is not enough to update all the information that scheduler caches. Scheduler must know what pods are placed on what nodes. For example, in order to check inter-pod affinity/anti-affinity it needs to know the placement of all the pods, including those scheduled by other schedulers.
But this information is present here: So it seems that your PR will now make sure to get the scheduled pods' information directly from the pod informer.
So when I check the history, I see it was here:
have we seen any specific issues due to this change (like between default scheduler and DS controller)?
Yes, but scheduler relies on its pod informers to keep this location up to date. That's exactly why it is important for the scheduler to receive updates for all the pods in the cluster. Yes. We missed the problem at the time. This is problematic only in clusters that use multiple schedulers. I initially thought that DS controller could cause issues as well, but DS controller does not set , so it must not cause an issue. We have seen instances of scheduler's missing certain events in cluster and as a result scheduler's cache getting stale, but I doubt if it is related to this particular issue. This only affects clusters with more than one scheduler. If you run the test that I have written in with the head, it will fail. That tests simulate a multi-scheduler scenario.


Question: import ( "flag" "fmt" <ins> "os" </ins> "path" "strings" "testing"

We for uploading to GCS on , but not on successful e2e runs or failed test -up. We should definitely be dumping those logs for failed cluster-ups, and it would also be nice to throw in the successes too (good to have recent green for comparison).
Getting logs uploaded on a failed test -up seems really important for debugging.
+1 There were just a bunch of failures in scalability test suite (run 1623-1628) that failed because not all system pods were running. Since there are no logs, there is no easy way to debug it. It would be super useful to have all the logs for failed cluster-ups. cc
Unless I'm confused, it looks as if these logs are still not being uploaded. FYI.
For example: gs://kubernetes-jenkins/logs/kubernetes-e2e-gce/7500
That particular build was a timeout. When Jenkins kills the job, the log-collecting code never runs.
Another case we won't collect logs is if the cluster fails to initialize. (Which is a really important place to collect logs, but it's not a trivial fix.)
Moving this into post, since it seems unlikely to get fixed in the near term. feel free to correct me if I'm wrong.
We also don't collect logs on test job timeouts, which is frustrating.
+1
will collect the logs when the cluster fails to initialize. That's not all the cases listed here, but it's a good one.
I'd like to bump the priority for this further - we don't have any logs to help with debugging , which started to happen more often recently - it seems that didn't help. cc
I have too many P0s assigned to me already. This is at least 3rd in line right now.
This one is half year old and is P0 for a month...
From discussion with on hangouts, a vague plan: move log-dumping logic from into a standalone command in something like (feel free to bikeshed the name) update Jenkins to call this command as a post-build step probably remove and the debug stuff recently to . Right now the stuff is pretty GCE/GKE specific and uses ssh. We should probably change to a more provider-agnostic method, but this would be a good start.
Hm, actually, post-build step actually won't work, since we tear down the cluster first.


Question: import ( "flag" "fmt" <ins> "os" </ins> "path" "strings" "testing"

But making it a separate command can still let us call it in more places.
FYI - we also need logs from timed-out runs (not only the failed ones).
moving out into its own binary should help accomplish this.
I'm worried that the best way to do this is to write a bash script that grabs IPs from 's and functions and just does an ssh and cats the logs. Then we could point at that. It's sorta difficult to make work on its own.
I think at this point we're happy with anything that usually works. We can work on making in nice and shiny later.
We now get logs when kube-up fails, which recently happened :) Still some work to do here, I'll leave this issue open until timeouts are handled properly and points here.
- I can't thank you enough!
Ok, is in so we should get logs for kubernetes-e2e-* jobs and PR Jenkins jobs on timeout.
I'm gonna call this done.


Question: # Sets-up etcd encryption. # Configuration of etcd level encryption consists of the following steps: # 1. Writing encryption provider config to disk <del> # 2. Adding experimental-encryption-provider-config flag to kube-apiserver </del> <ins> # 2. Adding encryption-provider-config flag to kube-apiserver </ins> # 3. Add kms-socket-vol and kms-socket-vol-mnt to enable communication with kms-plugin (if requested) # # Expects parameters:

Ref Ref ref /sig auth /kind feature /milestone v1.11 /priority important-longterm /status approved-for-milestone cc
What's need to be done here? I can think only about renaming to and updating the docs. What else?
What's required to promote encryption at rest with static configs to beta?
an example of promoting flags is here: If there's a feature gate, we're going to want to enable it by default too.
/assign
submitted PRs for that: and
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.11 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Will this make it to Beta in 1.12? From sig-auth on 7/11, it sounded like this was missing a release shepherd. Is anyone owning pushing this forward?
I'll do my best to push this. edit: I did not realize that 1.12 feature freeze is today (or so I heard), I don't think I'll be able to make it :slightlyfrowningface:
Ack - that's what I was wondering (with the feature freeze). Thanks
- Is this making it to the 1.12 ?
The current plan is 1.13


Question: } } <ins> func TestUpdateNodeWithMultipleTaints(t *testing.T) { taint1 := createNoExecuteTaint(1) taint2 := createNoExecuteTaint(2) minute := int64(60) pod := testutil.NewPod("pod1", "node1") pod.Spec.Tolerations = []v1.Toleration{ {Key: taint1.Key, Operator: v1.TolerationOpExists, Effect: v1.TaintEffectNoExecute}, {Key: taint2.Key, Operator: v1.TolerationOpExists, Effect: v1.TaintEffectNoExecute, TolerationSeconds: &minute}, } podNamespacedName := types.NamespacedName{Namespace: pod.Namespace, Name: pod.Name} untaintedNode := testutil.NewNode("node1") doubleTaintedNode := testutil.NewNode("node1") doubleTaintedNode.Spec.Taints = []v1.Taint{taint1, taint2} singleTaintedNode := testutil.NewNode("node1") singleTaintedNode.Spec.Taints = []v1.Taint{taint1} stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(pod) holder := &nodeHolder{node: untaintedNode} controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (holder).getNode, getPodsAssignedToNode(fakeClientset)) controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) // no taint holder.setNode(untaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion with no taints") } // no taint -> infinitely tolerated taint holder.setNode(singleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion with permanently tolerated taint") } // infinitely tolerated taint -> temporarily tolerated taint holder.setNode(doubleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) == nil { t.Fatalf("pod not queued for deletion after addition of temporarily tolerated taint") } // temporarily tolerated taint -> infinitely tolerated taint holder.setNode(singleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion after removal of temporarily tolerated taint") } // verify pod is not deleted for _, action := range fakeClientset.Actions() { if action.GetVerb() == "delete" && action.GetResource().Resource == "pods" { t.Error("Unexpected deletion") } } close(stopCh) } </ins> func TestUpdateNodeWithMultiplePods(t *testing.T) { testCases := []struct { description         string

What happened: A Pod was scheduled for deletion because a taint w/ effect was observed in the Node the Pod was assigned to, and after the taint is removed from the Node, the Pod scheduled deletion isn't canceled. After scheduling pod deletion, the only cancels pod deletion if all taints are removed from the node, including any taints that the user relies on for their use-cases (example below). What you expected to happen: After the taint that triggered scheduling pod deletion is removed, cancels the scheduled pod deletion. How to reproduce it (as minimally and precisely as possible): A node is tainted w/ , so any workloads already assigned to it which don't tolerate this taint are evicted. A pod is created w/ tolerations: w/ unspecified and w/ . is assigned to . At some point in time after the previous steps took place, is tainted as . gets an update event for and observes two taints. It proceeds to calculate the minimum time (in seconds) that tolerates for the two taints which at this point is 300 seconds, and marks for deletion in ~300 seconds - this happens in-memory, in a timed-queue. At some point in time after the previous steps took place, and before the 300 seconds trigger happen, taint is removed from . gets an update event for and observes only one taint, . It proceeds to calculate the minimum time (in seconds) that tolerates for the this taint which is infinity, and returns, not canceling the previous deletion. It completely ignores the fact that the taint that triggered deletion is no longer observed. Anything else we need to know?: Worked w/ to figure this out after all pods in a node got evicted without apparent reason. Environment: Kubernetes version (use ): 1.17.3 (but looking at the code seems to affect all of 1.17.x and 1.18.x and master. Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): 5.x Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A cc (owners of this code)
There seems to be at least .


Question: } } <ins> func TestUpdateNodeWithMultipleTaints(t *testing.T) { taint1 := createNoExecuteTaint(1) taint2 := createNoExecuteTaint(2) minute := int64(60) pod := testutil.NewPod("pod1", "node1") pod.Spec.Tolerations = []v1.Toleration{ {Key: taint1.Key, Operator: v1.TolerationOpExists, Effect: v1.TaintEffectNoExecute}, {Key: taint2.Key, Operator: v1.TolerationOpExists, Effect: v1.TaintEffectNoExecute, TolerationSeconds: &minute}, } podNamespacedName := types.NamespacedName{Namespace: pod.Namespace, Name: pod.Name} untaintedNode := testutil.NewNode("node1") doubleTaintedNode := testutil.NewNode("node1") doubleTaintedNode.Spec.Taints = []v1.Taint{taint1, taint2} singleTaintedNode := testutil.NewNode("node1") singleTaintedNode.Spec.Taints = []v1.Taint{taint1} stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(pod) holder := &nodeHolder{node: untaintedNode} controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (holder).getNode, getPodsAssignedToNode(fakeClientset)) controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) // no taint holder.setNode(untaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion with no taints") } // no taint -> infinitely tolerated taint holder.setNode(singleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion with permanently tolerated taint") } // infinitely tolerated taint -> temporarily tolerated taint holder.setNode(doubleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) == nil { t.Fatalf("pod not queued for deletion after addition of temporarily tolerated taint") } // temporarily tolerated taint -> infinitely tolerated taint holder.setNode(singleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion after removal of temporarily tolerated taint") } // verify pod is not deleted for _, action := range fakeClientset.Actions() { if action.GetVerb() == "delete" && action.GetResource().Resource == "pods" { t.Error("Unexpected deletion") } } close(stopCh) } </ins> func TestUpdateNodeWithMultiplePods(t *testing.T) { testCases := []struct { description         string

You can sort of get a feel for what's going on here by looking at these two code snippets: This is what happens in when the node does not have any NoExecute taints: If you have any NoExecute taints on the node processPodOnNode gets called and this routine is really, really eager to evict pods. So : We are currently working around this issue by changing over our NoExecute taints to NoSchedule.
/sig scheduling node
/cc
/remove-sig scheduling scheduler isn't involved in this case.
Which SIG owns the controller-manager or is the node lifecycle manager (and its NoExecuteTaintManager) owned by sig-node alone?
I think node lifecycle is jointly owned by sig-node and sig-cloud-provider
API machinery owns the mechanics of the controller manager (controller loop setup and management), but the individual controllers are SIG-specific
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Looks like we lost some momentum here and I'd like to avoid this falling into the abyss of bugs that never get fixed. When this bug is triggered the outcome is pretty terrible and once it does happen it is very difficult to dig into and understand what actually happened. Did we ever get this routed to the right owner?
adding sig-node leads for triage/routing. this seems like a clearly reproducible bug with significant correctness issues. /assign
thanks for the detailed report and reproducer. to help accelerate this, do you have a form of your reproducer we could add in the e2e suite (even if its failing initially)?
custom taints are considered disruptive, so it couldn't run in the main suite
It's been a while since I wrote tests for the e2e suite but if you are available over slack in case guidance is needed, I'm willing to put the effort.
Do all suites need to pass for a fix to be merged? I'd assume it is the case, otherwise how would a regression be prevented after the fix lands?


Question: } } <ins> func TestUpdateNodeWithMultipleTaints(t *testing.T) { taint1 := createNoExecuteTaint(1) taint2 := createNoExecuteTaint(2) minute := int64(60) pod := testutil.NewPod("pod1", "node1") pod.Spec.Tolerations = []v1.Toleration{ {Key: taint1.Key, Operator: v1.TolerationOpExists, Effect: v1.TaintEffectNoExecute}, {Key: taint2.Key, Operator: v1.TolerationOpExists, Effect: v1.TaintEffectNoExecute, TolerationSeconds: &minute}, } podNamespacedName := types.NamespacedName{Namespace: pod.Namespace, Name: pod.Name} untaintedNode := testutil.NewNode("node1") doubleTaintedNode := testutil.NewNode("node1") doubleTaintedNode.Spec.Taints = []v1.Taint{taint1, taint2} singleTaintedNode := testutil.NewNode("node1") singleTaintedNode.Spec.Taints = []v1.Taint{taint1} stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(pod) holder := &nodeHolder{node: untaintedNode} controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (holder).getNode, getPodsAssignedToNode(fakeClientset)) controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) // no taint holder.setNode(untaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion with no taints") } // no taint -> infinitely tolerated taint holder.setNode(singleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion with permanently tolerated taint") } // infinitely tolerated taint -> temporarily tolerated taint holder.setNode(doubleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) == nil { t.Fatalf("pod not queued for deletion after addition of temporarily tolerated taint") } // temporarily tolerated taint -> infinitely tolerated taint holder.setNode(singleTaintedNode) controller.handleNodeUpdate(nodeUpdateItem{"node1"}) // verify pod is not queued for deletion if controller.taintEvictionQueue.GetWorkerUnsafe(podNamespacedName.String()) != nil { t.Fatalf("pod queued for deletion after removal of temporarily tolerated taint") } // verify pod is not deleted for _, action := range fakeClientset.Actions() { if action.GetVerb() == "delete" && action.GetResource().Resource == "pods" { t.Error("Unexpected deletion") } } close(stopCh) } </ins> func TestUpdateNodeWithMultiplePods(t *testing.T) { testCases := []struct { description         string

i was more inquiring if there was a test that could automate the reproduction, so even just a linked commit may be helpful for someone that wants to pick this up. i am happy to help if you want to reach out over slack later this week.
looks like a unit test could reproduce the issue, opened a WIP fix in
our code is not publicly accessible, I'm sorry. PR linked by seems to cover both the fix and the test that proves it.
/remove-lifecycle frozen
frozen keeps the bot from auto-closing
Ah, sorry then.


Question: if !ftc.Exists() { return false } <del> return !ftc.IsDir() </del> <ins> pathType, err := ftc.hu.GetFileType(ftc.path) if err != nil { return false } return string(pathType) == string(v1.HostPathFile) </ins> } func (ftc *fileTypeChecker) MakeFile() error {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: This feature is lacking coverage, especially for containerized kubelet environments.
/assign since I'm the owner of this feature.
And just fixes a bug for containerized kubelet when the mount file is an empty file.
Are we going to enable these tests in v1.11? We're getting code freeze soon.
There is a local containerized cluster pull and CI job that we can probably leverage. Right now it only runs conformance tests, we may be able to add new suites to enable other tests. Cc who can point us to the owner for the containerized testing effort
yes, seems to be holding up pretty well - https://k8s-,%20master%20(dev)
In I made pull-kubernetes-local-e2e-containerized to run also all test with string. I'm enabling some tests (subpath!) in , feel free to suggest more tests.
Ref
IIUC, the default host path type is . And when host path type is , we expect container runtime to create the host path if it doesn't exist (docker is doing that today). Is that true? If that is, we should revert part of , because container runtime should still create host path if it does not exist. /cc I tried to enforce , and got a lot of errors , e.g.:
Correct. This is mainly for backwards compatibility.
I see. If so, I'll remove the CRI definition "If the hostPath doesn't exist (e.g. hostPath volume), runtimes should report errors". Thanks!
SGTM. Refer to .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale will help fix this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.


Question: if !ftc.Exists() { return false } <del> return !ftc.IsDir() </del> <ins> pathType, err := ftc.hu.GetFileType(ftc.path) if err != nil { return false } return string(pathType) == string(v1.HostPathFile) </ins> } func (ftc *fileTypeChecker) MakeFile() error {

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .


Question: if !ftc.Exists() { return false } <del> return !ftc.IsDir() </del> <ins> pathType, err := ftc.hu.GetFileType(ftc.path) if err != nil { return false } return string(pathType) == string(v1.HostPathFile) </ins> } func (ftc *fileTypeChecker) MakeFile() error {

Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library", "//staging/src/k8s.io/apiserver/pkg/admission:go_default_library", "//staging/src/k8s.io/apiserver/pkg/admission/initializer:go_default_library", <del> "//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library", </del> "//staging/src/k8s.io/client-go/informers:go_default_library", "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/listers/scheduling/v1:go_default_library", <ins> "//staging/src/k8s.io/component-base/featuregate:go_default_library", </ins> ], )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the documentation at the , I tried enabling quota for a namespace other than but I was still not able to create a pod with priorityClass in my namespace with following error: Error from server (Forbidden): error when creating "": pods "memory-demo" is forbidden: pods with system-cluster-critical priorityClass is not permitted in my-namespace namespace What you expected to happen: Expected pod to be created with namespace where I allowed quota. I can see that both ResourceQuota and Priority Plugins are enabled from logs. I0408 17:15:08. ] Loaded 8 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook. I0408 17:15:08. ] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota. How to reproduce it (as minimally and precisely as possible): Follow the steps at Anything else we need to know?: I think, the issue is with priorityPlugin where we are rejecting instead of letting resourceQuota plugin to handle the admission when is enabled. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig scheduling /cc
Same happens to me. Correction, it does not allow to use the default permissions outside of kube-system
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
See also:
/remove-lifecycle stale
/assign Per offline discussion with
Same issue here.


Question: "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library", "//staging/src/k8s.io/apiserver/pkg/admission:go_default_library", "//staging/src/k8s.io/apiserver/pkg/admission/initializer:go_default_library", <del> "//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library", </del> "//staging/src/k8s.io/client-go/informers:go_default_library", "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/listers/scheduling/v1:go_default_library", <ins> "//staging/src/k8s.io/component-base/featuregate:go_default_library", </ins> ], )

We are using namespaces other than kube-system for system pods as it is getting too crowded, therefore we would like to run cluster-node-critical pods in other namespaces.
Waiting for this to be available myself. No reason my Node Exporter dataset has to run in the namespace instead of my namespace.


Question: Expect(err).NotTo(HaveOccurred()) err = framework.WaitForPersistentVolumeClaimPhase( v1.ClaimBound, config.client, claim.Namespace, claim.Name, framework.Poll, 1*time.Minute) <del> Expect(claim.Spec.VolumeName).To(Equal(oldPV.Name)) </del> Expect(err).NotTo(HaveOccurred()) <ins> claim, err = config.client.Core().PersistentVolumeClaims(config.ns).Get(claim.Name, metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred()) Expect(claim.Spec.VolumeName).To(Equal(oldPV.Name)) </ins> // Delete the persistent volume claim: file will be cleaned up and volume be re-created. By("Deleting the persistent volume claim to clean up persistent volume and re-create one") <del> writeCmd, readCmd := createWriteAndReadCmds(volumePath, testFile, testFileContent) _, err = framework.NodeExec(node0.Name, writeCmd) </del> <ins> writeCmd, _ := createWriteAndReadCmds(volumePath, testFile, testFileContent) err = framework.IssueSSHCommand(writeCmd, framework.TestContext.Provider, node0) </ins> Expect(err).NotTo(HaveOccurred()) err = config.client.Core().PersistentVolumeClaims(claim.Namespace).Delete(claim.Name, &metav1.DeleteOptions{}) Expect(err).NotTo(HaveOccurred()) <ins> By("Waiting for a new PersistentVolume to be re-created") </ins> newPV, err := waitForLocalPersistentVolume(config.client, volumePath) Expect(err).NotTo(HaveOccurred()) Expect(newPV.UID).NotTo(Equal(oldPV.UID)) <del> result, err := framework.NodeExec(node0.Name, readCmd) </del> <ins> fileDoesntExistCmd := createFileDoesntExistCmd(volumePath, testFile) err = framework.IssueSSHCommand(fileDoesntExistCmd, framework.TestContext.Provider, node0) </ins> Expect(err).NotTo(HaveOccurred()) <del> Expect(result.Code).NotTo(BeZero(), "file should be deleted across local pv recreation, but exists") </del> }) }) })

8 tests failed, 2 jobs failed, 88 builds failed. Failure stats cover 1 day time range '3 Aug 2017 03:31 UTC' to '4 Aug 2017 03:31 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
This should have been fixed by /cc
The test panic is fixed, but now it's failing because it can't SSH into the node.
/assign


Question: output, err := hexCIDR(test.input) if test.expectErr { if err == nil { <del> t.Error("unexpected non-error") </del> <ins> t.Errorf("case %s: unexpected non-error", test.name) </ins> } } else { if err != nil { <del> t.Errorf("unexpected error: %v", err) </del> <ins> t.Errorf("case %s: unexpected error: %v", test.name, err) </ins> } if output != test.output { <del> t.Errorf("expected: %s, saw: %s", test.output, output) </del> <ins> t.Errorf("case %s: expected: %s, saw: %s", test.name, test.output, output) </ins> } <del> input, err := asciiCIDR(output) </del> <ins> } } } func TestAsciiCIDR(t *testing.T) { tests := []struct { name      string input     string output    string expectErr bool }{ { name:   "IPv4", input:  "01020000/ffff0000", output: "1.2.0.0/16", }, { name:   "IPv4 host", input:  "ac110002/ffffffff", output: "172.17.0.2/32", }, { name:   "IPv6", input:  "2001deadbeef00000000000000000000/ffffffffffffffff0000000000000000", output: "2001:dead:beef::/64", }, { name:   "IPv6 host", input:  "20010000000000000000000000000005/ffffffffffffffffffffffffffffffff", output: "2001::5/128", }, { name:      "invalid CIDR", input:     "malformed", expectErr: true, }, { name:      "non-hex IP", input:     "nonhex/32", expectErr: true, }, { name:      "non-hex mask", input:     "01020000/badmask", expectErr: true, }, } for _, test := range tests { output, err := asciiCIDR(test.input) if test.expectErr { if err == nil { t.Errorf("case %s: unexpected non-error", test.name) } } else { </ins> if err != nil { <del> t.Errorf("unexpected error: %v", err) </del> <ins> t.Errorf("case %s: unexpected error: %v", test.name, err) </ins> } <del> if input != test.input { t.Errorf("expected: %s, saw: %s", test.input, input) </del> <ins> if output != test.output { t.Errorf("case %s: expected: %s, saw: %s", test.name, test.output, output) </ins> } } }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): hexCIDR and CIDR Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): latest on master Environment: Cloud provider or hardware configuration: Bare-metal, 3 node config using KubeAdm OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Linux devstack-77 3.10.0-514.10.2.el7.x8664 SMP Fri Mar 3 00:04:05 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: KubeAdm Others: What happened: Investigating the code, hexCIDR() expects the CIDR passed to be for an IPv4 address and doesn't support an IPv6 CIDR. What you expected to happen: Support either IPv4 or IPv6 CIDRs. How to reproduce it (as minimally and precisely as possible): Anything else we need to know: Will be working on a solution for this. Note: In some of the call paths, a PodIP string is used and passed in as a CIDR with a "/32" appended. At some point, this would need to be updated as well, to use "/128" if an IPv6 PodIP was used.
I see there is an asciiCIDR() method too that needs addressing, but only in the UTs.


Question: output, err := hexCIDR(test.input) if test.expectErr { if err == nil { <del> t.Error("unexpected non-error") </del> <ins> t.Errorf("case %s: unexpected non-error", test.name) </ins> } } else { if err != nil { <del> t.Errorf("unexpected error: %v", err) </del> <ins> t.Errorf("case %s: unexpected error: %v", test.name, err) </ins> } if output != test.output { <del> t.Errorf("expected: %s, saw: %s", test.output, output) </del> <ins> t.Errorf("case %s: expected: %s, saw: %s", test.name, test.output, output) </ins> } <del> input, err := asciiCIDR(output) </del> <ins> } } } func TestAsciiCIDR(t *testing.T) { tests := []struct { name      string input     string output    string expectErr bool }{ { name:   "IPv4", input:  "01020000/ffff0000", output: "1.2.0.0/16", }, { name:   "IPv4 host", input:  "ac110002/ffffffff", output: "172.17.0.2/32", }, { name:   "IPv6", input:  "2001deadbeef00000000000000000000/ffffffffffffffff0000000000000000", output: "2001:dead:beef::/64", }, { name:   "IPv6 host", input:  "20010000000000000000000000000005/ffffffffffffffffffffffffffffffff", output: "2001::5/128", }, { name:      "invalid CIDR", input:     "malformed", expectErr: true, }, { name:      "non-hex IP", input:     "nonhex/32", expectErr: true, }, { name:      "non-hex mask", input:     "01020000/badmask", expectErr: true, }, } for _, test := range tests { output, err := asciiCIDR(test.input) if test.expectErr { if err == nil { t.Errorf("case %s: unexpected non-error", test.name) } } else { </ins> if err != nil { <del> t.Errorf("unexpected error: %v", err) </del> <ins> t.Errorf("case %s: unexpected error: %v", test.name, err) </ins> } <del> if input != test.input { t.Errorf("expected: %s, saw: %s", test.input, input) </del> <ins> if output != test.output { t.Errorf("case %s: expected: %s, saw: %s", test.name, test.output, output) </ins> } } }

Appears that it is expecting to reverse hex back to string, but test is not showing case of mask being properly applied. For example, "2001::5/64" when converted to hex and then back to string will be "2001::/64". Will update test to be more accurate.
/area networking
These labels do not exist in this repository: . <detailsIn response to : /area networking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details
/sig network
One question I have... Can the callers (bandwidth?) of the function work with an IPv6 based hex CIDR? In the future, if we support pod IPs that are IPv6 addresses, and we modify the code to add /128 vs /32, as is done currently, this would create an IPv6 based hex CIDR.
The only caller is currently the ingress/egress shaping function in kubelet. That code also needs to be updated for IPv6. For example, assumes IPv4 when adding CIDRs to the shaper list.
Agree.I wanted to separate this out, partly to make the change smaller, but also, when I looked at the callers (this shaping functionality) I wasn't sure if it would support IPv6. I looked a bit at the "tc" command and saw some notes that indicate that it doesn't support IPv6 yet. Is that still accurate? Would it be ok to tackle that separately?


Question: return updateServiceListener(serviceName, listenPort, true) } <del> // DeleteServiceListener Request addition of a listener for a service's health check </del> <ins> // DeleteServiceListener Request deletion of a listener for a service's health check </ins> func DeleteServiceListener(serviceName types.NamespacedName, listenPort int) bool { return updateServiceListener(serviceName, listenPort, false) }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? Currently, the 4th parameter name of updateServiceListener in kube-proxy is addOrDelete, However, from the func caller, we just pass for AddServiceListener, and for DeleteServiceListener, So, I think the name addOrDelete is unreasonable. I prefer to change the parameter name addOrDelete to add so that avoid making people confused. In addition, there is a typo, We should change addition to deletion! Also, there is a sentence breaking, we should change from No Service Endpoints Not Found =No Service Endpoints are found Thanks!
Fixed!


Question: continue } <del> epa := v1.EndpointAddress{ IP:       pod.Status.PodIP, NodeName: &pod.Spec.NodeName, TargetRef: &v1.ObjectReference{ Kind:            "Pod", Namespace:       pod.ObjectMeta.Namespace, Name:            pod.ObjectMeta.Name, UID:             pod.ObjectMeta.UID, ResourceVersion: pod.ObjectMeta.ResourceVersion, }} </del> <ins> epa := *podToEndpointAddress(pod) </ins> hostname := pod.Spec.Hostname if len(hostname) > 0 && pod.Spec.Subdomain == service.Name && service.Namespace == pod.Namespace {

/kind bug What happened: On a large cluster with enough pods, endpoints take as long as 10 minutes to show up after their associated pods and service are created. We believe that this is because every pod change results in an update of the addresses within the endpoint since they refer to a pod via a whose changes every time a pod change is observed. All these non-update updates eventually overwhelm the endpoint controller's queue rate limit and the queue starts to grow. While an increase in the queue's rate limit can alleviate the problem, an admin can keep a much lower limit if the non-update updates are dropped rather than processed. What you expected to happen: When a pod related to an endpoint is updated, I would expect the endpoint to update only if there is a change in the pod that relates to the endpoint, such as a change in the pod's IP address. How to reproduce it (as minimally and precisely as possible): a pod and associated service: the endpoint for changes: the watch running, and in another session, edit a pod to add a label: Then add another label on the line after the label, such as that the watched endpoint changed despite there not being any meaningful change to the fields the endpoint cares about. Anything else we need to know?: The PR addresses this issue. Environment: Kubernetes version (use ):
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig scalability
what ever happened with the effort to move leases off of Endpoints?


Question: import ( "fmt" <ins> "strconv" "strings" </ins> "k8s.io/klog/v2" utilexec "k8s.io/utils/exec"

Kubernetes should resize filesystem on a volume when it's created as a clone of a smaller volume / smaller snapshot is restored into it. Example: takes a snapshot of 1 GiB volume with ext3 filesystem on it. restores the snapshot into 2 GiB volume. 2.1 CSI driver restores the snapshot into a bigger volume, but ext3 filesystem still uses only 1 GiB of space. driver should make sure that the filesystem on the volume has been resized before the volume is mounted into a pod. There are few steps needed: [x] Clarify behavior of CSI call, [ ] Update CSI drivers to the new behavior. [x] Move filesystem resize code from kubernetes/kuberenetes to kubernetes/mount-utils, so all CSI drivers can re-use the same code [x] Enhance the code above to compare the volume size from block device perspective and from filesystem perspective. See for proof of concept. [ ] File issues in all community CSI drivers that we care about & fix them. [ ] Update csi-tests to (optionally?) check for volume size when cloning / restoring snapshot to a bigger size.
/assign /sig storage /priority important-soon
True, considering the CSI spec allows restore to a bigger size volume, we should have the support from CO.
This is hard, Kubernets PV controller overwrites on binding. Therefore the external-provisioner cannot create PVC as "marked for resize on node", i.e. with = = the real size of the new volume) and with = size of the original volume. We either need to change the binding mechanism not to overwrite or add a flag (annotation?) that marks the volume for resize on node, even when .
Talked to The external-provisioner could mark for resize on node: Set to the original volume size. Add condition to the PVC. As result, the extenal-provisioner needs to modify , which needs new RBAC and is a breaking change. The PV controller could respect , if set. We discussed and rejected adding a new flag / annotation to PV by the external-provisioner for kubelet to call NodeExpandVolume on the first mount. Such a flag would need to be cleared by kubelet and kubelet can't modify PVs.
From latest discussion last week, I believe the conclusion was to have external-provisioner set to the smaller size.


Question: import ( "fmt" <ins> "strconv" "strings" </ins> "k8s.io/klog/v2" utilexec "k8s.io/utils/exec"

Then we just let the rest of the normal resize flow continue. No need to have provisioner set the conditions.
we agreed to set conditions too because we want to be as close as possible to external-resizer. If there is some tooling that depends on condition for resizing pending on node, we don't want to break any such tooling. currently kubelet does not depend on condition for node expansion though.
Why can't we let resize controller set the condition when it sees pvc size != pv size?
hmm good point. I think that should work but we may have to verify that flow.
I think we can do that, but only with a change to the CSI spec wording, because the current implication is that should not be necessary.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
can you summarize the current conclusion/state of this? /remove-lifecycle rotten
Updated the issue with TODO list.
should this issue stay open to keep track of remaining issues?
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: import ( "fmt" <ins> "strconv" "strings" </ins> "k8s.io/klog/v2" utilexec "k8s.io/utils/exec"

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.


Question: import ( "fmt" <ins> "strconv" "strings" </ins> "k8s.io/klog/v2" utilexec "k8s.io/utils/exec"

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale /reopen
Reopened this issue. <detailsIn response to : /remove-lifecycle stale /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The issue has been marked as an important bug and triaged. Such issues are automatically marked as frozen when hitting the rotten state to avoid missing important bugs. Please send feedback to sig-contributor-experience at . /lifecycle frozen
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted /priority long-term what is the latest status on this?
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accepted /priority long-term what is the latest status on this? Instructions for interacting with me using PR comments are available .


Question: import ( "fmt" <ins> "strconv" "strings" </ins> "k8s.io/klog/v2" utilexec "k8s.io/utils/exec"

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-long-term
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority important-long-term Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello I've checked what is the current status of the drivers that were involved in this effort and I think we might be able to close this as all upstream work is done. Please take a look at the table below and let me know if anything is missing.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: var ( downwardAPIHugePages featuregate.Feature = "DownwardAPIHugePages" execProbeTimeout     featuregate.Feature = "ExecProbeTimeout" <ins> csiMigration         featuregate.Feature = "CSIMigration" </ins> ) func skipInternalf(caller int, format string, args ...interface{}) {

Right now, we have e2e test for that executes only CSI mock driver with artificial limit of 2 volumes per node. We need a test with real CSI driver(s). Proposal: Allow individual test storage drivers to specify if they support CSI limits. Add a test that picks a random node and creates as many pods with a PV that the volume can handle (128 on GCE!). Check that the node can't handle any other volume. Make this test part of . Add similar test(s) for in-tree volume plugins. Make sure the tests work with in-tree volume plugins migrated to CSI.
/sig storage
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
This sounds important /remove-lifecycle rotten /lifecycle frozen
/priority backlog
To add to this I believe we should test that an in-tree plugin and its "replacement" CSI Driver share the same attach limits (both max and current usage)
/cc
I see this test has been here: But I want to add a more basic/faster version of the test that simply checks if a driver has then has been set in CSINode, similar to wdyt, would it be useful?
This is a really old issue :-D. I think the required test has been implemented, however, we do not run it in CI, because it takes too much time to provision 128 PDs that a GCP node can handle. /unassign That sounds useful!
/assign opened to close this for good :)


Question: } }, "required": [ <del> "service", </del> "groupPriorityMinimum", "versionPriority" ],

/sig api-machinery /kind bug /cc this makes any updates on a "Local" apiservice failing by kubectl (unless you disable the client-side validation). a quick repro: (edit whatever a local apiservice) a label "foo: bar" will fail on the schema validation # "" was not valid: # * : Invalid value: "The edited file failed validation": ValidationError(): missing required field "service" in client-side v1.15.0 server-side v1.15.2 referencing the published openapi v2 for apiservice#spec below:
/label help-wanted
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can on this if no one else is currently working on it.
sure, feel free to tag me to request reviewing..
awesome, I will /assign
are you still working on this one?


Question: as.Nil(err) runContainerOpts3 := testManager.GetDeviceRunContainerOptions(newPod, &newPod.Spec.Containers[0]) as.Equal(1, len(runContainerOpts3.Envs)) <ins> // Requesting to create a pod that requests resourceName1 in init containers and normal containers // should succeed with devices allocated to init containers reallocated to normal containers. podWithPluginResourcesInInitContainers := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID: uuid.NewUUID(), }, Spec: v1.PodSpec{ InitContainers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity1, }, }, }, }, Containers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, }, }, } podsStub.updateActivePods([]*v1.Pod{podWithPluginResourcesInInitContainers}) err = testManager.Allocate(nodeInfo, &lifecycle.PodAdmitAttributes{Pod: podWithPluginResourcesInInitContainers}) as.Nil(err) podUID := string(podWithPluginResourcesInInitContainers.UID) initCont1 := podWithPluginResourcesInInitContainers.Spec.InitContainers[0].Name initCont2 := podWithPluginResourcesInInitContainers.Spec.InitContainers[1].Name normalCont1 := podWithPluginResourcesInInitContainers.Spec.Containers[0].Name normalCont2 := podWithPluginResourcesInInitContainers.Spec.Containers[1].Name initCont1Devices := testManager.podDevices.containerDevices(podUID, initCont1, resourceName1) initCont2Devices := testManager.podDevices.containerDevices(podUID, initCont2, resourceName1) normalCont1Devices := testManager.podDevices.containerDevices(podUID, normalCont1, resourceName1) normalCont2Devices := testManager.podDevices.containerDevices(podUID, normalCont2, resourceName1) as.True(initCont2Devices.IsSuperset(initCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont2Devices)) as.Equal(0, normalCont1Devices.Intersection(normalCont2Devices).Len()) </ins> } func TestSanitizeNodeAllocatable(t *testing.T) {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area hw-accelerators What happened: In the context of the device plugin, when using init containers in your pod spec, the current behavior will not reuse devices between init containers and regular containers. This is problematic as Pod scheduling is currently wrong: , the current predicate algorithm rightly assumes the maximum number of devices in use at one time is . However, the pod predicate ran on the node is the sum of devices used by regular and init containers. Meaning that your pod can bounce back between the scheduler and a node infinitely. Also given that many of the extended resources are typically more expensive that the built-in ones like CPU, Memory and Storage this bug has very real financial drawbacks in terms of resource usage (e.g: using 4 GPUs on a pod that only uses 2 GPUs at a time). Furthermore, this unexpected behavior was caught during implementation (but we did not catch the scheduling implications) and decision was taken to ignore it for the first implementation. We need to document it for 1.9 What you expected to happen: Devices can be reused by Init Containers Correct resource accounting In terms of the current behavior of the Allocate call to the device plugin, I'd expect the allocate call to be done before every Init containers start and once at the end of the last Init container (for regular containers). /assign
Yep, I realized the inconsistency between the predicate behavior of scheduler and that of node too. Having said that, considering we treat a pod as a whole when predicating currently, it's kind of difficult to reuse the runtime informations between init containers and regular containers. The real crux of the matter might be whether the scenario of requiring extended resources in init containers is common, if so, the issue should be addressed ASAP.
If I remembered correctly, the current design to improve DP includes a reconciliation in admit phase of Pod, it may be able to fix Allocate call for
can you point me to said design ? I'm not sure that the Allocate call for init containers is planned to be fixed as described.
Can you guys please point to the code where is happening. Looks to me that same logic is executed on both the scheduler and the node to calculate a pod's resource request and which is  On the node, function call tree looks like this: Is it possible to add few more words in explanation?


Question: as.Nil(err) runContainerOpts3 := testManager.GetDeviceRunContainerOptions(newPod, &newPod.Spec.Containers[0]) as.Equal(1, len(runContainerOpts3.Envs)) <ins> // Requesting to create a pod that requests resourceName1 in init containers and normal containers // should succeed with devices allocated to init containers reallocated to normal containers. podWithPluginResourcesInInitContainers := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID: uuid.NewUUID(), }, Spec: v1.PodSpec{ InitContainers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity1, }, }, }, }, Containers: []v1.Container{ { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, { Name: string(uuid.NewUUID()), Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceName(resourceName1): resourceQuantity2, v1.ResourceName(resourceName2): resourceQuantity2, }, }, }, }, }, } podsStub.updateActivePods([]*v1.Pod{podWithPluginResourcesInInitContainers}) err = testManager.Allocate(nodeInfo, &lifecycle.PodAdmitAttributes{Pod: podWithPluginResourcesInInitContainers}) as.Nil(err) podUID := string(podWithPluginResourcesInInitContainers.UID) initCont1 := podWithPluginResourcesInInitContainers.Spec.InitContainers[0].Name initCont2 := podWithPluginResourcesInInitContainers.Spec.InitContainers[1].Name normalCont1 := podWithPluginResourcesInInitContainers.Spec.Containers[0].Name normalCont2 := podWithPluginResourcesInInitContainers.Spec.Containers[1].Name initCont1Devices := testManager.podDevices.containerDevices(podUID, initCont1, resourceName1) initCont2Devices := testManager.podDevices.containerDevices(podUID, initCont2, resourceName1) normalCont1Devices := testManager.podDevices.containerDevices(podUID, normalCont1, resourceName1) normalCont2Devices := testManager.podDevices.containerDevices(podUID, normalCont2, resourceName1) as.True(initCont2Devices.IsSuperset(initCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont1Devices)) as.True(initCont2Devices.IsSuperset(normalCont2Devices)) as.Equal(0, normalCont1Devices.Intersection(normalCont2Devices).Len()) </ins> } func TestSanitizeNodeAllocatable(t *testing.T) {

PR that the mentioned change: Code in question is here:
Above looks incorrect description. Predicate that run on the scheduler and on the node is same i.e max(init, regular). Pod admission will pass at the node. Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'.
Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'. Absolutely correct, sorry to make you confused, I think Renaud name it mainly bacause the allocate func is supposed to be called during the predicate phase, that is, passed in as a part of PredicateAdmitHandler.
The original idea was onced listed in as a sub-task: Now it's removed (I don't know why ...), while I believe the idea was: device resources allocation during pod admit phase. This comment is also related if needed during reconcile in 1 and cache these devices info cached devices later during container create phase
Handle device resources allocation during pod admit phase. This comment (comment) is related Call Allocate() if needed during reconcile in 1 and cache these devices info Use cached devices later during container create phase Current code behaviors exactly as you described, but to address the issue, the key point might be to decide which devices of init containers could be reused by which regular containers, and generate cache accordingly in step2; when creating containers, we simply fetch runtime information from cache.
Thanks a lot for filing this issue, I think for 1.9, we probably want to list this as a known issue and may advise people to not request gpu resources in init containers. We can target to fix this problem in 1.10.
Agreed We don't have a clear solution for this tough. It's pretty clear to me that device resources allocation during pod admit phase is not the right model when init containers comes into the picture. GPUs are pretty simple example. On of the things we plan to do is scrub the GPU memory during the Allocate phase. If GPUs are re-used between initContainers and regular containers the current model doesn't fit.