Question: return nil } <del> // Ensures that the load balancer associated with the given service is deleted, // doing the deletion if necessary. Should always be retried upon failure. func (s *ServiceController) ensureLBDeleted(service *api.Service) error { // This is only needed because not all delete load balancer implementations // are currently idempotent to the LB not existing. if _, exists, err := s.balancer.GetTCPLoadBalancer(s.loadBalancerName(service), s.zone.Region); err != nil { return err } else if !exists { return nil } if err := s.balancer.DeleteTCPLoadBalancer(s.loadBalancerName(service), s.zone.Region); err != nil { return err } return nil } </del> // ListKeys implements the interface required by DeltaFIFO to list the keys we // already know about. func (s *serviceCache) ListKeys() []string {

The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.


Question: return nil } <del> // Ensures that the load balancer associated with the given service is deleted, // doing the deletion if necessary. Should always be retried upon failure. func (s *ServiceController) ensureLBDeleted(service *api.Service) error { // This is only needed because not all delete load balancer implementations // are currently idempotent to the LB not existing. if _, exists, err := s.balancer.GetTCPLoadBalancer(s.loadBalancerName(service), s.zone.Region); err != nil { return err } else if !exists { return nil } if err := s.balancer.DeleteTCPLoadBalancer(s.loadBalancerName(service), s.zone.Region); err != nil { return err } return nil } </del> // ListKeys implements the interface required by DeltaFIFO to list the keys we // already know about. func (s *serviceCache) ListKeys() []string {

I'll periodically check to see whether anything is leaked after this point. I apologize if the deletions affected any running tests - everything looked to only be from GKE and parallel e2e runs.


Question: start_kube_controller_manager start_kube_scheduler start_kube_addons <ins> start_cluster_autoscaler </ins> } 2>&1 | logger --priority daemon.info -t ${UPSTART_JOB} end script

For some reason they stopped working and we even don't create manifest file for cluster autoscaler. I used the following command to start the cluster:
It's possible it's related to


Question: // to the node. This list can be used to determine volumes that are either in-use // or have a mount/unmount operation pending. GetAttachedVolumes() []AttachedVolume <ins> // SyncReconstructedVolume check the volume.outerVolumeSpecName in asw and // the one populated from dsw , if they do not match, update this field from the value from dsw. SyncReconstructedVolume(volumeName v1.UniqueVolumeName, podName volumetypes.UniquePodName, outerVolumeSpecName string) </ins> } // MountedVolume represents a volume that has successfully been mounted to a pod.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --We have even encountered such case that pod continuously reports “Unable to attach or mount volumes for pod;...” when kubelet restarted. The reasons are listed in details as below: , as result of apiserver instantaneous unstablility, dswp get pvc‘s pv spec failed, then the pvc volume does not be to dsw. , syncStates() lists all the volume directories locally, and recontructs the volume from local volume, if the volume isn't in dsw, it will put the recontructed volume to asw, but now the volume.OuterVolumeSpecName isn't correct. chance, at this time, In , the fail to pvc volume is to dsw again normally in reconcile() -mountAttachVolumes(), it compares dsw and asw by volume's innerVolumename, it's all the same. but it could not sense the OuterVolumeSpecName dismatch. so it will not re-mount again to update asw cache , it checks pod's volume attach or mount success by comparing volume.OuterVolumeSpecName and , but this pv'c OuterVolumeSpecName isn't correct in asw. so pod continuously reports “Unable to attach or mount volumes for pod;...” reconcile() could sense the reconstruct volume's dismatch to the dsw, then re-mount again to make volume cache consistency Kubernetes version (use ): 1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
CC
I think this is a point that can be optimized. /cc
I will give a fix for this
Thanks for reporting the issue, this innerSpecName and outerSpecName are confusing and cause issues. Please ping me for review when you have a fix. Thanks!
Please don't triage your own issues immediately, give someone else a chance to review them :( /sig storage
okay, I know


Question: } }) <ins> It("should provision storage with non-default reclaim policy Retain", func() { framework.SkipUnlessProviderIs("gce", "gke") test := testsuites.StorageClassTest{ Name:           "HDD PD on GCE/GKE", CloudProviders: []string{"gce", "gke"}, Provisioner:    "kubernetes.io/gce-pd", Parameters: map[string]string{ "type": "pd-standard", }, ClaimSize:    "1Gi", ExpectedSize: "1Gi", PvCheck: func(volume *v1.PersistentVolume) error { return checkGCEPD(volume, "pd-standard") }, } class := newStorageClass(test, ns, "reclaimpolicy") retain := v1.PersistentVolumeReclaimRetain class.ReclaimPolicy = &retain claim := newClaim(test, ns, "reclaimpolicy") claim.Spec.StorageClassName = &class.Name pv := testsuites.TestDynamicProvisioning(test, c, claim, class) By(fmt.Sprintf("waiting for the provisioned PV %q to enter phase %s", pv.Name, v1.VolumeReleased)) framework.ExpectNoError(framework.WaitForPersistentVolumePhase(v1.VolumeReleased, c, pv.Name, 1*time.Second, 30*time.Second)) By(fmt.Sprintf("deleting the storage asset backing the PV %q", pv.Name)) framework.ExpectNoError(framework.DeletePDWithRetry(pv.Spec.GCEPersistentDisk.PDName)) By(fmt.Sprintf("deleting the PV %q", pv.Name)) framework.ExpectNoError(framework.DeletePersistentVolume(c, pv.Name), "Failed to delete PV ", pv.Name) framework.ExpectNoError(framework.WaitForPersistentVolumeDeleted(c, pv.Name, 1*time.Second, 30*time.Second)) }) </ins> It("should not provision a volume in an unmanaged GCE zone.", func() { framework.SkipUnlessProviderIs("gce", "gke") var suffix string = "unmananged"

Testgrid: https://k8s- https://k8s- https://k8s- https://k8s- Sample failures: This started failing across multiple test jobs last night. The culprit appears to be PR over to you. /kind failing-test /priority critical-urgent /sig storage
/milestone v1.13
/assign
GitHub didn't allow me to assign the following users: mkimuram. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think should delete the PVC and wait for it to be deleted.
Thank you for your comment. I haven't yet catching up, but as for the PVC deletion code in , which is changed in the PR, PVC deletion is done inside for delete policy case or for retain policy case. Isn't it enough? In addition, could you let me know how I can check the matrix for the failure that we are focusing? I couldn't find the failure for provisioning tests themselves, there.
Is my understanding right that the purpose of is to check whether resource is leaking by checking the diff of resource between the test? If so, is it correct that the sample output above means that one pd is leaking, not the leak of pv or pvc itself? I found that in the test for retain policy is removed in this PR when it is moved to , because it is gce specific logic and looks unnecessary. Do you think that it could be a cause of this issue? If so, should deleting the backend storage be generally required before removing the dynamic provisioned PV with retain policy?


Question: "description": "EndpointPort is a tuple that describes a single port.", "properties": { "appProtocol": { <del> "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and http://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", </del> <ins> "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and https://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", </ins> "type": "string" }, "name": {

We should make the change in PR API reference links to should use HTTPS, not HTTP Visit h and look at the IANA hyperlinks Once this fix is made, a PR to is needed to update the docs (by autogenerating them). This typically would be deferred until the next Kubernetes release. v1.23 not applicable not applicable not applicable not applicable not applicable
/sig docs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I opened this issue so that someone else can open a new PR that is equivalent to .
/retitle Update links to IANA for Service Name and Transport Protocol Port Number Registry
I would like to work on this issue. May I assign it to myself?
Anyone - including you - is welcome to work on this issue! So long as you've you can open a pull request to propose a fix.
Thanks
/assign
I have made the PR. I hope this works. If not then will be happy to do the necessary changes.
Had to close the previous PR because of issues, here is the new one I made.


Question: return i.doCmd(s, nil, ipvsCmdDelService) } <ins> // Flush deletes all existing services in the passed // handle. func (i *Handle) Flush() error { _, err := i.doCmdWithoutAttr(ipvsCmdFlush) return err } </ins> // NewDestination creates a new real server in the passed ipvs // service which should already be existing in the passed handle. func (i *Handle) NewDestination(s *Service, d *Destination) error {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently, we flush ipvs services by deleting them one by one. That's because there was no flush API in libnetwork. Now, libnetwork supports flush API, I think it's time to rsync. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig network
/area kube-proxy
/assign


Question: <del> apiVersion: v1 kind: Service metadata: name: elasticsearch-discovery labels: component: elasticsearch role: master spec: selector: component: elasticsearch role: master ports: - name: transport port: 9300 protocol: TCP </del>

Presently, the elasticsearch example Kubernetes configurations target elasticsearch v1.7.1, which is quite old. As of this writing, the latest elasticsearch release is v5.6.2, with many deployment changes since then. Could the elasticsearch examples be updated to target the v5 series of elasticsearch? See for a more up to date reference configuration.
cc
hey it's been a long while since I last contributed to the docs. i can do it soon but i need pointers on how to properly do it.
hey, so we need to update the stuff under examples/elasticsearch/ *&repos=kubernetes So basically update the images, yaml files etc and test that everything works (and submit a PR)
/sig testing
Ah, so there's no documentation updates anymore. OK I'll take care of it, assign it to me.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
GitHub didn't allow me to assign the following users: me. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign me Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign pires
/remove-lifecycle stale
it seems that the example moved to I am now reading that repo guidelines and will update this there. I'll also open a PR that cleans this folder in the current repo and closes this issue.


Question: }{ "backoffLimit 0 should have 1 pod active": { 1, 1, 0, <del> true, []int32{0}, </del> <ins> true, []int32{0}, v1.PodRunning, </ins> 1, 0, 0, nil, "", }, "backoffLimit 1 with restartCount 0 should have 1 pod active": { 1, 1, 1, <del> true, []int32{0}, </del> <ins> true, []int32{0}, v1.PodRunning, </ins> 1, 0, 0, nil, "", }, <del> "backoffLimit 1 with restartCount 1 should have 0 pod active": { </del> <ins> "backoffLimit 1 with restartCount 1 and podRunning should have 0 pod active": { </ins> 1, 1, 1, <del> true, []int32{1}, </del> <ins> true, []int32{1}, v1.PodRunning, </ins> 0, 0, 1, &jobConditionFailed, "BackoffLimitExceeded", }, <del> "too many job failures - single pod": { </del> <ins> "backoffLimit 1 with restartCount 1 and podPending should have 0 pod active": { 1, 1, 1, true, []int32{1}, v1.PodPending, 0, 0, 1, &jobConditionFailed, "BackoffLimitExceeded", }, "too many job failures with podRunning - single pod": { </ins> 1, 5, 2, <del> true, []int32{2}, </del> <ins> true, []int32{2}, v1.PodRunning, </ins> 0, 0, 1, &jobConditionFailed, "BackoffLimitExceeded", }, <del> "too many job failures - multiple pods": { </del> <ins> "too many job failures with podPending - single pod": { 1, 5, 2, true, []int32{2}, v1.PodPending, 0, 0, 1, &jobConditionFailed, "BackoffLimitExceeded", }, "too many job failures with podRunning - multiple pods": { 2, 5, 2, true, []int32{1, 1}, v1.PodRunning, 0, 0, 2, &jobConditionFailed, "BackoffLimitExceeded", }, "too many job failures with podPending - multiple pods": { </ins> 2, 5, 2, <del> true, []int32{1, 1}, </del> <ins> true, []int32{1, 1}, v1.PodPending, </ins> 0, 0, 2, &jobConditionFailed, "BackoffLimitExceeded", }, "not enough failures": { 2, 5, 3, <del> true, []int32{1, 1}, </del> <ins> true, []int32{1, 1}, v1.PodRunning, </ins> 2, 0, 0, nil, "", }, }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I create a job with initcontainer with restartPolicy : when the initContainer exec failed and it will be restarted, the pod Phase will be pending all the time. but in JobController, the only care the running pod. So this pod will be always restarted. What you expected to happen: when restart count to backoffLimit, the job failed. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig apps
/assign


Question: import ( "fmt" <ins> "os/exec" </ins> "reflect" "testing" "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/resource" <ins> "k8s.io/kubernetes/pkg/util/codeinspector" </ins> "k8s.io/kubernetes/plugin/pkg/scheduler/algorithm" "k8s.io/kubernetes/plugin/pkg/scheduler/schedulercache" )

I just noticed you forgot to register the image locality priority function in in I also missed it when I reviewed your PR. I have fixed this in . But it would be great if you could figure out a mechanism that would ensure that all predicate/priority functions that are tested (you can assume they will always live under ) are registered, and fail the test if not. That will allow us to prevent this mistake in the future.
Thanks! Will fix this soon!
Seems need to wait for in before I send PR? Otherwise this test will always fail ...


Question: "fmt" "net" "path/filepath" <ins> "strings" </ins> "sync" "testing" "time" authenticationv1 "k8s.io/api/authentication/v1" v1 "k8s.io/api/core/v1" <ins> storagev1 "k8s.io/api/storage/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/labels" </ins> "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/informers" clientset "k8s.io/client-go/kubernetes"

After the cluster was updated to enable PD CSI Driver, there was a race condition where AD Controller (kube-controller-manager) started to BEFORE the PD CSI Driver was up, thus AD Controller gave us the error Then AD Controller triggered the disk detach because the disk/volume was not populated in the desired state of world. Unfortunately the disk was in use during this detach and got corrupted. Since cluster upgrades can cause the controller manager and CSI drivers to start at different times, the AD controller needs to be more conservative when detaching We have not been able to reproduce this, it came up from a customer. /sig storage /assign <detailsk8s 1.23, CSI migration enabled </details<detailsGKE </details<details</details<details</details<details</details<details</details
/triage accepted
I think there's potentially two changes we should do: 1) for the initial DSW population, we should skip the check that if the csi driver is not installed. 2) in general, we should not skip volumes during dsw initial population if an error is returned. If we had some temporary api server error then that would cause volumes to not get to dsw and similar behavior happen. cc
for a repro, maybe you can try something like: a cluster with csi migration enabled (control plane + nodes) a pod with intree-volume and get it running the csi driver node daemonset. verify the driver is removed from CSINode kube-controller-manager
cc
/cc was trying to repro this, but on GKE with the managed driver which made it complicated.
Yeah maybe gce kube-up would be easier for a repro
If CSINode says that it has migrated GCP PD volume plugin to CSI, then IMO A/D controller should always attach GCP PD volumes to it using CSI, regardless if the CSI driver is present or not. In-tree volume plugin in the kubelet is disabled and can't mount any in-tree volumes anyway. To flip the migration feature gate / upgrade / downgrade kubelet, the cluster admin must drain the node first to make sure everything is detached. A/D controller should not see change of CSINode migration annotations with any volume attached to it.
/priority important-soon
Another option could be to process these volumes anyways but add them to DSOW with migration status as "uncertain", so as they can be re-processed later.


Question: "fmt" "net" "path/filepath" <ins> "strings" </ins> "sync" "testing" "time" authenticationv1 "k8s.io/api/authentication/v1" v1 "k8s.io/api/core/v1" <ins> storagev1 "k8s.io/api/storage/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/labels" </ins> "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/informers" clientset "k8s.io/client-go/kubernetes"

But surely it looks like might be easiest unless it has any downsides.
There wasn't a node update in this case---there was a control plane update which restarted the PD CSI driver, and it looks like the CSIDriver resource was updated or changed in some way which raced with the kube-controller-manager restart.
Is there anyway we can know if the csi driver is up and running(like a health check ping) before populating DSW?
We use the CSINode registration to determine that. The discussion here I think is concluding that we should not use that as a criteria for adding the volume to DSW. We only need to check that CSI migration is enabled on the node.
+1 we need to be very resilient against races, I think we have to assume a driver is in process of being installed at any point in the AD controller reconcilliation.
/assign sunnylovestiramisu
Reproduced with the steps: In /var/log/kube-controller-
Tested with the and doing the reproduce steps above. The error of does not show up anymore. The error message still get logged in the kube-controller- for debugging purpose: Ran Ran On the GCP console, the disk is used by node: Ran I redeploy the pd-csi-driver and then make sure it is up and running. Ran


Question: // It must return ("", cloudprovider.InstanceNotFound) if the instance does // not exist or is no longer running. func (ss *scaleSet) GetInstanceIDByNodeName(name string) (string, error) { <del> _, _, vm, err := ss.getVmssVM(name) </del> <ins> managedByAS, err := ss.isNodeManagedByAvailabilitySet(name) </ins> if err != nil { <del> if err == ErrorNotVmssInstance { glog.V(4).Infof("GetInstanceIDByNodeName: node %q is managed by availability set", name) // Retry with standard type because nodes are not managed by vmss. return ss.availabilitySet.GetInstanceIDByNodeName(name) } </del> <ins> glog.Errorf("Failed to check isNodeManagedByAvailabilitySet: %v", err) return "", err } if managedByAS { // vm is managed by availability set. return ss.availabilitySet.GetInstanceIDByNodeName(name) } _, _, vm, err := ss.getVmssVM(name) if err != nil { </ins> return "", err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: VMAS nodes are reporting error in vmss cluster (vmType set to vmss). What you expected to happen: Node should register correctly in the kubernetes cluster. How to reproduce it (as minimally and precisely as possible): Create a VMAS nodes inside vmss cluster. Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure


Question: t.Errorf("Expect node port type service, got none") } } <ins> func TestLoadBalanceSourceRanges(t *testing.T) { ipt := iptablestest.NewFake() ipvs := ipvstest.NewFake() ipset := ipsettest.NewFake(testIPSetVersion) fp := NewFakeProxier(ipt, ipvs, ipset, nil) svcIP := "10.20.30.41" svcPort := 80 svcLBIP := "1.2.3.4" svcLBSource := "10.0.0.0/8" svcPortName := proxy.ServicePortName{ NamespacedName: makeNSN("ns1", "svc1"), Port:           "p80", } epIP := "10.180.0.1" makeServiceMap(fp, makeTestService(svcPortName.Namespace, svcPortName.Name, func(svc *api.Service) { svc.Spec.Type = "LoadBalancer" svc.Spec.ClusterIP = svcIP svc.Spec.Ports = []api.ServicePort{{ Name:     svcPortName.Port, Port:     int32(svcPort), Protocol: api.ProtocolTCP, }} svc.Status.LoadBalancer.Ingress = []api.LoadBalancerIngress{{ IP: svcLBIP, }} svc.Spec.LoadBalancerSourceRanges = []string{ svcLBSource, } }), ) makeEndpointsMap(fp, makeTestEndpoints(svcPortName.Namespace, svcPortName.Name, func(ept *api.Endpoints) { ept.Subsets = []api.EndpointSubset{{ Addresses: []api.EndpointAddress{{ IP:       epIP, NodeName: strPtr(testHostname), }}, Ports: []api.EndpointPort{{ Name: svcPortName.Port, Port: int32(svcPort), }}, }} }), ) fp.syncProxyRules() // Check ipvs service and destinations services, err := ipvs.GetVirtualServers() if err != nil { t.Errorf("Failed to get ipvs services, err: %v", err) } found := false for _, svc := range services { fmt.Printf("address: %s:%d, %s", svc.Address.String(), svc.Port, svc.Protocol) if svc.Address.Equal(net.ParseIP(svcLBIP)) && svc.Port == uint16(svcPort) && svc.Protocol == string(api.ProtocolTCP) { destinations, _ := ipvs.GetRealServers(svc) if len(destinations) != 1 { t.Errorf("Unexpected %d destinations, expect 0 destinations", len(destinations)) } for _, ep := range destinations { if ep.Address.String() == epIP && ep.Port == uint16(svcPort) { found = true } } } } if !found { t.Errorf("Did not got expected loadbalance service") } // Check ipset entry expectIPSet := map[string]*utilipset.Entry{ KubeLoadBalancerSet: { IP:       svcLBIP, Port:     svcPort, Protocol: strings.ToLower(string(api.ProtocolTCP)), SetType:  utilipset.HashIPPort, }, KubeLoadBalancerMasqSet: { IP:       svcLBIP, Port:     svcPort, Protocol: strings.ToLower(string(api.ProtocolTCP)), SetType:  utilipset.HashIPPort, }, KubeLoadBalancerSourceCIDRSet: { IP:       svcLBIP, Port:     svcPort, Protocol: strings.ToLower(string(api.ProtocolTCP)), Net:      svcLBSource, SetType:  utilipset.HashIPPortNet, }, } for set, entry := range expectIPSet { ents, err := ipset.ListEntries(set) if err != nil || len(ents) != 1 { t.Errorf("Check ipset entries failed for ipset: %q", set) continue } if ents[0] != entry.String() { t.Errorf("Check ipset entries failed for ipset: %q", set) } } // Check iptables chain and rules kubeSvcRules := ipt.GetRules(string(kubeServicesChain)) kubeFWRules := ipt.GetRules(string(KubeFireWallChain)) if !hasJump(kubeSvcRules, string(KubeMarkMasqChain), KubeLoadBalancerMasqSet) { t.Errorf("Didn't find jump from chain %v match set %v to MASQUERADE", kubeServicesChain, KubeLoadBalancerMasqSet) } if !hasJump(kubeSvcRules, string(KubeFireWallChain), KubeLoadBalancerSet) { t.Errorf("Didn't find jump from chain %v match set %v to %v", kubeServicesChain, KubeLoadBalancerSet, KubeFireWallChain) } if !hasJump(kubeFWRules, "ACCEPT", KubeLoadBalancerSourceCIDRSet) { t.Errorf("Didn't find jump from chain %v match set %v to ACCEPT", kubeServicesChain, KubeLoadBalancerSourceCIDRSet) } } </ins> func TestOnlyLocalLoadBalancing(t *testing.T) { ipt := iptablestest.NewFake()

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Service loadbalancer source range don't work in ipvs proxy mode. Which result in gci-gce-ipvs CI failed. http://k8s- How to reproduce it (as minimally and precisely as possible): In gce environment. a service with backend. service type to service util service external IP is specified. the service from pod inside and pod out of the . Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): Install tools: cluster/kube- Others:
/kind bug /sig network /label ipvs
/area ipvs
The test seems to be flaky now. Can you take a look please?
I don't know why, but I think this flaky have the similiar reason with other normal service-test cases, like Those flaky are pretty annoying, problem is, I have run those cases in my own environment many times, never fail... : ( But still, I'll do my best to find the reason.


Question: // Sleep 1s to verify no detach are triggered after second pod is added in the future. time.Sleep(1000 * time.Millisecond) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> // verifyVolumeNoStatusUpdateNeeded(t, logger, generatedVolumeName, nodeName1, asw) // Add a third pod which tries to attach the volume to a different node.

After KCM restart, any Pod that reuses a volume that failed to detach gets stuck ContainerCreating with message: User "system:node:ip-10-0-243-" cannot get resource "volumeattachments" in API group "" at the cluster scope: no relationship found between node 'ip-10-0-243-' and this object See steps to reproduce below, it's quite a complex condition. It happens when a pod is restarted, a CSI driver "busy" (i.e. times out) + KCM restarted at a wrong time. Pods start just fine. a Pod that uses an attachable CSI volume. the Pod. Make sure it unmounts successfully, but any fails with a transient error, such as timeout. The external attacher continues to detach the volume at this point. kube-controller-manager (KCM) and start a new one. Wait for any leader election etc - it's important for the new KCM to populate A/D controller with no pod using the detaching volume. Let the A/D controller to observe detach error. a new Pod on the same node that uses the same volume. the CSI driver to detach the volume (i.e. stop injecting any transient errors). At this point: The external attacher succeeds detaching the volume, VolumeAttachment is deleted. The volume is marked as attached in node status (!!!) despite no VolumeAttachment exists. A/D controller does not even try to create a new VolumeAttachment, the volume is marked as attached in its Actual State of World (ASW). Kubelet tries to mount the volume, but WaitForAttach fails with: , basically forever. I created . The script expects that the cluster is created by , because it restarts KCM in the middle. It uses my hacked hostpath CSI driver to inject detach errors. Sources are at / the last commit(s) there. There are two issues that together lead to this situation: fails, , i.e. puts it back to . With the volume in , kubelet may try to mount it, while the external attacher tries to detach it. It is not as bad as it sounds, because kubelet can't read VolumeAttachment with the deletion timestamp and it will not call / of un-attached volume.


Question: // Sleep 1s to verify no detach are triggered after second pod is added in the future. time.Sleep(1000 * time.Millisecond) verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) <ins> verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </ins> // verifyVolumeNoStatusUpdateNeeded(t, logger, generatedVolumeName, nodeName1, asw) // Add a third pod which tries to attach the volume to a different node.

KCM starts, all VolumeAttachments that don't have a Pod (that's good), but also (that's probably bad). When the newly started KCM re-tries to detach the volume and gets an error again, it marks the volume as fully attached (same as the previous point) and adds it to . Now the volume is both mounted + attached in ASW and A/D controller actually stops here and waits forever for kubelet to unmount the volume. <detailsToday's k/k master, </details<detailsNone </details<detailsN/A, it's KCM issue. </details<details, but seen in a downstream Kubernetes distro too. </details<details</details<details</details
/sig storage
/assign
/triage accepted
I have the same problem: Seems restart of Kubelet solves this problem. Kubernetes v1.25.13


Question: return nil } _, err = p.client.Namespaces().Create(namespace) <del> if err != nil { </del> <ins> if err != nil && !errors.IsAlreadyExists(err) { </ins> return err } return nil

While running one of my soak tests I noticed the error message: I don't see why I should get this error message. Why should it matter if the namespace already exists? It was created as part of the service creation call just beforehand. And the second attempt to create this pod in exactly the same namespace works. Is this a bug?
Running a second time did not produce this error.
I agree it should not matter. The error is produced there, and I suspect there were multiple concurrent calls being made that try to create a resource in a namespace that had not yet existed. We need to update the code in question to swallow the error if the namespace already exists.
FYI - longer term, I would like to eliminate the use of autoprovision of namespace and require you to first create a namespace manually. what is there now is more to preserve existing behavior prior to namespace as object was introduced.


Question: OOMScoreAdj:            config.OOMScoreAdj, ConfigSyncPeriod:       config.ConfigSyncPeriod.Duration, HealthzServer:          healthzServer, <ins> UseEndpointSlices:      utilfeature.DefaultFeatureGate.Enabled(features.EndpointSlice), </ins> }, nil }

After this () has been merged and EnpointSlice feature is now enabled by default, we noticed Kube-proxy will not work on Windows. Only message I can see in the logs is: /sig windows
/cc /cc /cc
How about separating the above check into two if statements so that the log can pinpoint which field is false ?
It seems long term, EndpointSlice support should be so that is kept in sync with other implementations.
cc
Thanks for reporting this! Windows proxiers were still intended to be reading from Endpoints, but apparently that's not working as intended. EndpointSlices are moving back to disabled by default for 1.17, so that should fix the failing tests for now. I'll work on an actual fix for this as well, but it may not make it in until after KubeCon. /assign /sig network /priority important-soon
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
/assign cc
GitHub didn't allow me to assign the following users: jocelynberrendonner. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
: Please let me know if you need any help on the fix! I'll be more than happy to review/help test the fix on Windows.


Question: allErrs = append(allErrs, field.Invalid(fldPath.Child("deletionTimestamp"), newMeta.DeletionTimestamp, "field is immutable; may only be changed via deletion")) } <ins> // Finalizers cannot be added if the object is already being deleted. if oldMeta.DeletionTimestamp != nil { allErrs = append(allErrs, ValidateNoNewFinalizers(newMeta.Finalizers, oldMeta.Finalizers, fldPath.Child("finalizers"))...) } </ins> // Reject updates that don't specify a resource version if len(newMeta.ResourceVersion) == 0 { allErrs = append(allErrs, field.Invalid(fldPath.Child("resourceVersion"), newMeta.ResourceVersion, "must be specified for an update"))

says "If the deletionTimestamp of the object is non-nil, entries in this list can only be removed.". can you elucidate if I am assuming this does something different than reality
This is garbage collection finalizer field, not the namespace finalizer field that that new field was inspired from On Thursday, September 8, 2016, Tim Hockin wrote:
I don't parse that - is this not a general-purpose "refcount" thing? It seems to work that way, except not as documented
This is a general purpose field and it should work like Tim thinks it should work. can you take a look at this when you don't have anything pressing for 1.4?
The namespace finalizer should switch to using this field if it is currently doing something else.
I'm adding validation rule to prevent adding finalizers if deletionTimestamps is set. I'm still investigating why it can't be removed. This bug is strange.
Actually I guess it's a bug. It uses patch, and patch doesn't work on list of simple type, is a list of string.
Yep, it's a bug. I used to remove one finalizer and it worked as expected. I think this is a known bug of , we don't have a way to specify the mergekey of a list of simple type.
Is there a bug open for this? Frankly, if is going to fail for things like this, we should not be offering it as a feature, or it should use PUT rather than PATCH , Chao Xu wrote:
Created cannot use PUT, because it's doing a three way merge (current state, last configuration, current configuration).
These are intended for automated use, not human use, so kubectl edit not working is just unfortunate, not a blocker. Although it's really unfortunate that PATCH is so broken. , Chao Xu wrote:
Why does need to be 3-way? I don't think that is the semantic a user would expect from an edit operation AT ALL. It certainly surprised ME. WRT automated vs human - if PATCH doesn't work (worse - says it succeeded but didn't) for some class of fields, then PATCH doesn't work, and we should not support it. Full stop. It's going to end up causing someone an outage. These are not necessarily related to this bug, but I'd call them pretty serious flaws.


Question: allErrs = append(allErrs, field.Invalid(fldPath.Child("deletionTimestamp"), newMeta.DeletionTimestamp, "field is immutable; may only be changed via deletion")) } <ins> // Finalizers cannot be added if the object is already being deleted. if oldMeta.DeletionTimestamp != nil { allErrs = append(allErrs, ValidateNoNewFinalizers(newMeta.Finalizers, oldMeta.Finalizers, fldPath.Child("finalizers"))...) } </ins> // Reject updates that don't specify a resource version if len(newMeta.ResourceVersion) == 0 { allErrs = append(allErrs, field.Invalid(fldPath.Child("resourceVersion"), newMeta.ResourceVersion, "must be specified for an update"))

, Daniel Smith wrote:
Discussed in real life. PATCH works, it's just that there are certain operations you cannot express using the strategicmerge patch syntax. I was wrong. is building . So perhaps it uses PATCH over PUT due to the efficiency.
FWIW, I printed out the patch generated by when I removed one finalizer. The patch is empty, i.e., "{}".
, Chao Xu wrote: Well, no, if there are operations that can not be represented, it really DOESN'T work. I still think it should not use PUT, but I bet Brian can prove me wrong.


Question: "time" "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2017-12-01/compute" <ins> "github.com/golang/glog" </ins> "k8s.io/apimachinery/pkg/types" kwait "k8s.io/apimachinery/pkg/util/wait" <ins> "k8s.io/kubernetes/pkg/cloudprovider" </ins> ) const (

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: VirtualMachineScaleSetVM and VirtualMachine are different data structures (because of different API versions), so the disk attach operation seems duplicate between vm and vmss. That means although disk operations are in same logic, some duplication are still required now. They should use same api version in the future and should be merged into same code base, e.g. move sharing logic to Refer What you expected to happen: vmss and vm disk operations should share same code logic. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I have checked with currently we can’t merge, e.g. following two structs are different since they using different sdk-for-azure versions. While in the end when sdk-for-azure version is upgraded, above two versions should be same and at that time, we should merge functions like ‘AttachDisk’, ‘DetachDiskByName’, 'GetDiskLun', GetNextDiskLun, DisksAreAttached


Question: "k8s.io/kubernetes/pkg/client/record" client "k8s.io/kubernetes/pkg/client/unversioned" "k8s.io/kubernetes/pkg/controller/framework" <ins> "k8s.io/kubernetes/pkg/fields" </ins> "k8s.io/kubernetes/pkg/labels" "k8s.io/kubernetes/pkg/runtime" )

The test is in flaky suite and no one seems to care. We need an owner for this test. cc Last failure log:
do you think you can take a look at this?
This is happening because of a bad assumption. The assumption is: if the reflector's store is consistent with etcd, the store given back to clients of the framework.Controller is too. I'd really like a general fix applicable to all controllers for this problem, but fixing it just for the RCs is pretty easy. In more detail: This function reports when the reflector has listed from apiserver. But the store given to the reflector is the fifo of the framework.Controller. The store given back to the caller is still not consistent, and will not be till the framework.Controller puts all the objects from the fifo into the client state store (). The rc specific fix: Before starting the rc manager, seed the expectations store with an expectation like {rc:foo, adds: foo.Status.Replicas, deletes:0}. This will wait for status.Replicas number of ADD events till a timeout. The general fix: Somehow expose this through framework.Controller.HasSynced (clients don't care about the fifo anyway, shouldn't it just be an implementation detail?) ideas? cc'ing people who have written controllers that might be interested I don't think this (and generally anything flaky for 1 release) is a p0, I can fix it if you don't mind dealing with the flakes till I've cleared things off my plate. I'd be more than happy to review a fix.
Well, it's a P0 because test is flaky. E.g last run:
- did you have chance to look into it?
The "DaemonRestart" tests are still labeled at . I will send a PR to promote them.
are you still planning to send aPR to promote the DaemonRestart tests out of flaky? I think this is the right thing to do.
Ack; sorry, I lost this in the muck.


Question: // Synced returns true if the view is synced (for the first time) func (f *federatedInformerImpl) ClustersSynced() bool { <del> f.Lock() defer f.Unlock() </del> return f.clusterInformer.controller.HasSynced() }

https://k8s- Failed: TestNamespaceController {namespace}


Question: framework.Failf("failed dialing endpoint, %v", err) } <ins> ginkgo.By("pod-Service(hostNetwork): udp") ginkgo.By(fmt.Sprintf("dialing(udp) %v --> %v:%v (config.clusterIP)", config.TestContainerPod.Name, config.ClusterIP, e2enetwork.ClusterUDPPort)) err = config.DialFromTestContainer("udp", config.ClusterIP, e2enetwork.ClusterUDPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } ginkgo.By(fmt.Sprintf("dialing(udp) %v --> %v:%v (nodeIP)", config.TestContainerPod.Name, config.NodeIP, config.NodeUDPPort)) err = config.DialFromTestContainer("udp", config.NodeIP, config.NodeUDPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> ginkgo.By("node-Service(hostNetwork): http") ginkgo.By(fmt.Sprintf("dialing(http) %v (node) --> %v:%v (config.clusterIP)", config.NodeIP, config.ClusterIP, e2enetwork.ClusterHTTPPort))

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-kubernetes-e2e-gce-cos-k8sbeta-default Networking Granular Checks: Services should function for service endpoints using hostNetwork Triage looks like it's mostly flaking on this job only: Examples /sig network <!-- Please see the deflaking doc () for more guidance! --
/milestone v1.21 (marking in the current milestone until we get confirmation whether it is relevant to the release or not)
Slack Thread:
/assign
This is fixed by , let's wait some runs to confirm before closing
/milestone clear (since this has been deemed to be non release-blocking)
/triage accepted
/close
Closing this issue. <detailsIn response to : This is fixed by , let's wait some runs to confirm before closing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: framework.ExpectNoError(err) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. ginkgo.It("should be able to change the type and ports of a service [Slow] [DisabledForLargeClusters]", func() { </del> <ins> ginkgo.It("should be able to change the type and ports of a service [Slow]", func() { </ins> // requires cloud load-balancer support e2eskipper.SkipUnlessProviderIs("gce", "gke", "aws")

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is a followup from filed separately now that we're not tracking this as part of 1.9 release. The tests have been disabled with [DisabledForLargeClusters] tag for now, and this fact will be mentioned in the scalability validation report for v1.9. This issue tracks fixing the test so the tag can be removed and the test ran again in the large cluster. /assign /kind e2e-test-failure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Is this e2e test still broken?
Sorry for the confusion here. Currently this e2e test is disabled on extra large cluster due to some scalability issues with the load balancer related components. At the moment the issues haven't been completely resolved yet.
ok, adding priority so that I know it's not a 1.13 blocker. /priority important-longterm
cc - this is why the relevant tests for are not part of the e2e suite.
This issue is still valid --- some action items on top of my head: Figure out a plan to make service controller properly handle operations in parallel. Check on the status for cloud providers (especially GCP, which the test ran against) on handling large number of nodes as LB backends. /remove-triage unresolved
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: framework.ExpectNoError(err) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. ginkgo.It("should be able to change the type and ports of a service [Slow] [DisabledForLargeClusters]", func() { </del> <ins> ginkgo.It("should be able to change the type and ports of a service [Slow]", func() { </ins> // requires cloud load-balancer support e2eskipper.SkipUnlessProviderIs("gce", "gke", "aws")

/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /lifecycle frozen
/reopen Reopening as is being reverted in
Reopened this issue. <detailsIn response to : /reopen Reopening as is being reverted in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
these tests are flaking in xref
cc since these were enabled by
Yeah - though these failures doesn't seem to be scale related, they seem to be test related to me. This basically means nodes without public ips: It's pretty much the same thing as (just for a different test). should be addressing these kinds of errors. [Not a release blocker for sure]
BTW - has been opened directly for this one.
... so since is closed, Can we close this and re-enable for large clusters ? (i.e. remove the Tag in that disables this test) ? I'm not up to speed on the history but 2 related notes I came across digging into related test today... 's a couple of TODOs in the code referencing this specific issue, but your above direct issue is now closed, so we can close this as well right?... And if so, we can remove the TODO on the test... -enabling the test ... not sure wether the timing is good right now (given that is open right now , which sparked my stumbling upon this issue :))
We tried to enable all of them and that caused us a bunch of issues and those had to be reverted: A couple tests were enabled later in: and those seem fine. The remaining (mostly session-affinity ones) had some issue, which so far we didn't have time to debug and I haven't heard a good explanation why those were failing. It's possible that they were improved with , but I doubt those would really cause the failures we've seen in the past. So until this is debugged (or at least understood) I'm reluctant to enabling them.


Question: w := newLogWriter(stdoutBuf, stderrBuf, &LogOptions{timestamp: test.timestamp, bytes: int64(test.bytes)}) for i := 0; i < test.stdoutLines; i++ { msg.stream = runtimeapi.Stdout <del> if err := w.write(msg); err != nil { </del> <ins> if err := w.write(msg, true); err != nil { </ins> assert.EqualError(t, err, errMaximumWrite.Error()) } } for i := 0; i < test.stderrLines; i++ { msg.stream = runtimeapi.Stderr <del> if err := w.write(msg); err != nil { </del> <ins> if err := w.write(msg, true); err != nil { </ins> assert.EqualError(t, err, errMaximumWrite.Error()) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Docker has a hardcoded 16K limit on log line length, which splits lines longer than that into multiple messages in the log file. in , however, doesn't account for this when is set to and prints the timestamp in front of every entry in the log file instead of every logical line. This leads to random timestamps appearing in the middle of log lines, potentially breaking the applications that parse these logs. What you expected to happen: inserts timestamps once per logical line, so original log data is preserved. Alternatively, allow API to request the logs in a sane format like JSON, which would allow merging the lines back and combining them with the timestamps more intelligently on the application side. How to reproduce it (as minimally and precisely as possible): Create : apiVersion: v1 kind: Pod metadata: name: log-lines spec: containers: name: main image: busybox command: ['sh', '-c', 'while sleep 1; do echo -n ; for i in $(seq 1 ); do echo -n A; done; echo ; done'] Start the pod with Fetch the logs with , or using the API () Observe random timestamps inserted in the middle of the log lines. Environment: Kubernetes version (use ):
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I will work on it. /sig cli /remove-sig node
The logs are being corrupted by the API endpoint in kubelet, so shouldn't it be in scope of SIG node?
The log is not corrupted by k8s. The same log could be got using "docker container logs -t CONTAINERID" I think you should post your requirement to Docker.


Question: w := newLogWriter(stdoutBuf, stderrBuf, &LogOptions{timestamp: test.timestamp, bytes: int64(test.bytes)}) for i := 0; i < test.stdoutLines; i++ { msg.stream = runtimeapi.Stdout <del> if err := w.write(msg); err != nil { </del> <ins> if err := w.write(msg, true); err != nil { </ins> assert.EqualError(t, err, errMaximumWrite.Error()) } } for i := 0; i < test.stderrLines; i++ { msg.stream = runtimeapi.Stderr <del> if err := w.write(msg); err != nil { </del> <ins> if err := w.write(msg, true); err != nil { </ins> assert.EqualError(t, err, errMaximumWrite.Error()) } }

Please refer following code Line 50 ~ 52 if config.Timestamps { logLine = append([]byte(msg.Timestamp.Format(jsonmessage.RFC3339NanoFixed)+" "), logLine...) }
kubelet doesn't rely on Docker when reading logs, it reads the files itself (see the code linked in the issue). I don't think Docker would change the forced line splitting (see for example), so this should be fixed in Kubernetes.
Your are right and kubelet read log file from host such as /var/log/pods/pods/NAMESPACEPOD-NAMExxx/....(JSON format) I will post a PR for it. /remove-sig cli /sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .


Question: w := newLogWriter(stdoutBuf, stderrBuf, &LogOptions{timestamp: test.timestamp, bytes: int64(test.bytes)}) for i := 0; i < test.stdoutLines; i++ { msg.stream = runtimeapi.Stdout <del> if err := w.write(msg); err != nil { </del> <ins> if err := w.write(msg, true); err != nil { </ins> assert.EqualError(t, err, errMaximumWrite.Error()) } } for i := 0; i < test.stderrLines; i++ { msg.stream = runtimeapi.Stderr <del> if err := w.write(msg); err != nil { </del> <ins> if err := w.write(msg, true); err != nil { </ins> assert.EqualError(t, err, errMaximumWrite.Error()) } }

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. are you or anybody on this thread are still interested to work on this? Please re-open if so. Meanwhile closing this ahead of bot doing it because the inactivity. /close
Closing this issue. <detailsIn response to : I saw the attempt fixing this but it was scary (echo this comment: ). I suggested a way it may be improved. Also the proposed fix didn't introduce any e2e tests to validate that runtime behavior is not changed and runtimes keep the no-newline-on-splitted-log logic. Instructions for interacting with me using PR comments are available .


Question: w := newLogWriter(stdoutBuf, stderrBuf, &LogOptions{timestamp: test.timestamp, bytes: int64(test.bytes)}) for i := 0; i < test.stdoutLines; i++ { msg.stream = runtimeapi.Stdout <del> if err := w.write(msg); err != nil { </del> <ins> if err := w.write(msg, true); err != nil { </ins> assert.EqualError(t, err, errMaximumWrite.Error()) } } for i := 0; i < test.stderrLines; i++ { msg.stream = runtimeapi.Stderr <del> if err := w.write(msg); err != nil { </del> <ins> if err := w.write(msg, true); err != nil { </ins> assert.EqualError(t, err, errMaximumWrite.Error()) } }

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen will fix it.
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: w := newLogWriter(stdoutBuf, stderrBuf, &LogOptions{timestamp: test.timestamp, bytes: int64(test.bytes)}) for i := 0; i < test.stdoutLines; i++ { msg.stream = runtimeapi.Stdout <del> if err := w.write(msg); err != nil { </del> <ins> if err := w.write(msg, true); err != nil { </ins> assert.EqualError(t, err, errMaximumWrite.Error()) } } for i := 0; i < test.stderrLines; i++ { msg.stream = runtimeapi.Stderr <del> if err := w.write(msg); err != nil { </del> <ins> if err := w.write(msg, true); err != nil { </ins> assert.EqualError(t, err, errMaximumWrite.Error()) } }

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .


Question: w := newLogWriter(stdoutBuf, stderrBuf, &LogOptions{timestamp: test.timestamp, bytes: int64(test.bytes)}) for i := 0; i < test.stdoutLines; i++ { msg.stream = runtimeapi.Stdout <del> if err := w.write(msg); err != nil { </del> <ins> if err := w.write(msg, true); err != nil { </ins> assert.EqualError(t, err, errMaximumWrite.Error()) } } for i := 0; i < test.stderrLines; i++ { msg.stream = runtimeapi.Stderr <del> if err := w.write(msg); err != nil { </del> <ins> if err := w.write(msg, true); err != nil { </ins> assert.EqualError(t, err, errMaximumWrite.Error()) } }

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I received a bug report about this issue. The root cause is that the buffer reads up to a new line. If the line is greater than 4096 bytes and '--timestamps' is turrned on the kubelet will write the timestamp and the partial log line. We will need to refactor the ReadLogs function to allow for a partial line read.


Golang 1.13 uses TLS 1.13 by default (opt-out), which introduces RSA-PSS. RSA-PSS does not work with keys that are too small. The issue was uncovered during Fedora mass rebuild which uses Golang 1.13: See /sig api-machinery
What is the failure mode if a smaller key is used?
/sig auth /sig scalability
/sub


Question: "../examples/walkthrough": { "pod1": &api.Pod{}, "pod2": &api.Pod{}, <ins> "pod-with-http-healthcheck": &api.Pod{}, "service":                   &api.Service{}, "replication-controller":    &api.ReplicationController{}, </ins> }, }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: cAdvisor has a option to only collect docker related cgroups. Kubelet stats api now supports allocatable cgroup, /kubepods, for the runtime cgroup and the kubelet cgroup. But with settings, Kubelet stats api can not query pod related cgroups metrics. FYI: What you expected to happen: should be compatible with kubelet stats api.
/cc
/sig node /sig instrumentation
add to cadvisor in A comma-separated list of cgroup path prefix that needs to be collected even docker-only is specified bump cadvisor dep for Kubernetes setting to
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: "../examples/walkthrough": { "pod1": &api.Pod{}, "pod2": &api.Pod{}, <ins> "pod-with-http-healthcheck": &api.Pod{}, "service":                   &api.Service{}, "replication-controller":    &api.ReplicationController{}, </ins> }, }

/close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: var defaultKubernetesFeatureGates = map[featuregate.Feature]featuregate.FeatureSpec{ CrossNamespaceVolumeDataSource: {Default: false, PreRelease: featuregate.Alpha}, <ins> AllowServiceLBStatusOnNonLB: {Default: false, PreRelease: featuregate.Deprecated}, // remove after 1.29 </ins> AnyVolumeDataSource: {Default: true, PreRelease: featuregate.Beta}, // on by default in 1.24 APISelfSubjectReview: {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // GA in 1.28; remove in 1.30

xref I think this may be one of the few cases where we can actually change the behavior in a "breaking" way. User creates service with , is empty Controller or user updates the status subresource, sets (not wiped or validated, should be) User updates in any way, is wiped It's clear that the is not meant to be set when is not "LoadBalancer". The question is: 1) Fail validation? 2) Silently discard the value? 3) Warn and then discard the value? 4) Just warn and leave it. Because we allowed it in the past, but it is so volatile, I lean towards 3. and - the people most familair with this type. Edit: PrepareForUpdate() runs before warnings. So: (1) is easy (2) is easy (4) is slightly less easy than (2) (3) is slightly less easy than (4)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
OMG change it to clusterIP as expected Status is wiped update status as ClusterIP ... and if I edit the service back to loadbalancer it keeps the status??
/kind bug
/retitle Service can be set when !=
We talked about this one at the SIG Network meeting today. suggested we use the feature gate for deprecation option to phase the current (problematic) functionality out.
/triage accept
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: v1 "k8s.io/api/core/v1" rbacv1 "k8s.io/api/rbac/v1" storagev1 "k8s.io/api/storage/v1" <del> storagev1beta1 "k8s.io/api/storage/v1beta1" </del> apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime/schema"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-e2e-gci-gce-slow [sig-storage] Dynamic Provisioning DynamicProvisioner [Slow] should provision storage with different parameters 5/13 https://k8s- v1beta1.StorageClass got removed in We should remove the v1beta1 test.
/assign can you help take a look?
/triage accepted
/sig storage
I run the test with the latest from master without removing the beta test and it failed with a different error before getting to the beta test part: Do you have any idea what would've happened in my setup? I redeployed k/k using master HEAD
For this testcase: I saw that the provisioned PD has a size of 1.5Gi according to kubernetes: And in gcloud It's really interesting that the test was expecting 2Gi to be created, is this something that was changed recently?
The logs you posted are showing pd csi as the provisioner. Were you testing with the intree driver or csi driver? The CSI driver should to the nearest Gb. I believe intree does the .
I think I did intree because I didn't send as an argument in my command: It's interesting that the PVC has this annotation too:
mmm something interesting about my setup is that I created the cluster with CSIMigration turned on Do you think that might be forcing the usage of ?
Yes likely. Try with CSIMigration off and see if that helps. If so, then this potentially seems like a rounding bug with the csi driver
Thanks I tried with and it did indeed round up to 2Gi.


Question: {% set params = "--master=127.0.0.1:8080" + " " + cluster_name + " " + cluster_cidr + " " + allocate_node_cidrs + " " + service_cluster_ip_range + " " + terminated_pod_gc + " " + enable_garbage_collector + " " + cloud_provider  + " " + cloud_config + " " + service_account_key + " " + log_level + " " + root_ca_file -%} {% set params = params + " " + feature_gates -%} <ins> {% if pillar.get('enable_hostpath_provisioner', '').lower() == 'true' -%} {% set params = params + " --enable-hostpath-provisioner" %} {% endif -%} </ins> # test_args has to be kept at the end, so they'll overwrite any prior configuration {% if pillar['controller_manager_test_args'] is defined -%}

feature request Enables easier testing for pv/pvc, and all derivatives like pet sets. The only alternative is to use manually created pv of local type. I have time to work on it myself and created this feature to collect feedback/ideas on what provider should be enabled by default.
Since this was merged should we close this issue?


Question: {Name: "Name", Type: "string", Format: "name", Description: metav1.ObjectMeta{}.SwaggerDoc()["name"]}, {Name: "Status", Type: "string", Description: "Status of the cluster"}, {Name: "Age", Type: "string", Description: metav1.ObjectMeta{}.SwaggerDoc()["creationTimestamp"]}, <ins> {Name: "Labels", Type: "string", Description: "The labels of the cluster"}, </ins> } h.TableHandler(clusterColumnDefinitions, printCluster) h.TableHandler(clusterColumnDefinitions, printClusterList)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After labeling clusters in a Federation, when you type the field is empty. What you expected to happen: The fiels should contain information about the labels in the cluster How to reproduce it (as minimally and precisely as possible): a federation control pane in any Cloud Provider and any DNS provider your cluster: info aboyut the cluster with the flag: Note: if you describe it, you see the labels being : Environment: Kubernetes version (use ): 1.8.0 Cloud provider or hardware configuration**: anyone OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux Install tools: curl Others:
/sig multicluster
I'm working on it.


Question: "redis-sentinel-service":    &api.Service{}, }, "../examples/resourcequota": { <del> "resource-quota": &api.ResourceQuota{}, </del> <ins> "namespace": &api.Namespace{}, "limits":    &api.LimitRange{}, "quota":     &api.ResourceQuota{}, </ins> }, "../examples/rethinkdb": { "admin-pod":      &api.Pod{},

Resource quota example is missing readme. Was it once there and was accidentally deleted?
I wrote something up in Origin repo I need to repurpose for Kubernetes. Sent from my iPhone


Question: kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 1).resources.limits.cpu}}:{{end}}" "300m:" kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 1).resources.requests.cpu}}:{{end}}" "300m:" # Set limits on a local file without talking to the server <del> kubectl set resources -f hack/testdata/deployment-multicontainer.yaml -c=perl --limits=cpu=500m --requests=cpu=500m --dry-run -o yaml "${kube_flags[@]}" kubectl set resources deployment nginx-deployment -c=perl --limits=cpu=500m --requests=cpu=500m --dry-run -o yaml "${kube_flags[@]}" ! kubectl set resources deployment nginx-deployment -c=perl --limits=cpu=500m --requests=cpu=500m --local -o yaml "${kube_flags[@]}" </del> <ins> kubectl set resources deployment -f hack/testdata/deployment-multicontainer.yaml -c=perl --limits=cpu=300m --requests=cpu=300m --dry-run -o yaml "${kube_flags[@]}" </ins> kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 0).resources.limits.cpu}}:{{end}}" "200m:" kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 1).resources.limits.cpu}}:{{end}}" "300m:" kube::test::get_object_assert deployment "{{range.items}}{{(index .spec.template.spec.containers 1).resources.requests.cpu}}:{{end}}" "300m:"

from https://k8s-
Really suspect this is
This is putting us into test-go post-commit run is broken
cc:
marked this p0 - if a fix isn't easy let's try to identify what introduced this and revert.
This flake on: 2016-10-30 07:19 PDT First instance: : It is flaking frequently, blocking PRs from getting merged, and slowing the submit queue ( at 75%). As on-call, I'm going to try and identify the cluprit and revert. Can use all help in doing so.
can you investigate whether your additional tests could cause this (by breaking the deployment of those pods, for instance) , Saad Ali wrote:
PR looks most suspect to me. It mucked with , and the errors we are seeing in each of the failures is related to : That PR was merged at Oct 30, 2016, 6:18 AM PDT and the first flake in was at Oct 30, 2016, 7:19 AM PDT. Thoughts? CC
I'm going to initiate a revert of to see if that fixes the issue
IMO we can just change the flaky test to not get pods but some other resources, such as deployments. We can't really expect to clean up pods immediately.
I'm going to proceed with revert (). Please feel free to send a follow up PR to handle this more gracefully.
Going to leave this opened until we have a few hours of runs with the revert. Want to verify, with some degree of certainty, that the revert took care of this flake.
We had 15 green runs of in a row. Considering this flake fixed.


Question: // define volume limit to be 2 for this test var err error <del> init(testParameters{nodeSelectorKey: "node-attach-limit-csi", attachLimit: 2}) </del> <ins> nodeSelectorKey := fmt.Sprintf("attach-limit-csi-%s", f.Namespace.Name) init(testParameters{nodeSelectorKey: nodeSelectorKey, attachLimit: 2}) </ins> defer cleanup() nodeName := m.config.ClientNodeName attachKey := v1.ResourceName(volumeutil.GetCSIAttachLimitKey(m.provisioner))

Looks like CSI volume limit test is flaking. /sig storage /assign
This flake isn't happening in normal clusters, appears to be alpha only issue.


Question: // buildELBSecurityGroupList returns list of SecurityGroups which should be // attached to ELB created by a service. List always consist of at least <del> // 1 member which is an SG created for this service or a SG from the Global config. Extra groups can be // specified via annotation func (c *Cloud) buildELBSecurityGroupList(serviceName types.NamespacedName, loadBalancerName, annotation string) ([]string, error) { </del> <ins> // 1 member which is an SG created for this service or a SG from the Global config. // Extra groups can be specified via annotation, as can extra tags for any // new groups. func (c *Cloud) buildELBSecurityGroupList(serviceName types.NamespacedName, loadBalancerName string, annotations map[string]string) ([]string, error) { </ins> var err error var securityGroupID string

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig aws What happened: In k8s 1.7, when creating a service of type LoadBalancer, K8S creates the ELB and a Security Group for it. For these resources, it adds a tag, constructed dynamically, containing the cluster id: "${cluster_id}" with the value "true" What you expected to happen: We'd like to have also a statically named tag , as it was in 1.5 ("KubernetesCluster") with the value of the cluster id. How to reproduce it (as minimally and precisely as possible): Running on AWS cloud provider, create a service of type LoadBalancer. Anything else we need to know?: The reason for what we expect is that we have tight security constraints that require us to fine tune the IAM policies for the Instance profiles that the controllers use. With a statically named tag we were able, in 1.5, to restrict the permissions in the IAM policy to act only on the k8s managed resources (ELBs, security groups). Here is the relevant part of the IAM policy: With a dynamically named tag we are not able to do this any more as AWS IAM doesn't support something like: We are sharing the Instance Profiles among different k8s clusters, hence the "" in the value of the tag. Environment: Kubernetes version (use ):Cloud provider or hardware configuration*: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: kube-aws Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : <!-- This form is for bug reports and feature requests ONLY! -- /sig aws In k8s 1.7, when creating a service of type LoadBalancer, K8S creates the ELB and a Security Group for it. For these resources, it adds a tag, constructed dynamically, containing the cluster id: "${cluster_id}" with the value "true" We'd like to have also a statically named tag , as it was in 1.5 ("KubernetesCluster") with the value of the cluster id. Running on AWS cloud provider, create a service of type LoadBalancer.


Question: // buildELBSecurityGroupList returns list of SecurityGroups which should be // attached to ELB created by a service. List always consist of at least <del> // 1 member which is an SG created for this service or a SG from the Global config. Extra groups can be // specified via annotation func (c *Cloud) buildELBSecurityGroupList(serviceName types.NamespacedName, loadBalancerName, annotation string) ([]string, error) { </del> <ins> // 1 member which is an SG created for this service or a SG from the Global config. // Extra groups can be specified via annotation, as can extra tags for any // new groups. func (c *Cloud) buildELBSecurityGroupList(serviceName types.NamespacedName, loadBalancerName string, annotations map[string]string) ([]string, error) { </ins> var err error var securityGroupID string

The reason for what we expect is that we have tight security constraints that require us to fine tune the IAM policies for the Instance profiles that the controllers use. With a statically named tag we were able, in 1.5, to restrict the permissions in the IAM policy to act only on the k8s managed resources (ELBs, security groups). Here is the relevant part of the IAM policy: Environment: Kubernetes version (use ):OS (e.g. from /etc/os-release):Others: </details
I believe they are moving away from the tag since there are many resources that can be shared by multiple clusters. could you use an annotation?
We removed the condition in roles in the end. Thank you for your suggestion.
That annotation affects the ELB but not any security groups that are created. can you please reopen this? You reported a real issue, and even if it no longer affects you, it needs to be tracked and fixed.
Indeed, this might be a feature that someone else would need.
In fact, this is a feature I need!
Food for thought: should we propagate the tags on the ELB to the SG, or should we introduce an annotation for the SG specifically?


Question: var _ = SIGDescribe("PriorityPidEvictionOrdering [Slow] [Serial] [Disruptive][NodeFeature:Eviction]", func() { f := framework.NewDefaultFramework("pidpressure-eviction-test") f.NamespacePodSecurityLevel = admissionapi.LevelPrivileged <del> pressureTimeout := 2 * time.Minute </del> <ins> pressureTimeout := 3 * time.Minute </ins> expectedNodeCondition := v1.NodePIDPressure expectedStarvedResource := noStarvedResource

PriorityPidEvictionOrdering [Slow] [Serial] [Disruptive][NodeFeature:Eviction] when we run containers that should cause PIDPressure should eventually evict all of the correct pods /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node /assign /priority important-soon
fork-bomb-container-container exited with code 2. I tried to create a pod in my env of containerd: The pod can be running and it works as designed. I need to test it in cri-o later.
/cc
/triage accepted


Question: command: - /usr/local/bin/kube-proxy - --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf <ins> {{ .ExtraParams }} </ins> {{ .ClusterCIDR }} securityContext: privileged: true

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently, IPVS proxier does not run any e2e test cases. What you expected to happen: I think it's important to create test suite for IPVS in upstream CI which will help a lot in moving IPVS existing alpha. Anything else we need to know?: Let's work together. /sig network
Self /assign
There are to ways to do this work: , but doesn't support any kube-proxy config right now, so I had this pr , and also, need support it too, see this pr: kubernetes/kubernetes-anywhere build cluster. See this pr: kubernetes/test-infra The second way is much easier and faster, but I was told they gonna deprecated the , so in the long term, we still need and 's support. If anybody have better way or have some suggestion about this, please let me know, I'll be really appreciate, Thanks.


Question: allErrs = append(allErrs, field.Forbidden(fldPath.Child("x-kubernetes-int-or-string"), "must be false to be structural")) } <ins> // forbid reasoning about metadata because it can lead to metadata restriction we don't want if _, found := v.Properties["metadata"]; found { allErrs = append(allErrs, field.Forbidden(fldPath.Child("properties").Key("metadata"), "must not be specified in a nested context")) } </ins> return allErrs }

We allow CRD authors to write . Currently we don't restrict writing validation for , which means people can have . For example, people can have additive name constraints (require CR to be ) with the following schema: This brings us two questions: do we if "metadata" is included in the schema: simplest option is to publish standard metav1.ObjectMeta schema info, and use any CRD-specified "metadata" properties as additive validation (just like in-tree types do, publishing standard ObjectMeta schema info, and implementing type-specific name validation internally) alternatively, we can try merging user-specified metadata schema with the standard one. Technically people should only add extra validation to existing metadata properties (adding new fields to metadata won't work since they will be pruned by built-in validation first), but it's unclear to me how arbitrary people may want the metadata validation to be term question is whether to allow people specifying metadata validation at all, or just allow a subset (like name validation) in CRD v1 /sig api-machinery /area custom-resources cc
What is the difference between these two? I mean the end result is the same right? The only thing that can be is validation. Nothing else works unless you are considering extensions?
The former means we drop/hide user-specified metadata schema when we publish CRD OpenAPI. We still validate user-specified constraints on server side, but client-side validation won't see them. The latter enables client-side user-specified metadata validation.
If it's possible to merge with standard metadata schema info, or at the cost of not publishing the standard schema. My inclination would be to publish the complete standard schema for metadata/apiVersion/kind, which we know is complete and accurate, and use and user-specified schema for those fields as addition validations server side


Question: } return false, nil, nil } <del> mockMetadataClient := metadatafake.NewSimpleMetadataClient(scheme.Scheme) </del> <ins> mockMetadataClient := metadatafake.NewSimpleMetadataClient(metadatafake.NewTestScheme()) </ins> mockMetadataClient.PrependReactor("delete-collection", "flakes", ns1FlakesNotFound) mockMetadataClient.PrependReactor("list", "flakes", ns1FlakesNotFound)

Because of this scheme mutation, this leads to concurrent use of the same scheme to cause the program to panic. It would be ideal to do this in some concurrent safe manner or add a warning or similar
/sig api-machinery
/assign
/assign /triage accepted


Question: // This call is idempotent, and must not return an error if the sandbox has // already been removed. func (f *RemoteRuntime) RemovePodSandbox(ctx context.Context, req *kubeapi.RemovePodSandboxRequest) (*kubeapi.RemovePodSandboxResponse, error) { <del> err := f.RuntimeService.RemovePodSandbox(req.PodSandboxId) </del> <ins> err := f.RuntimeService.StopPodSandbox(req.PodSandboxId) </ins> if err != nil { return nil, err }

What happened: It's most visible in the test <img width="323" alt="Screen Shot 2020-05-20 at 6 09 10 PM" src="https://user-"It shows up in kind, kube- and kops jobs The regression was introduced somewhere within , possible culprits are: (FYI (FYI (FYI There are more tests affected, just not as prominently (visually inspecting ) [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data I wasn't sure whether to tag this as kind/regression or not, I'll leave that up to or someone /sig node /sig api-machimery /sig scheduling
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: It's most visible in the test (FYI (FYI (FYI StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data /sig api-machimery /sig scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.


Question: // This call is idempotent, and must not return an error if the sandbox has // already been removed. func (f *RemoteRuntime) RemovePodSandbox(ctx context.Context, req *kubeapi.RemovePodSandboxRequest) (*kubeapi.RemovePodSandboxResponse, error) { <del> err := f.RuntimeService.RemovePodSandbox(req.PodSandboxId) </del> <ins> err := f.RuntimeService.StopPodSandbox(req.PodSandboxId) </ins> if err != nil { return nil, err }

</details
/cc
will increase the time for deletion of pod and it should be expected because currently pod is deleted from apiserver before removing the sandbox container but after this fix pod will be deleted from the apiserver only after deleting the sandbox container.
/remove-sig scheduling doesn't affect scheduling running time at all.
Yeah, is almost definitely the culprit. Last time we changed that code path, it also caused a regression. The degradation in performance is expected, but a 5x slowdown is definitely larger than it should be.
do we know if that's because the status used by the path is stale, or if removing sandboxes is actually that slow?
if there's not a quick fix, should we revert and bring it back in once the root issue is understood?
I think a revert is appropriate in this case.
ok, I'll open it
open at
I agree with that 5x is something which shouldn't happen and i can take a look on why thats happening and that might require changing the way how the sandbox is currently deleted which is only during GC.


Question: function config-ip-firewall { echo "Configuring IP firewall rules" # The GCI image has host firewall which drop most inbound/forwarded packets. <del> # We need to add rules to accept all TCP/UDP packets. </del> <ins> # We need to add rules to accept all TCP/UDP/ICMP packets. </ins> if iptables -L INPUT | grep "Chain INPUT (policy DROP)" > /dev/null; then <del> echo "Add rules to accept all inbound TCP/UDP packets" </del> <ins> echo "Add rules to accept all inbound TCP/UDP/ICMP packets" </ins> iptables -A INPUT -w -p TCP -j ACCEPT iptables -A INPUT -w -p UDP -j ACCEPT <ins> iptables -A INPUT -w -p ICMP -j ACCEPT </ins> fi if iptables -L FORWARD | grep "Chain FORWARD (policy DROP)" > /dev/null; then <del> echo "Add rules to accept all forwarded TCP/UDP packets" </del> <ins> echo "Add rules to accept all forwarded TCP/UDP/ICMP packets" </ins> iptables -A FORWARD -w -p TCP -j ACCEPT iptables -A FORWARD -w -p UDP -j ACCEPT <ins> iptables -A FORWARD -w -p ICMP -j ACCEPT </ins> fi }

ref:
identified one problem: firewall is not open for icmp in GCI setup. I opened it manually. Still does not work.
Here are the symptoms: node can ping each other one node, it can ping all pods on it, and vice versa. one node or pod, it can ping all cbr0 gateways(10.X.X.1). pod, it cannot ping but able to curl. It cannot ping any other pod, even on the same bridge, but able to curl.
My first thought would be somethings blockign ping. your options are iptables, firewall, icmp is disabled alltogether (sysctl or somewhere in /proc/sys/net/ there should be a setting). Try traceroute and see where it gets to.
I have a working version now. GCI setup icmp ACCEPT rules in INPUT chain of filter table. But not in OUTPUT and FORWARD chain.
In summary, 2 problems: is not opened for icmp is filtering icmp on FORWARD chain. ACCEPT on INPUT chain is not enough Will submit a fix
Great detective work, Is this a ping-specific problem, or are there going to be problems with other types of outgoing traffic from pods on GCI as well?
I do not see any problem for udp/tcp traffic so far. e2e tests already covers lots of them. Unfortunately, no e2e covers ping tests and GCI happens to filter it.
We should add a test in networking suite for ping.
Is this missing a milestone 1.3?


Question: return result, nil } <del> // logPluginScores adds summarized plugin score logging. The goal of this block is to show the highest scoring plugins on // each node, and the average score for those plugins across all nodes. func logPluginScores(nodes []*v1.Node, scoresMap framework.PluginToNodeScores, pod *v1.Pod) { totalPluginScores := make(map[string]int64) for j := range scoresMap { for i := range nodes { totalPluginScores[j] += scoresMap[j][i].Score } } // Build a map of Nodes->PluginScores on that node nodeToPluginScores := make(framework.PluginToNodeScores, len(nodes)) for _, node := range nodes { nodeToPluginScores[node.Name] = make(framework.NodeScoreList, len(scoresMap)) } // Convert the scoresMap (which contains Plugins->NodeScores) to the Nodes->PluginScores map for plugin, nodeScoreList := range scoresMap { for _, nodeScore := range nodeScoreList { klog.V(10).InfoS("Plugin scored node for pod", "pod", klog.KObj(pod), "plugin", plugin, "node", nodeScore.Name, "score", nodeScore.Score) nodeToPluginScores[nodeScore.Name] = append(nodeToPluginScores[nodeScore.Name], framework.NodeScore{Name: plugin, Score: nodeScore.Score}) } } for node, scores := range nodeToPluginScores { // Get the top 3 scoring plugins for each node sort.Slice(scores, func(i, j int) bool { return scores[i].Score > scores[j].Score }) var topScores []pluginScores for _, score := range scores { pluginScore := pluginScores{ Plugin:           score.Name, Score:            score.Score, AverageNodeScore: float64(totalPluginScores[score.Name]) / float64(len(nodes)), } topScores = append(topScores, pluginScore) if len(topScores) == 3 { break } } klog.InfoS("Top 3 plugins for pod on node", "pod", klog.KObj(pod), "node", node, "scores", topScores, ) } } </del> // NewGenericScheduler creates a genericScheduler object. func NewGenericScheduler( cache internalcache.Cache,

Scheduler throughput drops significantly in http://perf- ! After , seems to be the curlspirit.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent /sig scheduling /sig scalability


Question: // Returns true if and only if changes were made // If the security group no longer exists, will return (false, nil) func (c *Cloud) removeSecurityGroupIngress(securityGroupID string, removePermissions []*ec2.IpPermission) (bool, error) { <ins> // We do not want to make changes to the Global defined SG if securityGroupID == c.cfg.Global.ElbSecurityGroup { return false, nil } </ins> group, err := c.findSecurityGroup(securityGroupID) if err != nil { glog.Warningf("Error retrieving security group: %q", err)

/kind bug What happened: When setting an elbSecurityGroup in for AWS, and deploying multiple services with differing ports, then the security group rules flap in AWS, causing services to become inaccessible (only one will be correctly configured at a time). What you expected to happen: I expect that the configured elbSecurityGroup's rules will not be touched at all. It must be pre-configured to permit access. Alternatively, all ports for all services are queried and managed together. How to reproduce it (as minimally and precisely as possible): See above. Anything else we need to know?: Most people are not running large clusters which may hit the AWS limits, so they won't use this option, nor see this issue. Environment: Kubernetes version (use ): 1.7.1 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 4.10.13-1-ARCH Install tools: Custom Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig aws
This is fixed by