Question: if podSpec.TerminationGracePeriodSeconds != nil && *podSpec.TerminationGracePeriodSeconds < 0 { warnings = append(warnings, fmt.Sprintf("%s: must be >= 0; negative values are invalid and will be treated as 1", fieldPath.Child("spec", "terminationGracePeriodSeconds"))) } <ins> if podSpec.Affinity != nil { if affinity := podSpec.Affinity.PodAffinity; affinity != nil { warnings = append(warnings, warningsForPodAffinityTerms(affinity.RequiredDuringSchedulingIgnoredDuringExecution, fieldPath.Child("spec", "affinity", "podAffinity", "requiredDuringSchedulingIgnoredDuringExecution"))...) warnings = append(warnings, warningsForWeightedPodAffinityTerms(affinity.PreferredDuringSchedulingIgnoredDuringExecution, fieldPath.Child("spec", "affinity", "podAffinity", "preferredDuringSchedulingIgnoredDuringExecution"))...) } if affinity := podSpec.Affinity.PodAntiAffinity; affinity != nil { warnings = append(warnings, warningsForPodAffinityTerms(affinity.RequiredDuringSchedulingIgnoredDuringExecution, fieldPath.Child("spec", "affinity", "podAntiAffinity", "requiredDuringSchedulingIgnoredDuringExecution"))...) warnings = append(warnings, warningsForWeightedPodAffinityTerms(affinity.PreferredDuringSchedulingIgnoredDuringExecution, fieldPath.Child("spec", "affinity", "podAntiAffinity", "preferredDuringSchedulingIgnoredDuringExecution"))...) } } return warnings } func warningsForPodAffinityTerms(terms []api.PodAffinityTerm, fieldPath *field.Path) []string { var warnings []string for i, t := range terms { if t.LabelSelector == nil { warnings = append(warnings, fmt.Sprintf("%s: a null labelSelector results in matching no pod", fieldPath.Index(i).Child("labelSelector"))) } } return warnings } func warningsForWeightedPodAffinityTerms(terms []api.WeightedPodAffinityTerm, fieldPath *field.Path) []string { var warnings []string for i, t := range terms { // warn if labelSelector is empty which is no-match. if t.PodAffinityTerm.LabelSelector == nil { warnings = append(warnings, fmt.Sprintf("%s: a null labelSelector results in matching no pod", fieldPath.Index(i).Child("podAffinityTerm", "labelSelector"))) } } </ins> return warnings }

Currently, when the Label Selector in the Pod Affinity is empty, no objects will be match. That means if it's in a required PodAffinity, the Pod will never be schedulable. And if it's in a required PodAntiAffinity, that PodAntiAffinity will be no-op. So, I'm thinking PodAffinity with empty Label Selector is very likely mistake. It'd be good if we could notice it for users. To prevent such a misconfiguration. Probably, we cannot filter out such a Pod creation in the validation for a backward compatibility. /assign /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm wondering if it has already discussed somewhere or not. Please just let me know if we have some decisions on it.
I am not sure, where do you propose to put the warning? If the pod is unschedulable, that will show up in the events and pod status.
Do you mean the place to implement this warning? I suspect ↓ is a suitable place:
A warning sounds useful, even if the pod is unschedulable, as it gives you a heads up at the time when you create the Pod. Whereas the Pod scheduling failure might be a bit harder to interpret.
ok, I thought you were referring to something that the scheduler will produce after attempting to schedule the pod. Yeah, if there is already a setup to print warnings on pod creation, it is certainly useful for this case.
Can you please also add it in topology spread?
Sure


Question: defer nc.evictorLock.Unlock() if status == v1.ConditionFalse { if !taintutils.TaintExists(node.Spec.Taints, NotReadyTaintTemplate) { <del> nc.zoneNoExecuteTainter[utilnode.GetZoneKey(node)].SetRemove(node.Name) </del> <ins> nc.zoneNoExecuteTainter[utilnode.GetZoneKey(node)].Remove(node.Name) </ins> } } if status == v1.ConditionUnknown { if !taintutils.TaintExists(node.Spec.Taints, UnreachableTaintTemplate) { <del> nc.zoneNoExecuteTainter[utilnode.GetZoneKey(node)].SetRemove(node.Name) </del> <ins> nc.zoneNoExecuteTainter[utilnode.GetZoneKey(node)].Remove(node.Name) </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Nodelifecycle controller does not set NoExecute taint to NotReady node when kubelet has been stopped What you expected to happen: effect: NoExecute key: Nodelifecycle controller sets NoExecute taint to NotReady node when kubelet has been stopped How to reproduce it (as minimally and precisely as possible): systemctl stop kubelet Anything else we need to know?: I use default args to start kube-controller-manager. I see log from kube-controller-manager: Environment: Kubernetes version (use ): [root ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): [root ~]# cat /etc/os-release NAME="CentOS Linux" VERSION="8 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="8" PLATFORMID="platform:el8" PRETTYNAME="CentOS Linux 8 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:8" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-8" CENTOSMANTISBTPROJECTVERSION="8" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="8" Kernel (e.g.


Question: defer nc.evictorLock.Unlock() if status == v1.ConditionFalse { if !taintutils.TaintExists(node.Spec.Taints, NotReadyTaintTemplate) { <del> nc.zoneNoExecuteTainter[utilnode.GetZoneKey(node)].SetRemove(node.Name) </del> <ins> nc.zoneNoExecuteTainter[utilnode.GetZoneKey(node)].Remove(node.Name) </ins> } } if status == v1.ConditionUnknown { if !taintutils.TaintExists(node.Spec.Taints, UnreachableTaintTemplate) { <del> nc.zoneNoExecuteTainter[utilnode.GetZoneKey(node)].SetRemove(node.Name) </del> <ins> nc.zoneNoExecuteTainter[utilnode.GetZoneKey(node)].Remove(node.Name) </ins> } }

): [root ~]# uname -a Linux liabio 4.18.0-147.5.1.el81.x8664 SMP Wed Feb 5 02:00:39 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle /wg cluster-lifecycle
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-cluster-lifecycle /wg sig-cluster-lifecycle /committee sig-cluster-lifecycle Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig cluster-lifecycle /sig node
Those labels are not set on the issue: <detailsIn response to : /remove-sig cluster-lifecycle /sig node </details
is NoExecute the expected taint for this condition?
- yes it is when running the taint-manager: evictions are usually safe eviction as we don't force delete pods
ref
I see the same, a node which has node condition set to Unknown, does not get an Unreachable taint with NoExecute effect. However I only see this on 1.19, works fine till 1.18.10 Till 1.18.10, such a node gets two Unreachable taints, one with effect NoExecute and one with effect NoSchedule. But on 1.19.3 it only gets the NoSchedule taint. I see that this is the only commit made in that area between 1.18 and 1.19: , could be related?
It totally breaks production readiness of all recent releases, yet doesn't seem to be regarded as a top priority issue. WTF?


Question: // find node-security-group for service var err error if len(lbaas.opts.NodeSecurityGroupIDs) == 0 { <del> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, nodes) </del> <ins> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, lbaas.network, nodes) </ins> if err != nil { return fmt.Errorf("failed to find node-security-group for loadbalancer service %s/%s: %v", apiService.Namespace, apiService.Name, err) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kubelet fails when Kubernetes master gets started using cloud-provider openstack und kubeadm init command. says: What you expected to happen: kubelet starts and manages the loadbalancer and its security group. How to reproduce it (as minimally and precisely as possible): Setup an instance on Openstack. Place the following cloud-config in And : As well as : Run: For further information: +-----------------+
--------------------------+ +-----------------+
--------------------------+ +-----------------+
--------------------------+ Environment: Kubernetes version (use ): 1.9.1 Cloud provider or hardware configuration: Openstack OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS Kernel (e.g. ): x86_64 Install tools: kubeadm
/sig cloud-provider
Looks like auto-probing for for (s), please see [1] and [2]. Apologies, the deprecation should have been handled better. [1] [2]
If I understand you correctly, this cloud-config would be enough: If I try this, no Loadbalancer gets created at all and the service which requests type LoadBalancer says: The service I use there is the nginx helm chart ().
Can you please comment? /assign
/sig openstack
From that, '' is your service name and '-202f-4d44-a538-' is your subnet id, can you ensure '-202f-4d44-a538-' exist? Is it "-c60c-48cb-a146-"? I suspect that failed to autoprobe subnet. Can you give me more logs from kube controller manager
Actually, -202f-4d44-a538- is a network, not a subnet. The subnet has the ID -c1d5-4cc1-8b98-
And here the relevant logs from kube-controller-manager,first: And then it keeps repeating:
Oh, I see. Oops, should get 'sub network id', not 'network id'. I will fix it soon.
I tried the fix in 1.9.2, but it does not work. I tried multiple setting combinations of providing and/or/neither , but none works. The Service does not get a LoadBalancer IP assigned.


Question: // find node-security-group for service var err error if len(lbaas.opts.NodeSecurityGroupIDs) == 0 { <del> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, nodes) </del> <ins> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, lbaas.network, nodes) </ins> if err != nil { return fmt.Errorf("failed to find node-security-group for loadbalancer service %s/%s: %v", apiService.Namespace, apiService.Name, err) }

Here are the logs and settings for the config version neither providing nor , since it seemed that's the way you recommended to me: cat /etc/kubernetes/pki/cloud-config kubectl get nodes journalctl -u kubelet.service kubectl describe svc ingress-nginx-ingress-controller Can you please re-open the issue? Do you need any other information?
Is it ok when you disable manage-security-group(manage-security-group=false)?
And please review .
/reopen
Yes, when I disable "manage-security-group" it assigns a LoadBalancer correctly. It also works, as soon as I run . I tried to check , but since I don't know go, I don't really understand what you are doing there and how it relates to my issue.
I check gophercloud.ErrResourceNotFound is not a interface, so the '' will return error and excute "" which is 'error occurred finding security group: ...'.
Okay, so you "just" print a better error message, right? But my question is, why does the error occur at all? In my understanding the kubelet should create a security-group on itself, if configured manage-security-groups=true and do not provide a security-group. However I also tried to define a valid security group id on my own, but in that case I again find these logs in kubelet (same as described first):
Okay, so you "just" print a better error message, right? But my question is, why does the error occur at all? In my understanding the kubelet should create a security-group on itself, if configured manage-security-groups=true and do not provide a security-group. No, not 'just print a better error message', openstack cloud provider can ensure security group for LB. Firstly, openstack cloud provider check whether security group does not exist, if not exist, it will create security group for LB. The "gophercloud.ErrResourceNotFound" means that security group not exist.
Now OpenStack cloud provider has not "node-security-group" variable. You should not specify it.
Please check this note:
Well, to me it does not look like the removal of any resources is the problem, rather that they do not get created at all. I can offer you a direct debugging session using Google Hangout or so, maybe that way it is easier to detect the real problem?
Have you see this log?


Question: // find node-security-group for service var err error if len(lbaas.opts.NodeSecurityGroupIDs) == 0 { <del> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, nodes) </del> <ins> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, lbaas.network, nodes) </ins> if err != nil { return fmt.Errorf("failed to find node-security-group for loadbalancer service %s/%s: %v", apiService.Namespace, apiService.Name, err) }

It means the security group we need to create does not exist, we should create it, not raise error, right? And I get 'non-interface type ErrResourceNotFound on left' error, 'err.(*gophercloud.ErrResourceNotFound)' is incorrect.
Yes, exactly. I also tried to fix the error manually, so first I created the sg-group . After that kubelet complained about empty pool-targets. After fixing that, it complained about invalid endpoint which pointed to the nginx server, don't know how to fix that one yet. I just want to point out: All that errors only occur if I activate manageSecurityGroups, otherwise the kubelet is able to assign a LoadBalancer to the IngressController.
Hi, how is going? I've deployed k8s 1.9.5 on OpenStack with kubespray and I'm struggling with this issue. If I do not set the LBAAS is created but it's not assigned to any Security Group. So, the service is not reachable from the outside world. I can fix it by hand creating the Security Group and assign it to the VIP. If I do set kubelet fails to start. According to it's a valid parameter... (?) And finally, if I do as in his last comment, the LBAAS remains in a PENDING state forever... Well, I wonder if there is any workaround to deal with this... I presume you're working on the fix, aren't you? Any information will be appreciate. Thanks
I'm hitting the same issue as yours with openstack loadbalancer with 1.9.4. Anyone know if fixes this issue? If so, when will it be part of a release?
Any updates on this?
I just tried to use this feature in kubernetes 1.10.2 and it still seems to be broken. The security groups do not get created and the service creation is stuck at pending without an ip address. I used this configuration and i also tried all the alternatives listed here. Either nothing gets created or the controller-manager crashes if i add node-security-group.
It's an issue with as well. The behaviour is as described by
Bump
fixes this for us.
So, If I do not enable manage-security-groups, This should work?


Question: // find node-security-group for service var err error if len(lbaas.opts.NodeSecurityGroupIDs) == 0 { <del> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, nodes) </del> <ins> lbaas.opts.NodeSecurityGroupIDs, err = getNodeSecurityGroupIDForLB(lbaas.compute, lbaas.network, nodes) </ins> if err != nil { return fmt.Errorf("failed to find node-security-group for loadbalancer service %s/%s: %v", apiService.Namespace, apiService.Name, err) }

I am getting the following error: Error creating load balancer (will retry): failed to ensure load balancer for service default/lbs-test: error creating LB floatingip {FloatingNetworkID:YYYY FloatingIP: PortID:XXXX FixedIP: TenantID:}: Resource not found
Yes. Don't specify and at all in config. The LoadBalancer(s) will by default apply the security rules in the security-group .
You need to specify to automatically add / alter security groups. I get the following error after I've upgraded to 1.9.10:
I ran into the same issue. It seems to be related to how k8's openstack driver deals with adding floating ips to the LB. For now I set the lb to internal in the service metadata then manually associate a floating ip . This could also be a queens magnum issue not providing all required cloud config info. I am not sure yet.


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/testapi" <del> "github.com/GoogleCloudPlatform/kubernetes/pkg/apiserver" </del> "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/fields" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" <del> "github.com/GoogleCloudPlatform/kubernetes/pkg/master" </del> "github.com/GoogleCloudPlatform/kubernetes/pkg/version" "github.com/GoogleCloudPlatform/kubernetes/pkg/watch" <del> "github.com/GoogleCloudPlatform/kubernetes/plugin/pkg/admission/admit" </del> ) func init() { requireEtcd() } <del> func RunAMaster(t *testing.T) (*master.Master, *httptest.Server) { helper, err := master.NewEtcdHelper(newEtcdClient(), testapi.Version()) if err != nil { t.Fatalf("unexpected error: %v", err) } var m *master.Master s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { m.Handler.ServeHTTP(w, req) })) m = master.New(&master.Config{ EtcdHelper:        helper, KubeletClient:     client.FakeKubeletClient{}, EnableLogsSupport: false, EnableProfiling:   true, EnableUISupport:   false, APIPrefix:         "/api", Authorizer:        apiserver.NewAlwaysAllowAuthorizer(), AdmissionControl:  admit.NewAlwaysAdmit(), }) return m, s } </del> func TestClient(t *testing.T) { <del> _, s := RunAMaster(t) </del> <ins> _, s := runAMaster(t) </ins> defer s.Close() ns := api.NamespaceDefault

In 0.13.2, the apiserver has automatically exposed metrics about the process, like start time, memory usage, processor usage, etc. This stopped working sometime between then and when 0.14.0 was cut. They're still correctly exported from the kubelet, interestingly enough. An e2e or integration test is also needed to catch this if it happens again.


Question: [A more robust example that follows Elasticsearch best-practices of separating nodes concern is also available](production_cluster/README.md).   Current pod descriptors use an `emptyDir` for storing data in each data node container. This is meant to be for the sake of simplicity and [should be adapted according to your storage needs](../../docs/design/persistent-storage.md).    Current pod descriptors use an `emptyDir` for storing data in each data node container. This is meant to be for the sake of simplicity and [should be adapted according to your storage needs](../../docs/design/persistent-storage.md).  ## Docker image

This includes but effects multiple docs.
This issue was also raised on the repository, see:
What's needed to fix this? Are the broken headers being generated automatically somewhere?
- The warning section is pre-pended to the md docs from the following script The fixes this issue, updating this script and all the docs... However, there is an open question as to whether "old" docs should be removed from this repo, which has prevented this PR from being merged.


Question: } return priority } <ins> // getGracePeriodOverrideTestPod returns a new Pod object containing a container // runs a shell script, hangs the process until a SIGTERM signal is received. // The script waits for $PID to ensure that the process does not exist. // If priorityClassName is scheduling.SystemNodeCritical, the Pod is marked as critical and a comment is added. </ins> func getGracePeriodOverrideTestPod(name string, node string, gracePeriod int64, priorityClassName string) *v1.Pod { pod := &v1.Pod{ TypeMeta: metav1.TypeMeta{

Followup feedback from In a few e2e node tests (e.g. tests of deletion grace period) we have some busybox pods that wait for SIGTERM and perform some action like a small sleep to simulate SIGTERM taking some time. These tests follow the pattern of -- see The will block the SIGTERM handler from being run until after the time is elapsed. We should consider to remove the and instead use to avoid blocking the SIGTERM handler, see pattern followed in n/a n/a No response n/a n/a <details</details<details</details<details</details<details</details
/sig node /cc
/assign
/triage accepted


Question: - name: socket-dir mountPath: /csi - name: csi-provisioner <del> image: registry.k8s.io/sig-storage/csi-provisioner:v3.3.0 </del> <ins> image: registry.k8s.io/sig-storage/csi-provisioner:v3.4.0 </ins> args: - "--v=5" - "--csi-address=/csi/csi.sock"

Seen in pull-kubernetes-e2e-gce-cos-alpha-features Specific job failure
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from the error message I don't know if it refers to the sig-storage error or to the sig-auth one
/sig storage
/kind failing-test
/assign From : ... I am not sure if it's related:
is the culprit. cannot be used inside a . This only gets caught when the test actually runs, otherwise Ginkgo never sees the incorrect .
I don’t understand why sometimes it can get the pv by namebut sometimes it failed
but then is a sig-auth test , not storage , sorry /sig auth this seems that was recently in
/remove sig-storage
/assign /priority important-soon
/unassign
is still failed. seems not fixing the whole problem. /reopen feel free to close if the ci is green.
Reopened this issue. <detailsIn response to : is still failed. /reopen feel free to close if the ci is green. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the failure now looks like it is just a sig storage issue:
cc
/assign looks like the new test is still an issue
I think this line is the problem: When the test creates the object, it may take time for the PV to be provisioned and to be bound to the PVC. So you should do another before this call so you can get the latest PVC object with the bound PV.
I opened to get PVC again before using .
Only do PersistentVolumeClaims.Get() is not enough, whether we need to wait for sometimes to wait it bound? because sometimes the testcase is successful and sometime is failed.
If the Pod was able to run, then the PVC has to be bound.
so I can't understand why sometimes the testcase is successful and sometimes is failed.


Question: - name: socket-dir mountPath: /csi - name: csi-provisioner <del> image: registry.k8s.io/sig-storage/csi-provisioner:v3.3.0 </del> <ins> image: registry.k8s.io/sig-storage/csi-provisioner:v3.4.0 </ins> args: - "--v=5" - "--csi-address=/csi/csi.sock"

By default, the DelayBinding is false, it will wait for PVC bound during creation. This is weird.
It waits for the claim to be bound, but the test case still needs to refresh its pvc object.
Because of the refresh is async, so after the pvc was bound, we need to get the pvc again, isn't it?


Question: package common import ( <ins> "encoding/json" </ins> "fmt" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/kubernetes/test/e2e/framework" imageutils "k8s.io/kubernetes/test/utils/image"

This issue proposes the writing of a new test for the API endpoints , with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this APIsnoop query, there are still some remaining ConfigMap endpoints which are untested. Create a ConfigMap with a static label Patch the ConfigMap with a new Label and updated data Get the ConfigMap to ensure it's patched List all ConfigMaps in all Namespaces find the ConfigMap(1) ensure that the ConfigMap is found and is patched Delete Namespaced ConfigMap(1) via a Collection with a LabelSelector This mock test uses client-go's standard functions for talking to the Event area of the API. Discover useragents: Display endpoints hit by the test: Show the change in coverage after the mock test: If a test with these calls gets merged, Conformance coverage will go up by 2 points /sig testing /sig architecture /area conformance
Bad to list all secrets without a label selector... look at label usage in secret test. Approved for work on 14th January Meeting.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/retitle Write CoreV1ConfigMap test+promote - +2 Coverage
please close this Issue in favour of and place in In Progress Issues on org dashboard
/close Close this Issue in favour of and place in In Progress Issues on org dashboard
Closing this issue. <detailsIn response to : /close Close this Issue in favour of and place in In Progress Issues on org dashboard Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close close this Issue in favor of and place in In Progress Issues on org dashboard


Question: return false, err }) } <ins> func unstructuredToResourceQuota(obj *unstructured.Unstructured) (*v1.ResourceQuota, error) { json, err := runtime.Encode(unstructured.UnstructuredJSONScheme, obj) if err != nil { return nil, err } rq := &v1.ResourceQuota{} err = runtime.DecodeInto(clientscheme.Codecs.LegacyCodec(v1.SchemeGroupVersion), json, rq) return rq, err } </ins>

[X] Test approval issue: [X] Test PR: 19 Oct 2022 [x] Two weeks soak end date: 2 Nov 2022 [x] Test promotion PR: According to this APIsnoop query, there are still three remaining ResourceQuota status endpoints which are untested. patchCoreV1NamespacedResourceQuotaStatusreplaceCoreV1NamespacedResourceQuotaStatusreadCoreV1NamespacedResourceQuotaStatus Using a number of existing e2e test practices a new has been created for 3 ResourceQuota Status endpoints. The e2e logs for this test are listed below. This query shows which Resource Status endpoints are hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 3 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: name := obj.GetName() DryRunCreateTest(t, rsc, obj, gvResource) <ins> DryRunCreateWithGenerateNameTest(t, rsc, obj, gvResource) </ins> if _, err := rsc.Create(context.TODO(), obj, metav1.CreateOptions{}); err != nil { t.Fatalf("failed to create stub for %s: %#v", gvResource, err)

Running kubectl create with dry-run returns unwanted uid metadata field, as well as a generated name when using generateName. This is inconsistent with the expectations as defined in which states I expected uid, and a generated name to be empty when using the dry-run create option. Expected metadata: Actual metadata: No response <detailsI've tested this from 1.19.0 through 1.23.0 </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
reopening, is reverted in to make sure requests with both name+generateName are handled correctly
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


Question: name := obj.GetName() DryRunCreateTest(t, rsc, obj, gvResource) <ins> DryRunCreateWithGenerateNameTest(t, rsc, obj, gvResource) </ins> if _, err := rsc.Create(context.TODO(), obj, metav1.CreateOptions{}); err != nil { t.Fatalf("failed to create stub for %s: %#v", gvResource, err)

/close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: SetTCPEstablishedTimeout(seconds int) error // SetTCPCloseWaitTimeout adjusts nf_conntrack_tcp_timeout_close_wait. SetTCPCloseWaitTimeout(seconds int) error <ins> // SetTCPBeLiberal adjusts nf_conntrack_tcp_be_liberal. SetTCPBeLiberal(value int) error </ins> // SetUDPTimeout adjusts nf_conntrack_udp_timeout. SetUDPTimeout(seconds int) error // SetUDPStreamTimeout adjusts nf_conntrack_udp_timeout_stream.

What happened: When kube-proxy is configured with a cluster-cidr range, it drops packets with an INVALID conntrack state for all sources on the KUBE-FORWARD chain. This is meant to prevent spurious retransmits in long-lived TCP connections to a service IP, but it prevents clusters from having asymmetrical routing on private subnets with services outside of a cluster. What you expected to happen: Packets on the KUBE-FORWARD chain with an INVALID conntrack state should be dropped for only sources inside of the cluster-cidr if set. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: Change to drop INVALID conntrack packets was in Backing out that commit and compiling is my current workaround. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I am reading but I don't fully grok what caused the problem in the first place, or why we think "only drop if it was in-cluster" is a correct fix? Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. Is this a conntrack bug? Should it be applying the DNAT?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I can try to give a bit more context to our setup, but the knowledge I have around this all dates back to Feb, unfortuately, so I may have forgotten details. We were replacing our mesos cluster with a kubernetes cluster in Feb and using a custom ingress solution (HAProxy w/an outside process configuring it for pod ips / mesos ip-ports) for both. The ingress nodes lived in the private subnet, but outside of the kubernetes cluster cidr range. When we upgraded from kubernetes 1.14 to kubernetes 1.15 (the first version with PR ) our ingress could no longer talk to any pod ips.


Question: SetTCPEstablishedTimeout(seconds int) error // SetTCPCloseWaitTimeout adjusts nf_conntrack_tcp_timeout_close_wait. SetTCPCloseWaitTimeout(seconds int) error <ins> // SetTCPBeLiberal adjusts nf_conntrack_tcp_be_liberal. SetTCPBeLiberal(value int) error </ins> // SetUDPTimeout adjusts nf_conntrack_udp_timeout. SetUDPTimeout(seconds int) error // SetUDPStreamTimeout adjusts nf_conntrack_udp_timeout_stream.

I discovered that the KUBE-FORWARD chain had a rule to drop invalid packets, which was in Compiling kubernetes 1.15 without this commit fixed our problems, and we have been carrying that patch forward ever since. I would like to get rid of our forked kubernetes, hence this ticket. When I was writing this ticket up, I also found also mentions the same issue with the iptables dropping INVALID conntrack state packets near the end, regarding asymmetrical routing. Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. I think you are correct, I can imagine packets from outside of a cluster could exhibit the same behavior as packets inside the cluster and cause spurious resets. However, I am wondering if kube-proxy should be managing packets from outside of the cluster on a node... That seems outside of the scope of kube-proxy.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I took a stab at implementing what I think is the correct behavior for kube-proxy in .
/assign
I think that another option is to set
I don't understand how that will help me here. The packets with a ctstate INVALID are not generated due to conntrack saturation (), but due to asymmetric routing in our network. See for an example of a similar setup.
are the packets on your asymmetric routing setup marked as invalid if you set ? it should only mark the ones that are out of window per the description
/triage accepted
I'd love to pin this down, but I don't have the means to set up a repro. Is there a way I can set up a way to experiment?
We were bit by this too after revisiting an old service that had been unused since the heady days of pre-, and would also like to see the proposed patch merged (or a configuration option to prevent insertion of the INVALID state drop rule - either suits us). I unfortunately don't have any useful input as to setting up for reproduction. In our case we have a pretty odd anycast set up with a small pool of (public) addresses managed by calico.


Question: SetTCPEstablishedTimeout(seconds int) error // SetTCPCloseWaitTimeout adjusts nf_conntrack_tcp_timeout_close_wait. SetTCPCloseWaitTimeout(seconds int) error <ins> // SetTCPBeLiberal adjusts nf_conntrack_tcp_be_liberal. SetTCPBeLiberal(value int) error </ins> // SetUDPTimeout adjusts nf_conntrack_udp_timeout. SetUDPTimeout(seconds int) error // SetUDPStreamTimeout adjusts nf_conntrack_udp_timeout_stream.

However I do seem to have a dirty workaround for small clusters. By setting notrack (ctstate UNTRACKED?) in the raw table for packets in our specific ranges that would otherwise match ctstate INVALID, the packets aren't dropped. Which may be of help to someone else who finds this issue.
We are seeing something very similar on large file downloads and our issue exactly fits this one's subject. These RESET (corrected after are happening intermittently and I am seeing similar behavior mentioned in the blog mentioned below. I am unsure if someone has found a workaround for this issue or not, so yeah my +1 to this issue being looked up by the community.
We're using as a solution for the connection reset we were seeing and might fix this issue.
Let me try to avoid that this thread derails and see if I can get current status: -proxy install rules to drop INVALID cstate connections, this solves the ~REJECT~ RESET problems are some scenarios, with asymmetric routing per example, that this rules may drop unrelated traffic. The request is to remove the rule that drops those INVALID states or to try to restrict it, and that can cause that people start to have the ~REJECT~ RESET problems and drops again. If we have a reproducer we can be more precise in the final solution, but removing one rule without any strong evidence in kube-proxy is very risky As a workaround for the asymetric routing problem, I suggested to set the nfconntracktcpbeliberal on the kernel: I think that will solve the problems, it will be good if someone can test it and provide feedback
No sadly, its my bad, it is not REJECT, it is RESET that's happening and conntrack has nothing to do with it, mostly (I hope). Kubeproxy is doing what is intended but specifically when there are external agents in play, the packet doesn't sends an ACK back to the response of SYN-ACK. Our case, we hit A to initiate a download, A calls B to get the action completed to generate download data for the user finally via their browser. A and B both are on k8s, we saw A starts and a complete cycle goes for sometime until there multiple parallel downloads requested where few gets completed and few gets a RESET.


Question: SetTCPEstablishedTimeout(seconds int) error // SetTCPCloseWaitTimeout adjusts nf_conntrack_tcp_timeout_close_wait. SetTCPCloseWaitTimeout(seconds int) error <ins> // SetTCPBeLiberal adjusts nf_conntrack_tcp_be_liberal. SetTCPBeLiberal(value int) error </ins> // SetUDPTimeout adjusts nf_conntrack_udp_timeout. SetUDPTimeout(seconds int) error // SetUDPStreamTimeout adjusts nf_conntrack_udp_timeout_stream.

On primary debugging, it appears that RESET was getting invoked by the pod but pod is just sending a RST-ACK not RST itself. The missing packet is never found in the default route, unless we see that there is a chance where ACK is missed and we found some lines in tcpdump with missing sequence for ACKs. Thats why i doubt my issue is related here. One more weird coincidence is few downloads have spurious transmission.
Thankyou, I will take a look, but mostly I would like to wait before using it, debug more to understand the actual cause and then act on it.
sorry , I wanted to say RESET, thanks for catching it. If you think is different and you can provide information and how to reproduce it, I suggest you to open a new issue
I will sure do, when it is different. For now I would like to watch and submit my findings if it is related. PS: sysctl -w net.netfilter.nfconntracktcpbeliberal=1 is already implemented and we are testing if it does something helpful. so far it doesn't looks like it. thanks for your help.
/area kube-proxy
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I have the feeling we are also affected by this with a standard calico installation: every node announces a /122 IPv6 network to the upstream router the upstream router uses ECMP to route the packet back packets that are sent from* the pod and returned via asymmetric path from outside seem to be dropped
and if you set ?
Same behaviour with set on all nodes, the outgoing pod traffic does not return. And I instantly see packets going back into the drop rule:
:thinking:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.


Question: SetTCPEstablishedTimeout(seconds int) error // SetTCPCloseWaitTimeout adjusts nf_conntrack_tcp_timeout_close_wait. SetTCPCloseWaitTimeout(seconds int) error <ins> // SetTCPBeLiberal adjusts nf_conntrack_tcp_be_liberal. SetTCPBeLiberal(value int) error </ins> // SetUDPTimeout adjusts nf_conntrack_udp_timeout. SetUDPTimeout(seconds int) error // SetUDPStreamTimeout adjusts nf_conntrack_udp_timeout_stream.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, are you still working on this? Same problem happened in our clusters, just because of asymmetrical routing.
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Also the command suggested by is helping us, but I still think this should be a bug of kube-proxy.
The problem here is: what happens if the solutions are exclusive or we regress? The addition of this rule fixed a legit bug that was documented and there is also a nice blog explaining the details Removing this rule without a repro and understanding what is the root cause of this issue is not really the best way to proceed. I think that people affected should try to find the root cause and explain the problem or just provide a way to reproduce it, so we can fix it
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


Question: SetTCPEstablishedTimeout(seconds int) error // SetTCPCloseWaitTimeout adjusts nf_conntrack_tcp_timeout_close_wait. SetTCPCloseWaitTimeout(seconds int) error <ins> // SetTCPBeLiberal adjusts nf_conntrack_tcp_be_liberal. SetTCPBeLiberal(value int) error </ins> // SetUDPTimeout adjusts nf_conntrack_udp_timeout. SetUDPTimeout(seconds int) error // SetUDPStreamTimeout adjusts nf_conntrack_udp_timeout_stream.

/close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The same problem happened in our cluster, the same as because of asymmetrical routing. the command suggested by doesn't work for us, and I delete the iptables rule and add new rule(refer to ), it works I think that makes sense I'll put up a reproduction of the gist link later on /reopen
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The main problem is how to be completely sure we don't regress, this comment from Tim is spot on
I'm trying to check that non-local connections will still reset without the INVALID Rule
I think the patch in is wrong; it changes the rule from "if INVALID then drop" to "if INVALID and from a pod, then drop" so that it doesn't affect non-kubernetes traffic, but I think the rule we actually need is "if INVALID and to a service, then drop". In particular, I believe we need to make sure that the rule applies to traffic with a cluster-external source and an in-cluster destination. It's possible that changing it to "if INVALID and to a pod, then drop" would also work. I skimmed and some linked bugs elsewhere but I was unable to confirm whether that rule would cover the reporters' use cases.
I think that is only INVALID and to a pod
In my case, cluster-external source to service is both ok with the INVALID rule or without the INVALID rule. It seems the rule doesn't affect on access service.


Question: SetTCPEstablishedTimeout(seconds int) error // SetTCPCloseWaitTimeout adjusts nf_conntrack_tcp_timeout_close_wait. SetTCPCloseWaitTimeout(seconds int) error <ins> // SetTCPBeLiberal adjusts nf_conntrack_tcp_be_liberal. SetTCPBeLiberal(value int) error </ins> // SetUDPTimeout adjusts nf_conntrack_udp_timeout. SetUDPTimeout(seconds int) error // SetUDPStreamTimeout adjusts nf_conntrack_udp_timeout_stream.

I add this rule but it doesn't work
So, based on that, the bug happens when a reply packet fails to get un-DNAT'ed, causing the client to then send a RST to the server, which matches the source/destination IP that the server knows of. But this could happen if you had a service with external IPs too; in that case the reply packet would be from the external endpoint IP to the node IP (because the packet would have been masqueraded), and if conntrack didn't unmap it correctly then the node would send a RST to the external IP, breaking the connection in just the same way. In this case there is no good way kube-proxy can recognize the packet as being its own, other than by using conntrack, which presumably won't work if the packet is INVALID?
It seems that there is no better way to solve this for external sources accessing the cluster CIDR, is it right? If so, Can we have a flag to decide whether to create The invalid rule? It would be nice that It's up to the user to decide whether to create the rule.
There's related discussion in . I think we should create whatever iptables rules we can that will fix up our own traffic without interfering with other traffic, and also document the sysctl that admins can use to change conntrack behavior to work around the problem. So probably, we should change the "drop if INVALID" rule to "drop if INVALID and to a pod", and also address the problem in , but not try to solve the problem with external IPs I mentioned in because we don't have a good way of fixing that without potentially breaking other things. (And it's kind of way more of an edge case anyway.) Note that can currently output rules for "from a pod", but not for "to a pod", so that will need some tweaking. (FWIW will also eventually need the ability to output nftables rules rather than iptables rules so that's another thing to think about if someone is going to change its API.)
So in the absence of a way to fix the problem with external IPs, can we have a flag to decide whether to create this rule or not? If true (default), create the drop if INVALID and to a pod rule. Otherwise, We do not create the rule.


Question: SetTCPEstablishedTimeout(seconds int) error // SetTCPCloseWaitTimeout adjusts nf_conntrack_tcp_timeout_close_wait. SetTCPCloseWaitTimeout(seconds int) error <ins> // SetTCPBeLiberal adjusts nf_conntrack_tcp_be_liberal. SetTCPBeLiberal(value int) error </ins> // SetUDPTimeout adjusts nf_conntrack_udp_timeout. SetUDPTimeout(seconds int) error // SetUDPStreamTimeout adjusts nf_conntrack_udp_timeout_stream.

I believe that even without this rule we can solve this problem by way of sysctl, right? I think we should do the following things We need a flag to disable the rule We should change the "drop if INVALID" rule to "drop if INVALID and to a pod" We document sysctl as a way to solve the problem.
If the rule is "drop if INVALID and to a pod", we shouldn't need a flag to disable it; it won't affect non-pod traffic, and it won't interfere with users that want to set the sysctl instead. (Setting the sysctl will cause packets to not be marked as INVALID so the rule we add just won't get hit.)
OK, drop if INVALID and to a pod is help us identify cluster traffic, right? We only drop cluster traffic if it's invalid, but this will not solve the problem of external IP. So we need a flag to disable the rule is drop if INVALID?
The same problem happened in our cluster, the same as because of asymmetrical routing None of the above methods solve the problem in our ENV. Are there any other workarounds?
Can you execute the following command on each node? I think this is a workaround.
Thank you for your reply I also tried removing this rule. However, k8s will reconfigure the deleted rules to the system at some point in time.
I worded that badly: I meant "external IP" as in "the endpoints are cluster-external IPs rather than pod IPs", not as in . So that would apply only to people using kube-proxy to load-balance traffic from clients outside the cluster to servers outside the cluster, which is a pretty weird use case. And nobody ever reported the packet RST bug for that case anyway. So I was arguing that we can just ignore that case. So: change the existing "drop if invalid" rule to "drop if invalid and to a pod" document the sysctl We don't need an option to disable the drop rule, because it won't affect non-k8s traffic, and it will automatically become a no-op if someone decides to use the sysctl instead.
Thank you for your response. I'd love to take this one. But I have a question, How to define k8s traffic?


Question: SetTCPEstablishedTimeout(seconds int) error // SetTCPCloseWaitTimeout adjusts nf_conntrack_tcp_timeout_close_wait. SetTCPCloseWaitTimeout(seconds int) error <ins> // SetTCPBeLiberal adjusts nf_conntrack_tcp_be_liberal. SetTCPBeLiberal(value int) error </ins> // SetUDPTimeout adjusts nf_conntrack_udp_timeout. SetUDPTimeout(seconds int) error // SetUDPStreamTimeout adjusts nf_conntrack_udp_timeout_stream.

If a pod in a cluster, we call it an underlay pod, its IP is not allocated from cluster CIDR, this is a real underlay IP, when it accesses other underlay pods in the cluster or pods that allocate IP from cluster CIDR(We call it overlay pod), is this k8s traffic? In our cluster, multiple types of CNI pods are implemented through multus running in a cluster, because of the problem of asymmetrical routing and the rule that makes them inaccessible, it seems that the above method will not solve this problem.
For kube-proxy, this is done by the . If you are in a cluster where there is no simple way to discriminate pod from non-pod traffic, then you'd have to run kube-proxy with , and then kube-proxy will see that returns and so it won't try to distinguish local from non-local traffic, and will do some things (like internal/external traffic policy) slightly differently as a result. So in this case, if there was no functioning localDetector, it would also not create the "drop invalid" rule
Thanks for the details, this is probably the case we want.
/assign
I did not understand how you could get asymmetric routing in K8s, but reading up more carefully I saw: If I understand this correctly, this is a load-balancer outside K8s that load-balance directly to POD addresses. To get asymmetric routing some kind of Direct Server Return (DSR) must also be used. So, to get asymmetric routing: from an external source is directed to an external LB LB is configured with POD-IPs as targets and selects one packet is routed to a K8s node, e.g. with ECMP. But is may not be the node where the target POD is executing packet is handled by the POD, but the response packet may be directed to some other GW This will give asymmetric routing. But it's not the every-day K8s network setup for sure. Given the above it's fairly easy to reproduce the problem. You don't even have to setup a load-balancer. The biggest problem may be that you need two ways in to the cluster. Example: ! I setup direct routes that ensures asymmetric routing, which can be checked with on vm-201 and vm-202.


Question: SetTCPEstablishedTimeout(seconds int) error // SetTCPCloseWaitTimeout adjusts nf_conntrack_tcp_timeout_close_wait. SetTCPCloseWaitTimeout(seconds int) error <ins> // SetTCPBeLiberal adjusts nf_conntrack_tcp_be_liberal. SetTCPBeLiberal(value int) error </ins> // SetUDPTimeout adjusts nf_conntrack_udp_timeout. SetUDPTimeout(seconds int) error // SetUDPStreamTimeout adjusts nf_conntrack_udp_timeout_stream.

From I can now directly ping POD-IPs, but due to the problem in this issue, I can't make a TCP connection: The POD with address runs on . A similar setup as above can be used to verify a PR. Now, there is a simple work-around: use proxy-mode=ipvs The update with the rule never made it to the ipvs proxier (a rather common case :cry: ), but in this case it may save your day. The same as above with proxy-mode=ipvs; The server on 5001 responds with the POD name.
Yeah, Our production environment also solved this problem by switching to IPVS mode. In IPVS mode, there is no the DROP rule.
The sysctl solves the problem in , but not the problem in this issue. Those problems are kind of related, but rather distant.
The same affects outgoing traffic if packets are not masqueraded to the node-ip, as should be a common setup for ipv6, please see For Calico I think you can setup BGP for clusterCIDRs and route incoming packets to PODs to the correct K8s node, instead of using ECMP. This would solve the problem, and save the extra hop.
I met the same issue. KUBE-FORWARD drop my packages.
temptatively solution
The issue fixed by , I'll close it, you can feel free to open it if have any questions. /close
Closing this issue. <detailsIn response to : The issue fixed by , I'll close it, you can feel free to open it if have any questions. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <del> #!/usr/bin/env bash # Copyright 2016 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. # delete all bazel related files not in vendor/ find "${KUBE_ROOT}"  (  -name BUILD  -o -name BUILD.bazel  -o -name '*.bzl'  )  -not  (  -path "${KUBE_ROOT}"'/vendor*'  )  -delete # remove additional one-off bazel related files # NOTE: most of these will be pairs of symlinked location in "${KUBE_ROOT}/" # and the actual location in "${KUBE_ROOT}/build/root/" rm -f  "${KUBE_ROOT}/build/root/BUILD.root"  "${KUBE_ROOT}/WORKSPACE"  "${KUBE_ROOT}/build/root/WORKSPACE"  "${KUBE_ROOT}/.bazelrc"  "${KUBE_ROOT}/build/root/.bazelrc"  "${KUBE_ROOT}/.bazelversion"  "${KUBE_ROOT}/build/root/.bazelversion"  "${KUBE_ROOT}/.kazelcfg.json"  "${KUBE_ROOT}/build/root/.kazelcfg.json" </del>

This is intended to be removed once there are no active branches with bazel. 1.19 was still supported last I checked. We have some other cleanup pending the final turndown of bazel branches.
19 just had it's last patch release a week ago , these are removed now.


Question: } } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {


Question: } } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {


Question: } } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {


Question: } } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {


Question: } } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {


Question: } } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {


Question: } } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {


Question: } } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {


Question: } } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {


Question: } } <ins> if s.Config.Conntrack.TCPBeLiberal { if err := ct.SetTCPBeLiberal(1); err != nil { return err } } </ins> if s.Config.Conntrack.UDPTimeout.Duration > 0 { timeout := int(s.Config.Conntrack.UDPTimeout.Duration / time.Second) if err := ct.SetUDPTimeout(timeout); err != nil {


Question: fi if ! _out="$(diff -Naupr -x "BUILD" -x "OWNERS" -x "AUTHORS*" -x "CONTRIBUTORS*" vendor ${_kubetmp}/vendor)"; then <del> echo "Your vendored results are different:" echo "${_out}" echo "Godeps Verify failed." </del> <ins> echo "Your vendored results are different:" >&2 echo "${_out}" >&2 echo "Godeps Verify failed." >&2 </ins> echo "${_out}" > vendordiff.patch <del> echo "If you're seeing this locally, run the below command to fix your directories:" echo "patch -p0 < vendordiff.patch" echo "(The above output can be saved as godepdiff.patch if you're not running this locally)" echo "(The patch file should also be exported as a build artifact if run through CI)" </del> <ins> echo "If you're seeing this locally, run the below command to fix your directories:" >&2 echo "patch -p0 < vendordiff.patch" >&2 echo "(The above output can be saved as godepdiff.patch if you're not running this locally)" >&2 echo "(The patch file should also be exported as a build artifact if run through CI)" >&2 </ins> KEEP_TMP=true if [[ -f vendordiff.patch && -d "${ARTIFACTS_DIR:-}" ]]; then echo "Copying patch to artifacts.."

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: When running on my commits locally I got the following error: Affected PRs:
/sig testing
/cc
There's likely something up with your local environment. These jobs are running correctly in CI, although you need to look at the build log for output


Question: // TODO: move it to k8s.io/utils/net, this is the same as current AddIPOffset() // but using netip.Addr instead of net.IP func addOffsetAddress(address netip.Addr, offset uint64) (netip.Addr, error) { <del> addressBig := big.NewInt(0).SetBytes(address.AsSlice()) r := big.NewInt(0).Add(addressBig, big.NewInt(int64(offset))) addr, ok := netip.AddrFromSlice(r.Bytes()) </del> <ins> addressBytes := address.AsSlice() addressBig := big.NewInt(0).SetBytes(addressBytes) r := big.NewInt(0).Add(addressBig, big.NewInt(int64(offset))).Bytes() // r must be 4 or 16 bytes depending of the ip family // bigInt conversion to bytes will not take this into consideration // and drop the leading zeros, so we have to take this into account. lenDiff := len(addressBytes) - len(r) if lenDiff > 0 { r = append(make([]byte, lenDiff), r...) } else if lenDiff < 0 { return netip.Addr{}, fmt.Errorf("invalid address %v", r) } addr, ok := netip.AddrFromSlice(r) </ins> if !ok { <del> return netip.Addr{}, fmt.Errorf("invalid address %v", r.Bytes()) </del> <ins> return netip.Addr{}, fmt.Errorf("invalid address %v", r) </ins> } return addr, nil }

We ran into an issue with an old version of k8s 1.18 that we were running where subnet for cluster IP is given as . The control plane comes up just fine but then when you try to create a k8s service, it fails due to a broken behavior with the IP Allocator Since this is not really a supported version of the k8s, I went around digging up a bit more to see what is the regular behavior and found this to be very inconsistent across a few release branches. bash go test -timeout 30m -run ^TestAllocateNextFC$ -v === RUN TestAllocateNextFC ? --- PASS: TestAllocateNextFC (0.00s) PASS ok 0.566s go func TestAllocateNextFC(t *testing.T) { , cidr, err := netutils.ParseCIDRSloppy("fc::/112") if err != nil { t.Fatal(err) } // Need to be replaced with the suitable handler for that version of the branch r, err := newTestAllocator(cidr) if err != nil { t.Fatal(err) } defer r.Destroy() ip, err := r.AllocateNext() if err != nil { t.Fatalf("%s", err.Error()) } fmt.Println(ip.String()) } Add the above Test case under the IP allocator test case No response_ <detailsNA </details<detailsNA </details<detailsNA </details<detailsNA </details<detailsNA </details<detailsNA </details
/sig network /assign /kind bug
Thanks for helping me with this issue.
/triage accepted /assign


Question: "${DOCKER[@]}" rm -f -v "$1" >/dev/null 2>&1 || true } <ins> # Validate a release version # # Globals: #   None # Arguments: #   version # Returns: #   If version is a valid release version # Sets: #  BASH_REMATCH, so you can do something like: #    local -r version_major="${BASH_REMATCH[1]}" #    local -r version_minor="${BASH_REMATCH[2]}" #    local -r version_patch="${BASH_REMATCH[3]}" function kube::release::parse_and_validate_release_version() { local -r version_regex="^v(0|[1-9][0-9]*).(0|[1-9][0-9]*).(0|[1-9][0-9]*)$" local -r version="${1-}" [[ "${version}" =~ ${version_regex} ]] || { kube::log::error "Invalid release version: '${version}'" return 1 } } </ins> # --------------------------------------------------------------------------- # Building

(cc Despite doing very, very similar things, doesn't rely on . It should, or the logic should otherwise be consolidated. I'm going to maybe work on this to make easier.


Question: # Setup the container runtime on the node. It supports both # Docker and containerd. function Setup-ContainerRuntime { <ins> Install-Pigz </ins> if (${env:CONTAINER_RUNTIME} -eq "containerd") { Install_Containerd Configure_Containerd

<!-- Please only use this template for submitting enhancement requests --What would you like to be : pigz for GCE Windows Why is this needed: Pigz improves the image layer extraction performance for multi-core systems. Pigz is usually preinstalled on Linux. This enhancement is scoped specifically for the GKE Windows node scripts. For Context:
/assign me
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
GitHub didn't allow me to assign the following users: me. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign me Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows /assign jeremyje
s/GKE/GCE/? as in kube-up?
Replaced but I don't understand the kube-up part.
is how one creates a GCE cluster from this repo. Otherwise we'd be talking about something that isn't part of kubernetes.


Question: var outTimestamp time.Time var err error if tc.metric.singleObject != nil { <del> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetObjectMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.singleObject) </del> <ins> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetObjectMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.singleObject, selector) </ins> } else if tc.metric.selector != nil { if tc.metric.targetUtilization > 0 { <del> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetExternalMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.selector) </del> <ins> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetExternalMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.selector, selector) </ins> } else if tc.metric.perPodTargetUtilization > 0 { outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetExternalPerPodMetricReplicas(tc.currentReplicas, tc.metric.perPodTargetUtilization, tc.metric.name, testNamespace, tc.metric.selector) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: We are starting to use the HPA (horizontal pod autoscaling) for one of our components. What is special about this particular component is that the warm up period is very long (sometimes even more than 30 minutes). During that time, the pod is in Running state but its containers are in Not Ready state. This pod is of course not able to receive any traffic so the metric we use in the HPA is ignoring those pods that are not Ready. The equation used by HPA to calculate the new number of replicas is as follows Our problem is that the equation used by HPA to calculate the new number of replicas is including in the "current replica count" the pods that are "Not Ready" so the value for "new replica count" is not correct. Since this warm up is so long we end up by having a lot of more instances of our services of what it should be necessary. They eventually get deleted once gradually the metrics start going down but that will take a lot of time and a big waste of resources without mentioning that with a changing traffic this HPA will never stabilize What you expected to happen: I would like to have a flag somewhere where I tell HPA to don't include in the "current replica count" the pods that are in "Not Ready" state How to reproduce it (as minimally and precisely as possible): Have a component that takes a lot of time to start and being able to receive traffic. Then define a metric that will ignore the pods that are Not Ready. Finally create a metric that will scale that component based on the mentioned metric Anything else we need to know?: Environment: Kubernetes version 1.7 Cloud provider or hardware configuration: Azure
/sig autoscaling
Hi - thanks for reaching out about this! It looks like this section of the , handles when pods aren't ready. I think you this behavior in , so I'm interested in your thoughts on whats best here. If we do end up adding a flag for this, I'd be happy to write the code :) Also, purely for my own curiosity, I'm curious what type of pod is taking 30 minutes to become ready. If your traffic pattern varies at a rate faster than 30 minutes, it may be difficult for auto-scaling to be helpful.
can you also share a bit more about your HPA?


Question: var outTimestamp time.Time var err error if tc.metric.singleObject != nil { <del> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetObjectMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.singleObject) </del> <ins> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetObjectMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.singleObject, selector) </ins> } else if tc.metric.selector != nil { if tc.metric.targetUtilization > 0 { <del> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetExternalMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.selector) </del> <ins> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetExternalMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.selector, selector) </ins> } else if tc.metric.perPodTargetUtilization > 0 { outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetExternalPerPodMetricReplicas(tc.currentReplicas, tc.metric.perPodTargetUtilization, tc.metric.name, testNamespace, tc.metric.selector) }

What kind of metric type is it? Based on the description I'm suspecting that it is an Object metric, am I right?
So answer about why our process takes that much time to start. Basically this is a server that in order to do what he needs to do, he needs to load a lot of data in memory (several GBs) and then perform some warm up that are pretty intensive. The idea is that he doesn't need to grab (almost) anything from outside to attend a request so it's latency is extremely low (in the order of millisecond) By the way. Because of this we have another problem which is that there is not an specific flag per HPA object that we could tell k8s to don't try to scale up again passed some time so the new pods have enough time to get ready before the next scale happens. I know that there is a cluster wide flag but we will need to have a different one by resource that we scale. Now to answer You pointed another interesting topic. The metric we use is % Utilization. We are able to calculate the level of utilization of our pods and we want to keep this metric in an specific value. About the flag. I suggested the flag so it doesn't change the existing behavior by default but I honestly think that the default behavior should be to don't include the NonReady pods as part of the calculation
do you think we can add this change to 1.10? it's critical bug/feature for us and we are happy to help with code/testing.
It's really late for 1.10, especially since we don't have any code here, and I'd like to know more about what's going on here, and especially see if it reproduces on Kubernetes 1.9 (we've had HPA controller bugs in the past). FWIW, currently unready pods are treated as consuming 0% of their utilization on scale up, and it was intended that this should prevent such issues. That is, the formula is actually: If the pods are marked as not ready, I find it strange that they would be affecting the scaling that much, unless your other pods are under real heavy load (i.e. I would assume that the unready pods would skew the scaling back down so that the number of pods just stayed constant).
I believe there is different code branch here.


Question: var outTimestamp time.Time var err error if tc.metric.singleObject != nil { <del> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetObjectMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.singleObject) </del> <ins> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetObjectMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.singleObject, selector) </ins> } else if tc.metric.selector != nil { if tc.metric.targetUtilization > 0 { <del> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetExternalMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.selector) </del> <ins> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetExternalMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.selector, selector) </ins> } else if tc.metric.perPodTargetUtilization > 0 { outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetExternalPerPodMetricReplicas(tc.currentReplicas, tc.metric.perPodTargetUtilization, tc.metric.name, testNamespace, tc.metric.selector) }

metric definition { "type": "Object", "object": { "target": { "kind": "Service", "name": "prometheus-exporter" }, "metricName": "PercentUtilization", "targetValue": "20" } } which should handle as and doesn't check Pod state.
I think proper handling of unready pods is a difficult problem that will require significant work. I suspect we may find out we need to start distinguishing pods that are booting up from pods that are 'broken' and it took us quite a while to solve analogous problem with nodes in Cluster Autoscaler (to be clear: I don't know if we will have to do that, that's just my intuition for now). That being said I agree the formula in GetObjectMetricReplicas will generally lead to scaling-up too much if the pod takes longer to initialize then backoff period in HPA (3 minutes). Assume it takes pod 4 minutes to boot up and we're currently running 2 pods. Metric value is 20 and target is 10. In this case: scales-up to 4 pods (20/10 utilization 2 pods). are starting up, they're not doing anything yet. 3 minutes later HPA scales-up to 8 pods (20/10 utilization 4 pods). If we only counted ready pods the scale-up in 3 would not occur, because the desired replicas would still be 4 (20/10 * 2). My intuition would be that this is the expected behavior of HPA. Also if there is no space in the cluster we will just keep scaling-up until we hit max replicas. The only case where current behavior would be more correct than filtering out unready pods in this formula is if someone uses a metric that is calculated based on total number of replicas. Do we advertise that as the right way to use HPA? If we add to Object metrics in 1.11 this usage pattern will stop making sense. To sum up - I think we will need to change the formula to only count ready pods at some point. I can see how current behavior could be a problem in any use-case except the one described in paragraph above. I'm inclined to agree with and provide some way to change the behavior in 1.10 (a flag or an annotation?), treating it as a bug fix.


Question: var outTimestamp time.Time var err error if tc.metric.singleObject != nil { <del> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetObjectMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.singleObject) </del> <ins> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetObjectMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.singleObject, selector) </ins> } else if tc.metric.selector != nil { if tc.metric.targetUtilization > 0 { <del> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetExternalMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.selector) </del> <ins> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetExternalMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.selector, selector) </ins> } else if tc.metric.perPodTargetUtilization > 0 { outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetExternalPerPodMetricReplicas(tc.currentReplicas, tc.metric.perPodTargetUtilization, tc.metric.name, testNamespace, tc.metric.selector) }

WDYT
Ah, yeah, I missed that it was object metrics before. I'm not entirely convinced we need a flag here -- it might just make sense to bring Object metrics in line with all the other metrics types. On the other hand, there's a subtlety here -- the other metric types don't quite ignore unready pods -- they just ignore the metrics from unready pods, which causes a heavy bias in the direction of not scaling, but an overwhelming spike can still cause scaling. That behavior is not as achievable using object metrics. I'm inclined to just ignore that subtlety, though, and say that object metrics should ignore unready pods entirely.
I think this is what we should do long term. I was worried about breaking someone who relies on targetValue + averaged metric to get effect similar to targetAverageValue and though about providing something for transitional period, but if you're ok with it I'd rather just change it as you say.
I think it's safe to change as-is -- that feature's still in beta, and it's not an API change, so it shouldn't just hard-break anyone. Plus, we've change stuff like this before (when we did the initial resource metrics change).
As said, we are more than ready to help with the fix/test if required. This since this is a major blocking for us in the implementation of the HPA in our side
is anyone actively working on implementing this? If not, I'm happy to. I think I have a fairly clear picture of the change we need to make and will likely have time early this week to get it done.
(Also, from the conversation above, to we want to tag this as a release blocker for 1.10)?
I probably won't be able to work on this before the end of this week. Happy to do a review if you jump in and implement it.
I don't think this qualifies as a release blocking bug (but I'm willing to have someone try and convince me otherwise ;-)). There doesn't seem to be any behavior here that's explicitly broken -- it's just sub-optimal in some use cases. I'd definitely like to address this in 1.11, though.
it's not release blocker, but to share our experience ;).


Question: var outTimestamp time.Time var err error if tc.metric.singleObject != nil { <del> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetObjectMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.singleObject) </del> <ins> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetObjectMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.singleObject, selector) </ins> } else if tc.metric.selector != nil { if tc.metric.targetUtilization > 0 { <del> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetExternalMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.selector) </del> <ins> outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetExternalMetricReplicas(tc.currentReplicas, tc.metric.targetUtilization, tc.metric.name, testNamespace, tc.metric.selector, selector) </ins> } else if tc.metric.perPodTargetUtilization > 0 { outReplicas, outUtilization, outTimestamp, err = replicaCalc.GetExternalPerPodMetricReplicas(tc.currentReplicas, tc.metric.perPodTargetUtilization, tc.metric.name, testNamespace, tc.metric.selector) }

without this feature we cannot use HPA as is and we need to figure out how to build/run fork (which can be challenging because everything inside controller-manager). also we are planning to upgrade to 1.10 quickly, but not sure for 1.11 (e.g. we are two versions behind now).
Sounds like a plan :) Will try and get this done tomorrow morning!
- I submitted a pr for this - would love your eyes on it so you can see if you think it meets your use case :)
/assign
GitHub didn't allow me to assign the following users: mattjmcnaughton. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I checked the code and it seems good to me. About the comment about scaling down if the readyPods<CurrentPods, we should be fine since our metric is already not including the pods that are not ready.
Is there any advance in this ticket? I'm wondering if this made it to k8s 1.10 or if we need to wait for k8s 1.11?
The PR is lgtm-ed and it will merge once the code freeze is lifted. Meaning it will go to 1.11. To get it into 1.10 you would need to get approval for milestone from (sig-autoscaling lead).


Question: BuildImageRegistry:      "k8s.gcr.io/build-image", InvalidRegistry:         "invalid.com/invalid", GcEtcdRegistry:          "k8s.gcr.io", <del> GcHttpdRegistry:         "k8s.gcr.io/e2e-test-images", </del> GcRegistry:              "k8s.gcr.io", SigStorageRegistry:      "k8s.gcr.io/sig-storage", GcrReleaseRegistry:      "gcr.io/gke-release",

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --/sig windows /cc After was merged I think.
I agree that looks like the issue. It started failing between the two commits: on this run:
/triage accepted
/assign


Question: BuildImageRegistry:      "k8s.gcr.io/build-image", InvalidRegistry:         "invalid.com/invalid", GcEtcdRegistry:          "k8s.gcr.io", <del> GcHttpdRegistry:         "k8s.gcr.io/e2e-test-images", </del> GcRegistry:              "k8s.gcr.io", SigStorageRegistry:      "k8s.gcr.io/sig-storage", GcrReleaseRegistry:      "gcr.io/gke-release",

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Kubernetes e2e suite: [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should have a working scale subresource [Conformance] Kubernetes e2e suite: [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform canary updates and phased rolling updates of template modifications [Conformance] and a few others.
/sig windows
/triage accepted
/cc should be resolved by
Currently blocked because there are some issues pushing the httpd test images to . Slack thread:
Latest thread:
The job failing is:
is where the fixes for publishing this image is being tracked


Question: } } <del> // Case 4: Schedule Pod with NetworkUnavailable toleration. </del> <ins> // Case 4: Schedule Pod with NetworkUnavailable and NotReady toleration. </ins> networkDaemonPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "network-daemon-pod",

Is this a BUG REPORT or FEATURE REQUEST?: What happened: admission control plugin in apiserver. a deployment with 1 replica. The pod is successfully launched. By default the admission control plugin will add following toleration in the pod. the node where the above pod is running. of image locality policy and above toleration in the pod, scheduler still choose the node to launch the pod. The pod is in state and failed to create. What you expected to happen: The pod should be scheduled to healthy node and created successfully. How to reproduce it (as minimally and precisely as possible): See above steps. Anything else we need to know?: Environment: Kubernetes version (use ): v1.10 Cloud provider or hardware configuration: openstack OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-121-generic Install tools: N/A Others:
/sig scheduler
/sig scheduling /sig node
I think that we should avoid scheduling pods to nodes which is not ready even it can tolerate and , comments
As the pod already claimed to tolerate NotReady, this is an expected behavior. P.S. If the eviction dose not happen after 5m, that should be the bug. Please reopen the issue and modify the description. This is what tolerations is designed for :) And there are many use cases for pod to be scheduled to NotReady nodes (mostly during cluster deployment).
I think maybe you can configure effect to avoid scheduling to those nodes.
scheduler still choose the NotReady node to launch the pod For running, the toleration is used for graceful termination by ; if is enabled, it's better to also taint node with for .
Wondering how the pod can start to run if the Node is not ready at deployment time? Can you specify a use case here?
It won't need to start, system pod pending will be expected :) The point is, freedom for users to decide what tolerations they want matters here. I don't think it's reasonable to combine with in this case.
Perhaps, add to default toleration of admission control plugin could eliminate the confusion.
We should align the behavior when enabled or disabled : if is disabled, will not schedule pod to not-ready node. if is enabled, we should also taint node with .


Question: } } <del> // Case 4: Schedule Pod with NetworkUnavailable toleration. </del> <ins> // Case 4: Schedule Pod with NetworkUnavailable and NotReady toleration. </ins> networkDaemonPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "network-daemon-pod",

with timeout is used for graceful termination of 'running pods' for feature. We should not re-schedule pod to not-ready unless tolerating it explicitly.
reopen this for more discussion :)
Let me extracted the mixed problems in this issue :-) First one: These two default tolerations attached to the pod by means: if the node is tainted as or , wait for 300s before the pod is evicted from the node. While according to the issue description, is working as expected (pods are not evicted immediately). This part should be closed. Second one: The discussion raised is actually about taint during eviction: if TaintNodesByCondition is enabled, we should also taint node with NotReady:NoSchedule. My thoughts is: it's reasonable to taint node with as well for consistency consideration (but I'm hesitate if only for eliminating confusion as pods are expected to pending on NotReady nodes) I think it should be clearer now.
I open a PR If TaintNodesByCondition is enable, taint node with NotReady:NoSchedule and Unreachable:NoSchedule.
, we did not catch 1.11 release, so this issue will be handled in 1.12 :)
in 1.9.7enable the TaintNodesByCondition and taint nodee.g. node01 with NotReady:NoSchedulethen pod also can not be scheduled to node01 even when node01 is healthy. What you expected to happen: The pod should be scheduled to node01 when it is healthy node and not be scheduled to node01 when it is not ready. Thanks
How did you do that? by kubectl? The fix is not ready until 1.12.
YesI tried "kubectl taint nodes node01 ",and also tried "kubectl taint nodes node01 "
this feature did not handle taint manually
Ok, thanks a lot! So I can not solve the problem manually now and what I should do is just waiting for the version 1.12, right?


Question: assert := assert.New(t) <del> plugMgr := newInitializedVolumePlugMgr() </del> <ins> plugMgr, rootDir := newInitializedVolumePlugMgr(t) if rootDir != "" { defer os.RemoveAll(rootDir) } </ins> plug, err := plugMgr.FindPluginByName(flockerPluginName) assert.NoError(err)

Unit tests on our CI have apparently been failing for 2 weeks and no one noticed (oops!)... The PR that seems to have caused the failures is . The failing test is - . Full Test Log: I'm not sure why this is failing on our CI and not on shippable. Has anyone else seen something similar? That test only seems to fail on the second run through with . Our unit tests run in a container built specifically for testing k8s-mesos: The Dockerfile for the env: Is anyone else able to repro with the above env or their own?
/cc
that test was around a week ago, it's doesn't seem possible to me that it was failing since 2 weeks ago. Anyway, we have few duplications of this issue: this one And the PR where is working on: It's a little bit difficult to me to follow all the conversations in all the differente places about the same bug, could it be possible to unificate some? Since you provided the environment I will try and take a look to reproduce it locally. Thanks!
I think I've got a working patch in
Closing in favor of &


Question: assert := assert.New(t) <del> plugMgr := newInitializedVolumePlugMgr() </del> <ins> plugMgr, rootDir := newInitializedVolumePlugMgr(t) if rootDir != "" { defer os.RemoveAll(rootDir) } </ins> plug, err := plugMgr.FindPluginByName(flockerPluginName) assert.NoError(err)

reports the following failure after syncing with latest code: --- FAIL: TestSetUpAtInternal (0.00s) Error Trace: Error: Not equal: "expected-to-be-set-properly" (expected) != "" (actual) FAIL FAIL 0.067s cc/
CC
Taking a look. What do you mean with failed sometimes? Is it a random failure?
Can you please provide more information. This test seems to be consistently green in our CI runs.
The test is failing on this assertion: but as you can see on the previous line I am checking that there was no error on that call. So the function is returning for the error checking but it's not setting . Only place where it can happen is here: Is the not been clean between runs?
When I said "the only place where it can happen" it was because the other s are being returned after a check. I think that there is no other way to get there.
PS: I run the test suite 100 times and I didn't get any error: (That branch is based on the branch)
can you test to see if the patch in fixes things for you?
See for reproducible test environment and failure logs.
Based on the solution in it sounds like this test creates files the first time through which causes the tests to fail on the second time through.


Question: recorder := recorderWithFakeClock(api.EventSource{Component: "eventTest"}, eventBroadcaster, clock) for index, item := range table { clock.Step(1 * time.Second) <del> logWatcher1 := eventBroadcaster.StartLogging(t.Logf) // Prove that it is useful logWatcher2 := eventBroadcaster.StartLogging(func(formatter string, args ...interface{}) { </del> <ins> logWatcher := eventBroadcaster.StartLogging(func(formatter string, args ...interface{}) { </ins> if e, a := item.expectLog, fmt.Sprintf(formatter, args...); e != a { t.Errorf("Expected '%v', got '%v'", e, a) }

I wasn't able to find automatically create issue for this, but I may not know what to search for. Is our bot creating issues for unit test failures? - this failure looks to me as something in client testing libraries.
If it's exposed in JUnit output, then the bot should be making issues for it.
It's not exposed in junit due to :(
Lots of failures in , probably all this bug.
Yeah this seems to be happening a lot. Raising priority.
I thought was reverted but it's not. I'll revert it.
golang 1.6 doesn't fix the race:
Sent .


Question: glog.Errorf("Failed to check whether disk is attached. err: %s", err) return false, err } <del> for _, controllerType := range supportedSCSIControllerType { controllerkey, _ := getControllerKey(controllerType, vmdevices) if controllerkey == virtualDiskControllerKey { return true, nil } } return false, ErrNonSupportedControllerType </del> <ins> return true, nil </ins> } // Returns the object key that denotes the controller object to which vmdk is attached.

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): v1.4.x v1.5.x v1.6.x master Environment: vSphere based install with vSphere Cloud Provider enabled. Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): NA Kernel (e.g. ):NA Install tools:kubernetes-anywhere Others: What happened: vSphere Cloud Provider configures additional SCSI adapters to scale the number of volumes that can be attached to a VM (1 adapter = 16 disks). The code within the cloud provider does not handle this correctly. Additional disks attach successfully to the VM but when k8s verifies the attach status it fails leading to k8s retrying the attach that never succeeds. In addition to this when the SCSI adapter is created for the first time the code does not use it to attach the disk leading to a failed attach disk request to the cloud provider. What you expected to happen: Additional disks should be to the VM upto the max number supported. How to reproduce it (as minimally and precisely as possible): Create a single node cluster and create a statefulset with 16 replicas or have a pod with more than 16 disks. Anything else we need to know:
A fix for this is coming up..
cc
Huh, I wonder if something changed between 1.3 and newer?


Question: glog.Errorf("Failed to check whether disk is attached. err: %s", err) return false, err } <del> for _, controllerType := range supportedSCSIControllerType { controllerkey, _ := getControllerKey(controllerType, vmdevices) if controllerkey == virtualDiskControllerKey { return true, nil } } return false, ErrNonSupportedControllerType </del> <ins> return true, nil </ins> } // Returns the object key that denotes the controller object to which vmdk is attached.

We had tested up to 40-something volumes on 1.3. There was a fun discussion internally because it would always fail at SCSI ID 7 and I had to explain SCSI to some younger engineers :)


Question: if !found { t.Errorf("Failed to find node %v in schedulerinternalcache.", node.Name) } <del> if cache.nodeTree.NumNodes != 1 || cache.nodeTree.Next() != node.Name { </del> <ins> if cache.nodeTree.NumNodes() != 1 || cache.nodeTree.Next() != node.Name { </ins> t.Errorf("cache.nodeTree is not updated correctly after adding node: %v", node.Name) }

Description: , part of the is exporting a field which allows to modify its value externally. This also means that this field is not thread-safe. Solution: Expose a method that returns this value in a thread-safe way and do not export the field. /kind bug /sig scheduling


Question: "create_clusterrole_test.go", "create_clusterrolebinding_test.go", "create_configmap_test.go", <ins> "create_cronjob_test.go", </ins> "create_deployment_test.go", "create_job_test.go", "create_namespace_test.go",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Why is this needed: Because of is not recommended to create cronjob (with flags) anymore. But there is no imperative command available for to create cronjob. /sig CLI <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/assign
GitHub didn't allow me to assign the following users: Pingan2017. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i wanna work on this. give me several days
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale