Question: } // getNodeSecurityGroupIDForLB lists node-security-groups for specific nodes <del> func getNodeSecurityGroupIDForLB(compute *gophercloud.ServiceClient, nodes []*v1.Node) ([]string, error) { nodeSecurityGroupIDs := sets.NewString() </del> <ins> func getNodeSecurityGroupIDForLB(compute *gophercloud.ServiceClient, network *gophercloud.ServiceClient, nodes []*v1.Node) ([]string, error) { secGroupNames := sets.NewString() </ins> for _, node := range nodes { nodeName := types.NodeName(node.Name) srv, err := getServerByName(compute, nodeName) if err != nil { <del> return nodeSecurityGroupIDs.List(), err </del> <ins> return []string{}, err </ins> } // use the first node-security-groups

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kubelet fails when Kubernetes master gets started using cloud-provider openstack und kubeadm init command. says: What you expected to happen: kubelet starts and manages the loadbalancer and its security group. How to reproduce it (as minimally and precisely as possible): Setup an instance on Openstack. Place the following cloud-config in And : As well as : Run: For further information: +-----------------+
--------------------------+ +-----------------+
--------------------------+ +-----------------+
--------------------------+ Environment: Kubernetes version (use ): 1.9.1 Cloud provider or hardware configuration: Openstack OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS Kernel (e.g. ): x86_64 Install tools: kubeadm
/sig cloud-provider
Looks like auto-probing for for (s), please see [1] and [2]. Apologies, the deprecation should have been handled better. [1] [2]
If I understand you correctly, this cloud-config would be enough: If I try this, no Loadbalancer gets created at all and the service which requests type LoadBalancer says: The service I use there is the nginx helm chart ().
Can you please comment? /assign
/sig openstack
From that, '' is your service name and '-202f-4d44-a538-' is your subnet id, can you ensure '-202f-4d44-a538-' exist? Is it "-c60c-48cb-a146-"? I suspect that failed to autoprobe subnet. Can you give me more logs from kube controller manager
Actually, -202f-4d44-a538- is a network, not a subnet. The subnet has the ID -c1d5-4cc1-8b98-
And here the relevant logs from kube-controller-manager,first: And then it keeps repeating:
Oh, I see. Oops, should get 'sub network id', not 'network id'. I will fix it soon.
I tried the fix in 1.9.2, but it does not work. I tried multiple setting combinations of providing and/or/neither , but none works. The Service does not get a LoadBalancer IP assigned.


Question: } // getNodeSecurityGroupIDForLB lists node-security-groups for specific nodes <del> func getNodeSecurityGroupIDForLB(compute *gophercloud.ServiceClient, nodes []*v1.Node) ([]string, error) { nodeSecurityGroupIDs := sets.NewString() </del> <ins> func getNodeSecurityGroupIDForLB(compute *gophercloud.ServiceClient, network *gophercloud.ServiceClient, nodes []*v1.Node) ([]string, error) { secGroupNames := sets.NewString() </ins> for _, node := range nodes { nodeName := types.NodeName(node.Name) srv, err := getServerByName(compute, nodeName) if err != nil { <del> return nodeSecurityGroupIDs.List(), err </del> <ins> return []string{}, err </ins> } // use the first node-security-groups

Here are the logs and settings for the config version neither providing nor , since it seemed that's the way you recommended to me: cat /etc/kubernetes/pki/cloud-config kubectl get nodes journalctl -u kubelet.service kubectl describe svc ingress-nginx-ingress-controller Can you please re-open the issue? Do you need any other information?
Is it ok when you disable manage-security-group(manage-security-group=false)?
And please review .
/reopen
Yes, when I disable "manage-security-group" it assigns a LoadBalancer correctly. It also works, as soon as I run . I tried to check , but since I don't know go, I don't really understand what you are doing there and how it relates to my issue.
I check gophercloud.ErrResourceNotFound is not a interface, so the '' will return error and excute "" which is 'error occurred finding security group: ...'.
Okay, so you "just" print a better error message, right? But my question is, why does the error occur at all? In my understanding the kubelet should create a security-group on itself, if configured manage-security-groups=true and do not provide a security-group. However I also tried to define a valid security group id on my own, but in that case I again find these logs in kubelet (same as described first):
Okay, so you "just" print a better error message, right? But my question is, why does the error occur at all? In my understanding the kubelet should create a security-group on itself, if configured manage-security-groups=true and do not provide a security-group. No, not 'just print a better error message', openstack cloud provider can ensure security group for LB. Firstly, openstack cloud provider check whether security group does not exist, if not exist, it will create security group for LB. The "gophercloud.ErrResourceNotFound" means that security group not exist.
Now OpenStack cloud provider has not "node-security-group" variable. You should not specify it.
Please check this note:
Well, to me it does not look like the removal of any resources is the problem, rather that they do not get created at all. I can offer you a direct debugging session using Google Hangout or so, maybe that way it is easier to detect the real problem?
Have you see this log?


Question: } // getNodeSecurityGroupIDForLB lists node-security-groups for specific nodes <del> func getNodeSecurityGroupIDForLB(compute *gophercloud.ServiceClient, nodes []*v1.Node) ([]string, error) { nodeSecurityGroupIDs := sets.NewString() </del> <ins> func getNodeSecurityGroupIDForLB(compute *gophercloud.ServiceClient, network *gophercloud.ServiceClient, nodes []*v1.Node) ([]string, error) { secGroupNames := sets.NewString() </ins> for _, node := range nodes { nodeName := types.NodeName(node.Name) srv, err := getServerByName(compute, nodeName) if err != nil { <del> return nodeSecurityGroupIDs.List(), err </del> <ins> return []string{}, err </ins> } // use the first node-security-groups

It means the security group we need to create does not exist, we should create it, not raise error, right? And I get 'non-interface type ErrResourceNotFound on left' error, 'err.(*gophercloud.ErrResourceNotFound)' is incorrect.
Yes, exactly. I also tried to fix the error manually, so first I created the sg-group . After that kubelet complained about empty pool-targets. After fixing that, it complained about invalid endpoint which pointed to the nginx server, don't know how to fix that one yet. I just want to point out: All that errors only occur if I activate manageSecurityGroups, otherwise the kubelet is able to assign a LoadBalancer to the IngressController.
Hi, how is going? I've deployed k8s 1.9.5 on OpenStack with kubespray and I'm struggling with this issue. If I do not set the LBAAS is created but it's not assigned to any Security Group. So, the service is not reachable from the outside world. I can fix it by hand creating the Security Group and assign it to the VIP. If I do set kubelet fails to start. According to it's a valid parameter... (?) And finally, if I do as in his last comment, the LBAAS remains in a PENDING state forever... Well, I wonder if there is any workaround to deal with this... I presume you're working on the fix, aren't you? Any information will be appreciate. Thanks
I'm hitting the same issue as yours with openstack loadbalancer with 1.9.4. Anyone know if fixes this issue? If so, when will it be part of a release?
Any updates on this?
I just tried to use this feature in kubernetes 1.10.2 and it still seems to be broken. The security groups do not get created and the service creation is stuck at pending without an ip address. I used this configuration and i also tried all the alternatives listed here. Either nothing gets created or the controller-manager crashes if i add node-security-group.
It's an issue with as well. The behaviour is as described by
Bump
fixes this for us.
So, If I do not enable manage-security-groups, This should work?


Question: } // getNodeSecurityGroupIDForLB lists node-security-groups for specific nodes <del> func getNodeSecurityGroupIDForLB(compute *gophercloud.ServiceClient, nodes []*v1.Node) ([]string, error) { nodeSecurityGroupIDs := sets.NewString() </del> <ins> func getNodeSecurityGroupIDForLB(compute *gophercloud.ServiceClient, network *gophercloud.ServiceClient, nodes []*v1.Node) ([]string, error) { secGroupNames := sets.NewString() </ins> for _, node := range nodes { nodeName := types.NodeName(node.Name) srv, err := getServerByName(compute, nodeName) if err != nil { <del> return nodeSecurityGroupIDs.List(), err </del> <ins> return []string{}, err </ins> } // use the first node-security-groups

I am getting the following error: Error creating load balancer (will retry): failed to ensure load balancer for service default/lbs-test: error creating LB floatingip {FloatingNetworkID:YYYY FloatingIP: PortID:XXXX FixedIP: TenantID:}: Resource not found
Yes. Don't specify and at all in config. The LoadBalancer(s) will by default apply the security rules in the security-group .
You need to specify to automatically add / alter security groups. I get the following error after I've upgraded to 1.9.10:
I ran into the same issue. It seems to be related to how k8's openstack driver deals with adding floating ips to the LB. For now I set the lb to internal in the service metadata then manually associate a floating ip . This could also be a queens magnum issue not providing all required cloud config info. I am not sure yet.


Question: expectedError: false, }, { <ins> desc: "valid: 1 search path with trailing period", dnsConfig: &core.PodDNSConfig{ Searches: []string{"custom."}, }, expectedError: false, }, { </ins> desc: "valid: 3 nameservers and 6 search paths", dnsConfig: &core.PodDNSConfig{ Nameservers: []string{"127.0.0.1", "10.0.0.10", "8.8.8.8"}, <del> Searches:    []string{"custom", "mydomain.com", "local", "cluster.local", "svc.cluster.local", "default.svc.cluster.local"}, </del> <ins> Searches:    []string{"custom", "mydomain.com", "local", "cluster.local", "svc.cluster.local", "default.svc.cluster.local."}, </ins> }, expectedError: false, },

What happened: When I create a pod with specified dnsConfig, the validation says my spec is invalid. And the error is like below: What you expected to happen: The spec should be accepted as it is totally fine to have trailing period. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.3
Despite it's possible to have domains with trailing periods in the field of the , I agree with about this topic: FQDN and domains are different things. Maybe enforcing this in kubernetes is not a bad idea since the resolver seems to ignore the trailing dots.
let’s say you have name server A, and the serach domain at client has , when you try to resolve name abc, the client will send to name server A. Now what happened if nameserver A can not resolve , it is going to try to append its own searches, for example, name server has its searches as , then it tries to resolve (Depends on the ndots options, this may be allowed). Instead, if you have the trailing period in searches, the second try won’t happen.
what I'm trying to point out is that my understanding is that the and doesn't matter if you add a trailing dot or not to the domain used in the field, thus server A in your example is always going to receive a query to an FQDN and not going to do a recursive search. If you do a test in your system you can capture the traffic and check that the query is always the same with or without a traling dot. Anyway, adding domains with dots is allowed so you are right, only wanted to go deeper into this topic and shared my thoughts :)


Question: # Defaults # local input_pkgs=() <del> while read -r file; do dir="$(dirname "${file}")" </del> <ins> while read -r dir; do </ins> pkg="$(cd "${dir}" && GO111MODULE=on go list -find .)" input_pkgs+=("${pkg}") done < <( <del> ( kube::codegen::internal::git_grep -l  </del> <ins> ( kube::codegen::internal::git_grep -l --null  </ins> -e '+k8s:defaulter-gen='  ":(glob)${root}"/'**/*.go'  || true  <del> ) | LC_ALL=C sort -u </del> <ins> ) | xargs -0 -n1 dirname  | LC_ALL=C sort -u </ins> ) if [ "${#input_pkgs[@]}" != 0 ]; then

function in script has following issues: function only supports file but we have multiple types file with name as in . Hence this function is not working in our case. For the mitigation there is already an open . I have asked for including our changes as well. We have multiple files under a for example and . Because of this for appears twice and it generates duplicate informers We only need unique elements before passing it to . I followed this approach for unique elements: We have packages under . Earlier we were generating client only for and . But with the new I am not able to do that. I am using the following call to function: It results in following flags to As there is no directory as , it doesnot generate anything. Is it still possible to call on specific folder as it was done with earlier scripts? One mitigation I can think of is to call function on whole and then remove the marker from the types for which genclient is not needed. But the issue is we won't be able to pass the flag for each pkg. function in script is not working as expected.
/sig api-machinery
/cc
/triage accepted
Hi . I knew that there were all sorts of clients out there doing all sorts of things with these scripts, so I am not surprised to find you. :) Do you have a PR which attempts to use kube_codegen, so I can repro your issues directly?
I think you have one too many in there Aside from that, the real problem here is that these old codegen tools make some deeeeeeep assumptions about the ability to string-join various arguments and get the right results. It's awful. I was trying to make it simpler in this script, which I did for all of the in-tree consumers, but out of tree people apparently don't always follow the same conventions. For example, we generate clients for every directory that requests it. For a reason I do not know, you have files like which say but you don't actually want a client? If you specify then it t
So this code expects that you generate an omnibus client. What if I add a flag, like which you can use to call it for each API to make distinct clients? In truth the codegen tools need to be fixed, but that has SO MUCH ripple I just can't right now.


Question: # Defaults # local input_pkgs=() <del> while read -r file; do dir="$(dirname "${file}")" </del> <ins> while read -r dir; do </ins> pkg="$(cd "${dir}" && GO111MODULE=on go list -find .)" input_pkgs+=("${pkg}") done < <( <del> ( kube::codegen::internal::git_grep -l  </del> <ins> ( kube::codegen::internal::git_grep -l --null  </ins> -e '+k8s:defaulter-gen='  ":(glob)${root}"/'**/*.go'  || true  <del> ) | LC_ALL=C sort -u </del> <ins> ) | xargs -0 -n1 dirname  | LC_ALL=C sort -u </ins> ) if [ "${#input_pkgs[@]}" != 0 ]; then

As the codegen was breaking us at several step, hence didn't prepared the PR.
Thanks for the PR. I have one more concern, is it possible to call and client-gen on the same pkg? With the current script, that is not possible because of the line It removes all the generated files first and then calls the . Hence the later clientset gen will remove the earlier generated clients.
Why are you generating a client for an internal version that isn't served?
My mistake. We need to stop using clients for internal versions.


Question: "sort" "strconv" "strings" <ins> "sync" "sync/atomic" </ins> "testing" "time"

pull-kubernetes-unit sometimes fails with data race. TestDirty I think the test was 3 months ago with the flaking state. There are two issues. First, updating and reading a bool-type variable is not thread-safe. and there is also another problem checking the state before completely syncing it. No response /sig api-machinery
/assign
/triage accepted


Question: return fmt.Errorf("failed to generate self-signed certificate for loopback connection: %v", err) } <ins> // Write to the front of SNICerts so that this overrides any other certs with the same name (*secureServingInfo).SNICerts = append([]dynamiccertificates.SNICertKeyContentProvider{certProvider}, (*secureServingInfo).SNICerts...) </ins> secureLoopbackClientConfig, err := (*secureServingInfo).NewLoopbackClientConfig(uuid.New().String(), certPem) switch { // if we failed and there's no fallback loopback client config, we need to fail case err != nil && *loopbackClientConfig == nil: <ins> (*secureServingInfo).SNICerts = (*secureServingInfo).SNICerts[1:] </ins> return err // if we failed, but we already have a fallback loopback client config (usually insecure), allow it

The code block does these actions: the self-signed cert for loopback connection the loopbackClientConfig the self-signed cert to the If the main cert () is empty, the loopbackClientConfig will not be created due to this check, loopbackClientConfig shouldn't be left empty, because we have created the SNICert, it's just that we haven't executed action 3 to add the SNICert to the list. Switching the order of action 2 and action 3 will make the loopbackClientConfig successfully created. /kind bug /sig auth api-machinery


Question: "pod_workers_test.go", "reason_cache_test.go", "runonce_test.go", <ins> "time_cache_test.go", </ins> ], embed = [":go_default_library"], deps = [

These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions


Question: } } } <ins> func TestCopyToPod(t *testing.T) { tf := cmdtesting.NewTestFactory() tf.Namespace = "test" ns := legacyscheme.Codecs codec := legacyscheme.Codecs.LegacyCodec(scheme.Versions...) tf.Client = &fake.RESTClient{ GroupVersion:         schema.GroupVersion{Group: "", Version: "v1"}, NegotiatedSerializer: ns, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { responsePod := &v1.Pod{} return &http.Response{StatusCode: http.StatusNotFound, Header: defaultHeader(), Body: ioutil.NopCloser(bytes.NewReader([]byte(runtime.EncodeOrDie(codec, responsePod))))}, nil }), } tf.ClientConfigVal = defaultClientConfig() buf := bytes.NewBuffer([]byte{}) errBuf := bytes.NewBuffer([]byte{}) cmd := NewCmdCp(tf, buf, errBuf) srcFile, err := ioutil.TempDir("", "test") if err != nil { t.Errorf("unexpected error: %v", err) t.FailNow() } defer os.RemoveAll(srcFile) tests := map[string]struct { dest        string expectedErr bool }{ "copy to directory": { dest:        "/tmp/", expectedErr: false, }, "copy to root": { dest:        "/", expectedErr: false, }, "copy to empty file name": { dest:        "", expectedErr: true, }, } for name, test := range tests { src := fileSpec{ File: srcFile, } dest := fileSpec{ PodNamespace: "pod-ns", PodName:      "pod-name", File:         test.dest, } t.Run(name, func(t *testing.T) { err = copyToPod(tf, cmd, buf, errBuf, src, dest) //If error is NotFound error , it indicates that the //request has been sent correctly. //Treat this as no error. if test.expectedErr && errors.IsNotFound(err) { t.Errorf("expected error but didn't get one") } if !test.expectedErr && !errors.IsNotFound(err) { t.Errorf("unexpected error: %v", err) } }) } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: panics What you expected to happen: Syntax error, as it is not legit cp syntax. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Occurs in every version 1.9.x and 1.10.x, but 1.8.x is safe. is similar, but it has a fix candidate already going in, that is not fixing this, so opening a separate issue. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: VM OS (e.g. from /etc/os-release): Ubuntu 18.04 Kernel (e.g. ): /sig cli
It seems to be fixed in 1.11 by
It was merged three months ago, any idea why it isn't in 1.10 too?
It was merged on March 31 and 1.10 was released on March 27: Can confirm it is not present in 1.10: Maybe we can cherry-pick it?
I've opened
Fixed by /close
Closing this issue. <detailsIn response to : Fixed by Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: numErrs: 0, }, { <ins> name: "valid ExternalName (trailing dot)", tweakSvc: func(s *core.Service) { s.Spec.Type = core.ServiceTypeExternalName s.Spec.ClusterIP = "" s.Spec.ExternalName = "foo.bar.example.com." }, numErrs: 0, }, { </ins> name: "invalid ExternalName clusterIP (valid IP)", tweakSvc: func(s *core.Service) { s.Spec.Type = core.ServiceTypeExternalName

What happened: The externalName of a Service is not allowed to end with a trailing dot. What you expected to happen: The externalName of a Service should allow a trailing dot. This is very useful to denote a fully qualified value of the CNAME record created by the service and prevent DNS-trial-and-error by appending different (cluster-local) search domains. How to reproduce it (as minimally and precisely as possible): Try to set as a externalName for a Service of type ExternalName. More details The reference reads: RFC-1123 section 6.1.4.3 also states: Also in the errata of RFC-1123 a reference to RFC-1034 is to reiterate on what a legal hostname is. Section 3.1 of RFC-1034 explains the use and validity of a trailing dot. So while one could argue that a valid RFC-1123 subdomain may have a trailing dot, it might still be best to not change the which is used in a lot of places, but instead change the validation locally. Anything else we need to know?: I'll send a PR. Environment: Not much applicable.
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: // If an array is empty, missing, or nil, default plugins at that extension point will be used. type PluginSet struct { // Enabled specifies plugins that should be enabled in addition to default plugins. <ins> // If the default plugin is also configured in the scheduler config file, the weight of plugin will // be overridden accordingly. </ins> // These are called after default plugins and in the same order specified here. // +listType=atomic Enabled []Plugin `json:"enabled,omitempty"`

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Start the scheduler with a scheduler config file which has a custom configuration for a specific plugin, the scheduler will fail to start and throw below exception, kube- error message, custom plugin config should take precedence over default plugin config. start a scheduler with a config file. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /assign
Does disabling the plugin and re-enabling the same plugin with a different weight work as a workaround?
This is WAI. You have to explicitly disable the default configuration.
I think so.
I not sure I agree with this, particularly, for someone who doesn't know details of code IMPL, e.g. an operator, Why bother to disable the plugin first and then re-enable the same plugin? there is no pointer to show they are different in a config file, as a developer, we know the reason behind the scenes, can we assume operator also know what that means to define a schedule config? By disabling the plugin and re-enable it means the operator has the knowledge this plugin is a default enabled plugin, should the operator check the source code to confirm this plugin's configuration? When this happens, given a different config for a specific plugin, should this deserve a corruption? I think scheduler should more robust to tolerate this.
We can improve the docs to include the default plugins. It's better for the API to be more predictable (always do the same thing). One thing to think about is order: if we do what your PR proposes, we are altering the order in which plugins are run, not just modifying the "weight" of the plugin.


Question: // If an array is empty, missing, or nil, default plugins at that extension point will be used. type PluginSet struct { // Enabled specifies plugins that should be enabled in addition to default plugins. <ins> // If the default plugin is also configured in the scheduler config file, the weight of plugin will // be overridden accordingly. </ins> // These are called after default plugins and in the same order specified here. // +listType=atomic Enabled []Plugin `json:"enabled,omitempty"`

I'm leaning towards closing this ?
It's the key point I'm concerned about: commented here So let me ask a question: for a given input like: What's the expected output? {DefaultPlugin1, 1}, {DefaultPlugin2, 200}, {DefaultPlugin3, 3}, OR {DefaultPlugin1, 1}, {DefaultPlugin3, 3}, {DefaultPlugin2, 200}
Today, it is 2.
So in other words, if a user wants to customize a plugin's weight and keep its order, the only way is to use and give a full list of plugins? If so, that's not a good UX IMO.
Well, order doesn't matter for Score, so not really a problem.
All right, so the only case users could disable PluginXYZ while also enabling PluginXYZ is Score plugins, I'm fine with the order issue. Back to this issue, my 2 cents is to change the semantics of to a more enforced manner (proposed in this issue) in a new API version - v1beta2 or GA. As IMO it's a real UX improvement. WDYT?
I'm not convinced it's a UX improvement. If the user "enables" a filter plugin that was a default (say, by mistake), then its order changes. At least first disabling the plugin makes it more explicit. cc who has been working on decisions based on CC version
I guess it depends on the implementation. What is the full proposal?
/remove-kind bug /kind feature
That's a fair point, esp. if we adjust default plugins from version to version. I think a good UX is to throw an error/warning to indicate that a default Filter plugin is being re-enabled - this's useful no matter which semantic employes.
I'd like to: keep current logic to warn users that "PluginXYZ" is already registered (by default), if it's a non-Score plugin apply the "overridden" semantics to directive, for Score plugins only btw: let's get a consensus first and better adopt it in new CC version.
I propose (only for new version): Fail when plugin is already registered in the custom configuration, and only warning if the plugin was already part of defaults. Override any weight, preserving order I think that keeps everything consistent.


Question: // If an array is empty, missing, or nil, default plugins at that extension point will be used. type PluginSet struct { // Enabled specifies plugins that should be enabled in addition to default plugins. <ins> // If the default plugin is also configured in the scheduler config file, the weight of plugin will // be overridden accordingly. </ins> // These are called after default plugins and in the same order specified here. // +listType=atomic Enabled []Plugin `json:"enabled,omitempty"`

I think this is what the last iteration of the PR is doing, but I only skimmed through it.
SGTM. To confirm, by "Fail when plugin is already registered in the custom configuration", this's only possible when users implement an out-of-tree plugin with a name identical to a default one, right?
I'm just referring to failing when the user repeats the same value twice. I think we can't prevent what you say.
This might be the reason why the order of the plugins should be reserved.
Unless it's score plugin like this issue, I don't know if there is any point to enable the default plugin in a config file. I think there is no need to distinguish score plugin with others, this will make the life easier, correct me?
Yeah, overall, is following this ideas, except that PR doesn't handle the repeated configuration, because if there is repeated value after the default plugin is overridden, it will hit the error below,
thanks for those valuable suggestion! So, we are still with beta1, and you suggest we can make some change in the new version, beta2? Is there any plan when this could happen? v1.22?
You are correct, there is no point, but it's good to have consistency. If it's allowed for Score, it should be allowed for other extension points. I agree Yes, that is the plan. It was 1.21, but we fall behind due to newer plans for deprecating plugins
let's say there are 3 default non-Score plugins A, B and C (in that order). I want to move B to the end of the list but I want to custom enable C too (for whatever reason; since we want to consistent with score plugins, let's allow this for other extension points too as said). if a non-Score plugin is custom disabled AND enabled, we should move it to the end of the list but if it's simply just enabled, then we should preserve order. if it's a score plugin, disabling and re-enabling has the same semantics as just simply enabling; they both will only change the weight. (idk if this has already been proposed; if it has been, consider this a vote of support :))
Are you saying that "disabled and enabled" should be treated the same as just "enabled"? That is, always move to the end of the list?


Question: // If an array is empty, missing, or nil, default plugins at that extension point will be used. type PluginSet struct { // Enabled specifies plugins that should be enabled in addition to default plugins. <ins> // If the default plugin is also configured in the scheduler config file, the weight of plugin will // be overridden accordingly. </ins> // These are called after default plugins and in the same order specified here. // +listType=atomic Enabled []Plugin `json:"enabled,omitempty"`

No, just "enabled" would NOT move things to the end of the list (it will preserve order), but "disabled and enabled" would move to the end of the list.
ok, that sounds good
Another problem here is if we don't want to overridden anything but just config a plugin which is accidently a default plugin, something like below, scheduler will fail as well, I think we need to De-dup here instead of just a warning here. I guess this PR should already cover this, I will go back to check it again, at least one more testcase is needed to cover this case. keep current logic to warn users that "PluginXYZ" is already registered (by default), if it's a non-Score plugin
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle stale let me revive the PR later.
/remove-lifecycle stale


Question: "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/component-base/metrics/testutil:go_default_library", "//test/e2e/framework/log:go_default_library", <ins> "//test/e2e/framework/pod:go_default_library", </ins> "//test/e2e/perftype:go_default_library", "//test/e2e/system:go_default_library", "//vendor/k8s.io/klog:go_default_library",

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce ci-kubernetes-e2e-gci-gce Which test(s) are flaking: [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly] [sig-storage] In-tree Volumes [Driver: emptydir] [Testpattern: Inline-volume (default fs)] subPath should support file as subpath [LinuxOnly] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-link] [Testpattern: Pre-provisioned PV (default fs)] volumes should allow exec of files on the volume [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: blockfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing single file [LinuxOnly] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-bindmounted] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Inline-volume (default fs)] subPath should be able to unmount after the subpath directory is deleted [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Inline-volume (default fs)] subPath should support non-existent path [sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should support existing directories when readOnly specified in the volumeSource [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)]


Question: "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/component-base/metrics/testutil:go_default_library", "//test/e2e/framework/log:go_default_library", <ins> "//test/e2e/framework/pod:go_default_library", </ins> "//test/e2e/perftype:go_default_library", "//test/e2e/system:go_default_library", "//vendor/k8s.io/klog:go_default_library",

subPath should be able to unmount after the subpath directory is deleted [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-link-bindmounted] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing single file [LinuxOnly] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (ext3)] volumes should allow exec of files on the volume [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (ext3)] volumes should store data Testgrid link: Reason for failure: Failure fetching metrics: Anything else we need to know: This appears to have restarted abruptly around 2/11-2/12 /sig storage /priority important-soon
xref I think we have a general failure to start containers reliably. Almost every instance of these test failures contained the pod status error of "OCI runtime start failed: container process is already dead: unknown"
after sweeping a bunch of the in-tree test failures, almost all seemed to have as a root cause, triggered by very short-lived containers used to check file content or permissions. The only test I saw failing that didn't seem to fit that was: [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory examples: I'd recommend repurposing this issue to address that specifically and leave the OCI issue for the short-lived container issue
Here's a PR for context: So, I think this PR is another example of flakes. Here are some links to previous runs: and same message: This prow: has the above message and a few more, all A lot of in-tree volume issues, but there were some persistent volume ones too.


Question: "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/component-base/metrics/testutil:go_default_library", "//test/e2e/framework/log:go_default_library", <ins> "//test/e2e/framework/pod:go_default_library", </ins> "//test/e2e/perftype:go_default_library", "//test/e2e/system:go_default_library", "//vendor/k8s.io/klog:go_default_library",

But, that showed up once so far from what I've seen.
The "error getting c-m metrics" failures appear to have just restarted:
/priority critical-urgent bumping priority since we're seeing significant numbers of failures that just started in the last two days
this is failing many test runs per day, do you have updated status on this?
I haven't had chance to fully debug this yet. looked into this little bit and he suspected that these tests started to flake same time around was merged. I should be able to take a closer look tomorrow.
This occurrence fails because it's trying to get the metrics from the kube-controller-manager pod but it is not ready until one minute later if we grep in the kube-apiserver for the controller pod name we can observe that after the pod is up it starts to get 200 instead of 404
keeping open until shows this is resolved
seen again in
this time the container is running but still it can't find it
the errors comes from the request to the apiserver on How can we check that the pod is found and that the apiserver found it?
the first successful get of /metrics logged by the controller manager in that run is at: We're also not checking the error in the once.Do to make sure the pod was running, and once running, we should probably then also check the metrics path on the pod. Maybe something like this:
still happening: message logging missed using the correct errors, so the messages aren't helpful:
what's this stack trace in the apiserver? is this related? a 503 shouldn't cause that, right?
again it seems is not really waiting because it fails just 200 ms after the INFO msg, I think is because it checks if the pod is running, but the message is that the pod doesn't really exist. The fact that there are 9 tests failing with the same error and we are using has to be related to the way Ginkgo use parallel.
It turns out that the method used to wait for the controller manager pod fails if the pod doesn't exist
Ah, so first we should wait for the pod to exist, then be ready, then return metrics
This start to look promising
yes, looks good. thanks for all the work /close
Closing this issue. <detailsIn response to : yes, looks good.


Question: "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/component-base/metrics/testutil:go_default_library", "//test/e2e/framework/log:go_default_library", <ins> "//test/e2e/framework/pod:go_default_library", </ins> "//test/e2e/perftype:go_default_library", "//test/e2e/system:go_default_library", "//vendor/k8s.io/klog:go_default_library",

thanks for all the work Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Fixed by


Question: <ins> // +build !windows /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package validation import ( "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig" ) // validateKubeletOSConfiguration validates os specific kubelet configuration and returns an error if it is invalid. func validateKubeletOSConfiguration(kc *kubeletconfig.KubeletConfiguration) error { return nil } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Pods stuck in terminating state on Windows node after deleting it What you expected to happen: Pods get terminated and should disappear from kubectl How to reproduce it (as minimally and precisely as possible): Run a pod with windows container Delete it Pods are still terminating Anything else we need to know?: Could you pls help to resolve this issue. K8s pods stuck as Terminating after getting deleted on windows node with windows container(IIS). Docker ps -a not showing this container, so it is completely deleted, but in kubectl it is marked as Terminating. For networking we using OVN. kubectl logs: kubelet logs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Hyper-V VMs OS (e.g. from /etc/os-release): Windows Node: Master Node: NAME="Ubuntu" VERSION="16.04.4 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.4 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux esd-kbrnts-lnx-1 4.13.0-37-generic ~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: OVN 2.7 on windows OVN 2.7.2 on linux Others: Docker Version on Windows node: Server Version: 17.06.2-ee-6
/sig windows
It needs sometime for kubelet to remove the pod. Could you wait a while and check whether the Pod removed?
Hi actually I have waited for 1 day, and it is not deleting. Can delete it only using --grace-period=0 --force. But I don't want to use this manual approach). Could you pls give some advice on this? Thanks in advance!
is a reference to this issue. Checking kubelet logs will always help to troubleshoot such problem.


Question: <ins> // +build !windows /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package validation import ( "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig" ) // validateKubeletOSConfiguration validates os specific kubelet configuration and returns an error if it is invalid. func validateKubeletOSConfiguration(kc *kubeletconfig.KubeletConfiguration) error { return nil } </ins>

Actually, I have attached kubelet log in the ticket and think the main reason is Also as there any option for kubelet to automatically force delete Terminating pods after some period? And --containerized is also not solving my issue. Thanks!
Kubelet won't delete the pod from apiserver. Since it is already not exist on the node, you could force delete it by kubectl.
It is clear, so how we can resolve this error: ? I think this is the core item of our issue. Thanks.
I'm not using OVN. Have you met this problem when using OVN?
adding Bob to comment on this. i have seen this issue before, but it has been quite a while.
Hi, Guys will be very grateful if you will provide some solution for it. We have tried to add to deployment spec but it doesn't help. Thanks.
The error message of is actually a warning message. It shouldn't block the removal of pods. Could you recheck the following things: the node, check whether the pod is still existing, e.g. by the pod exists, check if there are other error or warning message in kubelet logs; or else, the pod could be force removed safely from apiserver
Hi, I had posted this info in the ticket description. not showing this container, so it is completely deleted, but in kubectl it is displayed as Terminating.
i think you may get more support and faster responses in our slack channel for sig-windows.
So it could be safely force removed from API. And agreed with slack is a more faster way to such troubleshooting issues.
The issue seems to be related to . The only platform specific implementation for PodContainerManager is the one for . There is a default implementation (), that always returns true in the Exists call. Disabling the CgroupsPerQOS option on the Windows nodes resolves the issue.
Windows doesn't support cgroups. You should always disable it.
it is clear enough that Windows does not support cgroups). Unfortunately, it's not specified in the documentation. So to disable CgroupsPerQOS you should need to add this flags to kubelet: Maybe it will help someone... Thank you all for discussion!


Question: sleep 2 done <ins> # Basic sanity checking for (( i=0; i<${#MINION_NAMES[@]}; i++)); do # Make sure docker is installed gcutil ssh ${MINION_NAMES[$i]} which docker > /dev/null if [ "$?" != "0" ]; then echo "Docker failed to install on ${MINION_NAMES[$i]} your cluster is unlikely to work correctly" echo "Please run ./cluster/kube-down.sh and re-create the cluster. (sorry!)" exit 1 fi # Make sure the kubelet is running gcutil ssh ${MINION_NAMES[$i]} /etc/init.d/kubelet status if [ "$?" != "0" ]; then echo "Kubelet failed to install on ${MINION_NAMES[$i]} your cluster is unlikely to work correctly" echo "Please run ./cluster/kube-down.sh and re-create the cluster. (sorry!)" exit 1 fi done </ins> echo echo "Kubernetes cluster is running.  Access the master at:" echo

What happened: Deployment of node-local-dns in an ipv6-only cluster fails with the following iptables-related error Does node-local-dns support binding to IPv6 addresses? If not, can node-local-dns be used in an ipv6-only cluster? What you expected to happen: node-local-dns is up and running normally and binding to an ipv6 link-local address How to reproduce it (as minimally and precisely as possible): Create an ipv6-only cluster using nodes with single ipv6 stack Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.2 Cloud provider or hardware configuration: Openstack VMs OS (e.g: ): SLES 15 SP1 Kernel (e.g. ): 4.12.14-197.45-default Install tools: kubeadm Network plugin and version (if this is a network-related bug): calico v3.12.0 Others:
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
seems that node-local-dns only supports IPv4
looks like whatever's calling might need to have a log msg b4 it
/assign
Looks like there's already a logging statement for running iptables checks: Perhaps the log level was too low for that log message to get tripped?
hi Matt, ok so your looking at the logs? If you want you can into one of your kind nodes. Then edit run and if you wanna hack around in there, and open inside that container, which will allow you to play with the logging verbosity of Kubelet. If you decide its a good idea to surface this info then you can test there (). There are probably ways to plumb Kubelet args directly into kind also but that's just my off the cufff answer :)
/assign
/assign /assign
Not yet, node-local-dns is not supported to run on ipv6-only clusters.
/remove-triage unresolved
Thank you for letting me know, and thank you all for your efforts following through with the issue, I'll keep watch for any future updates on the feature.
I put together a patch to to make it more clear in this case what the issue is. Do you think it's worth moving forward with adding that log statement?


Question: sleep 2 done <ins> # Basic sanity checking for (( i=0; i<${#MINION_NAMES[@]}; i++)); do # Make sure docker is installed gcutil ssh ${MINION_NAMES[$i]} which docker > /dev/null if [ "$?" != "0" ]; then echo "Docker failed to install on ${MINION_NAMES[$i]} your cluster is unlikely to work correctly" echo "Please run ./cluster/kube-down.sh and re-create the cluster. (sorry!)" exit 1 fi # Make sure the kubelet is running gcutil ssh ${MINION_NAMES[$i]} /etc/init.d/kubelet status if [ "$?" != "0" ]; then echo "Kubelet failed to install on ${MINION_NAMES[$i]} your cluster is unlikely to work correctly" echo "Please run ./cluster/kube-down.sh and re-create the cluster. (sorry!)" exit 1 fi done </ins> echo echo "Kubernetes cluster is running.  Access the master at:" echo

(I know I'd have to redo the patch differently, I did it a really dumb way and sure there's a much much better way :) ) Happy to do the leg work on this! Thanks for the guidance! Edit: forgot to mention, this is only supposed to be a stopgap short-term solution before node-local-dns gets support for IPv6!
so , is what pramesh is suggesting as simple as running something in to figure out if there is an ipv4 address... and if not, ... fail ?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /assign I have a PR ready, but we need to update the KEP, the docs and add some testing, ... targeting 1.21


Question: return err } <del> for ix := range pods.Items { pod := &pods.Items[ix] writer := setupOutputWriter(cmd, out, path.Join(namespace, pod.Name, "logs.txt")) writer.Write([]byte(fmt.Sprintf("==== START logs for %s/%s ====n", pod.Namespace, pod.Name))) request, err := f.LogsForObject(pod, &api.PodLogOptions{}, timeout) </del> <ins> printContainer := func(writer io.Writer, container api.Container, pod *api.Pod) { writer.Write([]byte(fmt.Sprintf("==== START logs for container %s of pod %s/%s ====n", container.Name, pod.Namespace, pod.Name))) defer writer.Write([]byte(fmt.Sprintf("==== END logs for container %s of pod %s/%s ====n", container.Name, pod.Namespace, pod.Name))) request, err := f.LogsForObject(pod, &api.PodLogOptions{Container: container.Name}, timeout) </ins> if err != nil { <del> return err </del> <ins> // Print error and return. writer.Write([]byte(fmt.Sprintf("Create log request error: %sn", err.Error()))) return </ins> } data, err := request.DoRaw() if err != nil { <del> return err </del> <ins> // Print error and return. writer.Write([]byte(fmt.Sprintf("Request log error: %sn", err.Error()))) return </ins> } writer.Write(data) <del> writer.Write([]byte(fmt.Sprintf("==== END logs for %s/%s ====n", pod.Namespace, pod.Name))) </del> <ins> } for ix := range pods.Items { pod := &pods.Items[ix] containers := pod.Spec.Containers writer := setupOutputWriter(cmd, out, path.Join(namespace, pod.Name, "logs.txt")) for i := range containers { printContainer(writer, containers[i], pod) } </ins> } } dir := cmdutil.GetFlagString(cmd, "output-directory")

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): cluster-info, dump, "server rejected our request for an unknown reason" Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ): Install tools: kubeadm Others: What happened: kubectl cluster-info dump fails to get logs from pods with multiple containers. It exits at the first such pod it encounters (in lexical order). The kube-dns pod invariable triggers this. What you expected to happen: should gracefully handle multi-container pods, fetching logs for all of containers. a dump fail, kubectl should continue dumping as much of the information from remaining resources as possible. How to reproduce it (as minimally and precisely as possible): Run with a kube-dns pod running. Anything else we need to know:
I can reproduce the error by using the latest code, let me take some investigation on this first.


Question: Phase: api.NamespaceActive, }, } <del> store := cache.NewStore(cache.IndexFuncToKeyFuncAdapter(cache.MetaNamespaceIndexFunc)) </del> <ins> reactFunc := func(action testclient.Action) (runtime.Object, error) { switch { case action.Matches("get", "namespaces"): if getAction, ok := action.(testclient.GetAction); ok && getAction.GetName() == namespaceObj.Name { return namespaceObj, nil } case action.Matches("list", "namespaces"): return &api.NamespaceList{Items: []api.Namespace{*namespaceObj}}, nil } return nil, fmt.Errorf("No result for action %v", action) } store := cache.NewStore(cache.MetaNamespaceKeyFunc) </ins> store.Add(namespaceObj) <del> mockClient := &testclient.Fake{} </del> <ins> fakeWatch := watch.NewFake() mockClient := &testclient.Fake{Watch: fakeWatch, ReactFn: reactFunc} </ins> lfhandler := NewLifecycle(mockClient).(*lifecycle) lfhandler.store = store handler := admission.NewChainHandler(lfhandler) pod := api.Pod{ <del> ObjectMeta: api.ObjectMeta{Name: "123", Namespace: namespaceObj.Namespace}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: "123", Namespace: namespaceObj.Name}, Spec: api.PodSpec{ Volumes:    []api.Volume{{Name: "vol"}}, Containers: []api.Container{{Name: "ctr", Image: "image"}}, }, } badPod := api.Pod{ ObjectMeta: api.ObjectMeta{Name: "456", Namespace: "doesnotexist"}, </ins> Spec: api.PodSpec{ Volumes:    []api.Volume{{Name: "vol"}}, Containers: []api.Container{{Name: "ctr", Image: "image"}},

We should have a single NamespaceLifecycle plugin that enforces the Namespace rules now that all providers are off NamespaceAutoProvision. We may also want to make this no longer a user choice to configure and hard-wire the server to always run this check first to simplify configuration errors. For an example


Question: }, }, Capacity: v1.ResourceList{ <del> v1.ResourceCPU:     *resource.NewMilliQuantity(3000, resource.DecimalSI), v1.ResourceMemory:  *resource.NewQuantity(20E9, resource.BinarySI), v1.ResourceStorage: *resource.NewQuantity(500*mb, resource.BinarySI), v1.ResourcePods:    *resource.NewQuantity(0, resource.DecimalSI), </del> <ins> v1.ResourceCPU:    *resource.NewMilliQuantity(3000, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(20E9, resource.BinarySI), v1.ResourcePods:   *resource.NewQuantity(0, resource.DecimalSI), </ins> }, Allocatable: v1.ResourceList{ <del> v1.ResourceCPU:     *resource.NewMilliQuantity(2800, resource.DecimalSI), v1.ResourceMemory:  *resource.NewQuantity(19900E6, resource.BinarySI), v1.ResourceStorage: *resource.NewQuantity(500*mb, resource.BinarySI), v1.ResourcePods:    *resource.NewQuantity(0, resource.DecimalSI), </del> <ins> v1.ResourceCPU:    *resource.NewMilliQuantity(2800, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(19900E6, resource.BinarySI), v1.ResourcePods:   *resource.NewQuantity(0, resource.DecimalSI), </ins> }, }, }

Local storage is not expected to be available as a scheduling resource in v1.7. Kubelet is incorrectly exposing a resource named . This is incorrect because the feature is alpha and so it requires opt-in via feature flags and, the resource names are invalid. The expected resource names are listed - and . cc This is v1.7 blocker because we cannot expose a new resource in the API and rename it in a subsequent release.
CC I'll find someone on the storage team to address this.
/assign
GitHub didn't allow me to assign the following users: verult. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }) ginkgo.It("should provide Internet connection for containers [Feature:Networking-IPv4]", func() { <del> ginkgo.By("Running container which tries to ping 8.8.8.8") </del> <ins> ginkgo.By("Running container which tries to connect to 8.8.8.8") </ins> framework.ExpectNoError( <del> framework.CheckConnectivityToHost(f, "", "ping-test", "8.8.8.8", framework.IPv4PingCommand, 30)) </del> <ins> framework.CheckConnectivityToHost(f, "", "connectivity-test", "8.8.8.8", 53, 30)) </ins> }) ginkgo.It("should provide Internet connection for containers [Feature:Networking-IPv6][Experimental]", func() { <del> ginkgo.By("Running container which tries to ping 2001:4860:4860::8888") </del> <ins> ginkgo.By("Running container which tries to connect to 2001:4860:4860::8888") </ins> framework.ExpectNoError( <del> framework.CheckConnectivityToHost(f, "", "ping-test", "2001:4860:4860::8888", framework.IPv6PingCommand, 30)) </del> <ins> framework.CheckConnectivityToHost(f, "", "connectivity-test", "2001:4860:4860::8888", 53, 30)) </ins> }) // First test because it has no dependencies on variables created later on.

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: Since when has it been failing: Ever Reason for failure: Vm on Azure with no public IP assigned aren't able to perform a command to the internet. Anything else we need to know: The test (see ) tries to test the internet connection for a pod by pinging 8.8.8.8. But azure does not provide ICMP support for VM without a public IP therefore this test cannot succeed.(see ) A fix would be to change the test method to a tcp request instead of a ping request (the test already performs a http get to in its BeforeSuite (see ))
/sig network /sig testing /sig azure
you may refer to (Pods don't have connectivity to external networks )
cc WDYT about the proposal to change this from an ICMP call? We probably need input from sig-architecture as well since this as well /sig architecture
/triage unresolved
An alternative is to issue a TCP/UDP request to 8.8.8.8 on port 53.
ICMP is dropped by Azure for security reasons. This is intended. Can use curl as a replacement. I will fix the corresponding test.
SGTM
Can you please review the related PR? I'll in the PR again. Thanks!


Question: <ins> // +build !windows /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package debugger import "syscall" // compareSignal is the signal to trigger cache compare. For non-windows // environment it's SIGUSR2. var compareSignal = syscall.SIGUSR2 </ins>

PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. Main tracking issue: Currently, the feature is set up at the end of in . This setup should be moved into its own helper function, preferably in the package. /sig scheduling /kind cleanup /help
/assign /remove-help
I'd like to take this one
/remove-help
the PR I sent should resolve this one. :disappointed: Are any of the others still available?


Question: ) func performContainerRuntimeUnitOp(op containerRuntimeUnitOp) error { <del> ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) </del> <ins> ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) </ins> defer cancel() conn, err := dbus.NewWithContext(ctx)

E2eNode Suite.[It] [sig-node] MirrorPodWithGracePeriod when create a mirror pod and the container runtime is temporarily down during pod termination [NodeConformance] [Serial] [Disruptive] the mirror pod should terminate successfully Perma failure here: E2eNode Suite.[It] [sig-node] MirrorPodWithGracePeriod when create a mirror pod and the container runtime is temporarily down during pod termination [NodeConformance] [Serial] [Disruptive] the mirror pod should terminate successfully E2eNode Suite.[It] [sig-node] MirrorPodWithGracePeriod when create a mirror pod and the container runtime is temporarily down during pod termination [NodeConformance] [Serial] [Disruptive] the mirror pod should terminate successfully Since it was created at No response No response /sig node
/cc
/cc /cc is there anything specific to CRI-O that will make test fail
/assign
The test fails there: Hm, scraping the kubelet logs it looks like that we miss a metric: cc
/triage accepted /priority important-soon seeing this in the wild
failed for this in as well
It seems to fail on on :
Having a fix in mind in


Question: # function kube::codegen::gen_client() { local in_pkg_root="" <ins> local one_input_api="" </ins> local out_pkg_root="" local out_base="" # gengo needs the output dir must be $out_base/$out_pkg_root local clientset_subdir="clientset"

function in script has following issues: function only supports file but we have multiple types file with name as in . Hence this function is not working in our case. For the mitigation there is already an open . I have asked for including our changes as well. We have multiple files under a for example and . Because of this for appears twice and it generates duplicate informers We only need unique elements before passing it to . I followed this approach for unique elements: We have packages under . Earlier we were generating client only for and . But with the new I am not able to do that. I am using the following call to function: It results in following flags to As there is no directory as , it doesnot generate anything. Is it still possible to call on specific folder as it was done with earlier scripts? One mitigation I can think of is to call function on whole and then remove the marker from the types for which genclient is not needed. But the issue is we won't be able to pass the flag for each pkg. function in script is not working as expected.
/sig api-machinery
/cc
/triage accepted
Hi . I knew that there were all sorts of clients out there doing all sorts of things with these scripts, so I am not surprised to find you. :) Do you have a PR which attempts to use kube_codegen, so I can repro your issues directly?
I think you have one too many in there Aside from that, the real problem here is that these old codegen tools make some deeeeeeep assumptions about the ability to string-join various arguments and get the right results. It's awful. I was trying to make it simpler in this script, which I did for all of the in-tree consumers, but out of tree people apparently don't always follow the same conventions. For example, we generate clients for every directory that requests it. For a reason I do not know, you have files like which say but you don't actually want a client? If you specify then it t
So this code expects that you generate an omnibus client. What if I add a flag, like which you can use to call it for each API to make distinct clients? In truth the codegen tools need to be fixed, but that has SO MUCH ripple I just can't right now.


Question: # function kube::codegen::gen_client() { local in_pkg_root="" <ins> local one_input_api="" </ins> local out_pkg_root="" local out_base="" # gengo needs the output dir must be $out_base/$out_pkg_root local clientset_subdir="clientset"

As the codegen was breaking us at several step, hence didn't prepared the PR.
Thanks for the PR. I have one more concern, is it possible to call and client-gen on the same pkg? With the current script, that is not possible because of the line It removes all the generated files first and then calls the . Hence the later clientset gen will remove the earlier generated clients.
Why are you generating a client for an internal version that isn't served?
My mistake. We need to stop using clients for internal versions.


Question: v1.ResourceName("my.org"): resource.MustParse("10m"), }, }, <ins> skipRequestValueCheck: true, </ins> }, { name: "Invalid Resources with Limits",

Created limit range object with this parameters then tried to create this pod got message: Error from server (Invalid): error when creating "pod-to-be-": Pod "nginx-to-be-removed-4" is invalid: spec.containers[0].resources.requests: Invalid value: "700m": must be less than or equal to cpu limit Pod should be created with Requests & Limits of cpu equal to 700m (since it is in range of min/max cpu limitrange object) create limitrange object and pod as in description No response <details</details<detailson premise/AKS </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
You have a default CPU limit of 10m, and no explicit CPU limit on the container, but a request larger than the limit. The request must always be less than or equal to the limit. Your pod needs to also set a higher limit, or the limit range needs to not set a default limit.
/remove-kind bug /kind support
/cc
but this is not obvious behavior since there is min & max set in limit range. This is limits (not min/max) that should be overwritten by pod. limits should be set to requests in this case imho
limit range limits what the user can set, it doesn't perform any mutations other than the defaulting and doesn't claim to. Min/max enforce that pods cannot set higher / lower. The default ensures that if the pod doesn't set a limit it will have one. personally I would remove the default limit, especially at 10m, this will aggressively limit pods. it would be, if the pod set a limit that would be a substantial breaking change, I don't think we could do that short of a v2 API and I'm not sure it would be correct behavior even then. we could make it opt-in, but then, why not just set them to match on the pod to begin with?


Question: } info.portalIP = serviceIP info.portalPort = service.Spec.Port <del> if service.Spec.CreateExternalLoadBalancer { info.publicIP = service.Spec.PublicIPs } </del> <ins> info.publicIP = service.Spec.PublicIPs </ins> err = proxier.openPortal(service.Name, info) if err != nil { glog.Errorf("Failed to open portal for %q: %v", service.Name, err)

I'm not sure if it's the intended behaviour, but when I specify publicIPs for a service, external portals are only opened if createExternalLoadBalancer is set, too (this is checked in the OnUpdate function in ). After reading the docs, I would have expected that IPTables rules for external IPs are independent of the createExternalLoadBalancer flag. Also, we're running kubernetes locally, so setting createExternalLoadBalancer doesn't work since there's no cloud provider specified.
I was about to say it should work, but then I went and re-read the code and found at least one bug. Let me dig a bit more
You're right - the IPs actually propagate to the proxy, but it ignores them if createExternal... is not set. I think it is kosher to just do it. I'm prepping a PR
PR is up - if you feel bold you could test it in your environment...
Thanks, it's working fine with your patch.


Question: # Our minions will have a pool role to distinguish them from the master. cat <<EOF >/etc/salt/minion.d/grains.conf grains: <ins> network_mode: openvswitch </ins> node_ip: $MINION_IP etcd_servers: $MASTER_IP roles:

The salt configs assume that anything under "os_family = RedHat" will use the ovs mechanisms that are only set up for vagrant. This means that we can't use Fedora on GCE.
I can clean this up today.


Question: if err != nil { return nil, err } <del> err = json.Unmarshal(patched, clone) </del> <ins> restoredSet := &apps.StatefulSet{} err = json.Unmarshal(patched, restoredSet) </ins> if err != nil { return nil, err } <del> return clone, nil </del> <ins> return restoredSet, nil </ins> } // nextRevision finds the next valid revision number based on revisions. If the length of revisions

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After deleting a pod and scaling up the statefulset with different valid envs in the same container simultaneously, then the output events of executing the command is below: What you expected to happen: update successfully. How to reproduce it (as minimally and precisely as possible): a statefulset. apply this statefulset's yaml: After a while, execute , and the pod is running a new stateful's yaml, which a new env key and more replicas. the pod and apply a new stateful's yaml the statefulset Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
/sig apps
Meet same problem while using kube-addon-manager. vs and patchStrategy
+1
In some cases, add in the env which using can solve the problem.
/assign
I think the root cause of this is the strategic patch. As mentioned, we have to use the to fix the problem which tells the strategic patch to remove the field so the spec could pass the validation process of statefulsets. It would lead to API broken if we change the behaviour of strategic merge patch, I'll dig around in the apps validation process and try to find out the solution to this.
Thank you for your reply. According to your advice, I tried to add in my second statefulset's yaml, but I still could not update statefulset correctly. How to fix my yamls to get around the problem?


Question: if err != nil { return nil, err } <del> err = json.Unmarshal(patched, clone) </del> <ins> restoredSet := &apps.StatefulSet{} err = json.Unmarshal(patched, restoredSet) </ins> if err != nil { return nil, err } <del> return clone, nil </del> <ins> return restoredSet, nil </ins> } // nextRevision finds the next valid revision number based on revisions. If the length of revisions

/unassign Sorry, I do not have enough time for this, anyone interested could take it from here.
I don't think the problem comes from your YAML, the problem comes from the controller that doesn't update the fields properly.
/assign


Question: package storage import ( <del> "encoding/json" </del> "fmt" "math/rand" "path"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The local-volume bootstrapper/provisioner configuration format changed in kubernetes-incubator/external-storage, and its related e2e tests will fail when the provisioner & bootstrapper images are updated to v2.0.0. What you expected to happen: I'd expect the e2e tests to continue to work properly with the new version of the provisioner and boostrapper. How to reproduce it (as minimally and precisely as possible): Update local-volume provisioner and boostrapper images to v2.0.0, and then the provisioner e2e test will fail.
/sig storage
/assign


Question: glog.Fatal("Couldn't connect to docker.") } <del> cadvisorClient, err := cadvisor.NewClient("http://127.0.0.1:4194") if err != nil { glog.Errorf("Error on creating cadvisor client: %v", err) } </del> hostname := getHostname() if *rootDirectory == "" {

So today, cadvisor became wedged and just stopped paying attention to its socket. Not kubelet's fault. But when I would launch all I would get would be: It took me a bit of trolling through netstat and lsof to figure out that the problem was cadvisor being completely locked up. kill -9 cadvisor and everything started working well. I wonder if we should emit more messages during startup about what operation we are about to attempt, not just the results of operations, as I was blaming docker for a while....
kubelet shouldn't block on this. I'll take a look.
Any ideas what was locking cAdvisor? and was it just not responding to requests?
I don't have any idea why it was locked. If I find another one locked I'll hang onto it to debug. Journal had: obviously the 'kill' came from me. It would not respond to systemctl cadvisor stop. netstat looking at port 4194 showed me At which point I realized cadvisor wasn't handling its sockets right since so many long dead kubelet sockets hadn't been cleaned up, so I kill -9'd it.
I cut some interesting info from the cadvisor journal log (trying to cut hostnames) You can see the last message about failed to update stats for container ... happened Oct 08 01:06:01. It was then silent until I killed it Oct 08 19:20:13, even though I'm pretty sure dnf was still being launched every hour on the hour. So it was wedged for about 18 hours.
What kernel is this on? We've seen there is a kernel bug fixed in 3.12+ that sometimes hangs us forever on exit. We're not quite sure how it gets triggered...
16.3


Question: return nil, fmt.Errorf("claim.Spec.Selector is not supported for dynamic provisioning on Azure file") } <ins> if shareName == "" { // File share name has a length limit of 63, and it cannot contain two consecutive '-'s. name := util.GenerateVolumeName(a.options.ClusterName, a.options.PVName, 63) shareName = strings.Replace(name, "--", "-", -1) } </ins> // when use azure file premium, account kind should be specified as FileStorage accountKind := string(storage.StorageV2) if strings.HasPrefix(strings.ToLower(sku), "premium") { accountKind = string(storage.FileStorage) } <del> account, key, err := a.azureProvider.CreateFileShare(name, account, sku, accountKind, resourceGroup, location, requestGiB) </del> <ins> account, key, err := a.azureProvider.CreateFileShare(shareName, account, sku, accountKind, resourceGroup, location, requestGiB) </ins> if err != nil { return nil, err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: use an existing azure file share name should be possible in azure file plugin What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig azure /assign


Question: } } <ins> func runAMaster(t *testing.T) (*master.Master, *httptest.Server) { helper, err := master.NewEtcdHelper(newEtcdClient(), testapi.Version()) if err != nil { t.Fatalf("unexpected error: %v", err) } m := master.New(&master.Config{ EtcdHelper:        helper, KubeletClient:     client.FakeKubeletClient{}, EnableLogsSupport: false, EnableProfiling:   true, EnableUISupport:   false, APIPrefix:         "/api", Authorizer:        apiserver.NewAlwaysAllowAuthorizer(), AdmissionControl:  admit.NewAlwaysAdmit(), }) s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { m.Handler.ServeHTTP(w, req) })) return m, s } </ins>

In 0.13.2, the apiserver has automatically exposed metrics about the process, like start time, memory usage, processor usage, etc. This stopped working sometime between then and when 0.14.0 was cut. They're still correctly exported from the kubelet, interestingly enough. An e2e or integration test is also needed to catch this if it happens again.


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/api/meta" "github.com/GoogleCloudPlatform/kubernetes/pkg/runtime" "github.com/GoogleCloudPlatform/kubernetes/pkg/util/errors" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util/yaml" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/watch" )

If I stuff into a file and use that after -f, it works fine. cc
What's on stdin?
Bug, I'm only accepting json because I naively reused watch partitioner and we don't have test cases. Will fix tonight.
If that's the problem, why did it work when I gave it that file not over stdin?
Different code path. Stdin supports streaming json a la watch (multiple json objects in a stream), so you can do Unfortunately I missed yaml input and didn't use a segmenting yaml decoder (returning a stream of yaml objects).
Ah! Thanks for explanation. I think it would be good for us to ask for e2e runs before merging kubectl changes, since there's not quite enough coverage in test- yet.
Yeah, I'm slacking on that for these changes.
Fix incoming.
Fix provided. More tests to test-cmd... :)


Question: pvc = newClaim(test, ns, "default") var err error <del> metricsGrabber, err = metrics.NewMetricsGrabber(c, nil, false, false, true, false, false) </del> <ins> metricsGrabber, err = metrics.NewMetricsGrabber(c, nil, true, false, true, false, false) </ins> if err != nil { framework.Failf("Error creating metrics grabber : %v", err) } }) <ins> AfterEach(func() { framework.DeletePersistentVolumeClaim(c, pvc.Name, pvc.Namespace) }) </ins> It("should create prometheus metrics for volume provisioning and attach/detach", func() { var err error

In 1.8, we a new feature to expose PVC information through summary API so user can check the volume belongs to which PVC. We should add e2e test in under test/e2enode.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
GitHub didn't allow me to assign the following users: vkamra. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Can we please move this out of 1.8 milestone?
- I'm looking into adding this but yes we should move this out of 1.8. I don't have permissions to do that though.
ack Can we please move this out of 1.8?
Sorry am not sure why was this removed from 1.8 milestone. Are we not aiming to add the tests in 1.8? Or Are we disabling the feature or are we fine shipping it without any e2e tests in 1.8?
The test is in . I think we can keep this feature in 1.8


Question: float32(*registryPullQPS), *registryBurst) <ins> go func() { defer util.HandleCrash() // TODO: Monitor this connection, reconnect if needed? glog.V(1).Infof("Trying to create cadvisor client.") cadvisorClient, err := cadvisor.NewClient("http://127.0.0.1:4194") if err != nil { glog.Errorf("Error on creating cadvisor client: %v", err) return } glog.V(1).Infof("Successfully created cadvisor client.") k.SetCadvisorClient(cadvisorClient) }() </ins> // TODO: These should probably become more plugin-ish: register a factory func // in each checker's init(), iterate those here. health.AddHealthChecker(health.NewExecHealthChecker(k))


Question: readonly KNOWN_TOKENS_FILE="/srv/salt-overlay/salt/kube-apiserver/known_tokens.csv" readonly BASIC_AUTH_FILE="/srv/salt-overlay/salt/kube-apiserver/basic_auth.csv" <ins> function ensure-basic-networking() { # Deal with GCE networking bring-up race. (We rely on DNS for a lot, # and it's just not worth doing a whole lot of startup work if this # isn't ready yet.) until getent hosts metadata.google.internal &>/dev/null; do echo 'Waiting for functional DNS (trying to resolve metadata.google.internal)...' sleep 3 done until getent hosts $(hostname -f) &>/dev/null; do echo 'Waiting for functional DNS (trying to resolve my own FQDN)...' sleep 3 done until getent hosts $(hostname -i) &>/dev/null; do echo 'Waiting for functional DNS (trying to resolve my own IP)...' sleep 3 done echo "Networking functional on $(hostname) ($(hostname -i))" } </ins> function ensure-install-dir() { INSTALL_DIR="/var/cache/kubernetes-install" mkdir -p ${INSTALL_DIR}

Node 2's kubelet logs: controller-manager side: I have a cluster to hand off if someone wants to debug. It's not a transient state: node 2 just failed to ever post a valid status. cc cc (who looked at it briefly with me)
I will take a look soon.
Ok, why kubelet on node 2 talks to ip address of master instead of hostname? root ps -ef | grep kubelet root 1 3 17:35 ? 00:10:45 /usr/local/bin/kubelet --apiservers=https://10.240.49.47 --config=/etc/kubernetes/manifests --allowprivileged=False --v=2 --clusterdns=10.51.240.10 -- It looks like a cluster up issue to me, not a node problem.
It looks like failed to translate the master name into the fqdn (which is done ).
Fascinating. I think we're seeing instance-DNS flakiness, because that name is supposed to get converted ASAP. But what's weirder is ... that's not even the master:
The is definitely right:
Okay, this is messed up. Here's node 1, which is happy: Here's our unhappy node .... that decided to bring itself as a master:
haha... I checked kube-env and did suspect unhappy node treat itself as master node based on those variables. But I didn't check salt state :-)
I ... just can't get there. By the time we get here: , node-2 has set: My best guess is that flaked here: . I think we're seeing a systemic issue where we're racing with GCE network configuration.
I'm going to work up a PR that hopefully improves the reliability of this section to some extent.
<a href=""<img src="" title="made at "/</a
My working theory is that we raced with the configuration done normally by the GCE NetworkManager, leaving us in a bogus state when we made that check.
Why are we relying on inferring the master/node status from the hostname. It should be obvious from the variables set in the kube-env if you are the master or not. Masters don't get passed so why not just check that? Making sure DNS is working is probably also good for reliability...
On GKE, we actually pass the same kube-env on both sides for simplicity.


Question: readonly KNOWN_TOKENS_FILE="/srv/salt-overlay/salt/kube-apiserver/known_tokens.csv" readonly BASIC_AUTH_FILE="/srv/salt-overlay/salt/kube-apiserver/basic_auth.csv" <ins> function ensure-basic-networking() { # Deal with GCE networking bring-up race. (We rely on DNS for a lot, # and it's just not worth doing a whole lot of startup work if this # isn't ready yet.) until getent hosts metadata.google.internal &>/dev/null; do echo 'Waiting for functional DNS (trying to resolve metadata.google.internal)...' sleep 3 done until getent hosts $(hostname -f) &>/dev/null; do echo 'Waiting for functional DNS (trying to resolve my own FQDN)...' sleep 3 done until getent hosts $(hostname -i) &>/dev/null; do echo 'Waiting for functional DNS (trying to resolve my own IP)...' sleep 3 done echo "Networking functional on $(hostname) ($(hostname -i))" } </ins> function ensure-install-dir() { INSTALL_DIR="/var/cache/kubernetes-install" mkdir -p ${INSTALL_DIR}

We can obviously change that, but I was going for a change that didn't involve a deployment change on our side, too.


Question: // set status for Pods created on versions of kube older than 1.6 apiPodStatus.QOSClass = v1qos.GetPodQOS(pod) <ins> oldPodStatus, found := kl.statusManager.GetPodStatus(pod.UID) if !found { oldPodStatus = pod.Status } </ins> apiPodStatus.ContainerStatuses = kl.convertToAPIContainerStatuses( pod, podStatus, <del> pod.Status.ContainerStatuses, </del> <ins> oldPodStatus.ContainerStatuses, </ins> pod.Spec.Containers, len(pod.Spec.InitContainers) > 0, false, ) apiPodStatus.InitContainerStatuses = kl.convertToAPIContainerStatuses( pod, podStatus, <del> pod.Status.InitContainerStatuses, </del> <ins> oldPodStatus.InitContainerStatuses, </ins> pod.Spec.InitContainers, len(pod.Spec.InitContainers) > 0, true,

After , we observed pods moving from phase to . Presents on master and release-1.8 as it was picked in test script has an command for basically doing a CLI watch or you can look in the kubelet log to see the phase transition from to Thanks to for helping isolate this :+1: /sig node
/assign
taking a look. although at first glance it is strange this didnt happen with evicted pods prior to this.
Thanks. FYI, I did confirm that reverting the mentioned PR on master corrected the issue.
This showed up breaking a job controller that depended on the pod not transitioning back to pending. It's possible this can cause applications to fail / run multiple times in potentially dangerous ways. I'm not sure we should allow a status update that moves back to a non terminal state via validation. It's not a legal transition as we have defined phase. We should separately put a guard in place that will report a clear error to clients.
after thinking on it, I think that the situation with eviction is a little different in that the containers are terminated but they are not immediately cleaned up. It is the proximity between pod termination time and container removal time that is uncovering the issue, I think. AFAICT, some path is attempting to get the container status after the container has been removed. When this fails, something in the status manager gets messed up and it under the impression that the pod is new and the status needs to be reinitialized.
Here are the relevant logs showing the bug: Still trying to figure out what caused the status update.
Ok, I figured out why this is happening. When we compute a container's status, we assume that . So deleting a pod's containers before the pod object is removed causes the state to return to pending.
Maybe we should expand this that I to prevent init container status reset to include all containers with an oldStatus
Would that prevent updating a container that crashed and restarted? Edit: nvm, I think that might work
I a comment at
i also commented here that it would be good if syncPod in status manager protected against an invalid transfer of phase... see: can we do that in this same PR?
My hope is to cherrypick this to 1.8. Is that check something we want to cherrypick as well?


Question: // set status for Pods created on versions of kube older than 1.6 apiPodStatus.QOSClass = v1qos.GetPodQOS(pod) <ins> oldPodStatus, found := kl.statusManager.GetPodStatus(pod.UID) if !found { oldPodStatus = pod.Status } </ins> apiPodStatus.ContainerStatuses = kl.convertToAPIContainerStatuses( pod, podStatus, <del> pod.Status.ContainerStatuses, </del> <ins> oldPodStatus.ContainerStatuses, </ins> pod.Spec.Containers, len(pod.Spec.InitContainers) > 0, false, ) apiPodStatus.InitContainerStatuses = kl.convertToAPIContainerStatuses( pod, podStatus, <del> pod.Status.InitContainerStatuses, </del> <ins> oldPodStatus.InitContainerStatuses, </ins> pod.Spec.InitContainers, len(pod.Spec.InitContainers) > 0, true,

i dont see why not... absent that its hard to know this fix actually fixes it... or that we dont regress.
do we want to do it in or ? is on the other side of a channel where a stack trace on the caller of could be more informative.
both? i just know that we get the latest state as told to api server on the other end of the channel...


Question: return t.delegate.New(kind) } } <ins> func NewThirdPartyParameterCodec(p runtime.ParameterCodec) runtime.ParameterCodec { return &thirdPartyParameterCodec{p} } type thirdPartyParameterCodec struct { delegate runtime.ParameterCodec } func (t *thirdPartyParameterCodec) DecodeParameters(parameters url.Values, from unversioned.GroupVersion, into runtime.Object) error { return t.delegate.DecodeParameters(parameters, v1.SchemeGroupVersion, into) } func (t *thirdPartyParameterCodec) EncodeParameters(obj runtime.Object, to unversioned.GroupVersion) (url.Values, error) { return t.delegate.EncodeParameters(obj, v1.SchemeGroupVersion) } </ins>

A user report that it's broken in the slack channel:
Also and -- this probably broke with one of our recent refactors. The server should be accepting list options from any version, I thought?
FWIW, I think it's this line barf at the ListOptions: The parameter codec first decodes the query parameter to a versioned object and then convert it to the internal object. The code fails because it cannot create a "" ListOptions.
The server should accept list options in the API version it is sent. ListOptions should be registered in all third party groups.
The thirdPartyResourceDataCreator should handle "ListOptions", which surprises me that it doesn't then respect the group.
I think we should add a wrapper around the api.ParameterCodec here: Do something similar as what does.
Putting together a PR now and will let you guys review.
I guess my point is, the list options are just there from the URL query parameters. The user didn't send a group. Why do we assume that value for the group rather than "/v1" or something?
My understanding is we want to allow options to evolve differently in different groups. We do have v1.ListOptions and v1beta1.ListOptions. And if a client sends requeset to extensions/v1beta1 endpoint, then it's expected to encode the ListOptions to v1beta1 and then convert it to query parameters.
Mostly because the parameter codec is supposed to abstract that. "virtual" groups need to be implemented by the virtual "parameter codec". The statement on floating server objects (List, Delete etc) was that they have a locked API schema, and we have a mechanism to do that via the scheme which is registering them in a group. , Daniel Smith wrote:


Question: * A taint consists of a key, value, and effect. As an argument here, it is expressed as key=value:effect. * The key must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores, up to %[1]d characters. * The value must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores, up to %[2]d characters. <del> * The effect must be NoSchedule or PreferNoSchedule. </del> <ins> * The effect must be NoSchedule, PreferNoSchedule or NoExecute. </ins> * Currently taint can only apply to node.`) taint_example = templates.Examples(`

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: The help for needs to be updated to reflect the latest change. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
cc I will work on this together with


Question: const sysctlRouteLocalnet = "net/ipv4/conf/all/route_localnet" const sysctlBridgeCallIPTables = "net/bridge/bridge-nf-call-iptables" const sysctlVSConnTrack = "net/ipv4/vs/conntrack" <ins> const sysctlConnReuse = "net/ipv4/vs/conn_reuse_mode" </ins> const sysctlForward = "net/ipv4/ip_forward" const sysctlArpIgnore = "net/ipv4/conf/all/arp_ignore" const sysctlArpAnnounce = "net/ipv4/conf/all/arp_announce"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Service with clusterIP and a domain , the domain resolve to ingress-nginx. Anothor service will request , when use domain it works fine. I think using clusterIP directly will get better performance because it saves some time for dns resolve and ingress-nginx proxy, so I change domain to clusterIP. But the result was not expected, it got worse, very low throughput and high latency. What you expected to happen: On the above situation, clusterIP should have better performance than ingress-nginx. How to reproduce it (as minimally and precisely as possible): mode, do test with ab Anything else we need to know?: I install an ipvs enviroment manually and do some test, and found it possiblly because ipvs will use iptables to realize ! Environment: Kubernetes version (use ): 1.11.1 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): centos 7.2 Kernel (e.g. ): 3.10 Install tools: binary Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I try to change some kernel param and found is related to this issue , default . change to then complete about 3000k request, pretty good performance. So, should I change the param in production? Does it have any side effects
/area ipvs
Can you look at it?
, default 1. change to 0then complte about 3000k request, pretty good performance. Yes, seems you find the key to the problem - it's set to 0 in my environment and I have not found any issue so far(it's about IPVS module behavior and we will not find parameter in old kernels ). BTW, you can also try to set the ipvs conntrack() to 0 - it will improve performance as well.
Thanks for reply.


Question: const sysctlRouteLocalnet = "net/ipv4/conf/all/route_localnet" const sysctlBridgeCallIPTables = "net/bridge/bridge-nf-call-iptables" const sysctlVSConnTrack = "net/ipv4/vs/conntrack" <ins> const sysctlConnReuse = "net/ipv4/vs/conn_reuse_mode" </ins> const sysctlForward = "net/ipv4/ip_forward" const sysctlArpIgnore = "net/ipv4/conf/all/arp_ignore" const sysctlArpAnnounce = "net/ipv4/conf/all/arp_announce"

I am using kubernetes , and after set , clusterIP can not be reachable. Which version do I need ?
oops, have you ever configured Cluster-CIDR or iptables masq-all for kube-proxy?
Yes, I configure Cluster-CIDR
Sorry, Cluster IP still needs the IPVS conntrack and nodeport does not rely on it any more - we are trying to figure it out for Cluster IP.
Got it, thanks!
cc Should we tune the kernel parameter in kube-proxy? Given that many users complain about the performance issue, probably the answer is YES.
There is a nice article describing even more kernel parameters to tune ipvs:
Side effect like this comment: <img width="790" alt="hpaprize" src="https://user-"Note: time in picture below is UTC+8. ! Abviously every time scaledown , response time will increase. Any idea for this?
Thanks for reporting. Have you ever read the article posted by In this article, some ipvs parameters can be tuned, e.g.
yes, I had test these parameters not found in centos 7.2(kernel 3.10) will not affect the result is similar with , will be low throughput According to this issue ,I guess it may be the root cause, so I upgrade my setup from 1.11.1 to 1.12.2. But did not solve this problem, and found a new problem, rs delete failed after scaledown. I had open a new issue to this problem:
/assign
Hi, , I raised a pr for this issue. Please check. : ) Looking forward for your advise.
Could you please check out the HEAD to see if your issues have been resolved?
It's not a big surprise this benchmark performs better with . uses http/1.0 so is creating a new connection for every single request. With IPVS will reuse the existing connection entry if source port matches - sending repeated requests to the same real server rather than actually load balancing. So my guess is you will see an unbalanced load with using and , but it makes the benchmark look better. I'm not sure this benchmark is a realistic scenario. I suggest using a tool like instead which uses persistent connections and see what happens.
Apart from the load balancing issue, it's also possible the overhead of is bad when creating connections in a short period of time.