Question: extensions.Kind("ReplicaSet"):                  &ReplicaSetDescriber{c}, extensions.Kind("NetworkPolicy"):               &ExtensionsNetworkPolicyDescriber{c}, <ins> extensions.Kind("PodSecurityPolicy"):           &PodSecurityPolicyDescriber{c}, </ins> autoscaling.Kind("HorizontalPodAutoscaler"):    &HorizontalPodAutoscalerDescriber{c}, extensions.Kind("DaemonSet"):                   &DaemonSetDescriber{c}, extensions.Kind("Deployment"):                  &DeploymentDescriber{c, versionedClientsetForDeployment(c)},

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): describe psp Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): Environment: Doesn't matter. What happened: doesn't show PSP's fields: What you expected to happen: All PSP fields should be shown. How to reproduce it (as minimally and precisely as possible):
If you're agree with that than I (or :) ) could implement it.
yes, improve the describer. Look at the scc describer for formatting.
any ideas which SIG should own PSP?
In this case it should be sig-cli /sig cli
yeah, this is cli related. I'd go with sig-auth for actual behavior.


Question: value: 512M - name: HEAP_NEWSIZE value: 100M <ins> - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace </ins> volumes: - name: data emptyDir: {}

The namespace seems to be hard coded :
cc:
Thanks. We can improve it after v1 release.


Question: isMigratedOnNode := mpaSet.Has(pluginName) <del> if isMigratedOnNode { installed := false driverName, err := csiMigratedPluginManager.GetCSINameFromInTreeName(pluginName) if err != nil { return isMigratedOnNode, err } for _, driver := range csiNode.Spec.Drivers { if driver.Name == driverName { installed = true break } } if !installed { return true, fmt.Errorf("in-tree plugin %s is migrated on node %s but driver %s is not installed", pluginName, string(nodeName), driverName) } } </del> return isMigratedOnNode, nil }

After the cluster was updated to enable PD CSI Driver, there was a race condition where AD Controller (kube-controller-manager) started to BEFORE the PD CSI Driver was up, thus AD Controller gave us the error Then AD Controller triggered the disk detach because the disk/volume was not populated in the desired state of world. Unfortunately the disk was in use during this detach and got corrupted. Since cluster upgrades can cause the controller manager and CSI drivers to start at different times, the AD controller needs to be more conservative when detaching We have not been able to reproduce this, it came up from a customer. /sig storage /assign <detailsk8s 1.23, CSI migration enabled </details<detailsGKE </details<details</details<details</details<details</details<details</details
/triage accepted
I think there's potentially two changes we should do: 1) for the initial DSW population, we should skip the check that if the csi driver is not installed. 2) in general, we should not skip volumes during dsw initial population if an error is returned. If we had some temporary api server error then that would cause volumes to not get to dsw and similar behavior happen. cc
for a repro, maybe you can try something like: a cluster with csi migration enabled (control plane + nodes) a pod with intree-volume and get it running the csi driver node daemonset. verify the driver is removed from CSINode kube-controller-manager
cc
/cc was trying to repro this, but on GKE with the managed driver which made it complicated.
Yeah maybe gce kube-up would be easier for a repro
If CSINode says that it has migrated GCP PD volume plugin to CSI, then IMO A/D controller should always attach GCP PD volumes to it using CSI, regardless if the CSI driver is present or not. In-tree volume plugin in the kubelet is disabled and can't mount any in-tree volumes anyway. To flip the migration feature gate / upgrade / downgrade kubelet, the cluster admin must drain the node first to make sure everything is detached. A/D controller should not see change of CSINode migration annotations with any volume attached to it.
/priority important-soon
Another option could be to process these volumes anyways but add them to DSOW with migration status as "uncertain", so as they can be re-processed later.


Question: isMigratedOnNode := mpaSet.Has(pluginName) <del> if isMigratedOnNode { installed := false driverName, err := csiMigratedPluginManager.GetCSINameFromInTreeName(pluginName) if err != nil { return isMigratedOnNode, err } for _, driver := range csiNode.Spec.Drivers { if driver.Name == driverName { installed = true break } } if !installed { return true, fmt.Errorf("in-tree plugin %s is migrated on node %s but driver %s is not installed", pluginName, string(nodeName), driverName) } } </del> return isMigratedOnNode, nil }

But surely it looks like might be easiest unless it has any downsides.
There wasn't a node update in this case---there was a control plane update which restarted the PD CSI driver, and it looks like the CSIDriver resource was updated or changed in some way which raced with the kube-controller-manager restart.
Is there anyway we can know if the csi driver is up and running(like a health check ping) before populating DSW?
We use the CSINode registration to determine that. The discussion here I think is concluding that we should not use that as a criteria for adding the volume to DSW. We only need to check that CSI migration is enabled on the node.
+1 we need to be very resilient against races, I think we have to assume a driver is in process of being installed at any point in the AD controller reconcilliation.
/assign sunnylovestiramisu
Reproduced with the steps: In /var/log/kube-controller-
Tested with the and doing the reproduce steps above. The error of does not show up anymore. The error message still get logged in the kube-controller- for debugging purpose: Ran Ran On the GCP console, the disk is used by node: Ran I redeploy the pd-csi-driver and then make sure it is up and running. Ran


Question: if err != nil { return err } <ins> if existingRole.AggregationRule != nil { // the old role already moved to an aggregated role, so there are no custom rules to migrate at this point return nil } </ins> glog.V(1).Infof("migrating %v to %v", existingRole.Name, newName) existingRole.Name = newName existingRole.ResourceVersion = "" // clear this so the object can be created.

/kind bug What happened: I deployed a fresh custom HA kubernetes v1.10.2 cluster with 3 nodes (all controller and worker currently, more workers planned). I then created a namespace called "stummi" and a rolebinding with "user=stummi clusterrole=admin" in this namespace. User stummi (successfully authenticated by TLS client certificate) was not able to list pods or anything else in his namespace: While trying to figure things out, I downgraded to v1.8.11 (a version without cluster role aggregation, that worked) and also tried v1.11 alpha2 (which did not work). I never purged my etcd and I sadly do not have logs from before the different versions. Shortly after posting on slack and before writing this ticket, I dumped the cluster roles in question: Note the rules. What you expected to happen: I expected user stummi to be able to do most things in his namespace. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I posted this problem on slack and helped me and wanted this issue opened. /sig auth The cluster is deployed with ansible and I can give the playbooks when required. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: 3 virtual machines (2 cores, 8GB memory) OS (e.g. from /etc/os-release): Debian Stretch Kernel (e.g. ): Install tools: custom ansible playbooks Others: Please let me know if you need more information. I'm happy to provide everything you need to fix this :)
/assign


Question: "net/http" "net/url" "os" <ins> "path/filepath" </ins> "testing" <ins> "github.com/stretchr/testify/assert" </ins> ) // copied from k8s.io/client-go/transport/round_trippers_test.go

In kubectl v1.8.0+, schema info is cached in the location specified by (defaulting to ), written with world-writeable permissions (rw-rw-rw-). If is specified and pointed at a different location accessible to other users/groups, the written files may be modified by other users/groups and disrupt the kubectl invocation. CVSS score: CVSS:3.0/AV:L/AC:H/PR:L/UI:R/S:U/C:L/I:L/A:N (3.3, low) What versions are affected? kubectl v1.8.0+ What configurations are affected? Invocations that point at world-writeable locations Impact Malformed responses written to the cache directory can disrupt the kubectl invocation Workaround Use the default location in the $HOME directory or point it at a directory that is only accessible to desired users/groups. (original description follows) ==== What happened: The files inside of ".kube/http-cache" are world writeable (rw-rw-rw-). While the default for these files appears to be the home directory, using the "--cache-dir" flag could put these files into a place where world writeable files would allow any user / process to modify the cache files. Modification of the cache files could influence the kubectl utility in a negative way for other users. What you expected to happen: Apply stricter file permissions to the http-cache files. How to reproduce it (as minimally and precisely as possible): Run any generic kubectl command which is successful and then list the cache directory $ kubectl get pods --all-namespaces $ ls -la Anything else we need to know?


Question: "net/http" "net/url" "os" <ins> "path/filepath" </ins> "testing" <ins> "github.com/stretchr/testify/assert" </ins> ) // copied from k8s.io/client-go/transport/round_trippers_test.go

: I estimate this is a low severity security issue with a CVSS score of "3.3 / CVSS:3.0/AV:L/AC:L/PR:L/UI:N/S:U/C:N/I:L/A:N" - Environment: Linux Kubernetes version (use kubectl version):Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-26T12:49:28Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS. Running kube api server in hyperkube. OS (e.g: cat /etc/os-release): NAME="CentOS Linux" VERSION="7.1808 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="7" PRETTYNAME="CentOS Linux 7.1808 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:7" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-7" CENTOSMANTISBTPROJECTVERSION="7" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="7" OSTREEVERSION=7.1808 Kernel (e.g. uname -a): Linux hackit.internal 3.10.0-862.11.6.el7.x8664 SMP Tue Aug 14 21:49:04 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Manual installation. Others: n/a
/sig product-security
Closing this issue. <detailsIn response to : if you think this is serious, send an email thanks!


Question: "net/http" "net/url" "os" <ins> "path/filepath" </ins> "testing" <ins> "github.com/stretchr/testify/assert" </ins> ) // copied from k8s.io/client-go/transport/round_trippers_test.go

</details
Adjusted the CVSS score to to indicate user interaction is required (since they must specify a non-default cache option pointed at a shared writeable location)
There are two places this should be done: PathPerm and FilePerm should be set here: the temp file and final file should have stricter permissions (probably 0660) here:
/cc
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9.
GitHub didn't allow me to assign the following users: yuchengwu. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign I would like to work on release-1.9, as I just made a hot fix for our team's k8s version of 1.9. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
help fixing this is welcome, though it would only be fixed in releases 1.12+, per
/label official-cve-feed (Related to kubernetes/sig-security)


Question: return false, nil, nil }, }, <ins> { name:         "API watch error happen", volID:        "vol-005", attachID:     getAttachmentName("vol-005", testDriver, nodeName), shouldFail:   true, watcherError: true, reactor: func(action core.Action) (handled bool, ret runtime.Object, err error) { if action.Matches("get", "volumeattachments") { return true, makeTestAttachment(getAttachmentName("vol-005", testDriver, nodeName), nodeName, "vol-005"), nil } return false, nil, nil }, }, </ins> } for _, tc := range testCases {

A CSI plugin that the 1.10 controller manager can't detect (which could occur in a number of scenarios, such as when a rollout is being performed or a node is evacuated) seems to retry without backoff. In this case, I have a pod referencing a PVC -PV that references my CSI plugin, but no controller plugin running. The controller is probing every 100ms continuously for a single volume attachment. The controller should backoff. Have not tested with higher numbers of pods, but in a busy system this could cause the controller manager to burn CPU. Also, there isn't actually a great error telling the user why the probing is happening (I knew in this case, but an admin likely has no idea).
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /milestone v1.14 adding it to next milestone as per if it doesn't make it then it can be further moved
as KEP has been moved, unless an exception is made, /milestone v1.15
the linked kep above is unrelated to this issue. This is a bug and I don't think it needs an associated enhancement
sorry, got diverted by the comments on that KEP. /milestone v1.14
Hello from Bug Triage Shadow . I'd like to remind that code freeze is coming in 3 days (Friday, 8th March). Is this still planned to be fixed for 1.14?
We have not been tracking this for 1.14. /assign can you help triage and see if this is still an issue and if there is a quick fix?
I can't reproduce the issue, do you remember what you did? From the logs it seems that Watch channel consistently returns so new is called recursively (beware, 1.10 code, but it's very similar in master): Can it produce fairly stable 100ms delay between calls? Anyway, this is not usual code path, watcher does not return in most cases.


Question: return false, nil, nil }, }, <ins> { name:         "API watch error happen", volID:        "vol-005", attachID:     getAttachmentName("vol-005", testDriver, nodeName), shouldFail:   true, watcherError: true, reactor: func(action core.Action) (handled bool, ret runtime.Object, err error) { if action.Matches("get", "volumeattachments") { return true, makeTestAttachment(getAttachmentName("vol-005", testDriver, nodeName), nodeName, "vol-005"), nil } return false, nil, nil }, }, </ins> } for _, tc := range testCases {

if I read the code correctly, such recursive must end up with timeout - caller does not care about return value and caller's watch probably won't return anything useful. Is there a reason why we don't return error? This will kick in exponential backoff in A/D controller.
clearing the milestone for tracking purposes to let the fix land in its own time /milestone clear
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I found the error would happen when a pv is attahing(volumeattachment has created) and the csi-attacher is deleted at the same time. Then the log would print so frequently (because of recursion).
is this fixed by ?
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "testing" "github.com/imdario/mergo" <ins> "k8s.io/kubernetes/pkg/client/restclient" </ins> clientcmdapi "k8s.io/kubernetes/pkg/client/unversioned/clientcmd/api" )

Found in openshift, but I'm pretty sure the codepaths are the same. That means the token is getting ignored and in-cluster config is being used in preference to flags.
Is a similar issue?
Yep, kind of regression.
Sigh.
Yes, most likely the same cause.
I'm on it.
Unit test :)
Probably will want to cherrypick this into 1.5.
WIP in


Question: targetNumNodes := int32(framework.TestContext.CloudConfig.NumNodes - 1) ginkgo.By(fmt.Sprintf("decreasing cluster size to %d", targetNumNodes)) err = framework.ResizeGroup(group, targetNumNodes) <del> framework.ExpectEqual(err, nil) </del> <ins> framework.ExpectNoError(err) </ins> err = framework.WaitForGroupSize(group, targetNumNodes) <del> framework.ExpectEqual(err, nil) </del> <ins> framework.ExpectNoError(err) </ins> err = e2enode.WaitForReadyNodes(c, framework.TestContext.CloudConfig.NumNodes-1, 10*time.Minute) <del> framework.ExpectEqual(err, nil) </del> <ins> framework.ExpectNoError(err) </ins> targetNodes, err := e2enode.GetReadySchedulableNodes(c) framework.ExpectNoError(err) framework.ExpectEqual(len(targetNodes.Items), int(targetNumNodes))

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: kubernetes/node-problem-detector/node-problem-detector-ci: ci-npd-e2e-node See the Which test(s) are failing: [] NodeProblemDetector [NodeFeature:NodeProblemDetector] [] SystemLogMonitor should generate node condition and events for corresponding errors Since when has it been failing: Dec 03. was at 19:50 PST, was at 21:51 PST. Testgrid link: https://k8s- Reason for failure: Apparently is the culprit. Anything else we need to know: This is likely caused by some library upgrade. Nothing big.
/sig node
/assign
Seems like the failure is caused by . /cc /cc /cc /cc
/priority critical-urgent
The particular bug I care about has been fixed in . I'm gonna still work on submitting , as the same error should also be affecting a few other tests.


Question: assert.Equal(t, c.expected, real, fmt.Sprintf("TestCase[%d]: %s", i, c.desc)) } } <ins> func TestEnsureLoadBalancerDeleted(t *testing.T) { const vmCount = 8 const availabilitySetCount = 4 const serviceCount = 9 tests := []struct { desc              string service           v1.Service expectCreateError bool }{ { desc:    "external service should be created and deleted successfully", service: getTestService("test1", v1.ProtocolTCP, 80), }, { desc:    "internal service should be created and deleted successfully", service: getInternalTestService("test2", 80), }, { desc:    "annotated service with same resourceGroup should be created and deleted successfully", service: getResourceGroupTestService("test3", "rg", "", 80), }, { desc:              "annotated service with different resourceGroup shouldn't be created but should be deleted successfully", service:           getResourceGroupTestService("test4", "random-rg", "1.2.3.4", 80), expectCreateError: true, }, } az := getTestCloud() for i, c := range tests { clusterResources := getClusterResources(az, vmCount, availabilitySetCount) getTestSecurityGroup(az) if c.service.Annotations[ServiceAnnotationLoadBalancerInternal] == "true" { addTestSubnet(t, az, &c.service) } // create the service first. lbStatus, err := az.EnsureLoadBalancer(context.TODO(), testClusterName, &c.service, clusterResources.nodes) if c.expectCreateError { assert.NotNil(t, err, "TestCase[%d]: %s", i, c.desc) } else { assert.Nil(t, err, "TestCase[%d]: %s", i, c.desc) assert.NotNil(t, lbStatus, "TestCase[%d]: %s", i, c.desc) result, err := az.LoadBalancerClient.List(context.TODO(), az.Config.ResourceGroup) assert.Nil(t, err, "TestCase[%d]: %s", i, c.desc) assert.Equal(t, len(result), 1, "TestCase[%d]: %s", i, c.desc) assert.Equal(t, len(*result[0].LoadBalancingRules), 1, "TestCase[%d]: %s", i, c.desc) } // finally, delete it. err = az.EnsureLoadBalancerDeleted(context.TODO(), testClusterName, &c.service) assert.Nil(t, err, "TestCase[%d]: %s", i, c.desc) result, err := az.LoadBalancerClient.List(context.Background(), az.Config.ResourceGroup) assert.Nil(t, err, "TestCase[%d]: %s", i, c.desc) assert.Equal(t, len(result), 0, "TestCase[%d]: %s", i, c.desc) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: controller-manager is stuck in deleting an incorrectly configured LB type service. it cannot be recovered until restarted. What you expected to happen: controller-manager should recover. At least it should allow us to create the service without restarting it. How to reproduce it (as minimally and precisely as possible): put a random IP in above yaml and make sure the resource group doen't exist. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.4 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig azure
I also want to add that the above repro is stuck in 403. It's possible for it to be stuck in 404 as well when the SP has the right permission.
Can you share the log for 404?
Thanks for reporting the issue. /assign Do you mean other services couldn't be configured when this happens?
correct.
FYI, we had the same issue when the Idle Timeout annotation was set to "3" which isn't a supported case. Will this PR also cover this edge case ?
Unfortunately , this is still not fixed. The idle timeout is not 403 or 404. We need another fix for this (and possibly more for other annotations)


Question: "fmt" "io" "net" <ins> "net/http" </ins> "regexp" "time"

All of our external load balancer tests failed for the better part of a day due to the GCE project that our tests run in hitting its quota for number of target pools. While the target pools have been deleted, the fact that we hit our quota indicates that there must be a leak somewhere, and we need to plug it.
For reference, the previous issue was
Could the leakage be related to ? The quota exhaustion occurred about 22 hrs after that.
I'd guess that this is due to GetTCPLoadBalancer returning that the load balancer doesn't exist if the forwarding rule is gone, ignoring the target pool, while the service controller considers its deleting done if the cloud provider says that the load balancer doesn't exist. I that logic to the cloud provider because the other implementations of DeleteTCPLoadBalancer weren't/won't be idempotent in the sense of returning ok if the LB doesn't exist. To fix it, we'll either need to make sure all Delete implementations are idempotent and remove the Get call in the controller, or add a method along the lines of IsTCPLoadBalancerDeleted to ask the cloud provider whether all pieces of it have been completely cleaned up. We can't make GetTCPLoadBalancer check that because it's relied on to tell us whether or not the balancer is fully created.
Copying over my commentary from on why the GKE e2e tests are leaking so many forwarding rules and target pools vs the GCE tests:
Is there a way for the services test to wait for the service controller to clean up the external resources before finishing (without just doing a sleep)?
Not without polling GCE directly for whether the resources still exist, although that may be worth it as a workaround. What would be perfect is that clean-up-all-cloud-provider-resources API method that we talked about a couple months ago.
Can we not perhaps use GCE namespaces or projects as a cheap way of getting this here?
I don't believe so. We can't use projects due to the fact that projects can't be programatically created (they're intended to be long-lived). I'm not aware of anything along the lines of a GCE namespace other than networks, which won't work because target pools and forwarding rules unfortunately don't seem to be scoped to networks.


Question: "fmt" "io" "net" <ins> "net/http" </ins> "regexp" "time"

The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.


Question: "fmt" "io" "net" <ins> "net/http" </ins> "regexp" "time"

I'll periodically check to see whether anything is leaked after this point. I apologize if the deletions affected any running tests - everything looked to only be from GKE and parallel e2e runs.


Question: SSLContext ctx = SSLContext.getInstance("SSL"); ctx.init(null, trustAll, new SecureRandom()); <del> URL url = new URL(host + path + serviceName); </del> <ins> URL url = new URL(proto + host + ":" + port + path + serviceName); </ins> logger.info("Getting endpoints from " + url); HttpsURLConnection conn = (HttpsURLConnection)url.openConnection();

What would you like to be : There is a tab in the k8s perf dashaboard for the scheduler perf benchmarks, but it is showing nothing, we should fix that: http://perf- What this dashboard should be showing is multiple runs per day of We can run it every hour, and summarize the 24 runs of each day to show 50%, 90% and 99% percentile. Why is this needed: Having daily runs of the benchmarks is important to track performance regressions. /sig scheduling /assign /priority important-soon
Ref
/cc
The ci job's config is at And one of failed logs could be found at I tried to reproduce it on my local machine(because I do not have Gloud credentials, I run the task directly in the container): However it works well on my local machine. It is so weird. It will be very appreciated if anyone could help reproduce it with Gloud credentials.
can you try and reach out to the test-infra people? ping them on the slack channel perhaps.
yes, I have tried to get help from , he is busy now. I'll try more.
While working on reproducing this in order to file a bug against test-infra folks I think I found a solution. Will send PRs shortly. /assign
/retitle sig-scalability-benchmarks#scheduler fails consistently due to build issues
/sig scalability Testgrid link: https://k8s- Error logs:
A word about the solution, the job uses the deprecated script which is not supported anymore by test-infra oncall / folks. I've rewritten the job definition to use pod utilities and it started working (will require to get merged first).
Thanks
Thanks !


Question: while true; do if result=$(gcloud compute ${resource} list --project=${PROJECT} ${@:2}); then if [[ ! -z "${GREP_REGEX}" ]]; then <del> result=$(echo "${result}" | grep "${GREP_REGEX}") </del> <ins> result=$(echo "${result}" | grep "${GREP_REGEX}" || true) </ins> fi echo "${result}" return

The last 4 runs of https://k8s- have failed on the and steps with : https://k8s- https://k8s- https://k8s- https://k8s- The GCE version of this test suite appears to be fine. This is blocking the submit queue. CC
Is there a way to see what does /cluster/gce/list- output in the runs... exit status 1 is not very informative
Here is a non-gke failure: https://k8s-
Given that the last thing output from the listing of resource is: I would bet that the defined is failing to find a match from (see )
Hummm I ran it locally against but did not get the same failure
you might be more familiar with list-
This started happening on the 16th and became more regular on the 18th-- do you think the new janitor code might be causing this? More details here:
is giving me exit status 1, let me find which command it is.
Yeah, and have figured it out. Because of the janitor, we no longer have any default firewall rules in these projects (which might be a bug), and so now the is failing. I think in the distant past always printed out a header, even if nothing was found, though it hasn't done that in a long time. Fix for is to the on L52. We may also want to decide if we should be deleting the default firewalls.
We'll want to cherrypick that to the old branches, too.
hummm I'll let it skip default-firewall
Keeping open until we've verified fix


Question: } // Filter based on extender implemented predicate functions. The filtered list is <del> // expected to be a subset of the supplied list. failedNodesMap optionally contains // the list of failed nodes and failure reasons. </del> <ins> // expected to be a subset of the supplied list; otherwise the function returns an error. // failedNodesMap optionally contains the list of failed nodes and failure reasons. </ins> func (h *HTTPExtender) Filter( pod *v1.Pod, nodes []*v1.Node, nodeNameToInfo map[string]*schedulernodeinfo.NodeInfo,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Default scheduler crash if scheduler extender filter returns a not found node What you expected to happen: Default scheduler just should consider the not found node as not an candidate, instead of crash itself (panic). How to reproduce it (as minimally and precisely as possible): cluster only have 1 node: 0.0.0.1 scheduler extender (with nodeCacheCapable enabled) filter accidentally returns a candidate node: 0.0.0.2 scheduler recieved the candidate nodes: 0.0.0.2 from scheduler extender, but got below error: The is SuggestedHost: filteredNodes[0].Name: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scheduling


Question: // IsNotMountPoint determines if a directory is a mountpoint. func (mounter *Mounter) IsNotMountPoint(dir string) (bool, error) { <del> return IsNotMountPoint(mounter, dir) </del> <ins> return isNotMountPoint(mounter, dir) </ins> } // IsLikelyNotMountPoint determines if a directory is not a mountpoint.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There's , and . The former is used by local and nfs volume plugins, while the latter is used by , which is also used by local and nfs volume plugins. What you expected to happen: Can we consolidate these? This is really confusing and could easily be the source of bugs. <!-- DO NOT EDIT BELOW THIS LINE --/kind cleanup
cc
/assign I'd like to clean up code.


Question: kind: Service apiVersion: v1beta1 <del> id: nginxExample </del> <ins> id: nginx-example </ins> # the port that this service should serve on port: 8000 # just like the selector in the replication controller, # but this time it identifies the set of pods to load balance # traffic to. selector: <del> - name: nginx </del> <ins> name: nginx </ins> # the container on each pod to connect to, can be a name # (e.g. 'www') or a number (e.g. 80) containerPort: 80

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: cAdvisor has a option to only collect docker related cgroups. Kubelet stats api now supports allocatable cgroup, /kubepods, for the runtime cgroup and the kubelet cgroup. But with settings, Kubelet stats api can not query pod related cgroups metrics. FYI: What you expected to happen: should be compatible with kubelet stats api.
/cc
/sig node /sig instrumentation
add to cadvisor in A comma-separated list of cgroup path prefix that needs to be collected even docker-only is specified bump cadvisor dep for Kubernetes setting to
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: kind: Service apiVersion: v1beta1 <del> id: nginxExample </del> <ins> id: nginx-example </ins> # the port that this service should serve on port: 8000 # just like the selector in the replication controller, # but this time it identifies the set of pods to load balance # traffic to. selector: <del> - name: nginx </del> <ins> name: nginx </ins> # the container on each pod to connect to, can be a name # (e.g. 'www') or a number (e.g. 80) containerPort: 80

/close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <ins> func createSocketFile(socketDir string) (string, error) { testSocketFile := filepath.Join(socketDir, "mt.sock") // Switch to volume path and create the socket file // socket file can not have length of more than 108 character // and hence we must use relative path oldDir, _ := os.Getwd() err := os.Chdir(socketDir) if err != nil { return "", err } defer func() { os.Chdir(oldDir) }() _, socketCreateError := net.Listen("unix", "mt.sock") return testSocketFile, socketCreateError } </ins> func TestFindExistingPrefix(t *testing.T) { defaultPerm := os.FileMode(0750) tests := []struct {

Subpath unix socket tests can flake because of long socket path. On Linux the path limit should be 108 chars or less. /sig storage


Question: <del> load("@io_bazel_rules_go//go:def.bzl", "go_library") </del> <ins> load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test") </ins> go_library( name = "go_default_library",

Solution: I will add one.
/cc
xref
Thanks for sending a PR so quickly!
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code slush, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: If this issue is not resolved or labeled as by Wed, Nov 22 it will be moved out of the v1.9 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: <del> load("@io_bazel_rules_go//go:def.bzl", "go_library") </del> <ins> load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test") </ins> go_library( name = "go_default_library",

What would you like to be : Once a metric be deprecated, we should update the unit test accordingly if there is any. We need to work out a way for a unit test freely select codebase version, and also should make sure the unit test won't fail in the subsequent release. Why is this needed: Deprecated also needs to be protected. /sig instrumentation /assign


Question: (disk.ManagedDisk != nil && diskURI != "" && strings.EqualFold(*disk.ManagedDisk.ID, diskURI)) { // found the disk klog.V(2).Infof("azureDisk - detach disk: name %q uri %q", diskName, diskURI) <del> disks[i].ToBeDetached = to.BoolPtr(true) </del> <ins> if strings.EqualFold(ss.cloud.Environment.Name, "AZURESTACKCLOUD") { disks = append(disks[:i], disks[i+1:]...) } else { disks[i].ToBeDetached = to.BoolPtr(true) } </ins> bFoundDisk = true break }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: flag is not supported on Azure Stack, should switch to use original way to remove a disk from data disk list, refer to for more details What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


Question: REGISTRY ?= gcr.io/kubernetes-e2e-test-images GOARM=7 QEMUVERSION=v2.9.1 <del> GOLANG_VERSION=1.8.3 </del> <ins> GOLANG_VERSION=1.9.1 </ins> export ifndef WHAT

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature This is a followup on , but for Go 1.9 We should do the same kind of testing now to possibly catch bugs in either Go or Kubernetes when doing this kind of dependency version bump. Ideally we'd have automated tests against Go tip (), but we don't have that at the moment, so we have to do it kind of manually. We should also decide which k8s version we should target. k8s v1.8 or v1.9? What Go versions have been used so far? Kubernetes v1.7.0 - go1.8.3 Kubernetes v1.6.0 - go1.7.5 Kubernetes v1.5.0 - go1.7.4 Kubernetes v1.4.0 - go1.6.3 Kubernetes v1.3.0 - go1.6.2 Kubernetes v1.2.0 - go1.4.2 Seems like the trend has been to stay on the same Go version for two releases and then bump. That'd mean we'd switch to Go1.9 in k8s v1.9 Everyone happy with that? Or is there any reason for us to go start using go1.9 in k8s v1.8? In any case, we should indeed start testing go1.9 out now... cc
Let me know if you find any bugs
FWIW pretty tight to code-freeze I would prefer punting out a cycle fwiw.
I don't think we're that close to code freeze given that we have 5 (of 8) weeks left in this coding cycle But I'm happy to defer this to v1.9 if others think that. I guess the main reason for a bump would be scalability goals, but I'm not sure we have the need to expand above 5000 (and that Go would be the bottleneck in that case) for v1.8
made the Go benchmarks 9% faster with 15% larger binaries. I'm interested to see how that affects our numbers.
mid-stack inlining was not included in Go 1.9. It made the compiler too slow. It's being targeted to Go 1.10 now instead, after some compiler optimization work.
Oh, thanks for the correction!


Question: REGISTRY ?= gcr.io/kubernetes-e2e-test-images GOARM=7 QEMUVERSION=v2.9.1 <del> GOLANG_VERSION=1.8.3 </del> <ins> GOLANG_VERSION=1.9.1 </ins> export ifndef WHAT

Regardless, we need to run the scalability tests against the RC so we can report issues earlier-- the last upgrade was particularly painful because of an HTTP regression that we caught way too late.
yeah. I filed for that now
yeah, my recollection is that the Go releases usually occur too late in our release cadence and end up getting bumped to the next Kubernetes release. (Also we inevitably find something wrong, though testing RCs would help with that. Though I'm not sure anyone has cycles to test RCs and fix all of the issues that do come up.)
Based on our current time frame, it looks like go1.9 support will have to slip to k8s 1.9. is open to dig into and start things testing, but there's a number of things that have to move.. I'll try to document those this time around to help with future go version bumps. So far: bazel build rules needed to be updated to support go1.9 () bazel version possibly needs to be updated? some of our custom bazel rules broke
When done, we should update this:
/priority important-soon
Updating the dependency is blocked pending bazel 0.6.0 - .
Can we upgrade to 0.6.0 now? Does bazel 0.6.0 have multiarch support btw?
We can't update to depend on bazel 0.6.0+ until we can figure out how to support different versions of the bazelbuild and kubekins-e2e images in PR jobs in prow based on what branch is being tested. cc
Noting that we're working on branching jobs, is testing one option here:
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : New functionality.


Question: REGISTRY ?= gcr.io/kubernetes-e2e-test-images GOARM=7 QEMUVERSION=v2.9.1 <del> GOLANG_VERSION=1.8.3 </del> <ins> GOLANG_VERSION=1.9.1 </ins> export ifndef WHAT

</details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I approved this for v1.9, this is something we definitely want in order to keep up with Go (and for improved performance)
we're starting to use the forked images now /cc this should be more feasible very soon Edit: test-infra jobs have also been using bazel 0.6 (now 0.6.1) / a recent rules_go to soak things a bit
We've identified a failing test as a result of the WIP PR to upgrade the codebase to go1.9: The admission webhook test fails with certificate errors under go1.9.1, but runs successfully under go1.8.3. cc: who wanted to be alerted to potential bugs
- please see for a possible fix


Question: // remove all existing c.RemoveStreams(stream0, stream1) <ins> // remove nil stream should not crash c.RemoveStreams(nil) </ins> if len(c.streams) != 0 { t.Fatalf("should not have any streams, has %d", len(c.streams)) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Since upgrading to Kubernetes v1.19.11 (from v1.19.10) we see occasional panics in kubelet related to portforward streams: This issue seem to be introduced with should not panic and crash. We have not been able to successfully force this to happen, but it has happened 41 times in the last 2 weeks across 7 clusters (~100 nodes). Kubernetes version (use ):Cloud provider or hardware configuration: Self managed clusters on AWS. OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig node /cc
Hey thank you for the report! :pray: Uhh, thats unfortunate. The panic occurs there on L101: It looks like because works. The stack trace indicates the same: A on is a no-op, so would not be harmful. does also not seem to be , a crash in would have been visible in the stack trace. It may be possible that something assigned to the stream interface. Which means a check for will return , but accessing its fields is causing a segmentation fault. This does not explain why the crash is not visible in the trace.
With a quick look at the streaming code it's likely to be triggered by a timeout in waiting for streams: If the above case is hit then it goes down to , and if the timeout occurs before then it means that either or And then it attempts to remove the streams where at least one is : So it probably makes sense to check for either before passing to or inside the method where it crashes. This theory lines up with our kubelet logs, where I can see this just before the panic (didn't notice before):
Thank you for the summary working on a fix in
/triage accepted /assign
do you use dockershim within your cluster?
Yes, we still use dockershim.
Good, that's also why the panic happens within the kubelet.
Just to make sure I understand the scope of this issue, this only affects things like or other things that use the API, correct?
Exactly :+1:


Question: if _, err := dc.coreClient.CoreV1().Pods(tc.pod.Namespace).Create(ctx, tc.pod, metav1.CreateOptions{}); err != nil { t.Fatalf("Failed to create pod: %v", err) } <ins> dc.clock.Sleep(tc.timePassed) </ins> if err := dc.informerFactory.Core().V1().Pods().Informer().GetIndexer().Add(tc.pod); err != nil { t.Fatalf("Failed adding pod to indexer: %v", err) } <del> dc.clock.Sleep(tc.timePassed) </del> <ins> diff := "" </ins> if err := wait.Poll(100*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) { <del> return dc.stalePodDisruptionQueue.Len() == 0, nil </del> <ins> pod, err := dc.kubeClient.CoreV1().Pods(tc.pod.Namespace).Get(ctx, tc.pod.Name, metav1.GetOptions{}) if err != nil { t.Fatalf("Failed getting updated pod: %v", err) } diff = cmp.Diff(tc.wantConditions, pod.Status.Conditions, cmpopts.IgnoreFields(v1.PodCondition{}, "LastTransitionTime")) return diff == "", nil </ins> }); err != nil { <del> t.Fatalf("Failed waiting for worker to sync: %v", err) } pod, err := dc.kubeClient.CoreV1().Pods(tc.pod.Namespace).Get(ctx, tc.pod.Name, metav1.GetOptions{}) if err != nil { t.Fatalf("Failed getting updated pod: %v", err) } diff := cmp.Diff(tc.wantConditions, pod.Status.Conditions, cmpopts.IgnoreFields(v1.PodCondition{}, "LastTransitionTime")) if diff != "" { t.Errorf("Obtained pod conditions (-want,+got):n%s", diff) </del> <ins> t.Fatalf("Failed waiting for worker to sync: %v, (-want,+got):n%s", err, diff) </ins> } }) }

/kind failing-test <!-- If this is a flake, please add: /kind flake --/kind flake <!-- Please assign a SIG using: /sig SIG-NAME --/sig testing apps
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you PTAL, it seems this test was in It doesn't seem to have a high rate of failure , but seems flakes are legit
/assign
side note: it's really hard to make it fail in a fast machine :) I had to temporarily add a sleep in the reconciler.
have you tried Jordan's instructions compiling and running with ? for integration you should have an external etcd instance
I haven't. Good to know for next time :)


Question: k8s-app: nvidia-gpu-device-plugin addonmanager.kubernetes.io/mode: Reconcile spec: <ins> selector: matchLabels: k8s-app: nvidia-gpu-device-plugin </ins> template: metadata: labels:

This issue is here to satisfy PR requirements of issue association. A lot of times we need to do version bumps for minor bug fixes that don't deserve a separate issue in the kubernetes/kubernetes repo. /kind bug /sig node
/lifecycle frozen
all the linked PRs are emrged and no activity since more than 3 years /close
Closing this issue. <detailsIn response to : all the linked PRs are emrged and no activity since more than 3 years /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: containers: - name: fluentd-elasticsearch image: gcr.io/google_containers/fluentd-elasticsearch:1.5 <ins> resources: limits: cpu: 100m </ins> env: - name: "FLUENTD_ARGS" value: "-qq"

During debugging , I noticed that one of services pods: monitoring and logging pods stuck at pending sometimes forever: I logged into the node, and found the container is running happily on the node: Looks like status is never reported from kubelet. I checked kubelet log, then I found: Why we cannot delete mirror pod? and Created a new one? Because this regression, once we run into such state, the rest of test won't be triggered, and eventually timeout / failed.
checks those service pods in ready state, and prevent the test to run once it is pending. cc/ I think this is the real issue I talked to you earlier about PodStatus.
I spoke to offline. For some reason, kubelet thinks the spec of the mirror pod does not match the static pod locally. It then decides to delete the mirror pod and recreate it, repeatedly. As a result of that, it slows down (due to the QPS limit?) reporting statuses of other pods. This can be reproduced by running the e2e "Shell" test. I am looking into why the specs would suddenly mismatch. Did any recent PR modified the pod spec in apiserver?
The recently introduced default cpu share limit was not there in the static pod, causing the difference between its spec and the mirror pod's.
Seeing this as well. Makes the whole cluster slow and unusable for me. Sent from my iPhone
Is there a desired fix for this?
said she would add default resource limits to the static pod manifest files. If we go for that, I think it's reasonable to fail (not run the static pod) if resource limit is not specified. Also, the resource limits set in the mirror pods are not meaningful because kubelet creates containers based on the spec of the static pod. In general, static pods do not subject to any defaulting values applied at the apiserver.
Pretty sure that this is fixed by . Certainly testing against head since then works well for me. What version are you testing?
was fixing node status, and the problem had been around for a while. This issue is about pod status, which is still unfixed on head. These two are very different, and my cluster did include the fix :) This issue will slow down kubelet at times, but if you get lucky and the pod status (running) get sent out in time, the tests will still pass.


Question: containers: - name: fluentd-elasticsearch image: gcr.io/google_containers/fluentd-elasticsearch:1.5 <ins> resources: limits: cpu: 100m </ins> env: - name: "FLUENTD_ARGS" value: "-qq"

There is no way the issue is fixed by that PR. I am still seeing the issue on HEAD. The system gets worse the longer its up. I agree with the analysis of the issue in source PR Sent from my iPhone
I am on this one, and here is proposed solution in my mind so far (not final yet since I haven't put too much thoughts into it yet): 1) For all our static pods I am going to update them with some initial cpu limit for now. This should quickly stop flakiness caused by this particular issue. 2) Next step is that for static pods, if there is no initial cpu limit, kubelet should refuse to run. Before I do this, I need to send an announcement to our container-vm users about this potential breakage for the in the future. Another potential fix is we can ignore resource changes when comparing if a mirror pod is updated or not. I think is proposed a similiar one earlier. But I had a concern with it: this is defeat the initial purpose we created mirror pods and sync back to master. So if one made some resource changes on the static pods, you have to always ssh-ed into the node to see if the new value applied or not. Also eventually (post 1.0) those mirrored pod's resource limit is going to be used by scheduler when calculating if user pods are fit or not, we need them to be accurated. Another concern is in the code implementation itself to distinuguish mirror pods future from rest of pods. I guess this shouldn't be the big issue here. Any more suggestions here? One puzzle I had is that why master component pods doesn't have issue at all. I manually restart kube-apiserver pod, still doesn't have the issue.
Depending on time we could just remove the default limit from the limit range as a stop gap. Sent from my iPhone
Can you just bypass defaulting at the apiserver when you create mirror pods? I can't think of any other solution that works in general and is easily maintainable. Approach 2 you described in the previous entry works if we are defaulting cpu limit, but breaks as soon as we default anything else.


Question: containers: - name: fluentd-elasticsearch image: gcr.io/google_containers/fluentd-elasticsearch:1.5 <ins> resources: limits: cpu: 100m </ins> env: - name: "FLUENTD_ARGS" value: "-qq"

Same problem with ignoring resource differences when comparing mirror pod to static pod -- as long as defaulting only touches resources, that works, but if we change it to touch other stuff, then it breaks.
Commit I had can quickly fix current bleeding issue. Currently we only have master component pods and fluentd pod are file based by default. Master component pods doesn't create any issue now because mirror pods aren't created at the first place (due to recent change made by Also file-based pods will have issue only when its spec doesn't have initial value, and need to update (delete miss matched mirror pod, and recreate a new one). Please note that with my commit, everything is working even I manually update its cpu limit from 100m to 200m, there is no issue at all, and query pod shows accurate result with the new limit too: How badly we request all users using file-based pod feature providing default cpu limit for now? Eventually we want to get rid of file-based pods completely, also encouraging the user provides the initial resource limit is aligning with our direction in a long run anyway.
Just to clarify, what I was suggesting was for a "real fix" not the temporary measure to stop the bleeding (which you already have in your PR). But what do you think of what I suggested as the longer-term fix? My point is that today, we default CPU, but in the future, we may default anything. It seems that the most maintainable fix is to just not do defaulting when creating mirror pod from static pod. Is that possible? I wonder if the reason manual update works (what you described in your last comment) is because we don't go through the defaulting when you do an update, only a create? (Though that sounds like a bug.) IMO we don't need CPU default right now, especially since we already have kubelet flag for limiting pods per machine (though unfortunately it's currently set to 100 because we haven't reconfigured the tests that intentionally run lots of pods per node, to override the flag. we can change the default to 30 and add an override mechanism if this is important.)
Comparing a config file with the live state and expecting them to match is doomed. The recommended approach is described by . Kubelet could store the config as an annotation in the mirror pod and compare with that.


Question: containers: - name: fluentd-elasticsearch image: gcr.io/google_containers/fluentd-elasticsearch:1.5 <ins> resources: limits: cpu: 100m </ins> env: - name: "FLUENTD_ARGS" value: "-qq"

Does the scheduler favor nodes with the fewest pods even if they don't specify resources? LimitRange does cause that to happen.
Storing the config as an annotation sounds good to me. Filed
Although this issues was closed, it still appears in my environment
Same here, this happens from time to time in our environments, is there a way to debug to find the root cause?


Question: }, { manifest: KubeProxyDaemonSet, <del> data: struct{ ImageRepository, Arch, Version, ImageOverride, ClusterCIDR, MasterTaintKey, CloudTaintKey string }{ </del> <ins> data: struct{ ImageRepository, Arch, Version, ImageOverride, ExtraParams, ClusterCIDR, MasterTaintKey, CloudTaintKey string }{ </ins> ImageRepository: "foo", Arch:            "foo", Version:         "foo", ImageOverride:   "foo", <ins> ExtraParams:     "foo", </ins> ClusterCIDR:     "foo", MasterTaintKey:  "foo", CloudTaintKey:   "foo",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently, IPVS proxier does not run any e2e test cases. What you expected to happen: I think it's important to create test suite for IPVS in upstream CI which will help a lot in moving IPVS existing alpha. Anything else we need to know?: Let's work together. /sig network
Self /assign
There are to ways to do this work: , but doesn't support any kube-proxy config right now, so I had this pr , and also, need support it too, see this pr: kubernetes/kubernetes-anywhere build cluster. See this pr: kubernetes/test-infra The second way is much easier and faster, but I was told they gonna deprecated the , so in the long term, we still need and 's support. If anybody have better way or have some suggestion about this, please let me know, I'll be really appreciate, Thanks.


Question: Description string `json:"description,omitempty"` // Versions are versions for this third party object <del> // +optional </del> Versions []APIVersion `json:"versions,omitempty"` }

FYI. Currently the field of TPR is marked as which is wrong, actually if you try to create a TPR with empty versions, it will cause api server panic. Versions should be necessary field of TPR, we should validate this somehow.


Question: nodeTime = time.Now() bootTime, err = util.GetBootTime() <del> gomega.Expect(err).To(gomega.BeNil()) </del> <ins> framework.ExpectNoError(err) </ins> // Set lookback duration longer than node up time. // Assume the test won't take more than 1 hour, in fact it usually only takes 90 seconds.

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: kubernetes/node-problem-detector/node-problem-detector-ci: ci-npd-e2e-node See the Which test(s) are failing: [] NodeProblemDetector [NodeFeature:NodeProblemDetector] [] SystemLogMonitor should generate node condition and events for corresponding errors Since when has it been failing: Dec 03. was at 19:50 PST, was at 21:51 PST. Testgrid link: https://k8s- Reason for failure: Apparently is the culprit. Anything else we need to know: This is likely caused by some library upgrade. Nothing big.
/sig node
/assign
Seems like the failure is caused by . /cc /cc /cc /cc
/priority critical-urgent
The particular bug I care about has been fixed in . I'm gonna still work on submitting , as the same error should also be affecting a few other tests.


Question: genericConfig.DisabledPostStartHooks.Insert(rbacrest.PostStartHookName) } <del> webhookAuthResolver := func(delegate webhookconfig.AuthenticationInfoResolver) webhookconfig.AuthenticationInfoResolver { return webhookconfig.AuthenticationInfoResolverFunc(func(server string) (*rest.Config, error) { if server == "kubernetes.default.svc" { return genericConfig.LoopbackClientConfig, nil } ret, err := delegate.ClientConfigFor(server) if err != nil { return nil, err } if proxyTransport != nil && proxyTransport.Dial != nil { ret.Dial = proxyTransport.Dial } return ret, err }) </del> <ins> webhookAuthResolverWrapper := func(delegate webhookconfig.AuthenticationInfoResolver) webhookconfig.AuthenticationInfoResolver { return &webhookconfig.AuthenticationInfoResolverDelegator{ ClientConfigForFunc: func(server string) (*rest.Config, error) { if server == "kubernetes.default.svc" { return genericConfig.LoopbackClientConfig, nil } return delegate.ClientConfigFor(server) }, ClientConfigForServiceFunc: func(serviceName, serviceNamespace string) (*rest.Config, error) { if serviceName == "kubernetes" && serviceNamespace == "default" { return genericConfig.LoopbackClientConfig, nil } ret, err := delegate.ClientConfigForService(serviceName, serviceNamespace) if err != nil { return nil, err } if proxyTransport != nil && proxyTransport.Dial != nil { ret.Dial = proxyTransport.Dial } return ret, err }, } </ins> } pluginInitializers, err := BuildAdmissionPluginInitializers( s, client, sharedInformers, serviceResolver, <del> webhookAuthResolver, </del> <ins> webhookAuthResolverWrapper, </ins> ) if err != nil { return nil, nil, nil, nil, nil, fmt.Errorf("failed to create admission plugin initializer: %v", err)

We are getting the following error when the apiserver connects the webhook on localhost (configured via URL). We should only use the SSL tunnel for the connections to nodes when the webhooks are running as services. /sig api-machinery /kind bug /assign
/close


Question: // recent start error or not, State should be Terminated, LastTerminationState should be retrieved from second latest // terminated status. { <del> []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, []*kubecontainer.ContainerStatus{ </del> <ins> containers: []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, statuses: []*kubecontainer.ContainerStatus{ </ins> { Name:     "without-reason", State:    kubecontainer.ContainerStateExited,

is this a Node bug? cc as next oncall to follow up on this.
This is a node bug due to init containers
I may need help debugging - I see:
So I think the problem is in the error case - we're not properly recording that the start failed, and the next time we come in we somehow think the first container succeeded. I'll add some debugging to see if we can't recreate it.
Let's discuss on instead.


Question: // recent start error or not, State should be Terminated, LastTerminationState should be retrieved from second latest // terminated status. { <del> []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, []*kubecontainer.ContainerStatus{ </del> <ins> containers: []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, statuses: []*kubecontainer.ContainerStatus{ </ins> { Name:     "without-reason", State:    kubecontainer.ContainerStateExited,

Kubelet issues a docker inspect call after creating a container, but for very short lived containers there is a window in which they could exit before being inspected in the docker manager. If the container PID is 0, we should check the ExitCode and other relevant fields and handle them appropriately. [1] This is the suspected cause of several flakes: /cc
looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: I'd expect the second one to not have run in this case? Node logs:
Init containers have an implicit assumption that container statuses include results of previous SyncPod invocations. If that's not the case we may need to fix that (or at least mitigate it). On May 18, 2016, at 8:28 PM, Prashanth B wrote: looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: May 18 15:58:33.380: INFO: pod-init--1d4b-11e6-80a4- started at 2016-05-18T15:58:04-07:00 (2+1 container statuses recorded) May 18 15:58:33.380: INFO: Init container init1 ready: false, restart count 0 May 18 15:58:33.380: INFO: Init container init2 ready: true, restart count 0 I'd expect the second one to not have run in this case? Node logs: — You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub
This should be true. Is there any evidence that suggests the opposite? only shows conflicting view of whether init1 has run.
If the container exits already and it's not an infra container, kubelet should not error out. Although even if it does, I failed to see how this could affect the readiness of the init container without checking the code further.


Question: // recent start error or not, State should be Terminated, LastTerminationState should be retrieved from second latest // terminated status. { <del> []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, []*kubecontainer.ContainerStatus{ </del> <ins> containers: []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, statuses: []*kubecontainer.ContainerStatus{ </ins> { Name:     "without-reason", State:    kubecontainer.ContainerStateExited,

Each syncPod invocation, we inspect the list of init container in order looking for lastTerminationState. There could be a bug in that logic, or if we didn't get the full status, we might make a bad decision. When determining which container to run next, we always look for the last init container in the chain that has success, then run the next container (unless the pod is RestartNever). , Yu-Ju Hong wrote:
Saw this flake again. How do we verify that container status is not lost between syn pods?
... between calls to syncPods
The second sync clearly saw the init1 container, and decided to launch init2. See "InitContainersToKeep" in the message below.
containers to keep can include failed containers. , Yu-Ju Hong wrote:
Discussed on slack. In summary, the problem has to do with how we convert the kubecontainer.PodStatus to the api.PodStatus. If a container is terminated, the function makes the decisions: the pod restart policy is "Always", and there is a "failure reason" recorded for the given container, this means the pod worker tried to (re)start the container but failed. Set the state to "Waiting" and populate the reason. , the container state is simply "terminated". Init containers don't obey the restart policy like regular containers, and the logic will need to be amended to work with them. is working on a PR. In addition to that, kubelet should not error out if a user container exited before it could apply the oom score (which is best effort in this case). fixes that.
Ahh, thanks for digging into the issue. This is the exact the concern I had when I reviewed the initial implementation of init container, which blocks pr for a while. I even asked to help me verify this case, unfortunately we didn't reproduce the issue back then. :-(
I wish I'd seen the flakes in the runs prior to merge.
We didn't notice this until we started running the tests against docker v1.11.1, which is much faster and hit the race condition differently.
I suppose that's a good thing...
It is hard to reproduce this.


Question: // recent start error or not, State should be Terminated, LastTerminationState should be retrieved from second latest // terminated status. { <del> []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, []*kubecontainer.ContainerStatus{ </del> <ins> containers: []api.Container{{Name: "without-reason"}, {Name: "with-reason"}}, statuses: []*kubecontainer.ContainerStatus{ </ins> { Name:     "without-reason", State:    kubecontainer.ContainerStateExited,

The extra check of last terminated container in your initial code makes me nervous, but we couldn't reproduce it due to that small race windows until we have a faster docker running on the node. :-)
I got "Pods should invoke init containers on a RestartAlways pod" flake (referenced from ): Logs:
All the fixes went in. Please reopen this issue if you see any more flakes.


Question: } } <del> func k8sVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { </del> <ins> func isPkiVolumeMountNeeded() bool { // On some systems were we host-mount /etc/ssl/certs, it is also required to mount /etc/pki. This is needed // due to symlinks pointing from files in /etc/ssl/certs into /etc/pki/ if _, err := os.Stat("/etc/pki"); err == nil { return true } return false } func pkiVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { </ins> return api.Volume{ Name: "pki", VolumeSource: api.VolumeSource{ <ins> // TODO(phase1+) make path configurable HostPath: &api.HostPathVolumeSource{Path: "/etc/pki"}, }, } } func pkiVolumeMount() api.VolumeMount { return api.VolumeMount{ Name:      "pki", MountPath: "/etc/pki", } } func k8sVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { return api.Volume{ Name: "k8s", VolumeSource: api.VolumeSource{ </ins> HostPath: &api.HostPathVolumeSource{Path: kubeadmapi.GlobalEnvParams.KubernetesDir}, }, }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? controller /etc/ssl/certs kubeadm /etc/ssl/certs Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 5 custom built master I also use a custom built hyperkube image instead of the single app images for apiserver, controller, ... Environment: Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): CentOS 7.2 Kernel (e.g. ): Linux ma-kub8ms0 3.10.0-327.36.3.el7.x8664 SMP Mon Oct 24 16:09:20 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: deployed with Azure RM template and ansible What happened: controller is not able to do https, including calls to the Azure RM API What you expected to happen: https should work as usual How to reproduce it (as minimally and precisely as possible): Use kubeadm to install kubernetes on CentOS exec into the controller and try "curl " Anything else do we need to know: It looks like caused this issue. On CentOS, I see these files on the host OS: I suspect the symlinks to /etc/pki/... being the problem here, as they are not available inside the hyperkube Debian image. I also tried to verify this with It seems to not be based on Debian and thus gives a different error: I see 2 possible solutions: of host mounting /etc/ssl/certs into the controller pod, the ca certificates could be installed into the image.


Question: } } <del> func k8sVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { </del> <ins> func isPkiVolumeMountNeeded() bool { // On some systems were we host-mount /etc/ssl/certs, it is also required to mount /etc/pki. This is needed // due to symlinks pointing from files in /etc/ssl/certs into /etc/pki/ if _, err := os.Stat("/etc/pki"); err == nil { return true } return false } func pkiVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { </ins> return api.Volume{ Name: "pki", VolumeSource: api.VolumeSource{ <ins> // TODO(phase1+) make path configurable HostPath: &api.HostPathVolumeSource{Path: "/etc/pki"}, }, } } func pkiVolumeMount() api.VolumeMount { return api.VolumeMount{ Name:      "pki", MountPath: "/etc/pki", } } func k8sVolume(cfg *kubeadmapi.MasterConfiguration) api.Volume { return api.Volume{ Name: "k8s", VolumeSource: api.VolumeSource{ </ins> HostPath: &api.HostPathVolumeSource{Path: kubeadmapi.GlobalEnvParams.KubernetesDir}, }, }

This is already done for hyperkube and would work out of the box when the host mount would be omitted. could try to figure out if /etc/pki should also be host mounted and add this to the controller manifest if required. I'd be happy to provide a fix by myself, I just need to know which solution you would prefer.
Same problem seems to apply to the apiserver. Not sure however if it actually does https calls.
implements solution 2.
This issue was moved to kubernetes/kubeadm


Question: Namespace: pod.Namespace, } tests := []struct { <del> containers                   []api.Container statuses                     []*kubecontainer.ContainerStatus reasons                      map[string]error oldStatuses                  []api.ContainerStatus expectedState                map[string]api.ContainerState </del> <ins> containers    []api.Container statuses      []*kubecontainer.ContainerStatus reasons       map[string]error oldStatuses   []api.ContainerStatus expectedState map[string]api.ContainerState // Only set expectedInitState when it is different from expectedState expectedInitState            map[string]api.ContainerState </ins> expectedLastTerminationState map[string]api.ContainerState }{ // For container with no historical record, State should be Waiting, LastTerminationState should be retrieved from // old status from apiserver. { <del> []api.Container{{Name: "without-old-record"}, {Name: "with-old-record"}}, []*kubecontainer.ContainerStatus{}, map[string]error{}, []api.ContainerStatus{{ </del> <ins> containers: []api.Container{{Name: "without-old-record"}, {Name: "with-old-record"}}, statuses:   []*kubecontainer.ContainerStatus{}, reasons:    map[string]error{}, oldStatuses: []api.ContainerStatus{{ </ins> Name:                 "with-old-record", LastTerminationState: api.ContainerState{Terminated: &api.ContainerStateTerminated{}}, }}, <del> map[string]api.ContainerState{ </del> <ins> expectedState: map[string]api.ContainerState{ </ins> "without-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: startWaitingReason, }},


Question: Namespace: pod.Namespace, } tests := []struct { <del> containers                   []api.Container statuses                     []*kubecontainer.ContainerStatus reasons                      map[string]error oldStatuses                  []api.ContainerStatus expectedState                map[string]api.ContainerState </del> <ins> containers    []api.Container statuses      []*kubecontainer.ContainerStatus reasons       map[string]error oldStatuses   []api.ContainerStatus expectedState map[string]api.ContainerState // Only set expectedInitState when it is different from expectedState expectedInitState            map[string]api.ContainerState </ins> expectedLastTerminationState map[string]api.ContainerState }{ // For container with no historical record, State should be Waiting, LastTerminationState should be retrieved from // old status from apiserver. { <del> []api.Container{{Name: "without-old-record"}, {Name: "with-old-record"}}, []*kubecontainer.ContainerStatus{}, map[string]error{}, []api.ContainerStatus{{ </del> <ins> containers: []api.Container{{Name: "without-old-record"}, {Name: "with-old-record"}}, statuses:   []*kubecontainer.ContainerStatus{}, reasons:    map[string]error{}, oldStatuses: []api.ContainerStatus{{ </ins> Name:                 "with-old-record", LastTerminationState: api.ContainerState{Terminated: &api.ContainerStateTerminated{}}, }}, <del> map[string]api.ContainerState{ </del> <ins> expectedState: map[string]api.ContainerState{ </ins> "without-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: startWaitingReason, }},


Question: Namespace: pod.Namespace, } tests := []struct { <del> containers                   []api.Container statuses                     []*kubecontainer.ContainerStatus reasons                      map[string]error oldStatuses                  []api.ContainerStatus expectedState                map[string]api.ContainerState </del> <ins> containers    []api.Container statuses      []*kubecontainer.ContainerStatus reasons       map[string]error oldStatuses   []api.ContainerStatus expectedState map[string]api.ContainerState // Only set expectedInitState when it is different from expectedState expectedInitState            map[string]api.ContainerState </ins> expectedLastTerminationState map[string]api.ContainerState }{ // For container with no historical record, State should be Waiting, LastTerminationState should be retrieved from // old status from apiserver. { <del> []api.Container{{Name: "without-old-record"}, {Name: "with-old-record"}}, []*kubecontainer.ContainerStatus{}, map[string]error{}, []api.ContainerStatus{{ </del> <ins> containers: []api.Container{{Name: "without-old-record"}, {Name: "with-old-record"}}, statuses:   []*kubecontainer.ContainerStatus{}, reasons:    map[string]error{}, oldStatuses: []api.ContainerStatus{{ </ins> Name:                 "with-old-record", LastTerminationState: api.ContainerState{Terminated: &api.ContainerStateTerminated{}}, }}, <del> map[string]api.ContainerState{ </del> <ins> expectedState: map[string]api.ContainerState{ </ins> "without-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: startWaitingReason, }},


Question: Namespace: pod.Namespace, } tests := []struct { <del> containers                   []api.Container statuses                     []*kubecontainer.ContainerStatus reasons                      map[string]error oldStatuses                  []api.ContainerStatus expectedState                map[string]api.ContainerState </del> <ins> containers    []api.Container statuses      []*kubecontainer.ContainerStatus reasons       map[string]error oldStatuses   []api.ContainerStatus expectedState map[string]api.ContainerState // Only set expectedInitState when it is different from expectedState expectedInitState            map[string]api.ContainerState </ins> expectedLastTerminationState map[string]api.ContainerState }{ // For container with no historical record, State should be Waiting, LastTerminationState should be retrieved from // old status from apiserver. { <del> []api.Container{{Name: "without-old-record"}, {Name: "with-old-record"}}, []*kubecontainer.ContainerStatus{}, map[string]error{}, []api.ContainerStatus{{ </del> <ins> containers: []api.Container{{Name: "without-old-record"}, {Name: "with-old-record"}}, statuses:   []*kubecontainer.ContainerStatus{}, reasons:    map[string]error{}, oldStatuses: []api.ContainerStatus{{ </ins> Name:                 "with-old-record", LastTerminationState: api.ContainerState{Terminated: &api.ContainerStateTerminated{}}, }}, <del> map[string]api.ContainerState{ </del> <ins> expectedState: map[string]api.ContainerState{ </ins> "without-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: startWaitingReason, }},


Question: rs := &apps.ReplicaSet{} rc := &api.ReplicationController{} <ins> extGroup := schema.GroupVersion{Group: "apps", Version: "v1"} extCodec := legacyscheme.Codecs.LegacyCodec(extGroup) </ins> <del> extGroup := testapi.Apps defaultGroup := testapi.Default </del> <ins> defaultGroup := schema.GroupVersion{Group: "", Version: "v1"} defaultCodec := legacyscheme.Codecs.LegacyCodec(defaultGroup) </ins> fuzzInternalObject(t, schema.GroupVersion{Group: "apps", Version: runtime.APIVersionInternal}, rs, rand.Int63())

Execute make test WHAT=./pkg/api/testing/ And it failed with information: /kind failing-test
/sig api-machinery
what's the urgency of fixing this? Is it a release blocker?
i don't see it as a release blocker. but the problem is in client-go, so it's important to get fixed.


Question: kubeClient := testKubelet.fakeKubeClient kubeClient.ReactFn = testclient.NewSimpleFake(&api.NodeList{Items: []api.Node{ { <del> ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}, </ins> Spec:       api.NodeSpec{}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{

Looks like some kubelet unit tests perform a lookupIp which ends up depending on the health of the local resolver. Specifically has this failing: Which leads to the test in: And we recently swapped out the node names from testnode to 127.0.0.1, so probably the culprit?
/ref
Interesting. I did swap out the node names in the unit test as part of to be 127.0.0.1, party because I thought that it wouldn't have any problems resolving during unit tests.
/cc


Question: kubeClient := testKubelet.fakeKubeClient kubeClient.ReactFn = testclient.NewSimpleFake(&api.NodeList{Items: []api.Node{ { <del> ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}, </ins> Spec:       api.NodeSpec{}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{

cc
What is the environment where you are running the tests? They pass for me on my mac, a linux box, and shippable seems to be passing them as well.
I couldn 't reproduce the failure either.
I'm out right now; i'll try them when i get back. For the record, I run hack/test- Paul Morie wrote:
Ahem FTR This acronym is going to go quad-plat Paul Morie wrote:
I ran on both platforms and saw the tests pass.
I'm still seeing it. Won't have bandwidth to dig in personally until tomorrow.
seeing a similar failure after a git pull
I knew I wasn't crazy
environment: fedora 21, golang: 1.4, docker: 1.6
I think I see the problem. When I set the node name to in the test I was assuming it would parse as a valid ip address and not end up at the LookupIP call. But only the node name (and not the kubelet hostname) is being set (except in my new test where I ) so it's trying to do a lookup. Do you think you could test just changing the default hostname from "testnode" to "127.0.0.1" and see if that fixes it for you?
Trying now
I still get failures but no panic / threaddump.
gist coming
this looks like again?
Good that it isn't panicing any longer. I can dig into the test failures tomorrow (I think it should be relatively simple to resolve).
cool, thanks for looking
Passes in master now for me. Closing.
FYI, they will break again once goes in, and then we can merge my PR to see if that fixes it for you.
:+1:
is in -- are these tests breaking again?
I just rebased which I think will fix them.
They are breaking again.
is on it.


Question: # # We specify -X to avoid a race condition that can cause minion failure to # install.  See https://github.com/saltstack/salt-bootstrap/issues/270 <del> curl -L http://bootstrap.saltstack.com | sh -s -- -X </del> <ins> curl -L --connect-timeout 20 --retry 6 --retry-delay 10 http://bootstrap.saltstack.com | sh -s -- -X </ins>

Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
this appears to have always been broken, I'm going to disable the job for now. https://k8s-
Failed: https://k8s- Run so broken it didn't make JUnit output!


Question: if err := os.MkdirAll(dest, 0o755); err != nil { return err } <del> return utils.WithProcfd(c.root, m.Destination, func(procfd string) error { if err := mount(m.Source, m.Destination, procfd, "cgroup2", uintptr(m.Flags), m.Data); err != nil { // when we are in UserNS but CgroupNS is not unshared, we cannot mount cgroup2 (#2158) if errors.Is(err, unix.EPERM) || errors.Is(err, unix.EBUSY) { src := fs2.UnifiedMountpoint if c.cgroupns && c.cgroup2Path != "" { // Emulate cgroupns by bind-mounting // the container cgroup path rather than // the whole /sys/fs/cgroup. src = c.cgroup2Path } err = mount(src, m.Destination, procfd, "", uintptr(m.Flags)|unix.MS_BIND, "") if c.rootlessCgroups && errors.Is(err, unix.ENOENT) { err = nil } } return err } return nil </del> <ins> err = utils.WithProcfd(c.root, m.Destination, func(procfd string) error { return mount(m.Source, m.Destination, procfd, "cgroup2", uintptr(m.Flags), m.Data) </ins> }) <ins> if err == nil || !(errors.Is(err, unix.EPERM) || errors.Is(err, unix.EBUSY)) { return err } // When we are in UserNS but CgroupNS is not unshared, we cannot mount // cgroup2 (#2158), so fall back to bind mount. bindM := &configs.Mount{ Device:           "bind", Source:           fs2.UnifiedMountpoint, Destination:      m.Destination, Flags:            unix.MS_BIND | m.Flags, PropagationFlags: m.PropagationFlags, } if c.cgroupns && c.cgroup2Path != "" { // Emulate cgroupns by bind-mounting the container cgroup path // rather than the whole /sys/fs/cgroup. bindM.Source = c.cgroup2Path } // mountToRootfs() handles remounting for MS_RDONLY. // No need to set c.fd here, because mountToRootfs() calls utils.WithProcfd() by itself in mountPropagate(). err = mountToRootfs(bindM, c) if c.rootlessCgroups && errors.Is(err, unix.ENOENT) { // ENOENT (for `src = c.cgroup2Path`) happens when rootless runc is being executed // outside the userns+mountns. // // Mask `/sys/fs/cgroup` to ensure it is read-only, even when `/sys` is mounted // with `rbind,ro` (`runc spec --rootless` produces `rbind,ro` for `/sys`). err = utils.WithProcfd(c.root, m.Destination, func(procfd string) error { return maskPath(procfd, c.label) }) } return err </ins> } func doTmpfsCopyUp(m *configs.Mount, rootfs, mountLabel string) (Err error) {

opencontainers/runc has fixed three CVEs: Latest runc is pulled by k/k Check runc version in file No response n/a n/a <details</details<details</details<details</details<details</details
/assign /sig security architecture release
/triage accepted


Question: import ( "fmt" <del> "k8s.io/kubernetes/pkg/api" </del> "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/pkg/runtime" )

I started the federation-apiserver and went to and got the following output: Expected: I should have got an empty v1.ServiceList Any idea on what we are doing wrong? Have you guys seen this error?
I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote:
Deep copy generation was slightly broken as well - you may want to double check that they are registered. On May 23, 2016, at 5:50 PM, Daniel Smith wrote: I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote: — You are receiving this because you are on a team that was mentioned. Reply to this email directly or view it on GitHub
I got the same error with latest branch, as I was swamping on service controller, so I use a previous version and worked around it.
deep copies were indeed missing, but adding them hasnt fixed the issue. Will debug more. is my PR to add the deep copies cc Please feel free to file issues for any such problem you face. We need to fix all these issues. Its good if we know about them early. Thanks!
fixes this


Question: framework.ExpectNoError(framework.VerifyServeHostnameServiceUp(cs, ns, host, podNames1, svc1IP, servicePort)) // Restart apiserver <del> initialRestartCount, err := framework.GetApiserverRestartCount(cs) Expect(err).NotTo(HaveOccurred(), "failed to get apiserver's restart count") </del> By("Restarting apiserver") <del> if err := framework.RestartApiserver(cs.Discovery()); err != nil { </del> <ins> if err := framework.RestartApiserver(cs); err != nil { </ins> framework.Failf("error restarting apiserver: %v", err) } <del> By("Waiting for apiserver to be restarted") if err := framework.WaitForApiserverRestarted(cs, initialRestartCount); err != nil { framework.Failf("error while waiting for apiserver to be restarted: %v", err) } </del> By("Waiting for apiserver to come up by polling /healthz") if err := framework.WaitForApiserverUp(cs); err != nil { framework.Failf("error while waiting for apiserver up: %v", err)

This test is failing in both gce serial suite: http://k8s- /sig network /priority failing-test /priority critical-urgent /kind bug /status approved-for-milestone cc /assign PTAL
xref
/milestone v1.10
this also fails in http://k8s-
Failing to connect to Services at all. I can't even ping pod IPs:
So I can ping pods same-node and cross-node right up until it kills the apiserver. Rebooting fixes a given node. The apiserver demonstrably comes back - I can query it for info...
- I could use help - have to shift attention soon.
FWIW, I am testing on a Calico-enabled node
I'll take a look at this as well.
Weird, now even after a reboot, this condition is sticky. I have a cluster in this state, if you want to poke.
ugh, now it is resolved.
I just repro'ed without the e2e - have a cluster, proved ping works, killed apiserver ping fails.
Interesting - docker shows the pods not running, but apiserver says they are alive.
Update: docker sees no running pods. API server thinks kubelet is running them. kubelet is getting "" from apiserver, even after kubelet restart.
Update and recap: Bring up a cluster Run some pods Prove that pods are running and services work pkill -9 apiserver Wait. Watch kubelets "lose" pods that were assigned to them Observe that pods are not running and services do not work Observe that apiserver says the pods are running pkill -9 kubelet Observe that it is still broken kubectl delete one pod Observe that kubelet is fixed for that one pod, but not others Q: Why is kubelet not getting a full listing when it restarts?
What exactly does it mean for it to be fixed for a deleted pod?
Kubelet started running the pod that was deleted - it got the update for that one.
The cluster I was testing was a week or 2 old. I built a cluster from HEAD and I can not repro - said the same. Is this test running against head of master?
I also ran the test manually against my new cluster and it passed
FWIW I was also unable to reproduce this yesterday on a cluster from HEAD.


Question: framework.ExpectNoError(framework.VerifyServeHostnameServiceUp(cs, ns, host, podNames1, svc1IP, servicePort)) // Restart apiserver <del> initialRestartCount, err := framework.GetApiserverRestartCount(cs) Expect(err).NotTo(HaveOccurred(), "failed to get apiserver's restart count") </del> By("Restarting apiserver") <del> if err := framework.RestartApiserver(cs.Discovery()); err != nil { </del> <ins> if err := framework.RestartApiserver(cs); err != nil { </ins> framework.Failf("error restarting apiserver: %v", err) } <del> By("Waiting for apiserver to be restarted") if err := framework.WaitForApiserverRestarted(cs, initialRestartCount); err != nil { framework.Failf("error while waiting for apiserver to be restarted: %v", err) } </del> By("Waiting for apiserver to come up by polling /healthz") if err := framework.WaitForApiserverUp(cs); err != nil { framework.Failf("error while waiting for apiserver up: %v", err)

Did some investigations and had some findings for this specific failure: Traced down a bit on the history, the first occurrence was https://k8s- From the , seems suspicious because it modified . Also found reported a relevant symptom: The fix () for that was merged a day ago and may explained why we can't repro from HEAD anymore.
nice, is this issue reproducible without the fix ()?
Sorry should have had confirmed this, bringing up a cluster now.
Quick update: I was able to repro the same failure () from HEAD with reverted. Though this test is now flaky on some other errors: https://k8s- https://k8s- They all seem to failed while communicating with apiserver.
Thanks ! So fixed the issue and I am curious about what introduced the issue. Do you happen to know when did this test start to fail? Just trying to find out what went wrong in between. We have some other possibly related issues so finding out the root cause of this one would be super helpful.
/assign
The first occurrence (for that exact failure) I noticed was https://k8s-, which happened on 2018-02-15 09:49 PST.
You all rock! I see a couple greens on it now, but still mostly red?
The test is now flaky mostly on creating an exec pod after restarting apiserver. Some sample failures: To elaborate a bit more, a snippet of the test procedure is: ssh to master and (sending a SIGTERM) Keep polling on /healthz on apiserver until 200 is returned. Create an exec pod for later steps (and failed on various erros --- EOF, connection refused, apiserver terminating). According to above test failures, my suspicion is that the test is assuming the apiserver is restarted too soon. It is possible that apiserver captured SIGTERM and did a graceful termination, so that polling on /healthz may succeed even in the termination phase. I'm not surprised trying to create a pod via a terminating apiserver will hit those errors. Could anyone confirme the SIGTERM theory?
Spoke with folks, sounds like after SIGTERM apiserver does wait a bit for remaining requests to finish. I will take a stab to refine the test. cc
Sent .
Test is now passing on gce but failing on gke.


Question: framework.ExpectNoError(framework.VerifyServeHostnameServiceUp(cs, ns, host, podNames1, svc1IP, servicePort)) // Restart apiserver <del> initialRestartCount, err := framework.GetApiserverRestartCount(cs) Expect(err).NotTo(HaveOccurred(), "failed to get apiserver's restart count") </del> By("Restarting apiserver") <del> if err := framework.RestartApiserver(cs.Discovery()); err != nil { </del> <ins> if err := framework.RestartApiserver(cs); err != nil { </ins> framework.Failf("error restarting apiserver: %v", err) } <del> By("Waiting for apiserver to be restarted") if err := framework.WaitForApiserverRestarted(cs, initialRestartCount); err != nil { framework.Failf("error while waiting for apiserver to be restarted: %v", err) } </del> By("Waiting for apiserver to come up by polling /healthz") if err := framework.WaitForApiserverUp(cs); err != nil { framework.Failf("error while waiting for apiserver up: %v", err)

Apparently we don't have a way to trigger apiserver restart on gke at this moment. Sending a PR to skip this test on gke.
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Commented too quick. On gke we do a same-version master upgrade. Will send another fix :/