Question: Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"), }, }, <ins> "tcpBeLiberal": { SchemaProps: spec.SchemaProps{ Description: "tcpBeLiberal, if true, kube-proxy will configure conntrack to run in liberal mode for TCP connections and packets with out-of-window sequence numbers won't be marked INVALID.", Default:     false, Type:        []string{"boolean"}, Format:      "", }, }, </ins> "udpTimeout": { SchemaProps: spec.SchemaProps{ Description: "udpTimeout is how long an idle UDP conntrack entry in UNREPLIED state will remain in the conntrack table (e.g. '30s'). Must be greater than 0 to set.",

From I can now directly ping POD-IPs, but due to the problem in this issue, I can't make a TCP connection: The POD with address runs on . A similar setup as above can be used to verify a PR. Now, there is a simple work-around: use proxy-mode=ipvs The update with the rule never made it to the ipvs proxier (a rather common case :cry: ), but in this case it may save your day. The same as above with proxy-mode=ipvs; The server on 5001 responds with the POD name.
Yeah, Our production environment also solved this problem by switching to IPVS mode. In IPVS mode, there is no the DROP rule.
The sysctl solves the problem in , but not the problem in this issue. Those problems are kind of related, but rather distant.
The same affects outgoing traffic if packets are not masqueraded to the node-ip, as should be a common setup for ipv6, please see For Calico I think you can setup BGP for clusterCIDRs and route incoming packets to PODs to the correct K8s node, instead of using ECMP. This would solve the problem, and save the extra hop.
I met the same issue. KUBE-FORWARD drop my packages.
temptatively solution
The issue fixed by , I'll close it, you can feel free to open it if have any questions. /close
Closing this issue. <detailsIn response to : The issue fixed by , I'll close it, you can feel free to open it if have any questions. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: aggregatedManager.RemoveAPIService(s.Name) } <del> require.True(t, waitForEmptyQueue(testCtx.Done(), aggregatedManager)) </del> <ins> require.True(t, waitForQueueComplete(testCtx.Done(), aggregatedManager)) </ins> response, _, parsed := fetchPath(aggyService, "") if response.StatusCode != 200 {

pull-kubernetes-unit sometimes fails with data race. TestDirty I think the test was 3 months ago with the flaking state. There are two issues. First, updating and reading a bool-type variable is not thread-safe. and there is also another problem checking the state before completely syncing it. No response /sig api-machinery
/assign
/triage accepted


Question: // EndpointPort is a tuple that describes a single port. message EndpointPort { <del> // The name of this port (corresponds to ServicePort.Name). </del> <ins> // The name of this port.  This must match the 'name' field in the // corresponding ServicePort. </ins> // Must be a DNS_LABEL. // Optional only if one port is defined. // +optional

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When running a cluster service and manually created endpoints, the port name set on the cluster definition acts as an implicit selector - suitable end-points do not get mapped to the cluster service. What you expected to happen: Port name shouldn't be an implicit selector. How to reproduce it (as minimally and precisely as possible): Using the following spec file "": kubectl create -f , then describe the service. Output shows the endpoint as being correctly mapped Now run kubectl delete -f the and uncomment the name: foobar Re-run the kubectl create -f command. Now, checking the service gives the output below (end point isn't mapped): Anything else we need to know?: Adding the name: foobar to the endpoint definition makes the end point to be mapped again into the cluster. I.e. there is an implicit matching of the end-point and cluster port definitions. This is either a bug or a very surprising and undocumented feature Environment: kubectl version Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-03-25T02:52:13Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-01-16T18:14:49Z", GoVersion:"go1.10.7", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): root cat /etc/os-release NAME="Ubuntu" VERSION="18.04.1 LTS (Bionic Beaver)" Kernel (e.g.


Question: // EndpointPort is a tuple that describes a single port. message EndpointPort { <del> // The name of this port (corresponds to ServicePort.Name). </del> <ins> // The name of this port.  This must match the 'name' field in the // corresponding ServicePort. </ins> // Must be a DNS_LABEL. // Optional only if one port is defined. // +optional

): uname -a Linux 4.15.0-51-generic -Ubuntu SMP Wed May 15 14:27:21 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/remove-sig api-machinery /sig network The name mapping to the endpoints is intentional, but needs documentation
/triage unresolved
/assign
It may be under-documented, but it is not UNdocmented.
Also
working on a PR to fix doc


Question: } func (c *Cacher) terminateAllWatchers() { <del> glog.Warningf("Terminating all watchers from cacher %v", c.objectType) </del> c.Lock() defer c.Unlock() <del> c.watchers.terminateAll() </del> <ins> c.watchers.terminateAll(c.objectType) </ins> } func (c *Cacher) isStopped() bool {

Multiple, for example: Since See above The test tolerations are now to strict: No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think it's from if you check the cri-o versions the succeessful one was and the failing was
Working on a fix in
cc
/reopen need to bump in kube as well to fully get the fix
Reopened this issue. <detailsIn response to : /reopen </details


Question: return volumeLimits, nil } <del> if vmSizeList == nil { ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) defer cancel() result, err := az.VirtualMachineSizesClient.List(ctx, az.Location) if err != nil || result.Value == nil { klog.Errorf("failed to list vm sizes in GetVolumeLimits, plugin.host: %s, location: %s", plugin.host.GetHostName(), az.Location) return volumeLimits, nil } vmSizeList = result.Value } </del> volumeLimits = map[string]int64{ <del> util.AzureVolumeLimitKey: getMaxDataDiskCount(instanceType, vmSizeList), </del> <ins> util.AzureVolumeLimitKey: getMaxDataDiskCount(instanceType), </ins> } return volumeLimits, nil } <del> func getMaxDataDiskCount(instanceType string, sizeList *[]compute.VirtualMachineSize) int64 { if sizeList == nil { return defaultAzureVolumeLimit } </del> <ins> func getMaxDataDiskCount(instanceType string) int64 { </ins> vmsize := strings.ToUpper(instanceType) <del> for _, size := range *sizeList { if size.Name == nil || size.MaxDataDiskCount == nil { klog.Errorf("failed to get vm size in getMaxDataDiskCount") continue } if strings.ToUpper(*size.Name) == vmsize { klog.V(12).Infof("got a matching size in getMaxDataDiskCount, Name: %s, MaxDataDiskCount: %d", *size.Name, *size.MaxDataDiskCount) return int64(*size.MaxDataDiskCount) } </del> <ins> maxDataDiskCount, exists := maxDataDiskCountMap[vmsize] if exists { klog.V(12).Infof("got a matching size in getMaxDataDiskCount, VM Size: %s, MaxDataDiskCount: %d", vmsize, maxDataDiskCount) return maxDataDiskCount </ins> } <ins> klog.V(12).Infof("not found a matching size in getMaxDataDiskCount, VM Size: %s, use default volume limit: %d", vmsize, defaultAzureVolumeLimit) </ins> return defaultAzureVolumeLimit }

<!-- Please only use this template for submitting enhancement requests -- What would you like to be : related to Why is this needed: /assign /sig azure
here is the way about how to get MaxDataDiskCount: and then build kubelet, copy kubelet to agent node and then get MaxDataDiskCount data: There are 207 VM sizes totally.
Update: this command could get VM size list easily:


Question: "//pkg/kubelet/util/ioutils:go_default_library", "//pkg/security/apparmor:go_default_library", "//pkg/util/hash:go_default_library", <ins> "//pkg/util/parsers:go_default_library", </ins> "//pkg/util/term:go_default_library", "//vendor/github.com/blang/semver:go_default_library", "//vendor/github.com/docker/docker/api/types:go_default_library",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: We have a private registry configured (where all images including the sandbox one), are hosted. The registry credentials are being pulled by one of k8s credentials providers () With Kubernetes 1.7 sandbox image failed to be pulled with the following error message: Looks like with this only creds from docker config are being evaluated on the sandbox image pull What you expected to happen: Sandbox image pull should be attempted using credentials returned by not just the basic docker credential provider, but other credentials providers registered in the system. How to reproduce it (as minimally and precisely as possible): 1) Register private repo credentials with one of the k8s credential providers (we've tried with rancher). Configure the to use the image from the the private repo. 2) Make sure you are not logged in to this private repo on the kubelet host 3) Try to create a pod. Sandbox image pull will fail with an error mentioned above. Anything else we need to know?: Non-sandbox images from the same private repo were able to download correctly using private repo credentials provider. Environment: Kubernetes version (use ): 1.7.2 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node Just tested the same use case against k8s 1.5.x, it worked fine there and sandbox image was pulled using the credentials passed by the credential provider, not from docker config file. So it might have regressed in 1.7.x (haven't tested it on 1.6, so can't tell for sure)
Confirmed a bug introduced in dockershim. Will file a PR to fix this.


Question: "//pkg/kubelet/util/ioutils:go_default_library", "//pkg/security/apparmor:go_default_library", "//pkg/util/hash:go_default_library", <ins> "//pkg/util/parsers:go_default_library", </ins> "//pkg/util/term:go_default_library", "//vendor/github.com/blang/semver:go_default_library", "//vendor/github.com/docker/docker/api/types:go_default_library",

Could you also verify whether fixed your problem?
tested against your commits cherry-picked on top of 1.8 beta, the fix seems to work
thanks.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: } } <del> // PingCommand is the type to hold ping command. type PingCommand string const ( // IPv4PingCommand is a ping command for IPv4. IPv4PingCommand PingCommand = "ping" // IPv6PingCommand is a ping command for IPv6. IPv6PingCommand PingCommand = "ping6" ) </del> // CheckConnectivityToHost launches a pod to test connectivity to the specified // host. An error will be returned if the host is not reachable from the pod. // // An empty nodeName will use the schedule to choose where the pod is executed. <del> func CheckConnectivityToHost(f *Framework, nodeName, podName, host string, pingCmd PingCommand, timeout int) error { </del> <ins> func CheckConnectivityToHost(f *Framework, nodeName, podName, host string, port, timeout int) error { </ins> contName := fmt.Sprintf("%s-container", podName) command := []string{ <del> string(pingCmd), "-c", "3", // send 3 pings "-W", "2", // wait at most 2 seconds for a reply </del> <ins> "nc", "-vz", </ins> "-w", strconv.Itoa(timeout), host, <ins> strconv.Itoa(port), </ins> } pod := &v1.Pod{

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: Since when has it been failing: Ever Reason for failure: Vm on Azure with no public IP assigned aren't able to perform a command to the internet. Anything else we need to know: The test (see ) tries to test the internet connection for a pod by pinging 8.8.8.8. But azure does not provide ICMP support for VM without a public IP therefore this test cannot succeed.(see ) A fix would be to change the test method to a tcp request instead of a ping request (the test already performs a http get to in its BeforeSuite (see ))
/sig network /sig testing /sig azure
you may refer to (Pods don't have connectivity to external networks )
cc WDYT about the proposal to change this from an ICMP call? We probably need input from sig-architecture as well since this as well /sig architecture
/triage unresolved
An alternative is to issue a TCP/UDP request to 8.8.8.8 on port 53.
ICMP is dropped by Azure for security reasons. This is intended. Can use curl as a replacement. I will fix the corresponding test.
SGTM
Can you please review the related PR? I'll in the PR again. Thanks!


Question: // ServicePort contains information on service's port. type ServicePort struct { // The name of this port within the service. This must be a DNS_LABEL. <del> // All ports within a ServiceSpec must have unique names. This maps to // the 'Name' field in EndpointPort objects. </del> <ins> // All ports within a ServiceSpec must have unique names. When considering // the endpoints for a Service, this must match the 'name' field in the // EndpointPort. </ins> // Optional if only one ServicePort is defined on this service. // +optional Name string `json:"name,omitempty" protobuf:"bytes,1,opt,name=name"`


Question: // ServicePort contains information on service's port. type ServicePort struct { // The name of this port within the service. This must be a DNS_LABEL. <del> // All ports within a ServiceSpec must have unique names. This maps to // the 'Name' field in EndpointPort objects. </del> <ins> // All ports within a ServiceSpec must have unique names. When considering // the endpoints for a Service, this must match the 'name' field in the // EndpointPort. </ins> // Optional if only one ServicePort is defined on this service. // +optional Name string `json:"name,omitempty" protobuf:"bytes,1,opt,name=name"`


Question: <del> 1.2 </del> <ins> 1.3 </ins>

E2eNode Suite.[sig-node] Node Performance Testing [Serial] [Slow] Run node performance testing with pre-defined workloads TensorFlow workload From the testgrid, it has been 1/2 flaky since the start of history (03/20/2022) and has been failing constantly (0/2) since 04/01/2022 No response /sig node
/triage accepted /priority important-soon
The failing test seems to be expecting GPU on machine and the image seems to be broken. I tried the image manually: Logs:
/assign as discussed at SIG Node CI meeting, can you please take a look? I see that pod is now running for couple minutes and then stops. I wonder if this is something environment related - maybe pod was evicted or something. I doesn't seem to be able to repro the failure of the pod locally
GitHub didn't allow me to assign the following users: xmcqueen. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is no longer a flake. It has failed every run since the start of June. Locally this test runs perfectly with a legit correct local run of the test suite code, doing the full tensorflow run of 300 epochs. The failure in the testgrid job seems to be fast (173 sec to build and fail, locally its 160 sec before it starts the tensorflow pod). I wonder if there's a blocked port or something from the data initializing python program. It needs access to: . Here's the command it runs: Is it feasible to add debugging code, like a pre-check for that http port? It would be good to at least force it to fail with a meaningful error message.
Note that HEAD is now at image version 1.2, and it is still failing per . The successful local runs of the test mentioned above used image version 1.2.


Question: <del> 1.2 </del> <ins> 1.3 </ins>

Locally image version 1.1 does produce the cuda/GPU error shown above by :
More info is needed to find the RC for this test failure. I've put in a PR to get the test framework to preserve the container logs when the pod fails.
The pod logs show that it is timing out on an http connection, and it looks like it's the one where it tries to download the test data set for the tensorflow run. You can see it . Search down to . The data_download script in the image attempts to pull down a dataset from: . That url is live and healthy, so they might be blocking the gcloud.
This test depending on that external data doesn't look quite right :/
I'm discussing this test with sig-testing. I'll post an update later.
Talked to from sig-testing: he's contacting the remote to see if they are blocking us, because it looks like they are is pulling the script from a that is So I see three choices: upgrade this thing to a more recent edition delete it grab everything that's needed to run this and keep it in-house <- clear winner here what do you think: Maybe the folks over at sig-testing have a place to store the artifacts in-house? They are not massive (about 10MB see below) Size of data about 10MB: the tools and libs for the tensorflow run: the training and eval files for the testrun:
Here's a relevant issue in a related k8s repo about using/not using docker registries: Registries used in k/k CI should be on Kubernetes Community infra
looks good for the subject of putting up a new docker image containing the deps. This seems feasible and effective. I'd prefer it to deleting the test. I'll discuss it a bit more with a few others, but this seems like the way to go forward with this. TBS, I plan to take the above listed test requirements, and build a docker image containing all that is needed, and reset the test to pull the new docker image.
I've got some Dockerfile changes to fix this. I've tested it locally with no network and it does the full 300 epochs. PR coming soon.
Is it worth pushing up 182 files to get this green? It is about 6M of python, and 5M of a test dataset.


Question: <del> 1.2 </del> <ins> 1.3 </ins>

Its all stashed away in the to get the docker image to contain the prereqs. It is selected subdirectories of the quite old archived version (v1.9.0) of the code from here: current version v2.7.2 and two small test data sets from here:
OK I got it down to 43 files, including the 2 datasets. That's the drivers and the datasets and the libs they need. It all gets packed into the docker image.
quick question - is there another repository that is under sig-node that can host this image?
To be clear, the test's docker image is currently in a repo, the : This PR adds artifacts to an existing docker image in the above repo.
I've cancelled the above PR. I'll try to get something else going.
There has been a decent amount of discussion in slack relating to handling the test's required artifacts. The solution that seems best is to get some practical version of the test requirements into locally owned and controlled storage, so the endpoints that provide access to the assets will at least be under internal control. This is not code, and is not under development. It is a static dataset, that is now stored in an offsite archive, and will not require maintenance. The datasets are published with the "": (see bottom right) "This allows for the sharing and adaptation of the datasets for any purpose, provided that the appropriate credit is given." Here are some links to the dataset and where its hosted: https://archive-
I'd like to get the dataset that holds the test data (about 650k) hosted in GCS where I can get it via http. With that change it is expected that the test will go green. The failures show it is failing when it tries to download the artifact from the uncontrolled remote endpoint. Somebody, please help me get this artifact into an internally controlled server. Thanks
Was the idea of downloading this only on image creation in dockerfile rejected?
Thanks for the reminder I did not fairly evaluate that proposal. I just jumped to my earlier proposal. Thinking about this proposal I see that it would mean downloading the test data for inclusion into the docker image, while the image's test driver tooling is built to pull down the dataset itself from a remote.


Question: <del> 1.2 </del> <ins> 1.3 </ins>

That is a simple change to handle, while simply re-hosting the artifact on GCS would require only changing the endpoint. Also building the docker would probably still require network access to the artifact, which would probably require us to put it somewhere anyway. I assume, putting it into the docker image would best be done by downloading the test data artifact from somewhere. Would we then be storing the artifact twice - once in the docker image, and once in a locally controlled endpoint? If the docker build server has unrestricted access to the artifact, this proposal would be good. I don't have access to the servers to run tests. However, I could put up an RB for the Dockerfile to try it out. This is probably worth the effort. I'll get an PR going for trying to pull down the remote test-data artifact from the image build server, and bunde it into the docker image, and I'll link it to this ticket.
thank you. We do not change the image for a long time, so downloading it once would likely be enough. Once it is self-contained, it is even better for the continuous perf runs in tests.
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Currently the image builds for the first arch (amd64) but fails when it enters the cross compiling phase (arm64) and specifically when it tries to do apt installs from within the qemu-*-static emulator: The other images in node-perf fail to build in the same way. The jessie-dnsutils image build fails to build in the same way. I have tried pulling in newer base images up to buster, but it did not fix the problem. -Start focusing on the version of qemu. - qemu is version locked in the Makefile and has not changed in two years. There was an and the images were building at that time.


Question: <del> 1.2 </del> <ins> 1.3 </ins>

Try to find clues over there.
I tried pinning many relevant versions of from but none helped. Here's a clue: This ugly hack gets it past the observed error:
Note the previous PR was missing an important change, but I do not see any feasible way that could explain the current problem with failing, so I have not felt any urgency to put up a PR for it. FTR I need to bump the VERSION. I did not bump the version in I thought maybe the CI would bump it automatically.
There's an image version in . Its still pulling the old version and failing. Bump it from 1.2 to 1.3. So one more commit is needed.
The new image is present and runs perfectly. Tested it on local linux and in google cloud: It ran to completion. Note the new image URL. Its NOT the same as the old one:
podspec references the image Here's the . tf-wide-deep's url is built The above has staging in it. . Its already built and pushed to staging. It must be promoted when this new version number goes out.
FTR: My "testing" above did not catch the current problem because I did not run the full e2e_test binary. I only tested the image via a . AND I should have read way back at the beginning.
Local run of : has to change too.
Local test using the published 1.3 image run the standard way was successful with the above: ran the test successfully. So promote the image per bump the image version in the e2e test manifest change the
After the image promotion is merged, there's STILL one more commit coming: bump the image version in the e2e test manifest change the
its promoted and is found at the expected url local containerd cleared out local test was rerun pulling from the new prod registry local test using that new registry passes
its fixed. the test is green again.


Question: return ConformanceContainer{containers[0], cc.Client, pod.Spec.RestartPolicy, pod.Spec.Volumes, pod.Spec.NodeName, cc.Namespace, cc.podName}, nil } <del> func (cc *ConformanceContainer) GetStatus() (api.ContainerStatus, api.PodPhase, error) { </del> <ins> func (cc *ConformanceContainer) IsReady() (bool, error) { </ins> pod, err := cc.Client.Pods(cc.Namespace).Get(cc.podName) if err != nil { <del> return api.ContainerStatus{}, api.PodUnknown, err </del> <ins> return false, err </ins> } <ins> return api.IsPodReady(pod), nil } func (cc *ConformanceContainer) GetPhase() (api.PodPhase, error) { pod, err := cc.Client.Pods(cc.Namespace).Get(cc.podName) if err != nil { return api.PodUnknown, err } return pod.Status.Phase, nil } func (cc *ConformanceContainer) GetStatus() (api.ContainerStatus, error) { pod, err := cc.Client.Pods(cc.Namespace).Get(cc.podName) if err != nil { return api.ContainerStatus{}, err } </ins> statuses := pod.Status.ContainerStatuses <del> if len(statuses) != 1 { return api.ContainerStatus{}, api.PodUnknown, errors.New("Failed to get container status") </del> <ins> if len(statuses) != 1 || statuses[0].Name != cc.Container.Name { return api.ContainerStatus{}, fmt.Errorf("unexpected container statuses %v", statuses) </ins> } <del> return statuses[0], pod.Status.Phase, nil </del> <ins> return statuses[0], nil </ins> } func (cc *ConformanceContainer) Present() (bool, error) {

Slow conformance test has caused issues like We should make runtime/image conformance test run faster without breaking the test coverage, possibly by: Shorten consistently check interval . Remove redundant test cases such as ... /cc


Question: } type metricSet struct { <del> latencies        *prometheus.HistogramVec latenciesSummary *prometheus.SummaryVec </del> <ins> latencies                  *prometheus.HistogramVec deprecatedLatencies        *prometheus.HistogramVec latenciesSummary           *prometheus.SummaryVec deprecatedLatenciesSummary *prometheus.SummaryVec </ins> } func newMetricSet(name string, labels []string, helpTemplate string, hasSummary bool) *metricSet { <del> var summary *prometheus.SummaryVec </del> <ins> var summary, deprecatedSummary *prometheus.SummaryVec </ins> if hasSummary { summary = prometheus.NewSummaryVec( prometheus.SummaryOpts{ Namespace: namespace, Subsystem: subsystem, Name:      fmt.Sprintf("%s_admission_latencies_seconds_summary", name), <del> Help:      fmt.Sprintf(helpTemplate, "latency summary"), </del> <ins> Help:      fmt.Sprintf(helpTemplate, "latency summary in seconds"), MaxAge:    latencySummaryMaxAge, }, labels, ) deprecatedSummary = prometheus.NewSummaryVec( prometheus.SummaryOpts{ Namespace: namespace, Subsystem: subsystem, Name:      fmt.Sprintf("%s_admission_latencies_milliseconds_summary", name), Help:      fmt.Sprintf("(Deprecated) "+helpTemplate, "latency summary in milliseconds"), </ins> MaxAge:    latencySummaryMaxAge, }, labels,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Admission metrics name is and , the units from metrics name are , but actually the return metrics are in . What you expected to happen: Return metrics in true units. How to reproduce it (as minimally and precisely as possible): curl apiserver:8080/metrics Anything else we need to know?: Environment: Kubernetes version (use ): master branch Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign
/sig api-machinery
/cc
What about these? - -
metrics had been renamed to . And these unit are all fixed to seconds by .


Question: <ins> apiVersion: "v1beta1" id: "monitoring-heapsterController" kind: "ReplicationController" desiredState: replicas: 1 replicaSelector: name: "heapster" podTemplate: desiredState: manifest: version: "v1beta1" id: "monitoring-heapsterController" containers: - name: "heapster" image: "kubernetes/heapster:v0.5" labels: name: "heapster" uses: "monitoring-influxdb" labels: name: "heapster" </ins>

examples/monitoring/influx-grafana- and friends NEED to be replaced by a replication controller ASAP. As it stands they are bad examples. Also, they should probably move to contrib/monitoring to match contrib/logging and contrib/dns


Question: return devs.deviceIds } <ins> // Populates allocatedResources with the device resources allocated to the specified <podUID, contName>. func (pdev podDevices) addContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Union(devices.deviceIds) } } // Removes the device resources allocated to the specified <podUID, contName> from allocatedResources. func (pdev podDevices) removeContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Difference(devices.deviceIds) } } </ins> // Returns all of devices allocated to the pods being tracked, keyed by resourceName. func (pdev podDevices) devices() map[string]sets.String { ret := make(map[string]sets.String)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area hw-accelerators What happened: In the context of the device plugin, when using init containers in your pod spec, the current behavior will not reuse devices between init containers and regular containers. This is problematic as Pod scheduling is currently wrong: , the current predicate algorithm rightly assumes the maximum number of devices in use at one time is . However, the pod predicate ran on the node is the sum of devices used by regular and init containers. Meaning that your pod can bounce back between the scheduler and a node infinitely. Also given that many of the extended resources are typically more expensive that the built-in ones like CPU, Memory and Storage this bug has very real financial drawbacks in terms of resource usage (e.g: using 4 GPUs on a pod that only uses 2 GPUs at a time). Furthermore, this unexpected behavior was caught during implementation (but we did not catch the scheduling implications) and decision was taken to ignore it for the first implementation. We need to document it for 1.9 What you expected to happen: Devices can be reused by Init Containers Correct resource accounting In terms of the current behavior of the Allocate call to the device plugin, I'd expect the allocate call to be done before every Init containers start and once at the end of the last Init container (for regular containers). /assign
Yep, I realized the inconsistency between the predicate behavior of scheduler and that of node too. Having said that, considering we treat a pod as a whole when predicating currently, it's kind of difficult to reuse the runtime informations between init containers and regular containers. The real crux of the matter might be whether the scenario of requiring extended resources in init containers is common, if so, the issue should be addressed ASAP.
If I remembered correctly, the current design to improve DP includes a reconciliation in admit phase of Pod, it may be able to fix Allocate call for
can you point me to said design ? I'm not sure that the Allocate call for init containers is planned to be fixed as described.
Can you guys please point to the code where is happening. Looks to me that same logic is executed on both the scheduler and the node to calculate a pod's resource request and which is  On the node, function call tree looks like this: Is it possible to add few more words in explanation?


Question: return devs.deviceIds } <ins> // Populates allocatedResources with the device resources allocated to the specified <podUID, contName>. func (pdev podDevices) addContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Union(devices.deviceIds) } } // Removes the device resources allocated to the specified <podUID, contName> from allocatedResources. func (pdev podDevices) removeContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Difference(devices.deviceIds) } } </ins> // Returns all of devices allocated to the pods being tracked, keyed by resourceName. func (pdev podDevices) devices() map[string]sets.String { ret := make(map[string]sets.String)

PR that the mentioned change: Code in question is here:
Above looks incorrect description. Predicate that run on the scheduler and on the node is same i.e max(init, regular). Pod admission will pass at the node. Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'.
Problem might happen at allocation where allocated resources is not the max but the sum of (init, regular). And in that case, pod would hang there in fail state rather 'bounce back infinitely'. Absolutely correct, sorry to make you confused, I think Renaud name it mainly bacause the allocate func is supposed to be called during the predicate phase, that is, passed in as a part of PredicateAdmitHandler.
The original idea was onced listed in as a sub-task: Now it's removed (I don't know why ...), while I believe the idea was: device resources allocation during pod admit phase. This comment is also related if needed during reconcile in 1 and cache these devices info cached devices later during container create phase
Handle device resources allocation during pod admit phase. This comment (comment) is related Call Allocate() if needed during reconcile in 1 and cache these devices info Use cached devices later during container create phase Current code behaviors exactly as you described, but to address the issue, the key point might be to decide which devices of init containers could be reused by which regular containers, and generate cache accordingly in step2; when creating containers, we simply fetch runtime information from cache.
Thanks a lot for filing this issue, I think for 1.9, we probably want to list this as a known issue and may advise people to not request gpu resources in init containers. We can target to fix this problem in 1.10.
Agreed We don't have a clear solution for this tough. It's pretty clear to me that device resources allocation during pod admit phase is not the right model when init containers comes into the picture. GPUs are pretty simple example. On of the things we plan to do is scrub the GPU memory during the Allocate phase. If GPUs are re-used between initContainers and regular containers the current model doesn't fit.


Question: return devs.deviceIds } <ins> // Populates allocatedResources with the device resources allocated to the specified <podUID, contName>. func (pdev podDevices) addContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Union(devices.deviceIds) } } // Removes the device resources allocated to the specified <podUID, contName> from allocatedResources. func (pdev podDevices) removeContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Difference(devices.deviceIds) } } </ins> // Returns all of devices allocated to the pods being tracked, keyed by resourceName. func (pdev podDevices) devices() map[string]sets.String { ret := make(map[string]sets.String)

do you think if you have time to implement the solution you mentioned in the above comment We can try to get it into a 1.9 sub release. I think for now, lets just support device reuse among initContainers and normal containers. We can revisit this part when we need to guarantee an Allocate call per container.
Right what I am mentioning here is that it is not possible to do that. The point of the device plugin is so that we can execute vendor specific code before making the device available in the container (e.g: Scrub the GPU memory). I don't think the proposed solution addresses this problem.
Sure. But to clarify, I think if we dont't expect big shock on current implementation, we could only provide limited support on the point, that is, without re-generate runtime information for regular containers, only those whose requested resource is equal to or is a subset of that in init containers could make a resuse(e.g. if we have two init containers, each requests 2 devices, then a regular container that request 3 devices still could not reuse the devices.) Besides, if a initcontainer requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them. In sum, since we don't re-generate runtime information, we could only reuse devices per container, not per device, like we do in scheduler. Is that what you mean?
Hmm, that is a good question. I forgot that we now return the combined runtime information in AllocResponse. Then for now, perhaps we should just return an error in device plugin manager Allocate() in case both init containers and containers request the same device plugin resource?
You mean we give up reuse and prohibit users to request same kind of device in init containers and regular containers? that seems a bit weird. And if we decide to do so, we might need to do the check at the first beginning, i.e. in API validation, not wait until it goes into DevicePlugin.
I think for 1.9, we just return an error when both init containers and normal containers of a pod request the same device resource, and document this in the device plugin user guide.


Question: return devs.deviceIds } <ins> // Populates allocatedResources with the device resources allocated to the specified <podUID, contName>. func (pdev podDevices) addContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Union(devices.deviceIds) } } // Removes the device resources allocated to the specified <podUID, contName> from allocatedResources. func (pdev podDevices) removeContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Difference(devices.deviceIds) } } </ins> // Returns all of devices allocated to the pods being tracked, keyed by resourceName. func (pdev podDevices) devices() map[string]sets.String { ret := make(map[string]sets.String)

The reason is because in the current device plugin implementation, we don't have a good way to free devices used by init containers of an active pod. It is hard to do this kind of validation at API level or even scheduler because we can't know whether an extended resource is device resource or not. So for now, I think we should probably just return a clear error message during admission handler. I am not sure yet whether there will be strong use cases that need to share devices between init containers and normal containers. We can revisit this based on user feedback. If there is strong user demand to support this, something we could do is to differentiate init containers and normal containers in podDevices, and free up devices used by init containers when any normal container is about to run. That would request some code structure change, and we may wait to see whether there is strong demand to support this.
It is hard to do this at API level or even scheduler because we can't know whether an extended resource is device resource or not. Oh, you are right, I ignored the resource awareness issue. I also agree that it's hard to address the issue without big change to current implementation. But I still feel the inconsistency of behavior between API and kubelet might confusing users. However, its effect should be sustainable, since we haven't received feedback about init containers up to now.
Right. So for now, perhaps just return an error when this happens, to give clear error signals.
Oh, I'm afraid we miss something, the check may not work as expected. As I commented, because init-containers run sequentially, we did reuse between init containers in , that is, if we have two init containers, each request 1 device, then we expect 1 device in total in scheduler, but expect 2 in deviceplugin. The check could not cover the scenario.
Thinking about this problem a bit more, I wonder whether we may tackle this issue from the following approaches, ordered by their complexities: 1) Just return an error whenever an init container requests a device plugin resource, and document in the device plugin user guide that requiring device plugin resource in init containers are not supported. This may be the short-term change we can try to get into 1.9. 2) Maintain a list of devices allocated to init containers in ManagerImpl.Allocate().


Question: return devs.deviceIds } <ins> // Populates allocatedResources with the device resources allocated to the specified <podUID, contName>. func (pdev podDevices) addContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Union(devices.deviceIds) } } // Removes the device resources allocated to the specified <podUID, contName> from allocatedResources. func (pdev podDevices) removeContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Difference(devices.deviceIds) } } </ins> // Returns all of devices allocated to the pods being tracked, keyed by resourceName. func (pdev podDevices) devices() map[string]sets.String { ret := make(map[string]sets.String)

Try to allocate from this list first for later init containers and normal containers. The required no. of devices we pass to ManagerImpl.devicesToAllocate() should subtract the no. of devices we have allocated to init containers. This may require some code structure change around ManagerImpl.Allocate(), ManagerImpl.allocateContainerResources(), and ManagerImpl.devicesToAllocate(). However, Allocate gRPC call is still issued during Admission all together, and it is possible that we issue an Allocate gRPC for [dev1] (say dev1 allocated to initContianer1), followed by another Allocate gRPC call for [dev1, dev2] (say dev1 and dev2 allocated to initContainer2), with none of containers started or finished. This assumes containers running in the same pod share the same security context, and device plugin doesn't need to perform per-container allocation operations. We just use Allocate gRPC call to learn container runtime settings. 3) Take a further step from 2), during Admission, we just determine what devices to allocate to containers without issuing the actual Allocate gRPC calls. I.e., we only fill deviceIds field in podDevices.deviceAllocateInfo but leaves allocResp field empty. During ManagerImpl.GetDeviceRunContainerOptions(), we check for this case and issue the actual Allocate gRPC call for the container. This ensures we can issue a gRPC Allocate call per container. However, if a pod is restarted multiple times, we don't guarantee Allocate gRPC call will be issued during subsequent pod runs. 4) Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. My suggestion is to implement 1) now and gather some user feedbacks to determine whether we need to implement the following approaches.


Question: return devs.deviceIds } <ins> // Populates allocatedResources with the device resources allocated to the specified <podUID, contName>. func (pdev podDevices) addContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Union(devices.deviceIds) } } // Removes the device resources allocated to the specified <podUID, contName> from allocatedResources. func (pdev podDevices) removeContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Difference(devices.deviceIds) } } </ins> // Returns all of devices allocated to the pods being tracked, keyed by resourceName. func (pdev podDevices) devices() map[string]sets.String { ret := make(map[string]sets.String)

I see this a very big limitation. Special devices/resources in init containers is a very common use case IMO. Therefore we should try to fix this. This approach perfectly makes sense to me. What is your concern in implementing this? Even today, IIUC, Allocate call is made per container. Is there any other value that this approach is adding over 1) ? I am not able to see any gains with 2) or 3) over 1). Comments please.
I feel we may not be able to implement 2) for 1.9 timeframe. Option 2) also doesn't address the case mentioned above: Option 3) covers that case for pods without restarts. Option 4) fully covers the requirement we we will scrub old container data before giving it to a new container, even if they both run in the same pod, bug it does have the mentioned drawback during device plugin failure.
Take a further step from 3), we can check for overlapping device allocations in ManagerImpl.GetDeviceRunContainerOptions() and invalidate their allocResp so that an Allocate gRPC call will be issued next time for containers using re-allocated devices. E.g., for containerA, if we find it is allocated with [dev1, dev2] and dev1 is also allocated to containerB, we set containerB's allocResp to nil so that next time before containerB is running, we will issue an Allocate gRPC call for dev1. Note that this approach does have the drawback that device plugin failure would cause previously running pods not be able to restart, so we need to carefully evaluate our use cases to see whether we need to implement this. How about making the allocation call optional? That is , if we find a contaienr using re-allocated devices, we'd try to issue an Allocate gRPC call first, and still use the cached setting if allocation failed. And, if we decide not to address the issue in v1.9, which means we have enough time to work on it later, I think approach 2) might not be a good choice. Because it has a premise that devices are stateless, recycle actions before reuse(e.g: Scrub the GPU memory mentioned above) could not be supported. And this way, as I commented above, we could only reuse devices per container, not per device, like we do in scheduler.


Question: return devs.deviceIds } <ins> // Populates allocatedResources with the device resources allocated to the specified <podUID, contName>. func (pdev podDevices) addContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Union(devices.deviceIds) } } // Removes the device resources allocated to the specified <podUID, contName> from allocatedResources. func (pdev podDevices) removeContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Difference(devices.deviceIds) } } </ins> // Returns all of devices allocated to the pods being tracked, keyed by resourceName. func (pdev podDevices) devices() map[string]sets.String { ret := make(map[string]sets.String)

I noticed that you have already add the limitation into the , so I think it's already the decision that we add the limitation as a short-time solution in v1.9 So I posted a PR to add the check in code,, PTAL :)
Use Case: Let say there is a which node has two Devices of a resource type X and the device IDs are X1 and X2. User pod has one init container, lets call it cont-init and one regular container, lets call it cont-reg. Both cont-init and cont-reg requests 2 unit of resource X. cont-init will run some vendor specific code on allocated Devices and then cont-reg will use these 'initiated' devices. I think this is the use case that mentioned. And i feel 2) can handle this. 2) approach will allocate both the devices, X1 and X2, to both the containers. When the containers start, init container, cont-init, will be run first and will run to completion and here devices will get initialized. Strictly after this cont-reg will be started and now this regular container has initialized devices. Where i am wrong?
I guess in RenaudWasTaken 's case, the initialization task is considered to be managed in the plugin, not in init-container, and in init-container we only use the device, so then we might need to call plugin to do recycle action before the device reused by reguler containers. In practical use, both of the two scenarios may exist, it depends on the request of user. Besides, if we don't issue an Allocate gRPC call, we could not ensure a ”complete reuse“: say an init-container requests 2 devices, and we have two regular containers, each requests 1 device, then due to the lack of runtime info re-generation, the devices could only be reused by one of the containers, not both of them.
It is the init containers' job to initialize things and if in some special case device plugin is doing the initialization part, in that case, init containers will not be needed and serve no purpose in pod spec. Specific to example of memory scrubbing, i think this also makes more sense to be done in init container. Once Device's memory is scrubbed, it can be used by regular container. Let try to run your use case with approach 2).


Question: return devs.deviceIds } <ins> // Populates allocatedResources with the device resources allocated to the specified <podUID, contName>. func (pdev podDevices) addContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Union(devices.deviceIds) } } // Removes the device resources allocated to the specified <podUID, contName> from allocatedResources. func (pdev podDevices) removeContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Difference(devices.deviceIds) } } </ins> // Returns all of devices allocated to the pods being tracked, keyed by resourceName. func (pdev podDevices) devices() map[string]sets.String { ret := make(map[string]sets.String)

There are three containers in pod, cont-init(requested 2 unit or resource X), cont-reg1(requested 1 unit of resource X) and cont-reg2(requested 1 unit of resource X). Now in approach 2), gRPC Allocate call is made for cont-init requesting X1 and X2. Next, gRPC Allocate call is made for cont-reg1. This time resource id will be send in Allocate from the previously allocated [X1, X2]. Lets say X1 is requested for cont-reg1 Next, gRPC Allocate for cont-reg2 and resource with ID X2. All this allocation info gets stored in internal cache. Now when container runs, runtimeOptions will be collected from internal cache. Where do you see a problem in this flow?
/cc
Yep, it could work if the plugin behavior the way you described, but I'm not sure if all kinds of devices could support "pre-allocation", that is, doing allocation on a same device multiple times at the same time, to prepare runtime informations for several containers that reuse the device later. Also I have no idea if all plugin developers consider their plugins "stateless", that is, managing initialization in init-containers. As I don't have enough knowledge on hardware, would like to hear opinions from other guys :)
This would defeat the purpose of a container. In the case of GPUs, when you run a container you expect the GPU to always be in the same state. Furthermore GPUs can't be shared (exclusive usage) between containers meaning that Init containers should behave in the same way as regular containers. With the only difference being that they are executed sequentially and before regular containers. Additionally, operations to setup a device before making them available in a container, whether functionally (load the driver, insert libraries, ....) or security wise (scrub the memory), should be handled by the infrastructure not the users pod spec.
I agree that it's a sane behavior to forbid the use of Device Plugin Resources in Init Containers for the 1.9 release
moving forward,
/cc I think what long-term solution we decide to implement will largely depend on use-case requirements from different device plugins. Would like to hear some thoughts from different device plugin developers on this. Can people cc-ed in this issue take a look at the proposed options mentioned in and provide some feedbacks on which one may better fit your use cases?


Question: return devs.deviceIds } <ins> // Populates allocatedResources with the device resources allocated to the specified <podUID, contName>. func (pdev podDevices) addContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Union(devices.deviceIds) } } // Removes the device resources allocated to the specified <podUID, contName> from allocatedResources. func (pdev podDevices) removeContainerAllocatedResources(podUID, contName string, allocatedResources map[string]sets.String) { containers, exists := pdev[podUID] if !exists { return } resources, exists := containers[contName] if !exists { return } for resource, devices := range resources { allocatedResources[resource] = allocatedResources[resource].Difference(devices.deviceIds) } } </ins> // Returns all of devices allocated to the pods being tracked, keyed by resourceName. func (pdev podDevices) devices() map[string]sets.String { ret := make(map[string]sets.String)

Answering on behalf of and Solution 1 makes sense for the moment Solution 2 defeats the purpose of the device plugin, which is setting up a device before making it available to a container. solution 4 seems to be a sane solution. As mentioned in my previous comment, as long as devices are exclusive, I expect the devices exposed in my container to always be in the same state at the start of my container. When / If we implement a mechanism for sharing a GPU / devices across containers then the scenario described by is something that we can have. It doesn't however fit the current situation.
Solution number 1 seems totally reasonable short-term. Long term, simplifying and documenting the relationship between plugin callbacks and container lifecycle would help plugin authors.
For reference, I implemented option 2 solution in PR The change itself is small. This is an alternative short-term fix we may consider to get into a 1.9 patch release. I think the main benefit of taking this fix compared to Option 1 solution is that we can change the underlying implementation to support init containers later with less user-facing impact.
No mater we take PR or PR for 1.9, I think we need to have this issue approved for 1.9 milestone. or or can any of you approve this issue for 1.9? Thanks!


Question: util.ReallyCrash = true util.InitLogs() defer util.FlushLogs() <ins> if *reportDir != "" { if err := os.MkdirAll(*reportDir, 0755); err != nil { glog.Errorf("Failed creating report directory: %v", err) } defer coreDump(*reportDir) } </ins> if testContext.Provider == "" { glog.Info("The --provider flag is not set.  Treating as a conformance test.  Some tests may not be run.")

We for uploading to GCS on , but not on successful e2e runs or failed test -up. We should definitely be dumping those logs for failed cluster-ups, and it would also be nice to throw in the successes too (good to have recent green for comparison).
Getting logs uploaded on a failed test -up seems really important for debugging.
+1 There were just a bunch of failures in scalability test suite (run 1623-1628) that failed because not all system pods were running. Since there are no logs, there is no easy way to debug it. It would be super useful to have all the logs for failed cluster-ups. cc
Unless I'm confused, it looks as if these logs are still not being uploaded. FYI.
For example: gs://kubernetes-jenkins/logs/kubernetes-e2e-gce/7500
That particular build was a timeout. When Jenkins kills the job, the log-collecting code never runs.
Another case we won't collect logs is if the cluster fails to initialize. (Which is a really important place to collect logs, but it's not a trivial fix.)
Moving this into post, since it seems unlikely to get fixed in the near term. feel free to correct me if I'm wrong.
We also don't collect logs on test job timeouts, which is frustrating.
+1
will collect the logs when the cluster fails to initialize. That's not all the cases listed here, but it's a good one.
I'd like to bump the priority for this further - we don't have any logs to help with debugging , which started to happen more often recently - it seems that didn't help. cc
I have too many P0s assigned to me already. This is at least 3rd in line right now.
This one is half year old and is P0 for a month...
From discussion with on hangouts, a vague plan: move log-dumping logic from into a standalone command in something like (feel free to bikeshed the name) update Jenkins to call this command as a post-build step probably remove and the debug stuff recently to . Right now the stuff is pretty GCE/GKE specific and uses ssh. We should probably change to a more provider-agnostic method, but this would be a good start.
Hm, actually, post-build step actually won't work, since we tear down the cluster first.


Question: util.ReallyCrash = true util.InitLogs() defer util.FlushLogs() <ins> if *reportDir != "" { if err := os.MkdirAll(*reportDir, 0755); err != nil { glog.Errorf("Failed creating report directory: %v", err) } defer coreDump(*reportDir) } </ins> if testContext.Provider == "" { glog.Info("The --provider flag is not set.  Treating as a conformance test.  Some tests may not be run.")

But making it a separate command can still let us call it in more places.
FYI - we also need logs from timed-out runs (not only the failed ones).
moving out into its own binary should help accomplish this.
I'm worried that the best way to do this is to write a bash script that grabs IPs from 's and functions and just does an ssh and cats the logs. Then we could point at that. It's sorta difficult to make work on its own.
I think at this point we're happy with anything that usually works. We can work on making in nice and shiny later.
We now get logs when kube-up fails, which recently happened :) Still some work to do here, I'll leave this issue open until timeouts are handled properly and points here.
- I can't thank you enough!
Ok, is in so we should get logs for kubernetes-e2e-* jobs and PR Jenkins jobs on timeout.
I'm gonna call this done.


Question: echo "       or find someone who can help solve the tag problem!" echo "" <del> if [[ "${VERSION_PATCH}" == "0" ]]; then echo "- Send branch: ${current_branch} as a PR to master" </del> <ins> if [[ "${VERSION_PATCH}" != "0" ]]; then echo "- Send branch: ${current_branch} as a PR to ${release_branch} <-- NOTE THIS" </ins> echo "- Get someone to review and merge that PR" <ins> echo "" fi echo "- I created the branch ${backmerge} for you. What I don't know is if this is" echo "  the latest version. If it is, AND ONLY IF IT IS, submit this branch as a pull" echo "  request to master:" echo "" echo "   git push <personal> ${backmerge}" echo "" echo "  and get someone to approve that PR. I know this branch looks odd. The purpose of this" echo "  branch is to get the tag for the version onto master for things like 'git describe'." echo "" echo "  IF THIS IS NOT THE LATEST VERSION YOU WILL CAUSE TIME TO GO BACKWARDS. DON'T DO THAT, PLEASE." echo "" if [[ "${VERSION_PATCH}" == "0" ]]; then </ins> echo "- Push the new release branch" echo "   git push ${push_url} ${current_branch}:${release_branch}" <del> else echo "- Send branch: ${current_branch} as a PR to ${release_branch}" echo "- Get someone to review and merge that PR" echo "" echo "Now you need to back merge the release branch into master. This should" echo "only be done if you are committing to the latest release branch. If the" echo "latest release branch is, for example, release-0.10 and you are adding" echo "a commit to release-0.9, you may skip the remaining instructions" echo "" echo "We do this back merge so that master will always show the latest version." echo "The version in master would, for exampe show v0.10.2+ instead of v0.10.0+" echo "It is not enough to just edit the version file in pkg/version/base.go in a" echo "seperate PR. Doing it this way means that git will see the tag you just" echo "pushed as an ancestor of master, even though the tag is on on a release" echo "branch. The tag will thus be found by tools like git describe" echo "" echo "- Update so you see that merge in ${fetch_remote}" echo "   git remote update" echo "- Create and check out a new branch based on master" echo "   git checkout -b merge-${release_branch}-to-master ${fetch_remote}/master" echo "- Merge the ${release_branch} into your merge-${release_branch}-to-master branch:" echo "   git merge -s recursive -X ours ${fetch_remote}/${release_branch}" echo "   - It's possible you have merge conflicts that weren't resolved by the merge strategy." echo "     - You will almost always want to take what is in HEAD" echo "   - If you are not SURE how to solve these correctly, ask for help." echo "   - It is possible to break other people's work if you didn't understand why" echo "     the conflict happened and the correct way to solve it." echo "- Send merge-${release_branch}-to-master as a PR to master" echo "- Take the afternoon off" </del> fi

doesn't work with our existing release flow. I thought it did, but, from cc closing out .
Actually, I just realized my last suggestion in isn't the worst. The pull requests to master are essentially only there to get the tags backmerged, so we can construct a release branch like so: and then a release branch backmerge branch like so: and it'll still work.


Question: continue } <ins> if zone, ok := test.Parameters["zone"]; ok { framework.ExpectNotEqual(len(zone), 0, "expect at least one zone") } </ins> ginkgo.By("Testing " + test.Name) suffix := fmt.Sprintf("%d", i) test.Client = c

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --If the provide is , run the below storage testcase, Hit below exception, At first glance, it looks like the return value is not correct, but looking into how the testcases was wrote, the root case should be that the testcase below doesn't support the provide, SSD PD on GCE/GKE gp2 EBS on AWS We should provide the correct error message which will be useful for trouble shooting instead. Kubernetes version (use ): All Cloud provider or hardware configuration: local OS (e.g: ): All Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing /sig storage /assign
NOTE: the "local" provider is specifically hack/local-up- any other cluster that is not one of "gce" etc. should use the default provider / "skeleton" provider (I know the naming is bad, but it's sort of an API now)
/remove-sig testing


Question: <ins> // +build !windows /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package gce_pd import ( "os" "syscall" ) // Determine if a directory is a mountpoint, by comparing the device for the directory // with the device for it's parent.  If they are the same, it's not a mountpoint, if they're // different, it is. func isMountPoint(file string) (bool, error) { stat, err := os.Stat(file) if err != nil { return false, err } rootStat, err := os.Lstat(file + "/..") if err != nil { return false, err } // If the directory has the same device as parent, then it's not a mountpoint. return stat.Sys().(*syscall.Stat_t).Dev != rootStat.Sys().(*syscall.Stat_t).Dev, nil } </ins>

This whole set of tests (if it can't be run) should be omitted from the test execution, otherwise folks running on Mac can't verify everything else is clean in a sane fashion. Passive aggressive assign to Tim, I may get around to fixing this before he does.
Last I checked Travis + Go didn't work. We should check again, and enable OS X builds in Travis if it's possible.
You could argue that if the unsupported mount () is being called in a test, there is a problem anyway - it has to run as root. Will investigate. On Jan 25, 2015 9:09 AM, "Brendan Burns" wrote:
Without a Mac to test on, I feel like this would be better fixed by someone who uses Mac... Probably: just rename to I just discovered "implicit build constraints", sigh. Go is crossing from user-friendly to sort of absurd, IMO.


Question: // The name of the expected domain gmsaDomain = "k8sgmsa.lan" <ins> // The shared folder on the expected domain for file-writing test gmsaSharedFolder = "write_test" </ins> ) var _ = SIGDescribe("[Feature:Windows] GMSA Full [Serial] [Slow]", func() {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --There was a regression in containerd that required a fix for gmsa working with UNC paths. See the fix at We should have a test that validates this functionality. See for set up and place to add this test. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted /priority important-long-term /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --#### What happened: There was a regression in containerd that required a fix for gmsa working with UNC paths. See the fix at We should have a test that validates this functionality. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted /priority important-long-term /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : There was a regression in containerd that required a fix for gmsa working with UNC paths. See the fix at We should have a test that validates this functionality. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g.


Question: // The name of the expected domain gmsaDomain = "k8sgmsa.lan" <ins> // The shared folder on the expected domain for file-writing test gmsaSharedFolder = "write_test" </ins> ) var _ = SIGDescribe("[Feature:Windows] GMSA Full [Serial] [Slow]", func() {

): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /triage accepted /priority important-long-term /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-longterm
/assign
if you have questions feel free to reachout on slack in sig-windows
Will do, thanks!
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


Question: // The name of the expected domain gmsaDomain = "k8sgmsa.lan" <ins> // The shared folder on the expected domain for file-writing test gmsaSharedFolder = "write_test" </ins> ) var _ = SIGDescribe("[Feature:Windows] GMSA Full [Serial] [Slow]", func() {

/lifecycle rotten [1]:
/remove-lifecycle rotten
/remove-lifecycle rotten
/assign
I have thought about this a bit and think it might be easiest to just expose a share on the domain controller. This does push more requirements into the configuration that needs to happen outside of the tests but it will make the tests a lot cleaner IMO.
/assign
GitHub didn't allow me to assign the following users: iXinqi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: find -L . -not (  (  -path './_artifacts/*'  <del> -o -path './bazel-*/*'  </del> -o -path './_output/*'  -o -path './_gopath/*'  -o -path './cmd/kubeadm/test/*'

This is intended to be removed once there are no active branches with bazel. 1.19 was still supported last I checked. We have some other cleanup pending the final turndown of bazel branches.
19 just had it's last patch release a week ago , these are removed now.


Question:
`   <img src="http://kubernetes.io/img/warning.png" alt="WARNING"   <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"  width="25" height="25">  <img src="http://kubernetes.io/img/warning.png" alt="WARNING"   <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"  width="25" height="25">  <img src="http://kubernetes.io/img/warning.png" alt="WARNING"   <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"  width="25" height="25">  <img src="http://kubernetes.io/img/warning.png" alt="WARNING"   <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"  width="25" height="25">  <img src="http://kubernetes.io/img/warning.png" alt="WARNING"   <img src="http://kubernetes.io/kubernetes/img/warning.png" alt="WARNING"  width="25" height="25">
PLEASE NOTE: This document applies to the HEAD of the source tree

This includes but effects multiple docs.
This issue was also raised on the repository, see:
What's needed to fix this? Are the broken headers being generated automatically somewhere?
- The warning section is pre-pended to the md docs from the following script The fixes this issue, updating this script and all the docs... However, there is an open question as to whether "old" docs should be removed from this repo, which has prevented this PR from being merged.


Question: shopt -s dotglob cp -af "${KUBE_TEMP}"/* "${KUBE_ROOT}" shopt -u dotglob <del> # Replace with placeholder docs kube::util::set-placeholder-gen-docs </del>

Instead, have a presubmit test that verifies that the docs can be generated. This will reduce the number of conflicts & rebases, therefore increase speed through the queue. Confirmed with that nothing depends on these docs at the moment. fyi, some manual process might have to change once this is in.
Yes, we want to generate the docs into , anyway. cc
Excellent. One closer step to YAML generation, I hope. :)
was interested in working on this.
Want to get started on this?
is this still a goal of ours?
I've seen a thread this week-end about that. I think docs have moved but we still need to generate them there.
can this issue be closed then?
related to ?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen There's still cruft in the docs directory.
/remove-help I don't think this meets . Please add back via if I'm incorrect
/sig docs /sig release /remove-area test-infra /area build-release /area kubeadm /area kubectl All of the following dirs/files have the following content. Do we need them? This file is autogenerated, but we've stopped checking such files into the repository to reduce the need for rebases. Please run hack/generate- to populate this file. docs/admin: .md files for kube components and kubeadm commands man/man1: .1 files for same user-guide/kubectl: .md files for kubectl commands yaml/kubectl: .yaml files for kubectl commands
/help Now that we have /good-first-issue, I think this does meet the criteria for help wanted. Probably need to talk to someone from sig-docs and figure out if/how these are used.
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command.


Question: shopt -s dotglob cp -af "${KUBE_TEMP}"/* "${KUBE_ROOT}" shopt -u dotglob <del> # Replace with placeholder docs kube::util::set-placeholder-gen-docs </del>

<detailsIn response to : /help Now that we have /good-first-issue, I think this does meet the criteria for help wanted. Probably need to talk to someone from sig-docs and figure out if/how these are used. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-priority important-soon /priority important-longterm This hasn't been committed to for any release cycle in a while. I hope someone does commit to it, but I personally cannot, so I'm bumping down to the "hopefully next time" priority. /kind cleanup in addition to docs This impacts merge velocity of any kubectl/kubeadm changes because the docs/OWNERS file is small. We're going to work around by adding OWNERS files for the relevant sigs to the relevant dirs. But if these files don't have to exist in the first place, that'd be even better.
as discovered today this would not work as OWNERS in sub-folders do not solve the problem with the /docs/.generated_docs file which is in the parent folder. i've been looking into a solution to remove the placeholders completely and make verify-docs only consist of kube::util::gen-docs. this will: ensure that godoc errors and such still block presubmits the contributors are not blocked having to commit placeholder files
/lifecycle active
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle active as PR is still in flight.


Question: } } <ins> func resourceContainerSpec(memory, cpu string) v1.ResourceRequirements { return v1.ResourceRequirements{ Requests: allocatableResources(memory, cpu), } } </ins> func resourcePodSpecWithoutNodeName(memory, cpu string) v1.PodSpec { return v1.PodSpec{ Containers: []v1.Container{{

Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement /sig apps Description: is a key function of DaemonSet; it's better to have more unit test case for it, e.g. node status, pod on the node. There's several test cases to the DaemonSet's finial decision, which are kind of integration test to me; prefer to provide more unit test case for this key function.
GitHub didn't allow me to assign the following users: mathspanda. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
do you have any specific cases or shall I figure out something? I can take a look at this issue if you don't mind.
, , thanks very much for our effort on this enhancement; as you known, DaemonSet will use scheduler's predicates to decide which node can place pods; there's several predicates errors , the purpose of this issue is to trigger every error in unit test for code coverage.
Sure, thanks. I'll cover these predicates.
Also I've found interesting issue when I wanted to add test for predicate. It looks like is not handled or maybe I did something wrong, I couldn't find check. According following comments it should fail because AWS EBS forbids any two pods mounting the same volume ID : Test case: Any thoughts? is it correct behavior?


Question: script: - KUBE_RACE="-race" KUBE_COVER="-cover -covermode=atomic" KUBE_TIMEOUT='-timeout 60s' ./hack/test-go.sh - PATH=$HOME/gopath/bin:./third_party/etcd:$PATH ./hack/test-cmd.sh <ins> - PATH=$HOME/gopath/bin:./third_party/etcd:$PATH ./hack/verify-gendocs.sh </ins> - PATH=$HOME/gopath/bin:./third_party/etcd:$PATH ./hack/test-integration.sh notifications:

Manually updated docs have a tendency to drift from actual implementation. It shouldn't be too hard to add a script to generate the markdown from a . We could then run the script manually or as a commit hook when changes are made to pkg/kubectl.
A big chunk of that is cut/paste from the help output anyway.. Brendan On Jan 9, 2015 4:20 PM, "Jeff Lowdermilk" wrote:
Why are we including the help output? Seems like cobra should do that, or we should just tell the user to ask for help if they want it.
It's useful to see generated markdown of help output for the full command tree, e.g. if you want to link to the syntax on irc or something. Anyway, this was quite easy to do, I'll have a PR shortly.


Question: } func (p *podPreemptorImpl) setNominatedNodeName(pod *v1.Pod, nominatedNodeName string) error { <ins> klog.V(3).Infof("Setting nominated node name for %s/%s to "%s"", pod.Namespace, pod.Name, nominatedNodeName) if pod.Status.NominatedNodeName == nominatedNodeName { return nil } </ins> podCopy := pod.DeepCopy() <ins> oldData, err := json.Marshal(podCopy) if err != nil { return err } </ins> podCopy.Status.NominatedNodeName = nominatedNodeName <del> _, err := p.Client.CoreV1().Pods(pod.Namespace).UpdateStatus(context.TODO(), podCopy, metav1.UpdateOptions{}) </del> <ins> newData, err := json.Marshal(podCopy) if err != nil { return err } patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, &v1.Pod{}) if err != nil { return fmt.Errorf("failed to create merge patch for pod %q/%q: %v", pod.Namespace, pod.Name, err) } _, err = p.Client.CoreV1().Pods(pod.Namespace).Patch(context.TODO(), pod.Name, types.StrategicMergePatchType, patchBytes, metav1.PatchOptions{}, "status") </ins> return err }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Unknown yet. Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated /sig scheduling /help
Can I give it a try?
Sure, feel free to take it. /assign ahmad-diaa /remove-help
I keep trying to get it fail locally by running the following command: but it never does, am I doing something wrong? Is there something else that I should be doing and would be more helpful?
Sometimes the flake is hard to reproduce in local env. You may have to read the error log of the flakes thoroughly to get some clues. BTW: you can also run the test multiple times using the directive:
After looking through the logs I came up with some information that might be useful As far as I understand, the reason for the flake is the of the medium priority pod not being cleared That is because the update request for removing returns the following error: So my current guess is that might be outdated in some cases As I am not really familiar with the logic, I would like to know whether or not those are valid points so that I could dig deeper. What do you think?
If that's the case, you can set here: You can raise a PR and manually try for ~20 times to see you can reproduce the flake.
found another issue tracking this too. it seems it has been flaky for a while.
Sorry, didn't mean to take this one for you. I worked the duplicate issue and submitted a PR. Feel free to review that PR, or submit your own if you have a different solution.
This sometimes happens. But we usually take the first PR unless it's clearly worked by someone. In terms of which style to resolve the Conflict error, I'm asking here:
Sorry, didn't mean to take this one for you. I worked the duplicate issue and submitted a PR. Feel free to review that PR, or submit your own if you have a different solution. No worries, I haven't started implementing yet and your PR seems to be following the correct approach as well according to this:


Question: } func (p *podPreemptorImpl) setNominatedNodeName(pod *v1.Pod, nominatedNodeName string) error { <ins> klog.V(3).Infof("Setting nominated node name for %s/%s to "%s"", pod.Namespace, pod.Name, nominatedNodeName) if pod.Status.NominatedNodeName == nominatedNodeName { return nil } </ins> podCopy := pod.DeepCopy() <ins> oldData, err := json.Marshal(podCopy) if err != nil { return err } </ins> podCopy.Status.NominatedNodeName = nominatedNodeName <del> _, err := p.Client.CoreV1().Pods(pod.Namespace).UpdateStatus(context.TODO(), podCopy, metav1.UpdateOptions{}) </del> <ins> newData, err := json.Marshal(podCopy) if err != nil { return err } patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, &v1.Pod{}) if err != nil { return fmt.Errorf("failed to create merge patch for pod %q/%q: %v", pod.Namespace, pod.Name, err) } _, err = p.Client.CoreV1().Pods(pod.Namespace).Patch(context.TODO(), pod.Name, types.StrategicMergePatchType, patchBytes, metav1.PatchOptions{}, "status") </ins> return err }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: [TODO] Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig node These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.


Question: } func (p *podPreemptorImpl) setNominatedNodeName(pod *v1.Pod, nominatedNodeName string) error { <ins> klog.V(3).Infof("Setting nominated node name for %s/%s to "%s"", pod.Namespace, pod.Name, nominatedNodeName) if pod.Status.NominatedNodeName == nominatedNodeName { return nil } </ins> podCopy := pod.DeepCopy() <ins> oldData, err := json.Marshal(podCopy) if err != nil { return err } </ins> podCopy.Status.NominatedNodeName = nominatedNodeName <del> _, err := p.Client.CoreV1().Pods(pod.Namespace).UpdateStatus(context.TODO(), podCopy, metav1.UpdateOptions{}) </del> <ins> newData, err := json.Marshal(podCopy) if err != nil { return err } patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, &v1.Pod{}) if err != nil { return fmt.Errorf("failed to create merge patch for pod %q/%q: %v", pod.Namespace, pod.Name, err) } _, err = p.Client.CoreV1().Pods(pod.Namespace).Patch(context.TODO(), pod.Name, types.StrategicMergePatchType, patchBytes, metav1.PatchOptions{}, "status") </ins> return err }

Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: # A set of helpers for starting/running etcd for tests <del> ETCD_VERSION=${ETCD_VERSION:-3.5.0-rc.0} </del> <ins> ETCD_VERSION=${ETCD_VERSION:-3.5.0} </ins> ETCD_HOST=${ETCD_HOST:-127.0.0.1} ETCD_PORT=${ETCD_PORT:-2379} export KUBE_INTEGRATION_ETCD_URL="http://${ETCD_HOST}:${ETCD_PORT}"

/sig scalability This issue is tracking process of upgrading etcd to 3.5.0 planned for 1.22 K8s release. Etcd has recently released Beta versions and is starting the process of correctness and performance testing. To reduce change of finding performance regression late, we will start qualifying Beta Etcd releases on GKE master branch and will upgrade to 3.5.0 as soon as it is available. Steps: [x] Do preliminary scale testing of etcd v3.5.0-beta.3 release [x] Release etcd v3.5.0-beta.3 image (, ) [x] Release etcd v3.5.0-rc.0 image () [x] Upgrade etcd server to v3.5.0-rc.0 () [x] Wait for 3.5.0 release and upgrade etcd client () [x] Release etcd v3.5.0 image (, ) [x] Upgrade etcd server to v3.5.0 /milestone v1.22
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area etcd
Does this meant that the 3.5.0 should be promoted like
Yes, created
/area release-eng release-team /sig release For tracking: cc:
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /area release-eng release-team /sig release cc: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I do think this can be pointed in the Major Themes for the 1.22 release. I would like your opinions on this. If you do think it has a place in it, I will add it to the .
How can I be a contributor to this issue? what are the technical checklist that needs to be achieved to close out on the issue?


Question: # A set of helpers for starting/running etcd for tests <del> ETCD_VERSION=${ETCD_VERSION:-3.5.0-rc.0} </del> <ins> ETCD_VERSION=${ETCD_VERSION:-3.5.0} </ins> ETCD_HOST=${ETCD_HOST:-127.0.0.1} ETCD_PORT=${ETCD_PORT:-2379} export KUBE_INTEGRATION_ETCD_URL="http://${ETCD_HOST}:${ETCD_PORT}"

PR to update Etcd is ready, but we found a blocker for merging . There is one test failing I will take a look into it on Monday, still more eyes on the problem could speed up resolution.
We'll also need a new kube-cross image w/ etcd v3.5.0. I'm working on that here:
We just need to upgrade the server to use new image like in
ok, looks like is busy. Let me take a look /assign
Thanks.


Question: apiVersion: extensions/v1beta1 kind: DaemonSet metadata: <del> name: fluentd-es </del> <ins> name: fluentd-es-v2.0.1 </ins> namespace: kube-system labels: k8s-app: fluentd-es <del> version: v2.0.0 </del> <ins> version: v2.0.1 </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec:

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: [root EFK]# kubectl logs -f fluentd-es-jrsf7 -n kube-system /var/lib/gems/2.1.0/gems/ffi- block in ffilib': Could not open library '': cannot open shared object file: No such file or directory. (LoadError) Could not open library '': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': libsystemd- cannot open shared object file: No such file or directory from /var/lib/gems/2.1.0/gems/ffi-map' from /var/lib/gems/2.1.0/gems/ffi- ffilib' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Native' from /var/lib/gems/2.1.0/gems/systemd-journal- <module:Id128' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Systemd' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/systemd-journal-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from rescue in require' fromrequire' from /var/lib/gems/2.1.0/gems/fluent-plugin-systemd- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/fluentd-block in tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-tryloadplugin' from /var/lib/gems/2.1.


Question: apiVersion: extensions/v1beta1 kind: DaemonSet metadata: <del> name: fluentd-es </del> <ins> name: fluentd-es-v2.0.1 </ins> namespace: kube-system labels: k8s-app: fluentd-es <del> version: v2.0.0 </del> <ins> version: v2.0.1 </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec:

0/gems/fluentd- newimpl' from /var/lib/gems/2.1.0/gems/fluentd-newinput' from /var/lib/gems/2.1.0/gems/fluentd- addsource' from /var/lib/gems/2.1.0/gems/fluentd-block in configure' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-configure' from /var/lib/gems/2.1.0/gems/fluentd- configure' from /var/lib/gems/2.1.0/gems/fluentd-runconfigure' from /var/lib/gems/2.1.0/gems/fluentd- runconfigure' from /var/lib/gems/2.1.0/gems/fluentd-block in start' from /var/lib/gems/2.1.0/gems/fluentd- call' from /var/lib/gems/2.1.0/gems/fluentd-mainprocess' from /var/lib/gems/2.1.0/gems/fluentd- start' from /var/lib/gems/2.1.0/gems/fluentd-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/fluentd-0.12.39/bin/fluentd:5:in <top (required)' from /usr/local/bin/fluentd:23:inload' from /usr/local/bin/fluentd:23:in <main' What you expected to happen: fluentd-es run success How to reproduce it (as minimally and precisely as possible): kubectl create -f fluentd-es- the image is google-containers/fluentd-elasticsearch:v2.0.0 Anything else we need to know?