Question: if !allowedTransitions[pair{from, to}] { t.Errorf("observed transition %q -> %q for %v", from, to, n.Name) } <del> source.Delete(n) </del> <ins> if deletePod(n) { source.Delete(n) } </ins> }, DeleteFunc: func(obj interface{}) { testDoneWG.Done()

I'll do that and some stress testing to see if it goes away.
3589 runs so far, 10 failures and all 10 failures seem to confirm the above behavior.
Guarding against this: runs so far, 0 failures I will clean this up and send out a fix shortly.
Awesome, thanks! , krousey wrote:
Accidentally left the test case running. Good news:
It's probably safe to say that if there's remaining corner cases, your setup is not going to exercise them! :) , krousey wrote:


Question: // } if err := wait.Poll(200*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) { <del> _, err := getPod(testCtx.ClientSet, waitingPod.Name, waitingPod.Namespace) return apierrors.IsNotFound(err), nil </del> <ins> w := false permitPlugin.fh.IterateOverWaitingPods(func(wp framework.WaitingPod) { w = true }) return !w, nil </ins> }); err != nil { <del> t.Error("Expected the waiting pod to get preempted and deleted") </del> <ins> t.Error("Expected the waiting pod to get preempted") } // Expect the waitingPod to be still present. if _, err := getPod(testCtx.ClientSet, waitingPod.Name, waitingPod.Namespace); err != nil { t.Error("Get waiting pod in waiting pod failed.") } // Expect the runningPod to be deleted physically. _, err = getPod(testCtx.ClientSet, runningPod.Name, runningPod.Namespace) if err != nil && !errors.IsNotFound(err) { t.Error("Get running pod failed.") } if err == nil { t.Error("Running pod still exist.") </ins> } if permitPlugin.numPermitCalled == 0 { t.Errorf("Expected the permit plugin to be called.") } permitPlugin.reset() <del> testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{waitingPod, preemptorPod}) </del> <ins> testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{waitingPod, runningPod, preemptorPod}) </ins> } func initTestSchedulerForFrameworkTest(t *testing.T, testCtx *testutils.TestContext, nodeCount int, opts ...scheduler.Option) *testutils.TestContext {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via -- We may delete podS though they may be waiting in Do not delete waiting pods if they are not scheduled. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/cc WDYT ?
It sounds to me that you prefer to prioritize waiting Pod over other assigned Pods. In that case, you may want to exclude waiting Pods in the earlier Filtering/Sorting Victims step, instead of simply disregard the deletion in PrepareCandidate(). In general, when Permit plugins work along with PostFilter (preemption), the logic could be complicated. There may not be a general preemption implementation (probably in the future we can exercise pluginArgs to provide different options). So for now the best bet is to implement your own PostFilter plugin.
Got it! But for the default preemption plugin, we delete the victims to free resources. If pod has assigned, we should delete it. If pod has not assigned, we just need reject it in WaitingPod because the pod just exists in the cache.
Oh, your point was to verify if it's a waiting pod first, and then delete it? If so, then it makes sense to me. Feel free to raise a PR. /assign
Yes, I will open a PR to do this


Question: allErrs = append(allErrs, validateProbe(ctr.ReadinessProbe, idxPath.Child("readinessProbe"))...) allErrs = append(allErrs, validateContainerPorts(ctr.Ports, idxPath.Child("ports"))...) allErrs = append(allErrs, ValidateEnv(ctr.Env, idxPath.Child("env"))...) <ins> allErrs = append(allErrs, ValidateEnvFrom(ctr.EnvFrom, idxPath.Child("envFrom"))...) </ins> allErrs = append(allErrs, ValidateVolumeMounts(ctr.VolumeMounts, volumes, idxPath.Child("volumeMounts"))...) allErrs = append(allErrs, validatePullPolicy(ctr.ImagePullPolicy, idxPath.Child("imagePullPolicy"))...) allErrs = append(allErrs, ValidateResourceRequirements(&ctr.Resources, idxPath.Child("resources"))...)

Scenario: Expected results: pod would not be created as [0] is invalid Actual results: pod is created. Turns out we are not doing any validation of envFrom in container spec today.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
I don't know that we can tighten without failing pods.
The validation code exists, but there was no wiring done from validateContainers.
Pods fail on kubelet already. So those pods were not doing much already. We had a BZ on this.
Fail how - fail to even start, or fail to get the input?


Question: klog.Warningf("Bad device-number %q for interface %sn", numStr, macID) continue } <ins> macIDs = append(macIDs, macID) macDevNum[macID] = num } // Sort macIDs by interface device-number sort.Slice(macIDs, func(i, j int) bool { return macDevNum[macIDs[i]] < macDevNum[macIDs[j]] }) for _, macID := range macIDs { </ins> ipPath := path.Join("network/interfaces/macs/", macID, "local-ipv4s") <del> macIPs[num], err = c.metadata.GetMetadata(ipPath) </del> <ins> internalIPs, err := c.metadata.GetMetadata(ipPath) </ins> if err != nil { return nil, fmt.Errorf("error querying AWS metadata for %q: %q", ipPath, err) } <del> } for i := 0; i < len(macIPs); i++ { internalIPs := macIPs[i] if internalIPs == "" { continue } </del> for _, internalIP := range strings.Split(internalIPs, "n") { if internalIP == "" { continue

What happened: does not handle "holes" in interface device-number correctly, dropping later interfaces. Create instance with eth0, eth1, eth2 ENIs as usual. Remove (eg: by selectively deleting pods) eth1. Only the addresses on eth0 are reported by NodeAddresses. What you expected to happen: Addresses on eth0 and eth2 should be reported.
/sig cloud-provider These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.


Question: /* Release: v1.26 <del> Testname: CSIInlineVolumes should support ephemeral CSIDrivers Description: CSIDriver resources with ephemeral VolumeLifecycleMode should support create, get, list, and delete operations. */ framework.ConformanceIt("should support ephemeral VolumeLifecycleMode in CSIDriver API", func(ctx context.Context) { // Create client client := f.ClientSet.StorageV1().CSIDrivers() defaultFSGroupPolicy := storagev1.ReadWriteOnceWithFSTypeFSGroupPolicy // Driver that supports only Ephemeral driver1 := &storagev1.CSIDriver{ ObjectMeta: metav1.ObjectMeta{ Name: "inline-driver-" + string(uuid.NewUUID()), Labels: map[string]string{ "test": f.UniqueName, }, }, Spec: storagev1.CSIDriverSpec{ VolumeLifecycleModes: []storagev1.VolumeLifecycleMode{storagev1.VolumeLifecycleEphemeral}, FSGroupPolicy:        &defaultFSGroupPolicy, }, } // Driver that supports both Ephemeral and Persistent driver2 := &storagev1.CSIDriver{ ObjectMeta: metav1.ObjectMeta{ Name: "inline-driver-" + string(uuid.NewUUID()), Labels: map[string]string{ "test": f.UniqueName, }, }, Spec: storagev1.CSIDriverSpec{ VolumeLifecycleModes: []storagev1.VolumeLifecycleMode{ storagev1.VolumeLifecyclePersistent, storagev1.VolumeLifecycleEphemeral, }, FSGroupPolicy: &defaultFSGroupPolicy, }, } ginkgo.By("creating") createdDriver1, err := client.Create(ctx, driver1, metav1.CreateOptions{}) framework.ExpectNoError(err) createdDriver2, err := client.Create(ctx, driver2, metav1.CreateOptions{}) framework.ExpectNoError(err) _, err = client.Create(ctx, driver1, metav1.CreateOptions{}) if !apierrors.IsAlreadyExists(err) { framework.Failf("expected 409, got %#v", err) } ginkgo.By("getting") retrievedDriver1, err := client.Get(ctx, createdDriver1.Name, metav1.GetOptions{}) framework.ExpectNoError(err) framework.ExpectEqual(retrievedDriver1.UID, createdDriver1.UID) retrievedDriver2, err := client.Get(ctx, createdDriver2.Name, metav1.GetOptions{}) framework.ExpectNoError(err) framework.ExpectEqual(retrievedDriver2.UID, createdDriver2.UID) ginkgo.By("listing") driverList, err := client.List(ctx, metav1.ListOptions{LabelSelector: "test=" + f.UniqueName}) framework.ExpectNoError(err) framework.ExpectEqual(len(driverList.Items), 2, "filtered list should have 2 items, got: %s", driverList) ginkgo.By("deleting") for _, driver := range driverList.Items { err := client.Delete(ctx, driver.Name, metav1.DeleteOptions{}) framework.ExpectNoError(err) retrievedDriver, err := client.Get(ctx, driver.Name, metav1.GetOptions{}) switch { case apierrors.IsNotFound(err): // Okay, normal case. case err != nil: framework.Failf("expected 404, got %#v", err) case retrievedDriver.DeletionTimestamp != nil: // Okay, normal case. default: framework.Failf("CSIDriver should have been deleted or have DeletionTimestamp, but instead got: %s", retrievedDriver) } } }) /* Release: v1.26 </del> Testname: CSIInlineVolumes should support Pods with inline volumes Description: Pod resources with CSIVolumeSource should support create, get, list, patch, and delete operations.

[X] test approval issue : [X] test pr : 26 May 2023 [x] two weeks soak end date : 9 June 2023 [x] test promotion pr : [x] remove endpoint from [x] End of release 1.28 Release: Remove duplicate Conformance test the do not cover Patch, Replace, DeleteCollection. According to following APIsnoop query, there are still 3 CSIDriver endpoints that are not tested for Conformance. Using a number of existing e2e test practices a new has been created to address these 3 endpoints. The e2e logs for this test are listed below. This query shows the following csiDriver endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen For end of release 1.28 Release: Remove duplicate Conformance test the do not cover Patch, Replace, DeleteCollection.
Reopened this issue. <detailsIn response to : /reopen For end of release 1.28 Release: Remove duplicate Conformance test the do not cover Patch, Replace, DeleteCollection. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package apimachinery import ( "k8s.io/apimachinery/pkg/version" "k8s.io/kubernetes/test/e2e/framework" "regexp" "github.com/onsi/ginkgo" ) var _ = SIGDescribe("server version", func() { f := framework.NewDefaultFramework("server-version") ginkgo.It("should find the server version", func() { ginkgo.By("Request ServerVersion") var version *version.Info version, err := f.ClientSet.Discovery().ServerVersion() framework.ExpectNoError(err, "Fail to access ServerVersion") ginkgo.By("Confirm major version") re := regexp.MustCompile("[1-9]") framework.ExpectEqual(re.FindString(version.Major), version.Major, "unable to find major version") framework.Logf("Major version: %v", version.Major) ginkgo.By("Confirm minor version") re = regexp.MustCompile("[^0-9]+") cleanMinorVersion := re.ReplaceAllString(version.Minor, "") framework.Logf("cleanMinorVersion: %v", cleanMinorVersion) re = regexp.MustCompile("[0-9]+") framework.ExpectEqual(re.FindString(version.Minor), cleanMinorVersion, "unable to find minor version") framework.Logf("Minor version: %v", version.Minor) }) }) </ins>

3 July 2020 [x] Two weeks soak end date: 20 July 2020 According to this APIsnoop query, there are still an endpoint which is untested. Get the server version (hits api path /version) Check that the major version is current Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen was Merge but the merge was reversed by Issue and Test grid restarted.
Reopened this issue. <detailsIn response to : /reopen was Merge but the merge was reversed by Issue and Test grid restarted. </details
Automatic retest and Merge again


Question: # REVISION provides a version number fo this image and all it's bundled # artifacts. It should start at zero for each LATEST_ETCD_VERSION and increment # for each revision of this image at that etcd version. <del> REVISION?=0 </del> <ins> REVISION?=1 </ins> # IMAGE_TAG Uniquely identifies k8s.gcr.io/etcd docker image with a tag of the form "<etcd-version>-<revision>". IMAGE_TAG=$(LATEST_ETCD_VERSION)-$(REVISION)

What would you like to be : Have debian-iptables with iptables 1.8.5+ Why is this needed: To fix (nodelocaldns + iptables-nft) More informations Found a bug in iptables-nft, fixed in iptables 1.8.5, and now available in buster-backports https://alioth-
/sig api-machinery
ping as you did the last 2 updates
Yep, this is on my list. Will take of it next week. /sig release network /area release-eng /milestone v1.20 /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Opened to bump the debian-iptables images in kubernetes/kubernetes. /remove-triage unresolved
Here are the digests for the new image ():
Please reopen if you run into any issues with the new images. /close
Closing this issue. <detailsIn response to : Please reopen if you run into any issues with the new images. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err != nil { t.Fatalf("Failed to install plugin %s: %v", confFile, err) } <del> networkConfig := fmt.Sprintf(`{ "name": "%s", "type": "%s", "capabilities": {"portMappings": true, "bandwidth": true}  }`, confName, binName) </del> <ins> networkConfig := fmt.Sprintf(`{ "name": "%s", "type": "%s", "capabilities": {"portMappings": true, "bandwidth": true, "ipRanges": true}  }`, confName, binName) </ins> _, err = f.WriteString(networkConfig) if err != nil { t.Fatalf("Failed to write network config file (%v)", err)

/kind feature The CNI "capability arguments" parameter - currently used to pass port mappings - has a few new capabilities: bandwidth limiting and network address management. The dockershim's networking driver should use them. The network address is just passing the PodCIDR dynamically to the plugin, so it no longer has to retrieve it from the APIServer. Bandwidth limiting is already implemented in Kubenet. It is configured via annotations. There is no upgrade risk in providing these values; non-compliant plugins will simply ignore them. Administrators need to explicitly enable each capability individually in their CNI configuration file. To avoid skew between runtimes, the other runtimes should also be upgraded to pass these. We may also wish to look in to a way to add this more directly to the CRI, so we don't have this skew problem, but that's a different issue.
/sig network
/assign
fyi.
Looks like this was never done for CRI?
cc


Question: registry = "k8s.gcr.io/build-image", repository = "debian-base", # Ensure the digests above are updated to match a new tag <del> tag = "v2.1.3",  # ignored, but kept here for documentation </del> <ins> tag = "buster-v1.2.0",  # ignored, but kept here for documentation </ins> ) container_pull(


Question: nodeInstancePrefix:       nodeInstancePrefix, useMetadataServer:        useMetadataServer, operationPollRateLimiter: operationPollRateLimiter, <del> }, nil </del> <ins> } gce.manager = &GCEServiceManager{gce} return gce, nil </ins> } // Initialize takes in a clientBuilder and spawns a goroutine for watching the clusterid configmap.

Unit tests are needed for the GCE cloud provider disk interface.
Assigned to
/assign
GitHub didn't allow me to assign the following users: verult. Note that only can be assigned.. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details


Question: return true } <ins> // responsibleForPod returns true if the pod has asked to be scheduled by the given scheduler. func responsibleForPod(pod *v1.Pod, schedulerName string) bool { return schedulerName == pod.Spec.SchedulerName } </ins> // assignedPodLister filters the pods returned from a PodLister to // only include those that have a node name set. type assignedPodLister struct {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Scheduler's pod informer has a selector that . This is incorrect. Scheduler must watch all the pods of the cluster, including those scheduled by other schedulers in order to update its cache of the state of the cluster. Otherwise, it will miss important events and the amount of available resources, pods on nodes, ... which are needed for scheduler to work correctly, will not be updated properly. /sig scheduling /assign
cc
For updating it cache, shouldn't the default scheduler rely on node info/status? Also its not very clear, what information the default scheduler needs from other pods (not to be scheduled by it) that it makes decision on? Just trying to understand the issue more clearly.
Scheduler watches informer events to update its cache. Node status is not enough to update all the information that scheduler caches. Scheduler must know what pods are placed on what nodes. For example, in order to check inter-pod affinity/anti-affinity it needs to know the placement of all the pods, including those scheduled by other schedulers.
But this information is present here: So it seems that your PR will now make sure to get the scheduled pods' information directly from the pod informer.
So when I check the history, I see it was here:
have we seen any specific issues due to this change (like between default scheduler and DS controller)?
Yes, but scheduler relies on its pod informers to keep this location up to date. That's exactly why it is important for the scheduler to receive updates for all the pods in the cluster. Yes. We missed the problem at the time. This is problematic only in clusters that use multiple schedulers. I initially thought that DS controller could cause issues as well, but DS controller does not set , so it must not cause an issue. We have seen instances of scheduler's missing certain events in cluster and as a result scheduler's cache getting stale, but I doubt if it is related to this particular issue. This only affects clusters with more than one scheduler. If you run the test that I have written in with the head, it will fail. That tests simulate a multi-scheduler scenario.


Question: ProxyConfig    string }{ MasterEndpoint: masterEndpoint, <del> ProxyConfig:    proxyBytes, </del> <ins> ProxyConfig:    prefixBytes.String(), </ins> }) if err != nil { return fmt.Errorf("error when parsing kube-proxy configmap template: %v", err)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: MarshalToYamlForCodecs adds an indent to the yaml output. This code can be combined with PrintBytesWithLinePrefix What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign rpothier
/sig cluster-lifecycle
/priority important-soon


Question: }, expectedStatusCode: http.StatusServiceUnavailable, }, <ins> "service unresolveable": { user: &user.DefaultInfo{ Name:   "username", Groups: []string{"one", "two"}, }, path:            "/request/path", serviceResolver: &mockedRouter{err: fmt.Errorf("unresolveable")}, apiService: &apiregistration.APIService{ ObjectMeta: metav1.ObjectMeta{Name: "v1.foo"}, Spec: apiregistration.APIServiceSpec{ Service:  &apiregistration.ServiceReference{Name: "bad-service", Namespace: "test-ns"}, Group:    "foo", Version:  "v1", CABundle: testCACrt, }, Status: apiregistration.APIServiceStatus{ Conditions: []apiregistration.APIServiceCondition{ {Type: apiregistration.Available, Status: apiregistration.ConditionTrue}, }, }, }, expectedStatusCode: http.StatusServiceUnavailable, }, </ins> "fail on bad serving cert": { user: &user.DefaultInfo{ Name:   "username",

https://k8s- https://k8s- https://k8s- https://k8s- https://k8s- ... A lot of them /cc
/assign
Still see this failure after is merged, although less frequently: https://k8s- https://k8s- Reopen.
Looks like the client doesn't differentiate between 500 and 503 here: It should. Will open a fix shortly
opened
fix merged, will monitor https://k8s-
Thanks! :)
haven't seen this in the 12 hours since the fix merged, was seeing it every 2-3 hours before. /close


Question: "github.com/aws/aws-sdk-go/service/kms" "github.com/aws/aws-sdk-go/service/sts" "gopkg.in/gcfg.v1" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/klog" "k8s.io/apimachinery/pkg/api/resource"

E2e tests occasionally flake on AWS. It happens randomly to all tests that create a pod with a volume, write something to the volume, delete the pod and create a new one that reads from the volume. For example: /D controller detaches a volume from a node (= the old pod died) after that (under 1s) it tries to attach the same volume to the same node (=the new pod is scheduled and coincidentally it uses the same node). In most cases this works, however, few times per day it happens that the second attach (in Kubernetes terms, see below) does not succeed and the test times out. AWS cloud provider in Kubernetes heavily uses DescribeInstances and DescribeVolumes to check that a volume has been attached or detached: In step 1. above, Kubernetes issues AWS API call and polls until it reports that the volume has been detached. In step 2. above, Kubernetes issues AWS API call to check what devices are used on a node and then it tries to find a free device for the volume that should be attached. In this bug, in step 2 reports the volume as detaching, even that reported the volume as detached in step 1. There was no call in between! Kubernetes does not expect that this could ever happen and does not try attach the volume (it's not reported as detached, so why bother, right?). This is the code that finds out that the volume is not detached ("detaching" != "detached"): And few lines below we skip : We (OpenShift) run lot of Kubernetes e2e tests on AWS. We run them in a single AWS project and I guess we hit some timing issues / races on AWS side. We hit also AWS API quota pretty hard, but it does not seem to be related to this bug, I mention it only to illustrate the amount of things that run in parallel and that AWS may not like that load. Environment: Kubernetes v1.13.4 /kind bug /sig aws /sig storage
Flow from AWS API perspective: So far so good. = volume vol- is back detaching: All timing is related to a response received, processed and logged by Kubernetes, which may be different to time corresponding request was sent. Everything in this report happens in sequence, there is no pararellism for a single volume.


Question: "github.com/aws/aws-sdk-go/service/kms" "github.com/aws/aws-sdk-go/service/sts" "gopkg.in/gcfg.v1" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/klog" "k8s.io/apimachinery/pkg/api/resource"

(Other volumes may be and often are attached / detached in parallel to this flow).
this looks like a bug in AWS itself, seems to return old data randomly. Can you please check AWS side of things? And we probably need fix / workaround on Kubernetes side. How can we check that a volume has been 100% detached if returns inconsistent data? We can check if a volume is in and not report is as attached here: This will cause Kubernetes to re-use the same device name and call . How bulletproof is it going to be? And most importantly, what other parts of Kubernetes depend on being consistent?
I confirmed internally. This is expected behavior for and since ec2 API follows a where resource mutation might be not reflected immediately. This could explain why returns a volume in detaching state even after the previous call returns without the volume (detached) I think this won't work in the case when the volume is actually stuck in detaching state, since attaching a stuck volume could simply make it worse. Given that, how about, in call, we return error when notice the volume is still being returned from and we rely on the AD controller retry to check the state next time, and hopefully the state converges.
If can return stale data, depending on it in (or anywhere else) probably won't help, because we may again get new or stale data "randomly". We call in today to get fresh list of used/free devices. Depending on timing, two cases happen: Sometimes it returns that the volume is detaching and I can see in logs. Handling as you suggest would help here. Sometimes it returns that the volume is detached. We find a new device name and call AWS and start polling for finished attach. This then returns that the volume is detaching from the node with the old device name. So we're actually attaching a volume that's being detached and you wrote that's a bad thing. In my logs, I can see that Kubernetes recovers from this error: Volume plugin AttachVolume() fails with A/D controller re-tries AttachVolume() calls AttachDisk() that calls to find a free device. returns that the volume is actually being attached to the node (I can see in logs) AttachDisk() polls until the volume is attached. -it works, but to me it feels like it's just an accident. Checking can't help in this case.


Question: "github.com/aws/aws-sdk-go/service/kms" "github.com/aws/aws-sdk-go/service/sts" "gopkg.in/gcfg.v1" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/klog" "k8s.io/apimachinery/pkg/api/resource"

So, is there another way how to check that a volume is really detached? Or should we promote the error flow described above to official fix and add bunch of comments around so we won't remove / tweak the flow in future? What bad can happen when we call AWS on a volume that's detaching? AWS at least once reported that the volume is detached, so the volume probably is not be stuck.
I think it is only bad if the volume was previously stuck. But feels acceptable for me if it is being detached and the previous call in at least confirmed once. Confirming internally. But I suspect there isn't any. As long as the volume is not stuck (unmounted properly), we should be able to attach a detaching volume especially when it is at least confirmed being detached once from call in . Even if the first time it doesn't work, retry should help recover the situation. How do you think?
According pointed, AWS recommends two actions to manage the eventual consistency model (k8s already does 1, but AFAIK not 2, except when there's an exponential backoff among calls: 1) Run the appropriate call before modifying a resource 2) Add wait time between subsequent commands, even if returns a green response (this does infer that might return stale data) Currently, k8s assumes a state based on a single green response, e.g., is called repeatedly with an exponential backoff until it returns a given volume state. What if it required at least two positive responses instead? What are the chances that would return stale data twice? I'd assume that two calls returning a detached volume (with exponential backoff in between, which enforces point 2 above) would be reliable enough. The downside is that this would increase the API usage, which is already high.
do you know if the native AWS calls may be more stale-data-proof?
Looking at implementation, it tries with constant backoff of 15 seconds (which I consider pretty large) and it returns when the first returns that the volume is in use, so basically the same as what Kubernetes does.
From what I saw, sometimes it happens that reports a volume as detached, subsequent reports that the volume is not attached, succeeds and only after that shows that the volume is (from the old device name) instead of . So Kubernetes got at least two confirmations that the volume is detached.


Question: "github.com/aws/aws-sdk-go/service/kms" "github.com/aws/aws-sdk-go/service/sts" "gopkg.in/gcfg.v1" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/klog" "k8s.io/apimachinery/pkg/api/resource"

What would be the correct delay to wait for second (third, ...) confirmation that the volume is really detached? IMO it would hurt our performance a lot.
The Wait function simply calls DescribeXX under the hood and uses some hard coded timeout. So it's similar to what kubernetes does like said.


Question: git  nfs-common  glusterfs-client  <ins> cifs-utils  </ins> && DEBIAN_FRONTEND=noninteractive apt-get upgrade -y  && DEBIAN_FRONTEND=noninteractive apt-get autoremove -y  && DEBIAN_FRONTEND=noninteractive apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

Kubernetes version (use ): 1.4.0 Environment: Cloud provider or hardware configuration: Azure What happened: Trying to mount an Azure file share by following the directions fails because hyperkube does not include cifs-utils (eg, ). Can this be to the hyperkube image, please?
is this related to GCI image missing nfs related package?
This should fix the reported issue:
Thanks for fixing this so quickly!
No, unrelated


Question: // Should the test restart petset clusters? // TODO: enable when we've productionzed bringup of pets in this e2e. restartCluster = false <ins> // Timeout for reads from databases running on pets. readTimeout = 60 * time.Second </ins> ) // Time: 25m, slow by design.

https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
The test failed because it read nothing back from the key for the "member with index 2". Looking at , I notice that it's writing to the "member with index 0" and expecting to read it back from the "member with index 2". I'm curious, is each index supposed to represent a separate redis instance? What is the consistency model here?
/cc
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
It's master slave redis, you should be able to create the same petset with this: , fyi
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: // Should the test restart petset clusters? // TODO: enable when we've productionzed bringup of pets in this e2e. restartCluster = false <ins> // Timeout for reads from databases running on pets. readTimeout = 60 * time.Second </ins> ) // Time: 25m, slow by design.

https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: -cover -covermode="${KUBE_COVERMODE}"  -coverprofile="${cover_report_dir}/{}/${cover_profile}"  "${cover_params[@]+${cover_params[@]}}"  <del> "${KUBE_GO_PACKAGE}/{}" || true </del> <ins> "${KUBE_GO_PACKAGE}/{}" </ins> COMBINED_COVER_PROFILE="${cover_report_dir}/combined-coverage.out" {

Take great care merging! Example PR run (look at the "succeeded" builds): https://travis-
Somewhat related, (bug) and (fix, as yet unmerged) mean that "guestbook" and "shell" e2e tests are being silently skipped on all platforms since Wed last week (4/15). should merge shortly and resolve the issue.
From the Travis run, So it looks like either the shell script is eating the exit code, or the test is exiting 0 even when tests fail. Similarly for Shippable, it seems.
I think broke this. It appears that will always exit 0, even when tests fail or timeout. We should revert this change ASAP.
Fixed by


Question: AlphaFeatureGate *AlphaFeatureGate } <ins> // TODO: replace gcfg with json </ins> type ConfigGlobal struct { TokenURL  string `gcfg:"token-url"` TokenBody string `gcfg:"token-body"`


Question: } // Create SCs <del> scs := []*storagev1.StorageClass{ makeStorageClass(classImmediate, &modeImmediate), makeStorageClass(classWait, &modeWait), } for _, sc := range scs { </del> <ins> for _, sc := range sharedClasses { </ins> if _, err := clientset.StorageV1().StorageClasses().Create(sc); err != nil { t.Fatalf("Failed to create StorageClass %q: %v", sc.Name, err) }

Add test for "on PVC add", see Test for "on PVC update" is not needed, because it has already been tested in all volume scheduling tests.
/sig
docs:


Question: err = testsuites.WaitForControllerVolumeResize(ctx, pvc, m.cs, csiResizeWaitPeriod) framework.ExpectNoError(err, "While waiting for PV resize to finish") <ins> ginkgo.By("Waiting for PVC resize to finish") pvc, err = testsuites.WaitForFSResize(ctx, pvc, m.cs) framework.ExpectNoError(err, "while waiting for PVC to finish") </ins> ginkgo.By("Waiting for all remaining expected CSI calls") err = wait.Poll(time.Second, csiResizeWaitPeriod, func() (done bool, err error) { <del> _, index, err := compareCSICalls(ctx, trackedCalls, test.expectedCalls, m.driver.GetCalls) </del> <ins> var index int _, index, err = compareCSICalls(ctx, trackedCalls, test.expectedCalls, m.driver.GetCalls) </ins> if err != nil { return true, err }

Seen in pull-kubernetes-e2e-gce-cos-alpha-features Specific job failure
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from the error message I don't know if it refers to the sig-storage error or to the sig-auth one
/sig storage
/kind failing-test
/assign From : ... I am not sure if it's related:
is the culprit. cannot be used inside a . This only gets caught when the test actually runs, otherwise Ginkgo never sees the incorrect .
I don’t understand why sometimes it can get the pv by namebut sometimes it failed
but then is a sig-auth test , not storage , sorry /sig auth this seems that was recently in
/remove sig-storage
/assign /priority important-soon
/unassign
is still failed. seems not fixing the whole problem. /reopen feel free to close if the ci is green.
Reopened this issue. <detailsIn response to : is still failed. /reopen feel free to close if the ci is green. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the failure now looks like it is just a sig storage issue:
cc
/assign looks like the new test is still an issue
I think this line is the problem: When the test creates the object, it may take time for the PV to be provisioned and to be bound to the PVC. So you should do another before this call so you can get the latest PVC object with the bound PV.
I opened to get PVC again before using .
Only do PersistentVolumeClaims.Get() is not enough, whether we need to wait for sometimes to wait it bound? because sometimes the testcase is successful and sometime is failed.
If the Pod was able to run, then the PVC has to be bound.
so I can't understand why sometimes the testcase is successful and sometimes is failed.


Question: err = testsuites.WaitForControllerVolumeResize(ctx, pvc, m.cs, csiResizeWaitPeriod) framework.ExpectNoError(err, "While waiting for PV resize to finish") <ins> ginkgo.By("Waiting for PVC resize to finish") pvc, err = testsuites.WaitForFSResize(ctx, pvc, m.cs) framework.ExpectNoError(err, "while waiting for PVC to finish") </ins> ginkgo.By("Waiting for all remaining expected CSI calls") err = wait.Poll(time.Second, csiResizeWaitPeriod, func() (done bool, err error) { <del> _, index, err := compareCSICalls(ctx, trackedCalls, test.expectedCalls, m.driver.GetCalls) </del> <ins> var index int _, index, err = compareCSICalls(ctx, trackedCalls, test.expectedCalls, m.driver.GetCalls) </ins> if err != nil { return true, err }

By default, the DelayBinding is false, it will wait for PVC bound during creation. This is weird.
It waits for the claim to be bound, but the test case still needs to refresh its pvc object.
Because of the refresh is async, so after the pvc was bound, we need to get the pvc again, isn't it?


Question: } // NodeAllocatableRoot returns the literal cgroup path for the node allocatable cgroup <del> func NodeAllocatableRoot(cgroupRoot, cgroupDriver string) string { root := ParseCgroupfsToCgroupName(cgroupRoot) nodeAllocatableRoot := NewCgroupName(root, defaultNodeAllocatableCgroupName) </del> <ins> func NodeAllocatableRoot(cgroupRoot string, cgroupsPerQOS bool, cgroupDriver string) string { nodeAllocatableRoot := ParseCgroupfsToCgroupName(cgroupRoot) if cgroupsPerQOS { nodeAllocatableRoot = NewCgroupName(nodeAllocatableRoot, defaultNodeAllocatableCgroupName) } </ins> if libcontainerCgroupManagerType(cgroupDriver) == libcontainerSystemd { return nodeAllocatableRoot.ToSystemd() }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: k8s 1.17.2 config kubelet with those parameters cgroupDriver: cgroupfs cgroupRoot: /docker cgroupsPerQOS: false kubelet log always prints those E0303 17:25:03. ] Failed to get system container stats for "/docker": failed to get cgroup stats for "/docker": failed to get container info for "/docker": unknown container "/docker" E0303 17:25:03. ] eviction manager: failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics What you expected to happen: kubelet does not print those error information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node


Question: for _, extIP := range svcInfo.ExternalIPStrings() { staleServices.Insert(extIP) } <ins> for _, extIP := range svcInfo.LoadBalancerIPStrings() { staleServices.Insert(extIP) } </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In ipvs mode client traffic(UDP) to a pod behind LoadbalancerIP is blackholed after node having the pod is rebooted. The issue seems to be similar to the below iptable issue, since client continuously tries to connect to the loadbalancerIP (even before kubeproxy has applied all rules), and we see that stale conntracks entries remain even after rules are applied. After the stale conntrack entries are cleaned manually the traffic starts flowing : From the below code looks like ipvs currently clears stale conntrack entries for ExternalIPs but not LoadbalancerIP(?): ().String())-,for%20%2C%20extIP%20%3A%3D%20range%20svcInfo.ExternalIPStrings()%20%7B,-staleServices.Insert(extIP UDP traffic between client and pod(behind Loadbalancer IP) should have been established after node reboot. a POD acting as a UDP server and expose it through Loadbalancer service a UDP client to continuously send traffic to the LoadbalancerIP the node having the pod. after the new pod serving UDP is up, the UDP traffic from client would be blackholed(until stale conntrack entries are cleared). Kubernetes version (use ): 1.21.1 Cloud provider or hardware configuration: Manufacturer: Dell Inc. Product Name: PowerEdge R640 Version: Not Specified OS (e.g: ): SUSE Linux Enterprise Server 15 SP2 Kernel (e.g. ): Linux pool16-n108-wk16-n080 5.3.18-24.75.3..0.PTF.-default # 1 SMP Thu Sep 9 23:24:48 UTC 2021 () x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Calico v3.19.1-12- Others:
/sig network
/assign I'll try to reproduce this using but it may take some days.


Question: for _, extIP := range svcInfo.ExternalIPStrings() { staleServices.Insert(extIP) } <ins> for _, extIP := range svcInfo.LoadBalancerIPStrings() { staleServices.Insert(extIP) } </ins> } }

yeah, ipvs kube-proxy doesn't have some of the latest "conntrack stale" fixes like , mainly because I'm not very familiar with IPVS and I can't know if all of them are needed or only some of them.
/triage accepted is it a matter of inserting externalIPs in conntrack?
No, when a node is rebootes and udp messages to a loadBalancerIP enters before kube-proxy is started, then invalid (UNREPLIED) conntrack entrys are created. They will black-hole traffic, and since traffic keeps coming they will never timeout. This is not easy to reproduce. I found it best to direct incoming traffic to a node (2), and have one endpoint pod on another node (4). Then reboot the node where traffic enters (and let the pod live) while traffic is sent continously. After reboot, check the conntrack on the rebooted node. In this example 3 out of 10 connections are ok. Those are the fortunate ones that didn't send a packet in the small time span from node start to kube-proxy start. (about 1 pkt/sec is sent on all 10 connections)
/area kube-proxy /area ipvs


Question: func (im *realImageGCManager) Start() { ctx := context.Background() go wait.Until(func() { <del> // Initial detection make detected time "unknown" in the past. var ts time.Time if im.initialized { ts = time.Now() } _, err := im.detectImages(ctx, ts) </del> <ins> _, err := im.detectImages(ctx, time.Now()) </ins> if err != nil { klog.InfoS("Failed to monitor images", "err", err) <del> } else { im.initialized = true </del> } }, 5*time.Minute, wait.NeverStop)

when kubelet restart,all images which notused cleaned. but there is a flag in kubelet,,explainMinimum age for an unused image before it is garbage collected. Examples: '300ms', '10s' or '2h45m'. i set to ,it's means if a image not used,it will live 20m and then clean by kubeletif has disk pressure.if kubelet not restart, it work,but if restart ther is a bug, not work,image will be cleaneven if it's no enough time. work, image even if not used,it will live 20m and then clean by kubelet. 1make node trigger disk pressure. 2pull some image which not used by pod 3restart kubelet 4image cleaned by kubelet after around 10s i see the source code, and maybe there is a bug. in func,well detect image and update firstdetect time for every image. and in func,well check images livetime by firstDetected. if kubelet running normal,it work,because there is a func. in this line,,in first time,ts is zero. so image's firstDetected time is zero,and in freeSpace this check is always false,so image will be cleaned. maybe we can change to ,can fix this bug. <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
/assign
/triage accepted /priority backlog initial assessment of priority, subject to change


Question: apiServiceSuccess := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "success", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy2"}, }, }

When installing the metrics-server to implement the API and the prometheus-adapter to implement the API, it is possible to make the OpenAPI handler fail to initialize with this error log in apiserver: This will result in kubectl discovery calls to /openapi/v2 to fail with a 503: The OpenAPI handler should not fail to initialize if both and are serving the same path for different API groups/versions. I can provide exact installation steps if needed, however, a high-level repro is: install metrics-server to serve install prometheus-adapter to serve ensure resourceRules is populated in the prometheus-adapter config so that resource metrics are also served by restarting apiserver will result in the OpenAPI failure and 503 responses to /openapi/v2 path No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
Please let me know if I should reopen the issue in
/sig api-machinery
/sig instrumentation [metrics apis]
Would you mind taking a look when have time? Thank you^^ /triage accepted
I have a question. should be serving under . Why does an apiservice serving another gvk register paths under ? cc we've made the assumption that an APIService (such as metrics-server or prometheus-adapter) serves only gvks under its own registered group version. I'll do some more investigation but just wondering if prometheus doesn't follow that assumption?
the prometheus-adapter can also be configured to serve and replace the metrics-server by enabling these metrics, However, if the prometheus-adapter fronts and metrics-server fronts , it does seem like the correct behavior would be for prometheus-adapter to serve until an operator updates the APIService to reference prometheus-adapter rather than metrics-server. We're seeing this issue occur for upgrades between 1.27 and 1.28 so it does look this path duplication did not cause the OpenAPI handler to fail on 1.27.
I just upgraded from 1.27 to 1.28 and running into this problem. Some kubectl apply operations fail, and ArgoCD cannot load its custom resources. Any suggested workaround? Would removing the from the prometheus configuration resolve it? Or perhaps disable metrics-server?
Simply looking at the code diff between 1.27 and 1.28 I can't pinpoint the cause.


Question: apiServiceSuccess := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "success", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy2"}, }, }

Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question.


Question: apiServiceSuccess := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "success", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy2"}, }, }

In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.


Question: apiServiceSuccess := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "success", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy2"}, }, }

I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in


Question: binary | sha1 hash | md5 hash ------ | --------- | -------- <del> [kubernetes.tar.gz](https://storage.googleapis.com/kubernetes-releases/release/v1.3.0-alpha.2/kubernetes.tar.gz) | `305c8c2af7e99d463dbbe4208ecfe2b50585e796` | `aadb8d729d855e69212008f8fda628c0` </del> <ins> [kubernetes.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.3.0-alpha.2/kubernetes.tar.gz) | `305c8c2af7e99d463dbbe4208ecfe2b50585e796` | `aadb8d729d855e69212008f8fda628c0` </ins> ## Changes since v1.3.0-alpha.1

Perhaps a bug in
Thanks for catching this! I'm surprised it wasn't noticed earlier. It's an off by 1 (s) error. We pushed to gs://kubernetes-releases vs gs://kubernetes-release. The offending s has been removed. I'll clean up the release itself.
I thought this was right, but I think this is still an issue. I'm trying to release , and it's building wrong. I guess there were actually two problems; you caught one, but not the one I meant. This issue was about: should report itself as , but instead it reports itself as .
I'll look into this, since it's blocking alpha releases.
This seems contrary to , but doesn't have enough context for me to know why it was changed.
This was fixed by a long time ago.


Question: } func (c *dynamicResourceClient) Patch(name string, pt types.PatchType, data []byte, opts metav1.PatchOptions, subresources ...string) (*unstructured.Unstructured, error) { <ins> if len(name) == 0 { return nil, fmt.Errorf("name is required") } </ins> result := c.client.client. Patch(pt). AbsPath(append(c.makeURLSegments(name), subresources...)...).

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Providing an empty to the dynamic client's method results in a panic. What you expected to happen: An error should be returned on empty . How to reproduce it (as minimally and precisely as possible): Call with Anything else we need to know?: The expects a nonempty to be provided, but does not validate this. If an empty name is provided, this method will panic because it expects an , but the response will actually be an . Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: client-go 10.0.0
/sig api-machinery


Question: statusLock            sync.Mutex running               map[types.UID]bool terminating           map[types.UID]bool <ins> terminated            map[types.UID]bool </ins> terminationRequested  map[types.UID]bool removeRuntime         map[types.UID]bool removeContent         map[types.UID]bool

There appears to be a change of behavior / possible regression in k8s 1.22 with respect to pod lifecycle behavior. We've started to see some cases when multiple pods are being created (often as part of a k8s job) the following will happen: pods get created by e.g. job controller (e.g. say 5). Each pod in job requests majority of node memory so only one such pod can fit on node at a time. schedules [first] pod to node, rest (4 pods) stay in pending accepts first pod, runs it to completion pod get scheduled to the node rejects the pod during pod admission, e.g. with , despite the fact that the first pod has already completed successfully This results in many of pods backing the job getting into Failed phase and having to be recreated by job controller. The job can often get into BackofLimit exceeded to hitting backoffLimit. The root issue seems to be a mismatch of when kubelet actually considers the pod to complete and what it reports to API server (which causes the followup pods to be scheduled to it). I'm still investigating further, but this may be related to the pod lifecycle changes in 1.22 -- I've created a e2e test as minimal repro of this behavior -- (Test fails) 22 build log -- 22 kubelet log -- (Test passes) 21 build log -- 21 kubelet log -- <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --- Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.


Question: statusLock            sync.Mutex running               map[types.UID]bool terminating           map[types.UID]bool <ins> terminated            map[types.UID]bool </ins> terminationRequested  map[types.UID]bool removeRuntime         map[types.UID]bool removeContent         map[types.UID]bool

</details
/sig node
/cc
Looking at the kubelet log in 1.22 () here's what I see -- First pod starts up... starts dies after ~10 seconds since the job is completed (the container simply runs a ) Pod completes, status is updated to API server Next pod comes in but is rejected with despite the earlier pod already being completed/terminated
did the kubelet pod lifecycle rework result in a reordering of kubelet resource bookeeping w.r.t. pod status? it seems reasonable for the scheduler to send another pod to a kubelet once it reports the pod complete and the use of the resources is done from an API perspective
I am also seeing this same behavior, but with "OutOfPods" rather than "OutOfMemory", when running a job with parallelism=100, completions=400 on GKE (which has a 110 pod per node limit).
this is easily reproducible on a local cluster as well start with this: then create this job: result on 1.22 is the job hangs with 2 completions and a bunch of rejections by the kubelet: on 1.21, the same job completes all pods successfully in a couple minutes with no OutOfpods errors
verified introduced the OutOfPods issue
Shame we didn’t have an e2e which caught this given how easy the reproducer is. This looks to be the opposite of the previous cpu admission bug (ignoring things that are terminating but not terminated). We are weak on admission testing in general. This potentially implies admission is looking at the wrong set of pods (admitted + terminating). Note there is already a bug in 1.21 and before where force deleted pods are incorrectly ignored from the set of allocated resources while they are terminating, but this doesn’t immediately seem to be that issue (since i doubt those jobs are being force deleted)
yeah, there was no force deletion involved in (or any deletion at all, actually... the completed pods just sat there, completed)
The set of admitted pods considered for pod count is everything in the pod manager filtered by filterOutTerminatingPods, which calls CouldHaveRunningContainers which returns true once a pod is seen by the workers (shortly after admission, and until we’ve confirmed via syncTerminatingPod that no containers are running). Whatever the problem is it’s likely in the interaction of that.


Question: statusLock            sync.Mutex running               map[types.UID]bool terminating           map[types.UID]bool <ins> terminated            map[types.UID]bool </ins> terminationRequested  map[types.UID]bool removeRuntime         map[types.UID]bool removeContent         map[types.UID]bool

A pod the pod worker hasn’t seen yet is also filtered out (that means bursts of pods can get in, which seems the opposite of this problem, and is the bug i mentioned before).
Edit: one wrong thing spotted while looking - I'm not sure it fixes the issue you're seeing, but it's definitely wrong for admission (admission is filtering the known pod set down to remove pods that are terminated, but shouldn't filter pods that aren't admitted yet). This needs a node e2e reproducer that can run in serial (resource consumption is really hard to test from parallel, is slow in serial, and node e2e is back in a good state now) - / is that something either of you could help me with? I believe the reason the existing tests don't catch this is the job controller is creating pods fast enough that we see races in admission (multiple pods are admitted in one call to HandlePodAdditions) that are not caught in the current serial node e2e admission tests which rarely create more than one at a time.
Job pods still go through the scheduler to get assigned to a node, so even though the job controller can create pods in parallel, I'd still expect them to arrive at the node in an orderly fashion. It's still possible multiple arrive in a single HandlePodAdditions call, of course. It seems like kubelet admission is using laggy information that doesn't reflect pod completion the kubelet itself wrote to the pod API.
The bug turned out to be much simpler. Pods that are completed never go to the pod worker, so once a pod has reached terminal state it was being back to the list of active pods. The set of e2es that would have prevented this regression: node e2e that creates multiple pods at the same time and verifies that admission works correctly when HandlePodAdditions takes two pods at once node e2e that verifies that completed pods don't consume resources for admission purposes in the kubelet job controller e2e that attempts to verify some part of this reliably (ideally in a way that can be run in parallel e2e)
another related issue:
Linking here for completeness - a similar symptom might get exposed after this fix as described in The kubelet considers resources for terminating pods to be in use (they are!), but the scheduler ignores terminating pods and schedules new pods.


Question: statusLock            sync.Mutex running               map[types.UID]bool terminating           map[types.UID]bool <ins> terminated            map[types.UID]bool </ins> terminationRequested  map[types.UID]bool removeRuntime         map[types.UID]bool removeContent         map[types.UID]bool

Because the kubelet now considers terminating pods, it rejects those rapidly rescheduled pods. The follow up is more involved but requires the kubelet to allow pods to be potentially admitted that can't be run currently due to those terminating pods. See the linked issue.


Question: * * If updating an existing resource: * See `Conflict` from the `status` response section below on how to retrieve more information about the nature of the conflict. * GET and compare the fields in the pre-existing object, merge changes (if still valid according to preconditions), and retry with the updated request (including `ResourceVersion`). <ins> * `410 StatusGone` * Indicates that the item is no longer available at the server and no forwarding address is known. * Suggested client recovery behavior * Do not retry. Fix the request. </ins> * `422 StatusUnprocessableEntity` * Indicates that the requested create or update operation cannot be completed due to invalid data provided as part of the request. * Suggested client recovery behavior

We sorta fixed this-- now it's a real status error, but we left the status code 500, so it still produces stack traces and potentially confuses clients. we should figure out the correct status code.
I think there is a lot more going on related to this in . Clients are re-listing after the error but not rewatching.
/sub
It turned out this was not causing job e2e flakes. Just sending 500s that were confusing. So lowering priority.
How about a NewBadRequest error?
that certainly sounds like an improvement.


Question: package events import ( <ins> "context" </ins> "strconv" "testing" "time"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I wrote a test creating multiple isomorphic Events with an "". and stumbled upon a data race. The data race happens in the function. It seems that the Event Series field can be read and written at the same time by two different goroutines. . This seems to happen in only one scenario: (1) a singleton Event is created (2) an isomorphic Event is created (1) lock addition to cache (2) wait for the lock to be released (1) unlock (2) lock + the Series (1) the freshly created Series (1) singleton Event is now considered as a series That being said, this seems to lead to some data inconsistencies as the singleton Event would be created with an unexpected Series. <details<summaryData race</summary|================== WARNING: DATA RACE Write at 0x00c0001237a0 by goroutine 23: (eventBroadcasterImpl).recordToSink.func1.1() +0x536 (eventBroadcasterImpl).recordToSink.func1() +0x84 Previous read at 0x00c0001237a0 by goroutine 24: (Event).DeepCopyInto() +0x64 (Event).DeepCopy() +0x56 (Event).DeepCopyObject() +0x33 () +0xe8 (CreateActionImpl).DeepCopy() <autogenerated:1 +0xb7 (Fake).Invokes() +0x103 (FakeEvents).Create() +0x214 (EventSinkImpl).Create() +0x195 () +0x32c (eventBroadcasterImpl).attemptRecording() +0xe4 (eventBroadcasterImpl).recordToSink.func1() +0x9d Goroutine 23 (running) created at: (eventBroadcasterImpl).recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (eventBroadcasterImpl).StartEventWatcher.func2() +0x91 Goroutine 24 (running) created at: (eventBroadcasterImpl).


Question: package events import ( <ins> "context" </ins> "strconv" "testing" "time"

recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (eventBroadcasterImpl).StartEventWatcher.func2() +0x91 |================== |================== WARNING: DATA RACE Read at 0x00c0001236f0 by goroutine 23: (Event).DeepCopyInto() +0x64 (Event).DeepCopy() +0x72 () +0x4d () +0x98 (eventBroadcasterImpl).attemptRecording() +0xe4 (eventBroadcasterImpl).recordToSink.func1() +0x9d Previous write at 0x00c0001236f0 by goroutine 24: () +0x2e4 (eventBroadcasterImpl).attemptRecording() +0xe4 (eventBroadcasterImpl).recordToSink.func1() +0x9d Goroutine 23 (running) created at: (eventBroadcasterImpl).recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (eventBroadcasterImpl).StartEventWatcher.func2() +0x91 Goroutine 24 (running) created at: (eventBroadcasterImpl).recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (*eventBroadcasterImpl).StartEventWatcher.func2() +0x91 |================== </detailsWhat you expected to happen: No data race. How to reproduce it (as minimally and precisely as possible): You can run the following code snippet alongside any data race detector. I personally used . Anything else we need to know?: Environment: v0.25.0
/sig api-machinery
/assign Would you mind taking a look to this one? thanks!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity.


Question: package events import ( <ins> "context" </ins> "strconv" "testing" "time"

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We might also want to look into that again for /reopen /assign /unassign
/remove-lifecycle rotten
Reopened this issue. <detailsIn response to : We might also want to look into that again for /unassign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/cc
I was able to reproduce this issue with client-go v0.25.0. I updated the reproducer to be compatible with the latest version of the client and attached the data race trace.
are you gong to submit a fix for this one? thanks. /triage accepted
Yes, I am planning to submit a fix for this issue.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: package events import ( <ins> "context" </ins> "strconv" "testing" "time"

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
A PR is out to fix this issue: /remove-lifecycle stale


Question: printFlags := GoTemplatePrintFlags{ TemplateArgument: templateArg, } <ins> if !sort.StringsAreSorted(printFlags.AllowedFormats()) { t.Fatalf("allowed formats are not sorted") } </ins> p, err := printFlags.ToPrinter(tc.outputFormat) if tc.expectNoMatch {

/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "github.com/stretchr/testify/assert" ) <del> func TestGetAppArmorFS(t *testing.T) { // This test only passes on systems running AppArmor with the default configuration. // The test should be manually run if modifying the getAppArmorFS function. t.Skip() const expectedPath = "/sys/kernel/security/apparmor" actualPath, err := getAppArmorFS() assert.NoError(t, err) assert.Equal(t, expectedPath, actualPath) } func TestValidateHost(t *testing.T) { // This test only passes on systems running AppArmor with the default configuration. // The test should be manually run if modifying the getAppArmorFS function. t.Skip() assert.NoError(t, validateHost()) } </del> func TestValidateBadHost(t *testing.T) { hostErr := errors.New("expected host error") v := &validator{

As mentioned the return value of is not being used any longer and the entire method can be removed. The usage of was removed in Now there is no need to have the function and store it's result at all. /sig node /kind cleanup /good-first-issue /help-wanted KEP xref:
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : As mentioned the return value of is not being used any longer and the entire method can be removed. /kind cleanup /good-first-issue /help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/assign
I am having issues submitting the pull request. I removed that portion of the code under however this is my first time contributing and would like some help submitting this new updated version..
/assign
/assign
/assign
/assign


Question: } c.invalidateCachedPredicatesOnNodeUpdate(newNode, oldNode) <del> c.podQueue.MoveAllToActiveQueue() </del> <ins> // Only activate unschedulable pods if the node became more schedulable. // We skip the node property comparison when there is no unschedulable pods in the queue // to save processing cycles. We still trigger a move to active queue to cover the case // that a pod being processed by the scheduler is determined unschedulable. We want this // pod to be reevaluated when a change in the cluster happens. if c.podQueue.NumUnschedulablePods() == 0 || nodeSchedulingPropertiesChanged(newNode, oldNode) { c.podQueue.MoveAllToActiveQueue() } </ins> } func (c *configFactory) invalidateCachedPredicatesOnNodeUpdate(newNode *v1.Node, oldNode *v1.Node) {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Today, the scheduler re-processes unschedulable nodes . Node updates are sent by Kubelet frequently (every 10 seconds) and often times they don't change any of the node properties that could potentially make them feasible. We should be smarter about these updates and move pods to the schedulable queue only when certain properties of nodes change that could make them feasible. These properties include node allocatable, node conditions, taints, labels, .... Why is this needed: Improves performance of the scheduler and avoids doing unnecessary attempts to schedule pods. This is particularly helpful when a cluster has a lot of unschedulable pods. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature This is not a feature really. It is performance optimization, but in the absence of such a label, I need to use "feature" here. /sig scheduling
I can help to fix it. /assign mlmhl
The main reason for this problem is / update their status too frequently and most of these updates are heartbeats without any meaningful change. Recently sig-node introduces a new to alleviate this problem. With this mechanism, kubelet will split heartbeat and node status updates into two independent operations, and reduce the frequency of status updates to once per 1 minute(or more). In the long run this mechanism can indeed alleviate our problems, but at present it does not help us much as this is still an alpha feature and the node status updates frequency is still a bit high(per 1 minute right now).
Maybe we can try comparing old and new nodes after stripping their heartbeat info, like this:
I think that had sent earlier is more efficient. However, that PR does not filter out . In Wei's code, the heartbeat time is removed correctly. I think we should fix this issue and resubmit that PR.
could you resubmit a PR to: disregard in conditions comparison: and (optionally) disregard timestamp in taints comparison:
Thanks for all your advice, a new PR is submitted: This PR does not include your second suggestion(disregard timestamp in taints comparison) as I'm not sure if can be changed when node updating.


Question: } c.invalidateCachedPredicatesOnNodeUpdate(newNode, oldNode) <del> c.podQueue.MoveAllToActiveQueue() </del> <ins> // Only activate unschedulable pods if the node became more schedulable. // We skip the node property comparison when there is no unschedulable pods in the queue // to save processing cycles. We still trigger a move to active queue to cover the case // that a pod being processed by the scheduler is determined unschedulable. We want this // pod to be reevaluated when a change in the cluster happens. if c.podQueue.NumUnschedulablePods() == 0 || nodeSchedulingPropertiesChanged(newNode, oldNode) { c.podQueue.MoveAllToActiveQueue() } </ins> } func (c *configFactory) invalidateCachedPredicatesOnNodeUpdate(newNode *v1.Node, oldNode *v1.Node) {

If it can be changed, I will add this filter later, otherwise I think we can keep the current implementation to make the compare operation simpler and faster. What's more, just as I said in the PR's description, I'm not sure how much side effects this change has on scheduler's resource consumption, as it always copy node's conditions before comparison.
I am thinking whether we could move this logic(whether node became more schedulable) to kubelet, kubelet reports it through heartbeats.


Question: if err != nil { preserveDIP = false } <ins> // targetPort is zero if it is specified as a name in port.TargetPort. // Its real value would be got later from endpoints. targetPort := 0 if port.TargetPort.Type == intstr.Int { targetPort = port.TargetPort.IntValue() } </ins> info := &serviceInfo{ <del> clusterIP: net.ParseIP(service.Spec.ClusterIP), port:      int(port.Port), protocol:  port.Protocol, nodePort:  int(port.NodePort), // targetPort is zero if it is specified as a name in port.TargetPort. // Its real value would be got later from endpoints. targetPort: port.TargetPort.IntValue(), </del> <ins> clusterIP:  net.ParseIP(service.Spec.ClusterIP), port:       int(port.Port), protocol:   port.Protocol, nodePort:   int(port.NodePort), targetPort: targetPort, </ins> // Deep-copy in case the service instance changes loadBalancerStatus:       *service.Status.LoadBalancer.DeepCopy(), sessionAffinityType:      service.Spec.SessionAffinity,

There is no value by that conversion, even worse it can be misused.
should we use in such case?
I do not think this is a bug. 25% == .25 and int(.25) == 0. ;) Even in the case where the user puts in something like 100% I do not think it's a bug. Users should not put strings like "25%" into intOrString and expect to get the integer 25 out. Likely a user doing that is confused about what format the field takes in. I am not sure why the function GetValueFromIntOrPercent exists, and I bet I am not going to like the answer :)
No. First, this depends on how you do the rounding. If you are rounding up (MaxSurge in Deployments), this is 1. Then, the replica size of the deployment also plays a part. 25% of a controller with 1 replica is different from a controller with 4 replicas, etc.
Also, the fact that IntValue returns 0 for every percentage has nothing to do with any logic. It's purely because the conversion errors out:
the first line there was supposed to be a joke, sorry. :) I know that the actual reason for returning 0 is the % sign means it is not an int. More seriously: I actually agree with the parsing function that percentages are not ints. Users who want to do this should not use IntOrString. Is there a place in the API that wants to do this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen /remove-lifecycle stale
Opened to ensure all in-tree callers only call this when the type is Int (or verify it is a parseable number first), and to update the method documentation


Question: "--etcd-keyfile=" + filepath.Join(testCertsDir, "apiserver-etcd-client.key"), }, }, <ins> { name: "authorization-config extra-args", cfg: &kubeadmapi.ClusterConfiguration{ Networking:      kubeadmapi.Networking{ServiceSubnet: "bar", DNSDomain: "cluster.local"}, CertificatesDir: testCertsDir, APIServer: kubeadmapi.APIServer{ ControlPlaneComponent: kubeadmapi.ControlPlaneComponent{ ExtraArgs: []kubeadmapi.Arg{ {Name: "authorization-config", Value: "/path/to/authorization/config/file"}, }, }, }, }, endpoint: &kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "1.2.3.4"}, expected: []string{ "kube-apiserver", "--enable-admission-plugins=NodeRestriction", "--service-cluster-ip-range=bar", "--service-account-key-file=" + filepath.Join(testCertsDir, "sa.pub"), "--service-account-signing-key-file=" + filepath.Join(testCertsDir, "sa.key"), "--service-account-issuer=https://kubernetes.default.svc.cluster.local", "--client-ca-file=" + filepath.Join(testCertsDir, "ca.crt"), "--tls-cert-file=" + filepath.Join(testCertsDir, "apiserver.crt"), "--tls-private-key-file=" + filepath.Join(testCertsDir, "apiserver.key"), "--kubelet-client-certificate=" + filepath.Join(testCertsDir, "apiserver-kubelet-client.crt"), "--kubelet-client-key=" + filepath.Join(testCertsDir, "apiserver-kubelet-client.key"), "--enable-bootstrap-token-auth=true", "--secure-port=123", "--allow-privileged=true", "--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname", "--proxy-client-cert-file=" + filepath.FromSlash("/var/lib/certs/front-proxy-client.crt"), "--proxy-client-key-file=" + filepath.FromSlash("/var/lib/certs/front-proxy-client.key"), "--requestheader-username-headers=X-Remote-User", "--requestheader-group-headers=X-Remote-Group", "--requestheader-extra-headers-prefix=X-Remote-Extra-", "--requestheader-client-ca-file=" + filepath.Join(testCertsDir, "front-proxy-ca.crt"), "--requestheader-allowed-names=front-proxy-client", "--authorization-config=/path/to/authorization/config/file", "--advertise-address=1.2.3.4", fmt.Sprintf("--etcd-servers=https://127.0.0.1:%d", kubeadmconstants.EtcdListenClientPort), "--etcd-cafile=" + filepath.Join(testCertsDir, "etcd/ca.crt"), "--etcd-certfile=" + filepath.Join(testCertsDir, "apiserver-etcd-client.crt"), "--etcd-keyfile=" + filepath.Join(testCertsDir, "apiserver-etcd-client.key"), }, }, { // Note that we do not block it at this level but api server would fail to start. name: "authorization-config and authorization-mode extra-args", cfg: &kubeadmapi.ClusterConfiguration{ Networking:      kubeadmapi.Networking{ServiceSubnet: "bar", DNSDomain: "cluster.local"}, CertificatesDir: testCertsDir, APIServer: kubeadmapi.APIServer{ ControlPlaneComponent: kubeadmapi.ControlPlaneComponent{ ExtraArgs: []kubeadmapi.Arg{ {Name: "authorization-config", Value: "/path/to/authorization/config/file"}, {Name: "authorization-mode", Value: strings.Join([]string{ kubeadmconstants.ModeNode, kubeadmconstants.ModeRBAC, kubeadmconstants.ModeWebhook, }, ",")}, }, }, }, }, endpoint: &kubeadmapi.APIEndpoint{BindPort: 123, AdvertiseAddress: "1.2.3.4"}, expected: []string{ "kube-apiserver", "--enable-admission-plugins=NodeRestriction", "--service-cluster-ip-range=bar", "--service-account-key-file=" + filepath.Join(testCertsDir, "sa.pub"), "--service-account-signing-key-file=" + filepath.Join(testCertsDir, "sa.key"), "--service-account-issuer=https://kubernetes.default.svc.cluster.local", "--client-ca-file=" + filepath.Join(testCertsDir, "ca.crt"), "--tls-cert-file=" + filepath.Join(testCertsDir, "apiserver.crt"), "--tls-private-key-file=" + filepath.Join(testCertsDir, "apiserver.key"), "--kubelet-client-certificate=" + filepath.Join(testCertsDir, "apiserver-kubelet-client.crt"), "--kubelet-client-key=" + filepath.Join(testCertsDir, "apiserver-kubelet-client.key"), "--enable-bootstrap-token-auth=true", "--secure-port=123", "--allow-privileged=true", "--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname", "--proxy-client-cert-file=" + filepath.FromSlash("/var/lib/certs/front-proxy-client.crt"), "--proxy-client-key-file=" + filepath.FromSlash("/var/lib/certs/front-proxy-client.key"), "--requestheader-username-headers=X-Remote-User", "--requestheader-group-headers=X-Remote-Group", "--requestheader-extra-headers-prefix=X-Remote-Extra-", "--requestheader-client-ca-file=" + filepath.Join(testCertsDir, "front-proxy-ca.crt"), "--requestheader-allowed-names=front-proxy-client", "--authorization-config=/path/to/authorization/config/file", "--authorization-mode=Node,RBAC,Webhook", "--advertise-address=1.2.3.4", fmt.Sprintf("--etcd-servers=https://127.0.0.1:%d", kubeadmconstants.EtcdListenClientPort), "--etcd-cafile=" + filepath.Join(testCertsDir, "etcd/ca.crt"), "--etcd-certfile=" + filepath.Join(testCertsDir, "apiserver-etcd-client.crt"), "--etcd-keyfile=" + filepath.Join(testCertsDir, "apiserver-etcd-client.key"), }, }, </ins> } for _, rt := range tests {

TL;DR prevents us from using structured authz (setting flag) as those flags are mutually exclusive /cc (as the author of ) /cc FYI To use structured authz you need to define the authorization configuration using the flag. However this flag is mutually exclusive with and flags. (refer to & ) Kubeadam adds by default ( & ) which prevents us from using I should be able to define the flag and have structured authz working. Spin up a kind cluster using this kind config (make sure the images are 1.29 for structured authz to be avaliable) (For this bug, the content of doesn't matter.) However, the cluster become unhealthy. docker exec to the kind cluster container, where we can see the inner container of the apiserver has exited. Running we can see this error <img width="1720" alt="image" src=""No response_ <details</details<detailsN/A -I am using kind </details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
FYI, in case someone opens similar issue in Kind. I might be wrong but I think this effectively means that kind users cant use structured authz (unless it supports a mode to spin up a cluster without kubeadam).


Question: verifyAttachedVolume(t, attachedVolumes, generatedVolumeName2, string(volumeName), node2Name, devicePath2, true /* expectedMountedByNode */, false /* expectNonZeroDetachRequestedTime */) } <ins> // Test_SetNodeStatusUpdateNeededError expects the map nodesToUpdateStatusFor // to be empty if the SetNodeStatusUpdateNeeded is called on a node that // does not exist in the actual state of the world func Test_SetNodeStatusUpdateNeededError(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) asw := NewActualStateOfWorld(volumePluginMgr) nodeName := types.NodeName("node-1") // Act asw.SetNodeStatusUpdateNeeded(nodeName) // Assert nodesToUpdateStatusFor := asw.GetNodesToUpdateStatusFor() if len(nodesToUpdateStatusFor) != 0 { t.Fatalf("nodesToUpdateStatusFor should be empty as nodeName does not exist") } } </ins> func verifyAttachedVolume( t *testing.T, attachedVolumes []AttachedVolume,

Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: $ kubectl describe pods test-ipj93 Name: test-ipj93 Namespace: default Node: kubernetes-minion-group-yxra/10.128.0.3 Start Time: Mon, 28 Nov 2016 14:39:28 -0800 Labels: <noneStatus: Pending IP: Controllers: <noneContainers: busybox: Container ID: Image: busybox Image ID: Port: Command: sh -c while true; do echo (hostname)-(cat /proc/stat cut -f 4 -d" "); sleep 30; done QoS Tier: cpu: Burstable memory: BestEffort Requests: cpu: 100m State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment Variables: Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: test ReadOnly: false default-token-kjoqg: Type: Secret (a volume populated by a Secret) SecretName: default-token-kjoqg Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 57m 57m 1 {default-scheduler } Normal Scheduled Successfully assigned test-ipj93 to kubernetes-minion-group-yxra 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulling pulling image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Pulled Successfully pulled image "busybox" 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Created Created container with docker id ; Security:[seccomp=unconfined] 57m 57m 1 {kubelet kubernetes-minion-group-yxra} spec.containers{busybox} Normal Started Started container with docker id 47m 47m 6 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.


Question: verifyAttachedVolume(t, attachedVolumes, generatedVolumeName2, string(volumeName), node2Name, devicePath2, true /* expectedMountedByNode */, false /* expectNonZeroDetachRequestedTime */) } <ins> // Test_SetNodeStatusUpdateNeededError expects the map nodesToUpdateStatusFor // to be empty if the SetNodeStatusUpdateNeeded is called on a node that // does not exist in the actual state of the world func Test_SetNodeStatusUpdateNeededError(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) asw := NewActualStateOfWorld(volumePluginMgr) nodeName := types.NodeName("node-1") // Act asw.SetNodeStatusUpdateNeeded(nodeName) // Assert nodesToUpdateStatusFor := asw.GetNodesToUpdateStatusFor() if len(nodesToUpdateStatusFor) != 0 { t.Fatalf("nodesToUpdateStatusFor should be empty as nodeName does not exist") } } </ins> func verifyAttachedVolume( t *testing.T, attachedVolumes []AttachedVolume,

SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig.