Question: if vs, ok := managed.Fields()[self]; ok { delete(managed.Fields(), self) <del> managed.Times()[manager] = &metav1.Time{Time: time.Now().UTC()} </del> if previous, ok := managed.Fields()[manager]; ok { managed.Fields()[manager] = fieldpath.NewVersionedSet(vs.Set().Union(previous.Set()), vs.APIVersion(), vs.Applied()) } else { managed.Fields()[manager] = vs } <ins> // Update the time only if the manager's fieldSet has changed. if formerSet == nil || !managed.Fields()[manager].Set().Equals(formerSet.Set()) { managed.Times()[manager] = &metav1.Time{Time: time.Now().UTC()} } </ins> } return object, managed, nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I'm trying to automate the usage of to check for changes that need to be applied to the cluster based on a static set of manifests. However, for some manifests, if I run the output contains differences on fields called and , after the manifest was applied to the cluster. I ran . After that, I ran The following output was produced (I copied just the first diff, as many more followed): What you expected to happen: No differences should be detected. How to reproduce it (as minimally and precisely as possible): Run Then run Anything else we need to know?: This does not happen to all manifests. If I try with the manifest no differences are detected. I also tried with manifests loaded from disk, and was able to reproduce the behavior with the following manifest: With the above manifest the following occurs: Empirically, the differences observed always seem to be some seconds apart between the two versions. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: baremetalOS (e.g: ):Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig CLI
Ref: The newest version of has a different result still has a problem when diffing :
Sorry, I was unaware that there was a specific repository on which to report kubectl related issues. If needed, I can recreate this issue on the kubernetes/kubectl repository. Do let me know if this is preferred.
/assign
Hey, thanks for the report About the first issue, applying "". It looks like this yaml is dynamically generated and different every time you get it: So I think this one is totally working as intended. We're showing a difference because there is an actual difference. The second one is actually a bug that we are supposed to have fixed in 1.18. Let me look at it.
Ah, I see. That does account for the differences in the date field. Most unfortunate for me, but definitely not a bug in kubernetes/kubelet. My apologies, I should have checked that first.


Question: if vs, ok := managed.Fields()[self]; ok { delete(managed.Fields(), self) <del> managed.Times()[manager] = &metav1.Time{Time: time.Now().UTC()} </del> if previous, ok := managed.Fields()[manager]; ok { managed.Fields()[manager] = fieldpath.NewVersionedSet(vs.Set().Union(previous.Set()), vs.APIVersion(), vs.Applied()) } else { managed.Fields()[manager] = vs } <ins> // Update the time only if the manager's fieldSet has changed. if formerSet == nil || !managed.Fields()[manager].Set().Equals(formerSet.Set()) { managed.Times()[manager] = &metav1.Time{Time: time.Now().UTC()} } </ins> } return object, managed, nil

So the only actual issue is with the changing time field. Thanks for looking into that. Do let me know if there is anything I can do to help.
Can you show me the object before you run that diff? I'm curious what the first "fieldManager" is and why it's different from "kubectl-client-side-apply". Also, did you create the object using ?
I figured out what the problem is in that second message: the cpu goes from an to a : Note that I'm using a non-server-side capable apiserver version, but I think the problem is the same: we can't detect that this object has been changed because cpu goes from to , or from to and we consider this as a change. I have no clue how to solve that "bug".
Hi The test you made is comparing the local yaml against the version the apiserver currently has, correct? That would be similar to running the diff with the option to run server side, which I think yields a suspiciously similar result: I also suspect that it is for this reason that if I run the apply command with the dry-run flag, we always get incoming changes to apply: For comparison sake, the same test with the server-metrics manifest, that does not produce any diff changes, does not detect any incoming changes (notice all of the items are marked with unchanged): I also have no idea how to fix this properly, but perhaps running a yaml formatter on both sources (left and right) before comparison could produce better results? A canonical representation of both inputs might be better suited for comparison. Perhaps such a behavior could be controlled with a new enabling CLI flag. Again, thanks for looking into this. Having a more reliable diff behavior would go a long way for us to detect configuration mismatches against our configuration management system.
Sure thing! I spawned from this bug report to cover the "failed to create typed patch object" problem. I will try to test with the newly minted 1.19 release later today, to see if any previously observed behavior changes.
I recreated the cluster as version 1.19.0, and here are the results of the previous tests: Ran as expected Ran as expected Ran showing unexpected diff results
Thanks for all the details that's actually really useful for me to understand what's going on.
Some more context.


Question: if vs, ok := managed.Fields()[self]; ok { delete(managed.Fields(), self) <del> managed.Times()[manager] = &metav1.Time{Time: time.Now().UTC()} </del> if previous, ok := managed.Fields()[manager]; ok { managed.Fields()[manager] = fieldpath.NewVersionedSet(vs.Set().Union(previous.Set()), vs.APIVersion(), vs.Applied()) } else { managed.Fields()[manager] = vs } <ins> // Update the time only if the manager's fieldSet has changed. if formerSet == nil || !managed.Fields()[manager].Set().Equals(formerSet.Set()) { managed.Times()[manager] = &metav1.Time{Time: time.Now().UTC()} } </ins> } return object, managed, nil

A Gatekeeper user reported behavior where always shows a diff in the timestamp for of its CRD objects -- it looks like non-CRDs are unaffected. This happens when re-applying the static manifest found at
Thanks, this is useful, we'll look into it!


Question: "k8s.io/client-go/kubernetes/fake" core "k8s.io/client-go/testing" "k8s.io/client-go/tools/clientcmd" <del> bootstrapapi "k8s.io/cluster-bootstrap/token/api" </del> kubeadmapiv1beta1 "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1" )

Just got this while running the tests:
let's please move this to /close /sig cluster-lifecycle


Question: return subnet, exists, err } <ins> func (az *Cloud) getAzureLoadBalancer(name string) (lb network.LoadBalancer, exists bool, err error) { var realErr error lb, err = az.LoadBalancerClient.Get(az.ResourceGroup, name, "") exists, realErr = checkResourceExistsFromError(err) if realErr != nil { return lb, false, realErr } if !exists { return lb, false, nil } return lb, exists, err } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Internal Load Balancer created and associated with availability set but no target network ip configurations on Azure. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Create Kubernetes service of type LoadBalancer, annotate with "". Anything else we need to know?: Refer Environment: Kubernetes version (use ): 1.9.1 Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
/assign
/kind bug
Root cause: removes the logic for retrieve private ip. This is required however The controller-manager will crash every time new ilb was created, then will restart and get the IP then. Should get the ip in first round.


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package logs import ( "fmt" "sync" "github.com/go-logr/logr" ) var logRegistry = NewLogFormatRegistry() // LogFormatRegistry store klog format registry type LogFormatRegistry struct { registry map[string]logr.Logger mu       sync.Mutex } // NewLogFormatRegistry return new init LogFormatRegistry struct func NewLogFormatRegistry() *LogFormatRegistry { return &LogFormatRegistry{ registry: make(map[string]logr.Logger), mu:       sync.Mutex{}, } } // Register new log format registry to global logRegistry func (lfr *LogFormatRegistry) Register(name string, logger logr.Logger) error { lfr.mu.Lock() defer lfr.mu.Unlock() if _, ok := lfr.registry[name]; ok { return fmt.Errorf("log format: %s already exists", name) } lfr.registry[name] = logger return nil } // Get specified log format logger func (lfr *LogFormatRegistry) Get(name string) (logr.Logger, error) { lfr.mu.Lock() defer lfr.mu.Unlock() re, ok := lfr.registry[name] if !ok { return nil, fmt.Errorf("log format: %s does not exists", name) } return re, nil } // Set specified log format logger func (lfr *LogFormatRegistry) Set(name string, logger logr.Logger) { lfr.mu.Lock() defer lfr.mu.Unlock() lfr.registry[name] = logger } // Delete specified log format logger func (lfr *LogFormatRegistry) Delete(name string) { lfr.mu.Lock() defer lfr.mu.Unlock() delete(lfr.registry, name) } func init() { // Text format is default klog format logRegistry.Register("text", nil) } </ins>

What would you like to be : As part of should add logging configuration allowing to change logging implementation. Things to implement: --logging-format - flag to pick between available implementations. Defaults to "text" using unchanged klog. LoggingConfig - structure storing component configuration for logging LogFormatRegistry - Used for registration of additional log formats. New formats should be registered by providing unique name and implementation of interface LoggingConfig.Complete method should configure klog by passing logr interface implementation into klog.SetLogger method Why is this needed: Subtask of /help /wg component-standard
The klog 1.0.0 version looks like no SetLogger method, should update klog version in vendor?
yes, forgot about that. I remember saying that it will require some changes, but shouldn't be to hard. I will help with it if needed.
/assign /remove-help
Klog was upgraded to v2.


Question: if apierrs.IsMethodNotSupported(err) || apierrs.IsNotFound(err) || apierrs.IsForbidden(err) { continue } <ins> // skip unavailable servers if apierrs.IsServiceUnavailable(err) { continue } </ins> return false, err } unstructuredList, ok := obj.(*unstructured.UnstructuredList)

https://k8s- https://k8s- https://k8s- https://k8s- https://k8s- ... A lot of them /cc
/assign
Still see this failure after is merged, although less frequently: https://k8s- https://k8s- Reopen.
Looks like the client doesn't differentiate between 500 and 503 here: It should. Will open a fix shortly
opened
fix merged, will monitor https://k8s-
Thanks! :)
haven't seen this in the 12 hours since the fix merged, was seeing it every 2-3 hours before. /close


Question: TEMP_DIR:=$(shell mktemp -d) ifeq ($(ARCH),amd64) <del> BASEIMAGE?=k8s.gcr.io/build-image/debian-base:v2.1.3 </del> <ins> BASEIMAGE?=k8s.gcr.io/build-image/debian-base:buster-v1.2.0 </ins> endif ifeq ($(ARCH),arm) <del> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-arm:v2.1.3 </del> <ins> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-arm:buster-v1.2.0 </ins> endif ifeq ($(ARCH),arm64) <del> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-arm64:v2.1.3 </del> <ins> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-arm64:buster-v1.2.0 </ins> endif ifeq ($(ARCH),ppc64le) <del> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-ppc64le:v2.1.3 </del> <ins> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-ppc64le:buster-v1.2.0 </ins> endif ifeq ($(ARCH),s390x) <del> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-s390x:v2.1.3 </del> <ins> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-s390x:buster-v1.2.0 </ins> endif RUNNERIMAGE?=gcr.io/distroless/static:latest

What would you like to be : Have debian-iptables with iptables 1.8.5+ Why is this needed: To fix (nodelocaldns + iptables-nft) More informations Found a bug in iptables-nft, fixed in iptables 1.8.5, and now available in buster-backports https://alioth-
/sig api-machinery
ping as you did the last 2 updates
Yep, this is on my list. Will take of it next week. /sig release network /area release-eng /milestone v1.20 /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Opened to bump the debian-iptables images in kubernetes/kubernetes. /remove-triage unresolved
Here are the digests for the new image ():
Please reopen if you run into any issues with the new images. /close
Closing this issue. <detailsIn response to : Please reopen if you run into any issues with the new images. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: framework.ExpectError(err) gomega.Expect(err).To(gomega.ContainSubstring("Using in-cluster namespace")) gomega.Expect(err).To(gomega.ContainSubstring("Using in-cluster configuration")) <del> gomega.Expect(err).To(gomega.ContainSubstring("Authorization: Bearer invalid")) </del> gomega.Expect(err).To(gomega.ContainSubstring("Response Status: 401 Unauthorized")) ginkgo.By("trying to use kubectl with invalid server")

This issue was reported in the Description Kubernetes requires an authentication mechanism to enforce users’ privileges. One method of authentication, bearer tokens, are opaque strings used to associate a user with their having successfully authenticated previously. Any user with possession of this token may masquerade as the original user (the “bearer”) without further authentication. Within Kubernetes, the bearer token is captured within the hyperkube kube-apiserver system logs at high verbosity levels (--v 10). A malicious user with access to the system logs on such a system could masquerade as any user who has previously logged into the system. Exploit Scenario Alice logs into a Kubernetes cluster and is issued a Bearer token. The system logs her token. Eve, who has access to the logs but not the production Kubernetes cluster, replays Alice’s Bearer token, and can masquerade as Alice to the cluster. Recommendation Short term, remove the Bearer token from the log. Do not log any authentication credentials within the system, including tokens, private keys, or passwords that may be used to authenticate to the production Kubernetes cluster, regardless of the logging level. Long term, either implement policies that enforce code review to ensure that sensitive data is not exposed in logs, or implement logging filters that check for sensitive data and remove it prior to outputting the log. In either case, ensure that sensitive data cannot be trivially stored in logs. Anything else we need to know?: See for current status of all issues created from these findings. The vendor gave this issue an ID of TOB-K8S-001 and it was finding 6 of the report. The vendor considers this issue Medium Severity. To view the original finding, begin on page 31 of the Environment: Kubernetes version: 1.13.4
/sig auth /sig api-machinery
/remove-kind feature /kind bug
Should remote the "Authorization" header here?
for authorization headers, everything after the type should be masked (and potentially, only recognized auth types should be logged). Something like:
I think people would be unpleasantly surprised to see credentials in a logfile after turning up logging to debug an issue. /priority important-soon
Does either of you intend to createe PR to remove the "Authorization" header ?


Question: framework.ExpectError(err) gomega.Expect(err).To(gomega.ContainSubstring("Using in-cluster namespace")) gomega.Expect(err).To(gomega.ContainSubstring("Using in-cluster configuration")) <del> gomega.Expect(err).To(gomega.ContainSubstring("Authorization: Bearer invalid")) </del> gomega.Expect(err).To(gomega.ContainSubstring("Response Status: 401 Unauthorized")) ginkgo.By("trying to use kubectl with invalid server")

This issue seems CVE-worthy to me: loss of confidentiality (bearer tokens) + cross of privilege boundary (debug log level could be set by admin, then logs read later by low privileged user). My draft cvss would be: CVSS:3.0/AV:N/AC:H/PR:L/UI:N/S:U/C:H/I:N/A:N (5.3)
Two potential vectors: local user with access to system logs (local access, low privilege) (4.7) remote user with access to system logs (network access, high privilege): (4.4) CVE-2019- has been assigned to this issue.
If you want to, go ahead
- should this have the official CVE feed label ( retrospectively, I mean)?
probably? I have no idea how that label is supposed to get applied, but there appear to be several CVE issues that don't have it -
I think you're likely to know who to highlight this to. Sorry to mention you individually; please feel free to point out docs if there are any.
/label official-cve-feed thanks for the tag, the label now!
thanks for the useful filter. Currently in-scope issues are issues that were closed as fixed by code changes to k/k. So with that in mind, I have the labels to some of the closed issues too that conform to the scope defined here:
I would like to reply nearly four years later: There are valid cases to want to see the token, for example my current case, trying to debug token validation between the API server and other services.


Question: klog.V(2).InfoS("Preemptor Pod preempted victim Pod", "preemptor", klog.KObj(pod), "victim", klog.KObj(victim), "node", c.Name()) } <del> fh.EventRecorder().Eventf(victim, pod, v1.EventTypeNormal, "Preempted", "Preempting", "Preempted by a pod on node %v", c.Name()) </del> <ins> fh.EventRecorder().Eventf(victim, pod, v1.EventTypeNormal, "Preempted", "Preempting", "Preempted by pod %v on node %v", pod.UID, c.Name()) </ins> } fh.Parallelizer().Until(ctx, len(c.Victims().Pods), preemptPod, ev.PluginName)

/sig scheduling Include the Pod UID in the event message for preemption by kube-scheduler Minimizes the need for an end user to consult the ops team to learn about what caused their pods to be preempted, without compromising PII. From
/triage accepted
Can I pick this up?
Yes, just type . Note that we are in code freeze, so this won't be merged until about a month from now.
/assign
PII? I know that as personally identifying information (ie, data about humans).
BTW, ideally we'd also document the change.
In a multi-tenant environment, pod names could be sensitive information.


Question: - key: cloud.google.com/gke-accelerator operator: Exists tolerations: <del> - key: "nvidia.com/gpu" </del> <ins> - operator: "Exists" effect: "NoExecute" - operator: "Exists" </ins> effect: "NoSchedule" <del> operator: "Exists" </del> hostNetwork: true hostPID: true volumes:

This issue is here to satisfy PR requirements of issue association. A lot of times we need to do version bumps for minor bug fixes that don't deserve a separate issue in the kubernetes/kubernetes repo. /kind bug /sig node
/lifecycle frozen
all the linked PRs are emrged and no activity since more than 3 years /close
Closing this issue. <detailsIn response to : all the linked PRs are emrged and no activity since more than 3 years /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // another goroutine isn't already in action. func (kl *Kubelet) podKiller() { killing := sets.NewString() <del> resultCh := make(chan types.UID) defer close(resultCh) </del> <ins> // guard for the killing set lock := sync.Mutex{} </ins> for { select { case podPair, ok := <-kl.podKillingCh:

There is a defer calling in this method: defer close(resultCh) It may crash the goroutine created at line 777, which then will make the whole kubelet crash. A simple fix Another simple fix
cc do you think this make sense?
You could just drop the channel and go with a mutex to update the set.
indeed.


Question: }, } <del> elector, err := NewLeaderElector(lec) if err != nil { t.Fatal("Failed to create leader elector: ", err) </del> <ins> tests := []struct { name     string reactors []Reactor }{ { name: "release acquired lock on cancellation of update", reactors: []Reactor{ { verb:       "get", objectType: objectType, reaction: func(action fakeclient.Action) (handled bool, ret runtime.Object, err error) { gets++ if lockObj != nil { return true, lockObj, nil } return true, nil, errors.NewNotFound(action.(fakeclient.GetAction).GetResource().GroupResource(), action.(fakeclient.GetAction).GetName()) }, }, { verb:       "create", objectType: objectType, reaction: func(action fakeclient.Action) (handled bool, ret runtime.Object, err error) { lockObj = action.(fakeclient.CreateAction).GetObject() return true, lockObj, nil }, }, { verb:       "update", objectType: objectType, reaction: func(action fakeclient.Action) (handled bool, ret runtime.Object, err error) { updates++ // Second update (first renew) should return our canceled error // FakeClient doesn't do anything with the context so we're doing this ourselves if updates == 2 { close(onRenewCalled) <-onRenewResume return true, nil, context.Canceled } else if updates == 3 { // We update the lock after the cancellation to release it // This wg is to avoid the data race on lockObj defer wg.Done() close(onRelease) } lockObj = action.(fakeclient.UpdateAction).GetObject() return true, lockObj, nil }, }, }, }, { name: "release acquired lock on cancellation of get", reactors: []Reactor{ { verb:       "get", objectType: objectType, reaction: func(action fakeclient.Action) (handled bool, ret runtime.Object, err error) { gets++ if lockObj != nil { // Third and more get (first create, second renew) should return our canceled error // FakeClient doesn't do anything with the context so we're doing this ourselves if gets >= 3 { close(onRenewCalled) <-onRenewResume return true, nil, context.Canceled } return true, lockObj, nil } return true, nil, errors.NewNotFound(action.(fakeclient.GetAction).GetResource().GroupResource(), action.(fakeclient.GetAction).GetName()) }, }, { verb:       "create", objectType: objectType, reaction: func(action fakeclient.Action) (handled bool, ret runtime.Object, err error) { lockObj = action.(fakeclient.CreateAction).GetObject() return true, lockObj, nil }, }, { verb:       "update", objectType: objectType, reaction: func(action fakeclient.Action) (handled bool, ret runtime.Object, err error) { updates++ // Second update (first renew) should release the lock if updates == 2 { // We update the lock after the cancellation to release it // This wg is to avoid the data race on lockObj defer wg.Done() close(onRelease) } lockObj = action.(fakeclient.UpdateAction).GetObject() return true, lockObj, nil }, }, }, }, </ins> } <del> ctx, cancel := context.WithCancel(context.Background()) </del> <ins> for i := range tests { test := &tests[i] t.Run(test.name, func(t *testing.T) { wg.Add(1) resetVars() </ins> <del> go elector.Run(ctx) </del> <ins> resourceLockConfig := rl.ResourceLockConfig{ Identity:      "baz", EventRecorder: &record.FakeRecorder{}, } c := &fake.Clientset{} for _, reactor := range test.reactors { c.AddReactor(reactor.verb, objectType, reactor.reaction) } c.AddReactor("*", "*", func(action fakeclient.Action) (bool, runtime.Object, error) { t.Errorf("unreachable action. testclient called too many times: %+v", action) return true, nil, fmt.Errorf("unreachable action") }) lock, err := rl.New(objectType, "foo", "bar", c.CoreV1(), c.CoordinationV1(), resourceLockConfig) if err != nil { t.Fatal("resourcelock.New() = ", err) } </ins> <del> // Wait for us to become the leader select { case <-onNewLeader: case <-time.After(10 * time.Second): t.Fatal("failed to become the leader") } </del> <ins> lec.Lock = lock elector, err := NewLeaderElector(lec) if err != nil { t.Fatal("Failed to create leader elector: ", err) } </ins> <del> // Wait for renew (update) to be invoked select { case <-onRenewCalled: case <-time.After(10 * time.Second): t.Fatal("the elector failed to renew the lock") } </del> <ins> ctx, cancel := context.WithCancel(context.Background()) </ins> <del> // Cancel the context - stopping the elector while // it's running cancel() </del> <ins> go elector.Run(ctx) </ins> <del> // Resume the update call to return the cancellation // which should trigger the release flow close(onRenewResume) </del> <ins> // Wait for us to become the leader select { case <-onNewLeader: case <-time.After(10 * time.Second): t.Fatal("failed to become the leader") } // Wait for renew (update) to be invoked select { case <-onRenewCalled: case <-time.After(10 * time.Second): t.Fatal("the elector failed to renew the lock") } </ins> <del> select { case <-onRelease: case <-time.After(10 * time.Second): t.Fatal("the lock was not released") </del> <ins> // Cancel the context - stopping the elector while // it's running cancel() // Resume the tryAcquireOrRenew call to return the cancellation // which should trigger the release flow close(onRenewResume) select { case <-onRelease: case <-time.After(10 * time.Second): t.Fatal("the lock was not released") } wg.Wait() }) </ins> } }

The lock acquired by tryAcquireOrRenew is released when the leader ends leadership. However, due to the cancellation of the context, the lock may be set as an empty lock, so the Update cannot be run normally, resulting in a failure to release the lock. The lock can be released every time the leader ends. run this: and then you can got a log like these: and you when you watch the leases at the same time: you can see that was failed to release because of No response <details</details<details</details<details</details<details</details<details</detailsrelated packages: v1.3.0 v0.25.0 v0.25.0 v2.70.1
/sig api-machinery
/triage accepted thanks!
/unassign
/assign
/unassign
/unassign me


Question: func stringBody(body string) io.ReadCloser { return ioutil.NopCloser(bytes.NewReader([]byte(body))) } <ins> // Verify that resource.RESTClients constructed from a factory respect mapping.APIVersion func TestClientVersions(t *testing.T) { f := NewFactory(nil) versions := []string{ "v1beta1", "v1beta2", "v1beta3", } for _, version := range versions { mapping := &meta.RESTMapping{ APIVersion: version, } c, err := f.RESTClient(nil, mapping) if err != nil { t.Errorf("unexpected error: %v", err) } client := c.(*client.RESTClient) if client.APIVersion() != version { t.Errorf("unexpected Client APIVersion: %s %v", client.APIVersion, client) } } } </ins>

I stumbled across this while experimenting with kubeconfig load order/merging. There are various helpers constructed from a Factory that use , however this always uses the same default ClientConfig because doesn't use the passed in version param: Am I missing something or is this a bug? /cc
Looks like a bug, clientforversion should accept "" (which can provide a default) and SetKubernetesDefaults has the final fallback protection (by using latest.Version). If version != defaultConfig.Version, we should copy defaultConfig, set version, then apply kubedefaults. The reason we take version here is if a resource is specified via file which only exists in a newer version, ie user says: --api-version=v1beta1 -f RESTMapper can handle that newresource is only valid for v1beta3 and apply the right override, then ask for client. --api-version is expected to bubble up via default config.


Question: if err != nil || portal != "127.0.0.1:3260" || iqn != "eui.02004567A425678D" { t.Errorf("extractPortalAndIqn: got %v %s %s", err, portal, iqn) } <ins> devicePath = "[2001:db8:0:f101::1]:3260-iqn.2014-12.com.example:test.tgt00-lun-0" portal, iqn, err = extractPortalAndIqn(devicePath) if err != nil || portal != "[2001:db8:0:f101::1]:3260" || iqn != "iqn.2014-12.com.example:test.tgt00" { t.Errorf("extractPortalAndIqn: got %v %s %s", err, portal, iqn) } </ins> } func TestRemoveDuplicate(t *testing.T) {

Kubelet was not able to attach + mount an iSCSI PV when it has IPv6 server address. Volume is mounted + attached. Set up iSCSI target that's reachable over IPv6 and create a PV for it. For example: Kubelet logs: Notice around the IP address. Various iSCSI files in /sys and /dev/disk/by-path do not have these square brackets and kubelet is then not able to find the iSCSI host in /sys. That's where comes from. No response <detailsA recent master: console NAME="Fedora Linux" VERSION="36 (Workstation Edition)" </details<detailshack/local-up- </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-soon
/assign


Question: // Wait for the DaemonSet to be created before proceeding err = waitForDaemonSetAndControllerRevisionCreated(clientset, ds.Name, ds.Namespace) if err != nil { <del> t.Fatalf("Failed to create DeamonSet: %v", err) </del> <ins> t.Fatalf("Failed to create DaemonSet: %v", err) </ins> } ds, err = dsClient.Get(ds.Name, metav1.GetOptions{})

https://k8s- /sig apps /priority critical-urgent /assign PR that the test: /cc
GitHub didn't allow me to assign the following users: mortent. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : https://k8s- /priority critical-urgent /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake
I'm looking into this.
The test is still flaky, but with different symptom: An example flake: https://k8s- It's likely because the collision count was updated in a previous sync loop, but the in the cache store hasn't been updated and we used a stale collision count while calculating the new hash: /assign


Question: // the various afterEaches afterEaches map[string]AfterEachActionFunc <ins> // beforeEachStarted indicates that BeforeEach has started beforeEachStarted bool </ins> // configuration for framework's client Options Options

What happened: When running external CSI Driver tests with the Kubernetes E2E: We seem to be getting the following error. From initial investigation it looks like this happens in the during check but only when the test is being skipped. Hunch is that maybe since the test is skipped some resources are never created or they are cleaned up already and they are nil when the is called and expecting them. Kind of low priority since it doesn't make any tests actually fail. /cc /sig storage
see
I think should address this


Question: # etcd - name: "etcd" <del> version: 3.5.0-rc.0 </del> <ins> version: 3.5.0 </ins> refPaths: - path: cluster/gce/manifests/etcd.manifest match: etcd_docker_tag|etcd_version

/sig scalability This issue is tracking process of upgrading etcd to 3.5.0 planned for 1.22 K8s release. Etcd has recently released Beta versions and is starting the process of correctness and performance testing. To reduce change of finding performance regression late, we will start qualifying Beta Etcd releases on GKE master branch and will upgrade to 3.5.0 as soon as it is available. Steps: [x] Do preliminary scale testing of etcd v3.5.0-beta.3 release [x] Release etcd v3.5.0-beta.3 image (, ) [x] Release etcd v3.5.0-rc.0 image () [x] Upgrade etcd server to v3.5.0-rc.0 () [x] Wait for 3.5.0 release and upgrade etcd client () [x] Release etcd v3.5.0 image (, ) [x] Upgrade etcd server to v3.5.0 /milestone v1.22
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area etcd
Does this meant that the 3.5.0 should be promoted like
Yes, created
/area release-eng release-team /sig release For tracking: cc:
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /area release-eng release-team /sig release cc: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I do think this can be pointed in the Major Themes for the 1.22 release. I would like your opinions on this. If you do think it has a place in it, I will add it to the .
How can I be a contributor to this issue? what are the technical checklist that needs to be achieved to close out on the issue?


Question: # etcd - name: "etcd" <del> version: 3.5.0-rc.0 </del> <ins> version: 3.5.0 </ins> refPaths: - path: cluster/gce/manifests/etcd.manifest match: etcd_docker_tag|etcd_version

PR to update Etcd is ready, but we found a blocker for merging . There is one test failing I will take a look into it on Monday, still more eyes on the problem could speed up resolution.
We'll also need a new kube-cross image w/ etcd v3.5.0. I'm working on that here:
We just need to upgrade the server to use new image like in
ok, looks like is busy. Let me take a look /assign
Thanks.


Question: "fmt" "io" "net/url" <ins> "strings" </ins> "time" "github.com/spf13/cobra"

When attaching to an ephemeral container, prints a message suggesting that the container can be reattached after it exists. This is not true because an ephemeral container is never restarted. This cosmetic issue should be resolved for
/assign /sig node /triage accepted /priority important-longterm


Question: function ensure-local-disks() { <ins> # Skip if already mounted (a reboot) if ( grep "/mnt/ephemeral" /proc/mounts ); then echo "Found /mnt/ephemeral in /proc/mounts; skipping local disk initialization" return fi </ins> block_devices=() ephemeral_devices=$(curl --silent http://169.254.169.254/2014-11-05/meta-data/block-device-mapping/ | grep ephemeral)

On AWS we (will) now run our bootstrap script on every boot. The function to initialize ephemeral volumes doesn't tolerate being re-run, and we probably shouldn't re-run it anyway for speed. This does preserve the contents, which may be a good thing to save user's data (it doesn't belong on ephemeral drives, but...). On the other hand, if the ephemeral drives are corrupted, we lose the opportunity to reformat them and recover. Arguably, if the ephemeral storage is corrupted the right thing to do is to terminate the instance entirely.


Question: <ins> apiVersion: "v1beta1" kind: "ReplicationController" id: "monitoring-influxGrafanaController" desiredState: replicas: 1 replicaSelector: name: "influxGrafana" podTemplate: labels: name: "influxGrafana" desiredState: manifest: version: "v1beta1" id: "monitoring-influxGrafanaController" containers: - name: "influxdb" image: "kubernetes/heapster_influxdb:v0.2" ports: - containerPort: 8083 hostPort: 8083 - containerPort: 8086 hostPort: 8086 - name: "grafana" image: "kubernetes/heapster_grafana:v0.2" ports: - containerPort: 80 hostPort: 80 env: - name: "HTTP_USER" value: "admin" - name: "HTTP_PASS" value: "**None**" labels: name: "influxGrafana" </ins>

examples/monitoring/influx-grafana- and friends NEED to be replaced by a replication controller ASAP. As it stands they are bad examples. Also, they should probably move to contrib/monitoring to match contrib/logging and contrib/dns


Question: { Object: map[string]interface{}{ "apiVersion": "example.com/v1", <ins> "metadata":   map[string]interface{}{}, </ins> "kind":       "foo", "other":      "data2", },

If a CR converter is wired in and it modifies the input list, the delegating CR converter does not protect against this. A converter cannot effect the input list Not really possible externally - it's a coding issue that the current converters don't hit. No response <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery /assign
/triage accepted


Question: results := []reflect.Value{} switch node.Name { case "range": <del> j.stack = append(j.stack, j.cur) </del> j.beginRange++ results = input case "end": <del> if j.endRange < j.inRange { // inside a loop, break the current block </del> <ins> if j.inRange > 0 { </ins> j.endRange++ <del> break } // the loop is about to end, pop value and continue the following execution if len(j.stack) > 0 { j.cur, j.stack = j.stack[len(j.stack)-1], j.stack[:len(j.stack)-1] </del> } else { return results, fmt.Errorf("not in range, nothing to end") }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: (or in my instance) with does not correctly handle nested range statements. For example, one would expect that the following: would result in a list being output similar to this: but instead, results in a list being output like this: What you expected to happen: Nested range statements complete in the order in which they are specified in jsonpath query. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.11.0 Cloud provider or hardware configuration: OpenShift 3.11.141 on vSphere 6.5 OS (e.g: ): RHEL 7.7 Kernel (e.g. ): 3.10.0 Install tools:
/sig cli
/area kubectl
/assign
/sig api-machinery
/remove-sig cli
/remove-area cli /area api
Those labels are not set on the issue: <detailsIn response to : /remove-area cli /area api Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-area kubectl


Question: volume.MetricsNil } <del> func detachDiskLogError(cd *cinderVolume) { err := cd.manager.DetachDisk(&cinderVolumeUnmounter{cd}) if err != nil { glog.Warningf("Failed to detach disk: %v (%v)", cd, err) } } </del> func (b *cinderVolumeMounter) GetAttributes() volume.Attributes { return volume.Attributes{ ReadOnly:        b.readOnly,

while reading some cinder code, i came accross , we try to detach disk even if we didn't try to mount. Is this call mandatory ? (from what i recall AWS EBS do not have such mecanism) Also ping
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack
cc
that's was before attach/detach controller, a revisit is necessary
Sure i'll take care of it /assign


Question: "net" "strings" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/types" utilnet "k8s.io/apimachinery/pkg/util/net" utilruntime "k8s.io/apimachinery/pkg/util/runtime"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using Kubernetes 1.16.1, I started kube-proxy in iptables mode as a static pod on a master node. The master node was intentionally (and permanently) not available in the API as a Node object. kube-proxy started up and terminated soon after because it failed to (which is supposed to function as the bound address) and eventually returned an error: What you expected to happen: I expected the same behavior as in Kubernetes 1.15 where failure to look up the Node in the API would result in . How to reproduce it (as minimally and precisely as possible): Configure kube-proxy with iptables mode on Kubernetes 1.16 without the hosting Node being available in the Kubernetes API (under the OS hostname). Anything else we need to know?: PR (Fix for kube-proxy to wait for some duration for the node to be defined) introduced an additional . I believe this is the part that is causing the change in behavior: in Kubernetes 1.15, the check would be omitted and cause to set the default bind address to 127.0.0.1 (when iptables mode is set; I haven't checked other modes). The PR's primary purpose seems to be a different one (retrying node IP retrieval from the API), so my understanding is that the check was not really needed; unless I'm misunderstanding things which I don't want to rule out at all. The quick workaround was to explicitly configure kube-proxy with . However, I'm not entirely sure if that's a good solution long-term compared to letting Kubernetes do The Right Thing, which is what I'd like to clarify through this issue. Environment: Kubernetes version (use ): v1.16.1 Cloud provider or hardware configuration: Kubernetes running on DigitalOcean VMs (droplets) OS (e.g: ): Debian GNU/Linux 9 (stretch) Kernel (e.g.


Question: "net" "strings" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/types" utilnet "k8s.io/apimachinery/pkg/util/net" utilruntime "k8s.io/apimachinery/pkg/util/runtime"

): Linux master-k8s-1-16-test1 4.19.0-0.bpo.6-amd64 SMP Debian 4.19.67-2+deb10u1~bpo9+1 (2019-09-30) x86_64 GNU/Linux /sig network /cc who were involved in the referenced PR.
I made a similar comment in , wondering if this is a breaking change in v1.16 that we should revert? Previously if is nil, kube-proxy defaults to but now kube-proxy always exits with error instead of defaulting to local host. I think a lot of external cloud providers relied on this behavior because they needed to talk to API server prior to node addresses being set by the provider. Thoughts /assign
Opened to preserve behavior before v1.16 where node IP defaults to 127.0.01 is none is found. PTAL


Question: // TLSMinVersion is the minimum TLS version supported. // Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). TLSMinVersion string <ins> // rotateCertificates enables client certificate rotation. The Kubelet will request a // new certificate from the certificates.k8s.io API. This requires an approver to approve the // certificate signing requests. The RotateKubeletClientCertificate feature // must be enabled. RotateCertificates bool </ins> // serverTLSBootstrap enables server certificate bootstrap. Instead of self // signing a serving certificate, the Kubelet will request a certificate from // the certificates.k8s.io API. This requires an approver to approve the

/kind cleanup What happened: and both require opt-in configuration knobs to the kubelet. They are kinda global settings, you probably either opt into dynamic cert rotation or you don't, so they qualify for being in the struct. The knob for enabling dynamic serving certs was (I fully support that :+1:). What you expected to happen: I'd like the same to happen to  to be consistent. (Also, we're planning to use the config file for the kubelet by default in kubeadm soon, and it'd be nice to be able to set this value then in a structured manner) /assign


Question: // TLSMinVersion is the minimum TLS version supported. // Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). TLSMinVersion string <ins> // rotateCertificates enables client certificate rotation. The Kubelet will request a // new certificate from the certificates.k8s.io API. This requires an approver to approve the // certificate signing requests. The RotateKubeletClientCertificate feature // must be enabled. RotateCertificates bool </ins> // serverTLSBootstrap enables server certificate bootstrap. Instead of self // signing a serving certificate, the Kubelet will request a certificate from // the certificates.k8s.io API. This requires an approver to approve the

Flag name: Help text: <Warning: Beta featureAuto rotate the kubelet client certificates by requesting new certificates from the kube-apiserver when the certificate expiration approaches. This is part of migrating the Kubelet command-line to a Kubernetes-style API. The flag should either be migrated to the Kubelet's API group, or simply removed from the Kubelet. If this could be considered an instance-specific flag, or a descriptor of local topology managed by the Kubelet, see: If this flag is only registered in os-specific builds, see: /assign /sig node /kind feature /priority important-soon /milestone v1.11 /status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : New functionality. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: Context("when running a container with invalid image", func() { It("it should run with the expected status [Conformance]", func() { testContainer := api.Container{ <del> Image:           "foo.com/foo/foo", Command:         []string{"false"}, ImagePullPolicy: api.PullIfNotPresent, </del> <ins> Image:   "foo.com/foo/foo", Command: []string{"false"}, </ins> } <del> testStatus := testStatus{"invalid-image-rpa", api.RestartPolicyAlways, api.PodPending, ContainerStateWaiting, "==", 0, false} testContainer.Name = testStatus.Name </del> <ins> testCase := testCase{"invalid-image-rpa", api.RestartPolicyAlways, api.PodPending, ContainerStateWaiting, "==", 0, false} testContainer.Name = testCase.Name </ins> invalidImageContainer := ConformanceContainer{ Container:     testContainer, Client:        cl, <del> RestartPolicy: testStatus.RestartPolicy, </del> <ins> RestartPolicy: testCase.RestartPolicy, </ins> NodeName:      *nodeName, Namespace:     namespace, } <del> err := invalidImageContainer.Create() Expect(err).NotTo(HaveOccurred()) testCContainers = append(testCContainers, invalidImageContainer) </del> <ins> Expect(invalidImageContainer.Create()).To(Succeed()) defer invalidImageContainer.Delete() </ins> <del> var status api.ContainerStatus var phase api.PodPhase </del> <ins> Eventually(invalidImageContainer.GetPhase, retryTimeout, pollInterval).Should(Equal(testCase.Phase)) Consistently(invalidImageContainer.GetPhase, consistentCheckTimeout, pollInterval).Should(Equal(testCase.Phase)) </ins> <del> Consistently(func() api.PodPhase { if status, phase, err = invalidImageContainer.GetStatus(); err != nil { return api.PodPending } else { return phase } }, consistentCheckTimeout, pollInterval).Should(Equal(testStatus.Phase)) </del> <ins> status, err := invalidImageContainer.GetStatus() </ins> Expect(err).NotTo(HaveOccurred()) By("it should get the expected 'RestartCount'") <del> Expect(status.RestartCount).To(BeNumerically(testStatus.RestartCountOper, testStatus.RestartCount)) </del> <ins> Expect(status.RestartCount).To(BeNumerically(testCase.RestartCountOper, testCase.RestartCount)) </ins> By("it should get the expected 'Ready' status") <del> Expect(status.Ready).To(Equal(testStatus.Ready)) </del> <ins> Expect(status.Ready).To(Equal(testCase.Ready)) </ins> By("it should get the expected 'State'") <del> Expect(GetContainerState(status.State) & testStatus.State).NotTo(Equal(0)) </del> <ins> Expect(GetContainerState(status.State)).To(Equal(testCase.State)) </ins> By("it should be possible to delete [Conformance]") <del> err = invalidImageContainer.Delete() Expect(err).NotTo(HaveOccurred()) Eventually(func() bool { isPresent, err := invalidImageContainer.Present() return err == nil && !isPresent }, retryTimeout, pollInterval).Should(BeTrue()) </del> <ins> Expect(invalidImageContainer.Delete()).To(Succeed()) Eventually(invalidImageContainer.Present, retryTimeout, pollInterval).Should(BeFalse()) </ins> }) }) <del> AfterEach(func() { for _, cc := range testCContainers { cc.Delete() } }) </del> }) })

Slow conformance test has caused issues like We should make runtime/image conformance test run faster without breaking the test coverage, possibly by: Shorten consistently check interval . Remove redundant test cases such as ... /cc


Question: if err != nil { return nil, err } <ins> name := accessor.GetName() if len(name) == 0 { return nil, fmt.Errorf("name is required") } </ins> outBytes, err := runtime.Encode(unstructured.UnstructuredJSONScheme, obj) if err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Providing an empty to the dynamic client's method results in a panic. What you expected to happen: An error should be returned on empty . How to reproduce it (as minimally and precisely as possible): Call with Anything else we need to know?: The expects a nonempty to be provided, but does not validate this. If an empty name is provided, this method will panic because it expects an , but the response will actually be an . Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: client-go 10.0.0
/sig api-machinery


Question: // Name of the default http backend service defaultBackendName = "default-http-backend" <del> // GCEL7SrcRange is the IP src range from which the GCE L7 performs health checks. GCEL7SrcRange = "130.211.0.0/22" </del> // Cloud resources created by the ingress controller older than this // are automatically purged to prevent running out of quota. // TODO(37335): write soak tests and bump this up to a week.

From GCE email today: Current behavior: You are not required to add firewall rules for Network LB health checks today Action required: We are introducing three new ranges for health checking: 209.85.152.0/22, 209.85.204.0/22, and 35.191.0.0/16. You need to ensure that your firewall allows traffic from 209.85.152.0/22, 209.85.204.0/22, and 35.191.0.0/16. The health check probes to your backend instances will come from one of these ranges. We have until Jan 31, 2018 - 1 year.
/assign this issue explains the confusion I had several weeks ago when speaking with and you about L4 health checks. I didn't realize a firewall rule isn't currently required.
/assign
Is this done now?
Not yet, there is a WIP on .
/close


Question: patch, and delete operations. release: v1.26 file: test/e2e/storage/csi_inline.go <del> - testname: CSIInlineVolumes should support ephemeral CSIDrivers codename: '[sig-storage] CSIInlineVolumes should support ephemeral VolumeLifecycleMode in CSIDriver API [Conformance]' description: CSIDriver resources with ephemeral VolumeLifecycleMode should support create, get, list, and delete operations. release: v1.26 file: test/e2e/storage/csi_inline.go </del> - testname: CSIStorageCapacity API codename: '[sig-storage] CSIStorageCapacity  should support CSIStorageCapacities API operations [Conformance]'

[X] test approval issue : [X] test pr : 26 May 2023 [x] two weeks soak end date : 9 June 2023 [x] test promotion pr : [x] remove endpoint from [x] End of release 1.28 Release: Remove duplicate Conformance test the do not cover Patch, Replace, DeleteCollection. According to following APIsnoop query, there are still 3 CSIDriver endpoints that are not tested for Conformance. Using a number of existing e2e test practices a new has been created to address these 3 endpoints. The e2e logs for this test are listed below. This query shows the following csiDriver endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen For end of release 1.28 Release: Remove duplicate Conformance test the do not cover Patch, Replace, DeleteCollection.
Reopened this issue. <detailsIn response to : /reopen For end of release 1.28 Release: Remove duplicate Conformance test the do not cover Patch, Replace, DeleteCollection. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

What happened: When kube-proxy is configured with a cluster-cidr range, it drops packets with an INVALID conntrack state for all sources on the KUBE-FORWARD chain. This is meant to prevent spurious retransmits in long-lived TCP connections to a service IP, but it prevents clusters from having asymmetrical routing on private subnets with services outside of a cluster. What you expected to happen: Packets on the KUBE-FORWARD chain with an INVALID conntrack state should be dropped for only sources inside of the cluster-cidr if set. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: Change to drop INVALID conntrack packets was in Backing out that commit and compiling is my current workaround. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I am reading but I don't fully grok what caused the problem in the first place, or why we think "only drop if it was in-cluster" is a correct fix? Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. Is this a conntrack bug? Should it be applying the DNAT?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I can try to give a bit more context to our setup, but the knowledge I have around this all dates back to Feb, unfortuately, so I may have forgotten details. We were replacing our mesos cluster with a kubernetes cluster in Feb and using a custom ingress solution (HAProxy w/an outside process configuring it for pod ips / mesos ip-ports) for both. The ingress nodes lived in the private subnet, but outside of the kubernetes cluster cidr range. When we upgraded from kubernetes 1.14 to kubernetes 1.15 (the first version with PR ) our ingress could no longer talk to any pod ips.


Question: return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

I discovered that the KUBE-FORWARD chain had a rule to drop invalid packets, which was in Compiling kubernetes 1.15 without this commit fixed our problems, and we have been carrying that patch forward ever since. I would like to get rid of our forked kubernetes, hence this ticket. When I was writing this ticket up, I also found also mentions the same issue with the iptables dropping INVALID conntrack state packets near the end, regarding asymmetrical routing. Edit, OK I get the original problem, but still don't understand why packets from outside the pod range would not exhibit , thereby requiring yet another revisit of this problem. I think you are correct, I can imagine packets from outside of a cluster could exhibit the same behavior as packets inside the cluster and cause spurious resets. However, I am wondering if kube-proxy should be managing packets from outside of the cluster on a node... That seems outside of the scope of kube-proxy.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
I took a stab at implementing what I think is the correct behavior for kube-proxy in .
/assign
I think that another option is to set
I don't understand how that will help me here. The packets with a ctstate INVALID are not generated due to conntrack saturation (), but due to asymmetric routing in our network. See for an example of a similar setup.
are the packets on your asymmetric routing setup marked as invalid if you set ? it should only mark the ones that are out of window per the description
/triage accepted
I'd love to pin this down, but I don't have the means to set up a repro. Is there a way I can set up a way to experiment?
We were bit by this too after revisiting an old service that had been unused since the heady days of pre-, and would also like to see the proposed patch merged (or a configuration option to prevent insertion of the INVALID state drop rule - either suits us). I unfortunately don't have any useful input as to setting up for reproduction. In our case we have a pretty odd anycast set up with a small pool of (public) addresses managed by calico.


Question: return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

However I do seem to have a dirty workaround for small clusters. By setting notrack (ctstate UNTRACKED?) in the raw table for packets in our specific ranges that would otherwise match ctstate INVALID, the packets aren't dropped. Which may be of help to someone else who finds this issue.
We are seeing something very similar on large file downloads and our issue exactly fits this one's subject. These RESET (corrected after are happening intermittently and I am seeing similar behavior mentioned in the blog mentioned below. I am unsure if someone has found a workaround for this issue or not, so yeah my +1 to this issue being looked up by the community.
We're using as a solution for the connection reset we were seeing and might fix this issue.
Let me try to avoid that this thread derails and see if I can get current status: -proxy install rules to drop INVALID cstate connections, this solves the ~REJECT~ RESET problems are some scenarios, with asymmetric routing per example, that this rules may drop unrelated traffic. The request is to remove the rule that drops those INVALID states or to try to restrict it, and that can cause that people start to have the ~REJECT~ RESET problems and drops again. If we have a reproducer we can be more precise in the final solution, but removing one rule without any strong evidence in kube-proxy is very risky As a workaround for the asymetric routing problem, I suggested to set the nfconntracktcpbeliberal on the kernel: I think that will solve the problems, it will be good if someone can test it and provide feedback
No sadly, its my bad, it is not REJECT, it is RESET that's happening and conntrack has nothing to do with it, mostly (I hope). Kubeproxy is doing what is intended but specifically when there are external agents in play, the packet doesn't sends an ACK back to the response of SYN-ACK. Our case, we hit A to initiate a download, A calls B to get the action completed to generate download data for the user finally via their browser. A and B both are on k8s, we saw A starts and a complete cycle goes for sometime until there multiple parallel downloads requested where few gets completed and few gets a RESET.


Question: return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

On primary debugging, it appears that RESET was getting invoked by the pod but pod is just sending a RST-ACK not RST itself. The missing packet is never found in the default route, unless we see that there is a chance where ACK is missed and we found some lines in tcpdump with missing sequence for ACKs. Thats why i doubt my issue is related here. One more weird coincidence is few downloads have spurious transmission.
Thankyou, I will take a look, but mostly I would like to wait before using it, debug more to understand the actual cause and then act on it.
sorry , I wanted to say RESET, thanks for catching it. If you think is different and you can provide information and how to reproduce it, I suggest you to open a new issue
I will sure do, when it is different. For now I would like to watch and submit my findings if it is related. PS: sysctl -w net.netfilter.nfconntracktcpbeliberal=1 is already implemented and we are testing if it does something helpful. so far it doesn't looks like it. thanks for your help.
/area kube-proxy
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
I have the feeling we are also affected by this with a standard calico installation: every node announces a /122 IPv6 network to the upstream router the upstream router uses ECMP to route the packet back packets that are sent from* the pod and returned via asymmetric path from outside seem to be dropped
and if you set ?
Same behaviour with set on all nodes, the outgoing pod traffic does not return. And I instantly see packets going back into the drop rule:
:thinking:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.


Question: return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, are you still working on this? Same problem happened in our clusters, just because of asymmetrical routing.
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Also the command suggested by is helping us, but I still think this should be a bug of kube-proxy.
The problem here is: what happens if the solutions are exclusive or we regress? The addition of this rule fixed a legit bug that was documented and there is also a nice blog explaining the details Removing this rule without a repro and understanding what is the root cause of this issue is not really the best way to proceed. I think that people affected should try to find the root cause and explain the problem or just provide a way to reproduce it, so we can fix it
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


Question: return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

/close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The same problem happened in our cluster, the same as because of asymmetrical routing. the command suggested by doesn't work for us, and I delete the iptables rule and add new rule(refer to ), it works I think that makes sense I'll put up a reproduction of the gist link later on /reopen
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The main problem is how to be completely sure we don't regress, this comment from Tim is spot on
I'm trying to check that non-local connections will still reset without the INVALID Rule
I think the patch in is wrong; it changes the rule from "if INVALID then drop" to "if INVALID and from a pod, then drop" so that it doesn't affect non-kubernetes traffic, but I think the rule we actually need is "if INVALID and to a service, then drop". In particular, I believe we need to make sure that the rule applies to traffic with a cluster-external source and an in-cluster destination. It's possible that changing it to "if INVALID and to a pod, then drop" would also work. I skimmed and some linked bugs elsewhere but I was unable to confirm whether that rule would cover the reporters' use cases.
I think that is only INVALID and to a pod
In my case, cluster-external source to service is both ok with the INVALID rule or without the INVALID rule. It seems the rule doesn't affect on access service.


Question: return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

I add this rule but it doesn't work
So, based on that, the bug happens when a reply packet fails to get un-DNAT'ed, causing the client to then send a RST to the server, which matches the source/destination IP that the server knows of. But this could happen if you had a service with external IPs too; in that case the reply packet would be from the external endpoint IP to the node IP (because the packet would have been masqueraded), and if conntrack didn't unmap it correctly then the node would send a RST to the external IP, breaking the connection in just the same way. In this case there is no good way kube-proxy can recognize the packet as being its own, other than by using conntrack, which presumably won't work if the packet is INVALID?
It seems that there is no better way to solve this for external sources accessing the cluster CIDR, is it right? If so, Can we have a flag to decide whether to create The invalid rule? It would be nice that It's up to the user to decide whether to create the rule.
There's related discussion in . I think we should create whatever iptables rules we can that will fix up our own traffic without interfering with other traffic, and also document the sysctl that admins can use to change conntrack behavior to work around the problem. So probably, we should change the "drop if INVALID" rule to "drop if INVALID and to a pod", and also address the problem in , but not try to solve the problem with external IPs I mentioned in because we don't have a good way of fixing that without potentially breaking other things. (And it's kind of way more of an edge case anyway.) Note that can currently output rules for "from a pod", but not for "to a pod", so that will need some tweaking. (FWIW will also eventually need the ability to output nftables rules rather than iptables rules so that's another thing to think about if someone is going to change its API.)
So in the absence of a way to fix the problem with external IPs, can we have a flag to decide whether to create this rule or not? If true (default), create the drop if INVALID and to a pod rule. Otherwise, We do not create the rule.


Question: return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

I believe that even without this rule we can solve this problem by way of sysctl, right? I think we should do the following things We need a flag to disable the rule We should change the "drop if INVALID" rule to "drop if INVALID and to a pod" We document sysctl as a way to solve the problem.
If the rule is "drop if INVALID and to a pod", we shouldn't need a flag to disable it; it won't affect non-pod traffic, and it won't interfere with users that want to set the sysctl instead. (Setting the sysctl will cause packets to not be marked as INVALID so the rule we add just won't get hit.)
OK, drop if INVALID and to a pod is help us identify cluster traffic, right? We only drop cluster traffic if it's invalid, but this will not solve the problem of external IP. So we need a flag to disable the rule is drop if INVALID?
The same problem happened in our cluster, the same as because of asymmetrical routing None of the above methods solve the problem in our ENV. Are there any other workarounds?
Can you execute the following command on each node? I think this is a workaround.
Thank you for your reply I also tried removing this rule. However, k8s will reconfigure the deleted rules to the system at some point in time.
I worded that badly: I meant "external IP" as in "the endpoints are cluster-external IPs rather than pod IPs", not as in . So that would apply only to people using kube-proxy to load-balance traffic from clients outside the cluster to servers outside the cluster, which is a pretty weird use case. And nobody ever reported the packet RST bug for that case anyway. So I was arguing that we can just ignore that case. So: change the existing "drop if invalid" rule to "drop if invalid and to a pod" document the sysctl We don't need an option to disable the drop rule, because it won't affect non-k8s traffic, and it will automatically become a no-op if someone decides to use the sysctl instead.
Thank you for your response. I'd love to take this one. But I have a question, How to define k8s traffic?


Question: return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

If a pod in a cluster, we call it an underlay pod, its IP is not allocated from cluster CIDR, this is a real underlay IP, when it accesses other underlay pods in the cluster or pods that allocate IP from cluster CIDR(We call it overlay pod), is this k8s traffic? In our cluster, multiple types of CNI pods are implemented through multus running in a cluster, because of the problem of asymmetrical routing and the rule that makes them inaccessible, it seems that the above method will not solve this problem.
For kube-proxy, this is done by the . If you are in a cluster where there is no simple way to discriminate pod from non-pod traffic, then you'd have to run kube-proxy with , and then kube-proxy will see that returns and so it won't try to distinguish local from non-local traffic, and will do some things (like internal/external traffic policy) slightly differently as a result. So in this case, if there was no functioning localDetector, it would also not create the "drop invalid" rule
Thanks for the details, this is probably the case we want.
/assign
I did not understand how you could get asymmetric routing in K8s, but reading up more carefully I saw: If I understand this correctly, this is a load-balancer outside K8s that load-balance directly to POD addresses. To get asymmetric routing some kind of Direct Server Return (DSR) must also be used. So, to get asymmetric routing: from an external source is directed to an external LB LB is configured with POD-IPs as targets and selects one packet is routed to a K8s node, e.g. with ECMP. But is may not be the node where the target POD is executing packet is handled by the POD, but the response packet may be directed to some other GW This will give asymmetric routing. But it's not the every-day K8s network setup for sure. Given the above it's fairly easy to reproduce the problem. You don't even have to setup a load-balancer. The biggest problem may be that you need two ways in to the cluster. Example: ! I setup direct routes that ensures asymmetric routing, which can be checked with on vm-201 and vm-202.


Question: return rct.setIntSysCtl("nf_conntrack_tcp_timeout_close_wait", seconds) } <ins> func (rct realConntracker) SetTCPBeLiberal(value int) error { return rct.setIntSysCtl("nf_conntrack_tcp_be_liberal", value) } </ins> func (rct realConntracker) SetUDPTimeout(seconds int) error { return rct.setIntSysCtl("nf_conntrack_udp_timeout", seconds) }

From I can now directly ping POD-IPs, but due to the problem in this issue, I can't make a TCP connection: The POD with address runs on . A similar setup as above can be used to verify a PR. Now, there is a simple work-around: use proxy-mode=ipvs The update with the rule never made it to the ipvs proxier (a rather common case :cry: ), but in this case it may save your day. The same as above with proxy-mode=ipvs; The server on 5001 responds with the POD name.
Yeah, Our production environment also solved this problem by switching to IPVS mode. In IPVS mode, there is no the DROP rule.
The sysctl solves the problem in , but not the problem in this issue. Those problems are kind of related, but rather distant.
The same affects outgoing traffic if packets are not masqueraded to the node-ip, as should be a common setup for ipv6, please see For Calico I think you can setup BGP for clusterCIDRs and route incoming packets to PODs to the correct K8s node, instead of using ECMP. This would solve the problem, and save the extra hop.
I met the same issue. KUBE-FORWARD drop my packages.
temptatively solution
The issue fixed by , I'll close it, you can feel free to open it if have any questions. /close
Closing this issue. <detailsIn response to : The issue fixed by , I'll close it, you can feel free to open it if have any questions. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if e.Decorator != nil { e.Decorator(out) } <ins> if dryrun.IsDryRun(options.DryRun) { if err := dryrun.ResetMetadata(obj, out); err != nil { return nil, err } } </ins> return out, nil }

Running kubectl create with dry-run returns unwanted uid metadata field, as well as a generated name when using generateName. This is inconsistent with the expectations as defined in which states I expected uid, and a generated name to be empty when using the dry-run create option. Expected metadata: Actual metadata: No response <detailsI've tested this from 1.19.0 through 1.23.0 </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
reopening, is reverted in to make sure requests with both name+generateName are handled correctly
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


Question: if e.Decorator != nil { e.Decorator(out) } <ins> if dryrun.IsDryRun(options.DryRun) { if err := dryrun.ResetMetadata(obj, out); err != nil { return nil, err } } </ins> return out, nil }

/close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: kube := matches[0] framework.Logf("Get running kubelet with systemctl: %v, %v", string(stdout), kube) stdout, err = exec.Command("sudo", "systemctl", "restart", kube).CombinedOutput() <del> if err == nil { return } framework.Failf("Failed to restart kubelet with systemctl: %v, %v", err, stdout) </del> <ins> framework.ExpectNoError(err, "Failed to restart kubelet with systemctl: %v, %v", err, stdout) Eventually(func() ([]string, error) { return filepath.Glob("/var/lib/kubelet/device-plugins/nvidiaGPU*.sock") }, 5*time.Minute, framework.Poll).ShouldNot(ConsistOf(beforeSocks)) </ins> } func getDeviceId(f *framework.Framework, podName string, contName string, restartCount int32) string { // Wait till pod has been restarted at least restartCount times. Eventually(func() bool { p, err := f.PodClient().Get(podName, metav1.GetOptions{}) <del> framework.ExpectNoError(err) </del> <ins> if err != nil || len(p.Status.ContainerStatuses) < 1 { return false } </ins> return p.Status.ContainerStatuses[0].RestartCount >= restartCount <del> }, time.Minute, time.Second).Should(BeTrue()) </del> <ins> }, 5*time.Minute, framework.Poll).Should(BeTrue()) </ins> logs, err := framework.GetPodLogs(f.ClientSet, f.Namespace.Name, podName, contName) if err != nil { framework.Failf("GetPodLogs for pod %q failed: %v", podName, err)


Question: onRelease     = make(chan struct{}) lockObj runtime.Object <ins> gets    int </ins> updates int <ins> wg      sync.WaitGroup </ins> ) <ins> resetVars := func() { onNewLeader = make(chan struct{}) onRenewCalled = make(chan struct{}) onRenewResume = make(chan struct{}) onRelease = make(chan struct{}) </ins> <del> resourceLockConfig := rl.ResourceLockConfig{ Identity:      "baz", EventRecorder: &record.FakeRecorder{}, } c := &fake.Clientset{} c.AddReactor("get", objectType, func(action fakeclient.Action) (handled bool, ret runtime.Object, err error) { if lockObj != nil { return true, lockObj, nil } return true, nil, errors.NewNotFound(action.(fakeclient.GetAction).GetResource().GroupResource(), action.(fakeclient.GetAction).GetName()) }) // create lock c.AddReactor("create", objectType, func(action fakeclient.Action) (handled bool, ret runtime.Object, err error) { lockObj = action.(fakeclient.CreateAction).GetObject() return true, lockObj, nil }) c.AddReactor("update", objectType, func(action fakeclient.Action) (handled bool, ret runtime.Object, err error) { updates++ // Second update (first renew) should return our canceled error // FakeClient doesn't do anything with the context so we're doing this ourselves if updates == 2 { close(onRenewCalled) <-onRenewResume return true, nil, context.Canceled } else if updates == 3 { close(onRelease) } lockObj = action.(fakeclient.UpdateAction).GetObject() return true, lockObj, nil }) c.AddReactor("*", "*", func(action fakeclient.Action) (bool, runtime.Object, error) { t.Errorf("unreachable action. testclient called too many times: %+v", action) return true, nil, fmt.Errorf("unreachable action") }) lock, err := rl.New(objectType, "foo", "bar", c.CoreV1(), c.CoordinationV1(), resourceLockConfig) if err != nil { t.Fatal("resourcelock.New() = ", err) </del> <ins> lockObj = nil gets = 0 updates = 0 </ins> } lec := LeaderElectionConfig{ <del> Lock:          lock, </del> LeaseDuration: 15 * time.Second, RenewDeadline: 2 * time.Second, RetryPeriod:   1 * time.Second,

The lock acquired by tryAcquireOrRenew is released when the leader ends leadership. However, due to the cancellation of the context, the lock may be set as an empty lock, so the Update cannot be run normally, resulting in a failure to release the lock. The lock can be released every time the leader ends. run this: and then you can got a log like these: and you when you watch the leases at the same time: you can see that was failed to release because of No response <details</details<details</details<details</details<details</details<details</detailsrelated packages: v1.3.0 v0.25.0 v0.25.0 v2.70.1
/sig api-machinery
/triage accepted thanks!
/unassign
/assign
/unassign
/unassign me


Question: return fi mkdir -p "${KUBE_JUNIT_REPORT_DIR}" <del> # This filename isn't parsed by anything, and we must avoid # exceeding 255 character filename limit. KUBE_TEST_API # barely fits there and in coverage mode test names are # appended to generated file names, easily exceeding # 255 chars in length. So let's just use a sha1 hash of it. local KUBE_TEST_API_HASH KUBE_TEST_API_HASH="$(echo -n "${KUBE_TEST_API////-}"| ${SHA1SUM} |awk '{print $1}')" echo "${KUBE_JUNIT_REPORT_DIR}/junit_${KUBE_TEST_API_HASH}_$(kube::util::sortable_date)" </del> <ins> echo "${KUBE_JUNIT_REPORT_DIR}/junit_$(kube::util::sortable_date)" </ins> } verifyAndSuggestPackagePath() {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: unit tests with coverage succeed How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looking at : It is expecting group version string.
fixed in , clean up from


Question: if [[ -z "${KUBE_USER:-}" || -z "${KUBE_PASSWORD:-}" ]]; then gen-kube-basicauth fi <ins> # Make sure they don't contain any funny characters. if ! [[ "${KUBE_USER}" =~ ^[-._@a-zA-Z0-9]+$ ]]; then echo "Bad KUBE_USER string." exit 1 fi if ! [[ "${KUBE_PASSWORD}" =~ ^[-._@#%/a-zA-Z0-9]+$ ]]; then echo "Bad KUBE_PASSWORD string." exit 1 fi </ins> } function load-or-gen-kube-bearertoken() {

under kubetest.Up Nov 22 2023, last 7 runs failed No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
APIServer failed for
Yes it is failing for me too!
/sig testing
the generator config source must include the env to handle the "external" provider deprecation: /sig cloud-provider /area provider/gcp
/remove-sig testing /sig release
all other jobs seem green in the 1.29 blocking/informing, post that change. i will try sending a PR for this, but you need to find a non-USA approver if you want it merged before Monday. /assign
never mind, the generator is giving me errors: /unassign
/cc


Question: listerWatcher := newCacherListerWatcher(config.Storage, config.ResourcePrefix, config.NewListFunc) cacher := &Cacher{ <del> usable:        sync.RWMutex{}, storage:       config.Storage, watchCache:    watchCache, reflector:     cache.NewReflector(listerWatcher, config.Type, watchCache, 0), watcherIdx:    0, watchers:      make(map[int]*cacheWatcher), versioner:     config.Versioner, keyFunc:       config.KeyFunc, ListFromCache: config.ListFromCache, </del> <ins> usable:     sync.RWMutex{}, storage:    config.Storage, watchCache: watchCache, reflector:  cache.NewReflector(listerWatcher, config.Type, watchCache, 0), watcherIdx: 0, watchers:   make(map[int]*cacheWatcher), versioner:  config.Versioner, keyFunc:    config.KeyFunc, </ins> } cacher.usable.Lock() // See startCaching method for why explanation on it.

The label(s) cannot be applied. These labels are supported: <detailsIn response to : This one is trying to use with a generator to create an rc. This was deprecated and now is removed, so the test is failing now. /label sig/undefined Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/label failing-test
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label failing-test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test
At first I thought this was because generators were removed from . That still might be the cause of this, but I'm not 100% sure off-hand.
/sig cli
/assign
Funny thing about this is if you look in the build logs it is not immediately obvious that it is caused by generators being removed from because actually has sort of an unexpected behavior, where it tells you it is deprecated and gives you a pod instead... For example: Output:
/reopen should fix this is one of the affected jobs
Reopened this issue. <detailsIn response to : /reopen should fix this Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Since it's been a few weeks, I wanted to provide an update on this. I'm still waiting to confirm this is resolved by It looks like it is still running the old tests not marked as . Since 1.17.9 was released 14 days ago, and k8s-ci-robot only merged the PR 13 days ago, I'm guessing we need to wait until there is a 1.17.10 to see it skip the deprecated tests.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity.


Question: listerWatcher := newCacherListerWatcher(config.Storage, config.ResourcePrefix, config.NewListFunc) cacher := &Cacher{ <del> usable:        sync.RWMutex{}, storage:       config.Storage, watchCache:    watchCache, reflector:     cache.NewReflector(listerWatcher, config.Type, watchCache, 0), watcherIdx:    0, watchers:      make(map[int]*cacheWatcher), versioner:     config.Versioner, keyFunc:       config.KeyFunc, ListFromCache: config.ListFromCache, </del> <ins> usable:     sync.RWMutex{}, storage:    config.Storage, watchCache: watchCache, reflector:  cache.NewReflector(listerWatcher, config.Type, watchCache, 0), watcherIdx: 0, watchers:   make(map[int]*cacheWatcher), versioner:  config.Versioner, keyFunc:    config.KeyFunc, </ins> } cacher.usable.Lock() // See startCaching method for why explanation on it.

Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Float32 float32 // a base64 encoded characters ByteArray []byte <ins> // a member with an extension // +k8s:openapi-gen=x-kubernetes-member-tag:member_test WithExtension string </ins> } `) if err != nil {

Background: kubectl and the Kubernetes reference docs both depend on the spec to operate on Kubernetes resources. The lacks important information required by kubectl and the docs. Usecase: kubectl can identify the patchStrategy and mergeKey for resource fields kubectl can identify the columns to print for on resources docs can parse and display the default value for fields docs can identify top-level objects Solution: Add items to the comment tag: Example: The long term plan is to remove the and struct tags once all code has been updated to use the swagger spec instead. Until then, comments should be for the struct tags and . Some verification process will need to be written to make sure the struct tags and comments remain in sync.
I think this wouldn't fall under sig-apps. cc
I would not do the columns stuff - we're moving get to the server side. Top level objects is already available in some form by the client - specifically the list that we use (should use) to get the swagger doc tells us what the top level type is (or is that for something like explain). Merge key, patch strategy, default values all belong as "real" attributes, although we had not necessarily intended those to be arguments to k8s:openapi-gen, but actual tags in their own right (like "+optional")
Thanks Is there a design proposal / eta for this? This is for generating the docs from swagger. Right now the list of top level resources for the docs to display is hardcoded in a . I would like to move the majority of the metadata out of the config file so that: same tool can generate reference docs for federated apiservers without writing a custom config. don't need to manually maintain and update the config for each release - e.g. adding new types I have no strong opinion about this as long as they appear in the swagger so it is possible right tools the can use this data. Can you comment on why openapi-gen vs specific tags?
Describe in the server makes sense to me. IIUC, just prints out a set of columns. Was the idea that the server would just print out the columns or return a list of column / values or something more complicated? No need to answer if it is in the issue / proposal already.


Question: Float32 float32 // a base64 encoded characters ByteArray []byte <ins> // a member with an extension // +k8s:openapi-gen=x-kubernetes-member-tag:member_test WithExtension string </ins> } `) if err != nil {

is the bigger issue that touches on a rough overview, I can't find my rambling brain dump of the problem, but get would be rough tabular json, you could also get that as csv, and describe could be a richer but not quite full structured key/value structure.
regarding tags, they do not need to be in the same k8s:openapi-gen tag, it can be repeated: they can also be separate tags but still general like this: The idea is to have a general way to add arbitrary extension to a type so we don't need to change code generator for each specific tag. I have no strong feeling about the format, but I would like it to be general vs tags like +optional. The difference is +optional is a defined tag in OpenAPI () but extensions does not have any meaning for code generator. They better to be general so generate can pass them without change.
No, I mean we should define conventions in our API code that match the intent (optional) rather than have tons and tons of repeated code that is setting x-kubernete-describe-columns (or similar)
I spent some time thinking about your comments last night. I don't think this is mutually exclusive with , and could be complementary. IMHO this is general metadata associated with the type, and including it in the doesn't preclude supporting an api endpoint that pre-parses and formats the data. However, including it in the swagger does allow clients using the swagger to take advantage of it. Following is an example of where the swagger would be useful even if get was in the sever. A generated Java client could use the display columns for the "toString" method of the resource. Another use case that had mentioned to me was to allow some kubectl commands to operate on directly on local files. e.g. (strawman) could print the columns. I don't think we want to do this now, but it is an example of why keeping this object metadata private to the server limits what we can do. My hope is that this swagger tags solution will also oprove portable to easily work with ThirdPartyResources as well by writing the display-columns as an annotation on the TPR and keeping the same format. is a can of worms, and I would agree that the solution is capable of providing a richer experience for aggregating information across multiple resource types - e.g.