Question: sanitizedSpecVolID := kstrings.EscapeQualifiedNameForDisk(specVolID) return path.Join(host.GetVolumeDevicePluginDir(csiPluginName), sanitizedSpecVolID, "data") } <ins> // hasReadWriteOnce returns true if modes contains v1.ReadWriteOnce func hasReadWriteOnce(modes []api.PersistentVolumeAccessMode) bool { if modes == nil { return false } for _, mode := range modes { if mode == api.ReadWriteOnce { return true } } return false } </ins>

It would be a 1.12.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Approved for 1.12
Based on previous post, this is a summary: If then is not applied (because it could be an indication of a nonblock fs, or error) Done is provided if fstype is invalid Done if fstype is valid and then apply fsgroup logic else ignore fsGroup Done
- any updates on ?
Waiting for a review and approval.
you have a outstanding comment on the PR. Can you please answer that?
How does this approach relate to inline CSI volumes? There's no PersistentVolume involved, and hence no way to specify an access mode. , but my driver ignores it (always using tmpfs). I considered having my CSI driver read the mounting pod's spec to see if it has a security context with "fsGroup" set, since that detail is not supplied to the driver, as . My goal is to have the volume mounted with the files on it readable by the container's user, but there's no way for my driver to know who that user will be. Am I supposed to just use 0444 for the ownership bits, to allow everyone to read the files?


Question: new_storage_version=${test_data[5]} kube::log::status "Verifying ${resource}/${namespace}/${name} has updated storage version ${new_storage_version} in etcd" <del> curl -s http://${ETCD_HOST}:${ETCD_PORT}/v2/keys/registry/${resource}/${namespace}/${name} | grep ${new_storage_version} </del> <ins> curl -s http://${ETCD_HOST}:${ETCD_PORT}/v2/keys/${ETCD_PREFIX}/${resource}/${namespace}/${name} | grep ${new_storage_version} </ins> done killApiServer

removed this test. We need to continue testing this. This wasn't about testing upgrades, the prefix has nothing to do with upgrades. It's just there so people can share etcd. We maybe don't need to double everything but we do need to make sure that keeps working.
To be honest - I think that what we was testing before didn't give us completely any information. We didn't testing what information we are storing in etcd and where. That said, if we had a hardcoded prefix somewhere in our code, we wouldn't discover it at all, and those tests would still be passing. We should have a test that is directly checking etcd contents. I think it should be possible to write a simple unit test for it. Will try to put something together next week.


Question: dockertypes "github.com/docker/engine-api/types" ) <ins> // These two functions are OS specific (for now at least) func updateHostConfig(config *dockercontainer.HostConfig) { } func DefaultMemorySwap() int64 { return -1 } </ins> func getContainerIP(container *dockertypes.ContainerJSON) string { return "" }

What happened: Latest kubelet + Windows Docker version 1.14.0-dev refused to start any containers, error output: "invalid option: Windows does not support MemorySwap" How to reproduce it (as minimally and precisely as possible): It's always reproducible with Windows Docker 1.14.0-dev + latest kubelet code base. Anything else do we need to know: It didn't happen for Windows Docker 1.12.2-cs2-ws-beta. The reason behind the issue was because new code was for Windows Docker to check Resources parameters, it doesn't accept a non-zero value for MemorySwap: But kubelet was default to set an unlimited (-1) value. The issue was gone in my setup after applied a private fix to disable MemorySwap by set it to zero for Windows.


Question: } base := host hostURL, err := url.Parse(base) <del> if err != nil { return nil, "", err } if hostURL.Scheme == "" || hostURL.Host == "" { </del> <ins> if err != nil || hostURL.Scheme == "" || hostURL.Host == "" { </ins> scheme := "http://" if defaultTLS { scheme = "https://"

Hi, I have been trying to run api-server as an standalone service along with etcd. I am passing etcd-servers and cluster-ip-range as arguments, here is the error I am getting: etcd servers are up and running on that address. Any leads as to how to resolve this? TIA.
This is only happening with go1.8 version, not really sure what is causing issue while parsing the URI.
Caused by this change i believe - CC as 1.8beta1 is being worked on here -
Thanks
I'm getting this exact issue using Kubernetes 1.5.7 and Go 1.8.3. Reverting to Go 1.7 seems to resolve it. This happened after updating a Go 1.7-compiled 1.4.8 Kubernetes cluster to 1.5.7, not sure if that's a corner case. For reference, my stack trace:


Question: package main import ( <ins> goflag "flag" </ins> "fmt" "math/rand" "os" "time" <ins> "github.com/spf13/pflag" </ins> utilflag "k8s.io/apiserver/pkg/util/flag" "k8s.io/apiserver/pkg/util/logs" "k8s.io/kubernetes/cmd/kube-proxy/app"

There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
probably caused by
Could be a clash between these two?
cc
I still see the issue with local-cluster on the latest master: F0225 17:06:49. ] error: v1alpha1.KubeProxyConfiguration: FeatureGates: ReadMapCB: expect { or n, but found ", error found in byte of ......
looks like that's another issue, can you please file a new bug? with more logs? this one was about none of the hyperkube based processes starting up.
will do as it is also broke container kube-proxy as well. Same type of error.


Question: "context" "errors" "fmt" <ins> "net" "sync" </ins> "time" <del> v1 "k8s.io/api/core/v1" resourceapi "k8s.io/api/resource/v1alpha3" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" </del> <ins> "google.golang.org/grpc" "google.golang.org/grpc/connectivity" "google.golang.org/grpc/credentials/insecure" </ins> utilversion "k8s.io/apimachinery/pkg/util/version" <del> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/kubernetes" </del> "k8s.io/klog/v2" <del> "k8s.io/kubernetes/pkg/kubelet/pluginmanager/cache" </del> <ins> drapb "k8s.io/kubelet/pkg/apis/dra/v1alpha4" </ins> ) <del> // defaultClientCallTimeout is the default amount of time that a DRA driver has // to respond to any of the gRPC calls. kubelet uses this value by passing nil // to RegisterPlugin. Some tests use a different, usually shorter timeout to // speed up testing. // // This is half of the kubelet retry period (according to // https://github.com/kubernetes/kubernetes/commit/0449cef8fd5217d394c5cd331d852bd50983e6b3). const defaultClientCallTimeout = 45 * time.Second // RegistrationHandler is the handler which is fed to the pluginwatcher API. type RegistrationHandler struct { // backgroundCtx is used for all future activities of the handler. // This is necessary because it implements APIs which don't // provide a context. backgroundCtx context.Context kubeClient    kubernetes.Interface getNode       func() (*v1.Node, error) } </del> <ins> // NewDRAPluginClient returns a wrapper around those gRPC methods of a DRA // driver kubelet plugin which need to be called by kubelet. The wrapper // handles gRPC connection management and logging. Connections are reused // across different NewDRAPluginClient calls. func NewDRAPluginClient(pluginName string) (*Plugin, error) { if pluginName == "" { return nil, fmt.Errorf("plugin name is empty") } </ins> <del> var _ cache.PluginHandler = &RegistrationHandler{} // NewPluginHandler returns new registration handler. // // Must only be called once per process because it manages global state. // If a kubeClient is provided, then it synchronizes ResourceSlices // with the resource information provided by plugins. func NewRegistrationHandler(kubeClient kubernetes.Interface, getNode func() (*v1.Node, error)) *RegistrationHandler { handler := &RegistrationHandler{ // The context and thus logger should come from the caller. backgroundCtx: klog.NewContext(context.TODO(), klog.LoggerWithName(klog.TODO(), "DRA registration handler")), kubeClient:    kubeClient, getNode:       getNode, </del> <ins> existingPlugin := draPlugins.get(pluginName) if existingPlugin == nil { return nil, fmt.Errorf("plugin name %s not found in the list of registered DRA plugins", pluginName) </ins> } <del> // When kubelet starts up, no DRA driver has registered yet. None of // the drivers are usable until they come back, which might not happen // at all. Therefore it is better to not advertise any local resources // because pods could get stuck on the node waiting for the driver // to start up. // // This has to run in the background. go handler.wipeResourceSlices("") </del> <ins> return existingPlugin, nil } type Plugin struct { backgroundCtx context.Context cancel        func(cause error) </ins> <del> return handler </del> <ins> mutex                   sync.Mutex conn                    *grpc.ClientConn endpoint                string highestSupportedVersion *utilversion.Version clientCallTimeout       time.Duration </ins> } <del> // wipeResourceSlices deletes ResourceSlices of the node, optionally just for a specific driver. func (h *RegistrationHandler) wipeResourceSlices(driver string) { if h.kubeClient == nil { return } ctx := h.backgroundCtx logger := klog.FromContext(ctx) </del> <ins> func (p *Plugin) getOrCreateGRPCConn() (*grpc.ClientConn, error) { p.mutex.Lock() defer p.mutex.Unlock() </ins> <del> backoff := wait.Backoff{ Duration: time.Second, Factor:   2, Jitter:   0.2, Cap:      5 * time.Minute, Steps:    100, </del> <ins> if p.conn != nil { return p.conn, nil </ins> } <del> // Error logging is done inside the loop. Context cancellation doesn't get logged. _ = wait.ExponentialBackoffWithContext(ctx, backoff, func(ctx context.Context) (bool, error) { node, err := h.getNode() if apierrors.IsNotFound(err) { return false, nil } if err != nil { logger.Error(err, "Unexpected error checking for node") return false, nil } fieldSelector := fields.Set{resourceapi.ResourceSliceSelectorNodeName: node.Name} if driver != "" { fieldSelector[resourceapi.ResourceSliceSelectorDriver] = driver } err = h.kubeClient.ResourceV1alpha3().ResourceSlices().DeleteCollection(ctx, metav1.DeleteOptions{}, metav1.ListOptions{FieldSelector: fieldSelector.String()}) switch { case err == nil: logger.V(3).Info("Deleted ResourceSlices", "fieldSelector", fieldSelector) return true, nil case apierrors.IsUnauthorized(err): // This can happen while kubelet is still figuring out // its credentials. logger.V(5).Info("Deleting ResourceSlice failed, retrying", "fieldSelector", fieldSelector, "err", err) return false, nil default: // Log and retry for other errors. logger.V(3).Info("Deleting ResourceSlice failed, retrying", "fieldSelector", fieldSelector, "err", err) return false, nil } }) } // RegisterPlugin is called when a plugin can be registered. func (h *RegistrationHandler) RegisterPlugin(pluginName string, endpoint string, versions []string, pluginClientTimeout *time.Duration) error { // Prepare a context with its own logger for the plugin. // // The lifecycle of the plugin's background activities is tied to our // root context, so canceling that will also cancel the plugin. // // The logger injects the plugin name as additional value // into all log output related to the plugin. ctx := h.backgroundCtx </del> <ins> ctx := p.backgroundCtx </ins> logger := klog.FromContext(ctx) <del> logger = klog.LoggerWithValues(logger, "pluginName", pluginName) ctx = klog.NewContext(ctx, logger) logger.V(3).Info("Register new DRA plugin", "endpoint", endpoint) highestSupportedVersion, err := h.validateVersions(pluginName, versions) </del> <ins> network := "unix" logger.V(4).Info("Creating new gRPC connection", "protocol", network, "endpoint", p.endpoint) // grpc.Dial is deprecated. grpc.NewClient should be used instead. // For now this gets ignored because this function is meant to establish // the connection, with the one second timeout below. Perhaps that // approach should be reconsidered? //nolint:staticcheck conn, err := grpc.Dial( p.endpoint, grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithContextDialer(func(ctx context.Context, target string) (net.Conn, error) { return (&net.Dialer{}).DialContext(ctx, network, target) }), ) </ins> if err != nil { <del> return fmt.Errorf("version check of plugin %s failed: %w", pluginName, err) } var timeout time.Duration if pluginClientTimeout == nil { timeout = defaultClientCallTimeout } else { timeout = *pluginClientTimeout </del> <ins> return nil, err </ins> } <del> ctx, cancel := context.WithCancelCause(ctx) </del> <ins> ctx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() </ins> <del> pluginInstance := &Plugin{ backgroundCtx:           ctx, cancel:                  cancel, conn:                    nil, endpoint:                endpoint, highestSupportedVersion: highestSupportedVersion, clientCallTimeout:       timeout, </del> <ins> if ok := conn.WaitForStateChange(ctx, connectivity.Connecting); !ok { return nil, errors.New("timed out waiting for gRPC connection to be ready") </ins> } <del> // Storing endpoint of newly registered DRA Plugin into the map, where plugin name will be the key // all other DRA components will be able to get the actual socket of DRA plugins by its name. if draPlugins.add(pluginName, pluginInstance) { logger.V(1).Info("Already registered, previous plugin was replaced") } return nil </del> <ins> p.conn = conn return p.conn, nil </ins> } <del> func (h *RegistrationHandler) validateVersions( pluginName string, versions []string, ) (*utilversion.Version, error) { if len(versions) == 0 { return nil, errors.New("empty list for supported versions") } </del> <ins> func (p *Plugin) NodePrepareResources( ctx context.Context, req *drapb.NodePrepareResourcesRequest, opts ...grpc.CallOption, ) (*drapb.NodePrepareResourcesResponse, error) { logger := klog.FromContext(ctx) logger.V(4).Info("Calling NodePrepareResources rpc", "request", req) </ins> <del> // Validate version newPluginHighestVersion, err := utilversion.HighestSupportedVersion(versions) </del> <ins> conn, err := p.getOrCreateGRPCConn() </ins> if err != nil { <del> // HighestSupportedVersion includes the list of versions in its error // if relevant, no need to repeat it here. return nil, fmt.Errorf("none of the versions are supported: %w", err) } existingPlugin := draPlugins.get(pluginName) if existingPlugin == nil { return newPluginHighestVersion, nil </del> <ins> return nil, err </ins> } <del> if existingPlugin.highestSupportedVersion.LessThan(newPluginHighestVersion) { return newPluginHighestVersion, nil } return nil, fmt.Errorf("another plugin instance is already registered with a higher supported version: %q < %q", newPluginHighestVersion, existingPlugin.highestSupportedVersion) } // DeRegisterPlugin is called when a plugin has removed its socket, // signaling it is no longer available. func (h *RegistrationHandler) DeRegisterPlugin(pluginName string) { if p := draPlugins.delete(pluginName); p != nil { logger := klog.FromContext(p.backgroundCtx) logger.V(3).Info("Deregister DRA plugin", "endpoint", p.endpoint) // Clean up the ResourceSlices for the deleted Plugin since it // may have died without doing so itself and might never come // back. go h.wipeResourceSlices(pluginName) return } </del> <ins> ctx, cancel := context.WithTimeout(ctx, p.clientCallTimeout) defer cancel() </ins> <del> logger := klog.FromContext(h.backgroundCtx) logger.V(3).Info("Deregister DRA plugin not necessary, was already removed") </del> <ins> nodeClient := drapb.NewNodeClient(conn) response, err := nodeClient.NodePrepareResources(ctx, req) logger.V(4).Info("Done calling NodePrepareResources rpc", "response", response, "err", err) return response, err </ins> } <del> // ValidatePlugin is called by kubelet's plugin watcher upon detection // of a new registration socket opened by DRA plugin. func (h *RegistrationHandler) ValidatePlugin(pluginName string, endpoint string, versions []string) error { _, err := h.validateVersions(pluginName, versions) </del> <ins> func (p *Plugin) NodeUnprepareResources( ctx context.Context, req *drapb.NodeUnprepareResourcesRequest, opts ...grpc.CallOption, ) (*drapb.NodeUnprepareResourcesResponse, error) { logger := klog.FromContext(ctx) logger.V(4).Info("Calling NodeUnprepareResource rpc", "request", req) conn, err := p.getOrCreateGRPCConn() </ins> if err != nil { <del> return fmt.Errorf("invalid versions of plugin %s: %w", pluginName, err) </del> <ins> return nil, err </ins> } <del> return err </del> <ins> ctx, cancel := context.WithTimeout(ctx, p.clientCallTimeout) defer cancel() nodeClient := drapb.NewNodeClient(conn) response, err := nodeClient.NodeUnprepareResources(ctx, req) logger.V(4).Info("Done calling NodeUnprepareResources rpc", "response", response, "err", err) return response, err </ins> }

The PR above moved everything related to into because that was where most of the code already was, but didn't renamed the files to avoid a larger diff and merge conflicts. Once is merged, we can do the renaming. Code clarity. /sig node
/triage accepted /priority important-soon
/assign
a PR can be created for this now.
will do


Question: }) It("should write files of various sizes, verify size, validate content", func() { <del> fileSizes := []int64{1 * framework.MiB, 100 * framework.MiB, 1 * framework.GiB} </del> <ins> fileSizes := []int64{fileSizeSmall, fileSizeMedium, fileSizeLarge} </ins> err := testVolumeIO(f, cs, config, volSource, &podSec, testFile, fileSizes) Expect(err).NotTo(HaveOccurred()) })

The "Volume plugin streaming [Slow] NFS should write files of various sizes, verify size, validate content" e2e test is failing on our 2k-node gce clusters (https://k8s-) with the following error: cc
Could you take a look into it or reassign as apt? This is part of the correctness suite and large cluster tests are release-blocking.
/assign
is this run using the kubemark environment? I'll also take a look. It could be that the commands this test needs to execute may not work with kubemark. /assign
cc
No, this is not kubemark. This is on a real 2000-node cluster.
Also all other volume-related e2e tests are passing, except this one.
Looking at the log I see that the nfs test fails trying to verify the test file's content. The test exec's into the nfs client pod and executes: The return code from above is 127. The test then deletes the nfs client pod calling which works. The nfs-server pod is deleted in the AfterEach. For some reason the reported error is 137 but the returned err 127. The same command is exec'd for all of the tested storage plugins so not sure why it's failing just for nfs. Error 127 might be "command not found".
The other strange thing is that it's only failing in this suite and not the others.
When I look at the kubelet log, I see the exec command coming through: I also see these suspicious kubelet soon afterwards: So I think the grep command is causing the node to run out of memory and get evicted. I think we can fix this either by: Making the grep command more memory efficient. I'm guessing it's reading the whole file into memory or, Make this test Serial so that it doesn't run in parallel with other pods, since it consumes a lot of memory.
An unrelated issue, is that the node allocatable feature didn't actually evict the pod. Instead this pod got a system OOM kill. /cc
related to the memcg notifications, not node allocatable. Thanks!
any theories why we see this only for the nfs-io tests, and not the glusterfs-io test for instance?
the gluster test didn't run on this in this suite.


Question: }) It("should write files of various sizes, verify size, validate content", func() { <del> fileSizes := []int64{1 * framework.MiB, 100 * framework.MiB, 1 * framework.GiB} </del> <ins> fileSizes := []int64{fileSizeSmall, fileSizeMedium, fileSizeLarge} </ins> err := testVolumeIO(f, cs, config, volSource, &podSec, testFile, fileSizes) Expect(err).NotTo(HaveOccurred()) })

I'm suspecting that the reason why we didn't see a problem with this test on other suites is that the load could be higher here. I've asked to see if we can reduce the memory usage of the grep call.
pointed out that there are no newlines in the generated file, so when grep is called it could buffer the entire file in memory. Adding a newline at the end of the dd input file should do it.
/priority critical-urgent
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/scalability sig/storage: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details
It seems like even if the generated file is broken up by newlines, grep could still use on the order of hundreds of MB of memory for a 100MiB file (even with the -m1 and -F options). We could check the file line by line, but it would take a long time. A better solution would be to check the MD5 file hash. It's fast and uses little memory. The expected hashes will be hard-coded into the test, and if different file sizes are in the future, a hash needs to be computed and as well.
Thanks a lot for fixing this!
Ref


Question: import ( "fmt" <ins> "os" </ins> "strings" "syscall" "unsafe"

When a --resolv-conf (DNS resolver config) is passed to kubelet on a Windows node, the config content is used as part of sandbox initialization (through getHostDNSConfig()). Now with the recent changes (), it seems now to fail with an error like: E0315 10:22:16. 4764 ] "Error syncing pod, skipping" err="failed to "CreatePodSandbox" for "img-pull-(-b40a-459d-9033-)" with CreatePodSandboxError: "Failed to generate sandbox config for pod "img-pull-(-b40a-459d-9033-)": Unexpected resolver config value: "C:etckubernetescni". Expected "" or "Host"."" pod="img-puller-3842/img-pull-" podUID=-b40a-459d-9033- This is showing on all the runs of GCE testgrid (latest): Reading/using the passed resolv config on Windows Run a Windows pool with DNS resolv config being set as part of kubelet params. No response_ 27 alpha.3 1.27 beta.0 master <detailsGCE </details<details</details<details</details<details</details<details</details
/triage accepted /milestone v1.27 /sig node /sig network
/sig windows /cc
/priority criticial-urgent as this is likely a release blocking
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority criticial-urgent Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
Hmm, so you're setting the [1]. This PR introduced the capability to read the DNS config options from the Windows host registries while removing the capability to read from a file, introducing a regression. I've sent a PR addressing this issue: [1]
Is the file you are pointing to a UNIX-style or something else?
Was the file changing the behavior of the node?


Question: import ( "fmt" <ins> "os" </ins> "strings" "syscall" "unsafe"

Yeah, what used on GCE/GKE, seems to be a Linux-like conf, eg: nameserver 10.128.0.1 search c.<project_name.internal google.internal ` We haven't tried to remove from kubelet to see the behavior.
This Issue/PR has been marked as release blocker. Please have the fix merged by eod today otherwise the feature has to be reverted. For the Issue/PR owner: please have a revert/feature disable PR open today in case the fix failed to get in. Thank you! See the -- Kubernetes 1.27 release manager
I opened up a revert PR in case this PR does not merge by code freeze.
Thanks for the resolution! both 2019 and 2022 grids seems to be back to green now!
Thanks for confirming! I opened to ensure gets updated too


Question: t.Fatal(err) } j = convertNullTypeToNullable(j) <ins> j = stripIntOrStringType(j) </ins> openAPIJSON, err = json.Marshal(j) if err != nil { t.Fatal(err)

The server-side validation does not check the type of values with . Create a CRD like: Then create a CR: It is not rejected, but should because must not be a boolean.


Question: foo := newDeployment("foo", 1, nil, nil, nil, map[string]string{"foo": "bar"}) foo.Spec.Strategy.Type = extensions.RecreateDeploymentStrategyType <del> rs := newReplicaSet(foo, "foo-1", 1) pod := generatePodFromRS(rs) </del> <ins> rs1 := newReplicaSet(foo, "foo-1", 1) rs2 := newReplicaSet(foo, "foo-1", 1) pod1 := generatePodFromRS(rs1) pod2 := generatePodFromRS(rs2) </ins> f.dLister = append(f.dLister, foo) <del> f.rsLister = append(f.rsLister, rs) </del> <ins> // Let's pretend this is a different pod. The gist is that the pod lister needs to // return a non-empty list. f.podLister = append(f.podLister, pod1, pod2) c, _ := f.newController() enqueued := false c.enqueueDeployment = func(d *extensions.Deployment) { if d.Name == "foo" { enqueued = true } } c.deletePod(pod1) if enqueued { t.Errorf("expected deployment %q not to be queued after pod deletion", foo.Name) } } // TestPodDeletionPartialReplicaSetOwnershipEnqueueRecreateDeployment ensures that // the deletion of a pod will requeue a Recreate deployment iff there is no other // pod returned from the client in the case where a deployment has multiple replica // sets, some of which have empty owner references. func TestPodDeletionPartialReplicaSetOwnershipEnqueueRecreateDeployment(t *testing.T) { f := newFixture(t) foo := newDeployment("foo", 1, nil, nil, nil, map[string]string{"foo": "bar"}) foo.Spec.Strategy.Type = extensions.RecreateDeploymentStrategyType rs1 := newReplicaSet(foo, "foo-1", 1) rs2 := newReplicaSet(foo, "foo-2", 2) rs2.OwnerReferences = nil pod := generatePodFromRS(rs1) f.dLister = append(f.dLister, foo) f.rsLister = append(f.rsLister, rs1, rs2) f.objects = append(f.objects, foo, rs1, rs2) c, _ := f.newController() enqueued := false c.enqueueDeployment = func(d *extensions.Deployment) { if d.Name == "foo" { enqueued = true } } c.deletePod(pod) if !enqueued { t.Errorf("expected deployment %q to be queued after pod deletion", foo.Name) } } // TestPodDeletionPartialReplicaSetOwnershipDoesntEnqueueRecreateDeployment that the // deletion of a pod will not requeue a Recreate deployment iff there are other pods // returned from the client in the case where a deployment has multiple replica sets, // some of which have empty owner references. func TestPodDeletionPartialReplicaSetOwnershipDoesntEnqueueRecreateDeployment(t *testing.T) { f := newFixture(t) foo := newDeployment("foo", 1, nil, nil, nil, map[string]string{"foo": "bar"}) foo.Spec.Strategy.Type = extensions.RecreateDeploymentStrategyType rs1 := newReplicaSet(foo, "foo-1", 1) rs2 := newReplicaSet(foo, "foo-2", 2) rs2.OwnerReferences = nil pod := generatePodFromRS(rs1) f.dLister = append(f.dLister, foo) f.rsLister = append(f.rsLister, rs1, rs2) f.objects = append(f.objects, foo, rs1, rs2) </ins> // Let's pretend this is a different pod. The gist is that the pod lister needs to // return a non-empty list. f.podLister = append(f.podLister, pod)

IMO it feels like a bad practice to have mutating resource handlers. The deletePod handler is used for queueing a Recreate deployment once it has no pods running. It should merely look at the ReplicaSets that have owner refs as opposed to using dc.getReplicaSetsForDeployment. The orphaning/adoption dance is already happening inside the reconcilation loop.
I'd like to work on this
feel free to grab it. I can't assign the issue to you so I'll add myself; ping me on the PR when you have one.


Question: // GetZoneByNodeName gets cloudprovider.Zone by node name. func (ss *scaleSet) GetZoneByNodeName(name string) (cloudprovider.Zone, error) { <del> _, _, vm, err := ss.getVmssVM(name) </del> <ins> managedByAS, err := ss.isNodeManagedByAvailabilitySet(name) </ins> if err != nil { <del> if err == ErrorNotVmssInstance { glog.V(4).Infof("GetZoneByNodeName: node %q is managed by availability set", name) // Retry with standard type because nodes are not managed by vmss. return ss.availabilitySet.GetZoneByNodeName(name) } </del> <ins> glog.Errorf("Failed to check isNodeManagedByAvailabilitySet: %v", err) return cloudprovider.Zone{}, err } if managedByAS { // vm is managed by availability set. return ss.availabilitySet.GetZoneByNodeName(name) } _, _, vm, err := ss.getVmssVM(name) if err != nil { </ins> return cloudprovider.Zone{}, err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: VMAS nodes are reporting error in vmss cluster (vmType set to vmss). What you expected to happen: Node should register correctly in the kubernetes cluster. How to reproduce it (as minimally and precisely as possible): Create a VMAS nodes inside vmss cluster. Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure


Question: # CNI storage path for Windows nodes export WINDOWS_CNI_STORAGE_PATH="https://storage.googleapis.com/k8s-artifacts-cni/release" # CNI version for Windows nodes <del> export WINDOWS_CNI_VERSION="v0.8.7" </del> <ins> export WINDOWS_CNI_VERSION="v0.9.1" </ins> # Pod manifests directory for Windows nodes on Windows nodes. export WINDOWS_MANIFESTS_DIR="${WINDOWS_K8S_DIR}manifests" # Directory where cert/key files will be stores on Windows nodes.

The latest release of CNI plugins is , but we still have . I'll attempt to create a PR for this based on the .
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
- assigning to you :)


Question: // The name of this port. All ports in an EndpointSlice must have a unique // name. If the EndpointSlice is dervied from a Kubernetes service, this // corresponds to the Service.ports[].name. <del> // Name must either be an empty string or pass IANA_SVC_NAME validation: // * must be no more than 15 characters long // * may contain only [-a-z0-9] // * must contain at least one letter [a-z] // * it must not start or end with a hyphen, nor contain adjacent hyphens </del> <ins> // Name must either be an empty string or pass DNS_LABEL validation: // * must be no more than 63 characters long. // * must consist of lower case alphanumeric characters or '-'. // * must start and end with an alphanumeric character. </ins> // Default is empty string. Name *string `json:"name,omitempty" protobuf:"bytes,1,name=name"` // The IP protocol for this port.

What happened: No endpoints created in the EndpointSlice for my service due to the port on the service being too long. Logs show: EndpointSlice created has no endpoint: What you expected to happen: Service that works with Endpoints should work with EndpointSlice. If this is intentional, it should be clearly documented How to reproduce it (as minimally and precisely as possible): Create a service with a long port name Environment: Kubernetes version (use ): 1.16 Cloud provider or hardware configuration: kind
/sig networking
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /sig networking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
For a rough measure of impact, Istio had 5 ports that were over this limit (and many more that were right at 15 characters), so I suspect it may not be terribly rare.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This needs to be resolved before endpointslices graduate from alpha
/assign
/remove-triage unresolved


Question: } glog.V(5).Infof("namespace controller - deleteAllContentForGroupVersionResource - items remaining - namespace: %s, gvr: %v, items: %v", namespace, gvr, len(unstructuredList.Items)) if len(unstructuredList.Items) != 0 && estimate == int64(0) { <ins> // if any item has a finalizer, we treat that as a normal condition, and use a default estimation to allow for GC to complete. for _, item := range unstructuredList.Items { if len(item.GetFinalizers()) > 0 { glog.V(5).Infof("namespace controller - deleteAllContentForGroupVersionResource - items remaining with finalizers - namespace: %s, gvr: %v, finalizers: %v", namespace, gvr, item.GetFinalizers()) return finalizerEstimateSeconds, nil } } // nothing reported a finalizer, so something was unexpected as it should have been deleted. </ins> return estimate, fmt.Errorf("unexpected items still remain in namespace: %s for gvr: %v", namespace, gvr) } return estimate, nil

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "LRUExpireCache" Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT What happened: requires write lock since needs to manipulate its list to track recently used item. Currently it uses read lock so it may introduce race condition. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else do we need to know:
This was fixed in


Question: // the test so we can ensure that we clean up after // ourselves defer podClient.Delete(pod.Name, api.NewDeleteOptions(0)) <del> _, err = podClient.Create(pod) </del> <ins> pod, err = podClient.Create(pod) </ins> if err != nil { Failf("Failed to create pod: %v", err) } By("verifying the pod is in kubernetes") <ins> selector = labels.SelectorFromSet(labels.Set(map[string]string{"time": value})) options = api.ListOptions{ LabelSelector:   selector, ResourceVersion: pod.ResourceVersion, } </ins> pods, err = podClient.List(options) if err != nil { Failf("Failed to query for pods: %v", err)

cc
Yet another failure
There is already fix in flight
Ok. I see
This was closed 5 days ago but appears to not be fixed. Many PRs have referenced it as a flake since the fix went in.
- I referenced to it 2 after it was closed, but all of those failures were from runs before this was merged. So it's possible that it's already fixed.
When was the fix merged? I saw this a couple hours ago:
Oops, same test but the error is slightly different... Maybe this is a seperate flake
Yeah - it seems to be something different.
Also - I'm not claiming it's definitely fixed - I just said that we need to double check, because the two that I linked was from before merge.
I'll close this one for now. Please reopen the issue and cc us if the flake occurs again.


Question: return true }) if err != nil { <ins> if apiErr, ok := err.(awserr.Error); ok && apiErr.Code() == "DelegationSetNotReusable" || apiErr.Code() == "NoSuchDelegationSet" || apiErr.Code() == "InvalidInput" { return []dnsprovider.Zone{}, nil } </ins> return []dnsprovider.Zone{}, err } return zoneList, nil

/kind bug What happened: walteraa [6:55 AM] Hi, The controller manager of the federation using Google cloud DNS is crashing, the log showed is that: the hyperkube command is that: I have created this in the Google DNS Someone knows why the DNS zone query is getting the 404 error?? :disappointed: What you expected to happen: It should have created the missing domain. How to reproduce it (as minimally and precisely as possible): As per above. Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.4 Cloud provider or hardware configuration**: OpenStack (irrelevant) OS (e.g. from /etc/os-release): irrelevant Kernel (e.g. ): irrelevant Install tools: irrelevant Others: irrelevant
The bug is related to this: The above code assumes that the provider code returns no error but an empty list if the domain does not exist, but instead the provider returns a 404. Need to fix the provider and/or the above code.
/sig federation
I am unlikely to have time to work on this soon, so if anyone else wants to, go right ahead.
/assign
An additional information: I have created a which is based on the credential JSON file created through instruction. After that, I have configured the Federation controller manager to mount this secret as a volume in the container and set the environment variable to the JSON file inside this volume. Before doing that, I was getting an authentication error so the google DNS provider couldn't be instantiated, and after doing the instructions that I mentioned before, this authentication error was solved but now I'm getting 404 error when Google DNS provider tries access the specified managed zone. The file that I have generated follow this structure:


Question: func exportCustomMetricFromPod(f *framework.Framework, consumerName string, metricValue int) *common.ResourceConsumer { podAnnotations := map[string]string{ "prometheus.io/scrape": "true", <del> "prometheus.io/path":   "/Metrics", </del> <ins> "prometheus.io/path":   "/metrics", </ins> "prometheus.io/port":   "8080", } return common.NewMetricExporter(consumerName, f.Namespace.Name, podAnnotations, nil, metricValue, f.ClientSet, f.InternalClientset, f.ScalesGetter)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Used the endpoint /BumpMetric to create fake metrics for Prometheus. When I tried to curl /metrics I got the following response: Bad request. Not a POST request What you expected to happen: I expected to see something like this: # HELP info message. # TYPE gauge 0. How to reproduce it (as minimally and precisely as possible): Following the docs, after having it running inside kubernetes: Anything else we need to know?: Looks like introduced this by mistake. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig testing


Question: package route53 import ( <ins> "github.com/aws/aws-sdk-go/aws/awserr" </ins> "github.com/aws/aws-sdk-go/service/route53" "k8s.io/apimachinery/pkg/util/uuid"


Question: return info.HealthCheckNodePort } <ins> // GetNodePort is part of the ServicePort interface. func (info *BaseServiceInfo) GetNodePort() int { return info.NodePort } </ins> func (sct *ServiceChangeTracker) newBaseServiceInfo(port *v1.ServicePort, service *v1.Service) *BaseServiceInfo { onlyNodeLocalEndpoints := false if apiservice.RequestsOnlyLocalTraffic(service) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT What happened: When deleting/recreating pods backing a UDP service, traffic from external senders stops flowing. What you expected to happen: Traffic would continue to flow to new pods. How to reproduce it (as minimally and precisely as possible): Based on reading of similar issues and commit messages it seems like this would be most noticable on high-volume UDP applications, statsd and syslog are good examples. a deployment and service for a UDP app. In my test case, this would be the container. an external node, create a small test script utilizing a statsd library such as pystatsd. The most important part is that the sender uses the same source-port for every packet sent. Ex: the sender. Install tcpdump into the pod and watch for incoming packets, example from the test case: the pod tcpdump in the newly created pod. Observe that no new matching packets are flowing. existing conntrack rules on all nodes in the cluster. You should see traffic start flowing to the pod again. example command for clearing conntrack rules on the nodes: Anything else we need to know?: I have tested this only on GKE-managed clusters that utilize google forwarding-rules for external LBs, but I believe the issue may be present on any platform where iptables mode for kube-proxy is used. Tested and verified on 1.8.6-gke.0 and 1.7.11-gke.1. Also tested with both Google Container-optimized OS and the Ubuntu-based node images. I have only test this on GKE It appears this was a known issue and should have been cleared up by: From my reading of , it seems as if the func should exec() to clear up UDP entries on service endpoint change, but the only place I see that func called is from syncProxyRules(). That func looks complex and possibly like it is only run occasionally and not on service/endpoint changes. Environment: Kubernetes version (use ): (Also tested on 1.7.11-gke.1) Cloud provider or hardware configuration: GKE-managed clusters, google cloud. OS (e.g.


Question: return info.HealthCheckNodePort } <ins> // GetNodePort is part of the ServicePort interface. func (info *BaseServiceInfo) GetNodePort() int { return info.NodePort } </ins> func (sct *ServiceChangeTracker) newBaseServiceInfo(port *v1.ServicePort, service *v1.Service) *BaseServiceInfo { onlyNodeLocalEndpoints := false if apiservice.RequestsOnlyLocalTraffic(service) {

from /etc/os-release): tried both: and Kernel (e.g. ): and Install tools: Others:
/sig network
We also ran into this in kubernetes v1.7.6 using OpenShift 3.7:
~I believe this is the same as ~If so, it is fixed in 1.10: These are about HostPort, not NodePort, sorry for the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any other known workaround for this besides updating conntrack rules on the kube nodes? I tried deleting and recreating the service, but this didn't update the rules. When I restart the stream on the client side data flows correctly again (perhaps it's perceived as a new connection), but I have to restart the stream for each sender.
/remove-lifecycle stale
FWIW, this is not fixed in 1.10 - that fix was for HostPort, and this is still a bug for NodePort.
I am seeing this issue in 1.11. I am using LoadBalancer type.
has been working on a fix for this, FYI.
This issue is extremely impactful for telephony protocols that rely on static source ip and source port. Any update on a possible workaround? I was thinking a potential workaround might be to reduce the UDP timeout to something really short so the stale conntrack entries go away faster. Or maybe there is a way to flush conntrack on the event of pod deletion? Edit: Removed stupid idea of a workaround to disable conntrack for UDP. Can't disable conntrack for UDP because it's used for DNAT from node to container.
The workaround is to not use NodePorts. Other kinds of services are already correct.
not sure what you mean. I am seeing the issue with a LoadBalancer type so it does not affect NodePorts only. As a workaround I have created a small python script that watches the K8s API for pod deletions and clears the related conntrack entries when that happens. I can share it when I get to work on Monday if you want to try it out.


Question: return info.HealthCheckNodePort } <ins> // GetNodePort is part of the ServicePort interface. func (info *BaseServiceInfo) GetNodePort() int { return info.NodePort } </ins> func (sct *ServiceChangeTracker) newBaseServiceInfo(port *v1.ServicePort, service *v1.Service) *BaseServiceInfo { onlyNodeLocalEndpoints := false if apiservice.RequestsOnlyLocalTraffic(service) {

I've my script . Note that it's not tested much so don't use it in a production environment. Also it cleans up all conntrack entries for a deleted pod, not only the UDP related ones.
We are also seeing this issue with type loadbalancer. We are on 1.11. Our use-case is the following: multiple devices are sending to an aggreagator machine. This machine forwards the UDP packages to a loadBalancer in gcp. When debugging the service, I noticed the following: The traffic is assigned to one pod, and the packages are not loadbalanced (client and server are the same cfr use-case) When redeploying a new cluster, Conntrack is cleared on termination signal of the old pod. (client receives connection refused) red/black and highlander deployment strategies have the same effect. How would I expect this to be working then? New cluster is . Conntrack is cleared Data is routed to NEW pods old pods are terminated Best Regards, Tim
I am seeing this exact same issue on our cluster which has k8s v1.14.5 . We have endpoint nginx pods backing a UDP service with nodeport. Suddenly one day we see traffic not hitting into nginx pod but reaching the node hosting the nginx pod. We could simulate the problem by restarting the nginx pods. I am assuming with this issue marked as closed; and having made its way into v1.14; we should not be seeing this. Would also like to point out; the one thing, in addition we have in the service is ExternalTrafficPolicy set to Local. Can anyone please guide/help on why we are seeing this in v.1.14.5 of our cluster.
Seeing this with k8s v1.16.5


Question: kube::golang::setup_env <del> make -C "${KUBE_ROOT}" WHAT=cmd/genswaggertypedocs # Find binary genswaggertypedocs=$(kube::util::find-binary "genswaggertypedocs") if [[ ! -x "$genswaggertypedocs" ]]; then { echo "It looks as if you don't have a compiled genswaggertypedocs binary" echo echo "If you are running from a clone of the git repo, please run" echo "'make WHAT=cmd/genswaggertypedocs'." } >&2 exit 1 fi </del> _tmpdir="$(kube::realpath "$(mktemp -d -t swagger-docs.XXXXXX)")" function swagger_cleanup { rm -rf "${_tmpdir}"

When working on I a strange issue. It seems that while auto-generating conversions it works fine, but during things like or in the presubmits what is generated is different. Looking into test results: it seems that the order in which the files are processed is different then. /kind bug /sig api-machinery
FTR: The log from above has some debug-logging and panics and the conversion-generator logs are those with
/assign
modifies conversion-gen, but doesn't change the generated files. Did you intentionally exclude those changes? I wonder if it's similar to I checked out , ran I was able to reproduce the error in If that's the case, a workaround to fix the auto-generating is to first remove the affected files and re-generate them locally.
This is supposed to be no-op :) no auto-generated files should change That's not the fix. The problem is with the test jobs themselves: for generating swagger doc, they actually generate different things than they generate when you call "make generated-files" My PR is only a trigger for the problem, it's not a root cause. The auto-generated files are correct, but as part of running swagger docs, we're in-process generating a different stuff.
It looks that packages were missing. has a shorter / list compared to the other checks (e.g. ) when I ran . In the case of , the component-base config package was missing, thus the FIXME code path got hit. The list of packages was derived from . It seems that the dir was dirty during . I'm exploring if we can clean up the dir like does.
It seems that this one isn't yet fully fixed. I have updated the based on the comments on it and a similar issue on that (which I think has a very similar root cause). However, now "govet" and "staticcheck" are failing: [Ironically all other tests are passing, so I think this is again connected to some garbage somewhere] - can you please help with this?
As previously, I can reproduce it by:
I'm fairly sure at this point that the inputdirs param here is too restrictive:
It seems that this commit might be addressing this:
Heh - I think it helped me locally, but it doesn't help for - I probably did something wrong. But I'm fairly sure it's something along those lines that we need to do.


Question: kube::golang::setup_env <del> make -C "${KUBE_ROOT}" WHAT=cmd/genswaggertypedocs # Find binary genswaggertypedocs=$(kube::util::find-binary "genswaggertypedocs") if [[ ! -x "$genswaggertypedocs" ]]; then { echo "It looks as if you don't have a compiled genswaggertypedocs binary" echo echo "If you are running from a clone of the git repo, please run" echo "'make WHAT=cmd/genswaggertypedocs'." } >&2 exit 1 fi </del> _tmpdir="$(kube::realpath "$(mktemp -d -t swagger-docs.XXXXXX)")" function swagger_cleanup { rm -rf "${_tmpdir}"

- will you be able to help with it?
So yeah - I was completely wrong. Basically, running with with DBG it clearly is a problem with gen-conversion. And the problem is that we pass only limited input to it: I think we should always pass all inputs - but we should look a bit deeper into it.
OK - so this is the problematic part: We basically process only packages that are explicitly requested by the user and ignore ones that are imported. So in order to have the conversion func imported by the generator, the package where it lives has to be explicitly imported. So we should basically always pass all input packages to conversion generator.
I actually created a new one, as it seems to be a different thing. Closing in favor of
/triage accepted


Question: if err != nil { framework.Failf("Error getting node hostnames: %v", err) } <ins> ginkgo.By(fmt.Sprintf("Found %d SSH'able hosts", len(hosts))) </ins> testCases := []struct { cmd            string

ci-kubernetes-e2e-gce-scale-correctness Kubernetes e2e suite: [sig-node] SSH should SSH to all nodes and run commands kubetest.Test Jul 10 14:01:51 UTC+0200 Kubernetes e2e suite: [sig-node] SSH should SSH to all nodes and run commands: more . Test /sig node
/cc
/triage accepted
Uhhh, this may help here
/milestone v1.22
/sig scalability FYI
I got annoyed that I couldn't click through to see the other logs, so now you can, ref: (spyglass doesn't render build- because it's 500MB, and the limit for rendering is 100MB)
/assign SIG Scalability chairs/TLs, please find someone to address or call this a non-release-blocking failure
Assigning to current scalability oncall (): /assign
This is sig-node issue - should be addres by SIG-node folks. /sig node
is being discussed as a fix for this and I'm not sure it's correct
is it sig scalability's intent that 5k node clusters should support SSH?
The SSH test in question passes in the "default" job: So the fact that this failure is unique to a scalability job means to me that scalability needs to own at least some part of this
I did a little bit of digging and ci-kubernetes-e2e-gce-scale-correctness uses option in kubetest, which according to seems to be setting KUBESSHBASTION env. So I believe this is main difference between our scalability job and "default".
Debugging the test in and providing a fix there. The crictl host PR () is doing something different.


Question: } } <ins> func TestCacherNoLeakWithMultipleWatchers(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.WatchBookmark, true)() backingStorage := &dummyStorage{} cacher, _, err := newTestCacher(backingStorage, 1000) if err != nil { t.Fatalf("Couldn't create cacher: %v", err) } defer cacher.Stop() // Wait until cacher is initialized. cacher.ready.wait() pred := storage.Everything pred.AllowWatchBookmarks = true // run the collision test for 3 seconds to let ~2 buckets expire stopCh := make(chan struct{}) time.AfterFunc(3*time.Second, func() { close(stopCh) }) wg := &sync.WaitGroup{} wg.Add(1) go func() { defer wg.Done() for { select { case <-stopCh: return default: ctx, _ := context.WithTimeout(context.Background(), 3*time.Second) w, err := cacher.Watch(ctx, "pods/ns", "0", pred) if err != nil { t.Fatalf("Failed to create watch: %v", err) } w.Stop() } } }() wg.Add(1) go func() { defer wg.Done() for { select { case <-stopCh: return default: cacher.bookmarkWatchers.popExpiredWatchers() } } }() // wait for adding/removing watchers to end wg.Wait() // wait out the expiration period and pop expired watchers time.Sleep(2 * time.Second) cacher.bookmarkWatchers.popExpiredWatchers() cacher.bookmarkWatchers.lock.Lock() defer cacher.bookmarkWatchers.lock.Unlock() if len(cacher.bookmarkWatchers.watchersBuckets) != 0 { t.Errorf("unexpected bookmark watchers %v", len(cacher.bookmarkWatchers.watchersBuckets)) } } </ins> func testCacherSendBookmarkEvents(t *testing.T, watchCacheEnabled, allowWatchBookmarks, expectedBookmarks bool) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.WatchBookmark, watchCacheEnabled)() backingStorage := &dummyStorage{}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I try to backport bookmark feature to our internal branch. When I enable bookmark feature, apiserver leaks memory. I spend some time to dig the root cause and find the bookmarkWatchers of service cacher is keep increasing. I am not sure if it exists at upstream code. Please let me know if it is ok at latest code. What you expected to happen: Avoid memory leaking How to reproduce it (as minimally and precisely as possible): a cluster. not update service. will find kube-apiserver's memory keep increasing. Anything else we need to know?: Kubelet will watch service with bookmark, and add a watcher to bookmarkWatchers. and service init resource version is 0 Without updating service, we cannot update "lastProcessedResourceVersion", it will keep 0, bookmarkTimer branch will always be continued. As a result, it cannot trigger popExpiredWatchers, so the service watcher will keep increasing every watching. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scalability PLAT :)
How about making this change ?
- can you also please take a look?
Agree with u. That's same way I fix in our branch. But I wanna hear the suggestion of and .
! With the same bugfix wrote above, we can see the memory not increasing any more.
Given that watch-bookmarks doesn't create any additional go-routines (well, creates 1, but cannot leak them), it has to leak the data. The only data that can be leaked is exactly bookmarkWatchers. So I looked into the code and got into exactly the same conclusion - that we must be leaking watchers for resources that aren't changing. The fix as above makes sense (though we also need a unit test for it).
Thanks a lot.
has the fix by
Thanks for reporting the bug, I will look at this day later. In addition, also need to think carefully about whether need to send a bookmark when there are no events at all.
cc


Question: }) return args } <ins> func TestValidateKubeletFlags(t *testing.T) { tests := []struct { name   string error  bool labels map[string]string }{ { name:  "Invalid kubernetes.io label", error: true, labels: map[string]string{ "beta.kubernetes.io/metadata-proxy-ready": "true", }, }, { name:  "Valid label outside of kubernetes.io and k8s.io", error: false, labels: map[string]string{ "cloud.google.com/metadata-proxy-ready": "true", }, }, { name:   "Empty label list", error:  false, labels: map[string]string{}, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { err := ValidateKubeletFlags(&KubeletFlags{ NodeLabels: tt.labels, }) if tt.error && err == nil { t.Errorf("ValidateKubeletFlags should have failed with labels: %+v", tt.labels) } if !tt.error && err != nil { t.Errorf("ValidateKubeletFlags should not have failed with labels: %+v", tt.labels) } }) } } </ins>

xref In 1.16: change the metadata-proxy addon to use change the kube-proxy addon to use change the masq-agent addon to use stop setting , , and in kube-up
Hey I'm trying to start to contribute to kubernetes and this seem to be an easy task. Can I help with this one?


Question: } } <ins> // Check if the inspected image matches what we are looking for func matchImageTagOrSHA(inspected dockertypes.ImageInspect, image string) bool { // The image string follows the grammar specified here // https://github.com/docker/distribution/blob/master/reference/reference.go#L4 named, err := dockerref.ParseNamed(image) if err != nil { glog.V(4).Infof("couldn't parse image reference %q: %v", image, err) return false } _, isTagged := named.(dockerref.Tagged) digest, isDigested := named.(dockerref.Digested) if !isTagged && !isDigested { // No Tag or SHA specified, so just return what we have return true } if isTagged { // Check the RepoTags for an exact match for _, tag := range inspected.RepoTags { if tag == image { // We found a specific tag that we were looking for return true } } } if isDigested { algo := digest.Digest().Algorithm().String() sha := digest.Digest().Hex() // Look specifically for short and long sha(s) if strings.Contains(inspected.ID, algo+":"+sha) { // We found the short or long SHA requested return true } } glog.V(4).Infof("Inspected image (%q) does not match %s", inspected.ID, image) return false } </ins> // applyDefaultImageTag parses a docker image string, if it doesn't contain any tag or digest, // a default tag will be applied. func applyDefaultImageTag(image string) (string, error) {

We had rather strange case today when after rolling update one of the kubelet's started wrong container. For this specific replicationcontroller we are using What happened was that after rolling-update when i started to debug the issue i discovered that one of the kubelet's did not pull the image: When looking into what image it actually started i saw: When checking from the actual node i saw this It seems that instead of starting myimage with tag 2208 kubelet started image tagged as 1393 since the docker image id starts with 2208.


Question: resources []string namespace string <del> name      string </del> <ins> names     []string </ins> defaultNamespace bool requireNamespace bool

Currently, it could not delete multiple pods at once, need to delete them one by one, it's very inconvenient if I have tons of pods to delete. Unlike docker, docker can multiple stop/rm containers/images at once.
This is possible using label selectors as of , which is better if you have tons of pods.
I agree that using label selector is better to delete all of pods under the label. But to delete multile pods at once is still usable: I just want to delete 1/2 pods under the label, label selector will not work selector could not span over other selectors or pods Another reason is docker rm/stop x y z works, I will wonder why kubectl cannot. This is not urgent, but I hope it can be implemented.
I agree. Thanks for the suggestion.
Would this be a suitable task for someone new to k8s development? If so, I'd be willing to take a look at this, if someone could provide a couple of hints where to start.
Further to my last message, I have a functional work in progress I would like to share, if anyone is interested?
PRs are welcome. See
Yes, this is a reasonable place to start. Since its in the client, you won't get much orientation to the core server code, but it will be a good place to orient to the client side API. There us also prior art that you can model off of. The kubectl create ... command already supports multiple resources. A good command to adapter would be the 'label' command. You can find it in Many thanks in advance! Brendan On Feb 19, 2015 8:21 AM, "Eric Tune" wrote:


Question: if err := deployClient.Delete(deployment.ObjectMeta.Name, deleteOptions); err != nil { framework.Failf("failed to delete the deployment: %v", err) } <del> ginkgo.By("wait for 30 seconds to see if the garbage collector mistakenly deletes the rs") time.Sleep(30 * time.Second) </del> <ins> ginkgo.By("wait for deployment deletion to see if the garbage collector mistakenly deletes the rs") err = wait.PollImmediate(500*time.Millisecond, 1*time.Minute, func() (bool, error) { dList, err := deployClient.List(metav1.ListOptions{}) if err != nil { return false, fmt.Errorf("failed to list deployments: %v", err) } return len(dList.Items) == 0, nil }) if err != nil { framework.Failf("Failed to wait for the Deployment to be deleted: %v", err) } // Once the deployment object is gone, we'll know the GC has finished performing any relevant actions. </ins> objects := map[string]int{"Deployments": 0, "ReplicaSets": 1, "Pods": 2} ok, err := verifyRemainingObjects(f, objects) if err != nil {

/priority failing-test /priority important-soon /kind flake /sig-api-machinery this failure is currently one of the top flake and repros in both Presubmits and CI jobs. This test is also part of the Conformance suite, which makes the flake far more critical. Can one of you please help triage this issue.
/cc
/cc
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
flakes are continuing, hopefully logging gives us more info to work with
this test seems to flake quite a bit in the presubmits and its a conformance test on top of this. Can we please look into the new logs to see if we have more info to deflake this? Triage: Logs: https://k8s-
/cc
FWIW this test seems to pass consistently in our conformance dashboards - https://k8s-,%20master%20(dev). But seems to flake only in CI and Presubmits
Do those run at different parallelism levels? It could be load/timeout sensitive
I have also seen a general uptick in flakes in other e2e tests that could be sensitive to watch event latency, not just gc tests - for example, see the failures in https://k8s-
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hey, this test does not appear to be flaking anymore can we close this issue?
Closing based on non-response. /close
Closing this issue. <detailsIn response to : Closing based on non-response. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This was never actually fixed, looks like.


Question: } mockCadvisor.On("VersionInfo").Return(versionInfo, nil) expectedNode := &api.Node{ <del> ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}, </ins> Spec:       api.NodeSpec{}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{

Looks like some kubelet unit tests perform a lookupIp which ends up depending on the health of the local resolver. Specifically has this failing: Which leads to the test in: And we recently swapped out the node names from testnode to 127.0.0.1, so probably the culprit?
/ref
Interesting. I did swap out the node names in the unit test as part of to be 127.0.0.1, party because I thought that it wouldn't have any problems resolving during unit tests.
/cc


Question: } mockCadvisor.On("VersionInfo").Return(versionInfo, nil) expectedNode := &api.Node{ <del> ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}, </ins> Spec:       api.NodeSpec{}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{

cc
What is the environment where you are running the tests? They pass for me on my mac, a linux box, and shippable seems to be passing them as well.
I couldn 't reproduce the failure either.
I'm out right now; i'll try them when i get back. For the record, I run hack/test- Paul Morie wrote:
Ahem FTR This acronym is going to go quad-plat Paul Morie wrote:
I ran on both platforms and saw the tests pass.
I'm still seeing it. Won't have bandwidth to dig in personally until tomorrow.
seeing a similar failure after a git pull
I knew I wasn't crazy
environment: fedora 21, golang: 1.4, docker: 1.6
I think I see the problem. When I set the node name to in the test I was assuming it would parse as a valid ip address and not end up at the LookupIP call. But only the node name (and not the kubelet hostname) is being set (except in my new test where I ) so it's trying to do a lookup. Do you think you could test just changing the default hostname from "testnode" to "127.0.0.1" and see if that fixes it for you?
Trying now
I still get failures but no panic / threaddump.
gist coming
this looks like again?
Good that it isn't panicing any longer. I can dig into the test failures tomorrow (I think it should be relatively simple to resolve).
cool, thanks for looking
Passes in master now for me. Closing.
FYI, they will break again once goes in, and then we can merge my PR to see if that fixes it for you.
:+1:
is in -- are these tests breaking again?
I just rebased which I think will fix them.
They are breaking again.
is on it.


Question: . "github.com/onsi/gomega" ) <ins> func runLivenessTest(c *client.Client, podDescr *api.Pod) { defer GinkgoRecover() ns := "e2e-test-" + string(util.NewUUID()) By(fmt.Sprintf("Creating pod %s in namespace %s", podDescr.Name, ns)) _, err := c.Pods(ns).Create(podDescr) Expect(err).NotTo(HaveOccurred(), fmt.Sprintf("creating pod %s", podDescr.Name)) // At the end of the test, clean up by removing the pod. defer func() { By("deleting the pod") c.Pods(ns).Delete(podDescr.Name) }() // Wait until the pod is not pending. (Here we need to check for something other than // 'Pending' other than checking for 'Running', since when failures occur, we go to // 'Terminated' which can cause indefinite blocking.) By("waiting for the pod to be something other than pending") err = waitForPodNotPending(c, ns, podDescr.Name, 60*time.Second) Expect(err).NotTo(HaveOccurred(), fmt.Sprintf("starting pod %s in namespace %s", podDescr.Name, ns)) By(fmt.Sprintf("Started pod %s in namespace %s", podDescr.Name, ns)) // Check the pod's current state and verify that restartCount is present. By("checking the pod's current state and verifying that restartCount is present") pod, err := c.Pods(ns).Get(podDescr.Name) Expect(err).NotTo(HaveOccurred(), fmt.Sprintf("getting pod %s in namespace %s", podDescr.Name, ns)) initialRestartCount := pod.Status.Info["liveness"].RestartCount By(fmt.Sprintf("Initial restart count of pod %s is %d", podDescr.Name, initialRestartCount)) // Wait for at most 48 * 5 = 240s = 4 minutes until restartCount is incremented pass := false for i := 0; i < 48; i++ { // Wait until restartCount is incremented. time.Sleep(5 * time.Second) pod, err = c.Pods(ns).Get(podDescr.Name) Expect(err).NotTo(HaveOccurred(), fmt.Sprintf("getting pod %s", podDescr.Name)) restartCount := pod.Status.Info["liveness"].RestartCount By(fmt.Sprintf("Restart count of pod %s in namespace %s is now %d", podDescr.Name, ns, restartCount)) if restartCount > initialRestartCount { By(fmt.Sprintf("Restart count of pod %s in namespace %s increased from %d to %d during the test", podDescr.Name, ns, initialRestartCount, restartCount)) pass = true break } } if !pass { Fail(fmt.Sprintf("Did not see the restart count of pod %s in namespace %s increase from %d during the test", podDescr.Name, ns, initialRestartCount)) } } </ins> var _ = Describe("Pods", func() { <del> var ( c *client.Client ) </del> <ins> var c *client.Client </ins> BeforeEach(func() { var err error

Park of ginkgo-ization
Per IRC, is working on this
Pushed an update yesterday to address comments.
what's the status of this? There's an issue open to de-flake this test (), but it's tough to do that if the code is in the middle of being moved to ginkgo.
This has been merged.


Question: }, expected: &v1.Secret{ ObjectMeta: metav1.ObjectMeta{ <del> Name: "foo-94759gc65b", </del> <ins> Name: "foo-548cm7fgdh", </ins> }, Data: map[string][]byte{ <del> v1.DockerConfigKey: secretData, </del> <ins> v1.DockerConfigJsonKey: secretData, </ins> }, <del> Type: v1.SecretTypeDockercfg, </del> <ins> Type: v1.SecretTypeDockerConfigJson, </ins> }, expectErr: false, },

/kind bug We're having trouble retrieving images from a private registry on DockerCloud. For nearly a year the following has worked flawlessly. Last Friday I pulled down the 1.9.0 CLI (via brew) and since then have not been able to access any image in a private registry on DockerCloud. I've tried this on 10+ brand new clusters created with kops v1.8.0. The Kubernetes cluster version have been 1.8.0, 1.8.4, 1.8.5 and 1.9.0 (which is not supported by kops yet) and all exhibit the same problem. We create the secret as such: And verify it using: Which yields: And create a test service with with these contents: The pod fails immediatly with: I've reviewed the 1.9.0 release notes and nothing jumps out as the root cause. For reference I've tried putting quotes around the value when creating the secret to no avail. I've also tried prefixing the value with , so the full value would read: also to no avail. Environment: Kubernetes version: Cloud provider and installation info: AWS cluster built with Kops 1.8.0, using (to isolate this issue) nearly the default settings for everything.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have same problem, and I "solved" it creating a docker-registry secret in an old kubernetes version and pasting it on new kubernetes
I'm recently (I know it worked perfectly about last month) facing issues with as well. For me it ended in a similar manner saying: I was able to confirm, that this is a kubernetes CLI related issue and not end user. Based on and this I was able to create properly formatted auth information looking like this: . When encoding this one-liner using (MacOS workstation) and apply like this works fine and images can be pulled from private repository again.


Question: }, expected: &v1.Secret{ ObjectMeta: metav1.ObjectMeta{ <del> Name: "foo-94759gc65b", </del> <ins> Name: "foo-548cm7fgdh", </ins> }, Data: map[string][]byte{ <del> v1.DockerConfigKey: secretData, </del> <ins> v1.DockerConfigJsonKey: secretData, </ins> }, <del> Type: v1.SecretTypeDockercfg, </del> <ins> Type: v1.SecretTypeDockerConfigJson, </ins> }, expectErr: false, },

However this breaks all automation and can't be considered as solution Confirmed working with kubectl 1.8.5 and server 1.8.4 (guess 1.7.9 as well)
looks like there's an extra "auths" element when adding the secret using 1.9 kubectl
Added in /assign
Many thanks for the quick fix on this. Any idea when this might get rolled out? Also, it might be a good idea if this was to the known issue list for v1.9.0. It would have saved us a bunch of time trying to track down the root cause. Just trying to make sure no one else needs to do the same.
linking to the "v1.9.0 known issues" list
ack, asking Mehdy since he is the patch manager for v1.9 branch. Mehdy, any idea when we can ship 1.9.1 with this fix?
workarounds for folks until 1.9.1 is released: if you have a .dockerconfigjson for your private registry already, you can manually specify the type and data key (e.g. ) if you don't have a .dockerconfigjson file already, you can fix up the secret produced by manually: the type from to the data key from to the modified secret with
1 will be released tomorrow
I can confirm solution works, thanks so much!
I am still having issues in 1.9.1. This worked in the 1.8.x branches: results in the following error:
you have to remove the gitlab-registry secret before you can apply an update... type is immutable, and the fix in 1.9.1 required changing the type of the secret


Question: } } <del> azClientConfig := &azClientConfig{ subscriptionID:                 config.SubscriptionID, resourceManagerEndpoint:        env.ResourceManagerEndpoint, servicePrincipalToken:          servicePrincipalToken, rateLimiterReader:              operationPollRateLimiter, rateLimiterWriter:              operationPollRateLimiterWrite, CloudProviderBackoffRetries:    config.CloudProviderBackoffRetries, CloudProviderBackoffDuration:   config.CloudProviderBackoffDuration, ShouldOmitCloudProviderBackoff: config.shouldOmitCloudProviderBackoff(), } </del> az := Cloud{ Config:                 *config, Environment:            *env,

Currently there is no way to run without any identity (be it service principal or E/MSI). In reality code running on a worker node can use via - and only via - without the need to access ARM. Here is the proposed change a new mode authN = this mode, is basically telling the cloud provider to work any identity, this means only IMDS calls is allowed. This mode can be detected from if no SPN set and both MSI settings are set to . to use a config map within the cluster. This config map can be loaded on the cluster statically. or created - if not existent - in that then only is used in The net new result of the above is users bootstrapping clusters can configure the worker node to use without any identity on said worker nodes.
/sig azure These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by
In-tree AzureDisk volume plugin is also depending on Azure credential to fetch lun id. we should figure out how to solve it without credential
fetch disk lun id is also a dependency in kubelet code now, after my investigation, it could be removed on Linux node, while it cannot be removed on Windows node, need to change the k/k upstream code, there is a discussion on going here: , let' wait for the discussion result.
For Linux: 1- use IMDS my understanding is new api versions will have the disk/lun pairs. 2- don't use arm for the return of is the second parameter passed to . For Windows: I don't understand the dep on ARM for this. The above still apply, we just need to find a way to encode the value as ut return from that allow us to find the disk by on a windows machine. might be a good source for this. Ideally speaking. The value should be the same. But i will leave it for you to decide.
could be a good idea since currently it's a LUN num returned by Attach() and then it could be changed to the windows disk num, I will make the code change on v1.15 later, thanks for discussion.
Update: PR would remove the VM API calls when getting the LUN number for AzureDisk.


Question: // InstanceShutdownByProviderID returns true if the instance is in safe state to detach volumes func (vs *VSphere) InstanceShutdownByProviderID(ctx context.Context, providerID string) (bool, error) { <del> return false, cloudprovider.NotImplemented </del> <ins> nodeName, err := vs.GetNodeNameFromProviderID(providerID) if err != nil { glog.Errorf("Error while getting nodename for providerID %s", providerID) return false, err } vsi, err := vs.getVSphereInstance(convertToK8sType(nodeName)) if err != nil { return false, err } // Ensure client is logged in and session is valid if err := vs.nodeManager.vcConnect(ctx, vsi); err != nil { return false, err } vm, err := vs.getVMFromNodeName(ctx, convertToK8sType(nodeName)) if err != nil { glog.Errorf("Failed to get VM object for node: %q. err: +%v", nodeName, err) return false, err } isActive, err := vm.IsActive(ctx) if err != nil { glog.Errorf("Failed to check whether node %q is active. err: %+v.", nodeName, err) return false, err } return !isActive, nil </ins> } // InstanceID returns the cloud provider ID of the node with the specified Name.

As planned for 1.12 in kubernetes/features: After , the next step is to add the logic to the reconciler to detach the volumes when the nodeShutdown taint is applied. /assign /sig storage /kind feature looping
How do you know that node is Shutdown? is it a planned shutdown or a unplanned shutdown?
AFAIK regardless of whether it's planned or not, it's detected by the cloud provider. In case of bare-metal node it doesn't happen automatically, as there is no universal way to reliably determine if a node is powered off, so the node must be tainted as by some external source.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
any update on this issue?
, I am currently working with sig-node team on handling node shut down. I will make a proposal and discuss in sig-node meeting next Tuesday. Will post the update soon.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /kind bug /priority critical-urgent /assign /sig node
Those labels are not set on the issue: <detailsIn response to : /remove-kind feature Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
any update on this? is it going to be part of 1.15?
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .


Question: // InstanceShutdownByProviderID returns true if the instance is in safe state to detach volumes func (vs *VSphere) InstanceShutdownByProviderID(ctx context.Context, providerID string) (bool, error) { <del> return false, cloudprovider.NotImplemented </del> <ins> nodeName, err := vs.GetNodeNameFromProviderID(providerID) if err != nil { glog.Errorf("Error while getting nodename for providerID %s", providerID) return false, err } vsi, err := vs.getVSphereInstance(convertToK8sType(nodeName)) if err != nil { return false, err } // Ensure client is logged in and session is valid if err := vs.nodeManager.vcConnect(ctx, vsi); err != nil { return false, err } vm, err := vs.getVMFromNodeName(ctx, convertToK8sType(nodeName)) if err != nil { glog.Errorf("Failed to get VM object for node: %q. err: +%v", nodeName, err) return false, err } isActive, err := vm.IsActive(ctx) if err != nil { glog.Errorf("Failed to check whether node %q is active. err: %+v.", nodeName, err) return false, err } return !isActive, nil </ins> } // InstanceID returns the cloud provider ID of the node with the specified Name.

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/priority important-longterm
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/triage accepted /kind feature
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this can be closed now as we have non-graceful node shutdown.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if s.ServiceClusterIPRange.IP == nil { glog.Fatal("No --service-cluster-ip-range specified") } <ins> var ones, bits = s.ServiceClusterIPRange.Mask.Size() if bits-ones > 20 { glog.Fatal("Specified --service-cluster-ip-range is too large") } </ins> } func newEtcd(etcdConfigFile string, etcdServerList []string, interfacesFunc meta.VersionInterfacesFunc, defaultVersion, storageVersion, pathPrefix string) (etcdStorage storage.Interface, err error) {

If you spec a /8 for your service Cluster IPs, the bitmask becomes 2 MB, which is too big for etcd. We should enforce that the prefix be /12 or smaller or something like that.
Yes


Question: } <ins> func WaitForRootPaths(t *testing.T, ctx context.Context, client testClient, requirePaths, forbidPaths sets.Set[string]) error { return wait.PollUntilContextTimeout(ctx, 250*time.Millisecond, maxTimeout, true, func(ctx context.Context) (done bool, err error) { statusContent, err := client.Discovery().RESTClient().Get().AbsPath("/").SetHeader("Accept", "application/json").DoRaw(ctx) if err != nil { return false, err } rootPaths := metav1.RootPaths{} if err := json.Unmarshal(statusContent, &rootPaths); err != nil { return false, err } paths := sets.New(rootPaths.Paths...) if missing := requirePaths.Difference(paths); len(missing) > 0 { t.Logf("missing required root paths %v", sets.List(missing)) return false, nil } if present := forbidPaths.Intersection(paths); len(present) > 0 { t.Logf("present forbidden root paths %v", sets.List(present)) return false, nil } return true, nil }) } </ins> func WaitForGroups(ctx context.Context, client testClient, groups ...apidiscoveryv2beta1.APIGroupDiscovery) error { return WaitForResultWithCondition(ctx, client, func(groupList apidiscoveryv2beta1.APIGroupDiscoveryList) bool { for _, searchGroup := range groups {

reported that conformance testing with the latest version of sonobuoy against a version 1.29.0-alpha.3 cluster fails on the support the 1.17 Sample API Server using the current Aggregator test: , related to I found the reason why the test is working fine in Kubernetes CI jobs but fails with sonobuoy is because the service account used by sonobuoy "sonobuoy-serviceaccount" doesn't have read permission of the non-resource url "/", which is used by the test to confirm the removal of group path: The permission of "sonobuoy-serviceaccount" is as below: There may be two ways to fix it: the RBAC rule of sonobuoy to include "get" of "/". However, could other test runners encounter this issue? Is there a doc listing the required permissions for running e2e tests? the e2e to skip checking the removal of the group path if the service account doesn't have permission to access the root path. Which way do we usually prefer? Running e2e with sonobuoy should succeed. a v1.29.0-alpha.3 cluster. No response <details</details<details</details<details</details<details</details<details</details<details</details
implemented this solution.
/sig api-machinery
thank you for opening this issue and working on a solution. I'm probably not the right person to decide on which solution is preferred. From my perspective, either solution is fine.
Note: Sonobuoy is not owned by any Kubernetes SIG. At most it is between SIG Architecture and the CNCF to choose whether to recommend it for running tests. SIG Testing owns the e2e image that sonobuoy can be used to schedule but not sonobuoy. IMHO this is a bug in running the tests via a pod, which is not how we develop the tests (they're run outside the cluster, by a cluster admin account typically). I would say 1) is the correct answer. xref: /sig architecture /sig testing /triage accepted cc
thanks for your comment. I guess 1) means non of the existing sonobuoy versions can pass K8s conformance test against a cluster 1.29.0-alpha3, as the RBAC rule is hardcoded.


Question: } <ins> func WaitForRootPaths(t *testing.T, ctx context.Context, client testClient, requirePaths, forbidPaths sets.Set[string]) error { return wait.PollUntilContextTimeout(ctx, 250*time.Millisecond, maxTimeout, true, func(ctx context.Context) (done bool, err error) { statusContent, err := client.Discovery().RESTClient().Get().AbsPath("/").SetHeader("Accept", "application/json").DoRaw(ctx) if err != nil { return false, err } rootPaths := metav1.RootPaths{} if err := json.Unmarshal(statusContent, &rootPaths); err != nil { return false, err } paths := sets.New(rootPaths.Paths...) if missing := requirePaths.Difference(paths); len(missing) > 0 { t.Logf("missing required root paths %v", sets.List(missing)) return false, nil } if present := forbidPaths.Intersection(paths); len(present) > 0 { t.Logf("present forbidden root paths %v", sets.List(present)) return false, nil } return true, nil }) } </ins> func WaitForGroups(ctx context.Context, client testClient, groups ...apidiscoveryv2beta1.APIGroupDiscovery) error { return WaitForResultWithCondition(ctx, client, func(groupList apidiscoveryv2beta1.APIGroupDiscoveryList) bool { for _, searchGroup := range groups {

I will file an issue to sonobuoy first.
question here is what are the set of client permissions required to run the conformance suite? perhaps a level of access or a more granular recommendation should be documented. what is your take on the root path requirement as approver for: EDIT: slack thread:
I agree, we already require e.g. privileged pods for a long time in conformance tests and we should make it clear that we expect a highly privileged client.
+1 for what is saying, that we should expect that the user running the tests has required permissions. These tests are not built for being run unprivileged.
i wanted to PR the admin note in our conformance docs under sig-arch, but then i saw this: so broke this guarantee. sounds like SIG Arch needs to comment.
This statement surprises me. Root discovery is one of the least-privileged permissions you can have on a cluster. For years this was even open for unauthorized users on the internet (if your api endpoint has been exposed). In other words, just add the permission. While thinking about it, it is surprising that the user in question does not have this access already. Don't we allow that for every authorized user?
In contrast, I hope that lots of privileged operations are tested by conformance today. If we don't test things like admission webhooks for example, we have much bigger problems. And admission webhooks are basically equivalent to cluster-admin.
that is outdated, we use hostNetwork pods one some of the tests IIRC
looks like we don't. before this ticket, i was under the impression that the conformance suite already requires admin and that all testers (like Sonobuoy) use admin level credentials. it is non-privileged (e.g., does not require root on nodes, access to raw network interfaces, or cluster admin permissions) looks like we may have to change that sentence completely.
Would have expected it here: The PR in question explicitly fixes the result returned by . If that is part of conformance (IMO it should), every conformance test needs access it obviously. Whether we require admin, or we add it manually, either would work.


Question: } <ins> func WaitForRootPaths(t *testing.T, ctx context.Context, client testClient, requirePaths, forbidPaths sets.Set[string]) error { return wait.PollUntilContextTimeout(ctx, 250*time.Millisecond, maxTimeout, true, func(ctx context.Context) (done bool, err error) { statusContent, err := client.Discovery().RESTClient().Get().AbsPath("/").SetHeader("Accept", "application/json").DoRaw(ctx) if err != nil { return false, err } rootPaths := metav1.RootPaths{} if err := json.Unmarshal(statusContent, &rootPaths); err != nil { return false, err } paths := sets.New(rootPaths.Paths...) if missing := requirePaths.Difference(paths); len(missing) > 0 { t.Logf("missing required root paths %v", sets.List(missing)) return false, nil } if present := forbidPaths.Intersection(paths); len(present) > 0 { t.Logf("present forbidden root paths %v", sets.List(present)) return false, nil } return true, nil }) } </ins> func WaitForGroups(ctx context.Context, client testClient, groups ...apidiscoveryv2beta1.APIGroupDiscovery) error { return WaitForResultWithCondition(ctx, client, func(groupList apidiscoveryv2beta1.APIGroupDiscoveryList) bool { for _, searchGroup := range groups {

adding to milestone and making release blocking ... we should not release 1.29 in a state that the most commonly used conformance tool will fail
IMO we should clarify the state and go fix the tool, but we should keep the test and we should NOT add skip. Conformance docs are not a release artifact, and sonobuoy release lifecycle is not a conformance artifact. If we can't get sonobuoy working in a reasonable time, we already know the tests can be run by scheduling the image (which is ours) directly with kubectl instead and we can document that in the upstream conformance docs. We need to fix the for this image: ,%20master%20(dev) And it should probably be to release-blocking in the future.
it looks like the e2e change made in was not consciously made part of conformance, it was just trying to functionally test the bugfix being made. For 1.29, I think we should revert the change to and make an integration test that validates the fix instead. We can revisit whether content of is part of conformance or not for 1.30
I opened to revert the e2e change and move the functional test to an integration test
Filed to try to get the image job passing again.
We probably need another issue to track 1.30 follow-up?
If no one has done it, I could create another issue and follow up with the items that have been discussed here: outdated privilege requirement in commonly used conformance tools like sonobuoy aware of the required priviledge to avoid similar situations in the future. permission of GET "/" to authorized users via the role for root discovery.


Question: fakeDocker.VersionInfo = []string{} kubeClient.ReactFn = testclient.NewSimpleFake(&api.NodeList{Items: []api.Node{ <del> {ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}}, </del> <ins> {ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}}, </ins> }}).ReactFn mockCadvisor := testKubelet.fakeCadvisor machineInfo := &cadvisorApi.MachineInfo{


Question: fakeDocker.VersionInfo = []string{} kubeClient.ReactFn = testclient.NewSimpleFake(&api.NodeList{Items: []api.Node{ <del> {ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}}, </del> <ins> {ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}}, </ins> }}).ReactFn mockCadvisor := testKubelet.fakeCadvisor machineInfo := &cadvisorApi.MachineInfo{


Question: // a map[family]ip for the caller to release when everything else has // executed successfully func (rs *REST) handleClusterIPsForUpdatedService(oldService *api.Service, service *api.Service) (allocated map[api.IPFamily]string, toRelease map[api.IPFamily]string, err error) { <ins> // We don't want to upgrade (add an IP) or downgrade (remove an IP) // following a cluster downgrade/upgrade to/from dual-stackness // a PreferDualStack service following principle of least surprise // That means: PreferDualStack service will only be upgraded // if: // - changes type to RequireDualStack // - manually adding or removing ClusterIP (secondary) // - manually adding or removing IPFamily (secondary) if isMatchingPreferDualStackClusterIPFields(oldService, service) { return allocated, toRelease, nil // nothing more to do. } </ins> // use cases: // A: service changing types from ExternalName TO ClusterIP types ==> allocate all new // B: service changing types from ClusterIP types TO ExternalName ==> release all allocated

1) start with a single stack cluster 2) create a PreferDualStack service 3) configure API server with a 2nd service CIDR 4) Confirm the above service is not dual-stack 5) Add a label to the above service 6) Observe that the service becomes dual-stack I think the correct answer is to have a "repair loop" that upgrades (and downgrades!) PreferDualStack Services to match the config. I think this is a GA blocker.
/triage accepted The current thought is: of type with are special new breed of api objects that I don't think we have any similar precedent in the apis, i could be wrong. Their behavior is tied to how the cluster is configured. Thus their behavior must change as the cluster behavior change. a way the user is stating . If it is cluster then i am good with two ClusterIPs. If it is single stack cluster then I am also good with a single ClusterIP. One can assume that if the user was ok was with getting a additional IP when possible then they must be ok with taking it away when it is no longer possible. Based on the above I believe best course of action is create a new repair loop or piggyback on any of the existing (or better: fix it in rest stack during list and get calls). The new logic takes care of matching to the .
reproducer and regression test if somebody takes this, feel free to add it
ack - Scheduled to tackle 3rd week of May. Will definitely include that patch.


Question: //	    - ... zero or more // // * every specified field or array in s is also specified outside of value validation. <ins> // * metadata at the root can only restrict the name and generateName, and not be specified at all in nested contexts. </ins> // * additionalProperties at the root is not allowed. func ValidateStructural(s *Structural, fldPath *field.Path) field.ErrorList { allErrs := field.ErrorList{}

We allow CRD authors to write . Currently we don't restrict writing validation for , which means people can have . For example, people can have additive name constraints (require CR to be ) with the following schema: This brings us two questions: do we if "metadata" is included in the schema: simplest option is to publish standard metav1.ObjectMeta schema info, and use any CRD-specified "metadata" properties as additive validation (just like in-tree types do, publishing standard ObjectMeta schema info, and implementing type-specific name validation internally) alternatively, we can try merging user-specified metadata schema with the standard one. Technically people should only add extra validation to existing metadata properties (adding new fields to metadata won't work since they will be pruned by built-in validation first), but it's unclear to me how arbitrary people may want the metadata validation to be term question is whether to allow people specifying metadata validation at all, or just allow a subset (like name validation) in CRD v1 /sig api-machinery /area custom-resources cc
What is the difference between these two? I mean the end result is the same right? The only thing that can be is validation. Nothing else works unless you are considering extensions?
The former means we drop/hide user-specified metadata schema when we publish CRD OpenAPI. We still validate user-specified constraints on server side, but client-side validation won't see them. The latter enables client-side user-specified metadata validation.
If it's possible to merge with standard metadata schema info, or at the cost of not publishing the standard schema. My inclination would be to publish the complete standard schema for metadata/apiVersion/kind, which we know is complete and accurate, and use and user-specified schema for those fields as addition validations server side


Question: kr.CustomMustRegister(cs...) } <ins> // BuildVersion is a helper function that can be easily mocked. var BuildVersion = version.Get </ins> func newKubeRegistry(v apimachineryversion.Info) *kubeRegistry { r := &kubeRegistry{ PromRegistry:     prometheus.NewRegistry(),

What would you like to be : Once a metric be deprecated, we should update the unit test accordingly if there is any. We need to work out a way for a unit test freely select codebase version, and also should make sure the unit test won't fail in the subsequent release. Why is this needed: Deprecated also needs to be protected. /sig instrumentation /assign


Question: // Store in cache c.lock.Lock() <ins> defer c.lock.Unlock() </ins> c.cache[key] = data <del> c.lock.Unlock() </del> // Remove in flight entry c.inFlightLock.Lock() <ins> defer c.inFlightLock.Unlock() </ins> delete(c.inFlight, key) <del> c.inFlightLock.Unlock() </del> }() return result }

Failed due to an apparently unrelated PR: https://travis-
Problem here is that I left a small chance of a double-call in my parallelized cache. Didn't realize this was being tested. Will fix.