Question: // Plugin is an implementation of admission.Interface. type Plugin struct { *admission.Handler <del> client kubernetes.Interface lister schedulingv1listers.PriorityClassLister </del> <ins> client                          kubernetes.Interface lister                          schedulingv1listers.PriorityClassLister resourceQuotaFeatureGateEnabled bool nonPreemptingPriority           bool </ins> } var _ admission.MutationInterface = &Plugin{} var _ admission.ValidationInterface = &Plugin{} <ins> var _ genericadmissioninitializers.WantsFeatures = &Plugin{} </ins> var _ = genericadmissioninitializers.WantsExternalKubeInformerFactory(&Plugin{}) var _ = genericadmissioninitializers.WantsExternalKubeClientSet(&Plugin{})

We are using namespaces other than kube-system for system pods as it is getting too crowded, therefore we would like to run cluster-node-critical pods in other namespaces.
Waiting for this to be available myself. No reason my Node Exporter dataset has to run in the namespace instead of my namespace.


Question: // Store volume metadata for UnmountDevice. Keep it around even if the // driver does not support NodeStage, UnmountDevice still needs it. <del> if err = os.MkdirAll(deviceMountPath, 0750); err != nil && !corruptedDir { </del> <ins> if err = os.MkdirAll(deviceMountPath, 0750); err != nil { </ins> return errors.New(log("attacher.MountDevice failed to create dir %#v:  %v", deviceMountPath, err)) } klog.V(4).Info(log("created target path successfully [%s]", deviceMountPath))

What happened: A custom CSI driver implementing returns a failure response, indicating that it did not complete its intended steps to prepare the volume for the mounting pod. Let's say that these steps involve a filesystem at the target directory, data from a service, and that data to a set of files in the mounted filesystem. If the first filesystem mounting step succeeds, but either of the subsequent two steps fail, the volume is still not ready for use by the pod. However, if this CSI driver returns a failure response, and the CSI mounter comes back and, before invoking the RPC again, it finds a filesystem already mounted at the target directory, , and ceases further calls to to allow the CSI driver to try again. What you expected to happen: If a CSI driver returns a failure response to a RPC, Kubernetes should honor that declaration of failure and invoke the RPC again, disallowing the mounting pod from starting until this RPC succeeds. Even though it's possible to partially work around the current behavior by having the driver mount the filesystem as late as possible, and then attempting to unmount it if fails to populate the volume with its intended data, doing so makes writing drivers more complicated and forces them to do extra work. How to reproduce it (as minimally and precisely as possible): In a CSI driver (even one that serves just ephemeral, inline volumes), implement to mount a filesystem at the target path, then return a failure response. Confirm that Kubernetes does not invoke again, because there's now a filesystem mounted at the target path. Anything else we need to know?: This topic came up for discussion in the "Kubernetes" Slack team. There, helped investigate this complaint, and to discuss changing this behavior. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Others: csi-node-driver-registrar container image tag: v1.0.2
/sig storage
While the plugin should try to clean up as much as possible, I agree that kubelet should not be assuming that if there's a mount point, the publish is successful. NodePublish is supposed to be idempotent so it should theoretically be safe to call it when it's already mounted, although there is the possibility that plugins may have written against the K8s behavior.


Question: // Store volume metadata for UnmountDevice. Keep it around even if the // driver does not support NodeStage, UnmountDevice still needs it. <del> if err = os.MkdirAll(deviceMountPath, 0750); err != nil && !corruptedDir { </del> <ins> if err = os.MkdirAll(deviceMountPath, 0750); err != nil { </ins> return errors.New(log("attacher.MountDevice failed to create dir %#v:  %v", deviceMountPath, err)) } klog.V(4).Info(log("created target path successfully [%s]", deviceMountPath))

For the mount point checking here before calling NodePublishVolume, I am wondering it is just for performance optimization, or it has other purposes. Have concern about some drivers might depend on this check if they are not implemented with idenpotency.
Those drivers would be operating in violation of the specification. I'd prefer that we don't penalize already conforming and yet-to-be written drivers in order to accommodate these existing drivers—if they even exist.
More fallout in last night's testing: I introduced a bug into my driver that caused it to panic during ("Writing that data to a set of files in the mounted filesystem"). My handler function is set up to unmount a successfully mounted volume via if it doesn't complete steps 2 and 3 above, in order to deal with this issue's principal complaint. My driver container would crash due to this panic. The kubelet would call on my driver twice, it would crash twice, but the pod still started running. I'm not yet sure whether or not the filesystem got unmounted successfully via the on the first call, but somehow the pod started despite my handler function never completing successfully. I'm going to try to get some more debugging time on this later today to see if I can figure out what the kubelet was doing there.
Studying my code more, I see that my function only unmounted the mounted directory when the containing handler function was returning an error. When unwinding due to a panic, the named return value of type I was inspecting was still nil. Revising the criteria in that function covers both failure reasons (returning an and unwinding due to a panic). Regardless, this further underscores the problem: If a driver makes it as far as mounting a filesystem at the target directory, but fails afterward—including the process crashing and never responding positively to the RPC—the mounting pod will start. That's likely to then cause the pod to fail, for lack of the files it expected to find mounted.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.


Question: // Store volume metadata for UnmountDevice. Keep it around even if the // driver does not support NodeStage, UnmountDevice still needs it. <del> if err = os.MkdirAll(deviceMountPath, 0750); err != nil && !corruptedDir { </del> <ins> if err = os.MkdirAll(deviceMountPath, 0750); err != nil { </ins> return errors.New(log("attacher.MountDevice failed to create dir %#v:  %v", deviceMountPath, err)) } klog.V(4).Info(log("created target path successfully [%s]", deviceMountPath))

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: # have failed before getting to the "echo" in the block below. diff=$(find_files | xargs ${gofmt} -d -s 2>&1) || true if [[ -n "${diff}" ]]; then <del> echo "${diff}" </del> <ins> echo "${diff}" >&2 echo >&2 echo "Run ./hack/update-gofmt.sh" >&2 </ins> exit 1 fi

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: When running on my commits locally I got the following error: Affected PRs:
/sig testing
/cc
There's likely something up with your local environment. These jobs are running correctly in CI, although you need to look at the build log for output


Question: if len(keySplit) == 5 && keySplit[4] == "device-number" { for i, macElem := range m.aws.networkInterfacesMacs { if macParam == macElem { <del> return fmt.Sprintf("%dn", i), nil </del> <ins> n := i if n > 0 { // Introduce an artificial gap, just to test eg: [eth0, eth2] n++ } return fmt.Sprintf("%dn", n), nil </ins> } } }

What happened: does not handle "holes" in interface device-number correctly, dropping later interfaces. Create instance with eth0, eth1, eth2 ENIs as usual. Remove (eg: by selectively deleting pods) eth1. Only the addresses on eth0 are reported by NodeAddresses. What you expected to happen: Addresses on eth0 and eth2 should be reported.
/sig cloud-provider These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.


Question: continue } c.watchersBuffer = append(c.watchersBuffer, watcher) <del> // Given that we send bookmark event once at deadline-2s, never push again // after the watcher pops up from the buckets. Once we decide to change the // strategy to more sophisticated, we may need it here. </del> <ins> // Requeue the watcher for the next bookmark if needed. c.bookmarkWatchers.addWatcher(watcher) </ins> } } }

Currently, (best-effort) watch bookmarks are sent just right before the given watch is supposed to timeout: In we're adjusting how the size of watch cache is determined and make it dynamic and more duration-oriented (rather than based on some magic constants) - we're targeting to always keep 5 minutes of history in watch cache. With that change, we should get back to our bookmark policy: change it to (additionally to right before timeout) also send bookmarks every N seconds (setting N to slightly less than expected watch cache history period, so slightly less than X minutes we will use there) scale-test this change (to see if it doesn't introduce non-negligible overhead) if so permanently introduce this change /sig scalability api-machinery
/cc
/cc
/cc
I'm reverting as it broke bookmarks. I'm going to fix and re-submit for review hopefully later this week.


Question: if err != nil { return err } <ins> printService(o.Out, "Kubernetes master", o.Client.Host) </ins> services := r.Object.(*api.ServiceList).Items for _, service := range services { var link string

/kind bug What happened: The wording of kubectl cluster-info is misleading. When no cluster is running, it will state (in green) that the kubernetes master is running. What you expected to happen: I would expect a different message, like Or a How to reproduce it (as minimally and precisely as possible): Execute kubectl with no config files, and no local kubernetes cluster running on 8080
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, } <del> # In the bash-based build (build/lib/release.sh), the images built for amd64 do not use # an arch in their name (but other arches do), and the GCE cluster scripts # (which sideload the images via tarfiles) expect there not to be an arch. </del> # When pushing to gcr.io, we want to use an arch, since the archless name is now used for a # manifest list. Bazel doesn't support manifest lists (yet), so we can't do that either. <del> # For now, we use the archless name for the image tars saved in the server tarball, # to satisfy GCE and other similar providers. (If one were to pull the images via the manifest # list, the arch wouldn't appear in the name either.) </del> [multi_arch_container( name = binary, architectures = SERVER_PLATFORMS["linux"],

What happened: When we build with make to build k8s, it will use 'amd64' in the container names. This is not the case for bazel builds, which are still using something like 'kube-apiserver' instead of 'kube-apiserver-amd64' What you expected to happen: Images build with make or bazel should have 'amd64' in the container name How to reproduce it (as minimally and precisely as possible): make quick-release cat inside tar "RepoTags": ["" make bazel-release cat inside tar "RepoTags": [""
/sig cluster-lifecycle /assign
+1 we need to standardize both builds to generate the same. +1 to add to the bazel build
FYI
+1, cc
bazel / docker build parity SGTM.
+1. Perhaps we should document a standard, like: All docker images that are arch specific should have the arch in the name Arch is only able to be omitted when it's a manifest-based tag (multi-arch) Document legacy exceptions, and hopefully bring them into compliance /cc
i like it. where is the best spot to doc this?
-- please doc as updates to:
Created


Question: return job, nil } <del> func getTimeHash(scheduledTime time.Time) uint32 { timeHasher := adler32.New() hashutil.DeepHashObject(timeHasher, scheduledTime) return timeHasher.Sum32() </del> <ins> // Return Unix Epoch Time func getTimeHash(scheduledTime time.Time) int64 { return scheduledTime.Unix() </ins> } // makeCreatedByRefJson makes a json string with an object reference for use in "created-by" annotation value

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux What happened: Deployed a scheduled job from this manifest: After some time the jobs are no longer executed: What you expected to happen: Jobs are executed as defined in the manifest. How to reproduce it (as minimally and precisely as possible): Deploy a scheduled job and wait for some time.
Some logs from the controller manager:
Ping. More than 2 weeks and no action at all? Anything else i can do?
I'm also encountering this using both and as concurrency policies, fwiw.
I'm hitting this as well. Kube
Do the jobs start to reschedule if you delete old one? Try running . cc
From kube-controller-manager logs: The job is scheduled for 1:00. I think the "Multiple unmet start times" message is a clue of where things start going downhill. Also, notice how at 1:00 the controller wants to create job (for a whole day, it had failed on ).
Deleting (all) old jobs / pods fixes the problem temporarily. After some time the scheduled jobs face the same issue again.
There's a fix () that I've submitted against master to fix the Replace strategy, I'll mark it for inclusion in 1.4.
The other problem that I see here is this: It is possible that our is creating duplicates. I'll try to investigate that a bit more.
sorry for the delay in response time, I must have missed that while going through issues. Next time you're experiencing problems/have a question wrt Jobs/ScheduledJobs, just ping me directly in the issue.
No problem and I will. Thanks!
I'm still hitting this on
Fix in flight , I'll make sure to cherry-pick it back to 1.4
does this ship in to latest kubernetes release?
yes, this should be present in 1.5, 1.4 cherry-pick hasn't been merged yet.
so this issue is still present in v1.4.6?


Question: return job, nil } <del> func getTimeHash(scheduledTime time.Time) uint32 { timeHasher := adler32.New() hashutil.DeepHashObject(timeHasher, scheduledTime) return timeHasher.Sum32() </del> <ins> // Return Unix Epoch Time func getTimeHash(scheduledTime time.Time) int64 { return scheduledTime.Unix() </ins> } // makeCreatedByRefJson makes a json string with an object reference for use in "created-by" annotation value

yes - it is still present there. My fix was merged to 1.4 yesterday. You should wait for the next 1.4.x release.
thanks!
which according to should happen Dec 8th.
any news on the release?
See !msg/kubernetes-dev/o0At2-4KKQ8/_kPVM1XyDAAJ
so v1.4.7 will be released on that same day?
Look like should be available later today: !msg/kubernetes-dev/00lJpT9pxDc/YZfDzCxMDQAJ


Question: Annotations: newPodAnnotations(pod), } <ins> dnsServers, dnsSearches, useClusterFirstPolicy, err := m.runtimeHelper.GetClusterDNS(pod) if err != nil { return nil, err } podSandboxConfig.DnsConfig = &runtimeapi.DNSConfig{ Servers:  dnsServers, Searches: dnsSearches, } if useClusterFirstPolicy { podSandboxConfig.DnsConfig.Options = defaultDNSOptions } </ins> if !kubecontainer.IsHostNetworkPod(pod) { <del> dnsServers, dnsSearches, _, err := m.runtimeHelper.GetClusterDNS(pod) if err != nil { return nil, err } podSandboxConfig.DnsConfig = &runtimeapi.DNSConfig{ Servers:  dnsServers, Searches: dnsSearches, Options:  defaultDNSOptions, } </del> // TODO: Add domain support in new runtime interface hostname, _, err := m.runtimeHelper.GeneratePodHostNameAndDomain(pod) if err != nil {

Kubernetes version (use ): HEAD (~v1.6.0-beta.4) Environment: any What happened: I want to use ClusterFirstWithHostNet which was implemented in , but only dockertools (legacy/non-CRI) was updated to support this feature. The CRI dockershim should be modified to support this rewriting the file of Pods with HostNetwork=true as well. This is a bug in v1.6, I hope it can be prioritized to be fixed in v1.6.1, since it's blocking me from using CRI. Since CRI is enabled by default, it's expected that it should have feature parity with the earlier implementation.
Related
- I guess I didn't realize these crossed paths. Do you have time to take a look?
Will take a look
Thanks! Ping me directly on this.
Filed for supporting it in CRI.


Question: return done, nil }) <del> if pollErr == wait.ErrWaitTimeout { </del> <ins> if pollErr != nil { </ins> t.Fatalf("failed to get the expected healthz status of !OK for check: %s, error: %v, debug inner error: %v", checkName, pollErr, restErr) } }

{Failed === RUN TestKMSv2Healthz --- FAIL: TestKMSv2Healthz (167.38s) also shown in " TestKMSv2Healthz": 1, Not sure No response No response /sig auth
/assign /triage accepted Use for poll interval Change the poll timeout to 5m Update the poll error check (check if not nil)


Question: }, }, }, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { <del> pinged = true </del> <ins> time.Sleep(3 * time.Second) pinged.Store(true) </ins> service.ServeHTTP(w, r) })) testCtx, cancel := context.WithCancel(context.Background()) defer cancel() go aggregatedManager.Run(testCtx.Done()) <del> require.True(t, waitForEmptyQueue(testCtx.Done(), aggregatedManager)) </del> <ins> require.True(t, waitForQueueComplete(testCtx.Done(), aggregatedManager)) </ins> // immediately check for ping, since Run() should block for local services <del> if !pinged { </del> <ins> if !pinged.Load() { </ins> t.Errorf("service handler never pinged") } }

pull-kubernetes-unit sometimes fails with data race. TestDirty I think the test was 3 months ago with the flaking state. There are two issues. First, updating and reading a bool-type variable is not thread-safe. and there is also another problem checking the state before completely syncing it. No response /sig api-machinery
/assign
/triage accepted


Question: f := framework.NewDefaultFramework("volume-limits-on-node") ginkgo.BeforeEach(func() { e2eskipper.SkipUnlessProviderIs("aws", "gce", "gke") <ins> // If CSIMigration is enabled, then the limits should be on CSINodes, not Nodes, and another test checks this e2eskipper.SkipIfCSIMigrationEnabled() </ins> c = f.ClientSet framework.ExpectNoError(framework.WaitForAllNodesSchedulable(c, framework.TestContext.NodeSchedulableTimeout)) })

Right now, we have e2e test for that executes only CSI mock driver with artificial limit of 2 volumes per node. We need a test with real CSI driver(s). Proposal: Allow individual test storage drivers to specify if they support CSI limits. Add a test that picks a random node and creates as many pods with a PV that the volume can handle (128 on GCE!). Check that the node can't handle any other volume. Make this test part of . Add similar test(s) for in-tree volume plugins. Make sure the tests work with in-tree volume plugins migrated to CSI.
/sig storage
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
This sounds important /remove-lifecycle rotten /lifecycle frozen
/priority backlog
To add to this I believe we should test that an in-tree plugin and its "replacement" CSI Driver share the same attach limits (both max and current usage)
/cc
I see this test has been here: But I want to add a more basic/faster version of the test that simply checks if a driver has then has been set in CSINode, similar to wdyt, would it be useful?
This is a really old issue :-D. I think the required test has been implemented, however, we do not run it in CI, because it takes too much time to provision 128 PDs that a GCP node can handle. /unassign That sounds useful!
/assign opened to close this for good :)


Question: cidrBlocks := make([]string, 0, len(vpcs.Vpcs[0].CidrBlockAssociationSet)) for _, cidr := range vpcs.Vpcs[0].CidrBlockAssociationSet { <ins> if aws.StringValue(cidr.CidrBlockState.State) != ec2.VpcCidrBlockStateCodeAssociated { continue } </ins> cidrBlocks = append(cidrBlocks, aws.StringValue(cidr.CidrBlock)) } return cidrBlocks, nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If secondard CIDR are disassociated and associated again, AWS LoadBalancer creation fails. It's due to the VPC state is like so CIDRBlock is twice to securityGroup rules, which result in error What you expected to happen: only associated CIDR are considered when calculate VPC CIDR How to reproduce it (as minimally and precisely as possible): and then associated a secondary CIDR AWS CLB Service Anything else we need to know?: Environment: Kubernetes version (use ): <=1.18.0 Cloud provider or hardware configuration: aws OS (e.g: ): linux Kernel (e.g. ): linux Install tools: kops/eks Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.


Question: func newCmdConfigSetAuthInfo(out io.Writer, options *createAuthInfoOptions) *cobra.Command { cmd := &cobra.Command{ <del> Use:                   fmt.Sprintf("set-credentials NAME [--%v=path/to/certfile] [--%v=path/to/keyfile] [--%v=bearer_token] [--%v=basic_user] [--%v=basic_password] [--%v=provider_name] [--%v=key=value]", clientcmd.FlagCertFile, clientcmd.FlagKeyFile, clientcmd.FlagBearerToken, clientcmd.FlagUsername, clientcmd.FlagPassword, flagAuthProvider, flagAuthProviderArg), </del> <ins> Use: fmt.Sprintf( "set-credentials NAME [--%v=path/to/certfile] "+ "[--%v=path/to/keyfile] "+ "[--%v=bearer_token] "+ "[--%v=basic_user] "+ "[--%v=basic_password] "+ "[--%v=provider_name] "+ "[--%v=key=value] "+ "[--%v=exec_command] "+ "[--%v=exec_api_version] "+ "[--%v=arg] "+ "[--%v=key=value]", clientcmd.FlagCertFile, clientcmd.FlagKeyFile, clientcmd.FlagBearerToken, clientcmd.FlagUsername, clientcmd.FlagPassword, flagAuthProvider, flagAuthProviderArg, flagExecCommand, flagExecAPIVersion, flagExecArg, flagExecEnv, ), </ins> DisableFlagsInUseLine: true, Short:                 i18n.T("Sets a user entry in kubeconfig"), Long:                  createAuthInfoLong,

Now it's possible to configure third-party auth providers using command with option. It's nice to have the same possibility for the recently . Example: To specify the executable path we can run This will generate a new config for the user: Various auxiliary configuration options are also could be . For instance: to define an argument for the executable; to set an environment variable; to specify a desired API version. /kind feature
I'm going to start working on this when the API is promoted to beta.
/sig auth
/assign
Can someone share why you can't do : Is there a way to set map values (e.g. something that is -name: value ; versus something that is -context: )
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I would like to work on it, if this is still persist.
cc
This still seems useful and is outstanding. any objections to taking it over?
No, it's okay, I'm working on it. I'll post the code this week, sorry for the delay
I think this would help out with generating valid kubectl configurations for our EKS developers. I'm using Pulumi to build EKS environments and if all the flags are available, I think I can generate the necessary entries to assume a role. Thanks! Ex: name: k8s.developer user: exec: apiVersion: args: token -i cluster-eksCluster- -r arn:aws:iam:::role/k8sDevAdminRole- command: aws-iam-authenticator env: null
done :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/priority important-longterm


Question: return err } <ins> if string(data) == origBody { return nil } </ins> _, err = h.Client.CompareAndSwap(key, string(data), 0, origBody, index) if IsEtcdTestFailed(err) { continue

When you setup TLS assets, you need to make sure each master API server is using the same ones, unless you were running a single master. But this is a super common mistake, and it often breaks things. Because the masters validating tokens using that key, the controller-manager also has this service-account key, and it’s responsible for creating tokens for service accounts. So, depending on which apiserver the pod is talking to, that service-account token may, or may not work. This most commonly surfaces in kube2sky (the dns addon). Result: Tokens are invalidated regularly (~10 minutes) Expected: Warning about system having different certs (or some alert). Would this ever be a valid configuration?
Is this specifically for an HA setup (since you mention multiple apiservers)? Or do you mean that each master component should have the same cert?
Correct - it's the HA deployment.
I don't think that the apiservers should necessarily cross-check each other (and all other system components like the controller manager) and alert. If you are running an HA deployment, you should have something else making sure that your deployment is correct. On cluster creation, the startup scripts should do it.
I guess what I'm saying is that we know this is going to be a problem. There's almost no way to make this NOT a problem, yet it is a very easy bug to get into. So... with that in mind, what's the path for doing this? When you add a second master to your HA config, how does it establish trust with the first HA? Could we check at that point?
Yup, we do this on provision time with Ansible.


Question: "github.com/onsi/ginkgo/v2" <ins> "k8s.io/klog/v2" "k8s.io/klog/v2/ktesting" </ins> "k8s.io/kubernetes/test/e2e/framework" "k8s.io/kubernetes/test/e2e/framework/internal/output" testapiserver "k8s.io/kubernetes/test/utils/apiserver"

pull-kubernetes-unit Probably since it was merged in The test checks output from the E2E framework and sometimes some extra log output shows up that isn't expected: No response /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: // Assert // 4. When the volume is mounted again, expect that UnmountDevice operation did not clear devicePath if devicePath == "" { <del> t.Errorf("Expected WaitForAttach called with devicePath from Node.Status") </del> <ins> klog.ErrorS(nil, "Expected WaitForAttach called with devicePath from Node.Status") </ins> close(finished) return "", fmt.Errorf("Expected devicePath from Node.Status") }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --N/A Triage instead: Data race. links to appreciated links to specific failures in spyglass appreciated specific failure <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
previously:
The data race is for this time . fixed only by using a copy.
/assign
GitHub didn't allow me to assign the following users: Haleygo. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Seems noone work here, try to reproduce and fix. /assign
kubernetes# ./stress -p 20 - "TestRunPositive_VolumeMountControllerAttachEnabledRace" 15m25s: runs so far, 0 failures :( I cannot reproduce it, does anyone have better suggestions?
? I have no better idea.
-p 200 ,-p 300, I reproduced this fault (and another DATA RACE). I continued to test after adding deepCopy and found that the stress did not seem to recognize my modification, because the DATA RACE error line was the same as the original, I was a little bit confused....
confused: Use to compile stress, Add Info log to prove that the code has been modified: Call and fail the increased log is not printed (one timeout). As a result, I cannot verify whether I have fixed this problem!
I forgot to update , now the test continues. :-)


Question: "k8s.io/api/core/v1" extensions "k8s.io/api/extensions/v1beta1" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> "k8s.io/apimachinery/pkg/util/intstr" )

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The DaemonSet controller will schedule a pod on a node when the old pod is still in Terminating state. This breaks the expectation that only one pod will be running at a time per node. What you expected to happen: The DaemonSet controller should not let two pods running at the same time on the same node. How to reproduce it (as minimally and precisely as possible): Create a DaemonSet. Set terminationGracePeriodSeconds to be relatively long. Kill a pod. Watch a new pod start on the same node when the old pod is still terminating. Anything else we need to know?: Seems this behavior was introduced in Environment: Kubernetes version (use ): 1.7.2 Cloud provider or hardware configuration**: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps
/cc
cc
cc: I think the expected behavior is correct, the terminating pod could be performing an important step such as 'removing a lock' or 'cleanup' etc., we should allow that to gracefully die before creating a new one. IMHO we should revoke to old behavior, but why did we need this new behavior in the first place? That is not clear to me from the PR and Issue.
We don't currently guarantee at least or at most one scheduling for DaemonSet. Both behaviors are under consideration via issue and in the future work section of the . The reason for is to "Make sure that controllers are not adopting children that are marked for deletion" (). This is necessary for GC to work properly.
Well the behavior may be technically undefined, but can I request that in the future changes like this be mentioned in the release notes?
Yes, certainly.


Question: "k8s.io/api/core/v1" extensions "k8s.io/api/extensions/v1beta1" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> "k8s.io/apimachinery/pkg/util/intstr" )

Also "The DaemonSet controller should not let two pods running at the same time on the same node." is not necessarily the same thing as respecting termination prior to launching a new Pod. The former implies a more strict requirement than the latter. I think we should consider at least attempting to wait for a terminating Pod to be deleted prior to creating another Pod on the Node for same DaemonSet as default behavior.
will something like work then?


Question: </g> <g> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="278.614" y1="594.318" x2="229.688" y2="545.385"/> <del> <polygon style="fill: #000000" points="224.385,540.081 234.992,543.618 229.688,545.385 227.92,550.688 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="224.385,540.081 234.992,543.618 229.688,545.385 227.92,550.688 "/> </del> <ins> <polygon style="fill: #000000" points="224.385,540.081 234.991,543.618 229.688,545.385 227.92,550.688 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="224.385,540.081 234.991,543.618 229.688,545.385 227.92,550.688 "/> </ins> </g> <g> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="223.458" y1="443.5" x2="223.059" y2="458.767"/> <del> <polygon style="fill: #000000" points="222.863,466.265 218.126,456.137 223.059,458.767 228.123,456.399 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="222.863,466.265 218.126,456.137 223.059,458.767 228.123,456.399 "/> </del> <ins> <polygon style="fill: #000000" points="222.862,466.265 218.126,456.137 223.059,458.767 228.122,456.399 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="222.862,466.265 218.126,456.137 223.059,458.767 228.122,456.399 "/> </ins> </g> <g> <del> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="308.892" y1="503.5" x2="362.29" y2="504.346"/> <polygon style="fill: #000000" points="369.789,504.465 359.711,509.306 362.29,504.346 359.87,499.307 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="369.789,504.465 359.711,509.306 362.29,504.346 359.87,499.307 "/> </del> <ins> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="308.892" y1="503.5" x2="362.289" y2="504.346"/> <polygon style="fill: #000000" points="369.788,504.465 359.71,509.306 362.289,504.346 359.869,499.307 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="369.788,504.465 359.71,509.306 362.289,504.346 359.869,499.307 "/> </ins> </g> <g> <del> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="318.054" y1="544.587" x2="410.665" y2="606.112"/> <polygon style="fill: #000000" points="320.821,540.422 309.725,539.053 315.287,548.752 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="320.821,540.422 309.725,539.053 315.287,548.752 "/> <polygon style="fill: #000000" points="416.912,610.263 405.816,608.894 410.665,606.112 411.35,600.564 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="416.912,610.263 405.816,608.894 410.665,606.112 411.35,600.564 "/> </del> <ins> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="318.054" y1="544.587" x2="410.664" y2="606.112"/> <polygon style="fill: #000000" points="320.821,540.422 309.725,539.053 315.288,548.752 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="320.821,540.422 309.725,539.053 315.288,548.752 "/> <polygon style="fill: #000000" points="416.911,610.263 405.815,608.894 410.664,606.112 411.349,600.564 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="416.911,610.263 405.815,608.894 410.664,606.112 411.349,600.564 "/> </ins> </g> <g> <del> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="221.611" y1="130.49" x2="223.389" y2="379.764"/> </del> <ins> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="221.612" y1="130.49" x2="223.389" y2="379.764"/> </ins> <polygon style="fill: #000000" points="223.442,387.264 218.371,377.3 223.389,379.764 228.371,377.229 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="223.442,387.264 218.371,377.3 223.389,379.764 228.371,377.229 "/> </g>

Architecture picture: says kubecfg is deprecated. The architecture picture should be updated to show kubectl instead of kubecfg.
ref /cc


Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "regexp" "sync"

This issue is to discuss an e2e testing plan for Ephemeral Containers. /sig node
As a first step towards this issue, I am planning to create node-e2e tests (under ) to verify the following: With a running pod client adds a valid EphemeralContainer resource, and the ephemeral container starts from the ephemeral container, list processes of itself and other containers, and then signal other processes (if is ) from the ephemeral container, ping & curl other containers try to read/write to stdin/stdout of the ephemeral container verify resource request/limit values against calculated expectations check oom scores of processes in the ephemeral container verify volume mounts restarts according to policy, when process exits client tries to add EphemeralContainer resource with ports (should be rejected) client tries to add EphemeralContainer resource with probes (should be rejected) client tries to add EphemeralContainer resource with resources (should be rejected) client tries to add EphemeralContainer resource with securityContext (should be rejected) removes the ephemeral container, and the ephemeral container is stopped and deleted adds multiple ephemeral containers, should all startWith pod of non-running status (TBD)
Awesome, thanks! This is a great start, here are some things to look out for: Visibility should work, but I expect signaling to fail without which is not currently allowed. iirc was interested in how this would be tested for ephemeral containers. Can you point us towards any examples that would help us get started? Ephemeral containers will never restart (and the policy should always be never) These will be denied by the api server without ever making it to the kubelet, so they are better tested in the apiserver integration tests (and I think they are already). Removing ephemeral containers is not currently allowed. /assign
Validation of ephemeralContainers is well tested in Testing of removing ephemeral containers is in ,
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .


Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "regexp" "sync"

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign /priority important-soon
Reopened this issue. <detailsIn response to : /reopen /assign /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity.


Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "regexp" "sync"

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: readonly KUBE_CONTAINER_RSYNC_PORT=8730 # These are the default versions (image tags) for their respective base images. <del> readonly __default_debian_iptables_version=bullseye-v1.4.0 </del> <ins> readonly __default_distroless_iptables_version=v0.1.1 </ins> readonly __default_go_runner_version=v2.3.1-go1.18.3-bullseye.0 readonly __default_setcap_version=bullseye-v1.3.0

Istio folks have a iptables image here that is quite small In the past, we did talk about this, for example wrote it well what we could try: We could even do for running the hack script we have for switching the symlinks when vs : kube-proxy image is definitely one of the main causes of the cost we incur in our GCR repository, so we should definitely give this a shot again. /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is a sig/network-only issue. A gives; The looks like this (narroved): If I don't misinterpret this, K8s uses a unified server-image build and it should not be different for . The size of the binary for amd64 in is; It is stripped and can't easily be "made smaller" I guess.
We (Istio) would definitely be interested in sharing ownership (hopefully by upsteaming to distroless mainline) in the image we have if there is interest. Some other projects have expressed similar needs as well
!
kube-proxy make up 9% of current image data traffic
I threw together this little repo (based on our next-gen distroless infrastructure): It pushed which on x64 is roughly another 5x smaller (musl vs. glibc): If folks are interested in this, I'd be happy to create a repo under which is where we've been standing up new images, or you can create your own from pretty easily.
debian based please!! we have KEPs and stuff written that lay down that direction:
For a possible approach + above in that issue and for "distroless" debian based images with arbitrary C/C++ binaries + deps. Note that we need things besides just the distro binaries (see comment linked). and is pretty straightforward for one way to handle the debian binaries part.
I don't even know how we build the kube-proxy image. Where does that happen and form which dockerfile?


Question: readonly KUBE_CONTAINER_RSYNC_PORT=8730 # These are the default versions (image tags) for their respective base images. <del> readonly __default_debian_iptables_version=bullseye-v1.4.0 </del> <ins> readonly __default_distroless_iptables_version=v0.1.1 </ins> readonly __default_go_runner_version=v2.3.1-go1.18.3-bullseye.0 readonly __default_setcap_version=bullseye-v1.3.0

Note we need iptables, iptables-save and iptables-restore but also ipvsadm, shell, conntrack, ipset, and maybe other tools
It used to be in kubernetes/kubernetes under build/images, it's now in (Dockerfile etc there)
That's the base - where is the logic to turn that into a kube-proxy image?
Turning that image into a kube-proxy image is just adding the kube-proxy binary with a dockerfile and entirely too much bash which ultimately just does: (except api-server, where we do a setcap in )
/triage accepted Will take a look between this and next week
It doesn't even need bash; the version of the script in intentionally avoids bash-specific features (and gets tested against Alpine where /bin/sh is busybox (which I guess is "ash"?)). And the version in kubernetes/release appears to be just a stripped-down version of that script. (But actually, you should update the script to be based on the latest version anyway, to get the new logic in , in preparation for .)
Probably per one of the things we don't want to do right now is jump into a musl based image like Alpine, right? So I would take the case that we want to use distroless, and copy all the binaries we need to inside that container (conntrack, ipvs, iptables-*, etc etc). per iptables-wrapper, you said it's not bash specific so maybe we can use distroless + the smaller available shell? I will check some approaches here before jumping into changing the release/dockerfiles/whatever scripts, and check if I can run some comformance tests in a distroless k/proxy. Maybe starting with a good binary list is a way to cross check the binaries + libraries required for this distroless container. I did a list with the installed packages per kproxy needs, and some ldd on it: Will start working on something to assemble an image just with the binaries and required libraries, and check size + libraries
I agree no musl/alpine On Sun, Apr 17, 2022, 11:38 AM Ricardo Katz
If you identify the list of binaries, has logic to handle ldd for a binary (e.g. ignore vdso) + keep the copyright and package version info via dpkg.


Question: readonly KUBE_CONTAINER_RSYNC_PORT=8730 # These are the default versions (image tags) for their respective base images. <del> readonly __default_debian_iptables_version=bullseye-v1.4.0 </del> <ins> readonly __default_distroless_iptables_version=v0.1.1 </ins> readonly __default_go_runner_version=v2.3.1-go1.18.3-bullseye.0 readonly __default_setcap_version=bullseye-v1.3.0

We probably need a few more not in the list above for the other proxy modes?
Thanks this is gonna be very helpful! I will work on it using the script you provided during the week!
/cc
From script and some changes: There's a pretty good reduction of size (50%). But this is without kproxy binary, that is almost 40Mb. I will open a Draft PR in k/release so we can discuss over there the approach. BTW, anyone knows why kube-proxy binary is that big? Never digged into that, but curious about it.
https://dr- !
(updated with conntrack-tools) Why? The binary is statically linked so it does not need or use , it only needs some binaries to be executable and it doesn't matter if they are built with musl. They just have to work. I have sucessfully built and executed my own tests, both with proxy-mode and , using an image built with this : The image size on cluster including the binary (~40MB) is; (I use a private local registry) Compare to;
what about the different iptables-nft vs iptables-legacy binaries?
I have assumed that takes care of that (executed by the Dockerfile).
Re size: Are DWARF2 symbols stripped? Ideally k8s should produce artifacts without DWARF2 symbols and distribute them separatelly. Or produce the optional -debug artifacts. This can cut GCR/GCS costs, but introduce some debug/profiling complexity. The Go symbol table should be always included. Related
IIRC wrapper in prod needs both binaries existing in the container, and will use this in a way to check if the host machine is using iptables-legacy or iptables-nft.
Both exist in Alpine; The first example in is Alpine. can you please confirm that it works with Alpine?
I have altered my testenv so I can switch between legacy and nft and verified that the tests work with both and that shows the same in the container and the K8s node, e.g;
Side remark; to my suprise is not needed in the kube-proxy image
Tested the with K8s e2e;
There are / we're legal reasons to avoid Alpine. Have those been resolved? I haven't heard anything.


Question: readonly KUBE_CONTAINER_RSYNC_PORT=8730 # These are the default versions (image tags) for their respective base images. <del> readonly __default_debian_iptables_version=bullseye-v1.4.0 </del> <ins> readonly __default_distroless_iptables_version=v0.1.1 </ins> readonly __default_go_runner_version=v2.3.1-go1.18.3-bullseye.0 readonly __default_setcap_version=bullseye-v1.3.0

On Tue, Apr 19, 2022, 11:31 AM Lars Ekman
i don't think so. it's status quo on Alpine which is to avoid it totally.
Can I read about those somewhere? Are the legal reasons about Alpine? Or about musl?
Not my personal decision, and I'm not a lawyer let alone an OSS license expert but ... sharing what I've heard on the subject: IIRC for example Alpine does not make available the source code (AFAICT Alpine publishes source code for the packaging scripts, but not the package sources), or even the LICENSE of binary packages, which is problematic for compliance with licenses ... (Various common license require including the license text, and / or making source code available). Normal alpine images don't appear to install these doc packages: So those images don't even ship with license / copyright info files for packages which is ... not a great start. I'm not sure where a good public reference is for this, Google OSPO has some relevant docs internally about following license requirements when producing container images. So far the best public doc on the topic I've found is
Thanks!
TL:DR;
I wasn't suggesting switching to Alpine, I was just saying that I unit test the iptables-wrappers scripts against Alpine, which doesn't use bash, and so I know the scripts don't require bash. Yeah, I don't know the details of how distroless works, but if we're talking about debian packages then it looks like is 1/10 the size of (212k vs 2285k)
In Alpine (and Debian, and probably most distros other than Fedora), the "iptables" package includes both the legacy and nft binaries, so that's the only dependency.
Made a comment on bandwidth that may be better in this issue rahter than the PR;
Want to give a quick update on this (EDIT: I wrote this was going to be quick update, but ended up not being! sorry!): The image generation is working "fine", see: This process generates a temporary that can be used later with and/or changed in Kubernetes code to trigger the tests (see ) Due to the way iptables-wrapper-installer works, it calls iptables-legacy and iptables-nft to check if they are working fine.


Question: readonly KUBE_CONTAINER_RSYNC_PORT=8730 # These are the default versions (image tags) for their respective base images. <del> readonly __default_debian_iptables_version=bullseye-v1.4.0 </del> <ins> readonly __default_distroless_iptables_version=v0.1.1 </ins> readonly __default_go_runner_version=v2.3.1-go1.18.3-bullseye.0 readonly __default_setcap_version=bullseye-v1.3.0

We need to skip it (via --skip-integrity-check) in arch not amd64, as those are really specific kernel features and when called from arm64, as an example the command ends in an error. We need /bin/sleep to be present in distroless as well, due specifically to this image being used in this test: cloud-build timeout needs to be increased, it takes a long time for arm/s390/etc to build this The image can be optimized a bit more, removing some steps like linking the iptables binaries (as pointed by and removing some binaries like wc (as pointed by ) but right now I didn't wanted to change the image build AND the iptables-wrapper script. My take here would be: let's see now with sig-release if the build process is fine, get this image promoted and used before v1.25 All the tests but verify ran fine (verify due to dependency check, etc etc) - Here are the main tests with the distroless image: kind - kind ipv6 - gce ipvs - performance 100 nodes - fyi. I can bring this to next sig-net meeting, wanted to check: from sig/release perspective, is the PR fine? I will make some more changes here, but wanted even to check if this should be called debian-iptables:distroless or something like from sig/net if we want to move forward migrating the iptables-wrapper to shell script this release or not. I tend of agree with Rob that maybe we are getting too many changes in one release, and we can plan this for v1.26 (the PR is already there, I need to check Dan's comments only) Thanks
/assign
/reopen let's close it once we switch to the new image
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, Could you suggest if it's ok to use already? That would be super cool to use the image as a base for NodeLocalDNS (as it uses iptables). I already did manual smoke test in cluster - all is working correctly. Thanks!


Question: readonly KUBE_CONTAINER_RSYNC_PORT=8730 # These are the default versions (image tags) for their respective base images. <del> readonly __default_debian_iptables_version=bullseye-v1.4.0 </del> <ins> readonly __default_distroless_iptables_version=v0.1.1 </ins> readonly __default_go_runner_version=v2.3.1-go1.18.3-bullseye.0 readonly __default_setcap_version=bullseye-v1.3.0

Let's introduce it gradually, I suggest to wait until the image is promoted for kube-proxy
I've lost track of this, what do we need to do to promote it for kube-proxy?
I think Ricardo was on it But if someone is busy and others have time, let's sync next monday on slack and organize the work, so someone can get this to the finish line, there is no need to overload one person if there are other wanting to help
I didn’t saw there was a comment there. I can fix that, and wanted to plan with you folks next steps. I agree with Antonio and don’t want to block this on my availability :) I will check suggestion tomorrow and as we get this merged I’m more than happy to handle to someone else! On Sat, 16 Jul 2022 at 07:21 Antonio Ojea
I've opened a new PR to do the proper promotion. As soon as this is promoted I think we can start testing and migrating to distroless :) From here I think I can handle to someone else as I wont have much time and attention to this specific effort and I know this is important :) I'm still willing to migrate iptables-wrapper to go code (we started it in kubernetes-sigs/iptables-wrapper) so I will probably finish fixing Dan's reviews and we can discuss the distroless v0.2.0 being based entirely in Go :)
we should probably nudge a few of the subprojects to migrate, we just migrated KIND but I'm sure we have at least a few more out there like ip-masq-agent
I'm migrating nodelocaldns. I've got a couple of questions: nodelocaldns uses registry . Should I switch it to ? Is considered sort of a prod image registry? a multi-arch image? Because currently we use is it better to keep up with the new tags for ? So that we could quickly upgrade base image version as well? Thanks for the work!
YES! Make happy! This registry is actually a "proxy" that sig-k8s-infra is maintaining so people get directed to the closest Image registry, based on their cloud provider. As an example, with this users in AWS wont need to go to GCR to pull their image Yes, it is. It follows the same build pattern of debian-iptables. See Well, this is a good question.


Question: readonly KUBE_CONTAINER_RSYNC_PORT=8730 # These are the default versions (image tags) for their respective base images. <del> readonly __default_debian_iptables_version=bullseye-v1.4.0 </del> <ins> readonly __default_distroless_iptables_version=v0.1.1 </ins> readonly __default_go_runner_version=v2.3.1-go1.18.3-bullseye.0 readonly __default_setcap_version=bullseye-v1.3.0

AFAIK we don't have an event / notification system that sends a message (in Slack, as an example) whenever a base image gets promoted. It would be cool to have at least for debian-base, debian-iptables and distroless-iptables, probably it's worth to question sig-release on this :) ( maybe know)
Thanks for the answer! Regarding the - we use it as a compile image for dnsmasq. Will this image still be supported or should I find something other to replace this compile image?
Short answer: I imagine/suppose there is no deprecation plan for this image right now, as it is used as the compilation base for a bunch of other things (and it is base of etcd image) Long answer: In case of distroless-iptables the rationale was: we need to reduce costs (and by reducing costs, reducing image size) we need to use something with a smaller footprint and reduce the amount of packages, and as a result the amount of CVEs in image package. kube-proxy exists in all nodes (unless you use a CNI that implements its features) so it is a good candidate I guess right now debian-base will still be used (I don't remember on the top of my head if other kubernetes components are already running using distroless or compiled based on debian-base, as far as I can see at least etcd image depends on debian-base () so probably any effort on this image should be announced.


Question: func tabbedString(f func(io.Writer) error) (string, error) { out := new(tabwriter.Writer) <del> b := make([]byte, 1024) buf := bytes.NewBuffer(b) </del> <ins> buf := &bytes.Buffer{} </ins> out.Init(buf, 0, 8, 1, 't', 0) err := f(out)

kubectl describe emits 1024 NULL characters prior to actual text. This prevents all text-based processing tools from working on its output. /cc
This has got to be due to:
(IRL discussion: there's no reason to pass 1024 0 bytes to NewBuffer.)
Fixing in PR .


Question: Pod:        newNamedPod("5-static", "test1", "pod1", true), UpdateType: kubetypes.SyncPodUpdate, }) <ins> // Wait for the previous work to be delivered to the worker drainAllWorkers(podWorkers) </ins> channels.Channel("5-static").Hold() podWorkers.UpdatePod(UpdatePodOptions{ Pod:        newNamedPod("5-static", "test1", "pod1", true),

/kind failing-test <!-- If this is a flake, please add: /kind flake --/kind flake the test case was add in /cc smarterclayton <!-- Please assign a SIG using: /sig SIG-NAME --/sig node
/assign
bumping priority, seeing this in PR presubmit runs
/triage accepted


Question: package dockershim import ( <ins> "errors" </ins> "fmt" <ins> "io" </ins> "regexp" "strconv" "strings" <ins> "sync/atomic" </ins> dockertypes "github.com/docker/docker/api/types" dockercontainer "github.com/docker/docker/api/types/container"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Conformance test failed: xref What you expected to happen: I expected this conformance test to pass. How to reproduce it (as minimally and precisely as possible): We have configured docker's logging driver to use journald, and then ran sonobouy tests against a k8s 1.16.3 cluster. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Red Hat Enterprise Linux Server release 7.7 (Maipo) Kernel (e.g. ): 5.3.11- SMP Tue Nov 12 14:45:59 EST 2019 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig cli
/assign
I have a working patch that fixes this, will push a PR shortly.


Question: defaultLabels := map[string]string{"foo": "bar"} maxUnavailable := intstr.FromInt(1) period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug The change in a new field () to the Pod spec that is set to true by default. This is causing problems for DaemonSets in some situations. DaemonSets are using ControllerRevisions to keep track of previous configurations and each revision will contain among other things the Pod spec for the DaemonSet. If a DaemonSet is created with the enableServiceLinks property is set to true, the ControllerRevision will also contain this field. If the cluster is rolled back to a previous version of kubernetes, the DeamonSet struct does not have this field and so the ControllerRevision and the DaemonSet will no longer match. This leads to a new ControllerRevision being created and a rolling update of all pods in the DaemonSet. Would it be a better solution to rename the field to and enable service links whenever the field is not set? This would fix the problem with the DaemonSets unless the field is explicitly set before the rollback. This is causing the master upgrade tests to fail as reported in /sig apps /cc /cc /cc
I guess the change in podSpec and the defaulting will break all the workload resources as they are based on comparing the template's podSpec. ~Is the new value defaulted to true also for the DaemonSet's podSpec template? If not, controller might go hot loop like with mutating admission controllers.~ I don't like us making such changes to v1 podSpec. The other way around seems more sensible. /cc
Why is this a bad thing? We will continue to add fields to podSpec - the issue isn't the defaulting (this would happen regardless of which way it defaulted), it's that the controller can't really be version aware. The new value would default to true in the daemonset as well.
I'm not sure that a rollback requiring daemonsets to recreate is a bug. I think that's potentially "the best possible thing that we can do in that case".
/cc SIG Release blocking dashboard with upgrade test failures - https://k8s-
So the problem isn't the fact that a field has been . It the way in which the field was .


Question: defaultLabels := map[string]string{"foo": "bar"} maxUnavailable := intstr.FromInt(1) period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either ca So the problem isn't the fact that a field has been . It the way in which the field was . enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either case), you won't trigger the rolling restart on rollback or controller manager failoverse), you won't trigger the rolling restart on rollback or controller manager failover unless you explicitly opt out of the behavior. I don't see why the former behavior is more desirable than the latter.
For background/context, the choice between enable/disable was initially discussed here:
Are you saying that the issue is that default value is not the same as the Go "zero value" ? The concept of zero value is interesting but it forces APIs to bend into unnatural shapes to respect it. Since forever Kubernetes has explicitly said that we have version-specific default values which ARE NOT governed by zero values, but instead are governed by the API version. It is safe to assume that if a field is documented as having a default in a version, you can rely on that field being populated with that default, with a few exceptions.


Question: defaultLabels := map[string]string{"foo": "bar"} maxUnavailable := intstr.FromInt(1) period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

We generally ask for boolean fields to be "in the affirmative direction", as in this case. Now, we could revisit this. It would be a significant change in direction, but it's not impossible. But I don't think that solves your problem in general -- if this field were optional string, with a default value of "Hello there", you would be in the same boat right?
So the underlying issue is that we are defaulting into the PodSpec which gets adopted into all of the PodTemplateSpecs of every object. That sort of defaults the users declared intent. Why not just default the value into the Pod itself . Is the intention really to default this in all Templates or to make it optional in the Template and default it in the Pod itself?
/milestone v1.13
I can't speak to the ideology of whether to default the field in all PodSpecs or just the Pod, but your suggestion makes sense to me. I'm not sure of the past reasoning for adding other defaulted fields, most recently SchedulerName in Jan. 2017 (), to all PodSpecs instead of just to the Pod object. Did we not have these same tests in place then? This would effectively be saying we should never add new defaulted fields to PodSpec. If this is the plan, I can create a PR for the fix and I'll add a big warning comment to SetDefaults_PodSpec.
As an example of where we don't default the template we have a CronJob that embeds JobTemplate, we never default that template, only an actual Job gets defaulting. I agree with that not defaulting any templates does make sense.
Agree to only default this in pods. Agree to adding the comment. Most important defaulting happens in pods to keep higher level pod specs simpler. We may need to raise the general pattern in a discussion with api review in general, since this is a change from previous strategy, but the previous strategy was not explicitly chosen (to my recollection at least)
Hi all do we have an agreed path forward for fixing this issue in 1.13? if so can we have an ETA for the fix to land? the release team would like for the tests to get passing in the blocking suites in an effort to unmask any other failures that might show up once these are fixed.


Question: defaultLabels := map[string]string{"foo": "bar"} maxUnavailable := intstr.FromInt(1) period := int64(v1.DefaultTerminationGracePeriodSeconds) <del> enableServiceLinks := v1.DefaultEnableServiceLinks </del> defaultTemplate := v1.PodTemplateSpec{ Spec: v1.PodSpec{ DNSPolicy:                     v1.DNSClusterFirst,

/assign
We have a PR with a fix that will hopefully merge today or tomorrow. I will keep an eye on the dashboard to make sure it fixes the failing daemonset tests.
Thanks Can we link the PR here plz. /cc
Reopening, because of the auto-close. Tests are not passing yet. /reopen
Reopening this issue. <detailsIn response to : Reopening, because of the auto-close. Tests are not passing yet. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
PR that should (hopefully) close this issue is here:
DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- if it doesn't pop up again - /close
Closing this issue. <detailsIn response to : DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: echo "COPY nsswitch.conf /etc/" >> "${docker_file_path}" fi <del> "${DOCKER[@]}" build ${docker_build_opts:+"${docker_build_opts}"} -q -t "${docker_image_tag}" "${docker_build_path}" >/dev/null </del> <ins> if [[ "${arch}" == "linux/s390x" || "${arch}" == "linux/ppc64le" ]]; then DOCKER_CLI_EXPERIMENTAL=enabled "${DOCKER[@]}" buildx build --platform linux/"${arch}" --load ${docker_build_opts:+"${docker_build_opts}"} -q -t "${docker_image_tag}" "${docker_build_path}" >/dev/null else "${DOCKER[@]}" build ${docker_build_opts:+"${docker_build_opts}"} -q -t "${docker_image_tag}" "${docker_build_path}" >/dev/null fi </ins> # If we are building an official/alpha/beta release we want to keep # docker images and tag them appropriately. local -r release_docker_image_tag="${KUBE_DOCKER_REGISTRY-$docker_registry}/${binary_name}-${arch}:${KUBE_DOCKER_IMAGE_TAG-$docker_tag}"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When run below commands against master branch, only amd64 images are built. What you expected to happen: Images should be built successfully according to values set in . How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release


Question: client := clientValue.Load().(*clientv3.Client) ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) defer cancel() <del> if _, err := client.Cluster.MemberList(ctx); err != nil { return fmt.Errorf("error listing etcd members: %v", err) </del> <ins> // See https://github.com/etcd-io/etcd/blob/master/etcdctl/ctlv3/command/ep_command.go#L118 _, err := client.Get(ctx, path.Join(c.Prefix, "health")) if err == nil { return nil </ins> } <del> return nil </del> <ins> return fmt.Errorf("error getting data from etcd: %v", err) </ins> }, nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: I'm running a K8s 1.13 cluster and a partial netsplit happened due to a defective linecard dropping only TCP packets above a certain sizes between two masters, but the third could still communicate with both (so the fault made the network non-transitive). etcd was run on the physical network, but Kubernetes traffic runs on a Wireguard overlay for security and routing is provided by OSPF (so internal communication was not affected since it was tunelled over UDP). Etcd experienced some temporary issues due to the connections dropping only certain packets (above ~1000 bytes), but quickly dropped one host out of the quorum, at which point it was degraded but all the remaining hosts could communicate freely. Each master node has an etcd instance and an apiserver. All apiservers are anycasted inside the overlay to balance the load and provide redundancy and the announcement is coupled to the health check, so in case any apiserver fails no traffic will be routed to it. The problem was that the apiserver did not fail its healthcheck even though all etcd write requests timed out. The non-working apiserver was still accessible and got routed its share of the anycast traffic since that was routed over the UDP overlay. Thus ~1/3 of all workloads and nodes were stuck on a non-working apiserver until I nuked the non-working apiserver. TLDR: One etcd server started timing out all writes due to a non-transitive netsplit and the Kubernetes apiserver healthcheck did not fail, causing 1/3 of K8s clients to not have a working control plane until manual intervention. What you expected to happen: The health check should've failed (in my opinion, I'm happy to discuss if you think otherwise). How to reproduce it (as minimally and precisely as possible): Set up 3 etcds with corresponding apiservers, introduce a split between two (shouldn't have to be a packet size limit, a full disconnect should reproduce cleanly), observe that all three report healthy even though at least one of them no longer works. Anything else we need to know?


Question: client := clientValue.Load().(*clientv3.Client) ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) defer cancel() <del> if _, err := client.Cluster.MemberList(ctx); err != nil { return fmt.Errorf("error listing etcd members: %v", err) </del> <ins> // See https://github.com/etcd-io/etcd/blob/master/etcdctl/ctlv3/command/ep_command.go#L118 _, err := client.Get(ctx, path.Join(c.Prefix, "health")) if err == nil { return nil </ins> } <del> return nil </del> <ins> return fmt.Errorf("error getting data from etcd: %v", err) </ins> }, nil }

: Environment: Kubernetes version (use ): v1.13.0 Cloud provider or hardware configuration: Physical x86_64 servers OS (e.g. from /etc/os-release): Custom Kernel (e.g. ): Custom, based on 4.19. No patches affect this functionality. Install tools: Custom Others:
/sig api-machinery (maybe cluster-ops?)
cc
Looks like the etcdctl endpoint health command does a random read after fetching the list of members - - we should probably do the same?
Looks sane and would have prevented my issue. Should I PR this?
sure. thanks
/assign


Question: fi; # BEGIN_PROMETHEUS_TO_SD - name: prometheus-to-sd-exporter <del> image: gcr.io/google-containers/prometheus-to-sd:v0.1.3 </del> <ins> image: gcr.io/google-containers/prometheus-to-sd:v0.2.2 </ins> command: - /monitor <del> - --component=fluentd - --target-port=31337 </del> - --stackdriver-prefix={{ prometheus_to_sd_prefix }}/addons - --api-override={{ prometheus_to_sd_endpoint }} <del> - --whitelisted-metrics=stackdriver_successful_requests_count,stackdriver_failed_requests_count,stackdriver_ingested_entries_count,stackdriver_dropped_entries_count </del> <ins> - --source=fluentd:http://localhost:31337?whitelisted=stackdriver_successful_requests_count,stackdriver_failed_requests_count,stackdriver_ingested_entries_count,stackdriver_dropped_entries_count - --pod-id=$(POD_NAME) - --namespace-id=$(POD_NAMESPACE) env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace </ins> # END_PROMETHEUS_TO_SD nodeSelector: beta.kubernetes.io/fluentd-ds-ready: "true"

Currently there are several, and some of them have problems with excessive logging could you please do this?
Any estimate on this fix? Asking because I'm seeing the issue referenced in
It was already fixed in head and in 1.8 long time ago. Fix was merged to 1.7 recently, so we are just waiting for new patch release.


Question: verifyMetricCount(storageOpMetrics, updatedStorageMetrics, volumeOp) } }) <ins> It("should create volume metrics with the correct PVC ref", func() { var err error pvc, err = c.CoreV1().PersistentVolumeClaims(pvc.Namespace).Create(pvc) Expect(err).NotTo(HaveOccurred()) Expect(pvc).ToNot(Equal(nil)) claims := []*v1.PersistentVolumeClaim{pvc} pod := framework.MakePod(ns, claims, false, "") pod, err = c.CoreV1().Pods(ns).Create(pod) Expect(err).NotTo(HaveOccurred()) err = framework.WaitForPodRunningInNamespace(c, pod) framework.ExpectNoError(framework.WaitForPodRunningInNamespace(c, pod), "Error starting pod ", pod.Name) pod, err = c.CoreV1().Pods(ns).Get(pod.Name, metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred()) // Wait for `VolumeStatsAggPeriod' to grab metrics time.Sleep(1 * time.Minute) // Grab kubelet metrics from the node the pod was scheduled on kubeMetrics, err := metricsGrabber.GrabFromKubelet(pod.Spec.NodeName) Expect(err).NotTo(HaveOccurred(), "Error getting kubelet metrics : %v", err) framework.Logf("Deleting pod %q/%q", pod.Namespace, pod.Name) framework.ExpectNoError(framework.DeletePodWithWait(f, c, pod)) // Verify volume stat metrics were collected for the referenced PVC volumeStatKeys := []string{ kubeletmetrics.VolumeStatsUsedBytesKey, kubeletmetrics.VolumeStatsCapacityBytesKey, kubeletmetrics.VolumeStatsAvailableBytesKey, kubeletmetrics.VolumeStatsUsedBytesKey, kubeletmetrics.VolumeStatsInodesFreeKey, kubeletmetrics.VolumeStatsInodesUsedKey, } for _, key := range volumeStatKeys { kubeletKeyName := fmt.Sprintf("%s_%s", kubeletmetrics.KubeletSubsystem, key) verifyVolumeStatMetric(kubeletKeyName, pvc.Namespace, pvc.Name, kubeMetrics) } }) </ins> }) func verifyMetricCount(oldMetrics map[string]int64, newMetrics map[string]int64, metricName string) {

In 1.8, we a new feature to expose PVC information through summary API so user can check the volume belongs to which PVC. We should add e2e test in under test/e2enode.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
GitHub didn't allow me to assign the following users: vkamra. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Can we please move this out of 1.8 milestone?
- I'm looking into adding this but yes we should move this out of 1.8. I don't have permissions to do that though.
ack Can we please move this out of 1.8?
Sorry am not sure why was this removed from 1.8 milestone. Are we not aiming to add the tests in 1.8? Or Are we disabling the feature or are we fine shipping it without any e2e tests in 1.8?
The test is in . I think we can keep this feature in 1.8


Question: readonly KUBE_APISERVER_BASE_IMAGE="${KUBE_APISERVER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" readonly KUBE_CONTROLLER_MANAGER_BASE_IMAGE="${KUBE_CONTROLLER_MANAGER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" readonly KUBE_SCHEDULER_BASE_IMAGE="${KUBE_SCHEDULER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" <del> readonly KUBE_PROXY_BASE_IMAGE="${KUBE_PROXY_BASE_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/debian-iptables:$__default_debian_iptables_version}" </del> <ins> readonly KUBE_PROXY_BASE_IMAGE="${KUBE_PROXY_BASE_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/distroless-iptables:$__default_distroless_iptables_version}" </ins> # This is the image used in a multi-stage build to apply capabilities to Docker-wrapped binaries. readonly KUBE_BUILD_SETCAP_IMAGE="${KUBE_BUILD_SETCAP_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/setcap:$__default_setcap_version}"

Istio folks have a iptables image here that is quite small In the past, we did talk about this, for example wrote it well what we could try: We could even do for running the hack script we have for switching the symlinks when vs : kube-proxy image is definitely one of the main causes of the cost we incur in our GCR repository, so we should definitely give this a shot again. /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is a sig/network-only issue. A gives; The looks like this (narroved): If I don't misinterpret this, K8s uses a unified server-image build and it should not be different for . The size of the binary for amd64 in is; It is stripped and can't easily be "made smaller" I guess.
We (Istio) would definitely be interested in sharing ownership (hopefully by upsteaming to distroless mainline) in the image we have if there is interest. Some other projects have expressed similar needs as well
!
kube-proxy make up 9% of current image data traffic
I threw together this little repo (based on our next-gen distroless infrastructure): It pushed which on x64 is roughly another 5x smaller (musl vs. glibc): If folks are interested in this, I'd be happy to create a repo under which is where we've been standing up new images, or you can create your own from pretty easily.
debian based please!! we have KEPs and stuff written that lay down that direction:
For a possible approach + above in that issue and for "distroless" debian based images with arbitrary C/C++ binaries + deps. Note that we need things besides just the distro binaries (see comment linked). and is pretty straightforward for one way to handle the debian binaries part.
I don't even know how we build the kube-proxy image. Where does that happen and form which dockerfile?


Question: readonly KUBE_APISERVER_BASE_IMAGE="${KUBE_APISERVER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" readonly KUBE_CONTROLLER_MANAGER_BASE_IMAGE="${KUBE_CONTROLLER_MANAGER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" readonly KUBE_SCHEDULER_BASE_IMAGE="${KUBE_SCHEDULER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" <del> readonly KUBE_PROXY_BASE_IMAGE="${KUBE_PROXY_BASE_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/debian-iptables:$__default_debian_iptables_version}" </del> <ins> readonly KUBE_PROXY_BASE_IMAGE="${KUBE_PROXY_BASE_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/distroless-iptables:$__default_distroless_iptables_version}" </ins> # This is the image used in a multi-stage build to apply capabilities to Docker-wrapped binaries. readonly KUBE_BUILD_SETCAP_IMAGE="${KUBE_BUILD_SETCAP_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/setcap:$__default_setcap_version}"

Note we need iptables, iptables-save and iptables-restore but also ipvsadm, shell, conntrack, ipset, and maybe other tools
It used to be in kubernetes/kubernetes under build/images, it's now in (Dockerfile etc there)
That's the base - where is the logic to turn that into a kube-proxy image?
Turning that image into a kube-proxy image is just adding the kube-proxy binary with a dockerfile and entirely too much bash which ultimately just does: (except api-server, where we do a setcap in )
/triage accepted Will take a look between this and next week
It doesn't even need bash; the version of the script in intentionally avoids bash-specific features (and gets tested against Alpine where /bin/sh is busybox (which I guess is "ash"?)). And the version in kubernetes/release appears to be just a stripped-down version of that script. (But actually, you should update the script to be based on the latest version anyway, to get the new logic in , in preparation for .)
Probably per one of the things we don't want to do right now is jump into a musl based image like Alpine, right? So I would take the case that we want to use distroless, and copy all the binaries we need to inside that container (conntrack, ipvs, iptables-*, etc etc). per iptables-wrapper, you said it's not bash specific so maybe we can use distroless + the smaller available shell? I will check some approaches here before jumping into changing the release/dockerfiles/whatever scripts, and check if I can run some comformance tests in a distroless k/proxy. Maybe starting with a good binary list is a way to cross check the binaries + libraries required for this distroless container. I did a list with the installed packages per kproxy needs, and some ldd on it: Will start working on something to assemble an image just with the binaries and required libraries, and check size + libraries
I agree no musl/alpine On Sun, Apr 17, 2022, 11:38 AM Ricardo Katz
If you identify the list of binaries, has logic to handle ldd for a binary (e.g. ignore vdso) + keep the copyright and package version info via dpkg.


Question: readonly KUBE_APISERVER_BASE_IMAGE="${KUBE_APISERVER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" readonly KUBE_CONTROLLER_MANAGER_BASE_IMAGE="${KUBE_CONTROLLER_MANAGER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" readonly KUBE_SCHEDULER_BASE_IMAGE="${KUBE_SCHEDULER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" <del> readonly KUBE_PROXY_BASE_IMAGE="${KUBE_PROXY_BASE_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/debian-iptables:$__default_debian_iptables_version}" </del> <ins> readonly KUBE_PROXY_BASE_IMAGE="${KUBE_PROXY_BASE_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/distroless-iptables:$__default_distroless_iptables_version}" </ins> # This is the image used in a multi-stage build to apply capabilities to Docker-wrapped binaries. readonly KUBE_BUILD_SETCAP_IMAGE="${KUBE_BUILD_SETCAP_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/setcap:$__default_setcap_version}"

We probably need a few more not in the list above for the other proxy modes?
Thanks this is gonna be very helpful! I will work on it using the script you provided during the week!
/cc
From script and some changes: There's a pretty good reduction of size (50%). But this is without kproxy binary, that is almost 40Mb. I will open a Draft PR in k/release so we can discuss over there the approach. BTW, anyone knows why kube-proxy binary is that big? Never digged into that, but curious about it.
https://dr- !
(updated with conntrack-tools) Why? The binary is statically linked so it does not need or use , it only needs some binaries to be executable and it doesn't matter if they are built with musl. They just have to work. I have sucessfully built and executed my own tests, both with proxy-mode and , using an image built with this : The image size on cluster including the binary (~40MB) is; (I use a private local registry) Compare to;
what about the different iptables-nft vs iptables-legacy binaries?
I have assumed that takes care of that (executed by the Dockerfile).
Re size: Are DWARF2 symbols stripped? Ideally k8s should produce artifacts without DWARF2 symbols and distribute them separatelly. Or produce the optional -debug artifacts. This can cut GCR/GCS costs, but introduce some debug/profiling complexity. The Go symbol table should be always included. Related
IIRC wrapper in prod needs both binaries existing in the container, and will use this in a way to check if the host machine is using iptables-legacy or iptables-nft.
Both exist in Alpine; The first example in is Alpine. can you please confirm that it works with Alpine?
I have altered my testenv so I can switch between legacy and nft and verified that the tests work with both and that shows the same in the container and the K8s node, e.g;
Side remark; to my suprise is not needed in the kube-proxy image
Tested the with K8s e2e;
There are / we're legal reasons to avoid Alpine. Have those been resolved? I haven't heard anything.


Question: readonly KUBE_APISERVER_BASE_IMAGE="${KUBE_APISERVER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" readonly KUBE_CONTROLLER_MANAGER_BASE_IMAGE="${KUBE_CONTROLLER_MANAGER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" readonly KUBE_SCHEDULER_BASE_IMAGE="${KUBE_SCHEDULER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" <del> readonly KUBE_PROXY_BASE_IMAGE="${KUBE_PROXY_BASE_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/debian-iptables:$__default_debian_iptables_version}" </del> <ins> readonly KUBE_PROXY_BASE_IMAGE="${KUBE_PROXY_BASE_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/distroless-iptables:$__default_distroless_iptables_version}" </ins> # This is the image used in a multi-stage build to apply capabilities to Docker-wrapped binaries. readonly KUBE_BUILD_SETCAP_IMAGE="${KUBE_BUILD_SETCAP_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/setcap:$__default_setcap_version}"

On Tue, Apr 19, 2022, 11:31 AM Lars Ekman
i don't think so. it's status quo on Alpine which is to avoid it totally.
Can I read about those somewhere? Are the legal reasons about Alpine? Or about musl?
Not my personal decision, and I'm not a lawyer let alone an OSS license expert but ... sharing what I've heard on the subject: IIRC for example Alpine does not make available the source code (AFAICT Alpine publishes source code for the packaging scripts, but not the package sources), or even the LICENSE of binary packages, which is problematic for compliance with licenses ... (Various common license require including the license text, and / or making source code available). Normal alpine images don't appear to install these doc packages: So those images don't even ship with license / copyright info files for packages which is ... not a great start. I'm not sure where a good public reference is for this, Google OSPO has some relevant docs internally about following license requirements when producing container images. So far the best public doc on the topic I've found is
Thanks!
TL:DR;
I wasn't suggesting switching to Alpine, I was just saying that I unit test the iptables-wrappers scripts against Alpine, which doesn't use bash, and so I know the scripts don't require bash. Yeah, I don't know the details of how distroless works, but if we're talking about debian packages then it looks like is 1/10 the size of (212k vs 2285k)
In Alpine (and Debian, and probably most distros other than Fedora), the "iptables" package includes both the legacy and nft binaries, so that's the only dependency.
Made a comment on bandwidth that may be better in this issue rahter than the PR;
Want to give a quick update on this (EDIT: I wrote this was going to be quick update, but ended up not being! sorry!): The image generation is working "fine", see: This process generates a temporary that can be used later with and/or changed in Kubernetes code to trigger the tests (see ) Due to the way iptables-wrapper-installer works, it calls iptables-legacy and iptables-nft to check if they are working fine.


Question: readonly KUBE_APISERVER_BASE_IMAGE="${KUBE_APISERVER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" readonly KUBE_CONTROLLER_MANAGER_BASE_IMAGE="${KUBE_CONTROLLER_MANAGER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" readonly KUBE_SCHEDULER_BASE_IMAGE="${KUBE_SCHEDULER_BASE_IMAGE:-$KUBE_GORUNNER_IMAGE}" <del> readonly KUBE_PROXY_BASE_IMAGE="${KUBE_PROXY_BASE_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/debian-iptables:$__default_debian_iptables_version}" </del> <ins> readonly KUBE_PROXY_BASE_IMAGE="${KUBE_PROXY_BASE_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/distroless-iptables:$__default_distroless_iptables_version}" </ins> # This is the image used in a multi-stage build to apply capabilities to Docker-wrapped binaries. readonly KUBE_BUILD_SETCAP_IMAGE="${KUBE_BUILD_SETCAP_IMAGE:-$KUBE_BASE_IMAGE_REGISTRY/setcap:$__default_setcap_version}"

We need to skip it (via --skip-integrity-check) in arch not amd64, as those are really specific kernel features and when called from arm64, as an example the command ends in an error. We need /bin/sleep to be present in distroless as well, due specifically to this image being used in this test: cloud-build timeout needs to be increased, it takes a long time for arm/s390/etc to build this The image can be optimized a bit more, removing some steps like linking the iptables binaries (as pointed by and removing some binaries like wc (as pointed by ) but right now I didn't wanted to change the image build AND the iptables-wrapper script. My take here would be: let's see now with sig-release if the build process is fine, get this image promoted and used before v1.25 All the tests but verify ran fine (verify due to dependency check, etc etc) - Here are the main tests with the distroless image: kind - kind ipv6 - gce ipvs - performance 100 nodes - fyi. I can bring this to next sig-net meeting, wanted to check: from sig/release perspective, is the PR fine? I will make some more changes here, but wanted even to check if this should be called debian-iptables:distroless or something like from sig/net if we want to move forward migrating the iptables-wrapper to shell script this release or not. I tend of agree with Rob that maybe we are getting too many changes in one release, and we can plan this for v1.26 (the PR is already there, I need to check Dan's comments only) Thanks
/assign
/reopen let's close it once we switch to the new image
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, Could you suggest if it's ok to use already? That would be super cool to use the image as a base for NodeLocalDNS (as it uses iptables). I already did manual smoke test in cluster - all is working correctly. Thanks!