Question: procGetNetworkParams = iphlpapidll.MustFindProc("GetNetworkParams") ) <ins> func fileExists(filename string) (bool, error) { stat, err := os.Stat(filename) if os.IsNotExist(err) { return false, nil } if err != nil { return false, err } return stat.Mode().IsRegular(), nil } </ins> func getHostDNSConfig(resolverConfig string) (*runtimeapi.DNSConfig, error) { <del> if resolverConfig != "" && resolverConfig != hostResolvConf { err := fmt.Errorf(`Unexpected resolver config value: "%s". Expected "" or "%s".`, resolverConfig, hostResolvConf) </del> <ins> if resolverConfig == "" { // This handles "" by returning defaults. return getDNSConfig(resolverConfig) } isFile, err := fileExists(resolverConfig) if err != nil { err = fmt.Errorf(`Unexpected error while getting os.Stat for "%s" resolver config. Error: %w`, resolverConfig, err) </ins> klog.ErrorS(err, "Cannot get host DNS Configuration.") return nil, err } <ins> if isFile { // Get the DNS config from a resolv.conf-like file. return getDNSConfig(resolverConfig) } </ins> <del> var ( hostDNS, hostSearch []string err                 error ) </del> <ins> if resolverConfig != hostResolvConf { err := fmt.Errorf(`Unexpected resolver config value: "%s". Expected "", "%s", or a path to an existing resolv.conf file.`, resolverConfig, hostResolvConf) klog.ErrorS(err, "Cannot get host DNS Configuration.") return nil, err } // If we get here, the resolverConfig == hostResolvConf and that is not actually a file, so // it means to use the host settings. </ins> // Get host DNS settings <del> if resolverConfig == hostResolvConf { hostDNS, err = getDNSServerList() if err != nil { err = fmt.Errorf("Could not get the host's DNS Server List. Error: %w", err) klog.ErrorS(err, "Encountered error while getting host's DNS Server List.") return nil, err } hostSearch, err = getDNSSuffixList() if err != nil { err = fmt.Errorf("Could not get the host's DNS Suffix List. Error: %w", err) klog.ErrorS(err, "Encountered error while getting host's DNS Suffix List.") return nil, err } </del> <ins> hostDNS, err := getDNSServerList() if err != nil { err = fmt.Errorf("Could not get the host's DNS Server List. Error: %w", err) klog.ErrorS(err, "Encountered error while getting host's DNS Server List.") return nil, err } hostSearch, err := getDNSSuffixList() if err != nil { err = fmt.Errorf("Could not get the host's DNS Suffix List. Error: %w", err) klog.ErrorS(err, "Encountered error while getting host's DNS Suffix List.") return nil, err </ins> } return &runtimeapi.DNSConfig{ Servers:  hostDNS,

When a --resolv-conf (DNS resolver config) is passed to kubelet on a Windows node, the config content is used as part of sandbox initialization (through getHostDNSConfig()). Now with the recent changes (), it seems now to fail with an error like: E0315 10:22:16. 4764 ] "Error syncing pod, skipping" err="failed to "CreatePodSandbox" for "img-pull-(-b40a-459d-9033-)" with CreatePodSandboxError: "Failed to generate sandbox config for pod "img-pull-(-b40a-459d-9033-)": Unexpected resolver config value: "C:etckubernetescni". Expected "" or "Host"."" pod="img-puller-3842/img-pull-" podUID=-b40a-459d-9033- This is showing on all the runs of GCE testgrid (latest): Reading/using the passed resolv config on Windows Run a Windows pool with DNS resolv config being set as part of kubelet params. No response_ 27 alpha.3 1.27 beta.0 master <detailsGCE </details<details</details<details</details<details</details<details</details
/triage accepted /milestone v1.27 /sig node /sig network
/sig windows /cc
/priority criticial-urgent as this is likely a release blocking
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority criticial-urgent Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
Hmm, so you're setting the [1]. This PR introduced the capability to read the DNS config options from the Windows host registries while removing the capability to read from a file, introducing a regression. I've sent a PR addressing this issue: [1]
Is the file you are pointing to a UNIX-style or something else?
Was the file changing the behavior of the node?


Question: procGetNetworkParams = iphlpapidll.MustFindProc("GetNetworkParams") ) <ins> func fileExists(filename string) (bool, error) { stat, err := os.Stat(filename) if os.IsNotExist(err) { return false, nil } if err != nil { return false, err } return stat.Mode().IsRegular(), nil } </ins> func getHostDNSConfig(resolverConfig string) (*runtimeapi.DNSConfig, error) { <del> if resolverConfig != "" && resolverConfig != hostResolvConf { err := fmt.Errorf(`Unexpected resolver config value: "%s". Expected "" or "%s".`, resolverConfig, hostResolvConf) </del> <ins> if resolverConfig == "" { // This handles "" by returning defaults. return getDNSConfig(resolverConfig) } isFile, err := fileExists(resolverConfig) if err != nil { err = fmt.Errorf(`Unexpected error while getting os.Stat for "%s" resolver config. Error: %w`, resolverConfig, err) </ins> klog.ErrorS(err, "Cannot get host DNS Configuration.") return nil, err } <ins> if isFile { // Get the DNS config from a resolv.conf-like file. return getDNSConfig(resolverConfig) } </ins> <del> var ( hostDNS, hostSearch []string err                 error ) </del> <ins> if resolverConfig != hostResolvConf { err := fmt.Errorf(`Unexpected resolver config value: "%s". Expected "", "%s", or a path to an existing resolv.conf file.`, resolverConfig, hostResolvConf) klog.ErrorS(err, "Cannot get host DNS Configuration.") return nil, err } // If we get here, the resolverConfig == hostResolvConf and that is not actually a file, so // it means to use the host settings. </ins> // Get host DNS settings <del> if resolverConfig == hostResolvConf { hostDNS, err = getDNSServerList() if err != nil { err = fmt.Errorf("Could not get the host's DNS Server List. Error: %w", err) klog.ErrorS(err, "Encountered error while getting host's DNS Server List.") return nil, err } hostSearch, err = getDNSSuffixList() if err != nil { err = fmt.Errorf("Could not get the host's DNS Suffix List. Error: %w", err) klog.ErrorS(err, "Encountered error while getting host's DNS Suffix List.") return nil, err } </del> <ins> hostDNS, err := getDNSServerList() if err != nil { err = fmt.Errorf("Could not get the host's DNS Server List. Error: %w", err) klog.ErrorS(err, "Encountered error while getting host's DNS Server List.") return nil, err } hostSearch, err := getDNSSuffixList() if err != nil { err = fmt.Errorf("Could not get the host's DNS Suffix List. Error: %w", err) klog.ErrorS(err, "Encountered error while getting host's DNS Suffix List.") return nil, err </ins> } return &runtimeapi.DNSConfig{ Servers:  hostDNS,

Yeah, what used on GCE/GKE, seems to be a Linux-like conf, eg: nameserver 10.128.0.1 search c.<project_name.internal google.internal ` We haven't tried to remove from kubelet to see the behavior.
This Issue/PR has been marked as release blocker. Please have the fix merged by eod today otherwise the feature has to be reverted. For the Issue/PR owner: please have a revert/feature disable PR open today in case the fix failed to get in. Thank you! See the -- Kubernetes 1.27 release manager
I opened up a revert PR in case this PR does not merge by code freeze.
Thanks for the resolution! both 2019 and 2022 grids seems to be back to green now!
Thanks for confirming! I opened to ensure gets updated too


Question: kubeCfg.PodInfraContainerImage, float32(kubeCfg.RegistryPullQPS), int(kubeCfg.RegistryBurst), <del> containerLogsDir, </del> <ins> ContainerLogsDir, </ins> kubeDeps.OSInterface, klet.networkPlugin, klet,

Cluster logging assumes kubelet stores container log at a specific directory with specific name. Currently the path is: After the CRI work, the path will be: However, we don't have test coverage for this. We don't even know whether the log files are there or not. We should add a node e2e test for this, basically: 1) Create a container which prints logs 2) Check whether the expected log file is created. 3) Check whether the logs are collected in the file as expected. And it would be better to launch a pod to do the check instead of accessing local file system directly, which makes containerizing easier. /cc because you are collecting missing test. /cc
I will try to add a node-e2e test there
I think it's ok to add this for CRI. But there maybe problems on today's interface for rkt. If I'm not wrong, rkt does't store logs at same location with docker. cc/ Is this true?
Thanks! :)
After gets merged, we need to update the test to verify the cri log path.


Question: } return nil } <ins> // ClearEntriesForPortNAT uses the conntrack tool to delete the contrack entries // for connections specified by the {dest IP, port} pair. // Known issue: // https://github.com/kubernetes/kubernetes/issues/59368 func ClearEntriesForPortNAT(execer exec.Interface, dest string, port int, protocol v1.Protocol) error { if port <= 0 { return fmt.Errorf("Wrong port number. The port number must be greater then zero") } parameters := parametersWithFamily(utilnet.IsIPv6String(dest), "-D", "-p", protoStr(protocol), "--dport", strconv.Itoa(port), "--dst-nat", dest) err := Exec(execer, parameters...) if err != nil && !strings.Contains(err.Error(), NoConnectionToDelete) { return fmt.Errorf("error deleting conntrack entries for UDP port: %d, error: %v", port, err) } return nil } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT What happened: When deleting/recreating pods backing a UDP service, traffic from external senders stops flowing. What you expected to happen: Traffic would continue to flow to new pods. How to reproduce it (as minimally and precisely as possible): Based on reading of similar issues and commit messages it seems like this would be most noticable on high-volume UDP applications, statsd and syslog are good examples. a deployment and service for a UDP app. In my test case, this would be the container. an external node, create a small test script utilizing a statsd library such as pystatsd. The most important part is that the sender uses the same source-port for every packet sent. Ex: the sender. Install tcpdump into the pod and watch for incoming packets, example from the test case: the pod tcpdump in the newly created pod. Observe that no new matching packets are flowing. existing conntrack rules on all nodes in the cluster. You should see traffic start flowing to the pod again. example command for clearing conntrack rules on the nodes: Anything else we need to know?: I have tested this only on GKE-managed clusters that utilize google forwarding-rules for external LBs, but I believe the issue may be present on any platform where iptables mode for kube-proxy is used. Tested and verified on 1.8.6-gke.0 and 1.7.11-gke.1. Also tested with both Google Container-optimized OS and the Ubuntu-based node images. I have only test this on GKE It appears this was a known issue and should have been cleared up by: From my reading of , it seems as if the func should exec() to clear up UDP entries on service endpoint change, but the only place I see that func called is from syncProxyRules(). That func looks complex and possibly like it is only run occasionally and not on service/endpoint changes. Environment: Kubernetes version (use ): (Also tested on 1.7.11-gke.1) Cloud provider or hardware configuration: GKE-managed clusters, google cloud. OS (e.g.


Question: } return nil } <ins> // ClearEntriesForPortNAT uses the conntrack tool to delete the contrack entries // for connections specified by the {dest IP, port} pair. // Known issue: // https://github.com/kubernetes/kubernetes/issues/59368 func ClearEntriesForPortNAT(execer exec.Interface, dest string, port int, protocol v1.Protocol) error { if port <= 0 { return fmt.Errorf("Wrong port number. The port number must be greater then zero") } parameters := parametersWithFamily(utilnet.IsIPv6String(dest), "-D", "-p", protoStr(protocol), "--dport", strconv.Itoa(port), "--dst-nat", dest) err := Exec(execer, parameters...) if err != nil && !strings.Contains(err.Error(), NoConnectionToDelete) { return fmt.Errorf("error deleting conntrack entries for UDP port: %d, error: %v", port, err) } return nil } </ins>

from /etc/os-release): tried both: and Kernel (e.g. ): and Install tools: Others:
/sig network
We also ran into this in kubernetes v1.7.6 using OpenShift 3.7:
~I believe this is the same as ~If so, it is fixed in 1.10: These are about HostPort, not NodePort, sorry for the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any other known workaround for this besides updating conntrack rules on the kube nodes? I tried deleting and recreating the service, but this didn't update the rules. When I restart the stream on the client side data flows correctly again (perhaps it's perceived as a new connection), but I have to restart the stream for each sender.
/remove-lifecycle stale
FWIW, this is not fixed in 1.10 - that fix was for HostPort, and this is still a bug for NodePort.
I am seeing this issue in 1.11. I am using LoadBalancer type.
has been working on a fix for this, FYI.
This issue is extremely impactful for telephony protocols that rely on static source ip and source port. Any update on a possible workaround? I was thinking a potential workaround might be to reduce the UDP timeout to something really short so the stale conntrack entries go away faster. Or maybe there is a way to flush conntrack on the event of pod deletion? Edit: Removed stupid idea of a workaround to disable conntrack for UDP. Can't disable conntrack for UDP because it's used for DNAT from node to container.
The workaround is to not use NodePorts. Other kinds of services are already correct.
not sure what you mean. I am seeing the issue with a LoadBalancer type so it does not affect NodePorts only. As a workaround I have created a small python script that watches the K8s API for pod deletions and clears the related conntrack entries when that happens. I can share it when I get to work on Monday if you want to try it out.


Question: } return nil } <ins> // ClearEntriesForPortNAT uses the conntrack tool to delete the contrack entries // for connections specified by the {dest IP, port} pair. // Known issue: // https://github.com/kubernetes/kubernetes/issues/59368 func ClearEntriesForPortNAT(execer exec.Interface, dest string, port int, protocol v1.Protocol) error { if port <= 0 { return fmt.Errorf("Wrong port number. The port number must be greater then zero") } parameters := parametersWithFamily(utilnet.IsIPv6String(dest), "-D", "-p", protoStr(protocol), "--dport", strconv.Itoa(port), "--dst-nat", dest) err := Exec(execer, parameters...) if err != nil && !strings.Contains(err.Error(), NoConnectionToDelete) { return fmt.Errorf("error deleting conntrack entries for UDP port: %d, error: %v", port, err) } return nil } </ins>

I've my script . Note that it's not tested much so don't use it in a production environment. Also it cleans up all conntrack entries for a deleted pod, not only the UDP related ones.
We are also seeing this issue with type loadbalancer. We are on 1.11. Our use-case is the following: multiple devices are sending to an aggreagator machine. This machine forwards the UDP packages to a loadBalancer in gcp. When debugging the service, I noticed the following: The traffic is assigned to one pod, and the packages are not loadbalanced (client and server are the same cfr use-case) When redeploying a new cluster, Conntrack is cleared on termination signal of the old pod. (client receives connection refused) red/black and highlander deployment strategies have the same effect. How would I expect this to be working then? New cluster is . Conntrack is cleared Data is routed to NEW pods old pods are terminated Best Regards, Tim
I am seeing this exact same issue on our cluster which has k8s v1.14.5 . We have endpoint nginx pods backing a UDP service with nodeport. Suddenly one day we see traffic not hitting into nginx pod but reaching the node hosting the nginx pod. We could simulate the problem by restarting the nginx pods. I am assuming with this issue marked as closed; and having made its way into v1.14; we should not be seeing this. Would also like to point out; the one thing, in addition we have in the service is ExternalTrafficPolicy set to Local. Can anyone please guide/help on why we are seeing this in v.1.14.5 of our cluster.
Seeing this with k8s v1.16.5


Question: importpath = "k8s.io/kube-aggregator/pkg/apiserver", deps = [ "//vendor/github.com/golang/glog:go_default_library", <del> "//vendor/k8s.io/api/core/v1:go_default_library", </del> "//vendor/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apimachinery/announced:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apimachinery/registered:go_default_library",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Currently APIServiceRegistrationController watch service change and find the related APIServices related if any. Then it AddAPIService/RemoveAPIService for APIAggregator. I dig into the kube-aggregator and find it no need totally, because update APIServices status when related service or endpoint change. So if does not watch services, it will not miss any event for APIServices. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/cc
/kind enhancement /sig api-machinery
/assign
that does not follow. AvailableConditionController modifies the APIService object. APIServiceRegistrationController must still watch the APIService objects to observe those changes.
oh, I misread the description. You're saying APIServiceRegistrationController doesn't need to watch v1.Service objects


Question: if len(psp.Spec.AllowedFlexVolumes) > 0 { w.Write(LEVEL_1, "Allowed FlexVolume Types:t%sn", flexVolumesToString(psp.Spec.AllowedFlexVolumes)) } <ins> if len(psp.Spec.AllowedUnsafeSysctls) > 0 { w.Write(LEVEL_1, "Allowed Unsafe Sysctls:t%sn", sysctlsToString(psp.Spec.AllowedUnsafeSysctls)) } if len(psp.Spec.ForbiddenSysctls) > 0 { w.Write(LEVEL_1, "Forbidden Sysctls:t%sn", sysctlsToString(psp.Spec.ForbiddenSysctls)) } </ins> w.Write(LEVEL_1, "Allow Host Network:t%tn", psp.Spec.HostNetwork) w.Write(LEVEL_1, "Allow Host Ports:t%sn", hostPortRangeToString(psp.Spec.HostPorts)) w.Write(LEVEL_1, "Allow Host PID:t%tn", psp.Spec.HostPID)

/kind bug What happened: does not show the description for the new field in What you expected to happen: should describe the new field. How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version (use ): master
/sig auth
GitHub didn't allow me to assign the following users: fanzhangio. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If nobody has started yet, I wanna to pick it up.
was working on it AFAIR.
Indeed. if you don't mind, I'd take over.


Question: // applyPlatformSpecificDockerConfig applies platform-specific configurations to a dockertypes.ContainerCreateConfig struct. // The containerCleanupInfo struct it returns will be passed as is to performPlatformSpecificContainerCleanup <del> // after either: //   * the container creation has failed //   * the container has been successfully started //   * the container has been removed // whichever happens first. </del> <ins> // after either the container creation has failed or the container has been removed. </ins> func (ds *dockerService) applyPlatformSpecificDockerConfig(*runtimeapi.CreateContainerRequest, *dockertypes.ContainerCreateConfig) (*containerCleanupInfo, error) { return nil, nil } // performPlatformSpecificContainerCleanup is responsible for doing any platform-specific cleanup <del> // after either: //   * the container creation has failed //   * the container has been successfully started //   * the container has been removed // whichever happens first. // Any errors it returns are simply logged, but do not prevent the container from being started or // removed. </del> <ins> // after either the container creation has failed or the container has been removed. </ins> func (ds *dockerService) performPlatformSpecificContainerCleanup(cleanupInfo *containerCleanupInfo) (errors []error) { return }

What would you like to be : Periodically remove cleanupInfos for old containers from dockershim's map. One way of doing that would be to just forget about s older than say an hour, as they should always be short-lived. Why is this needed: introduced a new in-memory map for the dockershim, that could potentially (in pathological cases) cause memory leaks - for containers that use GMSA cred specs, get created successfully, but then never get started nor removed. This might be okay for the alpha release of the GMSA feature, but should be addressed in the beta. See
/assign wk8
/sig windows
/milestone v1.15
got moved out. let's move this issue with it as well /milestone v1.16
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
: yup we'd love to have this in 1.16, still waiting for kind reviewers on :) ...


Question: utilerrors "k8s.io/apimachinery/pkg/util/errors" serviceapi "k8s.io/kubernetes/pkg/api/v1/service" <ins> "github.com/Azure/azure-sdk-for-go/arm/compute" </ins> "github.com/Azure/azure-sdk-for-go/arm/network" "github.com/Azure/go-autorest/autorest/to" "github.com/golang/glog"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: In certain large cluster scenarios we would like to anticipate kubelet behavior hitting request limits, and backoff in response. E.g., What you expected to happen: This is a request for add'l backoff coverage, so nothing surprising. How to reproduce it (as minimally and precisely as possible): Build a cluster with as many nodes as your subscription will support on azure until kubelet behavior contributes to backend API request limits. Environment: Kubernetes version (use ): and Cloud provider or hardware configuration**: azure OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS (Xenial Xerus) Kernel (e.g. ): Linux k8s-pool7--7 4.4.0-83-generic -Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: acs-engine Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig azure


Question: fs.IntVar(&s.HardPodAffinitySymmetricWeight, "hard-pod-affinity-symmetric-weight", api.DefaultHardPodAffinitySymmetricWeight, "RequiredDuringScheduling affinity is not symmetric, but there is an implicit PreferredDuringScheduling affinity rule corresponding "+ "to every RequiredDuringScheduling affinity rule. --hard-pod-affinity-symmetric-weight represents the weight of implicit PreferredDuringScheduling affinity rule.") <ins> fs.MarkDeprecated("hard-pod-affinity-symmetric-weight", "This option was moved to the policy configuration file") </ins> fs.StringVar(&s.FailureDomains, "failure-domains", api.DefaultFailureDomains, "Indicate the "all topologies" set for an empty topologyKey when it's used for PreferredDuringScheduling pod anti-affinity.") fs.MarkDeprecated("failure-domains", "Doesn't have any effect. Will be removed in future version.") leaderelection.BindFlags(&s.LeaderElection, fs)

( noticed this) HardPodAffinitySymmetricWeight should be in the , not in KubeSchedulerConfiguration If/when fixing this, we need to be careful to do it in a backward-compatilble way.
Do you mean ? It seems scheduler policy config's args can not passed to FitPredicates; it need to update related interface. For the : mark to be deprecated and set scheduler policy config to HardPodAffinitySymmetricWeight if policy config is empty.
Are you working on this? if no, I can take it.
, nop; please go ahead.
/ / / , When reviewing the PR, there's one question on : the default value of is 1, which means the weight (1) will always add to priorities if term matched (), does it our expectation? And the user has to set it to 0 to disable it. IMO, the default value of should be 0.
The value 1 (instead of 0) was chosen so that if you say A must run with B, there is an automatic preference that B should run with A. This seems like reasonable behavior.
Thanks very much; reasonable case to me. But it seems we can not disable it: if set to 0, the default funcs will set it to default (1); similar result to un-set. Anyway, no user complain this now :).


Question:
PLEASE NOTE: This document applies to the HEAD of the source tree
 If you are using a released version of Kubernetes, you should refer to the docs that go with that version. Documentation for other releases can be found at [releases.k8s.io](http://releases.k8s.io).  --   # Generic Configuration Object ## Abstract This proposal proposes a new API resource, `ConfigMap`, that stores data used for the configuration of applications deployed on `Kubernetes`. The main focus points of this proposal are: * Dynamic distribution of configuration data to deployed applications. * Encapsulate configuration information and simplify `Kubernetes` deployments. * Create a flexible configuration model for `Kubernetes`. ## Motivation A `Secret`-like API resource is needed to store configuration data that pods can consume. Goals of this design: 1.  Describe a `ConfigMap` API resource 2.  Describe the semantics of consuming `ConfigMap` as environment variables 3.  Describe the semantics of consuming `ConfigMap` as files in a volume ## Use Cases 1. As a user, I want to be able to consume configuration data as environment variables 2. As a user, I want to be able to consume configuration data as files in a volume 3. As a user, I want my view of configuration data in files to be eventually consistent with changes to the data ### Consuming `ConfigMap` as Environment Variables Many programs read their configuration from environment variables.  `ConfigMap` should be possible to consume in environment variables.  The rough series of events for consuming `ConfigMap` this way is: 1. A `ConfigMap` object is created 2. A pod that consumes the configuration data via environment variables is created 3. The pod is scheduled onto a node 4. The kubelet retrieves the `ConfigMap` resource(s) referenced by the pod and starts the container processes with the appropriate data in environment variables ### Consuming `ConfigMap` in Volumes Many programs read their configuration from configuration files.  `ConfigMap` should be possible to consume in a volume.  The rough series of events for consuming `ConfigMap` this way is: 1. A `ConfigMap` object is created 2. A new pod using the `ConfigMap` via the volume plugin is created 3. The pod is scheduled onto a node 4. The Kubelet creates an instance of the volume plugin and calls its `Setup()` method 5. The volume plugin retrieves the `ConfigMap` resource(s) referenced by the pod and projects the appropriate data into the volume ### Consuming `ConfigMap`  Updates Any long-running system has configuration that is mutated over time.  Changes made to configuration data must be made visible to pods consuming data in volumes so that they can respond to those changes. The `resourceVersion` of the `ConfigMap` object will be updated by the API server every time the object is modified.  After an update, modifications will be made visible to the consumer container: 1. A `ConfigMap` object is created 2. A new pod using the `ConfigMap` via the volume plugin is created 3. The pod is scheduled onto a node 4. During the sync loop, the Kubelet creates an instance of the volume plugin and calls its `Setup()` method 5. The volume plugin retrieves the `ConfigMap` resource(s) referenced by the pod and projects the appropriate data into the volume 6. The `ConfigMap` referenced by the pod is updated 7. During the next iteration of the `syncLoop`, the Kubelet creates an instance of the volume plugin and calls its `Setup()` method 8. The volume plugin projects the updated data into the volume atomically It is the consuming pod's responsibility to make use of the updated data once it is made visible. Because environment variables cannot be updated without restarting a container, configuration data consumed in environment variables will not be updated. ### Advantages * Easy to consume in pods; consumer-agnostic * Configuration data is persistent and versioned * Consumers of configuration data in volumes can respond to changes in the data ## Proposed Design ### API Resource The `ConfigMap` resource will be added to the `extensions` API Group: ```go package api // ConfigMap holds configuration data for pods to consume. type ConfigMap struct { TypeMeta   `json:",inline"` ObjectMeta `json:"metadata,omitempty"` // Data contains the configuration data.  Each key must be a valid DNS_SUBDOMAIN or leading // dot followed by valid DNS_SUBDOMAIN. Data map[string]string `json:"data,omitempty"` } type ConfigMapList struct { TypeMeta `json:",inline"` ListMeta `json:"metadata,omitempty"` Items []ConfigMap `json:"items"` } ``` A `Registry` implementation for `ConfigMap` will be added to `pkg/registry/configmap`. ### Environment Variables The `EnvVarSource` will be extended with a new selector for `ConfigMap`: ```go package api // EnvVarSource represents a source for the value of an EnvVar. type EnvVarSource struct { // other fields omitted // Specifies a ConfigMap key ConfigMap *ConfigMapSelector `json:"configMap,omitempty"` } // ConfigMapSelector selects a key of a ConfigMap. type ConfigMapSelector struct { // The name of the ConfigMap to select a key from. ConfigMapName string `json:"configMapName"` // The key of the ConfigMap to select. Key string `json:"key"` } ``` ### Volume Source A new `ConfigMapVolumeSource` type of volume source containing the `ConfigMap` object will be added to the `VolumeSource` struct in the API: ```go package api type VolumeSource struct { // other fields omitted ConfigMap *ConfigMapVolumeSource `json:"configMap,omitempty"` } // ConfigMapVolumeSource represents a volume that holds configuration data type ConfigMapVolumeSource struct { // A list of configuration data keys to project into the volume in files Files []ConfigMapVolumeFile `json:"files"` } // ConfigMapVolumeFile represents a single file containing configuration data type ConfigMapVolumeFile struct { ConfigMapSelector `json:",inline"` // The relative path name of the file to be created. // Must not be absolute or contain the '..' path. Must be utf-8 encoded. // The first item of the relative path must not start with '..' Path string `json:"path"` } ``` **Note:** The update logic used in the downward API volume plug-in will be extracted and re-used in the volume plug-in for `ConfigMap`. ## Examples #### Consuming `ConfigMap` as Environment Variables ```yaml apiVersion: extensions/v1beta1 kind: ConfigMap metadata: name: etcd-env-config data: number_of_members: 1 initial_cluster_state: new initial_cluster_token: DUMMY_ETCD_INITIAL_CLUSTER_TOKEN discovery_token: DUMMY_ETCD_DISCOVERY_TOKEN discovery_url: http://etcd-discovery:2379 etcdctl_peers: http://etcd:2379 ``` This pod consumes the `ConfigMap` as environment variables: ```yaml apiVersion: v1 kind: Pod metadata: name: config-env-example spec: containers: - name: etcd image: openshift/etcd-20-centos7 ports: - containerPort: 2379 protocol: TCP - containerPort: 2380 protocol: TCP env: - name: ETCD_NUM_MEMBERS valueFrom: configMap: configMapName: etcd-env-config key: number_of_members - name: ETCD_INITIAL_CLUSTER_STATE valueFrom: configMap: configMapName: etcd-env-config key: initial_cluster_state - name: ETCD_DISCOVERY_TOKEN valueFrom: configMap: configMapName: etcd-env-config key: discovery_token - name: ETCD_DISCOVERY_URL valueFrom: configMap: configMapName: etcd-env-config key: discovery_url - name: ETCDCTL_PEERS valueFrom: configMap: configMapName: etcd-env-config key: etcdctl_peers ``` ### Consuming `ConfigMap` as Volumes `redis-volume-config` is intended to be used as a volume containing a config file: ```yaml apiVersion: extensions/v1beta1 kind: ConfigMap metadata: name: redis-volume-config data: redis.conf: "pidfile /var/run/redis.pidnport6379ntcp-backlog 511n databases 1ntimeout 0n" ``` The following pod consumes the `redis-volume-config` in a volume: ```yaml apiVersion: v1 kind: Pod metadata: name: config-volume-example spec: containers: - name: redis image: kubernetes/redis command: "redis-server /mnt/config-map/etc/redis.conf" ports: - containerPort: 6379 volumeMounts: - name: config-map-volume mountPath: /mnt/config-map volumes: - name: config-map-volume configMap: files: - path: "etc/redis.conf" configMapName: redis-volume-config key: redis.conf ``` ### Future Improvements In the future, we may add the ability to specify an init-container that can watch the volume contents for updates and respond to changes when they occur.  [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/proposals/configmap.md?pixel)]()

What happened: Visited It shows the release notes for v1.17 What you expected to happen: See the release notes for v1.18 How to reproduce it (as minimally and precisely as possible): Open in any browser or with curl
/sig docs
/assign


Question: // Run begins watching and syncing daemon sets. func (dsc *DaemonSetsController) Run(workers int, stopCh <-chan struct{}) { defer util.HandleCrash() <ins> glog.Infof("Starting Daemon Sets controller manager") controller.SyncAllPodsWithStore(dsc.kubeClient, dsc.podStore.Store) </ins> go dsc.dsController.Run(stopCh) go dsc.podController.Run(stopCh) go dsc.nodeController.Run(stopCh)

The test is in flaky suite and no one seems to care. We need an owner for this test. cc Last failure log:
do you think you can take a look at this?
This is happening because of a bad assumption. The assumption is: if the reflector's store is consistent with etcd, the store given back to clients of the framework.Controller is too. I'd really like a general fix applicable to all controllers for this problem, but fixing it just for the RCs is pretty easy. In more detail: This function reports when the reflector has listed from apiserver. But the store given to the reflector is the fifo of the framework.Controller. The store given back to the caller is still not consistent, and will not be till the framework.Controller puts all the objects from the fifo into the client state store (). The rc specific fix: Before starting the rc manager, seed the expectations store with an expectation like {rc:foo, adds: foo.Status.Replicas, deletes:0}. This will wait for status.Replicas number of ADD events till a timeout. The general fix: Somehow expose this through framework.Controller.HasSynced (clients don't care about the fifo anyway, shouldn't it just be an implementation detail?) ideas? cc'ing people who have written controllers that might be interested I don't think this (and generally anything flaky for 1 release) is a p0, I can fix it if you don't mind dealing with the flakes till I've cleared things off my plate. I'd be more than happy to review a fix.
Well, it's a P0 because test is flaky. E.g last run:
- did you have chance to look into it?
The "DaemonRestart" tests are still labeled at . I will send a PR to promote them.
are you still planning to send aPR to promote the DaemonRestart tests out of flaky? I think this is the right thing to do.
Ack; sorry, I lost this in the muck.


Question: }) It("should write files of various sizes, verify size, validate content", func() { <del> fileSizes := []int64{1 * framework.MiB, 100 * framework.MiB} </del> <ins> fileSizes := []int64{fileSizeSmall, fileSizeMedium} </ins> fsGroup := int64(1234) podSec := v1.PodSecurityContext{ FSGroup: &fsGroup,

The "Volume plugin streaming [Slow] NFS should write files of various sizes, verify size, validate content" e2e test is failing on our 2k-node gce clusters (https://k8s-) with the following error: cc
Could you take a look into it or reassign as apt? This is part of the correctness suite and large cluster tests are release-blocking.
/assign
is this run using the kubemark environment? I'll also take a look. It could be that the commands this test needs to execute may not work with kubemark. /assign
cc
No, this is not kubemark. This is on a real 2000-node cluster.
Also all other volume-related e2e tests are passing, except this one.
Looking at the log I see that the nfs test fails trying to verify the test file's content. The test exec's into the nfs client pod and executes: The return code from above is 127. The test then deletes the nfs client pod calling which works. The nfs-server pod is deleted in the AfterEach. For some reason the reported error is 137 but the returned err 127. The same command is exec'd for all of the tested storage plugins so not sure why it's failing just for nfs. Error 127 might be "command not found".
The other strange thing is that it's only failing in this suite and not the others.
When I look at the kubelet log, I see the exec command coming through: I also see these suspicious kubelet soon afterwards: So I think the grep command is causing the node to run out of memory and get evicted. I think we can fix this either by: Making the grep command more memory efficient. I'm guessing it's reading the whole file into memory or, Make this test Serial so that it doesn't run in parallel with other pods, since it consumes a lot of memory.
An unrelated issue, is that the node allocatable feature didn't actually evict the pod. Instead this pod got a system OOM kill. /cc
related to the memcg notifications, not node allocatable. Thanks!
any theories why we see this only for the nfs-io tests, and not the glusterfs-io test for instance?
the gluster test didn't run on this in this suite.


Question: }) It("should write files of various sizes, verify size, validate content", func() { <del> fileSizes := []int64{1 * framework.MiB, 100 * framework.MiB} </del> <ins> fileSizes := []int64{fileSizeSmall, fileSizeMedium} </ins> fsGroup := int64(1234) podSec := v1.PodSecurityContext{ FSGroup: &fsGroup,

I'm suspecting that the reason why we didn't see a problem with this test on other suites is that the load could be higher here. I've asked to see if we can reduce the memory usage of the grep call.
pointed out that there are no newlines in the generated file, so when grep is called it could buffer the entire file in memory. Adding a newline at the end of the dd input file should do it.
/priority critical-urgent
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/scalability sig/storage: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details
It seems like even if the generated file is broken up by newlines, grep could still use on the order of hundreds of MB of memory for a 100MiB file (even with the -m1 and -F options). We could check the file line by line, but it would take a long time. A better solution would be to check the MD5 file hash. It's fast and uses little memory. The expected hashes will be hard-coded into the test, and if different file sizes are in the future, a hash needs to be computed and as well.
Thanks a lot for fixing this!
Ref


Question: (disk.ManagedDisk != nil && diskURI != "" && strings.EqualFold(*disk.ManagedDisk.ID, diskURI)) { // found the disk klog.V(2).Infof("azureDisk - detach disk: name %q uri %q", diskName, diskURI) <del> disks[i].ToBeDetached = to.BoolPtr(true) </del> <ins> if strings.EqualFold(as.cloud.Environment.Name, "AZURESTACKCLOUD") { disks = append(disks[:i], disks[i+1:]...) } else { disks[i].ToBeDetached = to.BoolPtr(true) } </ins> bFoundDisk = true break }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: flag is not supported on Azure Stack, should switch to use original way to remove a disk from data disk list, refer to for more details What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


Question: klog.Errorf("cannot convert to *v1.Node: %v", t) return } <ins> klog.V(3).Infof("delete event for node %q", node.Name) </ins> // NOTE: Updates must be written to scheduler cache before invalidating // equivalence cache, because we could snapshot equivalence cache after the // invalidation and then snapshot the cache itself. If the cache is

This helps debugging scheduler's behavior, currently we don't have logs that indicate when the scheduler first saw a pod or a node. The log level should be v=3. /sig scheduling /cc
Do you mean add logging for callbacks in ?
yes, at least for add and delete events
Sure, I can also give this pr, before tomorrow
/assign
We actually have one v=2 log line for a Node being to the cache's tree: But it's probably good to keep this one.


Question: package fake <ins> import ( "fmt" "k8s.io/apimachinery/pkg/util/rand" ) </ins> const ( <del> defaultUnixEndpoint = "unix:///tmp/kubelet_remote.sock" </del> <ins> defaultUnixEndpoint = "unix:///tmp/kubelet_remote_%v.sock" </ins> ) // GenerateEndpoint generates a new unix socket server of grpc server. func GenerateEndpoint() (string, error) { <del> return defaultUnixEndpoint, nil </del> <ins> // use random int be a part fo file name return fmt.Sprintf(defaultUnixEndpoint, rand.Int()), nil </ins> }

Which jobs are flaking: periodics-kubernetes-bazel-test Which test(s) are flaking: //pkg/kubelet/cri/remote TestVersion Testgrid link: Reason for failure: Error: Received unexpected error: rpc error: code = Unavailable desc = transport is closing panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference Anything else we need to know: To reproduce with stress: go test -race -c ./pkg/kubelet/cri/remote stress - TestVersion Detailed logs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
I 'll give it a see and fix it :)
/assign
/remove-sig testing SIG node owns their unit tests. SIG testing owns the framework, infrastructure, etc.
/sig node


Question: return true, errors.New("not implemented") } <ins> func (m *execMounter) EvalHostSymlinks(pathname string) (string, error) { return "", errors.New("not implemented") } </ins> func (mounter *execMounter) PrepareSafeSubpath(subPath Subpath) (newHostPath string, cleanupAction func(), err error) { return subPath.Path, nil, nil }

/kind bug Using , if a block device is symlinked to the discovery directory e.g. , then MapVolume will fail with since the containerized kubelet cannot see the path on host. is mounted by containerized kubelet, so if before we try to map the device we set devicePath to the evaluation of symlink and end up with a devicePath of , containerized kubelet will see the device and successfully map the device. Are there any issues with this solution? This is on Kubernetes 1.10
/sig storage
ah yes containerized kubelet has many issues when dealing with symlinks. Check out Mounter.GetMode(), it calls a nsenter EvalSymlinks which should be able to do what you need. But actually, one thing to consider, is do you need to do the losetup on the host, or does inside the container work?
Yes it appears to work from inside the container, on the host I run which disappears after I delete the pod using the device. My understanding of running losetup in MapVolume is not that great but I suppose that as long as that /dev/loop0 is created we are fine. Thanks for the quick response, I'll create a PR using that function and tag you
Does it create it on the host or in the containerized kubelet? What happens when containerized kubelet is restarted? The purpose of losetup is to do "refcounting" on the block device to prevent possible naming collisions when devices are attached/detached under unhealthy conditions. Ie, if a device showed up as /dev/sdb and you gave that to a pod, and then for some reason it become unhealthy and disappeared (while the pod is still running), another device (maybe even another volume type) could get attached to the node and reuse /dev/sdb. Now the pod has access to a device it's not supposed to. With filesystem volumes, the refcounting is done with mount points, but for block device, loopback is the equivalent.
It is created on the host (and also visible in the container), and stays around through restart. I understand now, thank you.
Actually after I restart it becomes /vdb instead of /dev/vdb in the losetup output, idk what that is about.


Question: return true, errors.New("not implemented") } <ins> func (m *execMounter) EvalHostSymlinks(pathname string) (string, error) { return "", errors.New("not implemented") } </ins> func (mounter *execMounter) PrepareSafeSubpath(subPath Subpath) (newHostPath string, cleanupAction func(), err error) { return subPath.Path, nil, nil }

There is no such thing as /vdb and /dev/vdb is still intact of course. Seems inconsequential since it is disappearing after all pods using the device are deleted, i.e. it's still serving as a refcounter


Question: Description: The command 'kubectl version' MUST return the major, minor versions,  GitCommit, etc of the Client and the Server that the kubectl is configured to connect to. */ framework.ConformanceIt("should check is all data is printed ", func() { <del> version := framework.RunKubectlOrDie(ns, "version") requiredItems := []string{"Client Version:", "Server Version:", "Major:", "Minor:", "GitCommit:"} </del> <ins> versionString := framework.RunKubectlOrDie(ns, "version") // we expect following values for: Major -> digit, Minor -> numeric followed by an optional '+',  GitCommit -> alphanumeric requiredItems := []string{"Client Version: ", "Server Version: "} </ins> for _, item := range requiredItems { <del> if !strings.Contains(version, item) { framework.Failf("Required item %s not found in %s", item, version) </del> <ins> if matched, _ := regexp.MatchString(item+`version.Info{Major:"d", Minor:"d++?", GitVersion:"vd.d+.[dw-.+]+", GitCommit:"[0-9a-f]+"`, versionString); !matched { framework.Failf("Item %s value is not valid in %sn", item, versionString) </ins> } } })

only check that the output includes the correct keys - they don't confirm that it has appropriate values. Doing full validation on the version data, or at least enough to catch null-string values, would guard against failures like the bad functionality seen in OpenSUSE's binaries, which have been shipping with many of the fields left completely blank (as I discovered ).
/sig testing
/remove-sig testing /sig cli Edit: testing owns the framework, binaries, infrastructure etc. for testing. SIG CLI is nominally responsible for their test cases.
If SIGs are responsible for their own testcases, then this touches test cases for api-machinery as well (I'm not sure what the non-CLI version of this is, but there's definitely a schema adherence test that should fail at the API level). /sig api-machinery
/triage accepted /assign
GitHub didn't allow me to assign the following users: SaiHarshaK. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /triage accepted /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: // The first detach will be triggered after at least 50ms (maxWaitForUnmountDuration in test). // Right before detach operation is performed, the volume will be first removed from being reported // as attached on node status (RemoveVolumeFromReportAsAttached). After detach operation which is expected to fail, <del> // controller then added the volume back as attached. </del> <ins> // controller then treats the attachment as Uncertain. </ins> // Here it sleeps 100ms so that detach should be triggered already at this point. <del> // verifyVolumeReportedAsAttachedToNode will check volume is in the list of volume attached that needs to be updated // in node status. By calling this function (GetVolumesToReportAttached), node status should be updated, and the volume // will not need to be updated until new changes are applied (detach is triggered again) </del> time.Sleep(100 * time.Millisecond) <del> verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </del> <ins> verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateUncertain, asw) verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, false, asw, volumeAttachedCheckTimeout) </ins> // Add a second pod which tries to attach the volume to the same node. <del> // After adding pod to the same node, detach will not be triggered any more. </del> <ins> // After adding pod to the same node, detach will not be triggered any more, // the volume gets attached and reported as attached to the node. </ins> generatedVolumeName, podAddErr = dsw.AddPod(types.UniquePodName(podName2), controllervolumetesting.NewPod(podName2, podName2), volumeSpec, nodeName1) if podAddErr != nil { t.Fatalf("AddPod failed. Expected: <no error> Actual: <%v>", podAddErr)

After KCM restart, any Pod that reuses a volume that failed to detach gets stuck ContainerCreating with message: User "system:node:ip-10-0-243-" cannot get resource "volumeattachments" in API group "" at the cluster scope: no relationship found between node 'ip-10-0-243-' and this object See steps to reproduce below, it's quite a complex condition. It happens when a pod is restarted, a CSI driver "busy" (i.e. times out) + KCM restarted at a wrong time. Pods start just fine. a Pod that uses an attachable CSI volume. the Pod. Make sure it unmounts successfully, but any fails with a transient error, such as timeout. The external attacher continues to detach the volume at this point. kube-controller-manager (KCM) and start a new one. Wait for any leader election etc - it's important for the new KCM to populate A/D controller with no pod using the detaching volume. Let the A/D controller to observe detach error. a new Pod on the same node that uses the same volume. the CSI driver to detach the volume (i.e. stop injecting any transient errors). At this point: The external attacher succeeds detaching the volume, VolumeAttachment is deleted. The volume is marked as attached in node status (!!!) despite no VolumeAttachment exists. A/D controller does not even try to create a new VolumeAttachment, the volume is marked as attached in its Actual State of World (ASW). Kubelet tries to mount the volume, but WaitForAttach fails with: , basically forever. I created . The script expects that the cluster is created by , because it restarts KCM in the middle. It uses my hacked hostpath CSI driver to inject detach errors. Sources are at / the last commit(s) there. There are two issues that together lead to this situation: fails, , i.e. puts it back to . With the volume in , kubelet may try to mount it, while the external attacher tries to detach it. It is not as bad as it sounds, because kubelet can't read VolumeAttachment with the deletion timestamp and it will not call / of un-attached volume.


Question: // The first detach will be triggered after at least 50ms (maxWaitForUnmountDuration in test). // Right before detach operation is performed, the volume will be first removed from being reported // as attached on node status (RemoveVolumeFromReportAsAttached). After detach operation which is expected to fail, <del> // controller then added the volume back as attached. </del> <ins> // controller then treats the attachment as Uncertain. </ins> // Here it sleeps 100ms so that detach should be triggered already at this point. <del> // verifyVolumeReportedAsAttachedToNode will check volume is in the list of volume attached that needs to be updated // in node status. By calling this function (GetVolumesToReportAttached), node status should be updated, and the volume // will not need to be updated until new changes are applied (detach is triggered again) </del> time.Sleep(100 * time.Millisecond) <del> verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateAttached, asw) verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, true, asw, volumeAttachedCheckTimeout) </del> <ins> verifyVolumeAttachedToNode(t, generatedVolumeName, nodeName1, cache.AttachStateUncertain, asw) verifyVolumeReportedAsAttachedToNode(t, logger, generatedVolumeName, nodeName1, false, asw, volumeAttachedCheckTimeout) </ins> // Add a second pod which tries to attach the volume to the same node. <del> // After adding pod to the same node, detach will not be triggered any more. </del> <ins> // After adding pod to the same node, detach will not be triggered any more, // the volume gets attached and reported as attached to the node. </ins> generatedVolumeName, podAddErr = dsw.AddPod(types.UniquePodName(podName2), controllervolumetesting.NewPod(podName2, podName2), volumeSpec, nodeName1) if podAddErr != nil { t.Fatalf("AddPod failed. Expected: <no error> Actual: <%v>", podAddErr)

KCM starts, all VolumeAttachments that don't have a Pod (that's good), but also (that's probably bad). When the newly started KCM re-tries to detach the volume and gets an error again, it marks the volume as fully attached (same as the previous point) and adds it to . Now the volume is both mounted + attached in ASW and A/D controller actually stops here and waits forever for kubelet to unmount the volume. <detailsToday's k/k master, </details<detailsNone </details<detailsN/A, it's KCM issue. </details<details, but seen in a downstream Kubernetes distro too. </details<details</details<details</details
/sig storage
/assign
/triage accepted
I have the same problem: Seems restart of Kubelet solves this problem. Kubernetes v1.25.13


Question: eventRecorder := record.NewFakeRecorder(100) eventRecorder.IncludeObject = true <del> metadataClient := fakemetadata.NewSimpleMetadataClient(legacyscheme.Scheme) </del> <ins> metadataClient := fakemetadata.NewSimpleMetadataClient(fakemetadata.NewTestScheme()) </ins> tweakableRM := meta.NewDefaultRESTMapper(nil) tweakableRM.AddSpecific(

Because of this scheme mutation, this leads to concurrent use of the same scheme to cause the program to panic. It would be ideal to do this in some concurrent safe manner or add a warning or similar
/sig api-machinery
/assign
/assign /triage accepted


Question: } labels["e2e-run"] = string(RunId) <ins> // We don't use ObjectMeta.GenerateName feature, as in case of API call // failure we don't know whether the namespace was created and what is its // name. name, err := findAvailableNamespaceName(baseName, c) if err != nil { return nil, err } </ins> namespaceObj := &v1.Namespace{ ObjectMeta: metav1.ObjectMeta{ <del> GenerateName: fmt.Sprintf("e2e-tests-%v-", baseName), Namespace:    "", Labels:       labels, </del> <ins> Name:      name, Namespace: "", Labels:    labels, </ins> }, Status: v1.NamespaceStatus{}, }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In the test failed due to namespace leakage: we created e2e-tests-load-30-nodepods-1-pfn9m, but never got response from api server (got 0-length response instead) we retried and created e2e-tests-load-30-nodepods-1-m9tq5 e2e-tests-load-30-nodepods-1-m9tq5 to f.namespacesToDelete and leaked -pfn9m next test case failed because e2e-tests-load-30-nodepods-1-pfn9m wasn't deleted. What you expected to happen: While it's unclear why we got 0-length response instead ( investigates that), we always can get error while the namespace was created (e.g. connection reset, timeout, io error). We should handle this case in retrying correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.14.0-alpha.0.793+ Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Question: if err := os.WriteFile(certFixturePath, certBuffer.Bytes(), 0644); err != nil { return nil, nil, fmt.Errorf("failed to write cert fixture to %s: %v", certFixturePath, err) } <del> if err := os.WriteFile(keyFixturePath, keyBuffer.Bytes(), 0644); err != nil { </del> <ins> if err := os.WriteFile(keyFixturePath, keyBuffer.Bytes(), 0600); err != nil { </ins> return nil, nil, fmt.Errorf("failed to write key fixture to %s: %v", certFixturePath, err) } }

Hello, the SRC folks asked that I file this publicly: In the client-go repo, , function creates a private key and stores it into a file with mode : Typically keys are created with more restrictive modes to prevent other users or services on the system from being able to copy them before the user fixes the permissions. Of course, if a more restrictive umask is in place, this may not be a big deal, but it's an extremely fickle piece of the environment to rely upon for security. It's better to be explicit. Thanks
/kind bug Not sure if this is /sig api-machinery or /sig auth
is this related to us?
Hi IMO, mode make sense for the key file. I would like to work on this, waiting for Thanks!!
/triage accepted
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/lifecycle frozen /good-first-issue I think changing the code to use should be fine. Anyone is free to pick this up. Would be good to include a unit test.
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /lifecycle frozen /good-first-issue Instructions for interacting with me using PR comments are available .


Question: if err := os.WriteFile(certFixturePath, certBuffer.Bytes(), 0644); err != nil { return nil, nil, fmt.Errorf("failed to write cert fixture to %s: %v", certFixturePath, err) } <del> if err := os.WriteFile(keyFixturePath, keyBuffer.Bytes(), 0644); err != nil { </del> <ins> if err := os.WriteFile(keyFixturePath, keyBuffer.Bytes(), 0600); err != nil { </ins> return nil, nil, fmt.Errorf("failed to write key fixture to %s: %v", certFixturePath, err) } }

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Hello, this is my first ever open source contribution. I am trying to understand and get this fixed. Any suggestions would be helpful!
/assign
/assign


Question: "//staging/src/k8s.io/cri-api/pkg/apis/testing:go_default_library", "//vendor/google.golang.org/grpc:go_default_library", "//vendor/k8s.io/utils/exec:go_default_library", <del> ], </del> <ins> ] + select({ "@io_bazel_rules_go//go/platform:aix": [ "//staging/src/k8s.io/apimachinery/pkg/util/rand:go_default_library", ], "@io_bazel_rules_go//go/platform:android": [ "//staging/src/k8s.io/apimachinery/pkg/util/rand:go_default_library", ], "@io_bazel_rules_go//go/platform:darwin": [ "//staging/src/k8s.io/apimachinery/pkg/util/rand:go_default_library", ], "@io_bazel_rules_go//go/platform:dragonfly": [ "//staging/src/k8s.io/apimachinery/pkg/util/rand:go_default_library", ], "@io_bazel_rules_go//go/platform:freebsd": [ "//staging/src/k8s.io/apimachinery/pkg/util/rand:go_default_library", ], "@io_bazel_rules_go//go/platform:illumos": [ "//staging/src/k8s.io/apimachinery/pkg/util/rand:go_default_library", ], "@io_bazel_rules_go//go/platform:ios": [ "//staging/src/k8s.io/apimachinery/pkg/util/rand:go_default_library", ], "@io_bazel_rules_go//go/platform:js": [ "//staging/src/k8s.io/apimachinery/pkg/util/rand:go_default_library", ], "@io_bazel_rules_go//go/platform:linux": [ "//staging/src/k8s.io/apimachinery/pkg/util/rand:go_default_library", ], "@io_bazel_rules_go//go/platform:nacl": [ "//staging/src/k8s.io/apimachinery/pkg/util/rand:go_default_library", ], "@io_bazel_rules_go//go/platform:netbsd": [ "//staging/src/k8s.io/apimachinery/pkg/util/rand:go_default_library", ], "@io_bazel_rules_go//go/platform:openbsd": [ "//staging/src/k8s.io/apimachinery/pkg/util/rand:go_default_library", ], "@io_bazel_rules_go//go/platform:plan9": [ "//staging/src/k8s.io/apimachinery/pkg/util/rand:go_default_library", ], "@io_bazel_rules_go//go/platform:solaris": [ "//staging/src/k8s.io/apimachinery/pkg/util/rand:go_default_library", ], "//conditions:default": [], }), </ins> ) filegroup(


Question: gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= <ins> k8s.io/klog/v2 v2.80.1 h1:atnLQ121W371wYYFawwYx1aEY2eUfs4l3J72wtgAwV4= k8s.io/klog/v2 v2.80.1/go.mod h1:y1WjHnz7Dj687irZUWR/WLkLc5N1YHtjLdmgWjndZn0= </ins>

generation for local KEK based on counter; also rotate once a week even if the counter does not hit 2 million? using AES-GCM encryption of local KEK ref: The implementation would be to the directory.
/sig auth /triage accepted
/assign
GitHub didn't allow me to assign the following users: ibihim. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
thank you !
/milestone v1.26
Hi bug-triage shadow team member for 1.26 here. Looks like this issue is in the 1.26 milestone, but has been moved to 1.27. Is there something else or should this issue go to 1.27 as well? Thank you!
I just forgot to move this issue /milestone v1.27
/reopen is the partial implementation.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
My suggestion on the reference implementation, after reading 's comments is like so... The GRPC server, which implements the expects an implementation that satisfies . The local key logic can be a middleware. So it expects a it can fallback to, if it doesn't know a key or needs to encrypt its own local KEK. So the local implementation would be need to: have a storage of s have a reference to the current in use (if we put it into the cache, it might vanish) have a reference to another , which should be the remote KMS. So the sub-stories could look like: write Middleware write Transformers that can get exhausted (by time, by usage) write 's storage write some dummy for remote KMS? I have some sketches of what I am talking about .


Question: gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= <ins> k8s.io/klog/v2 v2.80.1 h1:atnLQ121W371wYYFawwYx1aEY2eUfs4l3J72wtgAwV4= k8s.io/klog/v2 v2.80.1/go.mod h1:y1WjHnz7Dj687irZUWR/WLkLc5N1YHtjLdmgWjndZn0= </ins>

I would implement that in terms of the interface so that it can be used both in testing (via a local key) and in real code (by delegating to a HSM).
Hi bug-triage team member for 1.27 here. This issue is still open but has had no recent activity, and no active PRs. Is everything on track for the Code Freeze in March 14th? Thank you!
/milestone clear
Could be closed, right? :)
We still need metrics and tracing support for the reference implementation (though it is all a lot less important now).
/retitle [KMSv2] Add metrics and tracing in for plugins
This is duplicate of /close
Closing this issue. <detailsIn response to : This is duplicate of Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: _, err = client.CoreV1().Pods(defaultNamespace).List(ctx, metav1.ListOptions{}) tt.assertErrFn(t, err) <del> err = os.WriteFile(apiServer.ServerOpts.Authentication.AuthenticationConfigFile, []byte(tt.newAuthConfigFn(t, oidcServer.URL(), string(caCert))), 0600) </del> <ins> // Create a temporary file tempFile, err := os.CreateTemp("", "tempfile") require.NoError(t, err) defer func() { _ = tempFile.Close() }() // Write the new content to the temporary file _, err = tempFile.Write([]byte(tt.newAuthConfigFn(t, oidcServer.URL(), string(caCert)))) require.NoError(t, err) // Atomically replace the original file with the temporary file err = os.Rename(tempFile.Name(), apiServer.ServerOpts.Authentication.AuthenticationConfigFile) </ins> require.NoError(t, err) if tt.waitAfterConfigSwap {

=== RUN TestStructuredAuthenticationConfigReload/oldinvalidconfigtonewvalidconfig first seen in 3/11 No response /sig api-machinery instrumentation
/sig auth /remove-sig api-machinery instrumentation /triage accepted /assign


Question: } } <del> allErrs = append(allErrs, validateValueValidation(s.ValueValidation, skipAnyOf, skipFirstAllOfAnyOf, fldPath)...) </del> <ins> allErrs = append(allErrs, validateValueValidation(s.ValueValidation, skipAnyOf, skipFirstAllOfAnyOf, lvl, fldPath)...) </ins> if s.XEmbeddedResource && s.Type != "object" { if len(s.Type) == 0 {

We allow CRD authors to write . Currently we don't restrict writing validation for , which means people can have . For example, people can have additive name constraints (require CR to be ) with the following schema: This brings us two questions: do we if "metadata" is included in the schema: simplest option is to publish standard metav1.ObjectMeta schema info, and use any CRD-specified "metadata" properties as additive validation (just like in-tree types do, publishing standard ObjectMeta schema info, and implementing type-specific name validation internally) alternatively, we can try merging user-specified metadata schema with the standard one. Technically people should only add extra validation to existing metadata properties (adding new fields to metadata won't work since they will be pruned by built-in validation first), but it's unclear to me how arbitrary people may want the metadata validation to be term question is whether to allow people specifying metadata validation at all, or just allow a subset (like name validation) in CRD v1 /sig api-machinery /area custom-resources cc
What is the difference between these two? I mean the end result is the same right? The only thing that can be is validation. Nothing else works unless you are considering extensions?
The former means we drop/hide user-specified metadata schema when we publish CRD OpenAPI. We still validate user-specified constraints on server side, but client-side validation won't see them. The latter enables client-side user-specified metadata validation.
If it's possible to merge with standard metadata schema info, or at the cost of not publishing the standard schema. My inclination would be to publish the complete standard schema for metadata/apiVersion/kind, which we know is complete and accurate, and use and user-specified schema for those fields as addition validations server side


Question: if err != nil { return fmt.Errorf("watch error:%v for volume %v", err, volumeHandle) } <del> var watcherClosed bool ch := watcher.ResultChan() defer func() { if !watcherClosed { watcher.Stop() } }() </del> <ins> ch := watcher.ResultChan() defer watcher.Stop() </ins> for { select { case event, ok := <-ch:

A CSI plugin that the 1.10 controller manager can't detect (which could occur in a number of scenarios, such as when a rollout is being performed or a node is evacuated) seems to retry without backoff. In this case, I have a pod referencing a PVC -PV that references my CSI plugin, but no controller plugin running. The controller is probing every 100ms continuously for a single volume attachment. The controller should backoff. Have not tested with higher numbers of pods, but in a busy system this could cause the controller manager to burn CPU. Also, there isn't actually a great error telling the user why the probing is happening (I knew in this case, but an admin likely has no idea).
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /milestone v1.14 adding it to next milestone as per if it doesn't make it then it can be further moved
as KEP has been moved, unless an exception is made, /milestone v1.15
the linked kep above is unrelated to this issue. This is a bug and I don't think it needs an associated enhancement
sorry, got diverted by the comments on that KEP. /milestone v1.14
Hello from Bug Triage Shadow . I'd like to remind that code freeze is coming in 3 days (Friday, 8th March). Is this still planned to be fixed for 1.14?
We have not been tracking this for 1.14. /assign can you help triage and see if this is still an issue and if there is a quick fix?
I can't reproduce the issue, do you remember what you did? From the logs it seems that Watch channel consistently returns so new is called recursively (beware, 1.10 code, but it's very similar in master): Can it produce fairly stable 100ms delay between calls? Anyway, this is not usual code path, watcher does not return in most cases.


Question: if err != nil { return fmt.Errorf("watch error:%v for volume %v", err, volumeHandle) } <del> var watcherClosed bool ch := watcher.ResultChan() defer func() { if !watcherClosed { watcher.Stop() } }() </del> <ins> ch := watcher.ResultChan() defer watcher.Stop() </ins> for { select { case event, ok := <-ch:

if I read the code correctly, such recursive must end up with timeout - caller does not care about return value and caller's watch probably won't return anything useful. Is there a reason why we don't return error? This will kick in exponential backoff in A/D controller.
clearing the milestone for tracking purposes to let the fix land in its own time /milestone clear
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I found the error would happen when a pv is attahing(volumeattachment has created) and the csi-attacher is deleted at the same time. Then the log would print so frequently (because of recursion).
is this fixed by ?
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> FROM debian:jessie RUN apt-get update RUN apt-get -qy install python-seqdiag make curl WORKDIR /diagrams RUN curl -sLo DroidSansMono.ttf https://googlefontdirectory.googlecode.com/hg/apache/droidsansmono/DroidSansMono.ttf ADD . /diagrams CMD bash -c 'make >/dev/stderr && tar cf - *.png' </ins>  No newline at end of file

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /priority failing-test /area test-infra What happened: job is failing. See https://k8s- https://k8s- https://k8s- https://k8s- What you expected to happen: test should pass
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/assign
Reiterating the mentions to trigger a notification: <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm guessing that something got messed up in the bazel cache when reverting from 0.5.2 to 0.4.5. I'm not sure how best to fix.
How about updating everything to bazel 0.5.3 at once? I know you're trying to do a step-by-step approach to avoid breakages, but I suspect it will be cheaper to just deal with a brief outage while we update? (aka update the bazelbuild image to 0.5.3 and then merge your PR with updated and working BUILD rules?)
there's at least one other issue I've discovered with the latest go rules - the binary no longer works, due to changes introduced in that might be arguably better (at least everything still builds?), though it's frustrating that it makes things more broken in a different way.


Question: export FAIL_ON_GCP_RESOURCE_LEAK="false" - 'gce-flannel': description: 'Run E2E tests on GCE using Flannel and the latest successful build. This suite is quarantined in a dedicated project because Flannel integration is experimental.' <ins> disable_job: true  # Issue #24520 </ins> # We don't really care to enforce a timeout for flannel tests. Any performance issues will show up in the other GCE builders. # This suite is to continuously check that flannel + Kubernetes works. timeout: 120

and show that we run this job every few minutes, but it never passes. I'm going to send out a PR to disable this job until someone has the time to investigate and resolve.
It's failing before tests even start:
At this point I don't think it's a flake, we needed flannel for a release and it's less important now so I'm not investigating why the cluster isn't coming up soon. I do believe if the cluster comes up a majority if not all of the tests will pass, and the ones that don't will flake for known reasons.


Question: "//pkg/api/v1/endpoints:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/api/discovery/v1alpha1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/api/equality:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: EndpointSlice for masters was not created after enabling EndpointSlice on a pre-existing cluster. Only those EndpointSlices managed by kube-controller-manager were created. What you expected to happen: EndpointSlice for masters should be created after enabling EndpointSlice on a pre-existing cluster, the same as the EndpointSlices managed by kube-controller-manager. How to reproduce it (as minimally and precisely as possible): kubeadm to create a cluster: the EndpointSlice feature by editing the kube-apiserver, kube-controller-manager, kube-proxy manifest kube-apiserver and kube-controller-manager is restarted, the EndpointSlice in namespace will be missing. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools:Network plugin and version (if this is a network-related bug): Others:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved /assign


Question: $KUBECFG delete services/redismaster $KUBECFG delete pods/redis-master-2 <del> POD_LIST_2=$($KUBECFG '-template={{range.Items}}{{.Meta}} {{end}}' list pods) </del> <ins> POD_LIST_2=$($KUBECFG '-template={{range.Items}}{{.Name}} {{end}}' list pods) </ins> echo "Pods running after shutdown: ${POD_LIST_2}" exit 0

What happened: is a subdirectory of the btrfs subvolume . kubelet fails to start with the following error: What you expected to happen: kubelet starts normally and doesn't fail with an error message How to reproduce it (as minimally and precisely as possible): Install a system using btrfs as file system for / Create a sub-volume for /var Start kubelet (even in disconnected mode) Anything else we need to know?: This is a follow-up issue of and has been promoted to fatal from onward, before it was a warning PR to fix the issue is Environment: Kubernetes version (use ): forward Cloud provider or hardware configuration: both bare metal and virtual OS (e.g. from /etc/os-release): SUSE 12+ or openSUSE (Tumbleweed/Leap) Kernel (e.g. ): Install tools: SUSE/openSUSE rpm Others:
/sig storage
fixed by /close
If I may - adding some notes - it might be helpful for others, until I get the time to do some more detailed tests and report a bug. I still had issues with kubelet not starting, with the same error - (version 1.13.4) My btrfs setup is that btrfs root is mounted at /, then I have also a subvolume /var and in it a subvolume /var/lib. The issue for me I think comes from the fact that I do not have a flat btrfs setup () but a nested one, and as such, I have as a mountpoint ONLY the root directory. Manually adding a mount entry for /var (heh - I should probably have mounted /var/lib) in /etc/fstab made it work: /var btrfs rw,relatime,ssd,space_cache,subvolid=259,subvol=/var 0 0` Also, even without this additional "intermediate" mount point I think it should have been working. I wonder if in , the loop 'for !found {' isn't exiting a tiny bit too fast (without testing for the root mountpoint itself) . Maybe it should be something like:
Has something changed in the code? Kubelet from kubernetes 1.20.1 is failing. Doing a mount as I described above fixes it.
see (open) and (closed) - apparently this was recently fixed (but isn't released yet).


Question: addBadCheck      bool }{ {"?verbose", "[+]ping oknhealthz check passedn", http.StatusOK, false}, <ins> {"?exclude=dontexist", "ok", http.StatusOK, false}, {"?exclude=bad", "ok", http.StatusOK, true}, {"?verbose=true&exclude=bad", "[+]ping okn[+]bad excluded: oknhealthz check passedn", http.StatusOK, true}, {"?verbose=true&exclude=dontexist", "[+]ping oknwarn: some health checks cannot be excluded: no matches for "dontexist"nhealthz check passedn", http.StatusOK, false}, </ins> {"/ping", "ok", http.StatusOK, false}, {"", "ok", http.StatusOK, false}, {"?verbose", "[+]ping okn[-]bad failed: reason withheldnhealthz check failedn", http.StatusInternalServerError, true},

What would you like to be : Add support to the kube-apiserver so that we can disable health-checks via query params. Why is this needed: Currently, the healthz endpoint is inflexible in that any and all health checks are currently executed when the endpoint is hit. Since the kube-apiserver only currently supports a liveness check, this means all persistent health check failures will eventually result in a kubelet driven restart. However, this may not be desirable behavior if you have additional monitoring services layered on top of your control plane. For instance, take the etcd failure from the perspective of the kube-apiserver. Currently, persistent failures during the etcd health check from the api-server healthz endpoint will cause the kube-apiserver to be restarted. This sometimes helps (in the case of a faulty etcd-client connection) but also can be detrimental (the kube-apiserver restarts until the etcd cluster is functioning, then the kube-apiserver bombards the etcd cluster with requests to update local caches, saturating the etcd cluster with requests, causing it to timeout on its own liveness probes and trigger a kubelet-driven restart). Let's say that we could now disable etcd from our liveness health check. Then if we had an external monitoring service, we could ask independently ask etcd if it was healthy and we could also ask the kube-apiserver if it thought etcd was unhealthy. Based on the combination of those answers, we could enable more intelligent responses to certain situation, like restarting the kube-apiserver when it thought etcd was unhealthy but etcd was actually reporting as healthy but not doing so when etcd was actually unhealthy. This feature is backwards-compatible and opt-in. There would be no impact to existing behavior unless these query params are actually used. /sig api-machinery /kind feature
/cc
SGTM
/assign


Question: Porter              = Config{e2eRegistry, "porter", "1.0"} PortForwardTester   = Config{e2eRegistry, "port-forward-tester", "1.0"} Redis               = Config{e2eRegistry, "redis", "1.0"} <del> ResourceConsumer    = Config{e2eRegistry, "resource-consumer", "1.4"} </del> <ins> ResourceConsumer    = Config{e2eRegistry, "resource-consumer", "1.5"} </ins> ResourceController  = Config{e2eRegistry, "resource-consumer/controller", "1.0"} ServeHostname       = Config{e2eRegistry, "serve-hostname", "1.1"} TestWebserver       = Config{e2eRegistry, "test-webserver", "1.0"}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Used the endpoint /BumpMetric to create fake metrics for Prometheus. When I tried to curl /metrics I got the following response: Bad request. Not a POST request What you expected to happen: I expected to see something like this: # HELP info message. # TYPE gauge 0. How to reproduce it (as minimally and precisely as possible): Following the docs, after having it running inside kubernetes: Anything else we need to know?: Looks like introduced this by mistake. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig testing


Question: func (s *Controller) getNodeConditionPredicate() NodeConditionPredicate { return func(node *v1.Node) bool { <del> // We add the master to the node list, but its unschedulable.  So we use this to filter // the master. if node.Spec.Unschedulable { return false } </del> if s.legacyNodeRoleFeatureEnabled { // As of 1.6, we will taint the master, but not necessarily mark it unschedulable. // Recognize nodes labeled as master, and filter them also, as we were doing previously.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: In order to migrate to a new version of kubernetes on GKE. I cordoned(Kubectl cordon <node) all nodes in the old node pool(1.8.9) in order to drain the node pool(pods on the node) with an intention to cause the pods to run on a new version of the node pool(1.10.2). As soon as I cordoned all the nodes in the old node pool, it caused the downtime of ingress(nginx). The nginx controller pods where running fine on the cordoned node pool. But the cordon caused the nodes to be removed from the target pool of ingress load balancer. Below are the documents that suggest the above same method that I followed as best practice for zero downtime upgrade. 1). 2). What you expected to happen: I expected "kubectl cordon <node" not to have the node/s removed from the target pool of the loadbalancer because the nginx controller pods where running absolutely fine on these nodes. How to reproduce it (as minimally and precisely as possible): 1).Cordon all the nodes in an old node pool that runs nginx controller. Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.2 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Container OS Kernel (e.g. ): 4.14.22+ Install tools: Others: Cordon causes to remove the nodes from the target pool of the ingress load balancer. The health checks fail on the nodes of the target pool, which should not be the case. Because the nginx controller pods run fine on these nodes.
Technically, this isn't a Kubernetes bug as this is working as intended; however, it does beg the question why we remove unschedulable nodes if they're marked as ready since we perform the to determine LB backend nodes. is it scheduable? is it not a master? is it not marked for exclusion? is it marked as ready?


Question: func (s *Controller) getNodeConditionPredicate() NodeConditionPredicate { return func(node *v1.Node) bool { <del> // We add the master to the node list, but its unschedulable.  So we use this to filter // the master. if node.Spec.Unschedulable { return false } </del> if s.legacyNodeRoleFeatureEnabled { // As of 1.6, we will taint the master, but not necessarily mark it unschedulable. // Recognize nodes labeled as master, and filter them also, as we were doing previously.

If any of those are a no, then it'll be removed from the pool on the next sync. Perhaps someone from can shed some light. /sig network
cc This problem extends beyond GCP/ingress controller and to anyone using pools to upgrade nodes. A typical workflow I see is you'd cordon a node pool so as you drain each node the pods don't go to another node in the pool that'll shortly be drained as well. The current logic does not support that and complicates cluster administration because cordoning does not respect PDB's.
/remove-sig apps /remove-sig gcp (since the service controller falls under sig networking)
Those labels are not set on the issue: <detailsIn response to : /remove-sig apps /remove-sig gcp Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Ref I thought a general practice for draining the old pool is to move over one node at a time in a rolling fashion. Shouldn't there always be certain backend pods being able to receive & serve traffic? cc
WRT draining, it is recommended to do it one node at a time ; however, in real world scenarios doing it one node at a time can cause a lot of avoidable container churn if you don't cordon off the nodes you at least plan to drain so that pods aren't being scheduled on a node you'll be draining after the next one. According to the comments in the service controller it seems the unschedulable part exists because it was how we differentiated masters in the past; however, we have labels for that now. We also have the service exclusion label (can't find when/if that'll be promoted though as i don't think there's a ticket on it).
I'm having problems with this issue as well. The fact that nodes are removed from target pools when cordoned means that I sometimes get brief interruptions of the traffic to a service when I cordon a node. I'm guessing the node isn't gracefully removed from the load balancer which results in some traffic being cut off trying to route through the recently cordoned node. The same problem occasionally happens when deleting nodes / node pools or when the cluster autoscaler deletes a node. We are running GKE ver. 1.10.2-gke.1.


Question: func (s *Controller) getNodeConditionPredicate() NodeConditionPredicate { return func(node *v1.Node) bool { <del> // We add the master to the node list, but its unschedulable.  So we use this to filter // the master. if node.Spec.Unschedulable { return false } </del> if s.legacyNodeRoleFeatureEnabled { // As of 1.6, we will taint the master, but not necessarily mark it unschedulable. // Recognize nodes labeled as master, and filter them also, as we were doing previously.

Our best defence against these issue have so far been to limit node changes as much as possible and not use any auto scaling.
I've posted a question regarding this issue in SO some weeks before it was created. The suggested action was to use node taints () instead of cordon, that way nodes will still be marked as .
Thank you for the link, it Certainly helped. Perhaps, if thats the case(as mentioned in the accepted answer of the above link), then to cordon all the nodes of old node pools should not be a recommended action, at least in the GKE upgrade docs.
only says: And I can't find what more it does in the documentation. So I would expect that it has exactly the same behaviour as a taint. Why is removing running pods from the balancer working as intended? If it isn't a bug then the documentation is certainly unclear in this regard.
This just caused a major outage for us too. I agree with the previous posts, while this may be working as the code intended, it does not follow the patterns and documentation k8s provides. When I cordon a node it should not impact the traffic going to the node or any pods on it. I should be able to gradually remove traffic by draining nodes/pods and respecting PDBs.
this issue touches on the fact we are removing masters as well and inadvertently removing cordoned nodes. The logic in the service controller mentions we only filter out unschedulable nodes since that's how masters were setup.
We wanted to replace all of the kubernetes cluster nodes to different node instance types, so we started off by cordoning all of the nodes, but it surprisingly caused an outage for us. As pointed out, the only documentation around cordoning is , which I took to mean that new pods can not be scheduled on those nodes. It was surprising behavior to find out that, not only does it not allow new pods to be scheduled on those nodes, but that it also causes the service controller to remove unschedulable nodes from load balancing pools. I understand that it's working as intended, but if that was also documented as part of the cordon operation, I would have been able to avoid an outage.
I finally caught up to this bug. Fun one. I agree that this is working-as-intended and that it stinks. We can do better.


Question: func (s *Controller) getNodeConditionPredicate() NodeConditionPredicate { return func(node *v1.Node) bool { <del> // We add the master to the node list, but its unschedulable.  So we use this to filter // the master. if node.Spec.Unschedulable { return false } </del> if s.legacyNodeRoleFeatureEnabled { // As of 1.6, we will taint the master, but not necessarily mark it unschedulable. // Recognize nodes labeled as master, and filter them also, as we were doing previously.

As a short-term workaround, you can taint the nodes before cordoning one-by-one. That will stop new work from arriving, but will leave old work there. If we agree on an appropriately named taint, we can have nginx tolerate it by default. Am I missing any reason why this would not work? If that flies, we should fix docs ( those are GCP docs). I am happy to help with that. Longer term maybe we can do something smarter. I am not sure what that would be, just now. Ideas?
Is there any reason we can't just leave the nodes in the load balancer pool when the node is cordoned? And when the pods are terminated the nodes can be removed? I may be missing some context here?
The presumption is that a node that is unschedulable is going away, or needs some other repair. Using it as a target for load-balancers means there could be sessions routed THROUGH that node. Even as the pods are drained. Right up until the node suddenly dies, causing an outage of a different flavor. What should really happen is that a node which becomes unschedulable starts draining connections. Once all the pods are moved off, the node can be removed from the pool entirely. I don't think any cloud providers, even GCP, handle this just right today. David Radcliffe <:
Wait, if you brought up a new nodepool, all those new nodes should be in the TargetPool. Cordoning the old nodepool should only terminate the existing connections, new connections should be fine. Something doesn't add up...
Was this using
Yes
Right, well, that explains it. So even draining won't do what you want - you need new connections to be allowed until you have capacity in the new pool. The good news is that you can do this entirely yourself, with taints. We just need nginx to tolerate the taint, which you can also specify yourself. The only real alternative I see would be, as you said, to leave nodes in the LB pool until the node dies. Given that we never specified this, it is probably different on every provider, which would make it hard to change (especially if the provider already provides clean drains while unschedulable).