Question: command := exec.Command("umount", target) output, err := command.CombinedOutput() if err != nil { <del> if err.Error() == errNoChildProcesses { if command.ProcessState.Success() { // We don't consider errNoChildProcesses an error if the process itself succeeded (see - k/k issue #103753). return nil } // Rewrite err with the actual exit error of the process. err = &exec.ExitError{ProcessState: command.ProcessState} } if mounter.withSafeNotMountedBehavior && strings.Contains(string(output), errNotMounted) { klog.V(4).Infof("ignoring 'not mounted' error for %s", target) return nil } return fmt.Errorf("unmount failed: %vnUnmounting arguments: %snOutput: %s", err, target, string(output)) </del> <ins> return checkUmountError(target, command, output, err, mounter.withSafeNotMountedBehavior) </ins> } return nil }

A status of Pod mounted NFS PV remain as terminating when a node shutdown and restart. A Pod is completely terminated. Deploy a Pod mounted NFS PV Shutdown the node where the Pod is running Wait for 5 min and the Pod status become Restart the node The Pod status is still and the Pod is never terminated completely <detailsPrepare a NFS Server.(I used ) Deploy a Pod mounted NFS PV like this. Shutdown the node where the Pod is running.(In this case ) And the status of node02 becomes . After 5 min later(by default), the Pod status becomes . (The time a Pod becomes status depends on .) Then, restart the node and the status of node02 becomes . But the Pod status is still and the Pod is never terminated completely. It seems that a faild to delete Pod because of trying to unmount a not mounted directory. Force delete Pod. Delete Pod's directory on the node. </detailsNo response <details</details<detailsnone(on premise) </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig node
It dose not happen using , so it seems to depend on NFS volume plugin.
/assign
I try to build and run kubelet with comment out this logic. Then it seems to get well. related: This change checks the result of instead of the mount point check. However, check logic is not implemented in and (these functions are used in NFS volume plugin), despite it is done in .
/remove-sig node /sig storage
Reopening, fix had to be reverted due to unit tests requiring root permissions ()
I am terribly sorry... Please wait a moment. I'll try to fix it.
no worries, just get to review the rework, making sure the units tests don't require root permissions
Thank you! Fixed in .
Are there any plans to backport this to older Kubernetes versions? This seems like a critical fix.


Question: } } } <ins> func TestFairEvaluationForNodes(t *testing.T) { defer algorithmpredicates.SetPredicatesOrderingDuringTest(order)() predicates := map[string]algorithmpredicates.FitPredicate{"true": truePredicate} numAllNodes := 500 nodeNames := make([]string, 0, numAllNodes) for i := 0; i < numAllNodes; i++ { nodeNames = append(nodeNames, strconv.Itoa(i)) } nodes := makeNodeList(nodeNames) g := makeScheduler(predicates, nodes) // To make numAllNodes % nodesToFind != 0 g.percentageOfNodesToScore = 30 nodesToFind := int(g.numFeasibleNodesToFind(int32(numAllNodes))) // Iterating over all nodes more than twice for i := 0; i < 2*(numAllNodes/nodesToFind+1); i++ { nodesThatFit, _, _, err := g.findNodesThatFit(context.Background(), framework.NewCycleState(), &v1.Pod{}) if err != nil { t.Errorf("unexpected error: %v", err) } if len(nodesThatFit) != nodesToFind { t.Errorf("got %d nodes filtered, want %d", len(nodesThatFit), nodesToFind) } if g.nextStartNodeIndex != (i+1)*nodesToFind%numAllNodes { t.Errorf("got %d lastProcessedNodeIndex, want %d", g.nextStartNodeIndex, (i+1)*nodesToFind%numAllNodes) } } } </ins>

What would you like to be : A unit test that verifies the evaluation order of the nodes in the scheduler. Why is this needed: The scheduler evaluates the nodes in a specific order until it finds a pre-defined minimum number of feasible nodes. To give all nodes a fair chance of being evaluated, in each scheduling cycle, the scheduler starts evaluating from where it left off in the previous cycle. This is captured by the parameter in  We need to add a unit test that captures regressions related to this behavior to avoid bugs like /sig scheduling /priority important-soon /help
/assign
/remove-help


Question: } } <ins> // in older kernels (< 3.18), the destination address family attribute doesn't exist so we must // assume it based on the destination address provided. if d.AddressFamily == 0 { // we can't check the address family using net stdlib because netlink returns // IPv4 addresses as the first 4 bytes in a []byte of length 16 where as // stdlib expects it as the last 4 bytes. addressFamily, err := getIPFamily(addressBytes) if err != nil { return nil, err } d.AddressFamily = addressFamily } </ins> // parse Address after parse AddressFamily incase of parseIP error if addressBytes != nil { ip, err := parseIP(addressBytes, d.AddressFamily)

Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error.


Question: } } <ins> // in older kernels (< 3.18), the destination address family attribute doesn't exist so we must // assume it based on the destination address provided. if d.AddressFamily == 0 { // we can't check the address family using net stdlib because netlink returns // IPv4 addresses as the first 4 bytes in a []byte of length 16 where as // stdlib expects it as the last 4 bytes. addressFamily, err := getIPFamily(addressBytes) if err != nil { return nil, err } d.AddressFamily = addressFamily } </ins> // parse Address after parse AddressFamily incase of parseIP error if addressBytes != nil { ip, err := parseIP(addressBytes, d.AddressFamily)

I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO.


Question: } } <ins> // in older kernels (< 3.18), the destination address family attribute doesn't exist so we must // assume it based on the destination address provided. if d.AddressFamily == 0 { // we can't check the address family using net stdlib because netlink returns // IPv4 addresses as the first 4 bytes in a []byte of length 16 where as // stdlib expects it as the last 4 bytes. addressFamily, err := getIPFamily(addressBytes) if err != nil { return nil, err } d.AddressFamily = addressFamily } </ins> // parse Address after parse AddressFamily incase of parseIP error if addressBytes != nil { ip, err := parseIP(addressBytes, d.AddressFamily)

However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).


Question: } } <ins> // in older kernels (< 3.18), the destination address family attribute doesn't exist so we must // assume it based on the destination address provided. if d.AddressFamily == 0 { // we can't check the address family using net stdlib because netlink returns // IPv4 addresses as the first 4 bytes in a []byte of length 16 where as // stdlib expects it as the last 4 bytes. addressFamily, err := getIPFamily(addressBytes) if err != nil { return nil, err } d.AddressFamily = addressFamily } </ins> // parse Address after parse AddressFamily incase of parseIP error if addressBytes != nil { ip, err := parseIP(addressBytes, d.AddressFamily)

FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks


Question: }) } <ins> // Wait for a pod to disappear by listing them explicitly. func waitForJobsPodToDisappear(c clientset.Interface, ns string, targetJob *batchv1.Job) error { return wait.Poll(framework.Poll, cronJobTimeout, func() (bool, error) { options := metav1.ListOptions{LabelSelector: fmt.Sprintf("controller-uid=%s", targetJob.UID)} pods, err := c.CoreV1().Pods(ns).List(options) if err != nil { return false, err } return len(pods.Items) == 0, nil }) } </ins> // Wait for a job to be replaced with a new one. func waitForJobReplaced(c clientset.Interface, ns, previousJobName string) error { return wait.Poll(framework.Poll, cronJobTimeout, func() (bool, error) {

What happened: There seems to be a regression after 1.13 release. A Cronjob with default settings does not clean up completed pods. The job themselves are getting deleted as expected, but the pods are left behind. This seems to be related to What you expected to happen: Expected the pods for now non-existent child jobs of a cronjob to be cleaned up. This is the way it used to work prior to 1.13/ How to reproduce it (as minimally and precisely as possible): run on this manifest: Wait, more than 3 minutes, observe, that while only 3 successful jobs are present, the pods from now deleted jobs are left behind and never get deleted. Anything else we need to know?: pointed out the following relevant places in the code base: deleting a job orphans pods by default and the cronjob controller doesn't do anything different than the default when deleting Environment: Kubernetes version (use ): 1.13.0 Cloud provider or hardware configuration: ubuntu VM in vSphere OS (e.g. from /etc/os-release): 16.04.5 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-135-generic Install tools: kubeadm <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig apps
Should include a known issues in 1.13.0 as well /priority critical-urgent
Lemme see if I can easily reproduce this.
/milestone v1.13
Fixes are in (master) and (1.13).


Question: testName:           "does not exist by provider id", existsByProviderID: false, providerIDErr:      nil, <del> existsByNodeName:   false, </del> <ins> hasInstanceID:      true, </ins> nodeNameErr:        errors.New("unimplemented"), expectedCalls:      []string{"instance-exists-by-provider-id"}, <ins> expectedNodeExists: false, </ins> node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name: "node0",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The change here removes exiternal id, but the call to instance id was not made used at all, it will always return false. At this time no node was able to register What you expected to happen: ProviderId was used in check How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment:
similar to are you using --cloud-provider in your controller manager? are you also setting it on your kubelet?
I'm using acs-engine to deploy the cluster. Config is as following: I've proposed a fix. PTAL
/sig cluster-lifecycle


Question: return &runtimeapi.ListPodSandboxResponse{Items: result}, nil } <del> // applySandboxLinuxOptions applies LinuxPodSandboxConfig to dockercontainer.HostConfig and dockercontainer.ContainerCreateConfig. func (ds *dockerService) applySandboxLinuxOptions(hc *dockercontainer.HostConfig, lc *runtimeapi.LinuxPodSandboxConfig, createConfig *dockertypes.ContainerCreateConfig, image string, separator rune) error { if lc == nil { return nil } // Apply security context. if err := applySandboxSecurityContext(lc, createConfig.Config, hc, ds.network, separator); err != nil { return err } // Set sysctls. hc.Sysctls = lc.Sysctls return nil } </del> func (ds *dockerService) applySandboxResources(hc *dockercontainer.HostConfig, lc *runtimeapi.LinuxPodSandboxConfig) error { hc.Resources = dockercontainer.Resources{ MemorySwap: DefaultMemorySwap(),


Question: "google.golang.org/cloud/compute/metadata" ) <del> const EXTERNAL_IP_METADATA_URL = "http://169.254.169.254/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip" </del> // GCECloud is an implementation of Interface, TCPLoadBalancer and Instances for Google Compute Engine. type GCECloud struct { service          *compute.Service

When we populate the addresses of the nodes in we call NodeAddresses, which for the gce provider gets the external IP from the which as expected puts the IP address of the master into the external IP field. The Legacy IP field is still filled in by asking the cloud provider for the instance data, so it contains the public IP of the node. /cc
If we merge this should get fixed, because the external IPs will be calculated on the nodes rather than on the master.
Since we are going to delay adding until I can rebase on top of it seems like it's worth fixing this separately.
This is blocking so assigning to him.


Question: pluginDir: pluginDir, watcher:   watcher, fs:        fs, <del> factory:   fakePluginFactory{error: false}, </del> <ins> factory:   fakePluginFactory{}, </ins> } prober.Init()

When kubelet is running if you create two volume plugin directories here: /usr/libexec/kubernetes/kubelet-plugins/volume/exec /usr/libexec/kubernetes/kubelet-plugins/volume/exec/cifs /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds You will see these expected logs: Now if you copy over the cifs plugin file to cifs directory and then copy the uds plugin the errors go away. The pods using the uds volume will function but any pods attempting to use the cifs volume will fail with the following error from kubelet: desiredStateOfWorld. err=failed to get Plugin from volumeSpec for volume "pv" err=no volume plugin matched Restarting kubelet fixes this problem as then both plugins are present and properly . A functional volume plugin should be to kubelet even if another volume plugin is missing or invalid. Fixing the invalid volume plugin currently only adds the one that was invalid and fixed and other valid plugins that were previously present are skipped. Start kubelet service on a node. Create two directories for two flex volume plugins in the /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ directory. Eg. nodeagent~uds and cifs Kubelet logs will now start complaining that the plugins are missing. Add the files to the plugin folders and then start a pod using the plugin. Notice that only the last plugin file copied is available to be used by pods. The first plugin will generate this error if used by a volume "err=no volume plugin matched" Performing a touch on the folder of the missing plugin will cause kubelet to properly index it. <details</details<detailsLocal kube cluster </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig k8s-infra
/sig storage /remove-sig k8s-infra


Question: framework.ExpectEqual(true, match, "failed to find a valid version for PreferredVersion") } }) <ins> ginkgo.It("should locate the groupVersion and a resource within each APIGroup", func(ctx context.Context) { tests := []struct { apiBasePath   string apiGroup      string apiVersion    string validResource string }{ { apiBasePath:   "/api", apiGroup:      "", apiVersion:    "v1", validResource: "namespaces", }, { apiBasePath:   "/apis", apiGroup:      "admissionregistration.k8s.io", apiVersion:    "v1", validResource: "validatingwebhookconfigurations", }, { apiBasePath:   "/apis", apiGroup:      "apiextensions.k8s.io", apiVersion:    "v1", validResource: "customresourcedefinitions", }, { apiBasePath:   "/apis", apiGroup:      "apiregistration.k8s.io", apiVersion:    "v1", validResource: "apiservices", }, { apiBasePath:   "/apis", apiGroup:      "apps", apiVersion:    "v1", validResource: "deployments", }, { apiBasePath:   "/apis", apiGroup:      "authentication.k8s.io", apiVersion:    "v1", validResource: "tokenreviews", }, { apiBasePath:   "/apis", apiGroup:      "authorization.k8s.io", apiVersion:    "v1", validResource: "selfsubjectaccessreviews", }, { apiBasePath:   "/apis", apiGroup:      "autoscaling", apiVersion:    "v1", validResource: "horizontalpodautoscalers", }, { apiBasePath:   "/apis", apiGroup:      "autoscaling", apiVersion:    "v2", validResource: "horizontalpodautoscalers", }, { apiBasePath:   "/apis", apiGroup:      "batch", apiVersion:    "v1", validResource: "jobs", }, { apiBasePath:   "/apis", apiGroup:      "certificates.k8s.io", apiVersion:    "v1", validResource: "certificatesigningrequests", }, { apiBasePath:   "/apis", apiGroup:      "coordination.k8s.io", apiVersion:    "v1", validResource: "leases", }, { apiBasePath:   "/apis", apiGroup:      "discovery.k8s.io", apiVersion:    "v1", validResource: "endpointslices", }, { apiBasePath:   "/apis", apiGroup:      "events.k8s.io", apiVersion:    "v1", validResource: "events", }, { apiBasePath:   "/apis", apiGroup:      "networking.k8s.io", apiVersion:    "v1", validResource: "ingresses", }, { apiBasePath:   "/apis", apiGroup:      "node.k8s.io", apiVersion:    "v1", validResource: "runtimeclasses", }, { apiBasePath:   "/apis", apiGroup:      "policy", apiVersion:    "v1", validResource: "poddisruptionbudgets", }, { apiBasePath:   "/apis", apiGroup:      "scheduling.k8s.io", apiVersion:    "v1", validResource: "priorityclasses", }, { apiBasePath:   "/apis", apiGroup:      "storage.k8s.io", apiVersion:    "v1", validResource: "csinodes", }, } for _, t := range tests { resourceList := &metav1.APIResourceList{} apiPath := path.Join(t.apiBasePath, t.apiGroup, t.apiVersion) ginkgo.By(fmt.Sprintf("Requesting APIResourceList from %q", apiPath)) err := f.ClientSet.Discovery().RESTClient().Get().AbsPath(apiPath).Do(ctx).Into(resourceList) framework.ExpectNoError(err, "Fail to access: %s", apiPath) gomega.Expect(resourceList.GroupVersion).To(gomega.Equal((schema.GroupVersion{Group: t.apiGroup, Version: t.apiVersion}).String())) foundResource := false for _, r := range resourceList.APIResources { if t.validResource == r.Name { foundResource = true break } } gomega.Expect(foundResource).To(gomega.BeTrue(), "Resource %q was not found inside of resourceListn%#v", t.validResource, resourceList.APIResources) } }) </ins> })

[X] test approval issue : [X] test pr : 3 May 2023 [X] two weeks soak end date : 16 May 2023 [X] test promotion pr : [X] remove endpoint form file According to following two APIsnoop queries, there are still 12 APIResources endpoints that are not tested for Conformance. Using a number of existing e2e test practices a new has been created to address these 12 endpoints. The e2e logs for this test are listed below. This query shows the following apiresources endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 12 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/open For tracking the rest of the processes
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <del> // SetRemove remove value from the set if value existed func (q *UniqueQueue) SetRemove(value string) { q.lock.Lock() defer q.lock.Unlock() if q.set.Has(value) { q.set.Delete(value) } } </del> // RateLimitedTimedQueue is a unique item priority queue ordered by // the expected next time of execution. It is also rate limited. type RateLimitedTimedQueue struct {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Nodelifecycle controller does not set NoExecute taint to NotReady node when kubelet has been stopped What you expected to happen: effect: NoExecute key: Nodelifecycle controller sets NoExecute taint to NotReady node when kubelet has been stopped How to reproduce it (as minimally and precisely as possible): systemctl stop kubelet Anything else we need to know?: I use default args to start kube-controller-manager. I see log from kube-controller-manager: Environment: Kubernetes version (use ): [root ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): [root ~]# cat /etc/os-release NAME="CentOS Linux" VERSION="8 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="8" PLATFORMID="platform:el8" PRETTYNAME="CentOS Linux 8 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:8" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-8" CENTOSMANTISBTPROJECTVERSION="8" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="8" Kernel (e.g.


Question: } } <del> // SetRemove remove value from the set if value existed func (q *UniqueQueue) SetRemove(value string) { q.lock.Lock() defer q.lock.Unlock() if q.set.Has(value) { q.set.Delete(value) } } </del> // RateLimitedTimedQueue is a unique item priority queue ordered by // the expected next time of execution. It is also rate limited. type RateLimitedTimedQueue struct {

): [root ~]# uname -a Linux liabio 4.18.0-147.5.1.el81.x8664 SMP Wed Feb 5 02:00:39 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle /wg cluster-lifecycle
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-cluster-lifecycle /wg sig-cluster-lifecycle /committee sig-cluster-lifecycle Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig cluster-lifecycle /sig node
Those labels are not set on the issue: <detailsIn response to : /remove-sig cluster-lifecycle /sig node </details
is NoExecute the expected taint for this condition?
- yes it is when running the taint-manager: evictions are usually safe eviction as we don't force delete pods
ref
I see the same, a node which has node condition set to Unknown, does not get an Unreachable taint with NoExecute effect. However I only see this on 1.19, works fine till 1.18.10 Till 1.18.10, such a node gets two Unreachable taints, one with effect NoExecute and one with effect NoSchedule. But on 1.19.3 it only gets the NoSchedule taint. I see that this is the only commit made in that area between 1.18 and 1.19: , could be related?
It totally breaks production readiness of all recent releases, yet doesn't seem to be regarded as a top priority issue. WTF?


Question: source "${KUBE_ROOT}/hack/lib/init.sh" kube::golang::setup_env <ins> kube::golang::setup_gomaxprocs </ins> # start the cache mutation detector by default so that cache mutators will be found KUBE_CACHE_MUTATION_DETECTOR="${KUBE_CACHE_MUTATION_DETECTOR:-true}"

fails in build environments without network connection due to missing in-tree vendored dependency . Build to succeed with without needing to download any go dependencies. In a network-less build environment, run . No response Building off of the v1.28.0-rc.0 tag N/A <details</detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
This is a hack/tools dependency here by , he may have more context
hack/tools aren't vendored. if you want to ensure those are available offline, will do that
My hack for building with OBS (), see
Is this actually a bug?
in I agree we probably don't want to calls in the build path... in code generation or verify tools, that's more ok, but the build target of a particular SHA should be able to execute hermetically
opened to scope down when we do this to test targets, and avoid the install if the env already explicitly set GOMAXPROCS
fix merged to master and head of release-1.28 branch


Question: local group_versions=() local input_pkgs=() <del> while read -r file; do dir="$(dirname "${file}")" </del> <ins> while read -r dir; do </ins> pkg="$(cd "${dir}" && GO111MODULE=on go list -find .)" leaf="$(basename "${dir}")" if grep -E -q '^v[0-9]+((alpha|beta)[0-9]+)?$' <<< "${leaf}"; then

function in script has following issues: function only supports file but we have multiple types file with name as in . Hence this function is not working in our case. For the mitigation there is already an open . I have asked for including our changes as well. We have multiple files under a for example and . Because of this for appears twice and it generates duplicate informers We only need unique elements before passing it to . I followed this approach for unique elements: We have packages under . Earlier we were generating client only for and . But with the new I am not able to do that. I am using the following call to function: It results in following flags to As there is no directory as , it doesnot generate anything. Is it still possible to call on specific folder as it was done with earlier scripts? One mitigation I can think of is to call function on whole and then remove the marker from the types for which genclient is not needed. But the issue is we won't be able to pass the flag for each pkg. function in script is not working as expected.
/sig api-machinery
/cc
/triage accepted
Hi . I knew that there were all sorts of clients out there doing all sorts of things with these scripts, so I am not surprised to find you. :) Do you have a PR which attempts to use kube_codegen, so I can repro your issues directly?
I think you have one too many in there Aside from that, the real problem here is that these old codegen tools make some deeeeeeep assumptions about the ability to string-join various arguments and get the right results. It's awful. I was trying to make it simpler in this script, which I did for all of the in-tree consumers, but out of tree people apparently don't always follow the same conventions. For example, we generate clients for every directory that requests it. For a reason I do not know, you have files like which say but you don't actually want a client? If you specify then it t
So this code expects that you generate an omnibus client. What if I add a flag, like which you can use to call it for each API to make distinct clients? In truth the codegen tools need to be fixed, but that has SO MUCH ripple I just can't right now.


Question: local group_versions=() local input_pkgs=() <del> while read -r file; do dir="$(dirname "${file}")" </del> <ins> while read -r dir; do </ins> pkg="$(cd "${dir}" && GO111MODULE=on go list -find .)" leaf="$(basename "${dir}")" if grep -E -q '^v[0-9]+((alpha|beta)[0-9]+)?$' <<< "${leaf}"; then

As the codegen was breaking us at several step, hence didn't prepared the PR.
Thanks for the PR. I have one more concern, is it possible to call and client-gen on the same pkg? With the current script, that is not possible because of the line It removes all the generated files first and then calls the . Hence the later clientset gen will remove the earlier generated clients.
Why are you generating a client for an internal version that isn't served?
My mistake. We need to stop using clients for internal versions.


Question: "    Driver:t%vn"+ "    FSType:t%vn"+ "    SecretRef:t%vn"+ <del> "    ReadOnly:t%vn", </del> <ins> "    ReadOnly:t%vn"+ </ins> "    Options:t%vn", flex.Driver, flex.FSType, flex.SecretRef, flex.ReadOnly, flex.Options) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: when i create PersistentVolume, yaml like that: than describe it, got : What you expected to happen: Like that: How to reproduce it (as minimally and precisely as possible): What's said in Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig storage
Already got PR to fix this. /assign


Question: return cmdutil.AddSourceToErr(fmt.Sprintf("retrieving modified configuration from:n%vnfor:", info), info.Source, err) } <ins> // Print object only if output format other than "name" is specified printObject := len(output) > 0 && !shortOutput </ins> if err := info.Get(); err != nil { if !errors.IsNotFound(err) { return cmdutil.AddSourceToErr(fmt.Sprintf("retrieving current configuration of:n%vnfrom server for:", info), info.Source, err)

Is this a BUG REPORT or FEATURE REQUEST?: Bug Report /kind bug What happened: Applying a file that has no changes compared to current will output something like , which muddles the output when using What you expected to happen: Using the -o flag should suppress the unchanged messages in stdout to prevent output from being muddled How to reproduce it (as minimally and precisely as possible): Given the file Will result in proper yaml being generated as expected Run again, and is outputted to stdout, and is invalid yaml Anything else we need to know?: I do need to check whether this still affects the most recent version kubectl Environment: Kubernetes version (use ): 1.8.6 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Darwin Install tools: Others:
confirmed on latest stable version (1.9.2)
Reiterating the mentions to trigger a notification: <detailsIn response to : confirmed on latest stable version (1.9.2) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have a solution to this, still waiting on my company to look over the agreement


Question: // Both names occurred in practice. goleak.IgnoreTopFunction("k8s.io/kubernetes/vendor/gopkg.in/natefinch/lumberjack%2ev2.(*Logger).millRun"), goleak.IgnoreTopFunction("gopkg.in/natefinch/lumberjack%2ev2.(*Logger).millRun"), <ins> // If there is an error during connection shutdown, SPDY keeps a // goroutine around for ten minutes, which gets reported as a leak // (the goroutine is cleaned up after the ten minutes). // // https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/moby/spdystream/connection.go#L732-L744 // // Ignore this reported leak. goleak.IgnoreTopFunction("github.com/moby/spdystream.(*Connection).shutdown"), </ins> ) stop, err := startEtcd(nil)

master-blocking: integration-master- This has been occasionally flaking for some time. Recent Failures: Another leak was fixed at /sig api-machinery /sig testing
/retitle [Flaking Test] integration-master (goroutine leak detection)
Some server is not stopping gracefully on that test
Similar reason:
Its either not terminating gracefully or there is a request that isn't being being served in a timely manner, because I also see a bunch of goroutines waiting on the netpoller.
/triage accepted (for SIG api-machinery)
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accepted (for SIG api-machinery) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this would be a release blocker, still can you confirm whether this would block the upcoming alpha 1 release cut?
Another failure:
Hey , The deadline for the alpha 1 release cut has been delayed to Monday, June 10 and we are still waiting for confirmation of this issue. Do you think this will be a release blocker?
/assign
/assign
integration test flake, we believe this PR fixes it: I'm not marking this fixed because it also mentions a integration test flake.


Question: // GetObjectMetricReplicas calculates the desired replica count based on a target metric utilization (as a milli-value) // for the given object in the given namespace, and the current replica count. <del> func (c *ReplicaCalculator) GetObjectMetricReplicas(currentReplicas int32, targetUtilization int64, metricName string, namespace string, objectRef *autoscaling.CrossVersionObjectReference) (replicaCount int32, utilization int64, timestamp time.Time, err error) { </del> <ins> func (c *ReplicaCalculator) GetObjectMetricReplicas(currentReplicas int32, targetUtilization int64, metricName string, namespace string, objectRef *autoscaling.CrossVersionObjectReference, selector labels.Selector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { </ins> utilization, timestamp, err = c.metricsClient.GetObjectMetric(metricName, namespace, objectRef) if err != nil { return 0, 0, time.Time{}, fmt.Errorf("unable to get metric %s: %v on %s %s/%s", metricName, objectRef.Kind, namespace, objectRef.Name, err)

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: We are starting to use the HPA (horizontal pod autoscaling) for one of our components. What is special about this particular component is that the warm up period is very long (sometimes even more than 30 minutes). During that time, the pod is in Running state but its containers are in Not Ready state. This pod is of course not able to receive any traffic so the metric we use in the HPA is ignoring those pods that are not Ready. The equation used by HPA to calculate the new number of replicas is as follows Our problem is that the equation used by HPA to calculate the new number of replicas is including in the "current replica count" the pods that are "Not Ready" so the value for "new replica count" is not correct. Since this warm up is so long we end up by having a lot of more instances of our services of what it should be necessary. They eventually get deleted once gradually the metrics start going down but that will take a lot of time and a big waste of resources without mentioning that with a changing traffic this HPA will never stabilize What you expected to happen: I would like to have a flag somewhere where I tell HPA to don't include in the "current replica count" the pods that are in "Not Ready" state How to reproduce it (as minimally and precisely as possible): Have a component that takes a lot of time to start and being able to receive traffic. Then define a metric that will ignore the pods that are Not Ready. Finally create a metric that will scale that component based on the mentioned metric Anything else we need to know?: Environment: Kubernetes version 1.7 Cloud provider or hardware configuration: Azure
/sig autoscaling
Hi - thanks for reaching out about this! It looks like this section of the , handles when pods aren't ready. I think you this behavior in , so I'm interested in your thoughts on whats best here. If we do end up adding a flag for this, I'd be happy to write the code :) Also, purely for my own curiosity, I'm curious what type of pod is taking 30 minutes to become ready. If your traffic pattern varies at a rate faster than 30 minutes, it may be difficult for auto-scaling to be helpful.
can you also share a bit more about your HPA?


Question: // GetObjectMetricReplicas calculates the desired replica count based on a target metric utilization (as a milli-value) // for the given object in the given namespace, and the current replica count. <del> func (c *ReplicaCalculator) GetObjectMetricReplicas(currentReplicas int32, targetUtilization int64, metricName string, namespace string, objectRef *autoscaling.CrossVersionObjectReference) (replicaCount int32, utilization int64, timestamp time.Time, err error) { </del> <ins> func (c *ReplicaCalculator) GetObjectMetricReplicas(currentReplicas int32, targetUtilization int64, metricName string, namespace string, objectRef *autoscaling.CrossVersionObjectReference, selector labels.Selector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { </ins> utilization, timestamp, err = c.metricsClient.GetObjectMetric(metricName, namespace, objectRef) if err != nil { return 0, 0, time.Time{}, fmt.Errorf("unable to get metric %s: %v on %s %s/%s", metricName, objectRef.Kind, namespace, objectRef.Name, err)

What kind of metric type is it? Based on the description I'm suspecting that it is an Object metric, am I right?
So answer about why our process takes that much time to start. Basically this is a server that in order to do what he needs to do, he needs to load a lot of data in memory (several GBs) and then perform some warm up that are pretty intensive. The idea is that he doesn't need to grab (almost) anything from outside to attend a request so it's latency is extremely low (in the order of millisecond) By the way. Because of this we have another problem which is that there is not an specific flag per HPA object that we could tell k8s to don't try to scale up again passed some time so the new pods have enough time to get ready before the next scale happens. I know that there is a cluster wide flag but we will need to have a different one by resource that we scale. Now to answer You pointed another interesting topic. The metric we use is % Utilization. We are able to calculate the level of utilization of our pods and we want to keep this metric in an specific value. About the flag. I suggested the flag so it doesn't change the existing behavior by default but I honestly think that the default behavior should be to don't include the NonReady pods as part of the calculation
do you think we can add this change to 1.10? it's critical bug/feature for us and we are happy to help with code/testing.
It's really late for 1.10, especially since we don't have any code here, and I'd like to know more about what's going on here, and especially see if it reproduces on Kubernetes 1.9 (we've had HPA controller bugs in the past). FWIW, currently unready pods are treated as consuming 0% of their utilization on scale up, and it was intended that this should prevent such issues. That is, the formula is actually: If the pods are marked as not ready, I find it strange that they would be affecting the scaling that much, unless your other pods are under real heavy load (i.e. I would assume that the unready pods would skew the scaling back down so that the number of pods just stayed constant).
I believe there is different code branch here.


Question: // GetObjectMetricReplicas calculates the desired replica count based on a target metric utilization (as a milli-value) // for the given object in the given namespace, and the current replica count. <del> func (c *ReplicaCalculator) GetObjectMetricReplicas(currentReplicas int32, targetUtilization int64, metricName string, namespace string, objectRef *autoscaling.CrossVersionObjectReference) (replicaCount int32, utilization int64, timestamp time.Time, err error) { </del> <ins> func (c *ReplicaCalculator) GetObjectMetricReplicas(currentReplicas int32, targetUtilization int64, metricName string, namespace string, objectRef *autoscaling.CrossVersionObjectReference, selector labels.Selector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { </ins> utilization, timestamp, err = c.metricsClient.GetObjectMetric(metricName, namespace, objectRef) if err != nil { return 0, 0, time.Time{}, fmt.Errorf("unable to get metric %s: %v on %s %s/%s", metricName, objectRef.Kind, namespace, objectRef.Name, err)

metric definition { "type": "Object", "object": { "target": { "kind": "Service", "name": "prometheus-exporter" }, "metricName": "PercentUtilization", "targetValue": "20" } } which should handle as and doesn't check Pod state.
I think proper handling of unready pods is a difficult problem that will require significant work. I suspect we may find out we need to start distinguishing pods that are booting up from pods that are 'broken' and it took us quite a while to solve analogous problem with nodes in Cluster Autoscaler (to be clear: I don't know if we will have to do that, that's just my intuition for now). That being said I agree the formula in GetObjectMetricReplicas will generally lead to scaling-up too much if the pod takes longer to initialize then backoff period in HPA (3 minutes). Assume it takes pod 4 minutes to boot up and we're currently running 2 pods. Metric value is 20 and target is 10. In this case: scales-up to 4 pods (20/10 utilization 2 pods). are starting up, they're not doing anything yet. 3 minutes later HPA scales-up to 8 pods (20/10 utilization 4 pods). If we only counted ready pods the scale-up in 3 would not occur, because the desired replicas would still be 4 (20/10 * 2). My intuition would be that this is the expected behavior of HPA. Also if there is no space in the cluster we will just keep scaling-up until we hit max replicas. The only case where current behavior would be more correct than filtering out unready pods in this formula is if someone uses a metric that is calculated based on total number of replicas. Do we advertise that as the right way to use HPA? If we add to Object metrics in 1.11 this usage pattern will stop making sense. To sum up - I think we will need to change the formula to only count ready pods at some point. I can see how current behavior could be a problem in any use-case except the one described in paragraph above. I'm inclined to agree with and provide some way to change the behavior in 1.10 (a flag or an annotation?), treating it as a bug fix.


Question: // GetObjectMetricReplicas calculates the desired replica count based on a target metric utilization (as a milli-value) // for the given object in the given namespace, and the current replica count. <del> func (c *ReplicaCalculator) GetObjectMetricReplicas(currentReplicas int32, targetUtilization int64, metricName string, namespace string, objectRef *autoscaling.CrossVersionObjectReference) (replicaCount int32, utilization int64, timestamp time.Time, err error) { </del> <ins> func (c *ReplicaCalculator) GetObjectMetricReplicas(currentReplicas int32, targetUtilization int64, metricName string, namespace string, objectRef *autoscaling.CrossVersionObjectReference, selector labels.Selector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { </ins> utilization, timestamp, err = c.metricsClient.GetObjectMetric(metricName, namespace, objectRef) if err != nil { return 0, 0, time.Time{}, fmt.Errorf("unable to get metric %s: %v on %s %s/%s", metricName, objectRef.Kind, namespace, objectRef.Name, err)

WDYT
Ah, yeah, I missed that it was object metrics before. I'm not entirely convinced we need a flag here -- it might just make sense to bring Object metrics in line with all the other metrics types. On the other hand, there's a subtlety here -- the other metric types don't quite ignore unready pods -- they just ignore the metrics from unready pods, which causes a heavy bias in the direction of not scaling, but an overwhelming spike can still cause scaling. That behavior is not as achievable using object metrics. I'm inclined to just ignore that subtlety, though, and say that object metrics should ignore unready pods entirely.
I think this is what we should do long term. I was worried about breaking someone who relies on targetValue + averaged metric to get effect similar to targetAverageValue and though about providing something for transitional period, but if you're ok with it I'd rather just change it as you say.
I think it's safe to change as-is -- that feature's still in beta, and it's not an API change, so it shouldn't just hard-break anyone. Plus, we've change stuff like this before (when we did the initial resource metrics change).
As said, we are more than ready to help with the fix/test if required. This since this is a major blocking for us in the implementation of the HPA in our side
is anyone actively working on implementing this? If not, I'm happy to. I think I have a fairly clear picture of the change we need to make and will likely have time early this week to get it done.
(Also, from the conversation above, to we want to tag this as a release blocker for 1.10)?
I probably won't be able to work on this before the end of this week. Happy to do a review if you jump in and implement it.
I don't think this qualifies as a release blocking bug (but I'm willing to have someone try and convince me otherwise ;-)). There doesn't seem to be any behavior here that's explicitly broken -- it's just sub-optimal in some use cases. I'd definitely like to address this in 1.11, though.
it's not release blocker, but to share our experience ;).


Question: // GetObjectMetricReplicas calculates the desired replica count based on a target metric utilization (as a milli-value) // for the given object in the given namespace, and the current replica count. <del> func (c *ReplicaCalculator) GetObjectMetricReplicas(currentReplicas int32, targetUtilization int64, metricName string, namespace string, objectRef *autoscaling.CrossVersionObjectReference) (replicaCount int32, utilization int64, timestamp time.Time, err error) { </del> <ins> func (c *ReplicaCalculator) GetObjectMetricReplicas(currentReplicas int32, targetUtilization int64, metricName string, namespace string, objectRef *autoscaling.CrossVersionObjectReference, selector labels.Selector) (replicaCount int32, utilization int64, timestamp time.Time, err error) { </ins> utilization, timestamp, err = c.metricsClient.GetObjectMetric(metricName, namespace, objectRef) if err != nil { return 0, 0, time.Time{}, fmt.Errorf("unable to get metric %s: %v on %s %s/%s", metricName, objectRef.Kind, namespace, objectRef.Name, err)

without this feature we cannot use HPA as is and we need to figure out how to build/run fork (which can be challenging because everything inside controller-manager). also we are planning to upgrade to 1.10 quickly, but not sure for 1.11 (e.g. we are two versions behind now).
Sounds like a plan :) Will try and get this done tomorrow morning!
- I submitted a pr for this - would love your eyes on it so you can see if you think it meets your use case :)
/assign
GitHub didn't allow me to assign the following users: mattjmcnaughton. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I checked the code and it seems good to me. About the comment about scaling down if the readyPods<CurrentPods, we should be fine since our metric is already not including the pods that are not ready.
Is there any advance in this ticket? I'm wondering if this made it to k8s 1.10 or if we need to wait for k8s 1.11?
The PR is lgtm-ed and it will merge once the code freeze is lifted. Meaning it will go to 1.11. To get it into 1.10 you would need to get approval for milestone from (sig-autoscaling lead).


Question: && clean-install $BUILD_DEPS  ca-certificates  libjemalloc1  <del> liblz4-1  </del> ruby  && echo 'gem: --no-document' >> /etc/gemrc  && gem install --file Gemfile  && apt-get purge -y --auto-remove  -o APT::AutoRemove::RecommendsImportant=false  $BUILD_DEPS  <del> && rm -rf /tmp/*  /var/lib/apt/lists/*  /usr/lib/ruby/gems/*/cache/*.gem  /var/log/*  /var/tmp/*  </del> <ins> && clean-apt  </ins> # Ensure fluent has enough file descriptors && ulimit -n 65536

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: [root EFK]# kubectl logs -f fluentd-es-jrsf7 -n kube-system /var/lib/gems/2.1.0/gems/ffi- block in ffilib': Could not open library '': cannot open shared object file: No such file or directory. (LoadError) Could not open library '': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': libsystemd- cannot open shared object file: No such file or directory from /var/lib/gems/2.1.0/gems/ffi-map' from /var/lib/gems/2.1.0/gems/ffi- ffilib' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Native' from /var/lib/gems/2.1.0/gems/systemd-journal- <module:Id128' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Systemd' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/systemd-journal-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from rescue in require' fromrequire' from /var/lib/gems/2.1.0/gems/fluent-plugin-systemd- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/fluentd-block in tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-tryloadplugin' from /var/lib/gems/2.1.


Question: && clean-install $BUILD_DEPS  ca-certificates  libjemalloc1  <del> liblz4-1  </del> ruby  && echo 'gem: --no-document' >> /etc/gemrc  && gem install --file Gemfile  && apt-get purge -y --auto-remove  -o APT::AutoRemove::RecommendsImportant=false  $BUILD_DEPS  <del> && rm -rf /tmp/*  /var/lib/apt/lists/*  /usr/lib/ruby/gems/*/cache/*.gem  /var/log/*  /var/tmp/*  </del> <ins> && clean-apt  </ins> # Ensure fluent has enough file descriptors && ulimit -n 65536

0/gems/fluentd- newimpl' from /var/lib/gems/2.1.0/gems/fluentd-newinput' from /var/lib/gems/2.1.0/gems/fluentd- addsource' from /var/lib/gems/2.1.0/gems/fluentd-block in configure' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-configure' from /var/lib/gems/2.1.0/gems/fluentd- configure' from /var/lib/gems/2.1.0/gems/fluentd-runconfigure' from /var/lib/gems/2.1.0/gems/fluentd- runconfigure' from /var/lib/gems/2.1.0/gems/fluentd-block in start' from /var/lib/gems/2.1.0/gems/fluentd- call' from /var/lib/gems/2.1.0/gems/fluentd-mainprocess' from /var/lib/gems/2.1.0/gems/fluentd- start' from /var/lib/gems/2.1.0/gems/fluentd-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/fluentd-0.12.39/bin/fluentd:5:in <top (required)' from /usr/local/bin/fluentd:23:inload' from /usr/local/bin/fluentd:23:in <main' What you expected to happen: fluentd-es run success How to reproduce it (as minimally and precisely as possible): kubectl create -f fluentd-es- the image is google-containers/fluentd-elasticsearch:v2.0.0 Anything else we need to know?


Question: && clean-install $BUILD_DEPS  ca-certificates  libjemalloc1  <del> liblz4-1  </del> ruby  && echo 'gem: --no-document' >> /etc/gemrc  && gem install --file Gemfile  && apt-get purge -y --auto-remove  -o APT::AutoRemove::RecommendsImportant=false  $BUILD_DEPS  <del> && rm -rf /tmp/*  /var/lib/apt/lists/*  /usr/lib/ruby/gems/*/cache/*.gem  /var/log/*  /var/tmp/*  </del> <ins> && clean-apt  </ins> # Ensure fluent has enough file descriptors && ulimit -n 65536

: Environment: Kubernetes version (use ):Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-21T08:23:22Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-21T08:08:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release):centos 7.3 Kernel (e.g. ):Linux node-21.46 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Yup, the situation with fluentd on the system with systemd is very sloppy. There's a hack to make it works sometimes, but it seems that it doesn't work. I'll look into it
/sig instrumentation /assign
Facing same issue.


Question: && clean-install $BUILD_DEPS  ca-certificates  libjemalloc1  <del> liblz4-1  </del> ruby  && echo 'gem: --no-document' >> /etc/gemrc  && gem install --file Gemfile  && apt-get purge -y --auto-remove  -o APT::AutoRemove::RecommendsImportant=false  $BUILD_DEPS  <del> && rm -rf /tmp/*  /var/lib/apt/lists/*  /usr/lib/ruby/gems/*/cache/*.gem  /var/log/*  /var/tmp/*  </del> <ins> && clean-apt  </ins> # Ensure fluent has enough file descriptors && ulimit -n 65536

Environment Details:kubectl version Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-03T15:31:10Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} OS (e.g. from /etc/os-release):Centos 7 (Core) Kernel (e.g. uname -a): Linux xxxx 3.10.0-327.el7.x8664 Error for reference /var/lib/gems/2.1.0/gems/ffi- block in ffilib': Could not open library '': cannot open shared object file: No such file or directory. (LoadError) Could not open library '': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': cannot open shared object file: No such file or directory. Could not open library 'libsystemd-': libsystemd- cannot open shared object file: No such file or directory from /var/lib/gems/2.1.0/gems/ffi-map' from /var/lib/gems/2.1.0/gems/ffi- ffilib' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Native' from /var/lib/gems/2.1.0/gems/systemd-journal- <module:Id128' from /var/lib/gems/2.1.0/gems/systemd-journal-<module:Systemd' from /var/lib/gems/2.1.0/gems/systemd-journal- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/systemd-journal-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.


Question: && clean-install $BUILD_DEPS  ca-certificates  libjemalloc1  <del> liblz4-1  </del> ruby  && echo 'gem: --no-document' >> /etc/gemrc  && gem install --file Gemfile  && apt-get purge -y --auto-remove  -o APT::AutoRemove::RecommendsImportant=false  $BUILD_DEPS  <del> && rm -rf /tmp/*  /var/lib/apt/lists/*  /usr/lib/ruby/gems/*/cache/*.gem  /var/log/*  /var/tmp/*  </del> <ins> && clean-apt  </ins> # Ensure fluent has enough file descriptors && ulimit -n 65536

0/gems/systemd-journal- <top (required)' fromrequire' from rescue in require' fromrequire' from /var/lib/gems/2.1.0/gems/fluent-plugin-systemd- <top (required)' fromrequire' from require' from /var/lib/gems/2.1.0/gems/fluentd-block in tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-tryloadplugin' from /var/lib/gems/2.1.0/gems/fluentd- newimpl' from /var/lib/gems/2.1.0/gems/fluentd-newinput' from /var/lib/gems/2.1.0/gems/fluentd- addsource' from /var/lib/gems/2.1.0/gems/fluentd-block in configure' from /var/lib/gems/2.1.0/gems/fluentd- each' from /var/lib/gems/2.1.0/gems/fluentd-configure' from /var/lib/gems/2.1.0/gems/fluentd- configure' from /var/lib/gems/2.1.0/gems/fluentd-runconfigure' from /var/lib/gems/2.1.0/gems/fluentd- runconfigure' from /var/lib/gems/2.1.0/gems/fluentd-block in start' from /var/lib/gems/2.1.0/gems/fluentd- call' from /var/lib/gems/2.1.0/gems/fluentd-mainprocess' from /var/lib/gems/2.1.0/gems/fluentd- start' from /var/lib/gems/2.1.0/gems/fluentd-<top (required)' from require' fromrequire' from /var/lib/gems/2.1.0/gems/fluentd-0.12.39/bin/fluentd:5:in <top (required)' from /usr/local/bin/fluentd:23:inload' from /usr/local/bin/fluentd:23:in <


Question: && clean-install $BUILD_DEPS  ca-certificates  libjemalloc1  <del> liblz4-1  </del> ruby  && echo 'gem: --no-document' >> /etc/gemrc  && gem install --file Gemfile  && apt-get purge -y --auto-remove  -o APT::AutoRemove::RecommendsImportant=false  $BUILD_DEPS  <del> && rm -rf /tmp/*  /var/lib/apt/lists/*  /usr/lib/ruby/gems/*/cache/*.gem  /var/log/*  /var/tmp/*  </del> <ins> && clean-apt  </ins> # Ensure fluent has enough file descriptors && ulimit -n 65536

main'
You can solve it temporary by removing I'm working on the solution
A solution here might be to base fluentd-es off an image with journald supporting XZ and LZ4 compression. I'm looking :)
Thanks


Question: } } } <del> // Check that the mounts are/are not propagated to the host. // Host can see mount from master cmd = fmt.Sprintf("test `cat %q/master/file` = master", hostDir) err = hostExec.IssueCommand(cmd, node) framework.ExpectNoError(err, "host should see mount from master") // Host can't see mount from slave cmd = fmt.Sprintf("test ! -e %q/slave/file", hostDir) err = hostExec.IssueCommand(cmd, node) framework.ExpectNoError(err, "host shouldn't see mount from slave") </del> <ins> // Find the kubelet PID to ensure we're working with the kubelet's mount namespace cmd = "pidof kubelet" kubeletPid, err := hostExec.IssueCommandWithResult(cmd, node) framework.ExpectNoError(err, "Checking kubelet pid") kubeletPid = strings.TrimSuffix(kubeletPid, "n") framework.ExpectEqual(strings.Count(kubeletPid, " "), 0, "kubelet should only have a single PID in the system (pidof returned %q)", kubeletPid) enterKubeletMountNS := fmt.Sprintf("nsenter -t %s -m", kubeletPid) // Check that the master and host mounts are propagated to the container runtime's mount namespace for _, mountName := range []string{"host", master.Name} { cmd := fmt.Sprintf("%s cat "%s/%s/file"", enterKubeletMountNS, hostDir, mountName) output, err := hostExec.IssueCommandWithResult(cmd, node) framework.ExpectNoError(err, "host container namespace should see mount from %s: %s", mountName, output) output = strings.TrimSuffix(output, "n") framework.ExpectEqual(output, mountName, "host container namespace should see mount contents from %s", mountName) } // Check that the slave, private, and default mounts are not propagated to the container runtime's mount namespace for _, podName := range []string{slave.Name, private.Name, defaultPropagation.Name} { cmd := fmt.Sprintf("%s test ! -e "%s/%s/file"", enterKubeletMountNS, hostDir, podName) output, err := hostExec.IssueCommandWithResult(cmd, node) framework.ExpectNoError(err, "host container namespace shouldn't see mount from %s: %s", podName, output) } </ins> }) })

The e2e test should be relaxed. It currently checks 3 different aspects of the volume mount flag: with do not receive mounts from outside their own mount namespace, nor do mount namespaces outside the container receive mounts created within the container. with receive new mounts from the host OS, the container runtime, and other containers with , but no mount namespaces outside the container receive mounts created within the container. with receive new mounts identical to containers with , but additionally any mounts made by these containers are: propagated to the container runtime where they can be propagated to other containers (related to 2) also propagated to the Host OS The first 3 aspects, up to and including 3.i, are critical for container operation and should still be ensured by an e2e test. This last check, 3.ii., is the part that I believe is overly strict. The current e2e test is too opinionated on the underlying node OS configuration. The requirement of propagation to the host OS precludes administrators or solution providers from modifying the Kubernetes installation to hide these Kubernetes-specific mountpoints, which can be for a variety of reasons: While access to container-specific mounts in Kubernetes (like secrets) today are protected by filesystem permissions, the fact that the locations are visible to unprivileged users via the 'mount' command can provide information which could be used in conjunction with other vulnerabilities to decipher secrets or other protected container mount contents. Locking access to even listing the mountpoints behind the ability to enter the mount namespace (which in turn requires and ) removes one level of potential access to container-specific mounts. These container overlay mounts, secret and configmap mounts, and namespace mounts are all internal implementation details around the way the Kubernetes and the Container Runtime allocate resources to containers. For normal day-to-day operation, there is no need for these special-use mounts to be visible in the host OS's top-level 'mount' or 'findmnt' output. After much investigation and conversation on both the sig-node and sig-storage mailing lists, as well as within Red Hat's OpenShift communities, we have yet to turn up a use case that relies on this container-to-host propagation. In fact, it may be seen as an anti-pattern and run contrary to the container-native way of doing things. Relying on a host OS to interact with a container-owned mountpoint seems suspect.


Question: } } } <del> // Check that the mounts are/are not propagated to the host. // Host can see mount from master cmd = fmt.Sprintf("test `cat %q/master/file` = master", hostDir) err = hostExec.IssueCommand(cmd, node) framework.ExpectNoError(err, "host should see mount from master") // Host can't see mount from slave cmd = fmt.Sprintf("test ! -e %q/slave/file", hostDir) err = hostExec.IssueCommand(cmd, node) framework.ExpectNoError(err, "host shouldn't see mount from slave") </del> <ins> // Find the kubelet PID to ensure we're working with the kubelet's mount namespace cmd = "pidof kubelet" kubeletPid, err := hostExec.IssueCommandWithResult(cmd, node) framework.ExpectNoError(err, "Checking kubelet pid") kubeletPid = strings.TrimSuffix(kubeletPid, "n") framework.ExpectEqual(strings.Count(kubeletPid, " "), 0, "kubelet should only have a single PID in the system (pidof returned %q)", kubeletPid) enterKubeletMountNS := fmt.Sprintf("nsenter -t %s -m", kubeletPid) // Check that the master and host mounts are propagated to the container runtime's mount namespace for _, mountName := range []string{"host", master.Name} { cmd := fmt.Sprintf("%s cat "%s/%s/file"", enterKubeletMountNS, hostDir, mountName) output, err := hostExec.IssueCommandWithResult(cmd, node) framework.ExpectNoError(err, "host container namespace should see mount from %s: %s", mountName, output) output = strings.TrimSuffix(output, "n") framework.ExpectEqual(output, mountName, "host container namespace should see mount contents from %s", mountName) } // Check that the slave, private, and default mounts are not propagated to the container runtime's mount namespace for _, podName := range []string{slave.Name, private.Name, defaultPropagation.Name} { cmd := fmt.Sprintf("%s test ! -e "%s/%s/file"", enterKubeletMountNS, hostDir, podName) output, err := hostExec.IssueCommandWithResult(cmd, node) framework.ExpectNoError(err, "host container namespace shouldn't see mount from %s: %s", podName, output) } </ins> }) })

Today systemd's CPU usage is adversely affected by a large number of mountpoints, especially versions released prior to December 2020. This feature of systemd, which scans all /proc/1/mount to invoke udev events for every change suffers from two issues. The first was an inefficiency in systemd’s processing logic. This was fixed in December 2020, and this fix reportedly reduced the systemd mount-scanning-related overhead by 30-50%. The second is a lack of granularity in the events the kernel sends to systemd about mountpoint changes, which necessitates a full rescan of all mountpoints even on a minor change or single mountpoint addition. The kernel change to remedy this seems unlikely to move forward in the near term. (See for more details) For example, a fairly small OpenShift deployment today (which does not yet have the systemd December 2020 fix) has upwards of 400 mountpoints on every node owned by Kubelet, the Container Runtime, and infrastructure containers like OVN or CSI operators. Hiding the container mountpoints reduces the steady-state systemd CPU utilization from ~10% to ~0% on every worker node. On a more heavily loaded system, with 1000’s of container mountpoints, systemd’s usage will cap out at 100% of a single CPU core. Even with the 30-50% reduction in systemd usage expected from the December 2020 fix, having systemd take up as much as 50-70% of a single CPU in the worst-case scenario is a lot of unnecessary overhead. Administrators of other OSes with other host OS services that monitor or process mountpoints indiscriminately would be able to hide these container-specific mountpoints from the OS at large, at their option. With this requirement relaxed, it would be possible to run multiple instances of Kubernetes on a single host without the various mountpoints interfering with one another. There may be other inter-instance difficulties besides the mount namespace, but this does move in that direction without resorting to more complicated solutions like trying to run Kubernetes inside of containers.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available .


Question: } } } <del> // Check that the mounts are/are not propagated to the host. // Host can see mount from master cmd = fmt.Sprintf("test `cat %q/master/file` = master", hostDir) err = hostExec.IssueCommand(cmd, node) framework.ExpectNoError(err, "host should see mount from master") // Host can't see mount from slave cmd = fmt.Sprintf("test ! -e %q/slave/file", hostDir) err = hostExec.IssueCommand(cmd, node) framework.ExpectNoError(err, "host shouldn't see mount from slave") </del> <ins> // Find the kubelet PID to ensure we're working with the kubelet's mount namespace cmd = "pidof kubelet" kubeletPid, err := hostExec.IssueCommandWithResult(cmd, node) framework.ExpectNoError(err, "Checking kubelet pid") kubeletPid = strings.TrimSuffix(kubeletPid, "n") framework.ExpectEqual(strings.Count(kubeletPid, " "), 0, "kubelet should only have a single PID in the system (pidof returned %q)", kubeletPid) enterKubeletMountNS := fmt.Sprintf("nsenter -t %s -m", kubeletPid) // Check that the master and host mounts are propagated to the container runtime's mount namespace for _, mountName := range []string{"host", master.Name} { cmd := fmt.Sprintf("%s cat "%s/%s/file"", enterKubeletMountNS, hostDir, mountName) output, err := hostExec.IssueCommandWithResult(cmd, node) framework.ExpectNoError(err, "host container namespace should see mount from %s: %s", mountName, output) output = strings.TrimSuffix(output, "n") framework.ExpectEqual(output, mountName, "host container namespace should see mount contents from %s", mountName) } // Check that the slave, private, and default mounts are not propagated to the container runtime's mount namespace for _, podName := range []string{slave.Name, private.Name, defaultPropagation.Name} { cmd := fmt.Sprintf("%s test ! -e "%s/%s/file"", enterKubeletMountNS, hostDir, podName) output, err := hostExec.IssueCommandWithResult(cmd, node) framework.ExpectNoError(err, "host container namespace shouldn't see mount from %s: %s", podName, output) } </ins> }) })

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig architecture storage /area conformance
I have a proof-of-concept of the relaxed test as well as some more and an OpenShift-based proof-of-concept of an actual implementation with the container mounts in their own namespace
/sig node
There's still a number of other issues, no? Is this a real motivation / use-case? There are other difficulties. Most of the difficulties with running Kubernetes inside containers are just the difficulties in sharing a kernel / host OS, which aren't related to this at all. The security and performance angles seem relevant, in any case.
Update: I mistakenly referred to this as a "conformance test" - Turns out it is not in fact a conformance test; just an e2e test :) Edited the original post to be more correct in this respect.
/remove-area conformance /area test
Fair point. I this (intentionally vague) section since some conversation on the sig-node mailing list with folks working on KIND suggested this may align with their goals in some way I'm not entirely clear on.
A good next step on this issue is to open a PR to the related e2e tests that documents why the tests are not good candidates for future inclusion in conformance. Is that something you can assist with ?
I have a branch that relaxes the test so it works in the current environment and a separated mount namespace environment... Can you explain a bit more what you mean about documenting this? Where would that kind of documentation go?


Question: "k8s.io/kubernetes/test/e2e/storage/utils" imageutils "k8s.io/kubernetes/test/utils/image" admissionapi "k8s.io/pod-security-admission/api" <ins> "github.com/onsi/ginkgo/v2" "github.com/onsi/gomega" </ins> ) // Validate PV/PVC, create and verify writer pod, delete the PVC, and validate the PV's

[X] test approval issue : [X] test pr : [x] two weeks soak end date : 11 Sept 2023 [x] test promotion pr : [x] remove endpoint from According to following APIsnoop query, there are two PersistentVolume and three PersistentVolumeClaim endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 13 endpoints. The e2e logs for this test are listed below. This query shows the following PV and PVC endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 13 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/cc
/reopen For tracking the rest of the steps to Promotion
Reopened this issue. <detailsIn response to : /reopen For tracking the rest of the steps to Promotion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: hostNetwork: true containers: - name: fluentd-gcp <del> image: gcr.io/google-containers/fluentd-gcp:2.0.6 </del> <ins> image: gcr.io/google-containers/fluentd-gcp:2.0.5 </ins> # If fluentd consumes its own logs, the following situation may happen: # fluentd fails to send a chunk to the server => writes it to the log => # tries to send this message to the server => fails to send a chunk and so on.

https://k8s-
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Most likely caused by the fluentd update /cc
The gce-gpu tests are failing for the same reason.
Ditto for pull-federation and federation-ci. pull-federation will need a cluster rebuild once the fix is incorporated upstream. /cc csbell
well that's exciting. cc
to revert fluentd to 2.0.5.
I don't understand what's special about these clusters, though. The PR passed presubmit tests a number of times.
none of the presubmit tests run COS images. I think that's the problem. From the error message, it seems like copying libsystemd is not enough, and more libraries need to be copied (e.g., libcap). Will need to build a new fluentd image to fix this.
yeah, I think you're right. I started a cluster myself and the pod is failing on the master (COS) but not on the nodes (CVM). curious that this broke between fluentd 2.0.5 and 2.0.6, which has no code changes and only updates the base image... I guess something materially changed in the base image.


Question: } err := clientretry.RetryOnConflict(UpdateNodeSpecBackoff, func() error { <ins> // TODO(wlan0): Move this logic to the route controller using the node taint instead of condition // Since there are node taints, do we still need this? // This condition marks the node as unusable until routes are initialized in the cloud provider if cnc.cloud.ProviderName() == "gce" { if err := nodeutil.SetNodeCondition(cnc.kubeClient, types.NodeName(node.Name), v1.NodeCondition{ Type:               v1.NodeNetworkUnavailable, Status:             v1.ConditionTrue, Reason:             "NoRouteCreated", Message:            "Node created without a route", LastTransitionTime: metav1.Now(), }); err != nil { return err } } </ins> curNode, err := cnc.kubeClient.CoreV1().Nodes().Get(node.Name, metav1.GetOptions{}) if err != nil { return err

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In , the controller add NodeNetworkUnavailable to the node conditions, and use to update it. But node status can not be updated by , it doesn't work because replace the node status with old status. To fix this bug, we should use to update node status. /sig cloud-provider /sig cluster-lifecycle /area cloudprovider What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I agree this is a bug. As a matter of interest though, how are you exercising this bug? It seems specific to cloud controller on gce. However that controller isn't actually enabled on gce. It would be good for me to understand your use case, to make sure we don't break it later.
I built my kubernetes cluster with external cloudprovider, but I found if I stop the route controller, pod will be scheduled on node even if the route is not for that node. I read the code and found that the node condition would not be add when using other cloudproviders. At first, I change "gce" in that line code to the name of my cloudprovider, but the problem also exists. Then I found that didn't take effect actually.


Question: REGISTRY?=staging-k8s.gcr.io IMAGE?=debian-hyperkube-base <del> TAG=0.8 </del> <ins> TAG=0.9 </ins> ARCH?=amd64 CACHEBUST?=1

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Upgraded to Kuberenetes 1.9 and now IPVS mode in kube-proxy stopped working due to What you expected to happen: Normal startup as it's been working flawless in 1.8.5 How to reproduce it (as minimally and precisely as possible): Use kube-proxy with IPVS service mode on 1.9.0 hyperkube image from GCR Anything else we need to know?: If i exec into the running kube-proxy pod and do then restart the container kube-proxy starts in ipvs mode fine Environment: Kubernetes version (use ):Cloud provider or hardware configuration: On-prem OS (e.g. from /etc/os-release): CoreOS Container Linux by CoreOS stable (1576.4.0) Kernel (e.g. ): 4.13.16-coreos-r2 Install tools: Own Others:
/sig network
According to jell in #sig-network these images are to be considered experimental and not used in production and might be missing binarys? , could someone shed some light on this
To be more precise: hyperkube binary can hide some options of underlying applications (e.g. of fix for that: ) and whole docker image of hyperkube - as this issue shows, can miss some useful binaries. That's why I named it as a tool to do experiments, while for real deployments probably a better choice would be to use underlying applications directly using own docker image, carefully tuned.
Is this connected to and by chance the changes hasn't propagated down to the public images yet?
image is based on (which is based on ) (and ) has a separate image based on But as has included support, IMO also should be extended by command, after this line.
Seems this issue already got fixed? Can close now?
Where do you get the info it's fixed, all my kube-proxy instances is still failbacking to iptables due to missing ipset that was present in 1.8.5
It was fixed by merging So, as it's in master, the fix will land in next release.
Ok, so it's fixed in the next version. I'll just switch away from Hyperkube then. Thanks
FYI: I've rebuilt hyperkube image for CoreOS with installed. Just for testing purposes. Everything works for me with it.


Question: REGISTRY?=staging-k8s.gcr.io IMAGE?=debian-hyperkube-base <del> TAG=0.8 </del> <ins> TAG=0.9 </ins> ARCH?=amd64 CACHEBUST?=1

Waiting for the next release. :relaxed:
So does have ipset. So please use that. However does not have ipset (see below). The commit above was merged well before we shipped v1.9.0, But hyperkube image does not use the debian-iptables, it uses debian-hyperkube-base image if you see Anyone want to log a PR for hyperkube image to have ipset?
Can we re-open this, only fixes this on debian, not amd64!
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
will need to reopen, then :)


Question: go_library( name = "go_default_library", <del> srcs = ["validation.go"], </del> <ins> srcs = [ "validation.go", ] + select({ "@io_bazel_rules_go//go/platform:android": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:darwin": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:dragonfly": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:freebsd": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:linux": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:nacl": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:netbsd": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:openbsd": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:plan9": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:solaris": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:windows": [ "validation_windows.go", ], "//conditions:default": [], }), </ins> importpath = "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/validation", deps = [ "//pkg/features:go_default_library",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Pods stuck in terminating state on Windows node after deleting it What you expected to happen: Pods get terminated and should disappear from kubectl How to reproduce it (as minimally and precisely as possible): Run a pod with windows container Delete it Pods are still terminating Anything else we need to know?: Could you pls help to resolve this issue. K8s pods stuck as Terminating after getting deleted on windows node with windows container(IIS). Docker ps -a not showing this container, so it is completely deleted, but in kubectl it is marked as Terminating. For networking we using OVN. kubectl logs: kubelet logs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Hyper-V VMs OS (e.g. from /etc/os-release): Windows Node: Master Node: NAME="Ubuntu" VERSION="16.04.4 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.4 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux esd-kbrnts-lnx-1 4.13.0-37-generic ~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: OVN 2.7 on windows OVN 2.7.2 on linux Others: Docker Version on Windows node: Server Version: 17.06.2-ee-6
/sig windows
It needs sometime for kubelet to remove the pod. Could you wait a while and check whether the Pod removed?
Hi actually I have waited for 1 day, and it is not deleting. Can delete it only using --grace-period=0 --force. But I don't want to use this manual approach). Could you pls give some advice on this? Thanks in advance!
is a reference to this issue. Checking kubelet logs will always help to troubleshoot such problem.


Question: go_library( name = "go_default_library", <del> srcs = ["validation.go"], </del> <ins> srcs = [ "validation.go", ] + select({ "@io_bazel_rules_go//go/platform:android": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:darwin": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:dragonfly": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:freebsd": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:linux": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:nacl": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:netbsd": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:openbsd": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:plan9": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:solaris": [ "validation_others.go", ], "@io_bazel_rules_go//go/platform:windows": [ "validation_windows.go", ], "//conditions:default": [], }), </ins> importpath = "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/validation", deps = [ "//pkg/features:go_default_library",

Actually, I have attached kubelet log in the ticket and think the main reason is Also as there any option for kubelet to automatically force delete Terminating pods after some period? And --containerized is also not solving my issue. Thanks!
Kubelet won't delete the pod from apiserver. Since it is already not exist on the node, you could force delete it by kubectl.
It is clear, so how we can resolve this error: ? I think this is the core item of our issue. Thanks.
I'm not using OVN. Have you met this problem when using OVN?
adding Bob to comment on this. i have seen this issue before, but it has been quite a while.
Hi, Guys will be very grateful if you will provide some solution for it. We have tried to add to deployment spec but it doesn't help. Thanks.
The error message of is actually a warning message. It shouldn't block the removal of pods. Could you recheck the following things: the node, check whether the pod is still existing, e.g. by the pod exists, check if there are other error or warning message in kubelet logs; or else, the pod could be force removed safely from apiserver
Hi, I had posted this info in the ticket description. not showing this container, so it is completely deleted, but in kubectl it is displayed as Terminating.
i think you may get more support and faster responses in our slack channel for sig-windows.
So it could be safely force removed from API. And agreed with slack is a more faster way to such troubleshooting issues.
The issue seems to be related to . The only platform specific implementation for PodContainerManager is the one for . There is a default implementation (), that always returns true in the Exists call. Disabling the CgroupsPerQOS option on the Windows nodes resolves the issue.
Windows doesn't support cgroups. You should always disable it.
it is clear enough that Windows does not support cgroups). Unfortunately, it's not specified in the documentation. So to disable CgroupsPerQOS you should need to add this flags to kubelet: Maybe it will help someone... Thank you all for discussion!


Question: return storage.NewInternalErrorf("unable to transform key %q: %v", kv.Key, err) } <ins> // Check if the request has already timed out before decode object select { case <-ctx.Done(): // parent context is canceled or timed out, no point in continuing return storage.NewTimeoutError(string(kv.Key), "request did not complete within requested timeout") default: } </ins> if err := appendListItem(v, data, uint64(kv.ModRevision), opts.Predicate, s.codec, s.versioner, newItemFunc); err != nil { recordDecodeError(s.groupResourceString, string(kv.Key)) return err

It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1.


Question: return storage.NewInternalErrorf("unable to transform key %q: %v", kv.Key, err) } <ins> // Check if the request has already timed out before decode object select { case <-ctx.Done(): // parent context is canceled or timed out, no point in continuing return storage.NewTimeoutError(string(kv.Key), "request did not complete within requested timeout") default: } </ins> if err := appendListItem(v, data, uint64(kv.ModRevision), opts.Predicate, s.codec, s.versioner, newItemFunc); err != nil { recordDecodeError(s.groupResourceString, string(kv.Key)) return err

We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ <del> "/apis/apiservicegroup/v2": {}, </del> <ins> "/apis/apiservicegroup/v1/path2": {}, </ins> }, }, },

When installing the metrics-server to implement the API and the prometheus-adapter to implement the API, it is possible to make the OpenAPI handler fail to initialize with this error log in apiserver: This will result in kubectl discovery calls to /openapi/v2 to fail with a 503: The OpenAPI handler should not fail to initialize if both and are serving the same path for different API groups/versions. I can provide exact installation steps if needed, however, a high-level repro is: install metrics-server to serve install prometheus-adapter to serve ensure resourceRules is populated in the prometheus-adapter config so that resource metrics are also served by restarting apiserver will result in the OpenAPI failure and 503 responses to /openapi/v2 path No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
Please let me know if I should reopen the issue in
/sig api-machinery
/sig instrumentation [metrics apis]
Would you mind taking a look when have time? Thank you^^ /triage accepted
I have a question. should be serving under . Why does an apiservice serving another gvk register paths under ? cc we've made the assumption that an APIService (such as metrics-server or prometheus-adapter) serves only gvks under its own registered group version. I'll do some more investigation but just wondering if prometheus doesn't follow that assumption?
the prometheus-adapter can also be configured to serve and replace the metrics-server by enabling these metrics, However, if the prometheus-adapter fronts and metrics-server fronts , it does seem like the correct behavior would be for prometheus-adapter to serve until an operator updates the APIService to reference prometheus-adapter rather than metrics-server. We're seeing this issue occur for upgrades between 1.27 and 1.28 so it does look this path duplication did not cause the OpenAPI handler to fail on 1.27.
I just upgraded from 1.27 to 1.28 and running into this problem. Some kubectl apply operations fail, and ArgoCD cannot load its custom resources. Any suggested workaround? Would removing the from the prometheus configuration resolve it? Or perhaps disable metrics-server?
Simply looking at the code diff between 1.27 and 1.28 I can't pinpoint the cause.


Question: SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ <del> "/apis/apiservicegroup/v2": {}, </del> <ins> "/apis/apiservicegroup/v1/path2": {}, </ins> }, }, },

Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question.