Question: `service`, a client can simply connect to $MYAPP_SERVICE_HOST on port $MYAPP_SERVICE_PORT. <ins> ## Service without selector Services, in addition to providing clean abstraction to access pods, can also abstract any kind of backend: - you want to have an external database cluster in production, but in test you use your own databases. - you want to point your service to a service in another [`namespace`](namespaces.md) or on another cluster. - you are migrating your workload to Kubernetes and some of your backends run outside of Kubernetes. In any of these scenarios you can define a service without a selector: ```json "kind": "Service", "apiVersion": "v1beta1", "id": "myapp", "port": 8765 ``` then you can explicitly map the service to a specific endpoint(s): ```json "kind": "Endpoints", "apiVersion": "v1beta1", "id": "myapp", "endpoints": ["173.194.112.206:80"] ``` Access to the service without a selector works the same as if it had selector. The traffic will be routed to endpoints defined by the user (`173.194.112.206:80` in case of this example). </ins> ## How do they work? Each node in a Kubernetes cluster runs a `service proxy`.  This application

I see a reason for the cluster services being visible and accessible for Kubernetes developer, but from an user perspective such services only make the system more complex: why do I see these services (pods,rc) ? are they related to my application ? should I care about them (especially when I use hosted Kubernetes) ? why does my container get all these environment variables for cluster services ?
I will update file to explain why one would have service without a selector.
Right now, every service goes everywhere for simplicity. Name spacing and filtering probably make sense in the long run, but it's not a high priority for v1.0 Brendan On Mar 4, 2015 5:02 AM, "Rafał Sokołowski" wrote:


Question: if plug.GetPluginName() != "testPlugin" { t.Errorf("Wrong name: %s", plug.GetPluginName()) } <ins> plug, err = vpm.FindPluginBySpec(nil) if err == nil { t.Errorf("Should return error if volume spec is nil") } volumeSpec := &Spec{} plug, err = vpm.FindPluginBySpec(volumeSpec) if err != nil { t.Errorf("Should return test plugin if volume spec is not nil") } </ins> } func Test_ValidatePodTemplate(t *testing.T) {

In volume management code, VolumeSpec is firstly needed to find the volume plugin. After the volume plugin is found, VolumeSpec is again used to attach or mount or check volume is still attached or not. This information is reconstructed during controller failure. The assumptions used here are If pod still exists, VolumeSpec should be reconstructed when controller is restarted. However, as discover from this bug, it might not be true if PVC is deleted in the meantime because pod only has PVC information and requires to use PVC to get PV information. By fixing Issue , we could fix this problem.If pod is already deleted, VolumeSpec is not really needed for detaching. But before detaching finishes, syncState loop might check this volume's attachement status and will use VolumeSpec during checking. One safe guard we can put here is to fail FindAttachablePluginBySpec if spec is nil. This function is used by multiple places such as attachdetachcontroller, desiredstateofworld, and actualstateofworld, oeperation_generator etc. Instead of adding nil check many places, I feel FindAttachablePluginBySpec or even better FindPluginBySpec (is called by FindAttachablePluginBySpec and other places). Open an issue for this
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: if host == "" { // No external IPs were found, let's try to use internal as plan B for _, a := range node.Status.Addresses { <del> if a.Type == v1.NodeInternalIP { </del> <ins> if a.Type == v1.NodeInternalIP && a.Address != "" { </ins> host = net.JoinHostPort(a.Address, sshPort) break }

Public IPs are expensive resources. Given that we're running scalability tests on 5k-node clusters, we are unnecessarily consuming 5k IPs which is a lot. We should switch our tests so that nodes have only private IP addresses. We are going to do that by configuring NAT so that nodes can talk to outside world (e.g. to download image from dockerhub). But that also requires changing some correctness tests which assume public IPs.
To summarize recent work: setting up private gce clusters with nat, feature is gated via the env test and dashboard with correctness tests on gce private cluster - https://k8s- 27 tests fail on fixing the tests - should fix majority of them
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
- is there anything missing to be done here?
No, we can close this one. It was done long time ago. Thanks for catching /close
Closing this issue. <detailsIn response to : No, we can close this one. It was done long time ago. Thanks for catching Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "context" "fmt" <ins> "strings" </ins> "sync" v1 "k8s.io/api/core/v1"

What happened: e2e tests detect whether PSP is enabled based on presence of PSP API objects. An addon that proactively includes a PSP in a manifest can trigger that detection and enable PSP-specific e2e tests that should not be run. What you expected to happen: In addition to detecting presence of PSP objects, consider dry-run creating a pod and verifying the annotation indicating the selected PSP policy is , indicating the admission plugin is active. /sig auth /assign


Question: return admission.NewForbidden(a, fmt.Errorf("node requested token bound to a pod scheduled on a different node")) } <ins> // Note: A token may only be bound to one object at a time. By requiring // the Pod binding, noderestriction eliminates the opportunity to spoof // a Node binding. Instead, kube-apiserver automatically infers and sets // the Node binding when it receives a Pod binding. See: // https://github.com/kubernetes/kubernetes/issues/121723 for more info. </ins> return nil }

recently merged, adding nodes as something we can put in bound object refs (yay!) Today while reading noderestriction code for a different reason, I realized we never extended noderestriction to block Kubelets from binding tokens to nodes other than themselves. Today, noderestriction does block Kubelets from binding tokens to Pods on other nodes, so perhaps it should do the same for bound node references. noderestriction to block it, probably. Read the code here: And note that we didn't modify in No response Not yet in any released version of K8s. N/A N/A N/A N/A N/A
/triage accepted
To summarize Slack discussion: TokenRequest can only bind to one object at a time. kubelet (node identity) is already required to bind to Pod by noderestriction. kube-apiserver automatically inserts the node information based on the Pod binding. kubelet does not set it. So this is technically still node bound. We should still add a comment in to avoid future confusion. There may be other reasons to add a check, like future proofing...


Question: FailedPreStopHook     = "FailedPreStopHook" UnfinishedPreStopHook = "UnfinishedPreStopHook" ) <del> // ToObjectReference takes an old style object reference and converts it to a client-go one func ToObjectReference(ref *v1.ObjectReference) *clientv1.ObjectReference { if ref == nil { return nil } return &clientv1.ObjectReference{ Kind:            ref.Kind, Namespace:       ref.Namespace, Name:            ref.Name, UID:             ref.UID, APIVersion:      ref.APIVersion, ResourceVersion: ref.ResourceVersion, FieldPath:       ref.FieldPath, } } </del>

Is this a BUG REPORT or FEATURE REQUEST?: What happened: Now the and has been consistent in by PR . There is no need to convert v1.ObjectReference to clientv1.ObjectReference in . What you expected to happen: Remove this function and cleanup the codes which call this function. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Will submit the cleanup after PR merged.
/sig api-machinery
/sub


Question: failedPodsObserved += failedPodsObservedOnNode } <del> // Remove pods assigned to not existing nodes when daemonset pods are scheduled by default scheduler. </del> <ins> // Remove unscheduled pods assigned to not existing nodes when daemonset pods are scheduled by scheduler. </ins> // If node doesn't exist then pods are never scheduled and can't be deleted by PodGCController. if utilfeature.DefaultFeatureGate.Enabled(features.ScheduleDaemonSetPods) { <del> podsToDelete = append(podsToDelete, getPodsWithoutNode(nodeList, nodeToDaemonPods)...) </del> <ins> podsToDelete = append(podsToDelete, getUnscheduledPodsWithoutNode(nodeList, nodeToDaemonPods)...) </ins> } // Label new pods using the hash label value of the current history when creating them

What happened: Occasionally we get a Daemonset stuck with less Current replicas than Desired, causing a Daemonset sync tends to fix it, e.g (add a new node). This happens to all our Daemonsets. I'm using fluentd as an example. There are no Daemonsets pods in pending nor are there any errors when describing the Daemonset What you expected to happen: The Daemonset Pod to be scheduled How to reproduce it (as minimally and precisely as possible): Reproducing this state is very tricky, I've had the most success by terminating nodes in our ASG causing a new one to come up. However I think I've identified why this is happening. The issue appears to be related to daemonset Pod deletion, a Pod gets deleted causing to trigger, this lowers the expectations by as expected. The delete pod event then triggers a sync of the daemonset, in which it finds which involves fetching pods from the informer cache This is somehow returning the deleted pod. You can see in the logs here thinks that it needs to delete the pod that was deleted. It is also setting the expectation back to after the delete pod event previously lowered it by 1. At this point you're stuck as the function can't get past this line As the expectation can not be satisfied as syncNodes set it back to and there's nothing it to get it back to . So I believe this is what's causing us to get less replicas than expected, the function is stuck due to the expectation so it can't perform a full sync to schedule the new pods. There's also not much to trigger a new sync once the expectation has expired until a new node comes up or one goes down. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ):
/sig apps
I investigated this as much as I could without getting some external input, it feels like it's some kind of race condition in the informer cache. If anyone has ideas of if places to dig deeper that would be handy.
Reiterating the mentions to trigger a notification: <detailsIn response to : I investigated this as much as I could without getting some external input, it feels like it's some kind of race condition in the informer cache. If anyone has ideas of if places to dig deeper that would be handy.


Question: failedPodsObserved += failedPodsObservedOnNode } <del> // Remove pods assigned to not existing nodes when daemonset pods are scheduled by default scheduler. </del> <ins> // Remove unscheduled pods assigned to not existing nodes when daemonset pods are scheduled by scheduler. </ins> // If node doesn't exist then pods are never scheduled and can't be deleted by PodGCController. if utilfeature.DefaultFeatureGate.Enabled(features.ScheduleDaemonSetPods) { <del> podsToDelete = append(podsToDelete, getPodsWithoutNode(nodeList, nodeToDaemonPods)...) </del> <ins> podsToDelete = append(podsToDelete, getUnscheduledPodsWithoutNode(nodeList, nodeToDaemonPods)...) </ins> } // Label new pods using the hash label value of the current history when creating them

Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This problem is definitely related to node termination, it's trying to delete the already deleted pod from the node that's been terminated. The problem seems to go away if is set, I think although this kind of race condition may have always been possible, it's being frequently caused via Which is finding the non-existent pod in the pod cache and seeing that it is assigned to a non-running node, resulting in a request to delete the non-existent pod. cc
Reiterating the mentions to trigger a notification: <detailsIn response to : This problem is definitely related to node termination, it's trying to delete the already deleted pod from the node that's been terminated. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign could you please take a look at this? It seems that it is related to the code you in .
Thanks for reporting the bug. Bug was introduced in which deleting pods without nodes. There is race condition between removing pod from cache and listing pods for DaemonSet. Bug occurs during following scenario: is deleted. for node is deleted by PodGCController. handler is executed by informer () set is to queue by delete handler () is processed and pods are taken from cache before deleted pod is removed. for deleted pod is not found so we add deleted pod to podsToDelete list (). are set to del:1() succeeds as pod is not found(). are never met so we need to wait till expectations expire. As fix I will send a change to so only pods that can't be deleted by PodGCController are deleted by DaemonSet controller.


Question: "k8s.io/apimachinery/pkg/util/sets" utilfeature "k8s.io/apiserver/pkg/util/feature" cliflag "k8s.io/component-base/cli/flag" <del> "k8s.io/klog" </del> "k8s.io/kubelet/config/v1beta1" "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/features"

xref In 1.16: change the metadata-proxy addon to use change the kube-proxy addon to use change the masq-agent addon to use stop setting , , and in kube-up
Hey I'm trying to start to contribute to kubernetes and this seem to be an easy task. Can I help with this one?


Question: "spec.version[v1].schema.openAPIV3Schema.properties[d]: Required value: because it is defined in spec.version[v1].schema.openAPIV3Schema.not.properties[d]", }, }, <ins> { desc: "metadata with non-properties", globalSchema: ` type: object properties: metadata: minimum: 42.0 `, expectedViolations: []string{ "spec.validation.openAPIV3Schema.properties[metadata]: Forbidden: must not specify anything other than name and generateName, but metadata is implicitly specified", "spec.validation.openAPIV3Schema.properties[metadata].type: Required value: must not be empty for specified object fields", }, }, { desc: "metadata with other properties", globalSchema: ` type: object properties: metadata: properties: name: pattern: "^[a-z]+$" labels: type: object maxLength: 4 `, expectedViolations: []string{ "spec.validation.openAPIV3Schema.properties[metadata]: Forbidden: must not specify anything other than name and generateName, but metadata is implicitly specified", "spec.validation.openAPIV3Schema.properties[metadata].type: Required value: must not be empty for specified object fields", "spec.validation.openAPIV3Schema.properties[metadata].properties[name].type: Required value: must not be empty for specified object fields", }, }, { desc: "metadata with name property", globalSchema: ` type: object properties: metadata: type: object properties: name: type: string pattern: "^[a-z]+$" `, expectedViolations: []string{}, }, { desc: "metadata with generateName property", globalSchema: ` type: object properties: metadata: type: object properties: generateName: type: string pattern: "^[a-z]+$" `, expectedViolations: []string{}, }, { desc: "metadata with name and generateName property", globalSchema: ` type: object properties: metadata: type: object properties: name: type: string pattern: "^[a-z]+$" generateName: type: string pattern: "^[a-z]+$" `, expectedViolations: []string{}, }, { desc: "metadata under junctors", globalSchema: ` type: object properties: metadata: type: object properties: name: type: string pattern: "^[a-z]+$" allOf: - properties: metadata: {} anyOf: - properties: metadata: {} oneOf: - properties: metadata: {} not: properties: metadata: {} `, expectedViolations: []string{ "spec.validation.openAPIV3Schema.anyOf[0].properties[metadata]: Forbidden: must not be specified in a nested context", "spec.validation.openAPIV3Schema.allOf[0].properties[metadata]: Forbidden: must not be specified in a nested context", "spec.validation.openAPIV3Schema.oneOf[0].properties[metadata]: Forbidden: must not be specified in a nested context", "spec.validation.openAPIV3Schema.not.properties[metadata]: Forbidden: must not be specified in a nested context", }, }, </ins> } for i := range tests {

We allow CRD authors to write . Currently we don't restrict writing validation for , which means people can have . For example, people can have additive name constraints (require CR to be ) with the following schema: This brings us two questions: do we if "metadata" is included in the schema: simplest option is to publish standard metav1.ObjectMeta schema info, and use any CRD-specified "metadata" properties as additive validation (just like in-tree types do, publishing standard ObjectMeta schema info, and implementing type-specific name validation internally) alternatively, we can try merging user-specified metadata schema with the standard one. Technically people should only add extra validation to existing metadata properties (adding new fields to metadata won't work since they will be pruned by built-in validation first), but it's unclear to me how arbitrary people may want the metadata validation to be term question is whether to allow people specifying metadata validation at all, or just allow a subset (like name validation) in CRD v1 /sig api-machinery /area custom-resources cc
What is the difference between these two? I mean the end result is the same right? The only thing that can be is validation. Nothing else works unless you are considering extensions?
The former means we drop/hide user-specified metadata schema when we publish CRD OpenAPI. We still validate user-specified constraints on server side, but client-side validation won't see them. The latter enables client-side user-specified metadata validation.
If it's possible to merge with standard metadata schema info, or at the cost of not publishing the standard schema. My inclination would be to publish the complete standard schema for metadata/apiVersion/kind, which we know is complete and accurate, and use and user-specified schema for those fields as addition validations server side


Question: // IsFitPredicateRegistered is useful for testing providers. func IsFitPredicateRegistered(name string) bool { <del> schedulerFactoryMutex.Lock() defer schedulerFactoryMutex.Unlock() </del> <ins> schedulerFactoryMutex.RLock() defer schedulerFactoryMutex.RUnlock() </ins> _, ok := fitPredicateMap[name] return ok }

What would you like to be : Currently Mutex lock is acquired in for accessing different maps. Some functions only read map entry: We can utilize RWMutex where the above function acquires read lock. Why is this needed: For concurrent calls to IsFitPredicateRegistered / IsPriorityFunctionRegistered, read lock would lower contention.
/sig scheduling
functions getFitPredicateFunctions, getPriorityFunctionConfigs, ListRegisteredFitPredicates, ListRegisteredPriorityFunctions and GetAlgorithmProvider can also use read lock.
I happened to raise earlier today. Anyone can finish resting ones.
Good to know,
Thanks for raising this ! My PR covered and . Feel free to raise another PR to cover others.
Once is merged, I will send my PR thanks
it's merged now.


Question: name = "go_default_library", srcs = ["event.go"], tags = ["automanaged"], <del> deps = ["//vendor/k8s.io/api/core/v1:go_default_library"], </del> ) filegroup(


Question: } func (dc *DeploymentController) syncRecreateDeployment(deployment extensions.Deployment) error { <del> // TODO: implement me. </del> <ins> newRC, err := dc.getNewRC(deployment) if err != nil { return err } oldRCs, err := dc.getOldRCs(deployment) if err != nil { return err } allRCs := append(oldRCs, newRC) // scale down old rcs scaledDown, err := dc.scaleDownOldRCsForRecreate(oldRCs, deployment) if err != nil { return err } if scaledDown { // Update DeploymentStatus return dc.updateDeploymentStatus(allRCs, newRC, deployment) } // scale up new rc scaledUp, err := dc.scaleUpNewRCForRecreate(newRC, deployment) if err != nil { return err } if scaledUp { // Update DeploymentStatus return dc.updateDeploymentStatus(allRCs, newRC, deployment) } </ins> return nil }

There's some practical limit on the number of concurrent access tokens that can be used in a Kubernetes cluster before it begins to spend all of its time in the authn stack verifying them. Absent fixes to it was < 500; with the fixes linked there it's still not infinitely scalable-- I still hit (what I believe to be) with ~ 5k unique tokens @ ~700QPS. This request is to this dimension an official name & definition, and list it with others, e.g. here: it it have a useful value (I think it should be 10k) /kind bug /sig scalability
/cc
oh and 4. add tests which verify it is as high as believed.
cc since projected tokens will increase the number of unique tokens
/sig auth
LRUs are not good at scaling and sharding has some major pitfalls as pointed out. I would suggest that we try to drop the LRU cache for successful authentication results in the token cache. The main benefit of using an LRU here is bound cache size. Entries in the token cache have a fast expiration independent of the LRU policy. The records probably take up O(2kb) (guesstimate) of memory. Given that these are successful attempts with presumably valid tokens, I don't think we are getting any benefit from the LRU. If we don't have a constraint on fixed cache size, we can back the token cache with a sync.Map and run a single background goroutine to periodically walk the map and remove expired entries. This would fix the problematic locking and eviction due to nonuniform distribution of tokens over shards. The failure cache is less important and we don't cache failures by default. If we consider this the "slowpath", we can probably leave it as. If that's not good enough, we have a couple options (with various tradeoffs) to improve it.
If off by default is not enough, more reasoning why the failed cache is not super useful: it might help with an accidental DoS but it doesn't prevent anything actively malicious DoS. I think we can focus our improvements on the success cache.
/assign He is already working on adding an explicit test for it.
has taken it over and is actively working on this. Unfortunately, I cannot assign this issue to him as he's not a k8s member yet
It's not closed - all testes must be submitted and appropriate documentation must be
Thanks everyone!!


Question: continue } <del> if o.Local || cmdutil.GetDryRunFlag(o.Cmd) { </del> <ins> if cmdutil.GetDryRunFlag(o.Cmd) { fmt.Fprintln(o.Err, "info: running in local mode...") </ins> return o.PrintObject(o.Cmd, o.Mapper, info.Object, o.Out) }

from https://k8s-
Really suspect this is
This is putting us into test-go post-commit run is broken
cc:
marked this p0 - if a fix isn't easy let's try to identify what introduced this and revert.
This flake on: 2016-10-30 07:19 PDT First instance: : It is flaking frequently, blocking PRs from getting merged, and slowing the submit queue ( at 75%). As on-call, I'm going to try and identify the cluprit and revert. Can use all help in doing so.
can you investigate whether your additional tests could cause this (by breaking the deployment of those pods, for instance) , Saad Ali wrote:
PR looks most suspect to me. It mucked with , and the errors we are seeing in each of the failures is related to : That PR was merged at Oct 30, 2016, 6:18 AM PDT and the first flake in was at Oct 30, 2016, 7:19 AM PDT. Thoughts? CC
I'm going to initiate a revert of to see if that fixes the issue
IMO we can just change the flaky test to not get pods but some other resources, such as deployments. We can't really expect to clean up pods immediately.
I'm going to proceed with revert (). Please feel free to send a follow up PR to handle this more gracefully.
Going to leave this opened until we have a few hours of runs with the revert. Want to verify, with some degree of certainty, that the revert took care of this flake.
We had 15 green runs of in a row. Considering this flake fixed.


Question: <ins> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Launches a container and verifies it can be reached. Assumes that # we're being called by hack/e2e-test.sh (we use some env vars it sets up). # TODO: Convert uses of gcutil to gcloud set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../.. source "${KUBE_ROOT}/cluster/kube-env.sh" source "${KUBE_ROOT}/cluster/$KUBERNETES_PROVIDER/util.sh" if [[ "$KUBERNETES_PROVIDER" != "gce" ]]; then echo "PD test is only run for GCE" return 0 fi disk_name="e2e-$(date +%H-%M-%s)" config="/tmp/${disk_name}.yaml" function teardown() { echo "Cleaning up test artifacts" ${KUBECFG} delete pods/testpd rm -rf ${config} echo "Waiting for disk to become unmounted" sleep 20 gcutil deletedisk -f --zone=${ZONE} ${disk_name} } trap "teardown" EXIT perl -p -e "s/%.*%/${disk_name}/g" ${KUBE_ROOT}/examples/gce-pd/testpd.yaml > ${config} # Create and mount the disk. gcutil adddisk --size_gb=10 --zone=${ZONE} ${disk_name} gcutil attachdisk --disk ${disk_name} ${MASTER_NAME} gcutil ssh ${MASTER_NAME} sudo rm -rf /mnt/tmp gcutil ssh ${MASTER_NAME} sudo mkdir /mnt/tmp gcutil ssh ${MASTER_NAME} sudo /usr/share/google/safe_format_and_mount /dev/disk/by-id/google-${disk_name} /mnt/tmp gcutil ssh ${MASTER_NAME} sudo umount /mnt/tmp gcloud compute instances detach-disk --disk ${disk_name} --zone ${ZONE} ${MASTER_NAME} ${KUBECFG} -c ${config} create pods pod_id_list=$($KUBECFG '-template={{range.items}}{{.id}} {{end}}' -l test=testpd list pods) # Pod turn up on a clean cluster can take a while for the docker image pull. all_running=0 for i in $(seq 1 24); do echo "Waiting for pods to come up." sleep 5 all_running=1 for id in $pod_id_list; do current_status=$($KUBECFG -template '{{.currentState.status}}' get pods/$id) || true if [[ "$current_status" != "Running" ]]; then all_running=0 break fi done if [[ "${all_running}" == 1 ]]; then break fi done if [[ "${all_running}" == 0 ]]; then echo "Pods did not come up in time" exit 1 fi </ins>

07-02 and 13-02 No response No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
I might be related to the npd.
The same test failures happening again on 25.02:
/sig node
/sig node /cc The restart of caused the flake. I also opened to track it in NPD repo.
I missed this log. It is caused by oom. Adding some memory may help. I submitted to fix it. Does mean killing metrics-server?
/remove-sig node Sorry, I misread the log. The NPD found the metric server is OOM. It is about the metrics server, not the NPD. Sorry for disturb. I may add some memory for metric-server and check why metric server memory changed. I heard that metrics server recently released a new version. I will check if it is related.
/sig node
OOM metrics shows the metric server is OOMed. This is odd, that only metrics-server-nanny is with a memory limit. It uses whose command is . metrics-server container has no memory limit, but it got killed by OOM. Does this indicate that the node is OOM? (edited)
The last run failed in node-problem-detector ContainerStateTerminated. I0227 05:03:31. ] PodStartupLatency: perc50: 14m29.018886827s, perc90: 28m44.625518807s, perc99: 32m2.19303496s; threshold 1h0m0s I0227 05:03:32. ]


Question: <ins> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Launches a container and verifies it can be reached. Assumes that # we're being called by hack/e2e-test.sh (we use some env vars it sets up). # TODO: Convert uses of gcutil to gcloud set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../.. source "${KUBE_ROOT}/cluster/kube-env.sh" source "${KUBE_ROOT}/cluster/$KUBERNETES_PROVIDER/util.sh" if [[ "$KUBERNETES_PROVIDER" != "gce" ]]; then echo "PD test is only run for GCE" return 0 fi disk_name="e2e-$(date +%H-%M-%s)" config="/tmp/${disk_name}.yaml" function teardown() { echo "Cleaning up test artifacts" ${KUBECFG} delete pods/testpd rm -rf ${config} echo "Waiting for disk to become unmounted" sleep 20 gcutil deletedisk -f --zone=${ZONE} ${disk_name} } trap "teardown" EXIT perl -p -e "s/%.*%/${disk_name}/g" ${KUBE_ROOT}/examples/gce-pd/testpd.yaml > ${config} # Create and mount the disk. gcutil adddisk --size_gb=10 --zone=${ZONE} ${disk_name} gcutil attachdisk --disk ${disk_name} ${MASTER_NAME} gcutil ssh ${MASTER_NAME} sudo rm -rf /mnt/tmp gcutil ssh ${MASTER_NAME} sudo mkdir /mnt/tmp gcutil ssh ${MASTER_NAME} sudo /usr/share/google/safe_format_and_mount /dev/disk/by-id/google-${disk_name} /mnt/tmp gcutil ssh ${MASTER_NAME} sudo umount /mnt/tmp gcloud compute instances detach-disk --disk ${disk_name} --zone ${ZONE} ${MASTER_NAME} ${KUBECFG} -c ${config} create pods pod_id_list=$($KUBECFG '-template={{range.items}}{{.id}} {{end}}' -l test=testpd list pods) # Pod turn up on a clean cluster can take a while for the docker image pull. all_running=0 for i in $(seq 1 24); do echo "Waiting for pods to come up." sleep 5 all_running=1 for id in $pod_id_list; do current_status=$($KUBECFG -template '{{.currentState.status}}' get pods/$id) || true if [[ "$current_status" != "Running" ]]; then all_running=0 break fi done if [[ "${all_running}" == 1 ]]; then break fi done if [[ "${all_running}" == 0 ]]; then echo "Pods did not come up in time" exit 1 fi </ins>

Loaded restart count threshold overrides: map[coredns:2 fluentd-gcp:999 konnectivity-agent:2 kube-controller-manager:1 kube-proxy:2 kube-scheduler:1 l7-lb-controller:1 metadata-proxy:2 prometheus-to-sd-exporter:2] I0227 05:03:32. ] ClusterOOMsTracker: gathering cluster OOMs tracking measurement E0227 05:03:32. ] TestMetrics: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0] I0227 05:03:32. ] Step "[step: 29] gathering measurements" ended W0227 05:03:32. ] Got errors during step execution: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] I0227 05:03:32. ] Waiting for the chaos monkey subroutine to end... I0227 05:03:32. ] Chaos monkey ended. I0227 05:03:32. ] I0227 05:03:32. ] Stopping DnsLookupLatency probe... I0227 05:03:47. ] Stopping InClusterNetworkLatency probe... I0227 05:04:22. ] Resources cleanup time: 50.539700215s E0227 05:04:22. ]
E0227 05:04:22. ] Test Finished E0227 05:04:22. ] Test: E0227 05:04:22. ] Status: Fail E0227 05:04:22. ]


Question: <ins> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Launches a container and verifies it can be reached. Assumes that # we're being called by hack/e2e-test.sh (we use some env vars it sets up). # TODO: Convert uses of gcutil to gcloud set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../.. source "${KUBE_ROOT}/cluster/kube-env.sh" source "${KUBE_ROOT}/cluster/$KUBERNETES_PROVIDER/util.sh" if [[ "$KUBERNETES_PROVIDER" != "gce" ]]; then echo "PD test is only run for GCE" return 0 fi disk_name="e2e-$(date +%H-%M-%s)" config="/tmp/${disk_name}.yaml" function teardown() { echo "Cleaning up test artifacts" ${KUBECFG} delete pods/testpd rm -rf ${config} echo "Waiting for disk to become unmounted" sleep 20 gcutil deletedisk -f --zone=${ZONE} ${disk_name} } trap "teardown" EXIT perl -p -e "s/%.*%/${disk_name}/g" ${KUBE_ROOT}/examples/gce-pd/testpd.yaml > ${config} # Create and mount the disk. gcutil adddisk --size_gb=10 --zone=${ZONE} ${disk_name} gcutil attachdisk --disk ${disk_name} ${MASTER_NAME} gcutil ssh ${MASTER_NAME} sudo rm -rf /mnt/tmp gcutil ssh ${MASTER_NAME} sudo mkdir /mnt/tmp gcutil ssh ${MASTER_NAME} sudo /usr/share/google/safe_format_and_mount /dev/disk/by-id/google-${disk_name} /mnt/tmp gcutil ssh ${MASTER_NAME} sudo umount /mnt/tmp gcloud compute instances detach-disk --disk ${disk_name} --zone ${ZONE} ${MASTER_NAME} ${KUBECFG} -c ${config} create pods pod_id_list=$($KUBECFG '-template={{range.items}}{{.id}} {{end}}' -l test=testpd list pods) # Pod turn up on a clean cluster can take a while for the docker image pull. all_running=0 for i in $(seq 1 24); do echo "Waiting for pods to come up." sleep 5 all_running=1 for id in $pod_id_list; do current_status=$($KUBECFG -template '{{.currentState.status}}' get pods/$id) || true if [[ "$current_status" != "Running" ]]; then all_running=0 break fi done if [[ "${all_running}" == 1 ]]; then break fi done if [[ "${all_running}" == 0 ]]; then echo "Pods did not come up in time" exit 1 fi </ins>

Errors: [measurement call TestMetrics - TestMetrics error: [action gather failed for SystemPodMetrics measurement: restart counts violation: RestartCount(node-problem-detector-tgl8h, node-problem-detector)=1, want <= 0; RestartCount(node-problem-detector-vpsmz, node-problem-detector)=1, want <= 0]] E0227 05:04:22. ]
I0227 05:04:22. ]
I0227 05:04:22. ] Running testing/huge- I0227 05:04:22. ]


Question: <ins> <?xml version="1.0" encoding="UTF-8" standalone="no"?> <!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> <!-- Generated by graphviz version 2.36.0 (20140111.2315) --> <!-- Title: tagged_release Pages: 1 --> <svg width="257pt" height="360pt" viewBox="0.00 0.00 257.33 360.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"> <g id="graph0" class="graph" transform="scale(0.649819 0.649819) rotate(0) translate(4 550)"> <title>tagged_release</title> <polygon fill="white" stroke="none" points="-4,4 -4,-550 392,-550 392,4 -4,4"/> <!-- ci012abc --> <g id="node1" class="node"><title>ci012abc</title> <ellipse fill="#ffffcc" stroke="black" cx="56" cy="-115" rx="42.7926" ry="42.7926"/> <text text-anchor="middle" x="56" y="-111.3" font-family="Consolas, Liberation Mono, Menlo, Courier, monospace" font-size="14.00">012abc</text> </g> <!-- pr101 --> <g id="node2" class="node"><title>pr101</title> <polygon fill="#ccccff" stroke="black" points="112,-255 0,-255 0,-219 112,-219 112,-255"/> <text text-anchor="middle" x="56" y="-233.3" font-family="Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" font-size="14.00">Merge PR #101</text> </g> <!-- ci012abc->pr101 --> <g id="edge1" class="edge"><title>ci012abc->pr101</title> <path fill="none" stroke="black" d="M56,-157.97C56,-174.83 56,-193.784 56,-208.789"/> <polygon fill="black" stroke="black" points="52.5001,-208.93 56,-218.93 59.5001,-208.93 52.5001,-208.93"/> </g> <!-- ci345cde --> <g id="node3" class="node"><title>ci345cde</title> <ellipse fill="#ffffcc" stroke="black" cx="62" cy="-359" rx="42.7926" ry="42.7926"/> <text text-anchor="middle" x="62" y="-355.3" font-family="Consolas, Liberation Mono, Menlo, Courier, monospace" font-size="14.00">345cde</text> </g> <!-- pr101->ci345cde --> <g id="edge2" class="edge"><title>pr101->ci345cde</title> <path fill="none" stroke="black" d="M56.8597,-255.193C57.5237,-268.473 58.4796,-287.592 59.3874,-305.748"/> <polygon fill="black" stroke="black" points="55.904,-306.17 59.8991,-315.982 62.8953,-305.82 55.904,-306.17"/> </g> <!-- pr102 --> <g id="node4" class="node"><title>pr102</title> <polygon fill="#ccccff" stroke="black" points="129,-474 17,-474 17,-438 129,-438 129,-474"/> <text text-anchor="middle" x="73" y="-452.3" font-family="Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" font-size="14.00">Merge PR #102</text> </g> <!-- ci345cde->pr102 --> <g id="edge3" class="edge"><title>ci345cde->pr102</title> <path fill="none" stroke="black" d="M66.8248,-401.668C67.8523,-410.542 68.9117,-419.692 69.8567,-427.853"/> <polygon fill="black" stroke="black" points="66.3936,-428.375 71.0207,-437.906 73.3472,-427.57 66.3936,-428.375"/> </g> <!-- pr100 --> <g id="node10" class="node"><title>pr100</title> <polygon fill="#ccccff" stroke="black" points="174,-546 62,-546 62,-510 174,-510 174,-546"/> <text text-anchor="middle" x="118" y="-524.3" font-family="Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" font-size="14.00">Merge PR #100</text> </g> <!-- pr102->pr100 --> <g id="edge6" class="edge"><title>pr102->pr100</title> <path fill="none" stroke="black" d="M84.1236,-474.303C89.355,-482.441 95.6999,-492.311 101.478,-501.299"/> <polygon fill="black" stroke="black" points="98.6526,-503.377 107.004,-509.896 104.541,-499.591 98.6526,-503.377"/> </g> <!-- version_commit --> <g id="node5" class="node"><title>version_commit</title> <ellipse fill="#ccffcc" stroke="black" cx="173" cy="-237" rx="42.7926" ry="42.7926"/> <text text-anchor="middle" x="173" y="-233.3" font-family="Consolas, Liberation Mono, Menlo, Courier, monospace" font-size="14.00">678fed</text> </g> <!-- dev_commit --> <g id="node6" class="node"><title>dev_commit</title> <ellipse fill="#ffffcc" stroke="black" cx="169" cy="-359" rx="42.7926" ry="42.7926"/> <text text-anchor="middle" x="169" y="-355.3" font-family="Consolas, Liberation Mono, Menlo, Courier, monospace" font-size="14.00">456dcb</text> </g> <!-- version_commit->dev_commit --> <g id="edge4" class="edge"><title>version_commit->dev_commit</title> <path fill="none" stroke="black" d="M171.601,-279.97C171.322,-288.326 171.027,-297.195 170.739,-305.844"/> <polygon fill="black" stroke="black" points="167.24,-305.74 170.405,-315.851 174.236,-305.973 167.24,-305.74"/> </g> <!-- dev_commit->pr100 --> <g id="edge8" class="edge"><title>dev_commit->pr100</title> <path fill="none" stroke="black" d="M158.36,-400.568C152.438,-422.433 144.719,-449.815 137,-474 134.253,-482.606 131.013,-491.906 127.994,-500.278"/> <polygon fill="black" stroke="black" points="124.616,-499.325 124.47,-509.919 131.191,-501.729 124.616,-499.325"/> </g> <!-- release_info --> <g id="node7" class="node"><title>release_info</title> <text text-anchor="middle" x="304" y="-240.8" font-family="Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" font-size="14.00">pkg/version/base.go:</text> <text text-anchor="middle" x="304" y="-225.8" font-family="Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" font-size="14.00">gitVersion = "v0.5";</text> </g> <!-- dev_info --> <g id="node8" class="node"><title>dev_info</title> <text text-anchor="middle" x="309" y="-362.8" font-family="Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" font-size="14.00">pkg/version/base.go:</text> <text text-anchor="middle" x="309" y="-347.8" font-family="Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" font-size="14.00">gitVersion = "v0.5-dev";</text> </g> <!-- pr99 --> <g id="node9" class="node"><title>pr99</title> <polygon fill="#ccccff" stroke="black" points="143,-36 39,-36 39,-0 143,-0 143,-36"/> <text text-anchor="middle" x="91" y="-14.3" font-family="Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" font-size="14.00">Merge PR #99</text> </g> <!-- pr99->ci012abc --> <g id="edge5" class="edge"><title>pr99->ci012abc</title> <path fill="none" stroke="black" d="M84.5805,-36.4245C81.5586,-44.6267 77.7879,-54.8615 73.9865,-65.1795"/> <polygon fill="black" stroke="black" points="70.6804,-64.0292 70.5075,-74.6226 77.2488,-66.4492 70.6804,-64.0292"/> </g> <!-- pr99->version_commit --> <g id="edge7" class="edge"><title>pr99->version_commit</title> <path fill="none" stroke="black" d="M97.4344,-36.0276C109.585,-68.1826 136.317,-138.924 154.498,-187.038"/> <polygon fill="black" stroke="black" points="151.318,-188.523 158.127,-196.64 157.866,-186.048 151.318,-188.523"/> </g> <!-- tag --> <g id="node11" class="node"><title>tag</title> <ellipse fill="#ffcccc" stroke="black" cx="226" cy="-115" rx="71.4873" ry="18"/> <text text-anchor="middle" x="226" y="-111.3" font-family="Helvetica Neue, Helvetica, Segoe UI, Arial, freesans, sans-serif" font-size="14.00">$ git tag -a v0.5</text> </g> <!-- tag->version_commit --> <g id="edge9" class="edge"><title>tag->version_commit</title> <path fill="none" stroke="black" d="M218.519,-132.939C212.168,-147.318 202.736,-168.673 194.103,-188.22"/> <polygon fill="black" stroke="black" points="190.784,-187.071 189.946,-197.632 197.188,-189.899 190.784,-187.071"/> </g> </g> </svg> </ins>

Right now, we have a mixture of user documentation and developer documentation in the top-level directory. We should make it easier to find a just user documentation. Step 1 would be to move developer documentation into a subdir like docs/development or something like that. User docs would stay in the top-level docs dir.
move to


Question: // If the ordinal could not be parsed (ord < 0), ignore the Pod. } <ins> // make sure to update the latest status even if there is an error later defer func() { // update the set's status statusErr := ssc.updateStatefulSetStatus(ctx, set, &status) if statusErr == nil { klog.V(4).InfoS("Updated status", "statefulSet", klog.KObj(set), "replicas", status.Replicas, "readyReplicas", status.ReadyReplicas, "currentReplicas", status.CurrentReplicas, "updatedReplicas", status.UpdatedReplicas) } else if updateErr == nil { updateErr = statusErr } else { klog.V(4).InfoS("Could not update status", "statefulSet", klog.KObj(set), "err", statusErr) } }() </ins> // for any empty indices in the sequence [0,set.Spec.Replicas) create a new Pod at the correct revision for ord := 0; ord < replicaCount; ord++ { if replicas[ord] == nil {

I create a ResourceQuota with two pods limit in my namespace. Then, I create a statefuleSet with 3 replicas. As expected, two pods are created successfully, and the third pod failed to create. But, the ReadyReplicas is not 2 ,but 1. The ReadyReplicas is correct 1Create a ResourceQuota with two pods limit in a new namespace. 2Create a statefulSet with 3 replicas 3Check if the ReadyReplicas of statefulSet matchs the number of stateful running pods. 1According to the logic of the statefulSet controller, new pod created in current sync is not be immediately counted into the ReadyReplicas. 2Failure to create pod will cause to return error, then skip updating status. <details</details<details</details<detailskind v1.23.3 </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign


Question: NoSnatTest               = ImageConfig{e2eRegistry, "no-snat-test", "1.0", true} NoSnatTestProxy          = ImageConfig{e2eRegistry, "no-snat-test-proxy", "1.0", true} NWayHTTP                 = ImageConfig{e2eRegistry, "n-way-http", "1.0", true} <del> // When these values are updated, also update cmd/kubelet/app/options/options.go Pause               = ImageConfig{gcRegistry, "pause", "3.0", true} </del> <ins> // When these values are updated, also update cmd/kubelet/app/options/container_runtime.go Pause               = ImageConfig{gcRegistry, "pause", "3.1", true} </ins> Porter              = ImageConfig{e2eRegistry, "porter", "1.0", true} PortForwardTester   = ImageConfig{e2eRegistry, "port-forward-tester", "1.0", true} Redis               = ImageConfig{e2eRegistry, "redis", "1.0", true}

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the pause-amd64:3.0 image on has an empty schema2 manifest, which prevents is from being pulled by proxy/caches which only support V2 registries. What you expected to happen: Being able to pull ` using a V2 manifest compatible client. Anything else we need to know?: The proxy/cahce is requesting this
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After talking with , he mentionned the following
AFAIK, does not support well. After inspecting the image, you can find is still using manifest v1.
/cc FYI
Im not sure which exact version x schéma X is needed. What I know is that we can download through the proxy/cache all other Kubernetes kube-system images, only the pause image is giving us trouble. Therefore, I would say that having the image available using the same format as the other kube-system images would be the request here.
Yeah, we haven't updated the pause image in a while, since it doesn't really ever change. I'm not sure whether rebuilding + pushing over the existing tag is safe, or if we should bump to or something. thoughts?
If we do rebuilding/updating the image, that would help and a bunch of similar issues.
it's better/save to go with new tag
I didn't realize there were changes to the pause binary that were never actually deployed. It seems like we should probably just do that (and bump the tag).
that sounds great. Any idea when the existing 3.0 tag could be pushed again?
We shouldn't push over the 3.0 tag; we should push a 3.1 or 4.0 tag instead. I'm not sure who owns the pause container who should drive this. Maybe sig-node and/or
It seems like it should be sig-node. I'll look into it.
The new container has been built, can you test with
Its working! thanks! I was able to pull it through our artifactory proxy , Lee Verberne <:
excellent, thanks for checking. /assign


Question: NoSnatTest               = ImageConfig{e2eRegistry, "no-snat-test", "1.0", true} NoSnatTestProxy          = ImageConfig{e2eRegistry, "no-snat-test-proxy", "1.0", true} NWayHTTP                 = ImageConfig{e2eRegistry, "n-way-http", "1.0", true} <del> // When these values are updated, also update cmd/kubelet/app/options/options.go Pause               = ImageConfig{gcRegistry, "pause", "3.0", true} </del> <ins> // When these values are updated, also update cmd/kubelet/app/options/container_runtime.go Pause               = ImageConfig{gcRegistry, "pause", "3.1", true} </ins> Porter              = ImageConfig{e2eRegistry, "porter", "1.0", true} PortForwardTester   = ImageConfig{e2eRegistry, "port-forward-tester", "1.0", true} Redis               = ImageConfig{e2eRegistry, "redis", "1.0", true}

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: pause does not reap zombie processes using the default pause container provided by What you expected to happen: should include the up to date pause code to reap zombie processes when using docker = 1.13.1 found here How to reproduce it (as minimally and precisely as possible): a.out will leave a zombie process that is not cleaned up Anything else we need to know?: Building the pause container from master solves this issue Related: Environment: Kubernetes version (use ): 1.7.0+ Cloud provider or hardware configuration**: N/A OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 4.10.13+ Install tools: Others: docker = 1.13.1
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
i get a simliar problem when i run crond service in pod container, every day it makes some [anacron] zombie progresses UID PID PPID C STIME TTY TIME CMD root 1 0 0 Nov14 ? 00:00:00 /pause root 7 0 0 Nov14 ? 00:00:00 /usr/local/bin/dumb-init /usr/lo root 13 7 0 Nov14 ? 00:00:00 nginx: master process /usr/local nobody 21 13 0 Nov14 ? 00:00:01 nginx: worker process nobody 22 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 23 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 24 13 0 Nov14 ? 00:00:05 nginx: worker process root 77 1 0 Nov14 ? 00:00:00 [anacron] root 1 0 00:01 ? 00:00:00 [anacron] <defunctroot 1 0 01:01 ? 00:00:00 [anacron] <defunctroot 1 0 02:01 ?


Question: NoSnatTest               = ImageConfig{e2eRegistry, "no-snat-test", "1.0", true} NoSnatTestProxy          = ImageConfig{e2eRegistry, "no-snat-test-proxy", "1.0", true} NWayHTTP                 = ImageConfig{e2eRegistry, "n-way-http", "1.0", true} <del> // When these values are updated, also update cmd/kubelet/app/options/options.go Pause               = ImageConfig{gcRegistry, "pause", "3.0", true} </del> <ins> // When these values are updated, also update cmd/kubelet/app/options/container_runtime.go Pause               = ImageConfig{gcRegistry, "pause", "3.1", true} </ins> Porter              = ImageConfig{e2eRegistry, "porter", "1.0", true} PortForwardTester   = ImageConfig{e2eRegistry, "port-forward-tester", "1.0", true} Redis               = ImageConfig{e2eRegistry, "redis", "1.0", true}

00:00:00 [anacron] <defunctroot 1 0 03:01 ? 00:00:00 [anacron] <defunctroot 0 0 11:49 ? 00:00:00 /bin/bash root 0 1 19:15 ? 00:00:00 /bin/bash root 0 19:15 ? 00:00:00 ps -ef !
Do we know why this was never resolved / was never deployed?
I just fought through this issue with a rabbitmq helm chart. I had to custom build a pause container from master to fix it. The rabbitmq helm chart uses for its liveness checks. It's a thin wrapper around a call to the rabbitmq-server daemon, which is not PID 1 (/pause is PID 1). The rabbitmq-server process forks some children to do the work and then they exit. Since the pause-amd64:3.0 is not reaping, those processes stick around as zombies. The liveness check runs every 5 seconds. After about 4 hours, there are about 27K zombies and the node begins failing to launch new containers because it's starved for resources. The pause container built from current master fixes that behavior, no zombies at all in the container.
The plan is to build a new version of pause-amd64 and then update the image referenced by Kubernetes. This means the default image will change in the next Kubernetes release. We can make the new image available once the freeze lifts and we can merge .
Looks like this made it into 1.10


Question: } func pvcDeletePolicyString(policy *apps.StatefulSetPersistentVolumeClaimRetentionPolicy) string { <ins> if policy == nil { return "nullPolicy" } </ins> const retain = apps.RetainPersistentVolumeClaimRetentionPolicyType const delete = apps.DeletePersistentVolumeClaimRetentionPolicyType switch {

The replicas is 3 but just create 1 pod 'web-0' and one pvc, then kube-controller-manager panic happend. Should create 3 pod and 3 pvc add '--feature-gates=StatefulSetAutoDeletePVC=true' arg for kube-controller-manager, then create statefulset with 2 replicas and volumeClaimTemplates. No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig apps
I can reproduce the issue and want to fix it. It seems like variables cause . /assign
/triage accepted
Great find! Does this only happen on an upgraded cluster? I'll spin up a cluster and see if the e2e test catches this problem.
The e2e tests are still passing on a 1.24 cluster, so I suspect this is an upgrade problem. I wonder if there are other holes here.
I installed the Kubernetes 1.25 cluster directly using and still reproduced this issue, so it was probably not caused by upgrading the cluster.
Interesting. I thought that would set the policy to be non-nil. Let's keep this bug open after you submit your PR as I think something else is going on here---I don't think we should be seeing the null policy in the first place.
I suspect that somewhere accidentally set it to nil, and I'll keep looking at this code.
I tried the following (and also with parallel pod management), and it didn't seem to cause the panic. This was a 1.24 cluster, maybe this is only happening on head versions?
I just tried this on a kind cluster running on k8s head, and the problem didn't seem to reproduce. But this was setting the feature flag default in I wonder----does the api server need to have the feature gate as well? That would explain why defaulting wasn't happening.
As far as I'm concerned,I just set the in kube-controller-manager manifests to reproduce this issue. I repeat the yaml used as follows: I tested that in statefulset yaml, setting or not would cause the component to panic. The panic logs are the same as above, so ignore them here.


Question: } func pvcDeletePolicyString(policy *apps.StatefulSetPersistentVolumeClaimRetentionPolicy) string { <ins> if policy == nil { return "nullPolicy" } </ins> const retain = apps.RetainPersistentVolumeClaimRetentionPolicyType const delete = apps.DeletePersistentVolumeClaimRetentionPolicyType switch {

statefulset with : The environment
I believe you---this doesn't seem to be well-documented, but it's my understanding that you need to set the feature gate on all of the system components. Obviously failing to do so shouldn't cause a panic, so thank you again for finding & fixing this bug! Let me ask around to see what the expected behavior is. I know there may be skew between kubelets and master versions, but I didn't expect to have to deal with skew between master components.
I read the code, and now I understand what you mean. This feature gate must be enabled in both the and components. We need to describe it in the to prevent someone from ignoring the component, which also needs to enable the feature gate.
I don't think we need all the system components, like the component, right? I read the original KEP documentation. This feature gate only involves components and :
I had a bit of discussion about this with others. Because upgrades don't occur atomically, any feature gate has to work regardless of how the other system components are configured (things shouldn't panic!). I suppose if the cluster administrator wants to track what needs to be enabled on each feature they can try to sequence which component has which feature gate. If it were me I'd try to make all the components consistent though.
This is certainly a method, but it is a bit more cumbersome for the administrator. It is best to have a place for the document to be queried by the administrator and avoid confusion. I don't think everyone is very familiar with the k8s mechanism. I suggest adding a partial description to this section of the : Before: After: I can submit this PR on .
This is going to be true of any api change. When you tried to use it with only adding the controller manager feature gate, I think you would be able to set the retain policy, but there wouldn't be any defaulting or validation (?).
I've seen other feature gates that say what components are needed to enable it to use the new feature. For example: If it's not necessary, then I close .
only adding the controller manager feature gate, I think you would be able to set the retain policy, but there wouldn't be any defaulting or validation (?). I remember just enabling it in kube-controller-manager.


Question: } func pvcDeletePolicyString(policy *apps.StatefulSetPersistentVolumeClaimRetentionPolicy) string { <ins> if policy == nil { return "nullPolicy" } </ins> const retain = apps.RetainPersistentVolumeClaimRetentionPolicyType const delete = apps.DeletePersistentVolumeClaimRetentionPolicyType switch {

it using the retain policy, but the statefulset does not have the policy from output of (It should be the apiserver authentication forbids it)
Yeah I think that makes sense. Thanks for finding this issue! After your PR with the panic fix is submitted, I think we can close this?
Yes. I agree. /close
Closing this issue. <detailsIn response to : Yes. I agree. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "os/exec" "path" "path/filepath" <del> "runtime" </del> "strconv" "strings" "sync"

What happened: Latest kubelet + Windows Docker version 1.14.0-dev refused to start any containers, error output: "invalid option: Windows does not support MemorySwap" How to reproduce it (as minimally and precisely as possible): It's always reproducible with Windows Docker 1.14.0-dev + latest kubelet code base. Anything else do we need to know: It didn't happen for Windows Docker 1.12.2-cs2-ws-beta. The reason behind the issue was because new code was for Windows Docker to check Resources parameters, it doesn't accept a non-zero value for MemorySwap: But kubelet was default to set an unlimited (-1) value. The issue was gone in my setup after applied a private fix to disable MemorySwap by set it to zero for Windows.


Question: // Write 10 lines to log file. // Let ReadLogs start. time.Sleep(50 * time.Millisecond) for line := 0; line < 10; line++ { // Write the first three lines to log file now := time.Now().Format(types.RFC3339NanoLenient)

Runs k8s with master golang: test Failure was observed on Aug 25th, it is passing after that. However it can be reproduced with stress tool. No response No response It is failing on both ppc and x86 architectures. Steps to reproduce: up golang tip version. clone kubernetes master /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
it is a sig node test, sig testing charter is not about doing or maintain tests, we need to remove that old mindset of having a QA department, that is not how software development works or should work :) Besides, thanks for spotting it and submittn a patch /sig node /remove-sig testing
Flaky in
merging this PR will fix the issue


Question: "k8s.io/kubernetes/pkg/kubelet/api/v1alpha1/stats" "k8s.io/kubernetes/test/e2e/framework" <ins> systemdutil "github.com/coreos/go-systemd/util" </ins> . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" "github.com/onsi/gomega/gstruct"

Kubernetes version (use ): 6 Environment: GCE What happened: I created a cluster for Heapster testing purposes to debug failing test in It turned out that Summary API does not export system/misc stats in SystemContainers section (though there are kubelet and docker stats). Everything works as expected on nodes. See also https://k8s-
What node OS were you using? We don't create the system (misc) container on systemd, so it's expected that GCI nodes don't report those stats.
The default configuration for 1.4, which is: master: GCI nodes: Container VM
Are the nodes running 1.4.6 too? There was a bug with this in 1.4.5, but it should have been fixed in 1.4.6.
Yes, both are running in 1.4.6. If you want to reproduce from Heapster current head run . It will take some time and as a side effect you should have a cluster with the mentioned problem. This was catch by integration tests and I temporary disabled the check here
if you have spare cycles, can you look into this?
friendly ping
Looking at it now
I have found a couple things so far. On the containerVM nodes, cadvisor is exporting a container with "name": "/system" On the GCI Master, cadvisor is exporting a container with "name": "" The has this: "{% set system_container = "--system-cgroups=/system" -%}" (/system is the default misc container name) So the real question is: Should cadvisor export "/system" instead of "" on GCI? Or should we be configuring GCI to use "--system-"?
I thought we fix this very issue for 1.4 branch by a while back. Are you running into the same issue with 1.5?
I think I told you the wrong thing earlier. We don't support the misc (/system) cgroup on systemd systems (I can't remember why, but it was an explicit decision - or might know). The is something different, I believe a base systemd cgroup (slice). As for the misc container on ContainerVM nodes, we've been testing for the misc container in 1.5+ in the summary api test (https://k8s-) for a while, and I haven't seen this issue.


Question: "k8s.io/kubernetes/pkg/kubelet/api/v1alpha1/stats" "k8s.io/kubernetes/test/e2e/framework" <ins> systemdutil "github.com/coreos/go-systemd/util" </ins> . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" "github.com/onsi/gomega/gstruct"

So if this is still reproducible, it's limited to 1.4. See this the bug was created for the post-fix.
Found the explanation here: The should use the flag to know the location that has the processes that are associated with , but it should not modify the cgroups of existing processes on the system during bootstrapping of the node. This is because is the on the host and it has not delegated authority to the to change how it manages .
IIUC this is intended behavior on systemd nodes. Based on the explanation provided, the kubelet doesnt have the authority to control "misc" processes on systemd.
+1. "misc" is just a bandaid for cvm. , David Ashpole <:
Ok, so I'm closing this issue.


Question: ```yaml kind: Service apiVersion: v1beta1 <del> id: nginxExample </del> <ins> # must be a DNS compatible name id: nginx-example </ins> # the port that this service should serve on port: 8000 # just like the selector in the replication controller, # but this time it identifies the set of pods to load balance # traffic to. selector: <del> - name: nginx </del> <ins> name: nginx </ins> # the container on each pod to connect to, can be a name # (e.g. 'www') or a number (e.g. 80) containerPort: 80

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: cAdvisor has a option to only collect docker related cgroups. Kubelet stats api now supports allocatable cgroup, /kubepods, for the runtime cgroup and the kubelet cgroup. But with settings, Kubelet stats api can not query pod related cgroups metrics. FYI: What you expected to happen: should be compatible with kubelet stats api.
/cc
/sig node /sig instrumentation
add to cadvisor in A comma-separated list of cgroup path prefix that needs to be collected even docker-only is specified bump cadvisor dep for Kubernetes setting to
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: ```yaml kind: Service apiVersion: v1beta1 <del> id: nginxExample </del> <ins> # must be a DNS compatible name id: nginx-example </ins> # the port that this service should serve on port: 8000 # just like the selector in the replication controller, # but this time it identifies the set of pods to load balance # traffic to. selector: <del> - name: nginx </del> <ins> name: nginx </ins> # the container on each pod to connect to, can be a name # (e.g. 'www') or a number (e.g. 80) containerPort: 80

/close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }) }) <ins> ginkgo.Context("when creating a pod in the host PID namespace", func() { makeHostPidPod := func(podName, image string, command []string, hostPID bool) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ RestartPolicy: v1.RestartPolicyNever, HostPID:       hostPID, Containers: []v1.Container{ { Image:   image, Name:    podName, Command: command, }, }, }, } } createAndWaitHostPidPod := func(podName string, hostPID bool) { podClient.Create(makeHostPidPod(podName, busyboxImage, []string{"sh", "-c", "pidof nginx || true"}, hostPID, )) podClient.WaitForSuccess(podName, framework.PodStartTimeout) } nginxPid := "" ginkgo.BeforeEach(func() { nginxPodName := "nginx-hostpid-" + string(uuid.NewUUID()) podClient.CreateSync(makeHostPidPod(nginxPodName, imageutils.GetE2EImage(imageutils.Nginx), nil, true, )) output := f.ExecShellInContainer(nginxPodName, nginxPodName, "cat /var/run/nginx.pid") nginxPid = strings.TrimSpace(output) }) ginkgo.It("should show its pid in the host PID namespace [NodeFeature:HostAccess]", func() { busyboxPodName := "busybox-hostpid-" + string(uuid.NewUUID()) createAndWaitHostPidPod(busyboxPodName, true) logs, err := e2epod.GetPodLogs(f.ClientSet, f.Namespace.Name, busyboxPodName, busyboxPodName) if err != nil { framework.Failf("GetPodLogs for pod %q failed: %v", busyboxPodName, err) } pids := strings.TrimSpace(logs) framework.Logf("Got nginx's pid %q from pod %q", pids, busyboxPodName) if pids == "" { framework.Failf("nginx's pid should be seen by hostpid containers") } pidSets := sets.NewString(strings.Split(pids, " ")...) if !pidSets.Has(nginxPid) { framework.Failf("nginx's pid should be seen by hostpid containers") } }) ginkgo.It("should not show its pid in the non-hostpid containers [NodeFeature:HostAccess]", func() { busyboxPodName := "busybox-non-hostpid-" + string(uuid.NewUUID()) createAndWaitHostPidPod(busyboxPodName, false) logs, err := e2epod.GetPodLogs(f.ClientSet, f.Namespace.Name, busyboxPodName, busyboxPodName) if err != nil { framework.Failf("GetPodLogs for pod %q failed: %v", busyboxPodName, err) } pids := strings.TrimSpace(logs) framework.Logf("Got nginx's pid %q from pod %q", pids, busyboxPodName) pidSets := sets.NewString(strings.Split(pids, " ")...) if pidSets.Has(nginxPid) { framework.Failf("nginx's pid should not be seen by non-hostpid containers") } }) }) </ins> ginkgo.Context("when creating a pod in the host IPC namespace", func() { makeHostIPCPod := func(podName, image string, command []string, hostIPC bool) *v1.Pod { return &v1.Pod{

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: , Which test(s) are failing: Since when has it been failing: since this PR introduced them Testgrid link: Note that this is the flakiest test in both dashboards, and failing quite frequently. Reason for failure: unknown Anything else we need to know: cc
/sig node
see: , both getting empty PID and multiple PIDs.
opened revert at
for discoverability for the release team /priority critical-urgent /milestone v1.17


Question: m.mutex.Lock() defer m.mutex.Unlock() <del> if _, exists := m.endpoints[resourceName]; exists { </del> <ins> if ep, exists := m.endpoints[resourceName]; exists { </ins> m.markResourceUnhealthy(resourceName) <del> klog.V(2).InfoS("Endpoint became unhealthy", "resourceName", resourceName, "endpoint", m.endpoints[resourceName]) } </del> <ins> klog.V(2).InfoS("Endpoint became unhealthy", "resourceName", resourceName, "endpoint", ep) </ins> <del> m.endpoints[resourceName].e.setStopTime(time.Now()) </del> <ins> ep.e.setStopTime(time.Now()) } </ins> } // PluginListAndWatchReceiver receives ListAndWatchResponse from a device plugin

Seen in and in No hits on stack strace nothing evident on the git history of the test itself, but it seems , so most probable something in the device manager has changed? /sig node /kind flake
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/cc as this flakes a lot in


Question: allErrs = append(allErrs, field.Duplicate(fldPath.Child("name"), driver.Name)) } driverNamesInSpecs.Insert(driver.Name) <del> topoKeys := make(sets.String) </del> <ins> topoKeys := sets.New[string]() </ins> for _, key := range driver.TopologyKeys { if len(key) == 0 { allErrs = append(allErrs, field.Required(fldPath, key))

It seems to be valid to add here. Would you like to open a pull request for it? this is like a cleanup, as the set insert will skip and the validation will do twice. is used to track bugs. I suggest you open a PR for that if you like.
/sig storage
I think adding ` == true and we didn't skip, it will insert set twice I believe? wasn't notice that at first glance. However, I can file a PR now
/triage accepted
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
Please feel free to submit a PR.


Question: e.clientConn.Close() } <del> // dial establishes the gRPC communication with the registered device plugin. </del> <ins> // dial establishes the gRPC communication with the registered device plugin. https://godoc.org/google.golang.org/grpc#Dial </ins> func dial(unixSocketPath string) (pluginapi.DevicePluginClient, *grpc.ClientConn, error) { <del> c, err := grpc.Dial(unixSocketPath, grpc.WithInsecure(), </del> <ins> c, err := grpc.Dial(unixSocketPath, grpc.WithInsecure(), grpc.WithBlock(), grpc.WithTimeout(10*time.Second), </ins> grpc.WithDialer(func(addr string, timeout time.Duration) (net.Conn, error) { return net.DialTimeout("unix", addr, timeout) }),

This is not 100% reproducable in my cluster. There is 20% chance of failing in my environment.
/area hw-accelerators mentioned a root cause that might be due to unit tests sharing the same Unix socket and interfering with each others. cc
Can you please share latest commit id of your code base using ?
Thanks very much, can you try the follow command please:
/assign
The latest commit id of my code base:
I found this is not 100% reproducable in my cluster. There is 20% chance of failing in my environment.
Result of go test command:
/sig scheduling
Thanks!
Hi, does these unit tests pass with PR
I run 100 times without failure.
Good to here that, Nice work! Thanks!


Question: type LRUExpireCache struct { cache *lru.Cache <del> lock  sync.RWMutex </del> <ins> lock  sync.Mutex </ins> } func NewLRUExpireCache(maxSize int) *LRUExpireCache {

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "LRUExpireCache" Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT What happened: requires write lock since needs to manipulate its list to track recently used item. Currently it uses read lock so it may introduce race condition. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else do we need to know:
This was fixed in


Question: "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/dynamic" "k8s.io/client-go/kubernetes"

reported that conformance testing with the latest version of sonobuoy against a version 1.29.0-alpha.3 cluster fails on the support the 1.17 Sample API Server using the current Aggregator test: , related to I found the reason why the test is working fine in Kubernetes CI jobs but fails with sonobuoy is because the service account used by sonobuoy "sonobuoy-serviceaccount" doesn't have read permission of the non-resource url "/", which is used by the test to confirm the removal of group path: The permission of "sonobuoy-serviceaccount" is as below: There may be two ways to fix it: the RBAC rule of sonobuoy to include "get" of "/". However, could other test runners encounter this issue? Is there a doc listing the required permissions for running e2e tests? the e2e to skip checking the removal of the group path if the service account doesn't have permission to access the root path. Which way do we usually prefer? Running e2e with sonobuoy should succeed. a v1.29.0-alpha.3 cluster. No response <details</details<details</details<details</details<details</details<details</details<details</details
implemented this solution.
/sig api-machinery
thank you for opening this issue and working on a solution. I'm probably not the right person to decide on which solution is preferred. From my perspective, either solution is fine.
Note: Sonobuoy is not owned by any Kubernetes SIG. At most it is between SIG Architecture and the CNCF to choose whether to recommend it for running tests. SIG Testing owns the e2e image that sonobuoy can be used to schedule but not sonobuoy. IMHO this is a bug in running the tests via a pod, which is not how we develop the tests (they're run outside the cluster, by a cluster admin account typically). I would say 1) is the correct answer. xref: /sig architecture /sig testing /triage accepted cc
thanks for your comment. I guess 1) means non of the existing sonobuoy versions can pass K8s conformance test against a cluster 1.29.0-alpha3, as the RBAC rule is hardcoded.


Question: "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/dynamic" "k8s.io/client-go/kubernetes"

I will file an issue to sonobuoy first.
question here is what are the set of client permissions required to run the conformance suite? perhaps a level of access or a more granular recommendation should be documented. what is your take on the root path requirement as approver for: EDIT: slack thread:
I agree, we already require e.g. privileged pods for a long time in conformance tests and we should make it clear that we expect a highly privileged client.
+1 for what is saying, that we should expect that the user running the tests has required permissions. These tests are not built for being run unprivileged.
i wanted to PR the admin note in our conformance docs under sig-arch, but then i saw this: so broke this guarantee. sounds like SIG Arch needs to comment.
This statement surprises me. Root discovery is one of the least-privileged permissions you can have on a cluster. For years this was even open for unauthorized users on the internet (if your api endpoint has been exposed). In other words, just add the permission. While thinking about it, it is surprising that the user in question does not have this access already. Don't we allow that for every authorized user?
In contrast, I hope that lots of privileged operations are tested by conformance today. If we don't test things like admission webhooks for example, we have much bigger problems. And admission webhooks are basically equivalent to cluster-admin.
that is outdated, we use hostNetwork pods one some of the tests IIRC
looks like we don't. before this ticket, i was under the impression that the conformance suite already requires admin and that all testers (like Sonobuoy) use admin level credentials. it is non-privileged (e.g., does not require root on nodes, access to raw network interfaces, or cluster admin permissions) looks like we may have to change that sentence completely.
Would have expected it here: The PR in question explicitly fixes the result returned by . If that is part of conformance (IMO it should), every conformance test needs access it obviously. Whether we require admin, or we add it manually, either would work.


Question: "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/dynamic" "k8s.io/client-go/kubernetes"

adding to milestone and making release blocking ... we should not release 1.29 in a state that the most commonly used conformance tool will fail
IMO we should clarify the state and go fix the tool, but we should keep the test and we should NOT add skip. Conformance docs are not a release artifact, and sonobuoy release lifecycle is not a conformance artifact. If we can't get sonobuoy working in a reasonable time, we already know the tests can be run by scheduling the image (which is ours) directly with kubectl instead and we can document that in the upstream conformance docs. We need to fix the for this image: ,%20master%20(dev) And it should probably be to release-blocking in the future.
it looks like the e2e change made in was not consciously made part of conformance, it was just trying to functionally test the bugfix being made. For 1.29, I think we should revert the change to and make an integration test that validates the fix instead. We can revisit whether content of is part of conformance or not for 1.30
I opened to revert the e2e change and move the functional test to an integration test
Filed to try to get the image job passing again.
We probably need another issue to track 1.30 follow-up?
If no one has done it, I could create another issue and follow up with the items that have been discussed here: outdated privilege requirement in commonly used conformance tools like sonobuoy aware of the required priviledge to avoid similar situations in the future. permission of GET "/" to authorized users via the role for root discovery.


Question: "attachRequired":       "attachRequired indicates this CSI volume driver requires an attach operation (because it implements the CSI ControllerPublishVolume() method), and that the Kubernetes attach detach controller should call the attach volume interface which checks the volumeattachment status and waits until the volume is attached before proceeding to mounting. The CSI external-attacher coordinates with CSI volume driver and updates the volumeattachment status when the attach operation is complete. If the CSIDriverRegistry feature gate is enabled and the value is specified to false, the attach operation will be skipped. Otherwise the attach operation will be called.nnThis field is immutable.", "podInfoOnMount":       "If set to true, podInfoOnMount indicates this CSI volume driver requires additional pod information (like podName, podUID, etc.) during mount operations. If set to false, pod information will not be passed on mount. Default is false. The CSI driver specifies podInfoOnMount as part of driver deployment. If true, Kubelet will pass pod information as VolumeContext in the CSI NodePublishVolume() calls. The CSI driver is responsible for parsing and validating the information passed in as VolumeContext. The following VolumeConext will be passed if podInfoOnMount is set to true. This list might grow, but the prefix will be used. "csi.storage.k8s.io/pod.name": pod.Name "csi.storage.k8s.io/pod.namespace": pod.Namespace "csi.storage.k8s.io/pod.uid": string(pod.UID) "csi.storage.k8s.io/ephemeral": "true" if the volume is an ephemeral inline volumen                                defined by a CSIVolumeSource, otherwise "false"nn"csi.storage.k8s.io/ephemeral" is a new feature in Kubernetes 1.16. It is only required for drivers which support both the "Persistent" and "Ephemeral" VolumeLifecycleMode. Other drivers can leave pod info disabled and/or ignore this field. As Kubernetes 1.15 doesn't support this field, drivers can only support one mode when deployed on such a cluster and the deployment determines which mode that is, for example via a command line parameter of the driver.nnThis field is immutable.", "volumeLifecycleModes": "VolumeLifecycleModes defines what kind of volumes this CSI volume driver supports. The default if the list is empty is "Persistent", which is the usage defined by the CSI specification and implemented in Kubernetes via the usual PV/PVC mechanism. The other mode is "Ephemeral". In this mode, volumes are defined inline inside the pod spec with CSIVolumeSource and their lifecycle is tied to the lifecycle of that pod. A driver has to be aware of this because it is only going to get a NodePublishVolume call for such a volume. For more information about implementing this mode, see https://kubernetes-csi.github.io/docs/ephemeral-local-volumes.html A driver can support one or more of these modes and more modes may be added in the future.nnThis field is immutable.", <del> "storageCapacity":      "If set to true, storageCapacity indicates that the CSI volume driver wants pod scheduling to consider the storage capacity that the driver deployment will report by creating CSIStorageCapacity objects with capacity information.nnThe check can be enabled immediately when deploying a driver. In that case, provisioning new volumes with late binding will pause until the driver deployment has published some suitable CSIStorageCapacity object.nnAlternatively, the driver can be deployed with the field unset or false and it can be flipped later when storage capacity information has been published.nnThis field is immutable.nnThis is a beta field and only available when the CSIStorageCapacity feature is enabled. The default is false.", </del> <ins> "storageCapacity":      "If set to true, storageCapacity indicates that the CSI volume driver wants pod scheduling to consider the storage capacity that the driver deployment will report by creating CSIStorageCapacity objects with capacity information.nnThe check can be enabled immediately when deploying a driver. In that case, provisioning new volumes with late binding will pause until the driver deployment has published some suitable CSIStorageCapacity object.nnAlternatively, the driver can be deployed with the field unset or false and it can be flipped later when storage capacity information has been published.nnThis field was immutable in Kubernetes <= 1.22 and now is mutable.nnThis is a beta field and only available when the CSIStorageCapacity feature is enabled. The default is false.", </ins> "fsGroupPolicy":        "Defines if the underlying volume supports changing ownership and permission of the volume before being mounted. Refer to the specific FSGroupPolicy values for additional details. This field is beta, and is only honored by servers that enable the CSIVolumeFSGroupPolicy feature gate.nnThis field is immutable.nnDefaults to ReadWriteOnceWithFSType, which will examine each volume to determine if Kubernetes should modify ownership and permissions of the volume. With the default policy the defined fsGroup will only be applied if a fstype is defined and the volume's access mode contains ReadWriteOnce.", "tokenRequests":        "TokenRequests indicates the CSI driver needs pods' service account tokens it is mounting volume for to do necessary authentication. Kubelet will pass the tokens in VolumeContext in the CSI NodePublishVolume calls. The CSI driver should parse and validate the following VolumeContext: "csi.storage.k8s.io/serviceAccount.tokens": {n  "<audience>": {n    "token": <token>,n    "expirationTimestamp": <expiration timestamp in RFC3339>,n  },n  ...n}nnNote: Audience in each TokenRequest should be different and at most one token is empty string. To receive a new token after expiry, RequiresRepublish can be used to trigger NodePublishVolume periodically.", "requiresRepublish":    "RequiresRepublish indicates the CSI driver wants `NodePublishVolume` being periodically called to reflect any possible change in the mounted volume. This field defaults to false.nnNote: After a successful initial NodePublishVolume call, subsequent calls to NodePublishVolume should only update the contents of the volume. New mount points will not be seen by a running container.",

Up to 1.21, that field is immutable. That means in practice, a CSIDriver object has to be deleted and then re-created with a different value instead. Mutation via delete/created is unnecessarily complex and as the only consumer of that field (the kube-scheduler) can handle mutation just fine (it'll always ask the informer cache for the current value), it's better to allow mutation.
/sig storage /assign
This is independent of allowing other fields to become mutable, but that might also make sense.
/triage accepted


Question: allErrs := errs.ValidationErrorList{} allErrs = append(allErrs, ValidateObjectMetaUpdate(&oldNamespace.ObjectMeta, &newNamespace.ObjectMeta).Prefix("metadata")...) newNamespace.Spec = oldNamespace.Spec <ins> if newNamespace.DeletionTimestamp.IsZero() { if newNamespace.Status.Phase != api.NamespaceActive { allErrs = append(allErrs, errs.NewFieldInvalid("Status.Phase", newNamespace.Status.Phase, "A namespace may only be in active status if it does not have a deletion timestamp.")) } } else { if newNamespace.Status.Phase != api.NamespaceTerminating { allErrs = append(allErrs, errs.NewFieldInvalid("Status.Phase", newNamespace.Status.Phase, "A namespace may only be in terminating status if it has a deletion timestamp.")) } } </ins> return allErrs }

There is a funny subtle behvaiour in get namespaces, in which if i call more than one time, it seems that the removal from Terminating state is accelarated. It seems this might indicate a bug, or maybe its not a bug and the eventual semantics of is a little funny sometimes. in any case, here's what we are seeing : 1) when i delete a namespace the first time usning kubectl, we get a state change to . that is correct, of course. 2) but ... if make the call a second time, then the same status line is immediately removed from . Expected behaviour : A second call to "kubectl delete ..." shouldn't have any effect on the timing of removal of a namespace from 'Terminating' state*... So i guess there is some "eventual consistency" subtle bug that is resolved by repeated calls to
suggests this may not be e bug, maybe its just the etcd eventual deleter thingy.
Looks to definitely be a bug. The 2nd delete seems to have interrupted the delete process and artifacts (like events) from the namespace still exist in etcd.
investigating...
oops, see the error, will submit a fix.
I have a fix, but need a test case, will submit PR tomorrow AM


Question: opts.Client = c opts.PodName = name opts.Namespace = ns <ins> // TODO: opts.Run sets opts.Err to nil, we need to find a better way stderr := opts.Err </ins> if err := opts.Run(); err != nil { <del> fmt.Fprintf(opts.Err, "Error attaching, falling back to logs: %vn", err) </del> <ins> fmt.Fprintf(stderr, "Error attaching, falling back to logs: %vn", err) </ins> req, err := f.LogsForObject(pod, &api.PodLogOptions{Container: ctrName}) if err != nil { return err

This was build from master earlier today.
cc This should be fixed before v1.4
This is critical
This needs to be tracked as a 1.4 blocker
ETA for resolution: This afternoon This is in the process of being cherrypicked


Question: ) const ( <del> defaultSandboxImage = "gcr.io/google_containers/pause-amd64:3.0" </del> <ins> defaultSandboxImage = "gcr.io/google_containers/pause-amd64:3.1" </ins> // Various default sandbox resources requests/limits. defaultSandboxCPUshares int64 = 2

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the pause-amd64:3.0 image on has an empty schema2 manifest, which prevents is from being pulled by proxy/caches which only support V2 registries. What you expected to happen: Being able to pull ` using a V2 manifest compatible client. Anything else we need to know?: The proxy/cahce is requesting this
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After talking with , he mentionned the following
AFAIK, does not support well. After inspecting the image, you can find is still using manifest v1.
/cc FYI
Im not sure which exact version x schéma X is needed. What I know is that we can download through the proxy/cache all other Kubernetes kube-system images, only the pause image is giving us trouble. Therefore, I would say that having the image available using the same format as the other kube-system images would be the request here.
Yeah, we haven't updated the pause image in a while, since it doesn't really ever change. I'm not sure whether rebuilding + pushing over the existing tag is safe, or if we should bump to or something. thoughts?
If we do rebuilding/updating the image, that would help and a bunch of similar issues.
it's better/save to go with new tag
I didn't realize there were changes to the pause binary that were never actually deployed. It seems like we should probably just do that (and bump the tag).
that sounds great. Any idea when the existing 3.0 tag could be pushed again?
We shouldn't push over the 3.0 tag; we should push a 3.1 or 4.0 tag instead. I'm not sure who owns the pause container who should drive this. Maybe sig-node and/or
It seems like it should be sig-node. I'll look into it.
The new container has been built, can you test with
Its working! thanks! I was able to pull it through our artifactory proxy , Lee Verberne <:
excellent, thanks for checking. /assign


Question: ) const ( <del> defaultSandboxImage = "gcr.io/google_containers/pause-amd64:3.0" </del> <ins> defaultSandboxImage = "gcr.io/google_containers/pause-amd64:3.1" </ins> // Various default sandbox resources requests/limits. defaultSandboxCPUshares int64 = 2

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: pause does not reap zombie processes using the default pause container provided by What you expected to happen: should include the up to date pause code to reap zombie processes when using docker = 1.13.1 found here How to reproduce it (as minimally and precisely as possible): a.out will leave a zombie process that is not cleaned up Anything else we need to know?: Building the pause container from master solves this issue Related: Environment: Kubernetes version (use ): 1.7.0+ Cloud provider or hardware configuration**: N/A OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 4.10.13+ Install tools: Others: docker = 1.13.1
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
i get a simliar problem when i run crond service in pod container, every day it makes some [anacron] zombie progresses UID PID PPID C STIME TTY TIME CMD root 1 0 0 Nov14 ? 00:00:00 /pause root 7 0 0 Nov14 ? 00:00:00 /usr/local/bin/dumb-init /usr/lo root 13 7 0 Nov14 ? 00:00:00 nginx: master process /usr/local nobody 21 13 0 Nov14 ? 00:00:01 nginx: worker process nobody 22 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 23 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 24 13 0 Nov14 ? 00:00:05 nginx: worker process root 77 1 0 Nov14 ? 00:00:00 [anacron] root 1 0 00:01 ? 00:00:00 [anacron] <defunctroot 1 0 01:01 ? 00:00:00 [anacron] <defunctroot 1 0 02:01 ?


Question: ) const ( <del> defaultSandboxImage = "gcr.io/google_containers/pause-amd64:3.0" </del> <ins> defaultSandboxImage = "gcr.io/google_containers/pause-amd64:3.1" </ins> // Various default sandbox resources requests/limits. defaultSandboxCPUshares int64 = 2

00:00:00 [anacron] <defunctroot 1 0 03:01 ? 00:00:00 [anacron] <defunctroot 0 0 11:49 ? 00:00:00 /bin/bash root 0 1 19:15 ? 00:00:00 /bin/bash root 0 19:15 ? 00:00:00 ps -ef !
Do we know why this was never resolved / was never deployed?
I just fought through this issue with a rabbitmq helm chart. I had to custom build a pause container from master to fix it. The rabbitmq helm chart uses for its liveness checks. It's a thin wrapper around a call to the rabbitmq-server daemon, which is not PID 1 (/pause is PID 1). The rabbitmq-server process forks some children to do the work and then they exit. Since the pause-amd64:3.0 is not reaping, those processes stick around as zombies. The liveness check runs every 5 seconds. After about 4 hours, there are about 27K zombies and the node begins failing to launch new containers because it's starved for resources. The pause container built from current master fixes that behavior, no zombies at all in the container.
The plan is to build a new version of pause-amd64 and then update the image referenced by Kubernetes. This means the default image will change in the next Kubernetes release. We can make the new image available once the freeze lifts and we can merge .
Looks like this made it into 1.10