Question: glog.Errorf("Failed to get hostname. err: %+v", err) return nil, err } <del> vs.vmUUID, err = getVMUUID() if err != nil { glog.Errorf("Failed to get uuid. err: %+v", err) return nil, err </del> <ins> if cfg.Global.VMUUID != "" { vs.vmUUID = cfg.Global.VMUUID } else { vs.vmUUID, err = getVMUUID() if err != nil { glog.Errorf("Failed to get uuid. err: %+v", err) return nil, err } </ins> } runtime.SetFinalizer(vs, logout) return vs, nil

Is this a BUG REPORT or FEATURE REQUEST?: What happened: Currently the vsphere cloud provider blocks the controller-manager from running if its running on a non-vmware host, because it reads and checks if its content starts with . What you expected to happen: The controller manager to start How to reproduce it (as minimally and precisely as possible): Start a on a non-vmware environment with the flags set Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig vmware
can you please share some more context- Why do you want vSphere cloud provider to work with non-vmware host (I believe you mean ESXi)? Where is this requirement coming from.
We are building a Kubernetes cluster management solution which does Kubernetes in Kubernetes and works by creating the control planes in one cluster, then creating worker instances on one of the supported platforms, among them vsphere. Currently the cluster hosting the control planes (We call it seed cluster) has to run on ESXi, otherwise the controller-manager crashes. Having the check in place for the Kubelets is completely fine and definitely makes sense, but for the controller-manager it should IMHO not be required.


Question: <del> apiVersion: v1 kind: ReplicationController metadata: name: es-client labels: component: elasticsearch role: client spec: replicas: 1 template: metadata: labels: component: elasticsearch role: client spec: serviceAccount: elasticsearch containers: - name: es-client securityContext: capabilities: add: - IPC_LOCK image: quay.io/pires/docker-elasticsearch-kubernetes:1.7.1-4 env: - name: KUBERNETES_CA_CERTIFICATE_FILE value: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt - name: NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: "CLUSTER_NAME" value: "myesdb" - name: NODE_MASTER value: "false" - name: NODE_DATA value: "false" - name: HTTP_ENABLE value: "true" ports: - containerPort: 9200 name: http protocol: TCP - containerPort: 9300 name: transport protocol: TCP volumeMounts: - mountPath: /data name: storage volumes: - name: storage emptyDir: {} </del>

Presently, the elasticsearch example Kubernetes configurations target elasticsearch v1.7.1, which is quite old. As of this writing, the latest elasticsearch release is v5.6.2, with many deployment changes since then. Could the elasticsearch examples be updated to target the v5 series of elasticsearch? See for a more up to date reference configuration.
cc
hey it's been a long while since I last contributed to the docs. i can do it soon but i need pointers on how to properly do it.
hey, so we need to update the stuff under examples/elasticsearch/ *&repos=kubernetes So basically update the images, yaml files etc and test that everything works (and submit a PR)
/sig testing
Ah, so there's no documentation updates anymore. OK I'll take care of it, assign it to me.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
GitHub didn't allow me to assign the following users: me. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign me Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign pires
/remove-lifecycle stale
it seems that the example moved to I am now reading that repo guidelines and will update this there. I'll also open a PR that cleans this folder in the current repo and closes this issue.


Question: return res, nil } <ins> // doCmdWithoutAttr a simple wrapper of netlink socket execute command func (i *Handle) doCmdWithoutAttr(cmd uint8) ([][]byte, error) { req := newIPVSRequest(cmd) req.Seq = atomic.AddUint32(&i.seq, 1) return execute(i.sock, req, 0) } </ins> func assembleDestination(attrs []syscall.NetlinkRouteAttr) (*Destination, error) { var d Destination

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently, we flush ipvs services by deleting them one by one. That's because there was no flush API in libnetwork. Now, libnetwork supports flush API, I think it's time to rsync. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig network
/area kube-proxy
/assign


Question: } return false, nil } <ins> // stripComments will transform a YAML file into JSON, thus dropping any comments // in it. Note that if the given file has a syntax error, the transformation will // fail and we will manually drop all comments from the file. func stripComments(file []byte) []byte { stripped, err := yaml.ToJSON(file) if err != nil { stripped = manualStrip(file) } return stripped } // manualStrip is used for dropping comments from a YAML file func manualStrip(file []byte) []byte { stripped := []byte{} for _, line := range bytes.Split(file, []byte("n")) { if bytes.HasPrefix(bytes.TrimSpace(line), []byte("#")) { continue } stripped = append(stripped, line...) stripped = append(stripped, 'n') } return stripped } </ins>

When trying to edit multiple resources, for example: will panic. Ref as the possible cause cc
if was the cause, any fix needs to happen in 1.1 as well, was already cherry picked there
Also shows that we need a test for this case.
is the cause. When we edit multiple resources (i.e., list-type objects), the list's Accessor.annotations (type ) is nil and will panic when UpdateApplyAnnotation tries to SetAnnotations ().


Question: t.Errorf("unexpected portallocator state: %d free", free) } } <ins> func TestCollectServiceNodePorts(t *testing.T) { tests := []struct { name        string serviceSpec corev1.ServiceSpec expected    []int }{ { name: "no duplicated nodePorts", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolTCP}, {NodePort: 112, Protocol: corev1.ProtocolUDP}, {NodePort: 113, Protocol: corev1.ProtocolUDP}, }, }, expected: []int{111, 112, 113}, }, { name: "duplicated nodePort with TCP protocol", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolTCP}, {NodePort: 111, Protocol: corev1.ProtocolTCP}, {NodePort: 112, Protocol: corev1.ProtocolUDP}, }, }, expected: []int{111, 111, 112}, }, { name: "duplicated nodePort with UDP protocol", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolUDP}, {NodePort: 111, Protocol: corev1.ProtocolUDP}, {NodePort: 112, Protocol: corev1.ProtocolTCP}, }, }, expected: []int{111, 111, 112}, }, { name: "duplicated nodePort with different protocol", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolTCP}, {NodePort: 112, Protocol: corev1.ProtocolTCP}, {NodePort: 111, Protocol: corev1.ProtocolUDP}, }, }, expected: []int{111, 112}, }, { name: "no duplicated port(with health check port)", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolTCP}, {NodePort: 112, Protocol: corev1.ProtocolUDP}, }, HealthCheckNodePort: 113, }, expected: []int{111, 112, 113}, }, { name: "nodePort has different protocol with duplicated health check port", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolUDP}, {NodePort: 112, Protocol: corev1.ProtocolTCP}, }, HealthCheckNodePort: 111, }, expected: []int{111, 112}, }, { name: "nodePort has same protocol as duplicated health check port", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolUDP}, {NodePort: 112, Protocol: corev1.ProtocolTCP}, }, HealthCheckNodePort: 112, }, expected: []int{111, 112, 112}, }, } for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { ports := collectServiceNodePorts(&corev1.Service{ ObjectMeta: metav1.ObjectMeta{Namespace: "one", Name: "one"}, Spec:       tc.serviceSpec, }) sort.Ints(ports) if !reflect.DeepEqual(tc.expected, ports) { t.Fatalf("Invalid resultnexpected: %vngot: %v", tc.expected, ports) } }) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I created several Workloads which require up to ten ports for both UDP and TCP protocol. Now my logs get spammed with . What you expected to happen: should consider -combination instead of only . How to reproduce it (as minimally and precisely as possible): Deploy any service with a node-port with both protocols, e.g.: Anything else we need to know?: This was already acknowledged by here: However, as this thread addresses a different bug and seems kind of stuck, i'd like to suggest that this might be addressed in a new context. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Deployed via Rancher v2.4.3
/sig api-machinery
/remove-sig api-machinery /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
How did you get into this situation? The allocator should not have granted the same node port twice?
Oh right, that issue. The only real way out is to de-allocate the NodePorts and then re-allocate them, I think
As this happens for all of my ports in the said Production Application (the one where my log gets spammed) as well as complete fresh installation (where i verified the bug), i think there's no "way out" except fixing the bug. As said, for me it's necessary to assign both protocols to the same port. This might be a niche-use-case, but checking the it's actually very likely for ports to allow tcp AND udp. I'm not sure about the history of this, but it might be that issue was the origin, which was adressed in PR ("Allow services which use same port, different protocol to use the same nodePort for both"). But this PR doesn't contain the adjustment of the said , so still considers the behaviour as invalid, while it's actually allowed and supported.


Question: t.Errorf("unexpected portallocator state: %d free", free) } } <ins> func TestCollectServiceNodePorts(t *testing.T) { tests := []struct { name        string serviceSpec corev1.ServiceSpec expected    []int }{ { name: "no duplicated nodePorts", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolTCP}, {NodePort: 112, Protocol: corev1.ProtocolUDP}, {NodePort: 113, Protocol: corev1.ProtocolUDP}, }, }, expected: []int{111, 112, 113}, }, { name: "duplicated nodePort with TCP protocol", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolTCP}, {NodePort: 111, Protocol: corev1.ProtocolTCP}, {NodePort: 112, Protocol: corev1.ProtocolUDP}, }, }, expected: []int{111, 111, 112}, }, { name: "duplicated nodePort with UDP protocol", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolUDP}, {NodePort: 111, Protocol: corev1.ProtocolUDP}, {NodePort: 112, Protocol: corev1.ProtocolTCP}, }, }, expected: []int{111, 111, 112}, }, { name: "duplicated nodePort with different protocol", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolTCP}, {NodePort: 112, Protocol: corev1.ProtocolTCP}, {NodePort: 111, Protocol: corev1.ProtocolUDP}, }, }, expected: []int{111, 112}, }, { name: "no duplicated port(with health check port)", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolTCP}, {NodePort: 112, Protocol: corev1.ProtocolUDP}, }, HealthCheckNodePort: 113, }, expected: []int{111, 112, 113}, }, { name: "nodePort has different protocol with duplicated health check port", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolUDP}, {NodePort: 112, Protocol: corev1.ProtocolTCP}, }, HealthCheckNodePort: 111, }, expected: []int{111, 112}, }, { name: "nodePort has same protocol as duplicated health check port", serviceSpec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{ {NodePort: 111, Protocol: corev1.ProtocolUDP}, {NodePort: 112, Protocol: corev1.ProtocolTCP}, }, HealthCheckNodePort: 112, }, expected: []int{111, 112, 112}, }, } for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { ports := collectServiceNodePorts(&corev1.Service{ ObjectMeta: metav1.ObjectMeta{Namespace: "one", Name: "one"}, Spec:       tc.serviceSpec, }) sort.Ints(ports) if !reflect.DeepEqual(tc.expected, ports) { t.Fatalf("Invalid resultnexpected: %vngot: %v", tc.expected, ports) } }) } } </ins>

/assign
Ahh, I thought you meant that you had accidentally been allocated the same nodePorts and I was very confused when you said "all of them". I have reproduced this issue, and I can see it by inspection. The obvious fix would be to collect ports split by protocol, but I think that's not right since we don't store it split. An even simpler fix would be for to build a . Then for each protocol, for each port, build that contains all the port numbers, but remove any duplicates ONLY IF they are duplicated across protocols. Duplicates in the same protocol should be retained (to report an error later). I don't have time to tackle this tonight (and write a test), but it seems like a good bug.
perhaps I can fix this problem /assign
things may be a little more complicated. Right now the is not aware of the protocol of the port so I guess simply rewriting function to return is not enough, the port allocator will still return on same node port even with different protocol. perhaps we need to refactor the to make it protocol-aware.
I don't think we need to do make the allocator be protocol aware. IIRC we allow you to use the same nodeport on different protocols ONLY within a single service. This means (I think) that we can do hyper-local deduplication as we collect the nodeports for each service. If we find the same nodeport number with different protocols, just remove the duplicates because it is allowed. I.e. a service that looks like: return - deduped. if ANY OTHER service tried to use it would be an error at the "rebuilt" allocator Likewise if a single Service claims to use the same nodeport number more than once on the same protocol, it should be an error: return - NOT deduped.
obviously, we want a great unit test here :P
ah got it! thanks for your detailed explanation!
Hi! I have filed a PR to resolve this issue, could you take a look?


Question: dir := cacheDir if len(dir) > 0 { version, err := clientset.Discovery().ServerVersion() <del> if err != nil { return nil, err </del> <ins> if err == nil { dir = path.Join(cacheDir, version.String()) } else { dir = "" // disable caching as a fallback </ins> } <del> dir = path.Join(cacheDir, version.String()) </del> } fedClient, err := clients.FederationClientForVersion(nil) if err != nil {

kubectl works mostly fine without /version. But a number of commands (apply, create, convert, ....) are wired to this: A 404 on /version let's the kubectl to fail hard. For environments where might be a security problem (cf. ) this is a nogo. Relates to minimal apiserver routes:
/cc
/version is desirable (we think all servers should have it) but it should not be required for the proper functioning of a client to be able to retrieve it.


Question: } } <ins> // createCustomResourceDefinition removes potentially stale storage so it gets re-created func (r *crdHandler) createCustomResourceDefinition(obj interface{}) { crd := obj.(*apiextensions.CustomResourceDefinition) r.customStorageLock.Lock() defer r.customStorageLock.Unlock() // this could happen if the create event is merged from create-update events r.removeStorage_locked(crd.UID) } // updateCustomResourceDefinition removes potentially stale storage so it gets re-created </ins> func (r *crdHandler) updateCustomResourceDefinition(oldObj, newObj interface{}) { oldCRD := oldObj.(*apiextensions.CustomResourceDefinition) newCRD := newObj.(*apiextensions.CustomResourceDefinition)

Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestWebhookConverterWithDefaulting /assign /priority important-soon /area custom-resources /sig api-machinery /kind flake /milestone v1.16
/cc
several of the webhook converter tests have similar flakes. it looks like the wait for storage version to become effective times out occasionally Rare flakes seen in all of these tests: TestWebhookConverterWithWatchCache/nontrivial-converter TestWebhookConverterWithWatchCache/metadata-mutating-converter TestWebhookConverterWithDefaulting/metadata-mutating-converter TestWebhookConverterWithWatchCache/noop-converter TestWebhookConverterWithDefaulting/empty-response TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-2 TestWebhookConverterWithoutWatchCache/noop-converter/check-0/v1beta2 TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-0/v1beta2
cc
Yikes, I’ll sort it out.
iiuc sending an empty patch gets short-circuited and doesn't bump CR generation, even if the storage version gets changed, which means we do not re-write to etcd and go through the encoding path this would be racing and doing noop's if the first pass of didn't get the expected storage version I will send a fix to make the patch do actual mutation
It doesn't bump generation, but the bytes to persist in etcd would still be different if the storage version had changed, so they should persist. You can verify that by changing the storage version on a CRD yourself, do an empty patch, and see if the resourceVersion changes.
you're right. Generation is for non-metadata change only, and the RV changes. I queried etcd locally and the apiVersion did change for an empty patch (sidetracking: another observation is after changing the storage version (say v1 -v2; noop convertor), empty-patching v2 endpoint would bump generation, while empty-patching v1 endpoint wouldn't)
another theory: our cached storage strategy could be stale if the CRD informer merges multiple events together (e.g. the informer does a re-list).


Question: } } <ins> // createCustomResourceDefinition removes potentially stale storage so it gets re-created func (r *crdHandler) createCustomResourceDefinition(obj interface{}) { crd := obj.(*apiextensions.CustomResourceDefinition) r.customStorageLock.Lock() defer r.customStorageLock.Unlock() // this could happen if the create event is merged from create-update events r.removeStorage_locked(crd.UID) } // updateCustomResourceDefinition removes potentially stale storage so it gets re-created </ins> func (r *crdHandler) updateCustomResourceDefinition(oldObj, newObj interface{}) { oldCRD := oldObj.(*apiextensions.CustomResourceDefinition) newCRD := newObj.(*apiextensions.CustomResourceDefinition)

Two scenarios: -create-update events merged into an update event: strategy with UID:a was created on demand for CRD Foo Foo was deleted and recreated strategy with UID:b was created on demand for the new Foo new Foo got updated (e.g. switching storage version) the strategy with UID:b is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the delete-create-update events into , the strategy with UID:a will be deleted, but the strategy with UID:b won't. -update events merged into a create event: strategy with UID:a was created on demand for CRD Foo Foo was updated (e.g. switching storage version) the strategy with UID:a is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the create-update events into , the strategy with UID:a won't be deleted (as we don't react on create events). another race found in , which could happen in a single update.
I think I :


Question: } func (tc *patchTestCase) runner(t *testing.T) { <ins> scheme := NewTestScheme() metav1.AddMetaToScheme(scheme) </ins> client := NewSimpleMetadataClient(scheme, tc.object) resourceInterface := client.Resource(schema.GroupVersionResource{Group: testGroup, Version: testVersion, Resource: testResource}).Namespace(testNamespace)

Because of this scheme mutation, this leads to concurrent use of the same scheme to cause the program to panic. It would be ideal to do this in some concurrent safe manner or add a warning or similar
/sig api-machinery
/assign
/assign /triage accepted


Question: SchedulingAlgorithmPremptionEvaluationDuration, PreemptionVictims, PreemptionAttempts, <ins> equivalenceCacheLookups, EquivalenceCacheWrites, </ins> } )

/kind feature The equivalence cache feature is planned to move to beta in 1.11, and it would be useful to have some metrics attached to it. The equivalence cache should improve scheduling throughput, so we want to be able to correlate e.g. throughput with eCache hit rate. /sig scheduling /sig instrumentation
/assign misterikkit
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: t.Fatalf("fail to enumerate network interface, %s", err) } if !ipv6 { <del> t.Fatalf("no ipv6 loopback interface") </del> <ins> t.Skip("no ipv6 loopback interface") </ins> } host, port, err := LoopbackHostPort("[ff06:0:0:0:0:0:0:c3]:443")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: TestLoopbackHostPortIPv6 fails when run in the machine where there is no IPv6 loopback device configured, with following error: Especially running in the containers where IPv6 is not really configured for the runtime What you expected to happen: Unit tests shouldn't really depend on the environment it is running, How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): master
/sig testing /cc
/assign
/remove-sig testing /sig api-machinery EDIT: This is a specific SIG's unit test that is environment dependent, not a systemic test issue or the framework etc.
/cc


Question: } patchedJS, retErr = jsonpatch.MergePatch(versionedJS, p.patchBytes) <ins> if retErr == jsonpatch.ErrBadJSONPatch { return nil, nil, errors.NewBadRequest(retErr.Error()) } </ins> return patchedJS, strictErrors, retErr default: // only here as a safety net - go-restful filters content-type

integration-master ci-kubernetes-integration-master.OverallChanges 11/19/2021 23:54 CET see this as example to get information about the failure No response No response_
great catch /sig api-machinery /priority critical-urgent it seems all tests fail with how can this fail consistently only in periodics and not in presubmits? is an external dependency?
I would say this is the origin Server Side Strict Field Validation
presubmits only run short tests, periodics run all long ones... looking if there's a long test that needs fixup
fix in


Question: http_archive( name = "io_bazel_rules_go", <del> sha256 = "a4ea00b71a6fc3bd381cbbf6eb83ec91fe8b32b1c622c048f1e6f0d965bb1a2d", strip_prefix = "rules_go-a280fbac1a0a4c67b0eee660b4fd1b3db7c9f058", urls = ["https://github.com/bazelbuild/rules_go/archive/a280fbac1a0a4c67b0eee660b4fd1b3db7c9f058.tar.gz"], </del> <ins> sha256 = "441e560e947d8011f064bd7348d86940d6b6131ae7d7c4425a538e8d9f884274", strip_prefix = "rules_go-c72631a220406c4fae276861ee286aaec82c5af2", urls = ["https://github.com/bazelbuild/rules_go/archive/c72631a220406c4fae276861ee286aaec82c5af2.tar.gz"], </ins> ) http_archive(

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature This is a followup on , but for Go 1.9 We should do the same kind of testing now to possibly catch bugs in either Go or Kubernetes when doing this kind of dependency version bump. Ideally we'd have automated tests against Go tip (), but we don't have that at the moment, so we have to do it kind of manually. We should also decide which k8s version we should target. k8s v1.8 or v1.9? What Go versions have been used so far? Kubernetes v1.7.0 - go1.8.3 Kubernetes v1.6.0 - go1.7.5 Kubernetes v1.5.0 - go1.7.4 Kubernetes v1.4.0 - go1.6.3 Kubernetes v1.3.0 - go1.6.2 Kubernetes v1.2.0 - go1.4.2 Seems like the trend has been to stay on the same Go version for two releases and then bump. That'd mean we'd switch to Go1.9 in k8s v1.9 Everyone happy with that? Or is there any reason for us to go start using go1.9 in k8s v1.8? In any case, we should indeed start testing go1.9 out now... cc
Let me know if you find any bugs
FWIW pretty tight to code-freeze I would prefer punting out a cycle fwiw.
I don't think we're that close to code freeze given that we have 5 (of 8) weeks left in this coding cycle But I'm happy to defer this to v1.9 if others think that. I guess the main reason for a bump would be scalability goals, but I'm not sure we have the need to expand above 5000 (and that Go would be the bottleneck in that case) for v1.8
made the Go benchmarks 9% faster with 15% larger binaries. I'm interested to see how that affects our numbers.
mid-stack inlining was not included in Go 1.9. It made the compiler too slow. It's being targeted to Go 1.10 now instead, after some compiler optimization work.
Oh, thanks for the correction!


Question: http_archive( name = "io_bazel_rules_go", <del> sha256 = "a4ea00b71a6fc3bd381cbbf6eb83ec91fe8b32b1c622c048f1e6f0d965bb1a2d", strip_prefix = "rules_go-a280fbac1a0a4c67b0eee660b4fd1b3db7c9f058", urls = ["https://github.com/bazelbuild/rules_go/archive/a280fbac1a0a4c67b0eee660b4fd1b3db7c9f058.tar.gz"], </del> <ins> sha256 = "441e560e947d8011f064bd7348d86940d6b6131ae7d7c4425a538e8d9f884274", strip_prefix = "rules_go-c72631a220406c4fae276861ee286aaec82c5af2", urls = ["https://github.com/bazelbuild/rules_go/archive/c72631a220406c4fae276861ee286aaec82c5af2.tar.gz"], </ins> ) http_archive(

Regardless, we need to run the scalability tests against the RC so we can report issues earlier-- the last upgrade was particularly painful because of an HTTP regression that we caught way too late.
yeah. I filed for that now
yeah, my recollection is that the Go releases usually occur too late in our release cadence and end up getting bumped to the next Kubernetes release. (Also we inevitably find something wrong, though testing RCs would help with that. Though I'm not sure anyone has cycles to test RCs and fix all of the issues that do come up.)
Based on our current time frame, it looks like go1.9 support will have to slip to k8s 1.9. is open to dig into and start things testing, but there's a number of things that have to move.. I'll try to document those this time around to help with future go version bumps. So far: bazel build rules needed to be updated to support go1.9 () bazel version possibly needs to be updated? some of our custom bazel rules broke
When done, we should update this:
/priority important-soon
Updating the dependency is blocked pending bazel 0.6.0 - .
Can we upgrade to 0.6.0 now? Does bazel 0.6.0 have multiarch support btw?
We can't update to depend on bazel 0.6.0+ until we can figure out how to support different versions of the bazelbuild and kubekins-e2e images in PR jobs in prow based on what branch is being tested. cc
Noting that we're working on branching jobs, is testing one option here:
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : New functionality.


Question: http_archive( name = "io_bazel_rules_go", <del> sha256 = "a4ea00b71a6fc3bd381cbbf6eb83ec91fe8b32b1c622c048f1e6f0d965bb1a2d", strip_prefix = "rules_go-a280fbac1a0a4c67b0eee660b4fd1b3db7c9f058", urls = ["https://github.com/bazelbuild/rules_go/archive/a280fbac1a0a4c67b0eee660b4fd1b3db7c9f058.tar.gz"], </del> <ins> sha256 = "441e560e947d8011f064bd7348d86940d6b6131ae7d7c4425a538e8d9f884274", strip_prefix = "rules_go-c72631a220406c4fae276861ee286aaec82c5af2", urls = ["https://github.com/bazelbuild/rules_go/archive/c72631a220406c4fae276861ee286aaec82c5af2.tar.gz"], </ins> ) http_archive(

</details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I approved this for v1.9, this is something we definitely want in order to keep up with Go (and for improved performance)
we're starting to use the forked images now /cc this should be more feasible very soon Edit: test-infra jobs have also been using bazel 0.6 (now 0.6.1) / a recent rules_go to soak things a bit
We've identified a failing test as a result of the WIP PR to upgrade the codebase to go1.9: The admission webhook test fails with certificate errors under go1.9.1, but runs successfully under go1.8.3. cc: who wanted to be alerted to potential bugs
- please see for a possible fix


Question: <ins> /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package features import ( utilfeature "k8s.io/apiserver/pkg/util/feature" "testing" ) func TestKubeFeatures(t *testing.T) { features := utilfeature.DefaultFeatureGate.DeepCopy().GetAll() for i := range features { featureName := string(i) if featureName == "AllAlpha" || featureName == "AllBeta" { continue } if _, ok := defaultKubernetesFeatureGates[i]; !ok { t.Errorf("The feature gate %q is not registered", featureName) } } } </ins>

This avoids drift between Kubernetes components and allows for cluster management tools that pass uniform feature gates to all components. xref:
/assign
Hey I f you stuck anywhere. Please free to involve me in this.


Question: "os/exec" "strings" <del> v1 "k8s.io/api/core/v1" </del> <ins> "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/uuid" utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/kubernetes/pkg/features"

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: , Which test(s) are failing: Since when has it been failing: since this PR introduced them Testgrid link: Note that this is the flakiest test in both dashboards, and failing quite frequently. Reason for failure: unknown Anything else we need to know: cc
/sig node
see: , both getting empty PID and multiple PIDs.
opened revert at
for discoverability for the release team /priority critical-urgent /milestone v1.17


Question: Namespace: api.NamespaceDefault, }, Spec: api.ReplicationControllerSpec{ <del> Replicas: 0, </del> <ins> Replicas: replicas, </ins> Template: &d.Spec.Template, }, }

There's some practical limit on the number of concurrent access tokens that can be used in a Kubernetes cluster before it begins to spend all of its time in the authn stack verifying them. Absent fixes to it was < 500; with the fixes linked there it's still not infinitely scalable-- I still hit (what I believe to be) with ~ 5k unique tokens @ ~700QPS. This request is to this dimension an official name & definition, and list it with others, e.g. here: it it have a useful value (I think it should be 10k) /kind bug /sig scalability
/cc
oh and 4. add tests which verify it is as high as believed.
cc since projected tokens will increase the number of unique tokens
/sig auth
LRUs are not good at scaling and sharding has some major pitfalls as pointed out. I would suggest that we try to drop the LRU cache for successful authentication results in the token cache. The main benefit of using an LRU here is bound cache size. Entries in the token cache have a fast expiration independent of the LRU policy. The records probably take up O(2kb) (guesstimate) of memory. Given that these are successful attempts with presumably valid tokens, I don't think we are getting any benefit from the LRU. If we don't have a constraint on fixed cache size, we can back the token cache with a sync.Map and run a single background goroutine to periodically walk the map and remove expired entries. This would fix the problematic locking and eviction due to nonuniform distribution of tokens over shards. The failure cache is less important and we don't cache failures by default. If we consider this the "slowpath", we can probably leave it as. If that's not good enough, we have a couple options (with various tradeoffs) to improve it.
If off by default is not enough, more reasoning why the failed cache is not super useful: it might help with an accidental DoS but it doesn't prevent anything actively malicious DoS. I think we can focus our improvements on the success cache.
/assign He is already working on adding an explicit test for it.
has taken it over and is actively working on this. Unfortunately, I cannot assign this issue to him as he's not a k8s member yet
It's not closed - all testes must be submitted and appropriate documentation must be
Thanks everyone!!


Question: "description": "EndpointPort represents a Port used by an EndpointSlice", "properties": { "appProtocol": { <del> "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and http://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", </del> <ins> "description": "The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and https://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.", </ins> "type": "string" }, "name": {

We should make the change in PR API reference links to should use HTTPS, not HTTP Visit h and look at the IANA hyperlinks Once this fix is made, a PR to is needed to update the docs (by autogenerating them). This typically would be deferred until the next Kubernetes release. v1.23 not applicable not applicable not applicable not applicable not applicable
/sig docs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I opened this issue so that someone else can open a new PR that is equivalent to .
/retitle Update links to IANA for Service Name and Transport Protocol Port Number Registry
I would like to work on this issue. May I assign it to myself?
Anyone - including you - is welcome to work on this issue! So long as you've you can open a pull request to propose a fix.
Thanks
/assign
I have made the PR. I hope this works. If not then will be happy to do the necessary changes.
Had to close the previous PR because of issues, here is the new one I made.


Question: }, []string{"runtime_handler"}, ) <ins> // RunningPodCount is a gauge that tracks the number of Pods currently running RunningPodCount = metrics.NewGauge( &metrics.GaugeOpts{ Subsystem:      KubeletSubsystem, Name:           "running_pod_count", Help:           "Number of pods currently running", StabilityLevel: metrics.ALPHA, }, ) // RunningContainerCount is a gauge that tracks the number of containers currently running RunningContainerCount = metrics.NewGaugeVec( &metrics.GaugeOpts{ Subsystem:      KubeletSubsystem, Name:           "running_container_count", Help:           "Number of containers currently running", StabilityLevel: metrics.ALPHA, }, []string{"container_state"}, ) </ins> ) var registerMetrics sync.Once

Some kubelet metrics use non-standard prometheus collectors unnecessarily. Specifically, we have which can be simple gauges. Since we cannot migrate non-standard prometheus collectors using the internal metrics stability framework, we will need to convert these metrics into standard prometheus variants. /sig instrumentation /assign
GitHub didn't allow me to assign the following users: solodov. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : Some kubelet metrics use non-standard prometheus collectors unnecessarily. Specifically, we have which can be simple gauges. Since we cannot migrate non-standard prometheus collectors using the internal metrics stability framework, we will need to convert these metrics into standard prometheus variants. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
recommends that we generate those metrics in the GenericPLEG during the relist. We can generate the data directly from the pods call there ().
I would be interested to work on this issue if no one else is working on it currently
Go for it!
/assign


Question: if class.DriverName != ctrl.name { return nil, nil } <del> // Check parameters. claimParameters, classParameters, err := ctrl.getParameters(ctx, claim, class) </del> <ins> // Check parameters. Record event to claim and pod if parameters are invalid. claimParameters, classParameters, err := ctrl.getParameters(ctx, claim, class, true) </ins> if err != nil { <ins> ctrl.eventRecorder.Event(pod, v1.EventTypeWarning, "Failed", fmt.Sprintf("claim %v: %v", claim.Name, err.Error())) </ins> return nil, err } return &ClaimAllocation{

When ResourceClaimParameters or ResourceClassParameters fail validation in resource driver during PodSchedulingContext sync loop, the error is visible only in the events of the PodSchedulingContext. The error should be visible on the object that fails validation as well, ResourceClaim or ResourceClass respectively. Creating ResourceClaim with unsupported parameters' values or unsupported No response <details</details<details</details<details</details<details</details<details</details<details</details
/assign
/sig node
/triage accepted
KEP:


Question: // try set ip families (for missing ip families) // we do it here, since we want this to be visible // even when dryRun == true <del> if err := rs.tryDefaultValidateServiceClusterIPFields(service); err != nil { </del> <ins> if err := rs.tryDefaultValidateServiceClusterIPFields(nil, service); err != nil { </ins> return nil, err }

1) start with a single stack cluster 2) create a PreferDualStack service 3) configure API server with a 2nd service CIDR 4) Confirm the above service is not dual-stack 5) Add a label to the above service 6) Observe that the service becomes dual-stack I think the correct answer is to have a "repair loop" that upgrades (and downgrades!) PreferDualStack Services to match the config. I think this is a GA blocker.
/triage accepted The current thought is: of type with are special new breed of api objects that I don't think we have any similar precedent in the apis, i could be wrong. Their behavior is tied to how the cluster is configured. Thus their behavior must change as the cluster behavior change. a way the user is stating . If it is cluster then i am good with two ClusterIPs. If it is single stack cluster then I am also good with a single ClusterIP. One can assume that if the user was ok was with getting a additional IP when possible then they must be ok with taking it away when it is no longer possible. Based on the above I believe best course of action is create a new repair loop or piggyback on any of the existing (or better: fix it in rest stack during list and get calls). The new logic takes care of matching to the .
reproducer and regression test if somebody takes this, feel free to add it
ack - Scheduled to tackle 3rd week of May. Will definitely include that patch.


Question: JessieDnsutils           = ImageConfig{e2eRegistry, "jessie-dnsutils", "1.0", true} Kitten                   = ImageConfig{e2eRegistry, "kitten", "1.0", true} Liveness                 = ImageConfig{e2eRegistry, "liveness", "1.0", true} <del> LogsGenerator            = ImageConfig{gcRegistry, "logs-generator", "v0.1.0", false} </del> <ins> LogsGenerator            = ImageConfig{e2eRegistry, "logs-generator", "1.0", true} </ins> Mounttest                = ImageConfig{e2eRegistry, "mounttest", "1.0", true} MounttestUser            = ImageConfig{e2eRegistry, "mounttest-user", "1.0", true} Nautilus                 = ImageConfig{e2eRegistry, "nautilus", "1.0", true}

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Failures - https://k8s- Part of pause is not used properly for tests for other arches than x86. As the current pause image is not a manifest image, so we need to use a different image for different archs like for , for etc.. With this, I'm using logs-generator from repo What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig testing
/assign


Question: execAffinityTestForLBServiceWithTransition(f, cs, svc) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. </del> // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { </del> <ins> ginkgo.It("should have session affinity work for LoadBalancer service with ESIPP off [Slow] [LinuxOnly]", func() { </ins> // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws")

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is a followup from filed separately now that we're not tracking this as part of 1.9 release. The tests have been disabled with [DisabledForLargeClusters] tag for now, and this fact will be mentioned in the scalability validation report for v1.9. This issue tracks fixing the test so the tag can be removed and the test ran again in the large cluster. /assign /kind e2e-test-failure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Is this e2e test still broken?
Sorry for the confusion here. Currently this e2e test is disabled on extra large cluster due to some scalability issues with the load balancer related components. At the moment the issues haven't been completely resolved yet.
ok, adding priority so that I know it's not a 1.13 blocker. /priority important-longterm
cc - this is why the relevant tests for are not part of the e2e suite.
This issue is still valid --- some action items on top of my head: Figure out a plan to make service controller properly handle operations in parallel. Check on the status for cloud providers (especially GCP, which the test ran against) on handling large number of nodes as LB backends. /remove-triage unresolved
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: execAffinityTestForLBServiceWithTransition(f, cs, svc) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. </del> // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { </del> <ins> ginkgo.It("should have session affinity work for LoadBalancer service with ESIPP off [Slow] [LinuxOnly]", func() { </ins> // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws")

/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /lifecycle frozen
/reopen Reopening as is being reverted in
Reopened this issue. <detailsIn response to : /reopen Reopening as is being reverted in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
these tests are flaking in xref
cc since these were enabled by
Yeah - though these failures doesn't seem to be scale related, they seem to be test related to me. This basically means nodes without public ips: It's pretty much the same thing as (just for a different test). should be addressing these kinds of errors. [Not a release blocker for sure]
BTW - has been opened directly for this one.
... so since is closed, Can we close this and re-enable for large clusters ? (i.e. remove the Tag in that disables this test) ? I'm not up to speed on the history but 2 related notes I came across digging into related test today... 's a couple of TODOs in the code referencing this specific issue, but your above direct issue is now closed, so we can close this as well right?... And if so, we can remove the TODO on the test... -enabling the test ... not sure wether the timing is good right now (given that is open right now , which sparked my stumbling upon this issue :))
We tried to enable all of them and that caused us a bunch of issues and those had to be reverted: A couple tests were enabled later in: and those seem fine. The remaining (mostly session-affinity ones) had some issue, which so far we didn't have time to debug and I haven't heard a good explanation why those were failing. It's possible that they were improved with , but I doubt those would really cause the failures we've seen in the past. So until this is debugged (or at least understood) I'm reluctant to enabling them.


Question: return fmt.Errorf("timed out waiting for apiserver to be restarted") } <del> func GetApiserverRestartCount(c clientset.Interface) (int32, error) { </del> <ins> func getApiserverRestartCount(c clientset.Interface) (int32, error) { </ins> label := labels.SelectorFromSet(labels.Set(map[string]string{"component": "kube-apiserver"})) listOpts := metav1.ListOptions{LabelSelector: label.String()} pods, err := c.CoreV1().Pods(metav1.NamespaceSystem).List(listOpts)

This test is failing in both gce serial suite: http://k8s- /sig network /priority failing-test /priority critical-urgent /kind bug /status approved-for-milestone cc /assign PTAL
xref
/milestone v1.10
this also fails in http://k8s-
Failing to connect to Services at all. I can't even ping pod IPs:
So I can ping pods same-node and cross-node right up until it kills the apiserver. Rebooting fixes a given node. The apiserver demonstrably comes back - I can query it for info...
- I could use help - have to shift attention soon.
FWIW, I am testing on a Calico-enabled node
I'll take a look at this as well.
Weird, now even after a reboot, this condition is sticky. I have a cluster in this state, if you want to poke.
ugh, now it is resolved.
I just repro'ed without the e2e - have a cluster, proved ping works, killed apiserver ping fails.
Interesting - docker shows the pods not running, but apiserver says they are alive.
Update: docker sees no running pods. API server thinks kubelet is running them. kubelet is getting "" from apiserver, even after kubelet restart.
Update and recap: Bring up a cluster Run some pods Prove that pods are running and services work pkill -9 apiserver Wait. Watch kubelets "lose" pods that were assigned to them Observe that pods are not running and services do not work Observe that apiserver says the pods are running pkill -9 kubelet Observe that it is still broken kubectl delete one pod Observe that kubelet is fixed for that one pod, but not others Q: Why is kubelet not getting a full listing when it restarts?
What exactly does it mean for it to be fixed for a deleted pod?
Kubelet started running the pod that was deleted - it got the update for that one.
The cluster I was testing was a week or 2 old. I built a cluster from HEAD and I can not repro - said the same. Is this test running against head of master?
I also ran the test manually against my new cluster and it passed
FWIW I was also unable to reproduce this yesterday on a cluster from HEAD.


Question: return fmt.Errorf("timed out waiting for apiserver to be restarted") } <del> func GetApiserverRestartCount(c clientset.Interface) (int32, error) { </del> <ins> func getApiserverRestartCount(c clientset.Interface) (int32, error) { </ins> label := labels.SelectorFromSet(labels.Set(map[string]string{"component": "kube-apiserver"})) listOpts := metav1.ListOptions{LabelSelector: label.String()} pods, err := c.CoreV1().Pods(metav1.NamespaceSystem).List(listOpts)

Did some investigations and had some findings for this specific failure: Traced down a bit on the history, the first occurrence was https://k8s- From the , seems suspicious because it modified . Also found reported a relevant symptom: The fix () for that was merged a day ago and may explained why we can't repro from HEAD anymore.
nice, is this issue reproducible without the fix ()?
Sorry should have had confirmed this, bringing up a cluster now.
Quick update: I was able to repro the same failure () from HEAD with reverted. Though this test is now flaky on some other errors: https://k8s- https://k8s- They all seem to failed while communicating with apiserver.
Thanks ! So fixed the issue and I am curious about what introduced the issue. Do you happen to know when did this test start to fail? Just trying to find out what went wrong in between. We have some other possibly related issues so finding out the root cause of this one would be super helpful.
/assign
The first occurrence (for that exact failure) I noticed was https://k8s-, which happened on 2018-02-15 09:49 PST.
You all rock! I see a couple greens on it now, but still mostly red?
The test is now flaky mostly on creating an exec pod after restarting apiserver. Some sample failures: To elaborate a bit more, a snippet of the test procedure is: ssh to master and (sending a SIGTERM) Keep polling on /healthz on apiserver until 200 is returned. Create an exec pod for later steps (and failed on various erros --- EOF, connection refused, apiserver terminating). According to above test failures, my suspicion is that the test is assuming the apiserver is restarted too soon. It is possible that apiserver captured SIGTERM and did a graceful termination, so that polling on /healthz may succeed even in the termination phase. I'm not surprised trying to create a pod via a terminating apiserver will hit those errors. Could anyone confirme the SIGTERM theory?
Spoke with folks, sounds like after SIGTERM apiserver does wait a bit for remaining requests to finish. I will take a stab to refine the test. cc
Sent .
Test is now passing on gce but failing on gke.


Question: return fmt.Errorf("timed out waiting for apiserver to be restarted") } <del> func GetApiserverRestartCount(c clientset.Interface) (int32, error) { </del> <ins> func getApiserverRestartCount(c clientset.Interface) (int32, error) { </ins> label := labels.SelectorFromSet(labels.Set(map[string]string{"component": "kube-apiserver"})) listOpts := metav1.ListOptions{LabelSelector: label.String()} pods, err := c.CoreV1().Pods(metav1.NamespaceSystem).List(listOpts)

Apparently we don't have a way to trigger apiserver restart on gke at this moment. Sending a PR to skip this test on gke.
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Commented too quick. On gke we do a same-version master upgrade. Will send another fix :/


Question: "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/component-base/config:go_default_library", "//staging/src/k8s.io/component-base/metrics/legacyregistry:go_default_library", <ins> "//staging/src/k8s.io/component-base/metrics/prometheus/workqueue:go_default_library", </ins> "//vendor/k8s.io/klog:go_default_library", ], )

<!-- Please only use this template for submitting enhancement requests --What would you like to be : kube-controller-manager currently does not expose work queue metrics Why is this needed: queuing delay and queue length are important information to debug control plane performance in large scale cluster /assign zhan849 /sig apps


Question: <del> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package main import ( "io" "io/ioutil" "log" "os" "os/signal" "path/filepath" "github.com/pkg/errors" ) func main() { env := envWithDefaults(map[string]string{ resultsDirEnvKey: defaultResultsDir, skipEnvKey:       defaultSkip, focusEnvKey:      defaultFocus, providerEnvKey:   defaultProvider, parallelEnvKey:   defaultParallel, ginkgoEnvKey:     defaultGinkgoBinary, testBinEnvKey:    defaultTestBinary, }) if err := configureAndRunWithEnv(env); err != nil { log.Fatal(err) } } // configureAndRunWithEnv uses the given environment to configure and then start the test run. // It will handle TERM signals gracefully and kill the test process and will // save the logs/results to the location specified via the RESULTS_DIR environment // variable. func configureAndRunWithEnv(env Getenver) error { // Ensure we save results regardless of other errors. This helps any // consumer who may be polling for the results. resultsDir := env.Getenv(resultsDirEnvKey) defer saveResults(resultsDir) // Print the output to stdout and a logfile which will be returned // as part of the results tarball. logFilePath := filepath.Join(resultsDir, logFileName) logFile, err := os.Create(logFilePath) if err != nil { return errors.Wrapf(err, "failed to create log file %v", logFilePath) } mw := io.MultiWriter(os.Stdout, logFile) cmd := getCmd(env, mw) log.Printf("Running command:n%vn", cmdInfo(cmd)) err = cmd.Start() if err != nil { return errors.Wrap(err, "starting command") } // Handle signals and shutdown process gracefully. go setupSigHandler(cmd.Process.Pid) return errors.Wrap(cmd.Wait(), "running command") } // setupSigHandler will kill the process identified by the given PID if it // gets a TERM signal. func setupSigHandler(pid int) { c := make(chan os.Signal, 1) signal.Notify(c, os.Interrupt) // Block until a signal is received. log.Println("Now listening for interrupts") s := <-c log.Printf("Got signal: %v. Shutting down test process (PID: %v)n", s, pid) p, err := os.FindProcess(pid) if err != nil { log.Printf("Could not find process %v to shut down.n", pid) return } if err := p.Signal(s); err != nil { log.Printf("Failed to signal test process to terminate: %vn", err) return } log.Printf("Signalled process %v to terminate successfully.n", pid) } // saveResults will tar the results directory and write the resulting tarball path // into the donefile. func saveResults(resultsDir string) error { log.Printf("Saving results at %vn", resultsDir) err := tarDir(resultsDir, filepath.Join(resultsDir, resultsTarballName)) if err != nil { return errors.Wrapf(err, "tar directory %v", resultsDir) } doneFile := filepath.Join(resultsDir, doneFileName) resultsTarball := filepath.Join(resultsDir, resultsTarballName) resultsTarball, err = filepath.Abs(resultsTarball) if err != nil { return errors.Wrapf(err, "failed to find absolute path for %v", resultsTarball) } return errors.Wrap( ioutil.WriteFile(doneFile, []byte(resultsTarball), os.FileMode(0777)), "writing donefile", ) } </del>

<!-- Please only use this template for submitting enhancement requests --What would you like to be modified: image to be based on anything other than Docker Hub sourced Why is this needed: With Docker Hub rate-limit policy in-place, we probably should consider removing this dependency to stay in sync with other base images in Kubernetes project. Folks building their own binaries (i.e. with ) might run into this as well. Notes: Prior discussion happened on An idea from is to use the should probably be renamed to avoid confusion with k/release (h/t Currently, is set to , so migrating to may not be a drop-in move However, , so maybe we don't have to start from scratch /sig testing /area conformance /sig release (cc'ing because builds conformance image, though my quick search seem to show there isn't any build pipelines referencing )
/assign
do we still want to do this? I read somewhere that the images built in Google Cloud are being mirrored from Docker Hub, right? This would mean that the API rate limiting will not hit us any more.
Womp yeah I think DockerHub rate limit is not really the big motivator at this point. However, I think I still want to consider using distroless / go-runner to build and produce conformance image to reduce the noise from CVE scanning done in downstream places. What do you think?
Yeah I think this seems a valuable addition. Are you planning to work on this issue? :)
I will, though probably won't start until next week. If you have ideas and would like to start early feel free to go for it! Otherwise, I'd be happy to work on and deliver this.
Thank you Wilson! Feel free to ping again if you need any assistance. /assign
/triage accepted
one thing I need a little help with is to find a reference for using go-runner image that's used everywhere -- the one that I've found is inside conformance directory which seems to be different from what everyone else is using
We maintain a go-runner in k/release:
oops, I forgot to get back on this last week -- I meant to say that I need a reference for image which I can use as , so something like on Dockerfile. Is that a possible thing to do?


Question: <del> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package main import ( "io" "io/ioutil" "log" "os" "os/signal" "path/filepath" "github.com/pkg/errors" ) func main() { env := envWithDefaults(map[string]string{ resultsDirEnvKey: defaultResultsDir, skipEnvKey:       defaultSkip, focusEnvKey:      defaultFocus, providerEnvKey:   defaultProvider, parallelEnvKey:   defaultParallel, ginkgoEnvKey:     defaultGinkgoBinary, testBinEnvKey:    defaultTestBinary, }) if err := configureAndRunWithEnv(env); err != nil { log.Fatal(err) } } // configureAndRunWithEnv uses the given environment to configure and then start the test run. // It will handle TERM signals gracefully and kill the test process and will // save the logs/results to the location specified via the RESULTS_DIR environment // variable. func configureAndRunWithEnv(env Getenver) error { // Ensure we save results regardless of other errors. This helps any // consumer who may be polling for the results. resultsDir := env.Getenv(resultsDirEnvKey) defer saveResults(resultsDir) // Print the output to stdout and a logfile which will be returned // as part of the results tarball. logFilePath := filepath.Join(resultsDir, logFileName) logFile, err := os.Create(logFilePath) if err != nil { return errors.Wrapf(err, "failed to create log file %v", logFilePath) } mw := io.MultiWriter(os.Stdout, logFile) cmd := getCmd(env, mw) log.Printf("Running command:n%vn", cmdInfo(cmd)) err = cmd.Start() if err != nil { return errors.Wrap(err, "starting command") } // Handle signals and shutdown process gracefully. go setupSigHandler(cmd.Process.Pid) return errors.Wrap(cmd.Wait(), "running command") } // setupSigHandler will kill the process identified by the given PID if it // gets a TERM signal. func setupSigHandler(pid int) { c := make(chan os.Signal, 1) signal.Notify(c, os.Interrupt) // Block until a signal is received. log.Println("Now listening for interrupts") s := <-c log.Printf("Got signal: %v. Shutting down test process (PID: %v)n", s, pid) p, err := os.FindProcess(pid) if err != nil { log.Printf("Could not find process %v to shut down.n", pid) return } if err := p.Signal(s); err != nil { log.Printf("Failed to signal test process to terminate: %vn", err) return } log.Printf("Signalled process %v to terminate successfully.n", pid) } // saveResults will tar the results directory and write the resulting tarball path // into the donefile. func saveResults(resultsDir string) error { log.Printf("Saving results at %vn", resultsDir) err := tarDir(resultsDir, filepath.Join(resultsDir, resultsTarballName)) if err != nil { return errors.Wrapf(err, "tar directory %v", resultsDir) } doneFile := filepath.Join(resultsDir, doneFileName) resultsTarball := filepath.Join(resultsDir, resultsTarballName) resultsTarball, err = filepath.Abs(resultsTarball) if err != nil { return errors.Wrapf(err, "failed to find absolute path for %v", resultsTarball) } return errors.Wrap( ioutil.WriteFile(doneFile, []byte(resultsTarball), os.FileMode(0777)), "writing donefile", ) } </del>

after searching around, I found this -- going to try and use and see how it works :smile:
Started a PR -- plenty of things I'd like to be discussed but we should probably have the discussions there
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale there is work in progress


Question: const connectionTimeout = 1 * time.Second <del> type connection interface { serverReachable(address string) bool parseServerList(serverList []string) error CheckEtcdServers() (bool, error) } </del> // EtcdConnection holds the Etcd server list type EtcdConnection struct { ServerList []string

/kind bug What happened: Every time a health check is called on the APIServer via the endpoint, an etcd healthcheck is performed. However, as this just starts at the first server in the list passed to the apiserver binary all the healthchecks generate read load on a single etcd member. What you expected to happen: The first server that is checked should either increment (and wrap around) each time or be randomised to spread load around the cluster
/assign
I think a random check is ok.
/sig api-machinery


Question: } missingIPs := desiredIPs.Difference(currentIPs) if len(missingIPs) > 0 { <del> glog.V(2).Infof("Current certificate is missing requested IP addresses %v, rotating now", missingIPs.List()) return time.Now() </del> <ins> glog.V(2).Infof("Current certificate is missing requested IP addresses %v", missingIPs.List()) return false </ins> } <ins> currentOrgs := sets.NewString(m.cert.Leaf.Subject.Organization...) desiredOrgs := sets.NewString(template.Subject.Organization...) missingOrgs := desiredOrgs.Difference(currentOrgs) if len(missingOrgs) > 0 { glog.V(2).Infof("Current certificate is missing requested orgs %v", missingOrgs.List()) return false } } return true } func (m *manager) certSatisfiesTemplate() bool { m.certAccessLock.RLock() defer m.certAccessLock.RUnlock() return m.certSatisfiesTemplateLocked() } // nextRotationDeadline returns a value for the threshold at which the // current certificate should be rotated, 80%+/-10% of the expiration of the // certificate. func (m *manager) nextRotationDeadline() time.Time { // forceRotation is not protected by locks if m.forceRotation { m.forceRotation = false return time.Now() } m.certAccessLock.RLock() defer m.certAccessLock.RUnlock() if !m.certSatisfiesTemplateLocked() { return time.Now() </ins> } notAfter := m.cert.Leaf.NotAfter

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I'm testing the new option in kubernetes 1.12 and it seems that kubelet always requests the signing of a new CSR and tries to rotate the certificates when it is started. This is happening even though there's a valid (brand new) certificate on disk issued by kube-controller-manager. From what I can tell, this is happening because kubelet implements and is set after a successful certificate rotation, but is never persisted to disk. If kubelet is restarted, then the last requested CSR is nil. What you expected to happen: Ideally, kubelet should only request the signing of a new CSR if it doesn't already have a serving certificate or it's about to expire. How to reproduce it (as minimally and precisely as possible): Restart a kubelet instance that had it's serving certificate issued by kube-controller-manager Check and see that it requested the signing of a new CSR Anything else we need to know?: kubelet is being run with . , , and are all omitted to use the default values. The TLS bootstrapping for client certificates is currently not being used--just the bootstrapping for serving certificates. Environment: Kubernetes version: v1.12.0 - built from source at the tag () Cloud provider or hardware configuration: Bare metal OS: CentOS 7.5.1804 Kernel: Install tools: Custom RPMs built from this git repo with no additional patches Others:
/sig node
/sig auth
Not sure if this is the ideal fix, but I've patched it so that it compares the current CSR template to the last requested template only if it doesn't already match the current cert.
cc
thanks for the proposed fix. I think it looks good. Could you send a PR? We should also remove entirely and only look at current cert and template. does that sound reasonable?
Sure thing, I'd be happy to do that. EDIT: I will submit a PR after approval from my employer.
I think that's probably ok, as long as it checks everything we include in the template and we consider a certificate missing anything we request to be cause for re-requesting another cert (an issuer couldn't choose to issue a cert that drops some SANs and us not request another cert) left some comments in
This regression is a frequent flyer.


Question: } missingIPs := desiredIPs.Difference(currentIPs) if len(missingIPs) > 0 { <del> glog.V(2).Infof("Current certificate is missing requested IP addresses %v, rotating now", missingIPs.List()) return time.Now() </del> <ins> glog.V(2).Infof("Current certificate is missing requested IP addresses %v", missingIPs.List()) return false </ins> } <ins> currentOrgs := sets.NewString(m.cert.Leaf.Subject.Organization...) desiredOrgs := sets.NewString(template.Subject.Organization...) missingOrgs := desiredOrgs.Difference(currentOrgs) if len(missingOrgs) > 0 { glog.V(2).Infof("Current certificate is missing requested orgs %v", missingOrgs.List()) return false } } return true } func (m *manager) certSatisfiesTemplate() bool { m.certAccessLock.RLock() defer m.certAccessLock.RUnlock() return m.certSatisfiesTemplateLocked() } // nextRotationDeadline returns a value for the threshold at which the // current certificate should be rotated, 80%+/-10% of the expiration of the // certificate. func (m *manager) nextRotationDeadline() time.Time { // forceRotation is not protected by locks if m.forceRotation { m.forceRotation = false return time.Now() } m.certAccessLock.RLock() defer m.certAccessLock.RUnlock() if !m.certSatisfiesTemplateLocked() { return time.Now() </ins> } notAfter := m.cert.Leaf.NotAfter

I'd love to see an integration test that would catch this (but that shouldn't block a fix).
Thanks for the comments! Now that I have the CLA signed, I'll address those and submit a PR.
Just submitted the PR:


Question: if longRunningRequestCheck(req) { return nil, "" } <del> return time.After(time.Minute), "" </del> <ins> return time.After(globalTimeout), "" </ins> } if secureLocation != "" {

this job failed with It seems it is imssing some dependency The job history confirms it is totally broken Originally posted by in
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing


Question: return volumeToMount.GenerateError("MapVolume failed", fmt.Errorf("Device path of the volume is empty")) } <ins> // When kubelet is containerized, devicePath may be a symlink at a place unavailable to // kubelet, so evaluate it on the host and expect that it links to a device in /dev, // which will be available to containerized kubelet. If still it does not exist, // AttachFileDevice will fail. If kubelet is not containerized, eval it anyway. mounter := og.GetVolumePluginMgr().Host.GetMounter(blockVolumePlugin.GetPluginName()) devicePath, err = mounter.EvalHostSymlinks(devicePath) if err != nil { return volumeToMount.GenerateError("MapVolume.EvalHostSymlinks failed", err) } </ins> // Map device to global and pod device map path volumeMapPath, volName := blockVolumeMapper.GetPodDeviceMapPath() mapErr = blockVolumeMapper.MapDevice(devicePath, globalMapPath, volumeMapPath, volName, volumeToMount.Pod.UID)

/kind bug Using , if a block device is symlinked to the discovery directory e.g. , then MapVolume will fail with since the containerized kubelet cannot see the path on host. is mounted by containerized kubelet, so if before we try to map the device we set devicePath to the evaluation of symlink and end up with a devicePath of , containerized kubelet will see the device and successfully map the device. Are there any issues with this solution? This is on Kubernetes 1.10
/sig storage
ah yes containerized kubelet has many issues when dealing with symlinks. Check out Mounter.GetMode(), it calls a nsenter EvalSymlinks which should be able to do what you need. But actually, one thing to consider, is do you need to do the losetup on the host, or does inside the container work?
Yes it appears to work from inside the container, on the host I run which disappears after I delete the pod using the device. My understanding of running losetup in MapVolume is not that great but I suppose that as long as that /dev/loop0 is created we are fine. Thanks for the quick response, I'll create a PR using that function and tag you
Does it create it on the host or in the containerized kubelet? What happens when containerized kubelet is restarted? The purpose of losetup is to do "refcounting" on the block device to prevent possible naming collisions when devices are attached/detached under unhealthy conditions. Ie, if a device showed up as /dev/sdb and you gave that to a pod, and then for some reason it become unhealthy and disappeared (while the pod is still running), another device (maybe even another volume type) could get attached to the node and reuse /dev/sdb. Now the pod has access to a device it's not supposed to. With filesystem volumes, the refcounting is done with mount points, but for block device, loopback is the equivalent.
It is created on the host (and also visible in the container), and stays around through restart. I understand now, thank you.
Actually after I restart it becomes /vdb instead of /dev/vdb in the losetup output, idk what that is about.


Question: return volumeToMount.GenerateError("MapVolume failed", fmt.Errorf("Device path of the volume is empty")) } <ins> // When kubelet is containerized, devicePath may be a symlink at a place unavailable to // kubelet, so evaluate it on the host and expect that it links to a device in /dev, // which will be available to containerized kubelet. If still it does not exist, // AttachFileDevice will fail. If kubelet is not containerized, eval it anyway. mounter := og.GetVolumePluginMgr().Host.GetMounter(blockVolumePlugin.GetPluginName()) devicePath, err = mounter.EvalHostSymlinks(devicePath) if err != nil { return volumeToMount.GenerateError("MapVolume.EvalHostSymlinks failed", err) } </ins> // Map device to global and pod device map path volumeMapPath, volName := blockVolumeMapper.GetPodDeviceMapPath() mapErr = blockVolumeMapper.MapDevice(devicePath, globalMapPath, volumeMapPath, volName, volumeToMount.Pod.UID)

There is no such thing as /vdb and /dev/vdb is still intact of course. Seems inconsequential since it is disappearing after all pods using the device are deleted, i.e. it's still serving as a refcounter


Question: // singleCallTimeout is how long to try single API calls (like 'get' or 'list'). Used to prevent // transient failures from failing tests. singleCallTimeout = 5 * time.Minute <ins> // sshBastionEnvKey is the environment variable key for running SSH commands via bastion. sshBastionEnvKey = "KUBE_SSH_BASTION" </ins> ) // GetSigner returns an ssh.Signer for the provider ("gce", etc.) that can be

ci-kubernetes-e2e-gce-scale-correctness Kubernetes e2e suite: [sig-node] SSH should SSH to all nodes and run commands kubetest.Test Jul 10 14:01:51 UTC+0200 Kubernetes e2e suite: [sig-node] SSH should SSH to all nodes and run commands: more . Test /sig node
/cc
/triage accepted
Uhhh, this may help here
/milestone v1.22
/sig scalability FYI
I got annoyed that I couldn't click through to see the other logs, so now you can, ref: (spyglass doesn't render build- because it's 500MB, and the limit for rendering is 100MB)
/assign SIG Scalability chairs/TLs, please find someone to address or call this a non-release-blocking failure
Assigning to current scalability oncall (): /assign
This is sig-node issue - should be addres by SIG-node folks. /sig node
is being discussed as a fix for this and I'm not sure it's correct
is it sig scalability's intent that 5k node clusters should support SSH?
The SSH test in question passes in the "default" job: So the fact that this failure is unique to a scalability job means to me that scalability needs to own at least some part of this
I did a little bit of digging and ci-kubernetes-e2e-gce-scale-correctness uses option in kubetest, which according to seems to be setting KUBESSHBASTION env. So I believe this is main difference between our scalability job and "default".
Debugging the test in and providing a fix there. The crictl host PR () is doing something different.


Question: } else if hours < 24*365*2 { return fmt.Sprintf("%dd", hours/24) } else if hours < 24*365*8 { <del> return fmt.Sprintf("%dy%dd", hours/24/365, (hours/24)%365) </del> <ins> dy := int(hours/24) % 365 if dy == 0 { return fmt.Sprintf("%dy", hours/24/365) } return fmt.Sprintf("%dy%dd", hours/24/365, dy) </ins> } return fmt.Sprintf("%dy", int(hours/24/365)) }

What happened: When the duration is exactly 2 years or 3 years, human readable duration is returned as or . But when the duration is exactly 4 years, 5 years, etc, the human readable duration is returned as , , etc. What you expected to happen: The format of an exact year should be consistent. Remove when duration is an exact year: should be should be Check other years... 4, 5, 6, 7... not sure if they also have the problem. How to reproduce it (as minimally and precisely as possible): Because this is time-dependent, it can be difficult to reproduce via the CLI, but you can see the problem illustrated in which shows the current behavior: The and test cases should be changed to want and , and the code should be changed to make that pass. Anything else we need to know?: Note, this is not a problem when the duration is exactly 1 year because the human readable duration is still shown in days. So nothing needs to change to handle the case of exactly 1 year. Environment: Kubernetes version (use ): 1.18 Cloud provider or hardware configuration: n/a OS (e.g: ): n/a Kernel (e.g. ): n/a Install tools: n/a Network plugin and version (if this is a network-related bug): n/a Others: n/a
/sig api-machinery
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Could I pick this up as my first issue?
Absolutely. Just use to assign it to yourself.
Also I just updated the issue description. Check the other exact years 4, 5, 6, 7 because they might also have the problem, I can't remember. I know 8 and beyond don't have the problem because it just stops showing days after 8 years.
/assign bitscuit
might recall a related PR?


Question: } else if hours < 24*365*2 { return fmt.Sprintf("%dd", hours/24) } else if hours < 24*365*8 { <del> return fmt.Sprintf("%dy%dd", hours/24/365, (hours/24)%365) </del> <ins> dy := int(hours/24) % 365 if dy == 0 { return fmt.Sprintf("%dy", hours/24/365) } return fmt.Sprintf("%dy%dd", hours/24/365, dy) </ins> } return fmt.Sprintf("%dy", int(hours/24/365)) }

(also, I have to leave the pedantic note: a year isn't always 365 days, so it's surprising to me to see the abbreviation exists at all)
I can't find it. Maybe I was remembering
Yeah it was probably that one. At first I had the tests and the fix. But it sat for a while. I changed it to just the tests of existing behavior so it could be approved more easily. Then I made this issue for the fix but decided it would probably be a good first time issue for someone instead of me doing it.
so should leap years be considered by the code?
There's no way to know how long a year is without knowing the date, so it's not a good unit--in the abstract, there's no way to know if it's a leap year or not. The go language deliberately omits it IIRC. Henry Li <:
What's the intention of HumanDuration? From what I've seen, it is just an approximation of a timestamp solely for humans to understand, so assuming 365 days per year sounds about right to me. This issue is about consistency though, so it could be solved the other way. Instead of abbreviating when it's an exact year (2y, 3y, etc.), exact years could just all have 0d (2y0d, 3y0d, etc.)
I think it is just used for a ballpark duration. It loses accuracy as the duration increases anyway because it changes the units. I guess my intention for opening the issue was to make it consistent with itself. For example it doesn't say 2d0h or 2h0m, so it should not say 2y0d.
(FTR I agree that Xy0d looks worse than Xy.) Brian Pursley <:
Hi I have made the required changes. Shall I raise the PR if you haven't started working in it ?
already assigned it, so I think it would be good to let him submit the PR, unless he unassigns or does not submit a PR in a reasonable amount of time.
Thanks I was planning on submitting the PR over the weekend
Hello, I am a first timer in open source contribution community. I think I made the required changes for that issue. However, I don't know how assignment rules work.


Question: package jsonpatch import ( <ins> "bytes" </ins> "encoding/json" "fmt" "reflect"

/kind bug What happened: Wrote a that returned a faulty json patch which specified an our of range index (e.G. when there is only one container). This resulted in the apiserver crashing. Stacktrace: What you expected to happen: No crash /sig-api-machinery
/sig api-machinery
This maybe a bug in . I will follow this question.
This bug had been fixed in PR: you can checkout it.
Thanks for your comment but I don't think the linked PR fixed the issue, as it got merged almost a year ago and I am running Kube 1.10.1 which does contain that patch but still panics. Just from quickly looking at the code I the linked PR I believe there are two things wrong: The check if the index is valid only happens when a negative index was given should be
Once the PR of is merged I will update kubernetes.
/kind bug


Question: }, } podWithHook := getPodWithHook("pod-with-poststart-https-hook", imageutils.GetPauseImageName(), lifecycle) <ins> // make sure we spawn the test pod on the same node as the webserver. nodeSelection := e2epod.NodeSelection{} e2epod.SetAffinity(&nodeSelection, targetNode) e2epod.SetNodeSelection(&podWithHook.Spec, nodeSelection) </ins> testPodWithHook(podWithHook) }) /*

release-master-informing: capz-windows-containerd-masterKubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] Kubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] Both the tests have been failing during most of the runs since 20th October Kubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] | 1m6s -- | -- {Oct 31 18:42:02.425: Timed out after 30.001s. Expected <errors.errorString | 0xc0015283b0: { s: "failed to match regexp "GET /echo?msg=prestop" in output "I1031 18:41:06. 8988 ] Started HTTP server on port 9090nI1031 18:41:06. 8988 ] Started UDP server on port 9091n"", } to be nil failed (0xc000f1d000) +0x469 () +0x335} Kubernetes e2e suite: [It] [sig-node] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart https hook properly [MinimumKubeletVersion:1.23] [NodeConformance] | 5m11s -- | -- {Oct 31 19:05:50.205: timed out while waiting for pod container-lifecycle-hook-6892/pod-with-poststart-https-hook to be running and ready failed (PodClient).CreateSync(0xc000eb5920, 0x22?) +0x94 (0xc0045e8c00) +0x73 () +0x325} No response_ /sig windows cc
/triage accepted /cc
These failures happen also with Kubernetes master + containerd master with Flannel CNI. Testgrid link:
I figured out why these new failures appear.


Question: }, } podWithHook := getPodWithHook("pod-with-poststart-https-hook", imageutils.GetPauseImageName(), lifecycle) <ins> // make sure we spawn the test pod on the same node as the webserver. nodeSelection := e2epod.NodeSelection{} e2epod.SetAffinity(&nodeSelection, targetNode) e2epod.SetNodeSelection(&podWithHook.Spec, nodeSelection) </ins> testPodWithHook(podWithHook) }) /*

I've submitted the fix via PR
This issue is also affecting 1.25 tests: If possible, we should cherry-pick to the release-1.25 branch once it's merged on master.


Question: # github.com/opencontainers/go-digest v1.0.0 ## explicit; go 1.13 github.com/opencontainers/go-digest <del> # github.com/opencontainers/runc v1.1.4 </del> <ins> # github.com/opencontainers/runc v1.1.5 </ins> ## explicit; go 1.16 github.com/opencontainers/runc/libcontainer github.com/opencontainers/runc/libcontainer/apparmor

opencontainers/runc has fixed three CVEs: Latest runc is pulled by k/k Check runc version in file No response n/a n/a <details</details<details</details<details</details<details</details
/assign /sig security architecture release
/triage accepted


Question: // WriteStaticPodManifests builds manifest objects based on user provided configuration and then dumps it to disk // where kubelet will pick and schedule them. func WriteStaticPodManifests(cfg *kubeadmapi.MasterConfiguration) error { <ins> volumes := []api.Volume{k8sVolume(cfg)} volumeMounts := []api.VolumeMount{k8sVolumeMount()} if isCertsVolumeMountNeeded() { volumes = append(volumes, certsVolume(cfg)) volumeMounts = append(volumeMounts, certsVolumeMount()) } if isPkiVolumeMountNeeded() { volumes = append(volumes, pkiVolume(cfg)) volumeMounts = append(volumeMounts, pkiVolumeMount()) } </ins> // Prepare static pod specs staticPodSpecs := map[string]api.Pod{ kubeAPIServer: componentPod(api.Container{ Name:          kubeAPIServer, Image:         images.GetCoreImage(images.KubeAPIServerImage, cfg, kubeadmapi.GlobalEnvParams.HyperkubeImage), Command:       getAPIServerCommand(cfg), <del> VolumeMounts:  []api.VolumeMount{certsVolumeMount(), k8sVolumeMount()}, </del> <ins> VolumeMounts:  volumeMounts, </ins> LivenessProbe: componentProbe(8080, "/healthz"), Resources:     componentResources("250m"), <del> }, certsVolume(cfg), k8sVolume(cfg)), </del> <ins> }, volumes...), </ins> kubeControllerManager: componentPod(api.Container{ Name:          kubeControllerManager, Image:         images.GetCoreImage(images.KubeControllerManagerImage, cfg, kubeadmapi.GlobalEnvParams.HyperkubeImage), Command:       getControllerManagerCommand(cfg), <del> VolumeMounts:  []api.VolumeMount{certsVolumeMount(), k8sVolumeMount()}, </del> <ins> VolumeMounts:  volumeMounts, </ins> LivenessProbe: componentProbe(10252, "/healthz"), Resources:     componentResources("200m"), <del> }, certsVolume(cfg), k8sVolume(cfg)), </del> <ins> }, volumes...), </ins> kubeScheduler: componentPod(api.Container{ Name:          kubeScheduler, Image:         images.GetCoreImage(images.KubeSchedulerImage, cfg, kubeadmapi.GlobalEnvParams.HyperkubeImage),

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? controller /etc/ssl/certs kubeadm /etc/ssl/certs Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 5 custom built master I also use a custom built hyperkube image instead of the single app images for apiserver, controller, ... Environment: Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): CentOS 7.2 Kernel (e.g. ): Linux ma-kub8ms0 3.10.0-327.36.3.el7.x8664 SMP Mon Oct 24 16:09:20 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: deployed with Azure RM template and ansible What happened: controller is not able to do https, including calls to the Azure RM API What you expected to happen: https should work as usual How to reproduce it (as minimally and precisely as possible): Use kubeadm to install kubernetes on CentOS exec into the controller and try "curl " Anything else do we need to know: It looks like caused this issue. On CentOS, I see these files on the host OS: I suspect the symlinks to /etc/pki/... being the problem here, as they are not available inside the hyperkube Debian image. I also tried to verify this with It seems to not be based on Debian and thus gives a different error: I see 2 possible solutions: of host mounting /etc/ssl/certs into the controller pod, the ca certificates could be installed into the image.


Question: // WriteStaticPodManifests builds manifest objects based on user provided configuration and then dumps it to disk // where kubelet will pick and schedule them. func WriteStaticPodManifests(cfg *kubeadmapi.MasterConfiguration) error { <ins> volumes := []api.Volume{k8sVolume(cfg)} volumeMounts := []api.VolumeMount{k8sVolumeMount()} if isCertsVolumeMountNeeded() { volumes = append(volumes, certsVolume(cfg)) volumeMounts = append(volumeMounts, certsVolumeMount()) } if isPkiVolumeMountNeeded() { volumes = append(volumes, pkiVolume(cfg)) volumeMounts = append(volumeMounts, pkiVolumeMount()) } </ins> // Prepare static pod specs staticPodSpecs := map[string]api.Pod{ kubeAPIServer: componentPod(api.Container{ Name:          kubeAPIServer, Image:         images.GetCoreImage(images.KubeAPIServerImage, cfg, kubeadmapi.GlobalEnvParams.HyperkubeImage), Command:       getAPIServerCommand(cfg), <del> VolumeMounts:  []api.VolumeMount{certsVolumeMount(), k8sVolumeMount()}, </del> <ins> VolumeMounts:  volumeMounts, </ins> LivenessProbe: componentProbe(8080, "/healthz"), Resources:     componentResources("250m"), <del> }, certsVolume(cfg), k8sVolume(cfg)), </del> <ins> }, volumes...), </ins> kubeControllerManager: componentPod(api.Container{ Name:          kubeControllerManager, Image:         images.GetCoreImage(images.KubeControllerManagerImage, cfg, kubeadmapi.GlobalEnvParams.HyperkubeImage), Command:       getControllerManagerCommand(cfg), <del> VolumeMounts:  []api.VolumeMount{certsVolumeMount(), k8sVolumeMount()}, </del> <ins> VolumeMounts:  volumeMounts, </ins> LivenessProbe: componentProbe(10252, "/healthz"), Resources:     componentResources("200m"), <del> }, certsVolume(cfg), k8sVolume(cfg)), </del> <ins> }, volumes...), </ins> kubeScheduler: componentPod(api.Container{ Name:          kubeScheduler, Image:         images.GetCoreImage(images.KubeSchedulerImage, cfg, kubeadmapi.GlobalEnvParams.HyperkubeImage),

This is already done for hyperkube and would work out of the box when the host mount would be omitted. could try to figure out if /etc/pki should also be host mounted and add this to the controller manifest if required. I'd be happy to provide a fix by myself, I just need to know which solution you would prefer.
Same problem seems to apply to the apiserver. Not sure however if it actually does https calls.
implements solution 2.
This issue was moved to kubernetes/kubeadm


Question: ## Protocols for Collaborative Development <del> Please read [this doc](docs/collab.md) for information on how we're running development for the project. </del> <ins> Please read [this doc](docs/devel/collab.md) for information on how we're running development for the project. </ins> ## Adding dependencies

Right now, we have a mixture of user documentation and developer documentation in the top-level directory. We should make it easier to find a just user documentation. Step 1 would be to move developer documentation into a subdir like docs/development or something like that. User docs would stay in the top-level docs dir.
move to


Question: } configurer = NewConfigurer(recorder, nodeRef, nil, testClusterDNS, testClusterDNSDomain, defaultResolvConf) <ins> configurer.getHostDNSConfig = fakeGetHostDNSConfigCustom </ins> for i, pod := range pods { var err error dnsConfig, err := configurer.GetPodDNS(pod)

When a --resolv-conf (DNS resolver config) is passed to kubelet on a Windows node, the config content is used as part of sandbox initialization (through getHostDNSConfig()). Now with the recent changes (), it seems now to fail with an error like: E0315 10:22:16. 4764 ] "Error syncing pod, skipping" err="failed to "CreatePodSandbox" for "img-pull-(-b40a-459d-9033-)" with CreatePodSandboxError: "Failed to generate sandbox config for pod "img-pull-(-b40a-459d-9033-)": Unexpected resolver config value: "C:etckubernetescni". Expected "" or "Host"."" pod="img-puller-3842/img-pull-" podUID=-b40a-459d-9033- This is showing on all the runs of GCE testgrid (latest): Reading/using the passed resolv config on Windows Run a Windows pool with DNS resolv config being set as part of kubelet params. No response_ 27 alpha.3 1.27 beta.0 master <detailsGCE </details<details</details<details</details<details</details<details</details
/triage accepted /milestone v1.27 /sig node /sig network
/sig windows /cc
/priority criticial-urgent as this is likely a release blocking
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority criticial-urgent Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
Hmm, so you're setting the [1]. This PR introduced the capability to read the DNS config options from the Windows host registries while removing the capability to read from a file, introducing a regression. I've sent a PR addressing this issue: [1]
Is the file you are pointing to a UNIX-style or something else?
Was the file changing the behavior of the node?


Question: } configurer = NewConfigurer(recorder, nodeRef, nil, testClusterDNS, testClusterDNSDomain, defaultResolvConf) <ins> configurer.getHostDNSConfig = fakeGetHostDNSConfigCustom </ins> for i, pod := range pods { var err error dnsConfig, err := configurer.GetPodDNS(pod)

Yeah, what used on GCE/GKE, seems to be a Linux-like conf, eg: nameserver 10.128.0.1 search c.<project_name.internal google.internal ` We haven't tried to remove from kubelet to see the behavior.
This Issue/PR has been marked as release blocker. Please have the fix merged by eod today otherwise the feature has to be reverted. For the Issue/PR owner: please have a revert/feature disable PR open today in case the fix failed to get in. Thank you! See the -- Kubernetes 1.27 release manager
I opened up a revert PR in case this PR does not merge by code freeze.
Thanks for the resolution! both 2019 and 2022 grids seems to be back to green now!
Thanks for confirming! I opened to ensure gets updated too


Question: CONTAINER_RUNTIME=${CONTAINER_RUNTIME:-"docker"} CONTAINER_RUNTIME_ENDPOINT=${CONTAINER_RUNTIME_ENDPOINT:-""} IMAGE_SERVICE_ENDPOINT=${IMAGE_SERVICE_ENDPOINT:-""} <ins> ENABLE_CRI=${ENABLE_CRI:-"true"} </ins> RKT_PATH=${RKT_PATH:-""} RKT_STAGE1_IMAGE=${RKT_STAGE1_IMAGE:-""} CHAOS_CHANCE=${CHAOS_CHANCE:-0.0}

When setting for other container runtime, won't start kubelet correctly because cri is disabled : Is there any reason of disabling cri at ? cc
no, that wasn't supposed to go in. I was testing something and it slipped in accidentally. I'll open a PR to fix.
Ah I see your PR for it. Thanks!


Question: } type gcepdSource struct { <del> diskName string </del> <ins> pvc *v1.PersistentVolumeClaim </ins> } func initGCEPD() volSource {

New tests for subpath consistently fail in ci-kubernetes-e2e-gce-multizone: https://k8s- /kind bug /sig storage
Is this a blocker for 1.10?
I would expect so, yes. /milestone v1.10
I'm on it /assign
The problem is likely due to inline GCE PD volumes not having enough zone information. Considering changing volumes to PVs instead.
fix was in progress in to change to a PV
/unassign
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: rolebindingetcd "k8s.io/kubernetes/pkg/registry/rbac/rolebinding/etcd" rolebindingpolicybased "k8s.io/kubernetes/pkg/registry/rbac/rolebinding/policybased" utilruntime "k8s.io/kubernetes/pkg/util/runtime" <ins> "k8s.io/kubernetes/pkg/util/wait" </ins> "k8s.io/kubernetes/plugin/pkg/auth/authorizer/rbac/bootstrappolicy" )

From etcd3: May need to retry poststarthook a few times to avoid bootstrapping failures like this.