Question: in.Spec.InitContainers = values // Call defaulters explicitly until annotations are removed <del> for i := range in.Spec.InitContainers { c := &in.Spec.InitContainers[i] SetDefaults_Container(c) </del> <ins> tmpPodTemp := &PodTemplate{ Template: PodTemplateSpec{ Spec: PodSpec{ HostNetwork:    in.Spec.HostNetwork, InitContainers: values, }, }, </ins> } <ins> SetObjectDefaults_PodTemplate(tmpPodTemp) in.Spec.InitContainers = tmpPodTemp.Template.Spec.InitContainers </ins> } if err := autoConvert_v1_PodTemplateSpec_To_api_PodTemplateSpec(in, out, s); err != nil {

I use ceph-ansible to install cluster so it may need to install additional daemons like mon before running mgr but the error, in this case, should be different, [Unit] Description=Ceph Manager [Service] EnvironmentFile=-/etc/environment ExecStartPre=-/usr/bin/podman stop ceph-mgr-storage2n1-la ExecStartPre=-/usr/bin/podman rm ceph-mgr-storage2n1-la ExecStart=/usr/bin/podman run --rm --net=host  --memory=16G  --cpus=4  -v /var/lib/ceph:/var/lib/ceph:z  -v /etc/ceph:/etc/ceph:z  -v /var/run/ceph:/var/run/ceph:z  -v /etc/localtime:/etc/localtime:ro  -e CLUSTER=ceph  -e CEPHDAEMON=MGR  -e CONTAINERIMAGE=  {{ cephmgrdockerextraenv }}  --name=ceph-mgr-storage2n1-la  ExecStopPost=-/usr/bin/podman stop ceph-mgr-storage2n1-la Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi- delighted <:
BTW, I projectatomic ppa and used apt-get -y podman to install. My podman version is 1.3.0 Sasha Litvak — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
Corrected file.


Question: in.Spec.InitContainers = values // Call defaulters explicitly until annotations are removed <del> for i := range in.Spec.InitContainers { c := &in.Spec.InitContainers[i] SetDefaults_Container(c) </del> <ins> tmpPodTemp := &PodTemplate{ Template: PodTemplateSpec{ Spec: PodSpec{ HostNetwork:    in.Spec.HostNetwork, InitContainers: values, }, }, </ins> } <ins> SetObjectDefaults_PodTemplate(tmpPodTemp) in.Spec.InitContainers = tmpPodTemp.Template.Spec.InitContainers </ins> } if err := autoConvert_v1_PodTemplateSpec_To_api_PodTemplateSpec(in, out, s); err != nil {

[Unit] Description=Ceph Manager [Service] EnvironmentFile=-/etc/environment ExecStartPre=-/usr/bin/podman stop ceph-mgr-storage2n1-la ExecStartPre=-/usr/bin/podman rm ceph-mgr-storage2n1-la ExecStart=/usr/bin/podman run --rm --net=host  --memory=16G  --cpus=4  -v /var/lib/ceph:/var/lib/ceph:z  -v /etc/ceph:/etc/ceph:z  -v /var/run/ceph:/var/run/ceph:z  -v /etc/localtime:/etc/localtime:ro  -e CLUSTER=ceph  -e CEPHDAEMON=MGR  -e CONTAINERIMAGE=  --name=ceph-mgr-storage2n1-la  ExecStopPost=-/usr/bin/podman stop ceph-mgr-storage2n1-la Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi- Giuseppe Scrivano <:
Would it have an effect system wide? Can I add it to a unit file or unit override instead? Giuseppe Scrivano <:
yes, I think it is better to add it only to the unit file
Why would runc work outside of systemd? And if it needs stderr, giving it Null in settings will redirect stderr to Null I presume? On Tue, May 14, 2019, 9:46 AM Giuseppe Scrivano <:
Running into same issues on Ubuntu 18.04.2 using cri-o-1.13 via PPA.
PR here: opencontainers/runc
Until it is avaialble with project atomic and such, can I use a null workaround? Giuseppe Scrivano <:
yes, but you will lose anything written to stderr. You could probably also use a file, like
What about actually mounting dev and using /dev/stderr ? On Tue, May 14, 2019, 3:11 PM Giuseppe Scrivano <:
is a symlink to . It is present, the issue here is that is a socket. The fix for runc got merged, could you use an updated runc?
I tried apt update/upgrade and restarting crio, but I'm still getting the issue.


Question: in.Spec.InitContainers = values // Call defaulters explicitly until annotations are removed <del> for i := range in.Spec.InitContainers { c := &in.Spec.InitContainers[i] SetDefaults_Container(c) </del> <ins> tmpPodTemp := &PodTemplate{ Template: PodTemplateSpec{ Spec: PodSpec{ HostNetwork:    in.Spec.HostNetwork, InitContainers: values, }, }, </ins> } <ins> SetObjectDefaults_PodTemplate(tmpPodTemp) in.Spec.InitContainers = tmpPodTemp.Template.Spec.InitContainers </ins> } if err := autoConvert_v1_PodTemplateSpec_To_api_PodTemplateSpec(in, out, s); err != nil {

Could it be that crio's runc is not updated yet? Edit Build new runc from and it is working now. Think crio hasn't updated it yet with their repository.
I'm running this version from PPA repo for ubuntu and it fails: could you point to version/commit or whatever helps identify the fixed version/snapshot ? Thanks in advance.
at the moment it is only fixed upstream. You'd need to build runc manually
Is there a version of the package on ppa we can downgrade to? On Wed, May 15, 2019, 4:35 AM Giuseppe Scrivano <:
could we update the runc version in the PPA repo?
Thank you. I seem them built but nut pushed to ppa yet.
should be available now.. let me know..
cri-o-runc (1.0.0-rc6-1~dev~ubuntu18.04~ppa42) bionic; urgency=medium autobuilt -- Lokesh Mandvekar (Bot) <, 15 May 2019 16:54:40 +0000
Thank you, everyone, for a quick turn around. Quick testing showed that fix is working, but more testing is needed. Lokesh Mandvekar <:
I confirm my initial tests with seem to confirm the regression is fixed. Many thanks
I also confirm the fix working for podman. Olivier Berger <:


Question: in.Spec.InitContainers = values // Call defaulters explicitly until annotations are removed <del> for i := range in.Spec.InitContainers { c := &in.Spec.InitContainers[i] SetDefaults_Container(c) </del> <ins> tmpPodTemp := &PodTemplate{ Template: PodTemplateSpec{ Spec: PodSpec{ HostNetwork:    in.Spec.HostNetwork, InitContainers: values, }, }, </ins> } <ins> SetObjectDefaults_PodTemplate(tmpPodTemp) in.Spec.InitContainers = tmpPodTemp.Template.Spec.InitContainers </ins> } if err := autoConvert_v1_PodTemplateSpec_To_api_PodTemplateSpec(in, out, s); err != nil {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): initContainers imagePullPolicy initContainers required value Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: cluster/kube- Others: What happened: When trying to create either a ReplicaSet or Replication Controller whose Pod template contains an Init Container, the validation fails saying is a required value. What you expected to happen: The create should succeed and the Init Container should receive a default . How to reproduce it (as minimally and precisely as possible): Try to create either of the following: ReplicaSet Replication Controller Anything else do we need to know: If I add to the Init Container in the ReplicaSet or Replication Controller Pod template, the create succeeds. When I create an Init Container without in a StatefulSet Pod template, it succeeds. When I create an Init Container without in a bare Pod, it succeeds:
Looks like defaulting not being applied on server? That is probably due to removing defaulting from convert. Surprised we didn't catch this earlier
Some values still not being defaulted
, I'm going to create a PR if none works on it.
We decode from bytes, set defaults and then convert; but for init containers, we delay the decode from "decode phase" to "convert phase" at .


Question: in.Spec.InitContainers = values // Call defaulters explicitly until annotations are removed <del> for i := range in.Spec.InitContainers { c := &in.Spec.InitContainers[i] SetDefaults_Container(c) </del> <ins> tmpPodTemp := &PodTemplate{ Template: PodTemplateSpec{ Spec: PodSpec{ HostNetwork:    in.Spec.HostNetwork, InitContainers: values, }, }, </ins> } <ins> SetObjectDefaults_PodTemplate(tmpPodTemp) in.Spec.InitContainers = tmpPodTemp.Template.Spec.InitContainers </ins> } if err := autoConvert_v1_PodTemplateSpec_To_api_PodTemplateSpec(in, out, s); err != nil {

The defaults has been handled for initContainer at . /cc I'm working to find a solution for that; if any suggestion, please let me know.
Would this bug affect the deletion of jobs with an init-container? I ran into this bug and adjusted my manifest to add the pull policy to the init-container. However, I now cannot delete said job claiming a field is immutable. Example:
, not sure now. Let me check it,
Same error here with version 1.5.1, 1.5.2 The Job "grafana-import-dashboards" is invalid: [0].imagePullPolicy: Required value With this setup: kubectl apply -f monitoring Not working even if I specify a pullpolicy in the yaml. If I roll back to 1.4.7, it is working. Thank you. Greg.
any updates on a PR to address this issue? Is anyone from able to pick this issue up otherwise?
, here's the latest status: there's is a PR to generate for each Object (), it works for Object (Pod, Job) now but List; I'm working to make it generating for List.
Why would you need List to work? No client side logic should be decoding lists except as a generic submission.
The List tests are failed in . There's no for List, so no code to decode init-container annotation, e.g. of should call 's recursively.
Thanks for the update
+1
Any update on this? I still getting this issue with the last version of 1.5.3 on GKE?
ping for the comments? should we merge the PR for 1.5?
, there's a PR () that handled that issue; but we're discussing not to merge it because initContainers had been moved to fields, and fields is the recommended way to use since 1.6. But I'd like to get & 's comments on 1.5.
This is still an issue in 1.5.3 when using annotations: Without the deployment would not be accepted
ping & :).
i dont think this is fixed in 1.5 I just encountered this bug on 1.5.3
, yes, the PR is NOT merged to any branch, so the issue is still here. I'll create a PR for 1.5.
Hi Anyone help this issue.


Question: in.Spec.InitContainers = values // Call defaulters explicitly until annotations are removed <del> for i := range in.Spec.InitContainers { c := &in.Spec.InitContainers[i] SetDefaults_Container(c) </del> <ins> tmpPodTemp := &PodTemplate{ Template: PodTemplateSpec{ Spec: PodSpec{ HostNetwork:    in.Spec.HostNetwork, InitContainers: values, }, }, </ins> } <ins> SetObjectDefaults_PodTemplate(tmpPodTemp) in.Spec.InitContainers = tmpPodTemp.Template.Spec.InitContainers </ins> } if err := autoConvert_v1_PodTemplateSpec_To_api_PodTemplateSpec(in, out, s); err != nil {

#cat mongo- apiVersion: v1 kind: Pod metadata: labels: name: mongo-svc-a name: mongo-deployment spec: replicas: 1 template: metadata: labels: name: mongo spec: containers: image: mongo name: mongo ports: name: mongo containerPort: hostPort: volumeMounts: mountPath: "/data/db" name: mongopv volumes: name: mongopv persistentVolumeClaim: claimName: mongopvc When i want create mongo db container by using this file folowing error is occured. The Pod "mongo-deployment" is invalid:spec.containers: Required value
, it seems there're two here, please remove the first one and have a try.
Hi my kubectl binary is configured at ubuntu 16.04 machine.


Question: in.Spec.InitContainers = values // Call defaulters explicitly until annotations are removed <del> for i := range in.Spec.InitContainers { c := &in.Spec.InitContainers[i] SetDefaults_Container(c) </del> <ins> tmpPodTemp := &PodTemplate{ Template: PodTemplateSpec{ Spec: PodSpec{ HostNetwork:    in.Spec.HostNetwork, InitContainers: values, }, }, </ins> } <ins> SetObjectDefaults_PodTemplate(tmpPodTemp) in.Spec.InitContainers = tmpPodTemp.Template.Spec.InitContainers </ins> } if err := autoConvert_v1_PodTemplateSpec_To_api_PodTemplateSpec(in, out, s); err != nil {

NAME LABELS STATUS 10.1 Ready,SchedulingDisabled 10.2 10.1.10.2 Ready 10.3 Ready apiVersion: v1 kind: Node metadata: creationTimestamp: 2017-03-03T14:41:52Z labels: 10.1.10.1 name: 10.1.10.1 resourceVersion: "" selfLink: /api/v1/nodes/10.1.10.1 uid: -001f-11e7-8e83- spec: externalID: 10.1.10.1 status: addresses: address: 10.1.10.1 type: LegacyHostIP address: 10.1.10.1 type: InternalIP capacity: cpu: "2" memory: 4041352Ki pods: "110" conditions: lastHeartbeatTime: 2017-04-05T18:58:42Z lastTransitionTime: 2017-03-03T14:41:52Z message: kubelet has sufficient disk space available reason: KubeletHasSufficientDisk status: "False" type: OutOfDisk lastHeartbeatTime: 2017-04-05T18:58:42Z lastTransitionTime: 2017-04-05T15:27:52Z message: kubelet is posting ready status reason: KubeletReady status: "True" type: Ready nodeInfo: bootID: -6124-4ac7-ab96- containerRuntimeVersion: docker://1.12.6 kernelVersion: 4.9.16-coreos-r1 kubeProxyVersion: v1.2.4+coreos.1 kubeletVersion: v1.2.4+coreos.1 machineID: osImage: Container Linux by CoreOS 1298.7.0 (Ladybug) systemUUID: C3FD9852-2DFB-21F4-CE07-2597A746D99D When i create pv for creation pod is shows pv is pending. Please give any suggestions.....
Is there solution of this problem with version 2.7. Still facing the same. what exactly expectation of helm upgrade --install relname chartsdirectory is is expected to work on first time installation in case with CI/CD Pipelines.


Question: exit 1 fi <ins> if [ ${#errors_expect_error[@]} -ne 0 ]; then { echo "Errors:" for err in "${errors_expect_error[@]}"; do echo "$err" done echo echo 'The above files need to use framework.ExpectError(err) instead of ' echo 'Expect(err).To(HaveOccurred()) or gomega.Expect(err).To(gomega.HaveOccurred())' echo } >&2 exit 1 fi </ins> echo 'Congratulations!  All e2e test source files are valid.'

What happened: After fixing golint failures under (now still in progress), we need to put for each and . Then common error checking code has become like: We can replace this code with which is more readable and easy to be understood what the code does. Now we have following e2e test directories which can be replaced: [x] test/e2e/apimachinery/ () [x] test/e2e/apps/ () [x] test/e2e/auth/ () [x] test/e2e/common/ () [x] test/e2e/node/ () [x] test/e2e/storage/ () [x] () [x] () [x] () [x] () [x] () [x] () [x] [x] test/e2e/storage/persistentvolumes- [x] () [x] () [x] [x] [x] [x] [x] ()
/sig testing /kind cleanup
/assign PR will be submitted soon.
Fix the following in
Fix in
Fix in
Hi, maybe I could take files beneath test/e2e/storage/vsphere/
oops, I'm finished and prepare for PR.
Thank you That's OK
Fix the following in
/assign
All related PRs were merged. I think this ticket can be closed. BTW, should we also replace to ?
Thanks for your work. thing is managed with We are almost finishing the work also :-)
/remove-kind bug
I found in these files. I'm working on :
I can work to the following item.


Question: } } <del> _, err = client.CoreV1().Pods("default").Create(reinvocationMarkerFixture) if err != nil { t.Fatal(err) } </del> for i, tt := range testCases { t.Run(tt.name, func(t *testing.T) { upCh := recorder.Reset() <del> ns := fmt.Sprintf("reinvoke-%d", i) _, err = client.CoreV1().Namespaces().Create(&v1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: ns}}) </del> <ins> testCaseID := strconv.Itoa(i) ns := "reinvoke-" + testCaseID nsLabels := map[string]string{"test-case": testCaseID} _, err = client.CoreV1().Namespaces().Create(&v1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: ns, Labels: nsLabels}}) if err != nil { t.Fatal(err) } // Write markers to a separate namespace to avoid cross-talk markerNs := ns + "-markers" markerNsLabels := map[string]string{"test-markers": testCaseID} _, err = client.CoreV1().Namespaces().Create(&v1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: markerNs, Labels: markerNsLabels}}) </ins> if err != nil { t.Fatal(err) } <ins> // Create a maker object to use to check for the webhook configurations to be ready. marker, err := client.CoreV1().Pods(markerNs).Create(newReinvocationMarkerFixture(markerNs)) if err != nil { t.Fatal(err) } fail := admissionv1beta1.Fail </ins> webhooks := []admissionv1beta1.MutatingWebhook{} for j, webhook := range tt.webhooks { <del> name := fmt.Sprintf("admission.integration.test.%d.%s", j, strings.TrimPrefix(webhook.path, "/")) fail := admissionv1beta1.Fail </del> endpoint := webhookServer.URL + webhook.path <ins> name := fmt.Sprintf("admission.integration.test.%d.%s", j, strings.TrimPrefix(webhook.path, "/")) </ins> webhooks = append(webhooks, admissionv1beta1.MutatingWebhook{ Name: name, ClientConfig: admissionv1beta1.WebhookClientConfig{

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig api-machinery /assign
/cc
want to take a look at the audit event issue first? I can dig in after that if issues remain.
yes, I'm looking at the audit event issue now
actually the failure message suggests that the missing audit log is caused by unexpected admission invocation: I think it's unrelated to auditing Note that there are two tests named "no reinvocation of webhook B when in-tree or prior webhook mutations" :( and from the audit log (), it also suggests that the test is hitting a wrong endpoint
Thanks I'll dig in.


Question: <ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package service import ( "context" "net" "time" "google.golang.org/grpc" "k8s.io/klog/v2" kmsapi "k8s.io/kms/apis/v2alpha1" ) // GRPCService is a gprc server that runs the kms v2 alpha 1 API. type GRPCService struct { addr    string timeout time.Duration server  *grpc.Server kmsService Service } var _ kmsapi.KeyManagementServiceServer = (*GRPCService)(nil) // NewGRPCService creates an instance of GRPCService. func NewGRPCService( address string, timeout time.Duration, kmsService Service, ) *GRPCService { klog.V(4).InfoS("KMS plugin configured", "address", address, "timeout", timeout) return &GRPCService{ addr:       address, timeout:    timeout, kmsService: kmsService, } } // ListenAndServe accepts incoming connections on a Unix socket. It is a blocking method. // Returns non-nil error unless Close or Shutdown is called. func (s *GRPCService) ListenAndServe() error { ln, err := net.Listen("unix", s.addr) if err != nil { return err } defer ln.Close() gs := grpc.NewServer( grpc.ConnectionTimeout(s.timeout), ) s.server = gs kmsapi.RegisterKeyManagementServiceServer(gs, s) klog.V(4).InfoS("kms plugin serving", "address", s.addr) return gs.Serve(ln) } // Shutdown performs a grafecul shutdown. Doesn't accept new connections and // blocks until all pending  RPCs are finished. func (s *GRPCService) Shutdown() { klog.V(4).InfoS("kms plugin shutdown", "address", s.addr) if s.server != nil { s.server.GracefulStop() } } // Close stops the server by closing all connections immediately and cancels // all active RPCs. func (s *GRPCService) Close() { klog.V(4).InfoS("kms plugin close", "address", s.addr) if s.server != nil { s.server.Stop() } } // Status sends a status request to specified kms service. func (s *GRPCService) Status(ctx context.Context, _ *kmsapi.StatusRequest) (*kmsapi.StatusResponse, error) { res, err := s.kmsService.Status(ctx) if err != nil { return nil, err } return &kmsapi.StatusResponse{ Version: res.Version, Healthz: res.Healthz, KeyId:   res.KeyID, }, nil } // Decrypt sends a decryption request to specified kms service. func (s *GRPCService) Decrypt(ctx context.Context, req *kmsapi.DecryptRequest) (*kmsapi.DecryptResponse, error) { klog.V(4).InfoS("decrypt request received", "id", req.Uid) plaintext, err := s.kmsService.Decrypt(ctx, req.Uid, &DecryptRequest{ Ciphertext:  req.Ciphertext, KeyID:       req.KeyId, Annotations: req.Annotations, }) if err != nil { return nil, err } return &kmsapi.DecryptResponse{ Plaintext: plaintext, }, nil } // Encrypt sends an encryption request to specified kms service. func (s *GRPCService) Encrypt(ctx context.Context, req *kmsapi.EncryptRequest) (*kmsapi.EncryptResponse, error) { klog.V(4).InfoS("encrypt request received", "id", req.Uid) encRes, err := s.kmsService.Encrypt(ctx, req.Uid, req.Plaintext) if err != nil { return nil, err } return &kmsapi.EncryptResponse{ Ciphertext:  encRes.Ciphertext, KeyId:       encRes.KeyID, Annotations: encRes.Annotations, }, nil } </ins>

generation for local KEK based on counter; also rotate once a week even if the counter does not hit 2 million? using AES-GCM encryption of local KEK ref: The implementation would be to the directory.
/sig auth /triage accepted
/assign
GitHub didn't allow me to assign the following users: ibihim. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
thank you !
/milestone v1.26
Hi bug-triage shadow team member for 1.26 here. Looks like this issue is in the 1.26 milestone, but has been moved to 1.27. Is there something else or should this issue go to 1.27 as well? Thank you!
I just forgot to move this issue /milestone v1.27
/reopen is the partial implementation.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
My suggestion on the reference implementation, after reading 's comments is like so... The GRPC server, which implements the expects an implementation that satisfies . The local key logic can be a middleware. So it expects a it can fallback to, if it doesn't know a key or needs to encrypt its own local KEK. So the local implementation would be need to: have a storage of s have a reference to the current in use (if we put it into the cache, it might vanish) have a reference to another , which should be the remote KMS. So the sub-stories could look like: write Middleware write Transformers that can get exhausted (by time, by usage) write 's storage write some dummy for remote KMS? I have some sketches of what I am talking about .


Question: <ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package service import ( "context" "net" "time" "google.golang.org/grpc" "k8s.io/klog/v2" kmsapi "k8s.io/kms/apis/v2alpha1" ) // GRPCService is a gprc server that runs the kms v2 alpha 1 API. type GRPCService struct { addr    string timeout time.Duration server  *grpc.Server kmsService Service } var _ kmsapi.KeyManagementServiceServer = (*GRPCService)(nil) // NewGRPCService creates an instance of GRPCService. func NewGRPCService( address string, timeout time.Duration, kmsService Service, ) *GRPCService { klog.V(4).InfoS("KMS plugin configured", "address", address, "timeout", timeout) return &GRPCService{ addr:       address, timeout:    timeout, kmsService: kmsService, } } // ListenAndServe accepts incoming connections on a Unix socket. It is a blocking method. // Returns non-nil error unless Close or Shutdown is called. func (s *GRPCService) ListenAndServe() error { ln, err := net.Listen("unix", s.addr) if err != nil { return err } defer ln.Close() gs := grpc.NewServer( grpc.ConnectionTimeout(s.timeout), ) s.server = gs kmsapi.RegisterKeyManagementServiceServer(gs, s) klog.V(4).InfoS("kms plugin serving", "address", s.addr) return gs.Serve(ln) } // Shutdown performs a grafecul shutdown. Doesn't accept new connections and // blocks until all pending  RPCs are finished. func (s *GRPCService) Shutdown() { klog.V(4).InfoS("kms plugin shutdown", "address", s.addr) if s.server != nil { s.server.GracefulStop() } } // Close stops the server by closing all connections immediately and cancels // all active RPCs. func (s *GRPCService) Close() { klog.V(4).InfoS("kms plugin close", "address", s.addr) if s.server != nil { s.server.Stop() } } // Status sends a status request to specified kms service. func (s *GRPCService) Status(ctx context.Context, _ *kmsapi.StatusRequest) (*kmsapi.StatusResponse, error) { res, err := s.kmsService.Status(ctx) if err != nil { return nil, err } return &kmsapi.StatusResponse{ Version: res.Version, Healthz: res.Healthz, KeyId:   res.KeyID, }, nil } // Decrypt sends a decryption request to specified kms service. func (s *GRPCService) Decrypt(ctx context.Context, req *kmsapi.DecryptRequest) (*kmsapi.DecryptResponse, error) { klog.V(4).InfoS("decrypt request received", "id", req.Uid) plaintext, err := s.kmsService.Decrypt(ctx, req.Uid, &DecryptRequest{ Ciphertext:  req.Ciphertext, KeyID:       req.KeyId, Annotations: req.Annotations, }) if err != nil { return nil, err } return &kmsapi.DecryptResponse{ Plaintext: plaintext, }, nil } // Encrypt sends an encryption request to specified kms service. func (s *GRPCService) Encrypt(ctx context.Context, req *kmsapi.EncryptRequest) (*kmsapi.EncryptResponse, error) { klog.V(4).InfoS("encrypt request received", "id", req.Uid) encRes, err := s.kmsService.Encrypt(ctx, req.Uid, req.Plaintext) if err != nil { return nil, err } return &kmsapi.EncryptResponse{ Ciphertext:  encRes.Ciphertext, KeyId:       encRes.KeyID, Annotations: encRes.Annotations, }, nil } </ins>

I would implement that in terms of the interface so that it can be used both in testing (via a local key) and in real code (by delegating to a HSM).
Hi bug-triage team member for 1.27 here. This issue is still open but has had no recent activity, and no active PRs. Is everything on track for the Code Freeze in March 14th? Thank you!
/milestone clear
Could be closed, right? :)
We still need metrics and tracing support for the reference implementation (though it is all a lot less important now).
/retitle [KMSv2] Add metrics and tracing in for plugins
This is duplicate of /close
Closing this issue. <detailsIn response to : This is duplicate of Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: set -o pipefail set -o xtrace <ins> retry() { for i in {1..5}; do "$@" && return 0 || sleep $i done "$@" } </ins> # Runs the unit and integration tests, producing JUnit-style XML test # reports in ${WORKSPACE}/artifacts. This script is intended to be run from # kubekins-test container with a kubernetes repo mapped in. See

Not sure if this concided with a Jenkins restart.
Not sure what we can do about this. Retry a few times?
GitHub seemed to hiccup last night and fail a bunch of our builds. I'm not sure a retry would've helped, unless we kept retrying for many minutes.
This is the sort of thing that just happens occasionally. We've seen this a lot in OpenShift CI runs. I'm not sure there's much you can do other than try to avoid external network access (e.g. bake tools into the VM image).
It seems to have occurred twice in the last 12h so retry w/ timeout can't hurt?
It can't hurt... but it may or may not help depending on the extent of the external issue
If the problem is really bad, we could toss it somewhere on GCS?
Looks like it failed on : We should probably put this in a retry loop or else do something more fancy


Question: q := NewPriorityQueue(nil) q.Add(&medPriorityPod) // Add a couple of pods to the unschedulableQ. <del> q.unschedulableQ.addOrUpdate(&unschedulablePod) q.unschedulableQ.addOrUpdate(affinityPod) </del> <ins> addOrUpdateUnschedulablePod(q, &unschedulablePod) addOrUpdateUnschedulablePod(q, affinityPod) </ins> // Simulate addition of an assigned pod. The pod has matching labels for // affinityPod. So, affinityPod should go to activeQ. q.AssignedPodAdded(&labelPod) <del> if q.unschedulableQ.get(affinityPod) != nil { </del> <ins> if getUnschedulablePod(q, affinityPod) != nil { </ins> t.Error("affinityPod is still in the unschedulableQ.") } if _, exists, _ := q.activeQ.Get(affinityPod); !exists { t.Error("affinityPod is not moved to activeQ.") } // Check that the other pod is still in the unschedulableQ. <del> if q.unschedulableQ.get(&unschedulablePod) == nil { </del> <ins> if getUnschedulablePod(q, &unschedulablePod) == nil { </ins> t.Error("unschedulablePod is not in the unschedulableQ.") } }

Which jobs are failing: pull-kubernetes-bazel-test Which test(s) are failing: bazel test //pkg/scheduler/internal/queue:godefaulttest Since when has it been failing: Testgrid link: Reason for failure: WARNING: DATA RACE Read at 0x00c000230db0 by goroutine 47: runtime.mapiterinit() +0x0 (PriorityQueue).flushUnschedulableQLeftover() +0x17a (PriorityQueue).flushUnschedulableQLeftover-fm() +0x41 () +0x61 () +0xce () +0x5a Previous write at 0x00c000230db0 by goroutine 45: runtime.mapassignfaststr() +0x0 (UnschedulablePodsMap).addOrUpdate() +0x30e () +0x601 testing.tRunner() +0x162 Goroutine 47 (running) created at: (PriorityQueue).run() +0x172 () +0xe0b () +0x50 () +0x5a5 testing.tRunner() +0x162 Goroutine 45 (running) created at: testing.(T).Run() +0x659 testing.runTests.func1() +0xa8 testing.tRunner() +0x162 testing.runTests() +0x4ee testing.(M).Run() +0x2ee () bazel-out/k8- +0x2fb Anything else we need to know: I'm very sorry about that. UnschedulablePodsMap is not thread-safe. In , lock will be acquired before operating UnschedulablePodsMap. But in , lock isn‘t be acquired. In the previous case, only a single thread runs so it's ok. But after add goroutine , there will be data race. So I need to acquire lock before operating UnschedulablePodsMap.
/sig testing /sig scheduling


Question: defer ws.Close() buf := &bytes.Buffer{} <del> for { var msg []byte if err := websocket.Message.Receive(ws, &msg); err != nil { if err == io.EOF { break </del> <ins> Eventually(func() error { for { var msg []byte if err := websocket.Message.Receive(ws, &msg); err != nil { if err == io.EOF { break } framework.Failf("Failed to read completely from websocket %s: %v", url.String(), err) </ins> } <del> framework.Failf("Failed to read completely from websocket %s: %v", url.String(), err) </del> <ins> if len(msg) == 0 { continue } if msg[0] != 1 { framework.Failf("Got message from server that didn't start with channel 1 (STDOUT): %v", msg) } buf.Write(msg[1:]) </ins> } <del> if len(msg) == 0 { continue </del> <ins> if buf.Len() == 0 { return fmt.Errorf("Unexpected output from server") </ins> } <del> if msg[0] != 1 { framework.Failf("Got message from server that didn't start with channel 1 (STDOUT): %v", msg) </del> <ins> if !strings.Contains(buf.String(), "nameserver") { return fmt.Errorf("Expected to find 'nameserver' in %q", buf.String()) </ins> } <del> buf.Write(msg[1:]) } if buf.Len() == 0 { framework.Failf("Unexpected output from server") } if !strings.Contains(buf.String(), "nameserver") { framework.Failf("Expected to find 'nameserver' in %q", buf.String()) } </del> <ins> return nil }, time.Minute, 10*time.Second).Should(BeNil()) </ins> }) It("should support retrieving logs from the container over websockets", func() {

https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .


Question: defer ws.Close() buf := &bytes.Buffer{} <del> for { var msg []byte if err := websocket.Message.Receive(ws, &msg); err != nil { if err == io.EOF { break </del> <ins> Eventually(func() error { for { var msg []byte if err := websocket.Message.Receive(ws, &msg); err != nil { if err == io.EOF { break } framework.Failf("Failed to read completely from websocket %s: %v", url.String(), err) </ins> } <del> framework.Failf("Failed to read completely from websocket %s: %v", url.String(), err) </del> <ins> if len(msg) == 0 { continue } if msg[0] != 1 { framework.Failf("Got message from server that didn't start with channel 1 (STDOUT): %v", msg) } buf.Write(msg[1:]) </ins> } <del> if len(msg) == 0 { continue </del> <ins> if buf.Len() == 0 { return fmt.Errorf("Unexpected output from server") </ins> } <del> if msg[0] != 1 { framework.Failf("Got message from server that didn't start with channel 1 (STDOUT): %v", msg) </del> <ins> if !strings.Contains(buf.String(), "nameserver") { return fmt.Errorf("Expected to find 'nameserver' in %q", buf.String()) </ins> } <del> buf.Write(msg[1:]) } if buf.Len() == 0 { framework.Failf("Unexpected output from server") } if !strings.Contains(buf.String(), "nameserver") { framework.Failf("Expected to find 'nameserver' in %q", buf.String()) } </del> <ins> return nil }, time.Minute, 10*time.Second).Should(BeNil()) </ins> }) It("should support retrieving logs from the container over websockets", func() {

https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
The test pod gets deleted by the test upon failure and so there is not enough information about the state of the test pod. I'm changing the test to not delete the pod and instead let the namespace deletion logic in the test framework handle deletions. I'm also increasing the pod lifetime from to a longer duration to avoid any possible race conditions in the test. The test will also retry if the logs output from the test pod does not contain non-space characters.


Question: defer ws.Close() buf := &bytes.Buffer{} <del> for { var msg []byte if err := websocket.Message.Receive(ws, &msg); err != nil { if err == io.EOF { break </del> <ins> Eventually(func() error { for { var msg []byte if err := websocket.Message.Receive(ws, &msg); err != nil { if err == io.EOF { break } framework.Failf("Failed to read completely from websocket %s: %v", url.String(), err) </ins> } <del> framework.Failf("Failed to read completely from websocket %s: %v", url.String(), err) </del> <ins> if len(msg) == 0 { continue } if msg[0] != 1 { framework.Failf("Got message from server that didn't start with channel 1 (STDOUT): %v", msg) } buf.Write(msg[1:]) </ins> } <del> if len(msg) == 0 { continue </del> <ins> if buf.Len() == 0 { return fmt.Errorf("Unexpected output from server") </ins> } <del> if msg[0] != 1 { framework.Failf("Got message from server that didn't start with channel 1 (STDOUT): %v", msg) </del> <ins> if !strings.Contains(buf.String(), "nameserver") { return fmt.Errorf("Expected to find 'nameserver' in %q", buf.String()) </ins> } <del> buf.Write(msg[1:]) } if buf.Len() == 0 { framework.Failf("Unexpected output from server") } if !strings.Contains(buf.String(), "nameserver") { framework.Failf("Expected to find 'nameserver' in %q", buf.String()) } </del> <ins> return nil }, time.Minute, 10*time.Second).Should(BeNil()) </ins> }) It("should support retrieving logs from the container over websockets", func() {

I'm lowering the priority of this test once my changes go in and see if the flake re-appears.
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
I suspect a race between container starting up and the test querying the logs. I'm adding a retry in the test to avoid the race. Lowering the priority since I still don't suspect any kubelet bugs.
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .


Question: FindBoundSatsified:   false, }, eventReason: "FailedScheduling", <del> expectError: makePredicateError("1 VolumeNodeAffinityConflict"), </del> <ins> expectError: makePredicateError("1 node(s) had volume node affinity conflict"), </ins> }, "unbound,no-matches": { volumeBinderConfig: &persistentvolume.FakeVolumeBinderConfig{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature The is not user friendly. There has been to make it better, but we still print in the pending message. We should change these errors to a more human readable message. For example, should be changed to something like . /sig scheduling


Question: ginkgo.By(fmt.Sprintf("SSH'ing to %d nodes and running %s", len(testhosts), testCase.cmd)) for _, host := range testhosts { <ins> ginkgo.By(fmt.Sprintf("SSH'ing host %s", host)) </ins> result, err := e2essh.SSH(testCase.cmd, host, framework.TestContext.Provider) stdout, stderr := strings.TrimSpace(result.Stdout), strings.TrimSpace(result.Stderr) if err != testCase.expectedError {

ci-kubernetes-e2e-gce-scale-correctness Kubernetes e2e suite: [sig-node] SSH should SSH to all nodes and run commands kubetest.Test Jul 10 14:01:51 UTC+0200 Kubernetes e2e suite: [sig-node] SSH should SSH to all nodes and run commands: more . Test /sig node
/cc
/triage accepted
Uhhh, this may help here
/milestone v1.22
/sig scalability FYI
I got annoyed that I couldn't click through to see the other logs, so now you can, ref: (spyglass doesn't render build- because it's 500MB, and the limit for rendering is 100MB)
/assign SIG Scalability chairs/TLs, please find someone to address or call this a non-release-blocking failure
Assigning to current scalability oncall (): /assign
This is sig-node issue - should be addres by SIG-node folks. /sig node
is being discussed as a fix for this and I'm not sure it's correct
is it sig scalability's intent that 5k node clusters should support SSH?
The SSH test in question passes in the "default" job: So the fact that this failure is unique to a scalability job means to me that scalability needs to own at least some part of this
I did a little bit of digging and ci-kubernetes-e2e-gce-scale-correctness uses option in kubetest, which according to seems to be setting KUBESSHBASTION env. So I believe this is main difference between our scalability job and "default".
Debugging the test in and providing a fix there. The crictl host PR () is doing something different.


Question: <del> apiVersion: extensions/v1beta1 </del> <ins> apiVersion: networking.k8s.io/v1beta1 </ins> kind: Ingress metadata: name: pre-shared-cert

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority important-soon /sig network
Ref:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
my fu points me to this commit [] Fix multiple conversion tests that belongs to this PR , that seems related can you confirm if that PR is causing this test to fail? (should we gate on this test?)
Yes, that PR is related to the failure. The test utility is trying to convert extensions/v1beta1 data read from various files directly to a go type. This was never a valid thing to do, and happened to silently succeed because the two structs had an identical structure. This is exactly the type of bug was designed to flush out
the correct fix is to update the test/e2e/testing- files to
/assign
It was my PR that broke this - I can also fix that.
Thanks, o it was the commit that was good but the test that was wrong :upsidedownface: , At least I got right the commit :sweat_smile:
is out for review (talked with offline, he was aware that I'll send that out)


Question: import ( "fmt" <ins> "strings" </ins> "testing" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Below k8s UT fail when run with master-golang. TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON error:=== RUN TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON I1124 05:38:14. ] [cpumanager] initializing new in-memory state store unexpected error while creatng checkpointState: could not restore state from checkpoint: json: unexpected end of JSON input, please drain this node and delete the CPU manager checkpoint file "/tmp/cpumanagerstatetest479660663/cpumanagercheckpointtest" before restarting Kubelet --- FAIL: TestCheckpointStateRestore/RestorecheckpointwithinvalidJSON (0.00s) TestStrategicMergePatchInvalid error:=== RUN TestStrategicMergePatchInvalid expected "invalid character 'b' looking for beginning of value", got "json: invalid character 'b' looking for beginning of value" --- FAIL: TestStrategicMergePatchInvalid (0.00s) TestJSONPatch error:=== RUN TestJSONPatch invalid-syntax: expected error invalid character 'i' looking for beginning of value, but got json: invalid character 'i' looking for beginning of value --- FAIL: TestJSONPatch (0.00s) TestV4ErrorDecoder expandless error:=== RUN TestV4ErrorDecoder wrong error for message "{": want="error stream protocol error: unexpected end of JSON input in "{"", got="error stream protocol error: json: unexpected end of JSON input in "{"" --- FAIL: TestV4ErrorDecoder (0.00s What you expected to happen: The tests should pass with no error. How to reproduce it (as minimally and precisely as possible): a stable version of golang installed master go from upstream git clone cd go/src cd ../.. cp -r ./go /usr/local export PATH=/usr/local/go/bin:$PATH kubernetes repo and run the unit tests. () Anything else we need to know?: These failures are occurring due to mismatch of error messages between upstream go and the k8s tests.


Question: import ( "fmt" <ins> "strings" </ins> "testing" )

There is an addition "json:" prefix to SyntaxError messages after this below commit: Environment: Kubernetes version (use ): latest source code from upstream Cloud provider or hardware configuration: OS (e.g: ): "CentOS Linux 8 (Core)" Kernel (e.g. ): Linux rajalakshmi-workspace 4.18.0-193.19.1.el82.ppc64le SMP Mon Sep 14 14:48:04 UTC 2020 ppc64le ppc64le ppc64le GNU/Linux Install tools: [root go]# go version go version devel + Fri Nov 20 00:54:43 2020 +0000 linux/ppc64le [root go]# Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
will fix it in Go 1.16 as release-blocker.
I would fix the test here to be more tolerant. exact string matches on error messages aren't really an API… I would not expect that to block the golang release
Agree too, we can think about having a substring match to avoid these issues instead!
/assign Lets see if we can fix this in the k8s tests


Question: ginkgo.Context("when gracefully shutting down with Pod priority", func() { const ( <del> pollInterval           = 1 * time.Second podStatusUpdateTimeout = 10 * time.Second </del> <ins> pollInterval                 = 1 * time.Second podStatusUpdateTimeout       = 10 * time.Second priorityClassesCreateTimeout = 10 * time.Second </ins> ) var (

Flaked a couple of times on serial lane with the same error. Test is relatively new. This run only has the single test flake: /sig node /kind flake /cc
/assign
New recent failures: /triage accepted


Question: // for the iSCSI connection. dirs2, err := io.ReadDir(sessionPath) if err != nil { <del> return nil, err </del> <ins> klog.Infof("Failed to process session %s, assuming this session is unavailable: %s", sessionName, err) continue </ins> } for _, dir2 := range dirs2 { // Skip over files that aren't the connection

If one target in multipath volume gets down and a pod that uses the volume is deleted and quickly recreated (e.g. by a Deployment), the new pod never starts. How to reproduce it (as minimally and precisely as possible): Deployment with single replica with one multipath iSCSI volume one target (e.g. switch the machine off). Multipath will mark appropriate path as failed: the deployment pod. will quickly start a new pod for the Deployment. new pod will be (or ) forever, What happened: Kubelet tries to WaitForAttach / SetUp volume for the new pod. The volume is still attached. Kubelet goes through all targets and tries to find out if it's logged in or not. Eventually it tries to read of the failed path and that returns error: What you expected to happen: Kubelet assumes that the path is failed / not logged in and tries to recover it. If it fails it continues with multipath with 2 available devices and 1 failed one. IMO, simple instead of will do its job just fine. /assign /sig storage cc for iSCSI expertise
Fix:


Question: return nil } <ins> // hasServedCRDVersion returns true if the given version is in the list of CRD's versions and the Served flag is set. func hasServedCRDVersion(spec *apiextensions.CustomResourceDefinitionSpec, version string) bool { for _, v := range spec.Versions { if v.Name == version { return v.Served } } return false } </ins>

Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestWebhookConverterWithDefaulting /assign /priority important-soon /area custom-resources /sig api-machinery /kind flake /milestone v1.16
/cc
several of the webhook converter tests have similar flakes. it looks like the wait for storage version to become effective times out occasionally Rare flakes seen in all of these tests: TestWebhookConverterWithWatchCache/nontrivial-converter TestWebhookConverterWithWatchCache/metadata-mutating-converter TestWebhookConverterWithDefaulting/metadata-mutating-converter TestWebhookConverterWithWatchCache/noop-converter TestWebhookConverterWithDefaulting/empty-response TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-2 TestWebhookConverterWithoutWatchCache/noop-converter/check-0/v1beta2 TestWebhookConverterWithoutWatchCache/nontrivial-converter/check-0/v1beta2
cc
Yikes, I’ll sort it out.
iiuc sending an empty patch gets short-circuited and doesn't bump CR generation, even if the storage version gets changed, which means we do not re-write to etcd and go through the encoding path this would be racing and doing noop's if the first pass of didn't get the expected storage version I will send a fix to make the patch do actual mutation
It doesn't bump generation, but the bytes to persist in etcd would still be different if the storage version had changed, so they should persist. You can verify that by changing the storage version on a CRD yourself, do an empty patch, and see if the resourceVersion changes.
you're right. Generation is for non-metadata change only, and the RV changes. I queried etcd locally and the apiVersion did change for an empty patch (sidetracking: another observation is after changing the storage version (say v1 -v2; noop convertor), empty-patching v2 endpoint would bump generation, while empty-patching v1 endpoint wouldn't)
another theory: our cached storage strategy could be stale if the CRD informer merges multiple events together (e.g. the informer does a re-list).


Question: return nil } <ins> // hasServedCRDVersion returns true if the given version is in the list of CRD's versions and the Served flag is set. func hasServedCRDVersion(spec *apiextensions.CustomResourceDefinitionSpec, version string) bool { for _, v := range spec.Versions { if v.Name == version { return v.Served } } return false } </ins>

Two scenarios: -create-update events merged into an update event: strategy with UID:a was created on demand for CRD Foo Foo was deleted and recreated strategy with UID:b was created on demand for the new Foo new Foo got updated (e.g. switching storage version) the strategy with UID:b is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the delete-create-update events into , the strategy with UID:a will be deleted, but the strategy with UID:b won't. -update events merged into a create event: strategy with UID:a was created on demand for CRD Foo Foo was updated (e.g. switching storage version) the strategy with UID:a is supposed to be deleted and , to reflect the last update. But if the CRD informer merges the create-update events into , the strategy with UID:a won't be deleted (as we don't react on create events). another race found in , which could happen in a single update.
I think I :


Question: "time" utilnet "k8s.io/apimachinery/pkg/util/net" <ins> utilrand "k8s.io/apimachinery/pkg/util/rand" </ins> "k8s.io/client-go/tools/cache"

[x] two weeks soak end date : 18 March 2021 According to this APIsnoop query, there are still some remaining RESOURCENAME endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Note: Community feedback for the e2e test has lead to a number of improvements including extending the test coverage to all outstanding endpoints listed above. As each endpoint is tested a ’watch’ confirms that the result is valid before testing the next endpoint. readCoreV1NamespacedServiceStatus (get /status) patchCoreV1NamespacedServiceStatus (patch /status) replaceCoreV1NamespacedServiceStatus (update /status) patchCoreV1NamespacedService Create a Service with a static label Patch the Service with a new label and updated data Get the Service to ensure it’s patched Upate the Service with a new label and updated data Get the Service to ensure it’s updated Delete Namespaced Service via a Collection with a LabelSelector This query by apisnoop shows that all outstanding endpoints as list at the start of this document have been hit by the e2e test. Note that the results do include other service endpoints that have been addressed in other conformance tests. If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
Can you please review the proposed test and advise if we could go ahead with a PR
/assign
It looks like it's patching metadata and spec but not status?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I've been retesting this mock test and apisnoop does log as been hit. From my testing below it looks like the patchStatus is working as expected or have I missed something?


Question: "time" utilnet "k8s.io/apimachinery/pkg/util/net" <ins> utilrand "k8s.io/apimachinery/pkg/util/rand" </ins> "k8s.io/client-go/tools/cache"

Using the following code before the patch request Gives this response Then after running the patch ServiceStatus Then running The response shows the patched changes.
During the Conformance Meeting on 12th of January 2021, reviewed approach with Clayton, and we will start on the test PR this week!
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
/triage accepted


Question: // MinionList is a list of minions. type MinionList struct { JSONBase `json:",inline" yaml:",inline"` <del> Items    []Minion `json:"minions,omitempty" yaml:"minions,omitempty"` </del> <ins> Items    []Minion `json:"items,omitempty" yaml:"items,omitempty"` </ins> } // Binding is written by a scheduler to cause a pod to be bound to a host.

should have the JSON attribute name "items", not "minions"
Doh! Fix now and break clients?
Define Items and Minions and use Convertors to resolve it?
Sure. I'll make a fix.


Question: } func IsMirrorPod(pod *api.Pod) bool { <del> if value, ok := pod.Annotations[kubetypes.ConfigMirrorAnnotationKey]; !ok { return false } else { return value == kubetypes.MirrorType } </del> <ins> _, ok := pod.Annotations[kubetypes.ConfigMirrorAnnotationKey] return ok } func getHashFromMirrorPod(pod *api.Pod) (string, bool) { hash, ok := pod.Annotations[kubetypes.ConfigMirrorAnnotationKey] return hash, ok } func getPodHash(pod *api.Pod) string { // The annotation exists for all static pods. return pod.Annotations[kubetypes.ConfigHashAnnotationKey] </ins> }

See for more context. Currently, kubelet compares the spec of a mirror pod (created via apiserver) with that of a static pod (config stored as a file) to decide whether the two are considered identical. If they differ, kubelet would delete the mirror pod and re-create it with the correct spec. Due to the fact that apiserver may set default values for some fields in the spec of the mirror pod, the specs could deviate undesirably. To avoid problems like this, we can store the config of a static pod as an annotation in the mirror pod, and compare the static pod with the annotation.
Would a hash of the config be sufficient if we're only going to compare? The whole spec would generally be more useful, but I wonder if that is too much data to put in an annotation.
This seems totally reasonable to me, but I'm still desperately curious why you can't just bypass defaulting when you generate mirror pod from static pod? (i.e. just use the static pod's configuration verbatim)
It's not kubelet who applies the default values here. The admission plugin was only run at the apiserver, and to bypass that, you have to ask apiserver to treat mirror pods differently and potentially at multiple places.
On May 22, 2015 10:38 AM, "Yu-Ju Hong" wrote: Understood, but is it really multiple places? If so then I agree it's a bad idea. Otherwise it might be worth considering as an alternative.
It's untenable for Kubelet to assume to know/circumvent all fields that will be set by other entities: admission control, defaulting, late initialization, auto-something, whatever. It needs to obey the same rules as other clients.
I don't understand why you would ever want to apply any of the things you mentioned to static pods. Static pods should be copied verbatim into the mirror pods, right? That's what I'm proposing.
What if the master is updated before the kubelet, and the master adds a new API field with a default value?
That case should be discussed in
I agree that mirror pod should reflect the true status of the static pod running on the node, and no default values should be set in this case.


Question: } func IsMirrorPod(pod *api.Pod) bool { <del> if value, ok := pod.Annotations[kubetypes.ConfigMirrorAnnotationKey]; !ok { return false } else { return value == kubetypes.MirrorType } </del> <ins> _, ok := pod.Annotations[kubetypes.ConfigMirrorAnnotationKey] return ok } func getHashFromMirrorPod(pod *api.Pod) (string, bool) { hash, ok := pod.Annotations[kubetypes.ConfigMirrorAnnotationKey] return hash, ok } func getPodHash(pod *api.Pod) string { // The annotation exists for all static pods. return pod.Annotations[kubetypes.ConfigHashAnnotationKey] </ins> }

However, I think it'd be harder to implement this way since, as stated above, that there are many things to circumvent in the apiserver itself. It could also be harder to keep this special case from being broken by a yet another new feature/plugin. Keeping the hash of the config in the annotation seems like a good compromise even though it may not reflect the true status 100% accurately. I think this is acceptable. What do you think?
Sure, if you think the annotation approach is easier, I would go with that. I guess in the case Brian mentoined (master adds a new API field with a default value), you would not update the mirror pod, because the static pod definition has not changed?
Yes, a hash would be sufficient.
It shouldn't be necessary to update the mirror pod in the case that a new default field was updated, because that field should be populated automatically by the apiserver on decode from etcd.
We should fix this so we don't have this problem again in 1.2.


Question: tester(unconfined) }) <del> By("Running a CAP_SYS_ADMIN pod", func() { </del> <ins> By("Running a SYS_ADMIN pod", func() { </ins> sysadmin := restrictedPod(f, "sysadmin") sysadmin.Spec.Containers[0].SecurityContext.Capabilities = &v1.Capabilities{ <del> Add: []v1.Capability{"CAP_SYS_ADMIN"}, </del> <ins> Add: []v1.Capability{"SYS_ADMIN"}, </ins> } sysadmin.Spec.Containers[0].SecurityContext.AllowPrivilegeEscalation = nil tester(sysadmin)

The pod security policy capability test is using wrong CAP format with prefix. In our document, we explicitly say: It fails for cri-containerd. I'll find out why it doesn't fail for Docker, maybe docker has handled it. But we should fix the test anyway.
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // General settings. fs.StringVar(&s.ContainerRuntime, "container-runtime", s.ContainerRuntime, "The container runtime to use. Possible values: 'docker', 'remote'.") fs.StringVar(&s.RuntimeCgroups, "runtime-cgroups", s.RuntimeCgroups, "Optional absolute name of cgroups to create and run the runtime in.") <del> _ = fs.Bool("redirect-container-streaming", false, "[REMOVED]") // TODO: Delete in v1.22 fs.MarkDeprecated("redirect-container-streaming", "Container streaming redirection has been removed from the kubelet as of v1.20, and this flag will be removed in v1.22. For more details, see http://git.k8s.io/enhancements/keps/sig-node/20191205-container-streaming-requests.md") </del> // Docker-specific settings. fs.StringVar(&s.DockershimRootDirectory, "experimental-dockershim-root-directory", s.DockershimRootDirectory, "Path to the dockershim root directory.")

[ ] OWNERSALIASES:581: # conformance aliases [x] // More info: ~~ [x] // See [x] fs.MarkDeprecated("redirect-container-streaming", "Container streaming redirection has been removed from the kubelet as of v1.20, and this flag will be removed in v1.22. For more details, see ") [x] // More info: [x] // More info: ~~ [x] [x] [x] CHANGELOG/CHANGELOG-[scalability validation report]: ~~ [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [ ] link: [ ] link: Some needs to be regenerated
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help
/kind documentation
/remove-kind bug
/good-first-issue
/assign I shall fix .
/assign i'm working in
i have create a pr for
i see , you need
/assign fixing
/assign
/assign CHANGELOG/CHANGELOG-
/assign CHANGELOG/CHANGELOG- Edit : Link is already resolved, unassigning
/assign CHANGELOG/CHANGELOG-
/assign ,111
Sorry, already assigned
/assign i will take
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
  is merged. Most KEP links are fixed. You may scan it again. fixed most of them.
/sig architecture
/assign I will work on and
/remove-help


Question: concurrencyLimit:            6, evalDuration:                time.Second * 20, expectedFair:                []bool{true}, <del> expectedFairnessMargin:      []float64{0.15}, </del> <ins> expectedFairnessMargin:      []float64{0.155}, </ins> expectAllRequests:           true, evalInqueueMetrics:          true, evalExecutingMetrics:        true,

pull-kubernetes-unit TestDifferentWidths As far back as triage board history shows No response /sig api-machinery
/cc /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available .


Question: concurrencyLimit:            6, evalDuration:                time.Second * 20, expectedFair:                []bool{true}, <del> expectedFairnessMargin:      []float64{0.15}, </del> <ins> expectedFairnessMargin:      []float64{0.155}, </ins> expectAllRequests:           true, evalInqueueMetrics:          true, evalExecutingMetrics:        true,

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Running into this one a fair bit in my PRs, opened as a possible fix
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen I got this in --
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .


Question: concurrencyLimit:            6, evalDuration:                time.Second * 20, expectedFair:                []bool{true}, <del> expectedFairnessMargin:      []float64{0.15}, </del> <ins> expectedFairnessMargin:      []float64{0.155}, </ins> expectAllRequests:           true, evalInqueueMetrics:          true, evalExecutingMetrics:        true,

/close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen Got this in
Reopened this issue. <detailsIn response to : /reopen Got this in </details
/remove-lifecycle rotten


Question: return hist.GetSampleSum() / float64(hist.GetSampleCount()) } <del> // Clear clears all fields of the wrapped histogram func (hist *Histogram) Clear() { if hist.SampleCount != nil { *hist.SampleCount = 0 } if hist.SampleSum != nil { *hist.SampleSum = 0 } for _, b := range hist.Bucket { if b.CumulativeCount != nil { *b.CumulativeCount = 0 } } } </del> // Validate makes sure the wrapped histogram has all necessary fields set and with valid values. func (hist *Histogram) Validate() error { if hist.SampleCount == nil || hist.GetSampleCount() == 0 {

In the new scheduler performance testing framework, (bool) is provided to benchmark performance of pods created in an "operator": (L11 below) However, the initialized Pods created in L5~L7 seem to be also benchmarked, although it doesn't specify . (please see the reproduce steps below). I think it's b/c once scheduler starts, the associated Prometheus artifacts are also up, and it's a single in-memory datastore. So no matter we start to collect the metric in which step, the prometheus data is already up-to-date. In the above workload config, the final data have benchmarked 1500 pods instead of 1000 pods. I guess it won't work either if running in the pod initialization step or not. Unless we manage a breakpoint at the end of each step, and calculate delta histogram for the step that specifies . Pods created in step without are not benchmarked. Apply the following diff: And then run: You will get an output sth. like: /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Could you please help confirm if it's an issue?
cc
Unfortunately, does not apply for metrics gathered from . As you mentioned. The default gatherer collects all metrics produced by the scheduler. It's the scheduler itself who knows pod specs and would have to decide which pods will get ignored when updating metrics. The question is if computing metrics for a subset of pods processed by the scheduler is a valid use case.
This is probably what we want. Is something like that possible ?
I think being able to absolutely ignore certain pods for metrics is an overkill.
The use case is to ignore the pods created in steps that don't specify , so as to collect the Pod metrics create at particular step(s) (). Otherwise, the semantics of option is quite misleading. Nope, that's not the intention. We don't need to target certain pods, like using label selector or whatever. Just as I mentioned, measure pods created at particular steps.


Question: return hist.GetSampleSum() / float64(hist.GetSampleCount()) } <del> // Clear clears all fields of the wrapped histogram func (hist *Histogram) Clear() { if hist.SampleCount != nil { *hist.SampleCount = 0 } if hist.SampleSum != nil { *hist.SampleSum = 0 } for _, b := range hist.Bucket { if b.CumulativeCount != nil { *b.CumulativeCount = 0 } } } </del> // Validate makes sure the wrapped histogram has all necessary fields set and with valid values. func (hist *Histogram) Validate() error { if hist.SampleCount == nil || hist.GetSampleCount() == 0 {

I would do something like the following: if a step has collectMetrics:false, we wait for them to finish scheduling, and then we take a snapshot of the histogram. Then for the next step that has collectMetrics:true, we take the diff with the latest snapshot to report the results. This might not be perfect, as we don't control if the metrics have finished syncing when we take the snapshot. But I think that's fine. Would doing this be possible? Now, if we have steps that run in parallel, one with collectMetrics:false and another with collectMetrics:true, then this doesn't work. Note that I don't know the details of how the gatherer works.
That adheres to what I have in my mind - we can only manage this on client side. I think it's fine. As we are already fetching the "snapshot" when pods get scheduled when equals true. True. An alternative is to design the case to be like: the step with to employ Pods that are all unschedulable, and then specify a custom metric "schedulere2eschedulingdurationseconds[result="unschedulable"]" to target a particular histogram instead of all histograms (default option) inside a histogramVec. Note this can only be possible when and some subsequent changes get in place.
We might just call Histogram.Clear() method which will clear what has been collected so far at the end of phase: Assuming individual phases have no pods in common and each phase can clear metrics without affecting other phases. Though, as said, it will work only for serial case.
WDYT
I'm afraid doesn't work. That was my initial thought as well, but no luck. I guess it's b/c, theoretically, we cannot reset a metric on the server side - Promoetheus' model doesn't support this. But let me know if I miss anything.
The performance tests assume all the metrics all gathered and accessed in the same binary. As long as that's the case, works.


Question: return hist.GetSampleSum() / float64(hist.GetSampleCount()) } <del> // Clear clears all fields of the wrapped histogram func (hist *Histogram) Clear() { if hist.SampleCount != nil { *hist.SampleCount = 0 } if hist.SampleSum != nil { *hist.SampleSum = 0 } for _, b := range hist.Bucket { if b.CumulativeCount != nil { *b.CumulativeCount = 0 } } } </del> // Validate makes sure the wrapped histogram has all necessary fields set and with valid values. func (hist *Histogram) Validate() error { if hist.SampleCount == nil || hist.GetSampleCount() == 0 {

We already call in the code:
That was my understanding as well, but no luck :( Here is a diff to call in case when "concreteOp.CollectMetrics" is false: <details<summaryDiff:</summary<p</p</detailsAnd below is the output - it's still 1500 (500 init pods plus 1000 measured pods) instead of 1000 being benchmarked: <details<summaryOutput:</summary<p</p</details
Also, if you run the tests and together, the result proves doesn't work as expected:
But I do find a way to fix this: .


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package runtime import ( "testing" "k8s.io/apimachinery/pkg/runtime/schema" ) func gv(group, version string) schema.GroupVersion { return schema.GroupVersion{Group: group, Version: version} } func gvk(group, version, kind string) schema.GroupVersionKind { return schema.GroupVersionKind{Group: group, Version: version, Kind: kind} } func gk(group, kind string) schema.GroupKind { return schema.GroupKind{Group: group, Kind: kind} } func TestCoercingMultiGroupVersioner(t *testing.T) { testcases := []struct { name           string target         schema.GroupVersion preferredKinds []schema.GroupKind kinds          []schema.GroupVersionKind expectKind     schema.GroupVersionKind }{ { name:           "matched preferred group/kind", target:         gv("mygroup", "__internal"), preferredKinds: []schema.GroupKind{gk("mygroup", "Foo"), gk("anothergroup", "Bar")}, kinds:          []schema.GroupVersionKind{gvk("yetanother", "v1", "Baz"), gvk("anothergroup", "v1", "Bar")}, expectKind:     gvk("mygroup", "__internal", "Bar"), }, { name:           "matched preferred group", target:         gv("mygroup", "__internal"), preferredKinds: []schema.GroupKind{gk("mygroup", ""), gk("anothergroup", "")}, kinds:          []schema.GroupVersionKind{gvk("yetanother", "v1", "Baz"), gvk("anothergroup", "v1", "Bar")}, expectKind:     gvk("mygroup", "__internal", "Bar"), }, { name:           "no preferred group/kind match, uses first kind in list", target:         gv("mygroup", "__internal"), preferredKinds: []schema.GroupKind{gk("mygroup", ""), gk("anothergroup", "")}, kinds:          []schema.GroupVersionKind{gvk("yetanother", "v1", "Baz"), gvk("yetanother", "v1", "Bar")}, expectKind:     gvk("mygroup", "__internal", "Baz"), }, } for _, tc := range testcases { t.Run(tc.name, func(t *testing.T) { v := NewCoercingMultiGroupVersioner(tc.target, tc.preferredKinds...) kind, ok := v.KindForGroupVersionKinds(tc.kinds) if !ok { t.Error("got no kind") } if kind != tc.expectKind { t.Errorf("expected %#v, got %#v", tc.expectKind, kind) } }) } } </ins>

What happened: Set up a HA cluster with 1.9 and 1.10 API servers (occurs on any apiserver skew boundary where a cohabitating resource's storage version changed groups) Observed errors when modifying workload objects: when modified via the 1.10 server, saw this error in the 1.9 server: when modified via the 1.9 server, saw this error in the 1.10 server: When this error was encountered, all watchers for the resource in question would be dropped and have to re-establish, and the internal watchers feeding the watch cache would drop and re-establish What you expected to happen: Watch events from all supported storage versions are processed correctly How to reproduce it (as minimally and precisely as possible): Start a server from master (for inspectability, store in json). With Create a deployment (e.g. kube-dns in kube-system) Establish a watch via extensions (kubectl get deployments.extensions --all-namespaces -w) Establish a watch via apps (kubectl get --all-namespaces -w) Tail the kube-apiserver log Change the apps/v1 deployment in etcd to extensions/v1beta1: Observer client watches drop and server log errors: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig api-machinery /cc /priority critical-urgent
error is triggered by this returning false: constructed here: the info about expected cohabitating groups is not included in the list of acceptable inputs


Question: <del> v1.8.3-3 </del> <ins> v1.9.1-1 </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature This is a followup on , but for Go 1.9 We should do the same kind of testing now to possibly catch bugs in either Go or Kubernetes when doing this kind of dependency version bump. Ideally we'd have automated tests against Go tip (), but we don't have that at the moment, so we have to do it kind of manually. We should also decide which k8s version we should target. k8s v1.8 or v1.9? What Go versions have been used so far? Kubernetes v1.7.0 - go1.8.3 Kubernetes v1.6.0 - go1.7.5 Kubernetes v1.5.0 - go1.7.4 Kubernetes v1.4.0 - go1.6.3 Kubernetes v1.3.0 - go1.6.2 Kubernetes v1.2.0 - go1.4.2 Seems like the trend has been to stay on the same Go version for two releases and then bump. That'd mean we'd switch to Go1.9 in k8s v1.9 Everyone happy with that? Or is there any reason for us to go start using go1.9 in k8s v1.8? In any case, we should indeed start testing go1.9 out now... cc
Let me know if you find any bugs
FWIW pretty tight to code-freeze I would prefer punting out a cycle fwiw.
I don't think we're that close to code freeze given that we have 5 (of 8) weeks left in this coding cycle But I'm happy to defer this to v1.9 if others think that. I guess the main reason for a bump would be scalability goals, but I'm not sure we have the need to expand above 5000 (and that Go would be the bottleneck in that case) for v1.8
made the Go benchmarks 9% faster with 15% larger binaries. I'm interested to see how that affects our numbers.
mid-stack inlining was not included in Go 1.9. It made the compiler too slow. It's being targeted to Go 1.10 now instead, after some compiler optimization work.
Oh, thanks for the correction!


Question: <del> v1.8.3-3 </del> <ins> v1.9.1-1 </ins>

Regardless, we need to run the scalability tests against the RC so we can report issues earlier-- the last upgrade was particularly painful because of an HTTP regression that we caught way too late.
yeah. I filed for that now
yeah, my recollection is that the Go releases usually occur too late in our release cadence and end up getting bumped to the next Kubernetes release. (Also we inevitably find something wrong, though testing RCs would help with that. Though I'm not sure anyone has cycles to test RCs and fix all of the issues that do come up.)
Based on our current time frame, it looks like go1.9 support will have to slip to k8s 1.9. is open to dig into and start things testing, but there's a number of things that have to move.. I'll try to document those this time around to help with future go version bumps. So far: bazel build rules needed to be updated to support go1.9 () bazel version possibly needs to be updated? some of our custom bazel rules broke
When done, we should update this:
/priority important-soon
Updating the dependency is blocked pending bazel 0.6.0 - .
Can we upgrade to 0.6.0 now? Does bazel 0.6.0 have multiarch support btw?
We can't update to depend on bazel 0.6.0+ until we can figure out how to support different versions of the bazelbuild and kubekins-e2e images in PR jobs in prow based on what branch is being tested. cc
Noting that we're working on branching jobs, is testing one option here:
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : New functionality.


Question: <del> v1.8.3-3 </del> <ins> v1.9.1-1 </ins>

</details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I approved this for v1.9, this is something we definitely want in order to keep up with Go (and for improved performance)
we're starting to use the forked images now /cc this should be more feasible very soon Edit: test-infra jobs have also been using bazel 0.6 (now 0.6.1) / a recent rules_go to soak things a bit
We've identified a failing test as a result of the WIP PR to upgrade the codebase to go1.9: The admission webhook test fails with certificate errors under go1.9.1, but runs successfully under go1.8.3. cc: who wanted to be alerted to potential bugs
- please see for a possible fix


Question: configs[CheckMetadataConcealment] = Config{e2eRegistry, "metadata-concealment", "1.2"} configs[CudaVectorAdd] = Config{e2eRegistry, "cuda-vector-add", "1.0"} configs[CudaVectorAdd2] = Config{e2eRegistry, "cuda-vector-add", "2.0"} <del> configs[DebianIptables] = Config{buildImageRegistry, "debian-iptables", "buster-v1.4.0"} </del> <ins> configs[DebianIptables] = Config{buildImageRegistry, "debian-iptables", "buster-v1.5.0"} </ins> configs[EchoServer] = Config{e2eRegistry, "echoserver", "2.2"} configs[Etcd] = Config{gcRegistry, "etcd", "3.4.13-0"} configs[GlusterDynamicProvisioner] = Config{dockerGluster, "glusterdynamic-provisioner", "v1.0"}

The based container images for are shipped with the wrong architecture within the manifest if not amd64: Refers to We updated the images a few minutes ago to buster-v1.3.0, which are now ready to be used as base for the multi-architecture images. If the images contain the right architectures, then it may magically happen that the kube-proxy images are fixed, too. This is because we already use the build syntax for it: /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release


Question: }, }, }, <ins> { name: "node port service with protocol sctp on a node with multiple nodeIPs", services: []*v1.Service{ makeTestService("ns1", "svc1", func(svc *v1.Service) { svc.Spec.Type = "NodePort" svc.Spec.ClusterIP = "10.20.30.41" svc.Spec.Ports = []v1.ServicePort{{ Name:     "p80", Port:     int32(80), Protocol: v1.ProtocolSCTP, NodePort: int32(3001), }} }), }, endpoints: []*v1.Endpoints{ makeTestEndpoints("ns1", "svc1", func(ept *v1.Endpoints) { ept.Subsets = []v1.EndpointSubset{{ Addresses: []v1.EndpointAddress{{ IP: "10.180.0.1", }}, Ports: []v1.EndpointPort{{ Name: "p80", Port: int32(80), }}, }} }), }, nodeIPs: []net.IP{ net.ParseIP("100.101.102.103"), net.ParseIP("100.101.102.104"), net.ParseIP("100.101.102.105"), net.ParseIP("2001:db8::1:1"), net.ParseIP("2001:db8::1:2"), net.ParseIP("2001:db8::1:3"), }, nodePortAddresses: []string{}, expectedIPVS: &ipvstest.FakeIPVS{ Services: map[ipvstest.ServiceKey]*utilipvs.VirtualServer{ { IP:       "10.20.30.41", Port:     80, Protocol: "SCTP", }: { Address:   net.ParseIP("10.20.30.41"), Protocol:  "SCTP", Port:      uint16(80), Scheduler: "rr", }, { IP:       "100.101.102.103", Port:     3001, Protocol: "SCTP", }: { Address:   net.ParseIP("100.101.102.103"), Protocol:  "SCTP", Port:      uint16(3001), Scheduler: "rr", }, { IP:       "100.101.102.104", Port:     3001, Protocol: "SCTP", }: { Address:   net.ParseIP("100.101.102.104"), Protocol:  "SCTP", Port:      uint16(3001), Scheduler: "rr", }, { IP:       "100.101.102.105", Port:     3001, Protocol: "SCTP", }: { Address:   net.ParseIP("100.101.102.105"), Protocol:  "SCTP", Port:      uint16(3001), Scheduler: "rr", }, { IP:       "2001:db8::1:1", Port:     3001, Protocol: "SCTP", }: { Address:   net.ParseIP("2001:db8::1:1"), Protocol:  "SCTP", Port:      uint16(3001), Scheduler: "rr", }, { IP:       "2001:db8::1:2", Port:     3001, Protocol: "SCTP", }: { Address:   net.ParseIP("2001:db8::1:2"), Protocol:  "SCTP", Port:      uint16(3001), Scheduler: "rr", }, { IP:       "2001:db8::1:3", Port:     3001, Protocol: "SCTP", }: { Address:   net.ParseIP("2001:db8::1:3"), Protocol:  "SCTP", Port:      uint16(3001), Scheduler: "rr", }, }, Destinations: map[ipvstest.ServiceKey][]*utilipvs.RealServer{ { IP:       "10.20.30.41", Port:     80, Protocol: "SCTP", }: { { Address: net.ParseIP("10.180.0.1"), Port:    uint16(80), Weight:  1, }, }, { IP:       "100.101.102.103", Port:     3001, Protocol: "SCTP", }: { { Address: net.ParseIP("10.180.0.1"), Port:    uint16(80), Weight:  1, }, }, { IP:       "100.101.102.104", Port:     3001, Protocol: "SCTP", }: { { Address: net.ParseIP("10.180.0.1"), Port:    uint16(80), Weight:  1, }, }, { IP:       "100.101.102.105", Port:     3001, Protocol: "SCTP", }: { { Address: net.ParseIP("10.180.0.1"), Port:    uint16(80), Weight:  1, }, }, { IP:       "2001:db8::1:1", Port:     3001, Protocol: "SCTP", }: { { Address: net.ParseIP("10.180.0.1"), Port:    uint16(80), Weight:  1, }, }, { IP:       "2001:db8::1:2", Port:     3001, Protocol: "SCTP", }: { { Address: net.ParseIP("10.180.0.1"), Port:    uint16(80), Weight:  1, }, }, { IP:       "2001:db8::1:3", Port:     3001, Protocol: "SCTP", }: { { Address: net.ParseIP("10.180.0.1"), Port:    uint16(80), Weight:  1, }, }, }, }, expectedIPSets: netlinktest.ExpectedIPSet{ kubeNodePortSetSCTP: { { IP:       "100.101.102.103", Port:     3001, Protocol: strings.ToLower(string(v1.ProtocolSCTP)), SetType:  utilipset.HashIPPort, }, { IP:       "100.101.102.104", Port:     3001, Protocol: strings.ToLower(string(v1.ProtocolSCTP)), SetType:  utilipset.HashIPPort, }, { IP:       "100.101.102.105", Port:     3001, Protocol: strings.ToLower(string(v1.ProtocolSCTP)), SetType:  utilipset.HashIPPort, }, { IP:       "2001:db8::1:1", Port:     3001, Protocol: strings.ToLower(string(v1.ProtocolSCTP)), SetType:  utilipset.HashIPPort, }, { IP:       "2001:db8::1:2", Port:     3001, Protocol: strings.ToLower(string(v1.ProtocolSCTP)), SetType:  utilipset.HashIPPort, }, { IP:       "2001:db8::1:3", Port:     3001, Protocol: strings.ToLower(string(v1.ProtocolSCTP)), SetType:  utilipset.HashIPPort, }, }, }, }, </ins> } for _, test := range tests {

What happened: SCTP nodeport service is not working for all IPs present in the node when ipvs is enabled. It is working only for node's InternalIP. What you expected to happen: SCTP nodeport service must work for all ips present in the node when ipvs is enabled. How to reproduce it (as minimally and precisely as possible): Create a kubernetes cluster with at least one node having multiple interfaces with different IPs. Start an SCTP server deployment. Then create a nodeport service for the same. Now try to reach the service using all the IPs present in that node. The service will be accessible only via node's InternalIP not any other IPs. Environment: Kubernetes version: v1.15.2 OS: ubuntu 18.04 Kernel: 4.15.0-54-generic Network plugin and version (if this is a network-related bug): ipset version v6.34
/sig network /area kube-proxy /area ipvs
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: // The application protocol for this port. // This field follows standard Kubernetes label syntax. // Un-prefixed names are reserved for IANA standard service names (as per <del> // RFC-6335 and http://www.iana.org/assignments/service-names). </del> <ins> // RFC-6335 and https://www.iana.org/assignments/service-names). </ins> // Non-standard protocols should use prefixed names such as // mycompany.com/my-custom-protocol. // +optional

We should make the change in PR API reference links to should use HTTPS, not HTTP Visit h and look at the IANA hyperlinks Once this fix is made, a PR to is needed to update the docs (by autogenerating them). This typically would be deferred until the next Kubernetes release. v1.23 not applicable not applicable not applicable not applicable not applicable
/sig docs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I opened this issue so that someone else can open a new PR that is equivalent to .
/retitle Update links to IANA for Service Name and Transport Protocol Port Number Registry
I would like to work on this issue. May I assign it to myself?
Anyone - including you - is welcome to work on this issue! So long as you've you can open a pull request to propose a fix.
Thanks
/assign
I have made the PR. I hope this works. If not then will be happy to do the necessary changes.
Had to close the previous PR because of issues, here is the new one I made.


Question: &unversioned.APIGroup{}, &unversioned.APIResourceList{}, ) <ins> addDeepCopyFuncs(scheme) addDefaultingFuncs(scheme) addConversionFuncs(scheme) </ins> }

I started the federation-apiserver and went to and got the following output: Expected: I should have got an empty v1.ServiceList Any idea on what we are doing wrong? Have you guys seen this error?
I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote:
Deep copy generation was slightly broken as well - you may want to double check that they are registered. On May 23, 2016, at 5:50 PM, Daniel Smith wrote: I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote: — You are receiving this because you are on a team that was mentioned. Reply to this email directly or view it on GitHub
I got the same error with latest branch, as I was swamping on service controller, so I use a previous version and worked around it.
deep copies were indeed missing, but adding them hasnt fixed the issue. Will debug more. is my PR to add the deep copies cc Please feel free to file issues for any such problem you face. We need to fix all these issues. Its good if we know about them early. Thanks!
fixes this


Question: for format := range columnsFormats { formats = append(formats, format) } <ins> sort.Strings(formats) </ins> return formats }

Similar to issue , building kubernetes1.23 resulted in 1 variation: builds should be fully determined by inputs on openSUSE or Debian, This bug was found while working on . <details23.0 </details<details</details<details</details<details</details<details</details<details</details
/sig release /area build-release
/assign
/sig cli
/triage accepted /assign


Question: // compute the context. Mutation is allowed. ValidatedPSPAnnotation is not taken into account. allowedPod, pspName, validationErrs, err := p.computeSecurityContext(ctx, a, pod, true, "") if err != nil { <del> return admission.NewForbidden(a, err) </del> <ins> return admission.NewForbidden(a, fmt.Errorf("PodSecurityPolicy: %w", err)) </ins> } if allowedPod != nil { *pod = *allowedPod

What happened: e2e tests detect whether PSP is enabled based on presence of PSP API objects. An addon that proactively includes a PSP in a manifest can trigger that detection and enable PSP-specific e2e tests that should not be run. What you expected to happen: In addition to detecting presence of PSP objects, consider dry-run creating a pod and verifying the annotation indicating the selected PSP policy is , indicating the admission plugin is active. /sig auth /assign


Question: "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/managedfields" "k8s.io/apimachinery/pkg/watch" <del> "k8s.io/kube-openapi/pkg/validation/spec" </del> "k8s.io/utils/ptr" )

Smoke tests run by publishing-bot on client-go master are failing Ref for failure logs: Tests defined for client-go, after publishing, Since commit No response As part of , when module is used from within the staging directory, can be used by traversing to the parent directory. But once the module is made as a separate repository as part of publishing, then traversing to repository using path will not work. No response /sig-api-machinery
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looking
I'll remove the swagger file dependency, I don't think we strictly need it.
/triage accepted