Question: // +groupName=samplecontroller.k8s.io // Package v1alpha1 is the v1alpha1 version of the API. <del> package v1alpha1 </del> <ins> package v1alpha1 // import "k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1" </ins>

Look at one of the directories under (Please don't pick more than one directory for one PR as it will cause issues for reviewers) Find all the files named (example find -name ) See the following link it has a this tells tools how this package needs to be imported. So we should ensure that each has an import Add a description appropriate to the package (similar to the one below) Look through sub directories and see if there are any other important directories that are likely to be used by outside users and add a Why are we doing this? We perfer to use vanity urls instead of directly using to refer to code, so this helps ensure that tools honor our wish. See It would be good to have some package level documentation since these directories under staging/ are meant to be for use by folks outside of kubernetes github org. /good-first-issue
/kind documentation /sig architecture
currently working my through Not going to use / assign since there could be a number of PRs for this issue.
/lifecycle active /good-first-issue edit: hmm..
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /lifecycle active /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
May I try on
IIUC this means that there needs to be a even at the root of the repo? Then how does work? :thinking:
Hi , can I make a PR for this directory ?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .


Question: // +groupName=samplecontroller.k8s.io // Package v1alpha1 is the v1alpha1 version of the API. <del> package v1alpha1 </del> <ins> package v1alpha1 // import "k8s.io/sample-controller/pkg/apis/samplecontroller/v1alpha1" </ins>

/lifecycle rotten
I guess this is still not done, was supposed to partially fix this issue. /reopen
Reopened this issue. <detailsIn response to : I guess this is still not done, was supposed to partially fix this issue. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /area code-organization


Question: # Arguments: a list of kubernetes packages to build. # Expected variables: ${build_args} should be set to an array of Go build arguments. # In addition, ${package} and ${platform} should have been set earlier, and if <del> # ${build_with_coverage} is set, coverage instrumentation will be enabled. </del> <ins> # ${KUBE_BUILD_WITH_COVERAGE} is set, coverage instrumentation will be enabled. </ins> # # Invokes Go to actually build some packages. If coverage is disabled, simply invokes # go install. If coverage is enabled, builds covered binaries using go test, temporarily # producing the required unit test files and then cleaning up after itself. # Non-covered binaries are then built using go install as usual. kube::golang::build_some_binaries() { <del> if [[ -n "${build_with_coverage:-}" ]]; then </del> <ins> if [[ -n "${KUBE_BUILD_WITH_COVERAGE:-}" ]]; then </ins> local -a uncovered=() for package in "$@"; do if kube::golang::is_instrumented_package "${package}"; then

/kind cleanup /sig testing /priority important-longterm This is a followup issue to address the "standard build globals" nit on At the very least I think we shouldn't capture our custom env var. Also think it'd be worth documenting the assumed variables in a list, similar to
cc
/assign


Question: # See the License for the specific language governing permissions and # limitations under the License. <del> # The line below points to debian:jessie as of 2019-10-23. The SHA should be # kept in sycn with debian_jessie definition in the WORKSPACE file. FROM debian@sha256:e25703ee6ab5b2fac31510323d959cdae31eebdf48e88891c549e55b25ad7e94 </del> <ins> # The line below points to distroless/base as of 2019-11-15. The SHA should be # kept in sycn with distroless_base definition in the WORKSPACE file. FROM gcr.io/distroless/base@sha256:7fa7445dfbebae4f4b7ab0e6ef99276e96075ae42584af6286ba080750d6dfe5 </ins> COPY kubemark /kubemark

There are CVEs happening in different images - we already migrated Kubmeark from busybox in , but we should go one step further and migrate the base image to distroless. There are some things that would need to be changed mentioned here: /sig scalability
Can I work on this? /assign
/remove-help
- sure that would be great.
Hi based on the discussion in , the kubemark image has dependency on and maybe more other utilities. So it seems impossible to migrate the base image to distroless? Or do we need something like debian-hyperkube-base?
The mount one we may get rid of - basically we should be able to switch to FakeMounter:


Question: hash := hash(podUpdate.nodeName()) select { case <-stopCh: <ins> tc.podUpdateQueue.Done(item) </ins> break case tc.podUpdateChannels[hash%workers] <- podUpdate: } <ins> tc.podUpdateQueue.Done(item) </ins> } }(stopCh)

We're seeing an issue with large memory growth in NoExecuteTaintManager when pods with tolerations are rapidly created/deleted. The item to the workqueue contains the full pod objects received from the informer: That means those pod objects are retained until they are processed. Additionally, it looks like the processing loop never removed the keys from the workqueue when it was done. It also means that multiple updates for the same pod result in multiple items in the workqueue, rather than enqueuing the pod to be processed once, like other controllers do. Simplifying this controller to enqueue pod updates only by name/namespace, and operate on the latest pod obtained from the store (like other controllers do) would speed up the processing and relieve the memory burden of the workqueue. /sig scheduling /cc
/assign
Thanks for finding this Jordan - looking into your fixes.
the memory issue was fixed by /close


Question: <del> { "apiVersion": "v1beta3", "kind": "ResourceQuota", "metadata": { "name": "quota" }, "spec": { "hard": { "memory": "1Gi", "cpu": "20", "pods": "10", "services": "5", "replicationcontrollers":"20", "resourcequotas":"1", "secrets":"10", "persistentvolumeclaims":"10" } } } </del>

Resource quota example is missing readme. Was it once there and was accidentally deleted?
I wrote something up in Origin repo I need to repurpose for Kubernetes. Sent from my iPhone


Question: return nil } <ins> // applyFSGroup applies the volume ownership it derives its logic // from https://github.com/kubernetes/kubernetes/issues/66323 // 1) if fstype is "", then skip fsgroup (could be indication of non-block filesystem) // 2) if fstype is provided and pv.AccessMode == ReadWriteOnly and !c.spec.ReadOnly then apply fsgroup func (c *csiMountMgr) applyFSGroup(fsType string, fsGroup *int64) error { if fsGroup != nil { if fsType == "" { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, fsType not provided")) return nil } accessModes := c.spec.PersistentVolume.Spec.AccessModes if c.spec.PersistentVolume.Spec.AccessModes == nil { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, access modes not provided")) return nil } if !hasReadWriteOnce(accessModes) { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, only support ReadWriteOnce access mode")) return nil } if c.readOnly { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, volume is readOnly")) return nil } err := volume.SetVolumeOwnership(c, fsGroup) if err != nil { return err } glog.V(4).Info(log("mounter.SetupAt fsGroup [%d] applied successfully to %s", *fsGroup, c.volumeID)) } return nil } </ins> // isDirMounted returns the !notMounted result from IsLikelyNotMountPoint check func isDirMounted(plug *csiPlugin, dir string) (bool, error) { mounter := plug.host.GetMounter(plug.GetPluginName())

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: fsgroup is only intended for RWO volumes because it recursively changes the permissions of all the directories in the volume. This can cause problems for nfs-type volumes because multiple pods could access the same volume and have the permissions changed out from underneath. What you expected to happen: Determine whether or not the plugin supports fsgroup either via: access mode. This is still a bit ambiguous for nfs volume types, that can theoretically support rwo, especially since Kubernetes does not do any sort of access mode enforcement. a new csi capability
CC
Are you saying that CSI internal driver should check driver capability for RWO, if supported, only then apply value ? What would the be ?
Yes one possible way is to infer fsgroup is supported for all rwo volumes, which may not be true? Like a nfs volume technically supports rwo but not fsgroup. So the other option is to make it a new CSI capability. I'm not sure exactly sure how best to describe it though, maybe something about volume ownership management.
Will it work if is not set then don't chown the ownership? I would think that for shared file system types(nfs, glusterfs) - won't be set. For raw block storage too - I think we shouldn't have to do this.
what do you mean by ? Shouldnt have to solve this issue ?
I meant - which is obvious. I just meant that the entire mechanism of changing ownership of files should apply only to volume types with block storage filesystems. I am just wondering if we can get away with detecting fsType and not have to introduce new capability in CSI.
agreed, do not map well to current csi.Capabilities enums which makes it hard to determine RWO when mounting. Perhaps, instead of a new capability, we revisit the existing capabilities to see if there are any way RWO can be reliably inferred from returned capabilities.


Question: return nil } <ins> // applyFSGroup applies the volume ownership it derives its logic // from https://github.com/kubernetes/kubernetes/issues/66323 // 1) if fstype is "", then skip fsgroup (could be indication of non-block filesystem) // 2) if fstype is provided and pv.AccessMode == ReadWriteOnly and !c.spec.ReadOnly then apply fsgroup func (c *csiMountMgr) applyFSGroup(fsType string, fsGroup *int64) error { if fsGroup != nil { if fsType == "" { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, fsType not provided")) return nil } accessModes := c.spec.PersistentVolume.Spec.AccessModes if c.spec.PersistentVolume.Spec.AccessModes == nil { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, access modes not provided")) return nil } if !hasReadWriteOnce(accessModes) { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, only support ReadWriteOnce access mode")) return nil } if c.readOnly { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, volume is readOnly")) return nil } err := volume.SetVolumeOwnership(c, fsGroup) if err != nil { return err } glog.V(4).Info(log("mounter.SetupAt fsGroup [%d] applied successfully to %s", *fsGroup, c.volumeID)) } return nil } </ins> // isDirMounted returns the !notMounted result from IsLikelyNotMountPoint check func isDirMounted(plug *csiPlugin, dir string) (bool, error) { mounter := plug.host.GetMounter(plug.GetPluginName())

Currently, this is what the code is doing which we knew needed to be revisited:
Users may not fully understand what fsgroup is and set it anyway, or a default psp with some fsgroup may be defined in which it will automatically be set on all pods. The first case we can attribute to user error, but the second case may not have a good workaround and could cause permission issues on rwx volumes. So I think we do need to solve this.
Yes, agreed this has to be solved. As you pointed out user error or improperly configured PSP can set value that does not match the correct attribute, causing permission issues. Here is what I think can be done without new CSI capabilities: , then do not apply fsgroup then: , apply fsgroup , do not apply fsgroup to void stated issue then: , apply fsgroup I think that may help deduce when to apply fsgroup.
If mount reference count <= 1, apply fsgroup If mount reference count 1, do not apply fsgroup to void stated issue A RWM volume (say NFS) can be mounted on multiple nodes. Do you have reliable count of mounts across all nodes?
In addition, that would be a change in semantics to how we support rwx volumes today. If we don't have a new CSI capability, then I think the best course of action is to only apply fsgroup for RWO volumes. Some questions remain: Do all RWO volumes support fsgroup? RWX is a superset of RWO. So if a user requests a RWO volume, it could still be satisfied by a RWX volume type. Can we detect this and still not apply the fsgroup even though the volume capability will say RWO?
If there's not a reliable way to enumerate node mounts (thinking about it, for CSI external drivers, the answer is NO). Theres is no reliable way to have k8s autocorrect a bad fsgroup/PV.AccessMode combo. So we would have to rely on the config provided by user/admin. I have come to similar conclusion as -- mainly, apply fsgroup only to volumes with access modes. The other k8s specified access modes, , cannot be reliably applied.


Question: return nil } <ins> // applyFSGroup applies the volume ownership it derives its logic // from https://github.com/kubernetes/kubernetes/issues/66323 // 1) if fstype is "", then skip fsgroup (could be indication of non-block filesystem) // 2) if fstype is provided and pv.AccessMode == ReadWriteOnly and !c.spec.ReadOnly then apply fsgroup func (c *csiMountMgr) applyFSGroup(fsType string, fsGroup *int64) error { if fsGroup != nil { if fsType == "" { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, fsType not provided")) return nil } accessModes := c.spec.PersistentVolume.Spec.AccessModes if c.spec.PersistentVolume.Spec.AccessModes == nil { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, access modes not provided")) return nil } if !hasReadWriteOnce(accessModes) { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, only support ReadWriteOnce access mode")) return nil } if c.readOnly { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, volume is readOnly")) return nil } err := volume.SetVolumeOwnership(c, fsGroup) if err != nil { return err } glog.V(4).Info(log("mounter.SetupAt fsGroup [%d] applied successfully to %s", *fsGroup, c.volumeID)) } return nil } </ins> // isDirMounted returns the !notMounted result from IsLikelyNotMountPoint check func isDirMounted(plug *csiPlugin, dir string) (bool, error) { mounter := plug.host.GetMounter(plug.GetPluginName())

Even if we were to introduce additional plugin capabilities that the CO can query (from driver) to find out what modes can be applied, there would still be the possibility of user/admin mis-configuration. Probably not, Is this where you think an additional capability would help ( ) That is a good point. Can't think of a way to detect/guess RWX when RWO is specified, this without the driver providing that info. Can we adopt the rule until we decide what capabilities would look like ?
I will just try to summarize the conversation from Slack. We can safely assume that only applies to block storage type with file system on top of it. So for all - volumes that need to have called MUST have a file system() on them. Is that assumption correct? If it is - can we not use presence of field or we can query the local volume for fs type (like how we query selinux relabelling capability) and call only when - volume is RW and has a valid block storage file system type. Will this not solve our use case? EDIT: renamed to .
We'll need to make it clear that is only for block-based storage systems, and the field cannot be reused for some multi-protocol file server (ie, nfsv3, nfsv4, smb, etc). For that case, then the plugin will need to expose its own new storageclass parameter + volume attribute to specify protocol.
Can we not use to get fsType - and use that information for calling ? It may be problematic to rely entirely on present inside csi source.
the only problem with is that in CSI volume info is intentionally opaque to the CSI code inside k8s. All mount logic is delegated to the driver.
Instead of a new capability or trying to deduce how best to apply fsgroup, why not pass the to the driver and let it decide how to apply it (either as a attribute for the mount or top level request param)? Right now, all mount logic has been delegated to the external CSI driver except for applying permission. Thoughts ?
that is true but we can still inspect file system on the mounted path because by the time gets called volume is already mounted. This should be no different than how selinux support is determined. what you think?
is this approved for 1.12 milestone? Thank you for working on this,
still discussing how best to approach this.


Question: return nil } <ins> // applyFSGroup applies the volume ownership it derives its logic // from https://github.com/kubernetes/kubernetes/issues/66323 // 1) if fstype is "", then skip fsgroup (could be indication of non-block filesystem) // 2) if fstype is provided and pv.AccessMode == ReadWriteOnly and !c.spec.ReadOnly then apply fsgroup func (c *csiMountMgr) applyFSGroup(fsType string, fsGroup *int64) error { if fsGroup != nil { if fsType == "" { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, fsType not provided")) return nil } accessModes := c.spec.PersistentVolume.Spec.AccessModes if c.spec.PersistentVolume.Spec.AccessModes == nil { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, access modes not provided")) return nil } if !hasReadWriteOnce(accessModes) { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, only support ReadWriteOnce access mode")) return nil } if c.readOnly { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, volume is readOnly")) return nil } err := volume.SetVolumeOwnership(c, fsGroup) if err != nil { return err } glog.V(4).Info(log("mounter.SetupAt fsGroup [%d] applied successfully to %s", *fsGroup, c.volumeID)) } return nil } </ins> // isDirMounted returns the !notMounted result from IsLikelyNotMountPoint check func isDirMounted(plug *csiPlugin, dir string) (bool, error) { mounter := plug.host.GetMounter(plug.GetPluginName())

It would be a 1.12.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Approved for 1.12
Based on previous post, this is a summary: If then is not applied (because it could be an indication of a nonblock fs, or error) Done is provided if fstype is invalid Done if fstype is valid and then apply fsgroup logic else ignore fsGroup Done
- any updates on ?
Waiting for a review and approval.
you have a outstanding comment on the PR. Can you please answer that?
How does this approach relate to inline CSI volumes? There's no PersistentVolume involved, and hence no way to specify an access mode. , but my driver ignores it (always using tmpfs). I considered having my CSI driver read the mounting pod's spec to see if it has a security context with "fsGroup" set, since that detail is not supplied to the driver, as . My goal is to have the volume mounted with the files on it readable by the container's user, but there's no way for my driver to know who that user will be. Am I supposed to just use 0444 for the ownership bits, to allow everyone to read the files?


Question: resources: {} terminationMessagePath: "/dev/termination-log" imagePullPolicy: IfNotPresent <del> capabilities: {} </del> securityContext: capabilities: {} privileged: false

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Vagrant OS (e.g. from /etc/os-release): CoreOS Kernel (e.g. ):Install tools: Vagrant & Others: What happened: What you expected to happen: Success How to reproduce it (as minimally and precisely as possible): Followed instructions at Tried to run Received: Anything else do we need to know: From googling, it seems several of the fields in the config are for when using OpenShift origin. But creating the replication controller is a prerequisite to launching OpenShift Origin. Chicken & egg problem.


Question: } } if len(unknownLabels) > 0 { <del> // TODO(liggitt): in 1.16, return an error klog.Warningf("unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels: %v", unknownLabels.List()) klog.Warningf("in 1.16, --node-labels in the 'kubernetes.io' namespace must begin with an allowed prefix (%s) or be in the specifically allowed set (%s)", strings.Join(kubeletapis.KubeletLabelNamespaces(), ", "), strings.Join(kubeletapis.KubeletLabels(), ", ")) </del> <ins> return fmt.Errorf("unknown 'kubernetes.io' or 'k8s.io' labels specified with --node-labels: %vn--node-labels in the 'kubernetes.io' namespace must begin with an allowed prefix (%s) or be in the specifically allowed set (%s)", unknownLabels.List(), strings.Join(kubeletapis.KubeletLabelNamespaces(), ", "), strings.Join(kubeletapis.KubeletLabels(), ", ")) </ins> } return nil

xref In 1.16: change the metadata-proxy addon to use change the kube-proxy addon to use change the masq-agent addon to use stop setting , , and in kube-up
Hey I'm trying to start to contribute to kubernetes and this seem to be an easy task. Can I help with this one?


Question: reference, target, current, <del> minPods, </del> <ins> *minPods, </ins> maxPods, translateTimestamp(hpa.CreationTimestamp), ); err != nil {

returns It's because we recently change HorizontalPodAutoscalerSpec.MinReplicas from int to *int.


Question: ginkgo.By("updating the pod") podClient.Update(pod.ObjectMeta.Name, func(pod *v1.Pod) { <del> pod.ObjectMeta.Annotations = map[string]string{"mysubpath": "mypath"} </del> <ins> if pod.ObjectMeta.Annotations == nil { pod.ObjectMeta.Annotations = make(map[string]string) } pod.ObjectMeta.Annotations["mysubpath"] = "mypath" </ins> }) ginkgo.By("waiting for pod running")

In our Tanzu Supervisor Cluster k8s setup, we observe that the "should verify that a failing subpath expansion can be modified during the lifecycle of a container" conformance test fails. The root-cause seems to be that the test is updating the pod annotations, thereby resulting in removing the other annotations present in the pod object. Removing the custom annotations that we use causes the test to fail during pod deletion. Should this test perhaps use Patch instead of Update so that the annotations are merged and not replaced? Thanks ginkgo.By("updating the pod") podClient.Update(pod.ObjectMeta.Name, func(pod *v1.Pod) { pod.ObjectMeta.Annotations = map[string]string{"mysubpath": "mypath"} })
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /sig conformance I personally agree that it should not remove existing annotations, but better ask the test owners
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig storage /sig conformance I personally agree that it should not remove existing annotations, but better ask the test owners Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
May I request the original author to weigh in on this? Thank you.
I dont have any issues with the code change to a merge but what would bother me is that conformance tests are being altered by outside influences which doesnt seem particularly right. I think this would be for the sig storage owners to decide
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale


Question: return fmt.Errorf("unimplemented") } <del> func (f *fakeVMSet) GetDiskLun(diskName, diskURI string, nodeName types.NodeName) (int32, error) { return -1, fmt.Errorf("unimplemented") } func (f *fakeVMSet) GetNextDiskLun(nodeName types.NodeName) (int32, error) { return -1, fmt.Errorf("unimplemented") } func (f *fakeVMSet) DisksAreAttached(diskNames []string, nodeName types.NodeName) (map[string]bool, error) { </del> <ins> func (f *fakeVMSet) GetDataDisks(nodeName types.NodeName) ([]compute.DataDisk, error) { </ins> return nil, fmt.Errorf("unimplemented") }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: VirtualMachineScaleSetVM and VirtualMachine are different data structures (because of different API versions), so the disk attach operation seems duplicate between vm and vmss. That means although disk operations are in same logic, some duplication are still required now. They should use same api version in the future and should be merged into same code base, e.g. move sharing logic to Refer What you expected to happen: vmss and vm disk operations should share same code logic. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I have checked with currently we can’t merge, e.g. following two structs are different since they using different sdk-for-azure versions. While in the end when sdk-for-azure version is upgraded, above two versions should be same and at that time, we should merge functions like ‘AttachDisk’, ‘DetachDiskByName’, 'GetDiskLun', GetNextDiskLun, DisksAreAttached


Question: exec, syncPeriod, minSyncPeriod, filterCIDRs(true, excludeCIDRs), strictARP, tcpTimeout, tcpFinTimeout, udpTimeout, masqueradeAll, masqueradeBit, localDetectors[1], hostname, nodeIP[1], <del> nil, nil, scheduler, nodePortAddresses, kernelHandler) </del> <ins> nil, nil, scheduler, nodePortAddresses6, kernelHandler) </ins> if err != nil { return nil, fmt.Errorf("unable to create ipv6 proxier: %v", err) }

What happened: In a dual-stack cluster when specifying both ipv6 and ipv4 addresses in the entries are inserted in ipvs for the ipv6 nodeports but with an invalid ipv4 address; The example shows the ipv4 loopback address but any ipv4 address in will cause a faulty entry. What you expected to happen: No ipv4 entry shall be inserted for the ipv6 nodeport. How to reproduce it (as minimally and precisely as possible): Configure with both ipv4 and ipv6 addresses Bring up a dual-stack cluster with proxy-mode Create a service with ipFamily:IPv6 Do Anything else we need to know?: Proxy-mode=iptables seems to work. With no ipv4 entries are inserted for ipv6 nodeports. In it is recommended to use but doing so makes kube-proxy enter rules for address (in addition to "127.0.0.1"). While these entries cause no harm they can't be used and should not be there. These entries are created for both proxy-mode "iptables" and "ipvs". This should not be handled in this issue but in another (new) issue if needed. Example; Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Kvm's on Ubuntu OS (e.g: ): xcluster Kernel (e.g. ):Install tools: "from scratch" Network plugin and version (if this is a network-related bug): calico/cni:v3.12.0 Others: /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/area ipvs /area ipv6
/remove-triage unresolved /assign I take this myself if no one objects.
The ipv4 address is obviously a ipv4-converted ipv6 from .
According to the code here (), we may need to filter the before passing it to .
Looks that that's it. Please go ahead and make a PR, or I can do it if you prefer. I will help test the PR of course. Here are some review comments right away; Use v1.IPFamily as param to the filter function rather than The filter function seem usable and generic and may be placed in some "util" package, but leave it as-is and let the maintainers decide that.
Hi feel free to send a PR please, as you have took this already :) I'd like to help review it!
The "addresses" are really CIDRs which make the fix even simpler.


Question: frameworkruntime.WithCaptureProfile(frameworkruntime.CaptureProfile(c.frameworkCapturer)), frameworkruntime.WithClusterEventMap(c.clusterEventMap), frameworkruntime.WithParallelism(int(c.parallellism)), <ins> frameworkruntime.WithExtenders(extenders), </ins> ) if err != nil { return nil, fmt.Errorf("initializing profiles: %v", err)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --here is supposed to be set through , but is not called from anywhere except tests, so always returns . should be set through here. Am I missing something? N/A None Kubernetes version (use ): master () Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: None
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
Thanks for pointing that out. The plumbing of extenders to the framework impact calling extenders on preemption. That seems to be a path not covered by tests end to end. Can you submit a PR to fix this? as you mentioned we need to pass the extenders on profile creation. We need to patch the fix back to v1.21 and v1.20 (the release where this was refactored). /cc
I have submitted a PR for a fix.


Question: v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <del> "k8s.io/apimachinery/pkg/util/sets" </del> "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/kubernetes/pkg/kubelet/events" "k8s.io/kubernetes/test/e2e/framework"

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: , Which test(s) are failing: Since when has it been failing: since this PR introduced them Testgrid link: Note that this is the flakiest test in both dashboards, and failing quite frequently. Reason for failure: unknown Anything else we need to know: cc
/sig node
see: , both getting empty PID and multiple PIDs.
opened revert at
for discoverability for the release team /priority critical-urgent /milestone v1.17


Question: AllNamespaces bool // true iff the action is namespaced but works on aggregate result for all namespaces } <ins> // An interface to see if one storage supports override its default verb for monitoring type StorageMetricsOverride interface { // OverrideMetricsVerb gives a storage object an opportunity to override the verb reported to the metrics endpoint OverrideMetricsVerb(oldVerb string) (newVerb string) } </ins> // An interface to see if an object supports swagger documentation as a method type documentable interface { SwaggerDoc() map[string]string

Is this a BUG REPORT or FEATURE REQUEST?: What happened: use kubectl search log with follow option get api latency (apiserverrequestlatenciessummary{quantile="0.99",resource="pods",verb="GET"} )high to give a alert in my cluster. the apiserver log is below and we can see it costs 1m25s. What you expected to happen: do not make this api latency account in getting pod request api latency How to reproduce it (as minimally and precisely as possible): use kubectl to search a pod log with follow option long enough and you will get this metric apiserverrequestlatenciessummary{quantile="0.99",resource="pods",verb="GET"} higher than the ordinary case. Anything else we need to know?: I am happy to fix it. Environment: Kubernetes version (use ): 1.6 Cloud provider or hardware configuration**: bare-metal OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): 4.4 Install tools: kubeadm Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig API Machinery
/area apiserver
/sig api-machinery
/assign Thanks... looking forward to your PR!
fixed with PTAL


Question: _, err = f.ClientSet.CoreV1().Endpoints(testNamespaceName).Get(context.TODO(), testEndpointName, metav1.GetOptions{}) framework.ExpectError(err, "should not be able to fetch Endpoint") }) <ins> ginkgo.It("should complete a service status lifecycle", func() { ns := f.Namespace.Name svcResource := schema.GroupVersionResource{Group: "", Version: "v1", Resource: "services"} svcClient := f.ClientSet.CoreV1().Services(ns) testSvcName := "test-service-" + utilrand.String(5) testSvcLabels := map[string]string{"test-service-static": "true"} testSvcLabelsFlat := "test-service-static=true" w := &cache.ListWatch{ WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { options.LabelSelector = testSvcLabelsFlat return cs.CoreV1().Services(ns).Watch(context.TODO(), options) }, } svcList, err := cs.CoreV1().Services("").List(context.TODO(), metav1.ListOptions{LabelSelector: testSvcLabelsFlat}) framework.ExpectNoError(err, "failed to list Services") ginkgo.By("creating a Service") testService := v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:   testSvcName, Labels: testSvcLabels, }, Spec: v1.ServiceSpec{ Type: "ClusterIP", Ports: []v1.ServicePort{{ Name:       "http", Protocol:   v1.ProtocolTCP, Port:       int32(80), TargetPort: intstr.FromInt(80), }}, }, } _, err = cs.CoreV1().Services(ns).Create(context.TODO(), &testService, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create Service") ginkgo.By("watching for the Service to be added") ctx, cancel := context.WithTimeout(context.Background(), svcReadyTimeout) defer cancel() _, err = watchtools.Until(ctx, svcList.ResourceVersion, w, func(event watch.Event) (bool, error) { if svc, ok := event.Object.(*v1.Service); ok { found := svc.ObjectMeta.Name == testService.ObjectMeta.Name && svc.ObjectMeta.Namespace == ns && svc.Labels["test-service-static"] == "true" if !found { framework.Logf("observed Service %v in namespace %v with labels: %v & ports %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Labels, svc.Spec.Ports) return false, nil } framework.Logf("Found Service %v in namespace %v with labels: %v & ports %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Labels, svc.Spec.Ports) return found, nil } framework.Logf("Observed event: %+v", event.Object) return false, nil }) framework.ExpectNoError(err, "Failed to locate Service %v in namespace %v", testService.ObjectMeta.Name, ns) framework.Logf("Service %s created", testSvcName) ginkgo.By("Getting /status") svcStatusUnstructured, err := f.DynamicClient.Resource(svcResource).Namespace(ns).Get(context.TODO(), testSvcName, metav1.GetOptions{}, "status") framework.ExpectNoError(err, "Failed to fetch ServiceStatus of Service %s in namespace %s", testSvcName, ns) svcStatusBytes, err := json.Marshal(svcStatusUnstructured) framework.ExpectNoError(err, "Failed to marshal unstructured response. %v", err) var svcStatus v1.Service err = json.Unmarshal(svcStatusBytes, &svcStatus) framework.ExpectNoError(err, "Failed to unmarshal JSON bytes to a Service object type") framework.Logf("Service %s has LoadBalancer: %v", testSvcName, svcStatus.Status.LoadBalancer) ginkgo.By("patching the ServiceStatus") lbStatus := v1.LoadBalancerStatus{ Ingress: []v1.LoadBalancerIngress{{IP: "203.0.113.1"}}, } lbStatusJSON, err := json.Marshal(lbStatus) framework.ExpectNoError(err, "Failed to marshal JSON. %v", err) _, err = svcClient.Patch(context.TODO(), testSvcName, types.MergePatchType, []byte(`{"metadata":{"annotations":{"patchedstatus":"true"}},"status":{"loadBalancer":`+string(lbStatusJSON)+`}}`), metav1.PatchOptions{}, "status") framework.ExpectNoError(err, "Could not patch service status", err) ginkgo.By("watching for the Service to be patched") ctx, cancel = context.WithTimeout(context.Background(), svcReadyTimeout) defer cancel() _, err = watchtools.Until(ctx, svcList.ResourceVersion, w, func(event watch.Event) (bool, error) { if svc, ok := event.Object.(*v1.Service); ok { found := svc.ObjectMeta.Name == testService.ObjectMeta.Name && svc.ObjectMeta.Namespace == ns && svc.Annotations["patchedstatus"] == "true" if !found { framework.Logf("observed Service %v in namespace %v with annotations: %v & LoadBalancer: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Annotations, svc.Status.LoadBalancer) return false, nil } framework.Logf("Found Service %v in namespace %v with annotations: %v & LoadBalancer: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Annotations, svc.Status.LoadBalancer) return found, nil } framework.Logf("Observed event: %+v", event.Object) return false, nil }) framework.ExpectNoError(err, "failed to locate Service %v in namespace %v", testService.ObjectMeta.Name, ns) framework.Logf("Service %s has service status patched", testSvcName) ginkgo.By("updating the ServiceStatus") var statusToUpdate, updatedStatus *v1.Service err = retry.RetryOnConflict(retry.DefaultRetry, func() error { statusToUpdate, err = svcClient.Get(context.TODO(), testSvcName, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to retrieve service %s", testSvcName) statusToUpdate.Status.Conditions = append(statusToUpdate.Status.Conditions, metav1.Condition{ Type:    "StatusUpdate", Status:  metav1.ConditionTrue, Reason:  "E2E", Message: "Set from e2e test", }) updatedStatus, err = svcClient.UpdateStatus(context.TODO(), statusToUpdate, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "nn Failed to UpdateStatus. %vnn", err) framework.Logf("updatedStatus.Conditions: %#v", updatedStatus.Status.Conditions) ginkgo.By("watching for the Service to be updated") ctx, cancel = context.WithTimeout(context.Background(), svcReadyTimeout) defer cancel() _, err = watchtools.Until(ctx, svcList.ResourceVersion, w, func(event watch.Event) (bool, error) { if svc, ok := event.Object.(*v1.Service); ok { found := svc.ObjectMeta.Name == testService.ObjectMeta.Name && svc.ObjectMeta.Namespace == ns && svc.Annotations["patchedstatus"] == "true" if !found { framework.Logf("Observed Service %v in namespace %v with annotations: %v & Conditions: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Annotations, svc.Status.LoadBalancer) return false, nil } for _, cond := range svc.Status.Conditions { if cond.Type == "StatusUpdate" && cond.Reason == "E2E" && cond.Message == "Set from e2e test" { framework.Logf("Found Service %v in namespace %v with annotations: %v & Conditions: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Annotations, svc.Status.Conditions) return found, nil } else { framework.Logf("Observed Service %v in namespace %v with annotations: %v & Conditions: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Annotations, svc.Status.LoadBalancer) return false, nil } } } framework.Logf("Observed event: %+v", event.Object) return false, nil }) framework.ExpectNoError(err, "failed to locate Service %v in namespace %v", testService.ObjectMeta.Name, ns) framework.Logf("Service %s has service status updated", testSvcName) ginkgo.By("patching the service") servicePatchPayload, err := json.Marshal(v1.Service{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{ "test-service": "patched", }, }, }) _, err = svcClient.Patch(context.TODO(), testSvcName, types.StrategicMergePatchType, []byte(servicePatchPayload), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch service. %v", err) ginkgo.By("watching for the Service to be patched") ctx, cancel = context.WithTimeout(context.Background(), svcReadyTimeout) defer cancel() _, err = watchtools.Until(ctx, svcList.ResourceVersion, w, func(event watch.Event) (bool, error) { if svc, ok := event.Object.(*v1.Service); ok { found := svc.ObjectMeta.Name == testService.ObjectMeta.Name && svc.ObjectMeta.Namespace == ns && svc.Labels["test-service"] == "patched" if !found { framework.Logf("observed Service %v in namespace %v with labels: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Labels) return false, nil } framework.Logf("Found Service %v in namespace %v with labels: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Labels) return found, nil } framework.Logf("Observed event: %+v", event.Object) return false, nil }) framework.ExpectNoError(err, "failed to locate Service %v in namespace %v", testService.ObjectMeta.Name, ns) framework.Logf("Service %s patched", testSvcName) ginkgo.By("deleting the service") err = cs.CoreV1().Services(ns).Delete(context.TODO(), testSvcName, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete the Service. %v", err) ginkgo.By("watching for the Service to be deleted") ctx, cancel = context.WithTimeout(context.Background(), svcReadyTimeout) defer cancel() _, err = watchtools.Until(ctx, svcList.ResourceVersion, w, func(event watch.Event) (bool, error) { switch event.Type { case watch.Deleted: if svc, ok := event.Object.(*v1.Service); ok { found := svc.ObjectMeta.Name == testService.ObjectMeta.Name && svc.ObjectMeta.Namespace == ns && svc.Labels["test-service-static"] == "true" if !found { framework.Logf("observed Service %v in namespace %v with labels: %v & annotations: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Labels, svc.Annotations) return false, nil } framework.Logf("Found Service %v in namespace %v with labels: %v & annotations: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Labels, svc.Annotations) return found, nil } default: framework.Logf("Observed event: %+v", event.Type) } return false, nil }) framework.ExpectNoError(err, "failed to delete Service %v in namespace %v", testService.ObjectMeta.Name, ns) framework.Logf("Service %s deleted", testSvcName) }) </ins> }) // execAffinityTestForSessionAffinityTimeout is a helper function that wrap the logic of

[x] two weeks soak end date : 18 March 2021 According to this APIsnoop query, there are still some remaining RESOURCENAME endpoints which are untested. with this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Note: Community feedback for the e2e test has lead to a number of improvements including extending the test coverage to all outstanding endpoints listed above. As each endpoint is tested a ’watch’ confirms that the result is valid before testing the next endpoint. readCoreV1NamespacedServiceStatus (get /status) patchCoreV1NamespacedServiceStatus (patch /status) replaceCoreV1NamespacedServiceStatus (update /status) patchCoreV1NamespacedService Create a Service with a static label Patch the Service with a new label and updated data Get the Service to ensure it’s patched Upate the Service with a new label and updated data Get the Service to ensure it’s updated Delete Namespaced Service via a Collection with a LabelSelector This query by apisnoop shows that all outstanding endpoints as list at the start of this document have been hit by the e2e test. Note that the results do include other service endpoints that have been addressed in other conformance tests. If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
Can you please review the proposed test and advise if we could go ahead with a PR
/assign
It looks like it's patching metadata and spec but not status?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I've been retesting this mock test and apisnoop does log as been hit. From my testing below it looks like the patchStatus is working as expected or have I missed something?


Question: _, err = f.ClientSet.CoreV1().Endpoints(testNamespaceName).Get(context.TODO(), testEndpointName, metav1.GetOptions{}) framework.ExpectError(err, "should not be able to fetch Endpoint") }) <ins> ginkgo.It("should complete a service status lifecycle", func() { ns := f.Namespace.Name svcResource := schema.GroupVersionResource{Group: "", Version: "v1", Resource: "services"} svcClient := f.ClientSet.CoreV1().Services(ns) testSvcName := "test-service-" + utilrand.String(5) testSvcLabels := map[string]string{"test-service-static": "true"} testSvcLabelsFlat := "test-service-static=true" w := &cache.ListWatch{ WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { options.LabelSelector = testSvcLabelsFlat return cs.CoreV1().Services(ns).Watch(context.TODO(), options) }, } svcList, err := cs.CoreV1().Services("").List(context.TODO(), metav1.ListOptions{LabelSelector: testSvcLabelsFlat}) framework.ExpectNoError(err, "failed to list Services") ginkgo.By("creating a Service") testService := v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:   testSvcName, Labels: testSvcLabels, }, Spec: v1.ServiceSpec{ Type: "ClusterIP", Ports: []v1.ServicePort{{ Name:       "http", Protocol:   v1.ProtocolTCP, Port:       int32(80), TargetPort: intstr.FromInt(80), }}, }, } _, err = cs.CoreV1().Services(ns).Create(context.TODO(), &testService, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create Service") ginkgo.By("watching for the Service to be added") ctx, cancel := context.WithTimeout(context.Background(), svcReadyTimeout) defer cancel() _, err = watchtools.Until(ctx, svcList.ResourceVersion, w, func(event watch.Event) (bool, error) { if svc, ok := event.Object.(*v1.Service); ok { found := svc.ObjectMeta.Name == testService.ObjectMeta.Name && svc.ObjectMeta.Namespace == ns && svc.Labels["test-service-static"] == "true" if !found { framework.Logf("observed Service %v in namespace %v with labels: %v & ports %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Labels, svc.Spec.Ports) return false, nil } framework.Logf("Found Service %v in namespace %v with labels: %v & ports %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Labels, svc.Spec.Ports) return found, nil } framework.Logf("Observed event: %+v", event.Object) return false, nil }) framework.ExpectNoError(err, "Failed to locate Service %v in namespace %v", testService.ObjectMeta.Name, ns) framework.Logf("Service %s created", testSvcName) ginkgo.By("Getting /status") svcStatusUnstructured, err := f.DynamicClient.Resource(svcResource).Namespace(ns).Get(context.TODO(), testSvcName, metav1.GetOptions{}, "status") framework.ExpectNoError(err, "Failed to fetch ServiceStatus of Service %s in namespace %s", testSvcName, ns) svcStatusBytes, err := json.Marshal(svcStatusUnstructured) framework.ExpectNoError(err, "Failed to marshal unstructured response. %v", err) var svcStatus v1.Service err = json.Unmarshal(svcStatusBytes, &svcStatus) framework.ExpectNoError(err, "Failed to unmarshal JSON bytes to a Service object type") framework.Logf("Service %s has LoadBalancer: %v", testSvcName, svcStatus.Status.LoadBalancer) ginkgo.By("patching the ServiceStatus") lbStatus := v1.LoadBalancerStatus{ Ingress: []v1.LoadBalancerIngress{{IP: "203.0.113.1"}}, } lbStatusJSON, err := json.Marshal(lbStatus) framework.ExpectNoError(err, "Failed to marshal JSON. %v", err) _, err = svcClient.Patch(context.TODO(), testSvcName, types.MergePatchType, []byte(`{"metadata":{"annotations":{"patchedstatus":"true"}},"status":{"loadBalancer":`+string(lbStatusJSON)+`}}`), metav1.PatchOptions{}, "status") framework.ExpectNoError(err, "Could not patch service status", err) ginkgo.By("watching for the Service to be patched") ctx, cancel = context.WithTimeout(context.Background(), svcReadyTimeout) defer cancel() _, err = watchtools.Until(ctx, svcList.ResourceVersion, w, func(event watch.Event) (bool, error) { if svc, ok := event.Object.(*v1.Service); ok { found := svc.ObjectMeta.Name == testService.ObjectMeta.Name && svc.ObjectMeta.Namespace == ns && svc.Annotations["patchedstatus"] == "true" if !found { framework.Logf("observed Service %v in namespace %v with annotations: %v & LoadBalancer: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Annotations, svc.Status.LoadBalancer) return false, nil } framework.Logf("Found Service %v in namespace %v with annotations: %v & LoadBalancer: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Annotations, svc.Status.LoadBalancer) return found, nil } framework.Logf("Observed event: %+v", event.Object) return false, nil }) framework.ExpectNoError(err, "failed to locate Service %v in namespace %v", testService.ObjectMeta.Name, ns) framework.Logf("Service %s has service status patched", testSvcName) ginkgo.By("updating the ServiceStatus") var statusToUpdate, updatedStatus *v1.Service err = retry.RetryOnConflict(retry.DefaultRetry, func() error { statusToUpdate, err = svcClient.Get(context.TODO(), testSvcName, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to retrieve service %s", testSvcName) statusToUpdate.Status.Conditions = append(statusToUpdate.Status.Conditions, metav1.Condition{ Type:    "StatusUpdate", Status:  metav1.ConditionTrue, Reason:  "E2E", Message: "Set from e2e test", }) updatedStatus, err = svcClient.UpdateStatus(context.TODO(), statusToUpdate, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "nn Failed to UpdateStatus. %vnn", err) framework.Logf("updatedStatus.Conditions: %#v", updatedStatus.Status.Conditions) ginkgo.By("watching for the Service to be updated") ctx, cancel = context.WithTimeout(context.Background(), svcReadyTimeout) defer cancel() _, err = watchtools.Until(ctx, svcList.ResourceVersion, w, func(event watch.Event) (bool, error) { if svc, ok := event.Object.(*v1.Service); ok { found := svc.ObjectMeta.Name == testService.ObjectMeta.Name && svc.ObjectMeta.Namespace == ns && svc.Annotations["patchedstatus"] == "true" if !found { framework.Logf("Observed Service %v in namespace %v with annotations: %v & Conditions: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Annotations, svc.Status.LoadBalancer) return false, nil } for _, cond := range svc.Status.Conditions { if cond.Type == "StatusUpdate" && cond.Reason == "E2E" && cond.Message == "Set from e2e test" { framework.Logf("Found Service %v in namespace %v with annotations: %v & Conditions: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Annotations, svc.Status.Conditions) return found, nil } else { framework.Logf("Observed Service %v in namespace %v with annotations: %v & Conditions: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Annotations, svc.Status.LoadBalancer) return false, nil } } } framework.Logf("Observed event: %+v", event.Object) return false, nil }) framework.ExpectNoError(err, "failed to locate Service %v in namespace %v", testService.ObjectMeta.Name, ns) framework.Logf("Service %s has service status updated", testSvcName) ginkgo.By("patching the service") servicePatchPayload, err := json.Marshal(v1.Service{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{ "test-service": "patched", }, }, }) _, err = svcClient.Patch(context.TODO(), testSvcName, types.StrategicMergePatchType, []byte(servicePatchPayload), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch service. %v", err) ginkgo.By("watching for the Service to be patched") ctx, cancel = context.WithTimeout(context.Background(), svcReadyTimeout) defer cancel() _, err = watchtools.Until(ctx, svcList.ResourceVersion, w, func(event watch.Event) (bool, error) { if svc, ok := event.Object.(*v1.Service); ok { found := svc.ObjectMeta.Name == testService.ObjectMeta.Name && svc.ObjectMeta.Namespace == ns && svc.Labels["test-service"] == "patched" if !found { framework.Logf("observed Service %v in namespace %v with labels: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Labels) return false, nil } framework.Logf("Found Service %v in namespace %v with labels: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Labels) return found, nil } framework.Logf("Observed event: %+v", event.Object) return false, nil }) framework.ExpectNoError(err, "failed to locate Service %v in namespace %v", testService.ObjectMeta.Name, ns) framework.Logf("Service %s patched", testSvcName) ginkgo.By("deleting the service") err = cs.CoreV1().Services(ns).Delete(context.TODO(), testSvcName, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete the Service. %v", err) ginkgo.By("watching for the Service to be deleted") ctx, cancel = context.WithTimeout(context.Background(), svcReadyTimeout) defer cancel() _, err = watchtools.Until(ctx, svcList.ResourceVersion, w, func(event watch.Event) (bool, error) { switch event.Type { case watch.Deleted: if svc, ok := event.Object.(*v1.Service); ok { found := svc.ObjectMeta.Name == testService.ObjectMeta.Name && svc.ObjectMeta.Namespace == ns && svc.Labels["test-service-static"] == "true" if !found { framework.Logf("observed Service %v in namespace %v with labels: %v & annotations: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Labels, svc.Annotations) return false, nil } framework.Logf("Found Service %v in namespace %v with labels: %v & annotations: %v", svc.ObjectMeta.Name, svc.ObjectMeta.Namespace, svc.Labels, svc.Annotations) return found, nil } default: framework.Logf("Observed event: %+v", event.Type) } return false, nil }) framework.ExpectNoError(err, "failed to delete Service %v in namespace %v", testService.ObjectMeta.Name, ns) framework.Logf("Service %s deleted", testSvcName) }) </ins> }) // execAffinityTestForSessionAffinityTimeout is a helper function that wrap the logic of

Using the following code before the patch request Gives this response Then after running the patch ServiceStatus Then running The response shows the patched changes.
During the Conformance Meeting on 12th of January 2021, reviewed approach with Clayton, and we will start on the test PR this week!
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
/triage accepted


Question: fi sudo -E "${GO_OUT}/hyperkube" kubelet ${priv_arg} <del> --enable-cri=false  </del> <ins> --enable-cri="${ENABLE_CRI}"  </ins> --v=${LOG_LEVEL}  --chaos-chance="${CHAOS_CHANCE}"  --container-runtime="${CONTAINER_RUNTIME}"

When setting for other container runtime, won't start kubelet correctly because cri is disabled : Is there any reason of disabling cri at ? cc
no, that wasn't supposed to go in. I was testing something and it slipped in accidentally. I'll open a PR to fix.
Ah I see your PR for it. Thanks!


Question: "/readyz/shutdown", } checkExpectedPathsAtRoot(server.URL, expectedPaths, t) <ins> // wait for health (max-in-flight-filter is initialized asynchronously, can take a few milliseconds to initialize) if err := wait.PollImmediate(100*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) { // healthz checks are installed in PrepareRun resp, err := http.Get(server.URL + "/healthz?exclude=wrapping-health&exclude=delegate-health") if err != nil { t.Fatal(err) } data, _ := ioutil.ReadAll(resp.Body) if http.StatusOK != resp.StatusCode { t.Logf("got %d", resp.StatusCode) t.Log(string(data)) return false, nil } return true, nil }); err != nil { t.Fatal(err) } </ins> checkPath(server.URL+"/healthz", http.StatusInternalServerError, `[+]ping ok [+]log ok [-]wrapping-health failed: reason withheld

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated [Will update shortly] links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Which jobs are flaking: bazel-test-master Which test(s) are flaking: gotest: Testgrid link: Reason for failure: links to appreciatedlinks to specific failures in spyglass appreciated /sig api-machinary Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/assign /triage accepted
Flaked twice ~2 weeks ago but now looking good: Since it's been relatively quiet, I'm going to go ahead and close this for now. /close
Closing this issue. <detailsIn response to : Flaked twice ~2 weeks ago but now looking good: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign I meet this test failure in And I can reproduce it with local stress testing. The reason is the same with
Reopened this issue. <detailsIn response to : /reopen /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <ins> func verifyContainerStatuses(statuses []api.ContainerStatus, state, lastTerminationState map[string]api.ContainerState) error { for _, s := range statuses { if !reflect.DeepEqual(s.State, state[s.Name]) { return fmt.Errorf("unexpected state: %s", diff.ObjectDiff(state[s.Name], s.State)) } if !reflect.DeepEqual(s.LastTerminationState, lastTerminationState[s.Name]) { return fmt.Errorf("unexpected last termination state %s", diff.ObjectDiff( lastTerminationState[s.Name], s.LastTerminationState)) } } return nil } // Test generateAPIPodStatus with different reason cache and old api pod status. </ins> func TestGenerateAPIPodStatusWithReasonCache(t *testing.T) { // The following waiting reason and message  are generated in convertStatusToAPIStatus() startWaitingReason := "ContainerCreating" <ins> initWaitingReason := "PodInitializing" </ins> testTimestamp := time.Unix(123456789, 987654321) testErrorReason := fmt.Errorf("test-error") emptyContainerID := (&kubecontainer.ContainerID{}).String() <del> verifyStatus := func(t *testing.T, c int, podStatus api.PodStatus, state, lastTerminationState map[string]api.ContainerState) { statuses := podStatus.ContainerStatuses for _, s := range statuses { if !reflect.DeepEqual(s.State, state[s.Name]) { t.Errorf("case #%d, unexpected state: %s", c, diff.ObjectDiff(state[s.Name], s.State)) } if !reflect.DeepEqual(s.LastTerminationState, lastTerminationState[s.Name]) { t.Errorf("case #%d, unexpected last termination state %s", c, diff.ObjectDiff( lastTerminationState[s.Name], s.LastTerminationState)) } } } </del> testKubelet := newTestKubelet(t) kubelet := testKubelet.kubelet pod := podWithUidNameNs("12345678", "foo", "new")

is this a Node bug? cc as next oncall to follow up on this.
This is a node bug due to init containers
I may need help debugging - I see:
So I think the problem is in the error case - we're not properly recording that the start failed, and the next time we come in we somehow think the first container succeeded. I'll add some debugging to see if we can't recreate it.
Let's discuss on instead.


Question: } } <ins> func verifyContainerStatuses(statuses []api.ContainerStatus, state, lastTerminationState map[string]api.ContainerState) error { for _, s := range statuses { if !reflect.DeepEqual(s.State, state[s.Name]) { return fmt.Errorf("unexpected state: %s", diff.ObjectDiff(state[s.Name], s.State)) } if !reflect.DeepEqual(s.LastTerminationState, lastTerminationState[s.Name]) { return fmt.Errorf("unexpected last termination state %s", diff.ObjectDiff( lastTerminationState[s.Name], s.LastTerminationState)) } } return nil } // Test generateAPIPodStatus with different reason cache and old api pod status. </ins> func TestGenerateAPIPodStatusWithReasonCache(t *testing.T) { // The following waiting reason and message  are generated in convertStatusToAPIStatus() startWaitingReason := "ContainerCreating" <ins> initWaitingReason := "PodInitializing" </ins> testTimestamp := time.Unix(123456789, 987654321) testErrorReason := fmt.Errorf("test-error") emptyContainerID := (&kubecontainer.ContainerID{}).String() <del> verifyStatus := func(t *testing.T, c int, podStatus api.PodStatus, state, lastTerminationState map[string]api.ContainerState) { statuses := podStatus.ContainerStatuses for _, s := range statuses { if !reflect.DeepEqual(s.State, state[s.Name]) { t.Errorf("case #%d, unexpected state: %s", c, diff.ObjectDiff(state[s.Name], s.State)) } if !reflect.DeepEqual(s.LastTerminationState, lastTerminationState[s.Name]) { t.Errorf("case #%d, unexpected last termination state %s", c, diff.ObjectDiff( lastTerminationState[s.Name], s.LastTerminationState)) } } } </del> testKubelet := newTestKubelet(t) kubelet := testKubelet.kubelet pod := podWithUidNameNs("12345678", "foo", "new")

Kubelet issues a docker inspect call after creating a container, but for very short lived containers there is a window in which they could exit before being inspected in the docker manager. If the container PID is 0, we should check the ExitCode and other relevant fields and handle them appropriately. [1] This is the suspected cause of several flakes: /cc
looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: I'd expect the second one to not have run in this case? Node logs:
Init containers have an implicit assumption that container statuses include results of previous SyncPod invocations. If that's not the case we may need to fix that (or at least mitigate it). On May 18, 2016, at 8:28 PM, Prashanth B wrote: looking at the logs 2 weird things: race described second container ran, which suggests inconsistent reporting of initialized status of init1 and choosing init2 as the next container ( vs ). I think 1 got it right and 2 got it wrong. From logs: May 18 15:58:33.380: INFO: pod-init--1d4b-11e6-80a4- started at 2016-05-18T15:58:04-07:00 (2+1 container statuses recorded) May 18 15:58:33.380: INFO: Init container init1 ready: false, restart count 0 May 18 15:58:33.380: INFO: Init container init2 ready: true, restart count 0 I'd expect the second one to not have run in this case? Node logs: — You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub
This should be true. Is there any evidence that suggests the opposite? only shows conflicting view of whether init1 has run.
If the container exits already and it's not an infra container, kubelet should not error out. Although even if it does, I failed to see how this could affect the readiness of the init container without checking the code further.


Question: } } <ins> func verifyContainerStatuses(statuses []api.ContainerStatus, state, lastTerminationState map[string]api.ContainerState) error { for _, s := range statuses { if !reflect.DeepEqual(s.State, state[s.Name]) { return fmt.Errorf("unexpected state: %s", diff.ObjectDiff(state[s.Name], s.State)) } if !reflect.DeepEqual(s.LastTerminationState, lastTerminationState[s.Name]) { return fmt.Errorf("unexpected last termination state %s", diff.ObjectDiff( lastTerminationState[s.Name], s.LastTerminationState)) } } return nil } // Test generateAPIPodStatus with different reason cache and old api pod status. </ins> func TestGenerateAPIPodStatusWithReasonCache(t *testing.T) { // The following waiting reason and message  are generated in convertStatusToAPIStatus() startWaitingReason := "ContainerCreating" <ins> initWaitingReason := "PodInitializing" </ins> testTimestamp := time.Unix(123456789, 987654321) testErrorReason := fmt.Errorf("test-error") emptyContainerID := (&kubecontainer.ContainerID{}).String() <del> verifyStatus := func(t *testing.T, c int, podStatus api.PodStatus, state, lastTerminationState map[string]api.ContainerState) { statuses := podStatus.ContainerStatuses for _, s := range statuses { if !reflect.DeepEqual(s.State, state[s.Name]) { t.Errorf("case #%d, unexpected state: %s", c, diff.ObjectDiff(state[s.Name], s.State)) } if !reflect.DeepEqual(s.LastTerminationState, lastTerminationState[s.Name]) { t.Errorf("case #%d, unexpected last termination state %s", c, diff.ObjectDiff( lastTerminationState[s.Name], s.LastTerminationState)) } } } </del> testKubelet := newTestKubelet(t) kubelet := testKubelet.kubelet pod := podWithUidNameNs("12345678", "foo", "new")

Each syncPod invocation, we inspect the list of init container in order looking for lastTerminationState. There could be a bug in that logic, or if we didn't get the full status, we might make a bad decision. When determining which container to run next, we always look for the last init container in the chain that has success, then run the next container (unless the pod is RestartNever). , Yu-Ju Hong wrote:
Saw this flake again. How do we verify that container status is not lost between syn pods?
... between calls to syncPods
The second sync clearly saw the init1 container, and decided to launch init2. See "InitContainersToKeep" in the message below.
containers to keep can include failed containers. , Yu-Ju Hong wrote:
Discussed on slack. In summary, the problem has to do with how we convert the kubecontainer.PodStatus to the api.PodStatus. If a container is terminated, the function makes the decisions: the pod restart policy is "Always", and there is a "failure reason" recorded for the given container, this means the pod worker tried to (re)start the container but failed. Set the state to "Waiting" and populate the reason. , the container state is simply "terminated". Init containers don't obey the restart policy like regular containers, and the logic will need to be amended to work with them. is working on a PR. In addition to that, kubelet should not error out if a user container exited before it could apply the oom score (which is best effort in this case). fixes that.
Ahh, thanks for digging into the issue. This is the exact the concern I had when I reviewed the initial implementation of init container, which blocks pr for a while. I even asked to help me verify this case, unfortunately we didn't reproduce the issue back then. :-(
I wish I'd seen the flakes in the runs prior to merge.
We didn't notice this until we started running the tests against docker v1.11.1, which is much faster and hit the race condition differently.
I suppose that's a good thing...
It is hard to reproduce this.


Question: } } <ins> func verifyContainerStatuses(statuses []api.ContainerStatus, state, lastTerminationState map[string]api.ContainerState) error { for _, s := range statuses { if !reflect.DeepEqual(s.State, state[s.Name]) { return fmt.Errorf("unexpected state: %s", diff.ObjectDiff(state[s.Name], s.State)) } if !reflect.DeepEqual(s.LastTerminationState, lastTerminationState[s.Name]) { return fmt.Errorf("unexpected last termination state %s", diff.ObjectDiff( lastTerminationState[s.Name], s.LastTerminationState)) } } return nil } // Test generateAPIPodStatus with different reason cache and old api pod status. </ins> func TestGenerateAPIPodStatusWithReasonCache(t *testing.T) { // The following waiting reason and message  are generated in convertStatusToAPIStatus() startWaitingReason := "ContainerCreating" <ins> initWaitingReason := "PodInitializing" </ins> testTimestamp := time.Unix(123456789, 987654321) testErrorReason := fmt.Errorf("test-error") emptyContainerID := (&kubecontainer.ContainerID{}).String() <del> verifyStatus := func(t *testing.T, c int, podStatus api.PodStatus, state, lastTerminationState map[string]api.ContainerState) { statuses := podStatus.ContainerStatuses for _, s := range statuses { if !reflect.DeepEqual(s.State, state[s.Name]) { t.Errorf("case #%d, unexpected state: %s", c, diff.ObjectDiff(state[s.Name], s.State)) } if !reflect.DeepEqual(s.LastTerminationState, lastTerminationState[s.Name]) { t.Errorf("case #%d, unexpected last termination state %s", c, diff.ObjectDiff( lastTerminationState[s.Name], s.LastTerminationState)) } } } </del> testKubelet := newTestKubelet(t) kubelet := testKubelet.kubelet pod := podWithUidNameNs("12345678", "foo", "new")

The extra check of last terminated container in your initial code makes me nervous, but we couldn't reproduce it due to that small race windows until we have a faster docker running on the node. :-)
I got "Pods should invoke init containers on a RestartAlways pod" flake (referenced from ): Logs:
All the fixes went in. Please reopen this issue if you see any more flakes.


Question: "net" "os" "strconv" <ins> "time" </ins> "github.com/golang/glog" "github.com/spf13/pflag"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: kube-cm kube-scheduler does not exit when kube-apiserver suspended What you expected to happen: kube-cm kube-scheduler should exit after LeaderElector stops leading How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig api-machinery /kind bug
/assign
/cc


Question: } if svc := h.ClientConfig.Service; svc != nil { <del> serverName := svc.Name + "." + svc.Namespace + ".svc" restConfig, err := cm.authInfoResolver.ClientConfigFor(serverName) </del> <ins> restConfig, err := cm.authInfoResolver.ClientConfigForService(svc.Name, svc.Namespace) </ins> if err != nil { return nil, err } cfg := rest.CopyConfig(restConfig) <ins> serverName := svc.Name + "." + svc.Namespace + ".svc" </ins> host := serverName + ":443" cfg.Host = "https://" + host if svc.Path != nil {

We are getting the following error when the apiserver connects the webhook on localhost (configured via URL). We should only use the SSL tunnel for the connections to nodes when the webhooks are running as services. /sig api-machinery /kind bug /assign
/close


Question: --pull  -t ${REGISTRY}/conformance-${ARCH}:${VERSION}  --build-arg BASEIMAGE=$(BASEIMAGE)  <ins> --build-arg RUNNERIMAGE=$(RUNNERIMAGE)  </ins> ${TEMP_DIR} rm -rf "${TEMP_DIR}"

<!-- Please only use this template for submitting enhancement requests --What would you like to be modified: image to be based on anything other than Docker Hub sourced Why is this needed: With Docker Hub rate-limit policy in-place, we probably should consider removing this dependency to stay in sync with other base images in Kubernetes project. Folks building their own binaries (i.e. with ) might run into this as well. Notes: Prior discussion happened on An idea from is to use the should probably be renamed to avoid confusion with k/release (h/t Currently, is set to , so migrating to may not be a drop-in move However, , so maybe we don't have to start from scratch /sig testing /area conformance /sig release (cc'ing because builds conformance image, though my quick search seem to show there isn't any build pipelines referencing )
/assign
do we still want to do this? I read somewhere that the images built in Google Cloud are being mirrored from Docker Hub, right? This would mean that the API rate limiting will not hit us any more.
Womp yeah I think DockerHub rate limit is not really the big motivator at this point. However, I think I still want to consider using distroless / go-runner to build and produce conformance image to reduce the noise from CVE scanning done in downstream places. What do you think?
Yeah I think this seems a valuable addition. Are you planning to work on this issue? :)
I will, though probably won't start until next week. If you have ideas and would like to start early feel free to go for it! Otherwise, I'd be happy to work on and deliver this.
Thank you Wilson! Feel free to ping again if you need any assistance. /assign
/triage accepted
one thing I need a little help with is to find a reference for using go-runner image that's used everywhere -- the one that I've found is inside conformance directory which seems to be different from what everyone else is using
We maintain a go-runner in k/release:
oops, I forgot to get back on this last week -- I meant to say that I need a reference for image which I can use as , so something like on Dockerfile. Is that a possible thing to do?


Question: --pull  -t ${REGISTRY}/conformance-${ARCH}:${VERSION}  --build-arg BASEIMAGE=$(BASEIMAGE)  <ins> --build-arg RUNNERIMAGE=$(RUNNERIMAGE)  </ins> ${TEMP_DIR} rm -rf "${TEMP_DIR}"

after searching around, I found this -- going to try and use and see how it works :smile:
Started a PR -- plenty of things I'd like to be discussed but we should probably have the discussions there
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale there is work in progress


Question: // TODO(#58034): This is not a static file, so it's not quite as straightforward as --google-json-key. // We need to figure out how ACR users can dynamically provide pull credentials before we can deprecate this. pflagRegister(global, local, "azure-container-registry-config") <ins> local.MarkDeprecated("azure-container-registry-config", "Use --image-credential-provider-config and --image-credential-provider-bin-dir to setup acr credential provider instead. Will be removed in a future release.") </ins> }

WRT , container registry credentials should be included in a docker file instead. We deprecated for this reason in , but is more tricky, because the file it references is not static. We need to figure out what the alternative is for dynamically providing pull credentials to nodes. /cc
I think we need to enable Managed Service Identity (MSI) for all instances before deprecating . To do this, we also need to grant VM access to Resource Group in Azure IAM (e.g. config Azure AD in acs-engine). WDYT?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I think this is still active. Right, /remove-lifecycle stale
Thanks, this is still active
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
-- What's the status of this?
Sorry, no progress yet. Any ideas on this?
-- pinging again on this one.
We're not making any progress yet in this release. Still seeking the possible solutions for non-MSI cases, secret is one option for this. /milestone v1.15 Related and
None one is working on this. Let's move this out of 1.15 /milestone v1.16
thanks. This missed v1.15 feature timeline. LGTM of moving to v1.16.
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 5 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?


Question: // TODO(#58034): This is not a static file, so it's not quite as straightforward as --google-json-key. // We need to figure out how ACR users can dynamically provide pull credentials before we can deprecate this. pflagRegister(global, local, "azure-container-registry-config") <ins> local.MarkDeprecated("azure-container-registry-config", "Use --image-credential-provider-config and --image-credential-provider-bin-dir to setup acr credential provider instead. Will be removed in a future release.") </ins> }

/milestone clear is preparing a KEP for this, but it's still not ready yet.
Will follow this KEP for implementation
This should be addressed by /milestone v1.18
Hello ! Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March, which is a few weeks from now. Is this issue still intended for 1.18? Thanks in advance!
C/O Code freeze for release 1.18 is starting in 5 March (less than 1 week from now) and we want to ensure that each issue has a chance to be finished on time. Is this still on track for this deadline?
/milestone v1.19 The work on the KEP hasn't started yet, so moving this to 1.19
Hello from Bug Triage team! Wanted to follow up on this issue with a friendly reminder that the code freeze for 1.19 is starting June 25th (about 4 weeks from now). As this issue is tagged for 1.19, is it still planned for this release?
/milestone v1.20
Hey Bug Triage here. Wanted to follow up on the status of this issue as we're approaching code freeze on 12.11.2020. This issue is tagged for 1.20, is it still planned for this release?
Hi Bug Triage here, seeing that there is not much recent activity here, and we are pretty late into code freeze, I am going to remove the milestone label. /milestone clear
/triage accepted /priority backlog I think we would be able to remove the flag once external kubelet credential provider promoted to GA
/kind deprecation
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available .


Question: // TODO(#58034): This is not a static file, so it's not quite as straightforward as --google-json-key. // We need to figure out how ACR users can dynamically provide pull credentials before we can deprecate this. pflagRegister(global, local, "azure-container-registry-config") <ins> local.MarkDeprecated("azure-container-registry-config", "Use --image-credential-provider-config and --image-credential-provider-bin-dir to setup acr credential provider instead. Will be removed in a future release.") </ins> }

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


When the master dials nodes, it is mutating the tlsConfig held by the master kubelet client, to add a server name. That means if two different nodes are dialed from a master started with , x509 verification will fail for the second one. Introduced in
this is a regression from 1.3 and breaks pod logs/exec/attach and node proxying on installations that use
We probably need to review that PR and look to see if any other deliberate copies were flagged incorrectly by govet.
Installations that depend on secure trust from externally provided kubelet server certs are the highest security rings. If we don't fix this we need to tell anyone using this flag not to upgrade until we release 1.4.1.
the rest of the changes in the govet cleanup PR looked fine to me. fix for this issue is in
Since we are telling folks they have to upgrade to kubectl 1.4.0 for rollingupdate to work it would definitely be a confusing message to say other folks should not upgrade to 1.4.0 at all.
If the fix is merged into master by this afternoon, lets discuss whether or not we should cherrypick it - e.g. risks involved, how we can manually qa to verify nothing else was broken, etc.
Risk involved: If you are using a secure cluster (defined as ensuring no one can MITM your nodes and masters) and you upgrade to 1.4.0 and it does not include this fix - you are broken.
What is the likelihood of interactions with other code? Is it possible this fix could introduce a breaking change to something else? Has the fix been manually verified? Are we confident that this will fully fixes the issue (vs other edge cases exist for secure clusters that this doesn't catch). Ideally I would like to see this cherrypicked into 1.4 this evening and then cut a new beta that includes it.
None. The single line that is currently broken was changed. The rest is a new function (with no side effects), and tests. No, it restores the original intent of the TLS config copy, and tests cover the cases we are using this with (secure, insecure, custom dialer) Yes. I swept for the same issue elsewhere and didn't find any issues. We test with a fully secured master-kubelet setup in Openshift and got a clean test run with this fix applied.
SGTM. Thanks for fixing this.


Question: "k8s.io/klog" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/wait" )

What would you like to be : Add support to the kube-apiserver so that we can disable health-checks via query params. Why is this needed: Currently, the healthz endpoint is inflexible in that any and all health checks are currently executed when the endpoint is hit. Since the kube-apiserver only currently supports a liveness check, this means all persistent health check failures will eventually result in a kubelet driven restart. However, this may not be desirable behavior if you have additional monitoring services layered on top of your control plane. For instance, take the etcd failure from the perspective of the kube-apiserver. Currently, persistent failures during the etcd health check from the api-server healthz endpoint will cause the kube-apiserver to be restarted. This sometimes helps (in the case of a faulty etcd-client connection) but also can be detrimental (the kube-apiserver restarts until the etcd cluster is functioning, then the kube-apiserver bombards the etcd cluster with requests to update local caches, saturating the etcd cluster with requests, causing it to timeout on its own liveness probes and trigger a kubelet-driven restart). Let's say that we could now disable etcd from our liveness health check. Then if we had an external monitoring service, we could ask independently ask etcd if it was healthy and we could also ask the kube-apiserver if it thought etcd was unhealthy. Based on the combination of those answers, we could enable more intelligent responses to certain situation, like restarting the kube-apiserver when it thought etcd was unhealthy but etcd was actually reporting as healthy but not doing so when etcd was actually unhealthy. This feature is backwards-compatible and opt-in. There would be no impact to existing behavior unless these query params are actually used. /sig api-machinery /kind feature
/cc
SGTM
/assign


Question: jig := framework.NewServiceTestJig(cs, serviceName) nodes := jig.GetNodes(framework.MaxNodesForEndpointsTests) <del> svc := jig.CreateOnlyLocalLoadBalancerService(namespace, serviceName, loadBalancerCreateTimeout, false, nil) </del> <ins> svc := jig.CreateOnlyLocalLoadBalancerService(namespace, serviceName, loadBalancerCreateTimeout, false, func(svc *v1.Service) { // Change service port to avoid collision with opened hostPorts // in other tests that run in parallel. if len(svc.Spec.Ports) != 0 { svc.Spec.Ports[0].TargetPort = intstr.FromInt(int(svc.Spec.Ports[0].Port)) svc.Spec.Ports[0].Port = 8081 } }) </ins> serviceLBNames = append(serviceLBNames, cloudprovider.GetLoadBalancerName(svc)) defer func() { jig.ChangeServiceType(svc.Namespace, svc.Name, v1.ServiceTypeClusterIP, loadBalancerCreateTimeout)

https://k8s- It's quite flaky.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/assign
/sig networking
There are two failures in gci-gce-slow, so it's not specific to gke. https://k8s- Also in gce-slow: https://k8s-
/sig network
This is strange, seems like this test is trying to reach the test-netexec pod through servicePublicIP:serivcePort, but the request hit the gce default backend instead. cc
There is another type of failure: Seems to be the port collision between test service and fluentd daemonset, which changed to use few days ago ().
cc
Checked the kube-proxy logs and found this is not the case. Port was not allocated (although that could happen in theory) to any service during test:
This turned out to be an interesting corner case. It happens at the beginning after we set up the GCE network LoadBalancer for OnlyLocal service --- when the LB still forwarding traffic onto nodes that don't have endpoints on. For more context, when network LB forward traffic to nodes, packet arrives node with below src:dst: If no endpoints are on the node, and there happens to have a pod using hostPort with the same port number, packet will be DNAT to that pod IP instead. These are observations and I haven't traced down to the exact iptables entries. I'm changing service port in test to avoid collision for now. cc


Question: {"name" : "ServiceSpreadingPriority", "weight" : 1}, {"name" : "EqualPriority", "weight" : 1} ], <del> "extenders":[ </del> <ins> "extenders" : [ </ins> { "urlPrefix": "http://127.0.0.1:12346/scheduler", "apiVersion": "v1beta1",

( noticed this) HardPodAffinitySymmetricWeight should be in the , not in KubeSchedulerConfiguration If/when fixing this, we need to be careful to do it in a backward-compatilble way.
Do you mean ? It seems scheduler policy config's args can not passed to FitPredicates; it need to update related interface. For the : mark to be deprecated and set scheduler policy config to HardPodAffinitySymmetricWeight if policy config is empty.
Are you working on this? if no, I can take it.
, nop; please go ahead.
/ / / , When reviewing the PR, there's one question on : the default value of is 1, which means the weight (1) will always add to priorities if term matched (), does it our expectation? And the user has to set it to 0 to disable it. IMO, the default value of should be 0.
The value 1 (instead of 0) was chosen so that if you say A must run with B, there is an automatic preference that B should run with A. This seems like reasonable behavior.
Thanks very much; reasonable case to me. But it seems we can not disable it: if set to 0, the default funcs will set it to default (1); similar result to un-set. Anyway, no user complain this now :).


Question: var _ = framework.KubeDescribe("Pods", func() { f := framework.NewDefaultFramework("pods") var podClient *framework.PodClient <ins> var dc dynamic.Interface </ins> ginkgo.BeforeEach(func() { podClient = f.PodClient() <ins> dc = f.DynamicClient </ins> }) /*

[x] Original approval issue: - Reverted due to flakes [x] Two weeks soak end date: 20 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Pod endpoints which are untested. a Pod with a static label the Pod with a new Label and updated data the Pod to ensure it's patched the Pod's status the Pod's status Ready condition to False the Pod's status to check if Ready is False all Pods in all Namespaces find the Pod(1) ensure that the Pod is found and is patched Namespaced Pod(1) via a Collection with a LabelSelector the Pod to check that it's deleted Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/reopen Test PR Merge Promotion PR to be created next
Reopened this issue. <detailsIn response to : /reopen Test PR Merge Promotion PR to be created next Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
reverted by due to flaky test
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/reopen PR on test grid for 2 weeks before promotion to conformance
Reopened this issue. <detailsIn response to : /reopen PR on test grid for 2 weeks before promotion to conformance Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: // rules separately to systemd) we can safely skip entries that don't // have a corresponding path. if _, err := os.Stat(entry.Path); err != nil { <del> logrus.Debugf("skipping device %s for systemd: %s", entry.Path, err) continue </del> <ins> // Also check /sys/dev so that we don't depend on /dev/{block,char} // being populated. (/dev/{block,char} is populated by udev, which // isn't strictly required for systemd). Ironically, this happens most // easily when starting containerd within a runc created container // itself. // We don't bother with securejoin here because we create entry.Path // right above here, so we know it's safe. if _, err := os.Stat("/sys" + entry.Path); err != nil { logrus.Warnf("skipping device %s for systemd: %s", entry.Path, err) continue } </ins> } } deviceAllowList = append(deviceAllowList, entry)

opencontainers/runc has fixed three CVEs: Latest runc is pulled by k/k Check runc version in file No response n/a n/a <details</details<details</details<details</details<details</details
/assign /sig security architecture release
/triage accepted


Question: <del> This file has moved to [https://github.com/kubernetes/examples/blob/master/staging/elasticsearch/production_cluster/README.md](https://github.com/kubernetes/examples/blob/master/staging/elasticsearch/production_cluster/README.md) </del>

Presently, the elasticsearch example Kubernetes configurations target elasticsearch v1.7.1, which is quite old. As of this writing, the latest elasticsearch release is v5.6.2, with many deployment changes since then. Could the elasticsearch examples be updated to target the v5 series of elasticsearch? See for a more up to date reference configuration.
cc
hey it's been a long while since I last contributed to the docs. i can do it soon but i need pointers on how to properly do it.
hey, so we need to update the stuff under examples/elasticsearch/ *&repos=kubernetes So basically update the images, yaml files etc and test that everything works (and submit a PR)
/sig testing
Ah, so there's no documentation updates anymore. OK I'll take care of it, assign it to me.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
GitHub didn't allow me to assign the following users: me. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign me Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign pires
/remove-lifecycle stale
it seems that the example moved to I am now reading that repo guidelines and will update this there. I'll also open a PR that cleans this folder in the current repo and closes this issue.


Question: } } <ins> // ensure KUBE-MARK-DROP chain exist but do not change any rules for _, ch := range iptablesEnsureChains { if _, err := proxier.iptables.EnsureChain(ch.table, ch.chain); err != nil { klog.Errorf("Failed to ensure that %s chain %s exists: %v", ch.table, ch.chain, err) return } } </ins> // // Below this point we will not return until we try to write the iptables rules. //

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-e2e-gci-gce-ipvs Which test(s) are failing: [sig-network] Services should only allow access from service loadbalancer source ranges [Slow] Testgrid link: https://k8s- Reason for failure: Anything else we need to know: The test is failing consistently, however, there is also test that is failing a lot but not always, may it be related?
/sig-network /area ipvs /cc /help wanted
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
/remove-triage unresolved
Hi, , turnout this is because mark drop rule in be deleted, I raied a pr to fix this issue, PTAL. : )
/close as duplicate of
Closing this issue. <detailsIn response to : /close as duplicate of Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen this was affecting ipvs only before , let's keep it open until we verify is not failing anymore
Reopened this issue. <detailsIn response to : /reopen this was affecting ipvs only before , let's keep it open until we verify is not failing anymore Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Ok I tested this out with kind in ipvs-mode with metallb. In iptables mode I had zero failures but ipvs does not pass at all during the timeout window. This is likely due to


Question: }, { "ImportPath": "github.com/docker/libnetwork/ipvs", <del> "Comment": "v0.8.0-dev.2-908-gd5c82231", "Rev": "d5c822319097cc01cc9bd5ffedd74c7ce7c894f2" </del> <ins> "Rev": "ba46b928444931e6865d8618dc03622cac79aa6f" </ins> }, { "ImportPath": "github.com/docker/libtrust",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently, we flush ipvs services by deleting them one by one. That's because there was no flush API in libnetwork. Now, libnetwork supports flush API, I think it's time to rsync. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig network
/area kube-proxy
/assign