Question:
several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.

return newGidTable, nil } <del> //createEndpointService create an endpoint and service in provided namespace. func (plugin *glusterfsPlugin) createEndpointService(namespace string, epServiceName string, hostips []string, pvcname string) (endpoint *v1.Endpoints, service *v1.Service, err error) { addrlist := make([]v1.EndpointAddress, len(hostips)) for i, v := range hostips { addrlist[i].IP = v } endpoint = &v1.Endpoints{ ObjectMeta: metav1.ObjectMeta{ Namespace: namespace, Name:      epServiceName, Labels: map[string]string{ "gluster.kubernetes.io/provisioned-for-pvc": pvcname, }, }, Subsets: []v1.EndpointSubset{{ Addresses: addrlist, Ports:     []v1.EndpointPort{{Port: 1, Protocol: "TCP"}}, }}, } kubeClient := plugin.host.GetKubeClient() if kubeClient == nil { return nil, nil, fmt.Errorf("glusterfs: failed to get kube client when creating endpoint service") } _, err = kubeClient.Core().Endpoints(namespace).Create(endpoint) if err != nil && errors.IsAlreadyExists(err) { glog.V(1).Infof("glusterfs: endpoint [%s] already exist in namespace [%s]", endpoint, namespace) err = nil } if err != nil { glog.Errorf("glusterfs: failed to create endpoint: %v", err) return nil, nil, fmt.Errorf("error creating endpoint: %v", err) } service = &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      epServiceName, Namespace: namespace, Labels: map[string]string{ "gluster.kubernetes.io/provisioned-for-pvc": pvcname, }, }, Spec: v1.ServiceSpec{ Ports: []v1.ServicePort{ {Protocol: "TCP", Port: 1}}}} _, err = kubeClient.Core().Services(namespace).Create(service) if err != nil && errors.IsAlreadyExists(err) { glog.V(1).Infof("glusterfs: service [%s] already exist in namespace [%s]", service, namespace) err = nil } if err != nil { glog.Errorf("glusterfs: failed to create service: %v", err) return nil, nil, fmt.Errorf("error creating service: %v", err) } return endpoint, service, nil } // deleteEndpointService delete the endpoint and service from the provided namespace. func (plugin *glusterfsPlugin) deleteEndpointService(namespace string, epServiceName string) (err error) { kubeClient := plugin.host.GetKubeClient() if kubeClient == nil { return fmt.Errorf("glusterfs: failed to get kube client when deleting endpoint service") } err = kubeClient.Core().Services(namespace).Delete(epServiceName, nil) if err != nil { glog.Errorf("glusterfs: error deleting service %s/%s: %v", namespace, epServiceName, err) return fmt.Errorf("error deleting service %s/%s: %v", namespace, epServiceName, err) } glog.V(1).Infof("glusterfs: service/endpoint %s/%s deleted successfully", namespace, epServiceName) return nil } </del> func (d *glusterfsVolumeDeleter) getGid() (int, bool, error) { gidStr, ok := d.spec.Annotations[volumehelper.VolumeGidAnnotationKey]


Question:
several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.

dynamicEndpoint = pvSpec.Glusterfs.EndpointsName } glog.V(3).Infof("glusterfs: dynamic namespace and endpoint : [%v/%v]", dynamicNamespace, dynamicEndpoint) <del> err = d.plugin.deleteEndpointService(dynamicNamespace, dynamicEndpoint) </del> <ins> err = d.deleteEndpointService(dynamicNamespace, dynamicEndpoint) </ins> if err != nil { glog.Errorf("glusterfs: error when deleting endpoint/service :%v", err) } else {


Question:
several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.

gidStr := strconv.FormatInt(int64(gid), 10) pv.Annotations = map[string]string{ <del> volumehelper.VolumeGidAnnotationKey:        gidStr, volumehelper.VolumeDynamicallyCreatedByKey: heketiAnn, glusterTypeAnn:                             "file", "Description":                              glusterDescAnn, v1.MountOptionAnnotation:                   "auto_unmount", </del> <ins> volumehelper.VolumeGidAnnotationKey: gidStr, "kubernetes.io/createdby":           "heketi-dynamic-provisioner", "gluster.org/type":                  "file", "Description":                       "Gluster: Dynamically provisioned PV", v1.MountOptionAnnotation:            "auto_unmount", </ins> } pv.Spec.Capacity = v1.ResourceList{


Question:
several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.

return pv, nil } <ins> func (p *glusterfsVolumeProvisioner) GetClusterNodes(cli *gcli.Client, cluster string) (dynamicHostIps []string, err error) { clusterinfo, err := cli.ClusterInfo(cluster) if err != nil { glog.Errorf("glusterfs: failed to get cluster details: %v", err) return nil, fmt.Errorf("failed to get cluster details: %v", err) } // For the dynamically provisioned volume, we gather the list of node IPs // of the cluster on which provisioned volume belongs to, as there can be multiple // clusters. for _, node := range clusterinfo.Nodes { nodei, err := cli.NodeInfo(string(node)) if err != nil { glog.Errorf("glusterfs: failed to get hostip: %v", err) return nil, fmt.Errorf("failed to get hostip: %v", err) } ipaddr := dstrings.Join(nodei.NodeAddRequest.Hostnames.Storage, "") dynamicHostIps = append(dynamicHostIps, ipaddr) } glog.V(3).Infof("glusterfs: hostlist :%v", dynamicHostIps) if len(dynamicHostIps) == 0 { glog.Errorf("glusterfs: no hosts found: %v", err) return nil, fmt.Errorf("no hosts found: %v", err) } return dynamicHostIps, nil } </ins> func (p *glusterfsVolumeProvisioner) CreateVolume(gid int) (r *v1.GlusterfsVolumeSource, size int, err error) { var clusterIDs []string capacity := p.options.PVC.Spec.Resources.Requests[v1.ResourceName(v1.ResourceStorage)]


Question:
several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.

return nil, 0, fmt.Errorf("error creating volume %v", err) } glog.V(1).Infof("glusterfs: volume with size: %d and name: %s created", volume.Size, volume.Name) <del> dynamicHostIps, err := getClusterNodes(cli, volume.Cluster) </del> <ins> dynamicHostIps, err := p.GetClusterNodes(cli, volume.Cluster) </ins> if err != nil { glog.Errorf("glusterfs: error [%v] when getting cluster nodes for volume %s", err, volume) return nil, 0, fmt.Errorf("error [%v] when getting cluster nodes for volume %s", err, volume)


Question:
several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.

// of volume creation. epServiceName := dynamicEpSvcPrefix + p.options.PVC.Name epNamespace := p.options.PVC.Namespace <del> endpoint, service, err := p.plugin.createEndpointService(epNamespace, epServiceName, dynamicHostIps, p.options.PVC.Name) </del> <ins> endpoint, service, err := p.createEndpointService(epNamespace, epServiceName, dynamicHostIps, p.options.PVC.Name) </ins> if err != nil { glog.Errorf("glusterfs: failed to create endpoint/service: %v", err) deleteErr := cli.VolumeDelete(volume.Id)


Question:
several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.

}, sz, nil } <ins> func (p *glusterfsVolumeProvisioner) createEndpointService(namespace string, epServiceName string, hostips []string, pvcname string) (endpoint *v1.Endpoints, service *v1.Service, err error) { addrlist := make([]v1.EndpointAddress, len(hostips)) for i, v := range hostips { addrlist[i].IP = v } endpoint = &v1.Endpoints{ ObjectMeta: metav1.ObjectMeta{ Namespace: namespace, Name:      epServiceName, Labels: map[string]string{ "gluster.kubernetes.io/provisioned-for-pvc": pvcname, }, }, Subsets: []v1.EndpointSubset{{ Addresses: addrlist, Ports:     []v1.EndpointPort{{Port: 1, Protocol: "TCP"}}, }}, } kubeClient := p.plugin.host.GetKubeClient() if kubeClient == nil { return nil, nil, fmt.Errorf("glusterfs: failed to get kube client when creating endpoint service") } _, err = kubeClient.Core().Endpoints(namespace).Create(endpoint) if err != nil && errors.IsAlreadyExists(err) { glog.V(1).Infof("glusterfs: endpoint [%s] already exist in namespace [%s]", endpoint, namespace) err = nil } if err != nil { glog.Errorf("glusterfs: failed to create endpoint: %v", err) return nil, nil, fmt.Errorf("error creating endpoint: %v", err) } service = &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      epServiceName, Namespace: namespace, Labels: map[string]string{ "gluster.kubernetes.io/provisioned-for-pvc": pvcname, }, }, Spec: v1.ServiceSpec{ Ports: []v1.ServicePort{ {Protocol: "TCP", Port: 1}}}} _, err = kubeClient.Core().Services(namespace).Create(service) if err != nil && errors.IsAlreadyExists(err) { glog.V(1).Infof("glusterfs: service [%s] already exist in namespace [%s]", service, namespace) err = nil } if err != nil { glog.Errorf("glusterfs: failed to create service: %v", err) return nil, nil, fmt.Errorf("error creating service: %v", err) } return endpoint, service, nil } func (d *glusterfsVolumeDeleter) deleteEndpointService(namespace string, epServiceName string) (err error) { kubeClient := d.plugin.host.GetKubeClient() if kubeClient == nil { return fmt.Errorf("glusterfs: failed to get kube client when deleting endpoint service") } err = kubeClient.Core().Services(namespace).Delete(epServiceName, nil) if err != nil { glog.Errorf("glusterfs: error deleting service %s/%s: %v", namespace, epServiceName, err) return fmt.Errorf("error deleting service %s/%s: %v", namespace, epServiceName, err) } glog.V(1).Infof("glusterfs: service/endpoint %s/%s deleted successfully", namespace, epServiceName) return nil } </ins> // parseSecret finds a given Secret instance and reads user password from it. func parseSecret(namespace, secretName string, kubeClient clientset.Interface) (string, error) { secretMap, err := volutil.GetSecretForPV(namespace, secretName, glusterfsPluginName, kubeClient)


Question:
several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.

return &cfg, nil } <del> // getClusterNodes() returns the cluster nodes of a given cluster func getClusterNodes(cli *gcli.Client, cluster string) (dynamicHostIps []string, err error) { clusterinfo, err := cli.ClusterInfo(cluster) if err != nil { glog.Errorf("glusterfs: failed to get cluster details: %v", err) return nil, fmt.Errorf("failed to get cluster details: %v", err) } // For the dynamically provisioned volume, we gather the list of node IPs // of the cluster on which provisioned volume belongs to, as there can be multiple // clusters. for _, node := range clusterinfo.Nodes { nodei, err := cli.NodeInfo(string(node)) if err != nil { glog.Errorf("glusterfs: failed to get hostip: %v", err) return nil, fmt.Errorf("failed to get hostip: %v", err) } ipaddr := dstrings.Join(nodei.NodeAddRequest.Hostnames.Storage, "") dynamicHostIps = append(dynamicHostIps, ipaddr) } glog.V(3).Infof("glusterfs: hostlist :%v", dynamicHostIps) if len(dynamicHostIps) == 0 { glog.Errorf("glusterfs: no hosts found: %v", err) return nil, fmt.Errorf("no hosts found: %v", err) } return dynamicHostIps, nil } </del>


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): local-up- Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: KVM OS (e.g. from /etc/os-release): fedora 25 Kernel (e.g. ): uname -a Linux k8sbuild-master 4.11.3-200.fc25.x8664 SMP Thu May 25 19:03:07 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others: What happened: local-up- fails to create kube-dns and StorageClass. What you expected to happen: local-up- is successfully executed. How to reproduce it (as minimally and precisely as possible): 1) Clone kubernetes from master branch 2) run local-up- $ export ENABLEHOSTPATHPROVISIONER="true"; ./hack/local-up- If you already have kubernetes , try this 1) make cleanmeta 2) run local-up- $ export ENABLEHOSTPATHPROVISIONER="true"; ./hack/local-up- Anything else we need to know: I think this problem is timing issue because sleeping 3 seconds before startkubedns can avoid this problem.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig testing /sig storage /sig network
/sig apps
I met the same issue we should add a function to check if all api-versions are up

DNS_SERVER_IP=${KUBE_DNS_SERVER_IP:-10.0.0.10} DNS_DOMAIN=${KUBE_DNS_NAME:-"cluster.local"} KUBECTL=${KUBECTL:-cluster/kubectl.sh} <del> WAIT_FOR_URL_API_SERVER=${WAIT_FOR_URL_API_SERVER:-10} </del> <ins> WAIT_FOR_URL_API_SERVER=${WAIT_FOR_URL_API_SERVER:-20} </ins> ENABLE_DAEMON=${ENABLE_DAEMON:-false} HOSTNAME_OVERRIDE=${HOSTNAME_OVERRIDE:-"127.0.0.1"} CLOUD_PROVIDER=${CLOUD_PROVIDER:-""}


Question:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): local-up- Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: KVM OS (e.g. from /etc/os-release): fedora 25 Kernel (e.g. ): uname -a Linux k8sbuild-master 4.11.3-200.fc25.x8664 SMP Thu May 25 19:03:07 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others: What happened: local-up- fails to create kube-dns and StorageClass. What you expected to happen: local-up- is successfully executed. How to reproduce it (as minimally and precisely as possible): 1) Clone kubernetes from master branch 2) run local-up- $ export ENABLEHOSTPATHPROVISIONER="true"; ./hack/local-up- If you already have kubernetes , try this 1) make cleanmeta 2) run local-up- $ export ENABLEHOSTPATHPROVISIONER="true"; ./hack/local-up- Anything else we need to know: I think this problem is timing issue because sleeping 3 seconds before startkubedns can avoid this problem.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig testing /sig storage /sig network
/sig apps
I met the same issue we should add a function to check if all api-versions are up

# this uses the API port because if you don't have any authenticator, you can't seem to use the secure port at all. # this matches what happened with the combination in 1.4. # TODO change this conditionally based on whether API_PORT is on or off <del> kube::util::wait_for_url "http://${API_HOST_IP}:${API_SECURE_PORT}/healthz" "apiserver: " 1 ${WAIT_FOR_URL_API_SERVER}  </del> <ins> kube::util::wait_for_url "https://${API_HOST_IP}:${API_SECURE_PORT}/healthz" "apiserver: " 1 ${WAIT_FOR_URL_API_SERVER}  </ins> || { echo "check apiserver logs: ${APISERVER_LOG}" ; exit 1 ; } # Create kubeconfigs for all components, using client certs


Question: Every time watch is reset, event exporter exports an entry about it to Stackdriver. To make it less irritating, this has to be done once per container run Related image PR: /cc

apiVersion: apps/v1beta1 kind: Deployment metadata: <del> name: event-exporter-v0.1.0 </del> <ins> name: event-exporter-v0.1.4 </ins> namespace: kube-system labels: k8s-app: event-exporter


Question: Every time watch is reset, event exporter exports an entry about it to Stackdriver. To make it less irritating, this has to be done once per container run Related image PR: /cc

containers: # TODO: Add resources in 1.8 - name: event-exporter <del> image: gcr.io/google-containers/event-exporter:v0.1.0-r2 </del> <ins> image: gcr.io/google-containers/event-exporter:v0.1.4 </ins> command: - '/event-exporter' - name: prometheus-to-sd-exporter


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If node is unreachable, NodeController only set , , to unknown; it's better to also set NetworkUnavailable & InodePressure to unknown.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
Isn't a known condition if node is unreachable?
is the status of container, e.g. CNI is not ready. If node is unreachable, it should be Unknown.
Thanks for explanation!
will not set NetworkUnavailable & InodePressure to unknown, refer to .

} } <del> // For a given node checks its conditions and tries to update it. Returns grace period to which given node // is entitled, state of current and last observed Ready Condition, and an error if it occurred. </del> <ins> // tryUpdateNodeStatus checks a given node's conditions and tries to update it. Returns grace period to // which given node is entitled, state of current and last observed Ready Condition, and an error if it occurred. </ins> func (nc *NodeController) tryUpdateNodeStatus(node *v1.Node) (time.Duration, v1.NodeCondition, *v1.NodeCondition, error) { var err error var gracePeriod time.Duration


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If node is unreachable, NodeController only set , , to unknown; it's better to also set NetworkUnavailable & InodePressure to unknown.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
Isn't a known condition if node is unreachable?
is the status of container, e.g. CNI is not ready. If node is unreachable, it should be Unknown.
Thanks for explanation!
will not set NetworkUnavailable & InodePressure to unknown, refer to .

// otherwise we leave it as it is. if savedCondition.LastTransitionTime != observedCondition.LastTransitionTime { glog.V(3).Infof("ReadyCondition for Node %s transitioned from %v to %v", node.Name, savedCondition.Status, observedCondition) transitionTime = nc.now() } else { transitionTime = savedNodeStatus.readyTransitionTimestamp


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If node is unreachable, NodeController only set , , to unknown; it's better to also set NetworkUnavailable & InodePressure to unknown.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
Isn't a known condition if node is unreachable?
is the status of container, e.g. CNI is not ready. If node is unreachable, it should be Unknown.
Thanks for explanation!
will not set NetworkUnavailable & InodePressure to unknown, refer to .

} // remaining node conditions should also be set to Unknown <del> remainingNodeConditionTypes := []v1.NodeConditionType{v1.NodeOutOfDisk, v1.NodeMemoryPressure, v1.NodeDiskPressure} </del> <ins> remainingNodeConditionTypes := []v1.NodeConditionType{ v1.NodeOutOfDisk, v1.NodeMemoryPressure, v1.NodeDiskPressure, // We don't change 'NodeInodePressure' condition, as it'll be removed in future. // v1.NodeInodePressure, // We don't change 'NodeNetworkUnavailable' condition, as it's managed on a control plane level. // v1.NodeNetworkUnavailable, } </ins> nowTimestamp := nc.now() for _, nodeConditionType := range remainingNodeConditionTypes { _, currentCondition := nodeutil.GetNodeCondition(&node.Status, nodeConditionType)


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If node is unreachable, NodeController only set , , to unknown; it's better to also set NetworkUnavailable & InodePressure to unknown.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
Isn't a known condition if node is unreachable?
is the status of container, e.g. CNI is not ready. If node is unreachable, it should be Unknown.
Thanks for explanation!
will not set NetworkUnavailable & InodePressure to unknown, refer to .

return 0 } <del> // This function is expected to get a slice of NodeReadyConditions for all Nodes in a given zone. </del> <ins> // ComputeZoneState returns a slice of NodeReadyConditions for all Nodes in a given zone. </ins> // The zone is considered: // - fullyDisrupted if there're no Ready Nodes, // - partiallyDisrupted if at least than nc.unhealthyZoneThreshold percent of Nodes are not Ready,


Question: There's a problem b/c pods are created to run on all nodes, but run sequentially, effectively multiplying testing time

"math" "time" <del> meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1" </del> "k8s.io/kubernetes/test/e2e/framework" "k8s.io/kubernetes/test/e2e/instrumentation"


Question: There's a problem b/c pods are created to run on all nodes, but run sequentially, effectively multiplying testing time

nodes := framework.GetReadySchedulableNodesOrDie(f.ClientSet).Items maxPodCount := 10 <del> jobDuration := 1 * time.Hour </del> <ins> jobDuration := 30 * time.Minute </ins> linesPerPodPerSecond := 100 <del> testDuration := 21 * time.Hour </del> <ins> // TODO(crassirostris): Increase to 21 hrs testDuration := 3 * time.Hour </ins> ingestionTimeout := testDuration + 30*time.Minute allowedRestarts := int(math.Ceil(float64(testDuration) / float64(time.Hour) * maxAllowedRestartsPerHour))


Question: There's a problem b/c pods are created to run on all nodes, but run sequentially, effectively multiplying testing time

podRunCount := maxPodCount*(int(testDuration/jobDuration)-1) + 1 linesPerPod := linesPerPodPerSecond * int(jobDuration.Seconds()) <ins> // pods is a flat array of all pods to be run and to expect in Stackdriver. </ins> pods := []*loggingPod{} <ins> // podsByRun is a two-dimensional array of pods, first dimension is the run // index, the second dimension is the node index. Since we want to create // an equal load on all nodes, for the same run we have one pod per node. podsByRun := [][]*loggingPod{} </ins> for runIdx := 0; runIdx < podRunCount; runIdx++ { <ins> podsInRun := []*loggingPod{} </ins> for nodeIdx, node := range nodes { podName := fmt.Sprintf("job-logs-generator-%d-%d-%d-%d", maxPodCount, linesPerPod, runIdx, nodeIdx) <del> pods = append(pods, newLoggingPod(podName, node.Name, linesPerPod, jobDuration)) </del> <ins> pod := newLoggingPod(podName, node.Name, linesPerPod, jobDuration) pods = append(pods, pod) podsInRun = append(podsInRun, pod) </ins> } <ins> podsByRun = append(podsByRun, podsInRun) </ins> } By("Running short-living pods") go func() { <del> for _, pod := range pods { pod.Start(f) </del> <ins> for runIdx := 0; runIdx < podRunCount; runIdx++ { // Starting one pod on each node. for _, pod := range podsByRun[runIdx] { pod.Start(f) } </ins> time.Sleep(podRunDelay) <del> defer f.PodClient().Delete(pod.Name, &meta_v1.DeleteOptions{}) </del> } // Waiting until the last pod has completed time.Sleep(jobDuration - podRunDelay + lastPodIngestionSlack)


Question:
/kind bug What happened: A pod with an empty namespace field was mutated by a pod preset in a different namespace. What you expected to happen: Pod presets should only operate on pods in the same namespace. How to reproduce it (as minimally and precisely as possible): Submit a pod with an empty namespace field and observe pod preset mutation. Anything else we need to know?: This is a fix for an alpha non-default feature. Fix submitted in
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery

return nil } <del> list, err := c.lister.PodPresets(pod.GetNamespace()).List(labels.Everything()) </del> <ins> list, err := c.lister.PodPresets(a.GetNamespace()).List(labels.Everything()) </ins> // Ignore if exclusion annotation is present if podAnnotations := pod.GetAnnotations(); podAnnotations != nil {


Question:
/kind bug What happened: A pod with an empty namespace field was mutated by a pod preset in a different namespace. What you expected to happen: Pod presets should only operate on pods in the same namespace. How to reproduce it (as minimally and precisely as possible): Submit a pod with an empty namespace field and observe pod preset mutation. Anything else we need to know?: This is a fix for an alpha non-default feature. Fix submitted in
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery

} } <ins> func TestAdmitEmptyPodNamespace(t *testing.T) { containerName := "container" pod := &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "mypod", Labels: map[string]string{ "security": "S2", }, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name: containerName, Env:  []api.EnvVar{{Name: "abc", Value: "value2"}, {Name: "ABCD", Value: "value3"}}, }, }, }, } pip := &settings.PodPreset{ ObjectMeta: v1.ObjectMeta{ Name:      "hello", Namespace: "different", // (pod will be submitted to namespace 'namespace') }, Spec: settings.PodPresetSpec{ Selector: v1.LabelSelector{ MatchExpressions: []v1.LabelSelectorRequirement{ { Key:      "security", Operator: v1.LabelSelectorOpIn, Values:   []string{"S2"}, }, }, }, Volumes: []api.Volume{{Name: "vol", VolumeSource: api.VolumeSource{EmptyDir: &api.EmptyDirVolumeSource{}}}}, Env:     []api.EnvVar{{Name: "abcd", Value: "value"}, {Name: "ABC", Value: "value"}}, EnvFrom: []api.EnvFromSource{ { ConfigMapRef: &api.ConfigMapEnvSource{ LocalObjectReference: api.LocalObjectReference{Name: "abc"}, }, }, { Prefix: "pre_", ConfigMapRef: &api.ConfigMapEnvSource{ LocalObjectReference: api.LocalObjectReference{Name: "abc"}, }, }, }, }, } originalPod, err := api.Scheme.Copy(pod) if err != nil { t.Fatal(err) } err = admitPod(pod, pip) if err != nil { t.Fatal(err) } // verify PodSpec has not been mutated if !reflect.DeepEqual(pod, originalPod) { t.Fatalf("Expected pod spec of '%v' to be unchanged", pod.Name) } } </ins> func admitPod(pod *api.Pod, pip *settings.PodPreset) error { informerFactory := informers.NewSharedInformerFactory(nil, controller.NoResyncPeriodFunc()) store := informerFactory.Settings().InternalVersion().PodPresets().Informer().GetStore()


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When trying to list object using field spec.SchedulerName, we are getting the below error message. Failed to list : field label not supported: spec.schedulerName What you expected to happen: To be able to fetch pods using the list and watch based on the field 'spec.schedulerName' How to reproduce it (as minimally and precisely as possible): Create a simple controller to with a list and watch method. Set the FieldSelector ( in the ListOptions structure ) in the ListFunc and the WatchFunc to 'spec.SchedulerName=custome-SchedulerName' Running this controller yields the above error. Anything else we need to know?: Description: Unable to fetch pods based on the field 'spec.schedulerName'. This was in 1.6 version, for multi scheduler support in kubernetes. We need to be able to fetch pods 1) Assigned to a particular custom scheduler. 2) Should be able to fetch the state transition of the pods scheduled by a custom scheduler using the fields selector. Whats the correct way to use 'schedulerName' fields for fetching objects? Or is there any other way for the custom scheduler to determine the above information from the kubernetes? This is an urgent issue, any quick help/suggestion will be of great help. Environment: Kubernetes version : Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-14T01:48:01Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Install tools: kubeadm
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
/cc
cc
SIG API Machinery doesn't own the contents of the API--we only need to be involved if the mechanism needs to be general.
Given the error message mentioned in the bug report at the top of this thread, do you have a suggestion of what we forgot to do when we the SchedulerName field to the PodSpec? i.e. to enable field selection on the schedulerName field?
/cc me

"spec.nodeName", "spec.restartPolicy", "spec.serviceAccountName", <ins> "spec.schedulerName", </ins> "status.phase", "status.hostIP", "status.podIP":


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When trying to list object using field spec.SchedulerName, we are getting the below error message. Failed to list : field label not supported: spec.schedulerName What you expected to happen: To be able to fetch pods using the list and watch based on the field 'spec.schedulerName' How to reproduce it (as minimally and precisely as possible): Create a simple controller to with a list and watch method. Set the FieldSelector ( in the ListOptions structure ) in the ListFunc and the WatchFunc to 'spec.SchedulerName=custome-SchedulerName' Running this controller yields the above error. Anything else we need to know?: Description: Unable to fetch pods based on the field 'spec.schedulerName'. This was in 1.6 version, for multi scheduler support in kubernetes. We need to be able to fetch pods 1) Assigned to a particular custom scheduler. 2) Should be able to fetch the state transition of the pods scheduled by a custom scheduler using the fields selector. Whats the correct way to use 'schedulerName' fields for fetching objects? Or is there any other way for the custom scheduler to determine the above information from the kubernetes? This is an urgent issue, any quick help/suggestion will be of great help. Environment: Kubernetes version : Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-14T01:48:01Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Install tools: kubeadm
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
/cc
cc
SIG API Machinery doesn't own the contents of the API--we only need to be involved if the mechanism needs to be general.
Given the error message mentioned in the bug report at the top of this thread, do you have a suggestion of what we forgot to do when we the SchedulerName field to the PodSpec? i.e. to enable field selection on the schedulerName field?
/cc me

// amount of allocations needed to create the fields.Set. If you add any // field here or the number of object-meta related fields changes, this should // be adjusted. <del> podSpecificFieldsSet := make(fields.Set, 6) </del> <ins> podSpecificFieldsSet := make(fields.Set, 7) </ins> podSpecificFieldsSet["spec.nodeName"] = pod.Spec.NodeName podSpecificFieldsSet["spec.restartPolicy"] = string(pod.Spec.RestartPolicy) <ins> podSpecificFieldsSet["spec.schedulerName"] = string(pod.Spec.SchedulerName) </ins> podSpecificFieldsSet["status.phase"] = string(pod.Status.Phase) podSpecificFieldsSet["status.podIP"] = string(pod.Status.PodIP) return generic.AddObjectMetaFieldsSet(podSpecificFieldsSet, &pod.ObjectMeta, true)


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When trying to list object using field spec.SchedulerName, we are getting the below error message. Failed to list : field label not supported: spec.schedulerName What you expected to happen: To be able to fetch pods using the list and watch based on the field 'spec.schedulerName' How to reproduce it (as minimally and precisely as possible): Create a simple controller to with a list and watch method. Set the FieldSelector ( in the ListOptions structure ) in the ListFunc and the WatchFunc to 'spec.SchedulerName=custome-SchedulerName' Running this controller yields the above error. Anything else we need to know?: Description: Unable to fetch pods based on the field 'spec.schedulerName'. This was in 1.6 version, for multi scheduler support in kubernetes. We need to be able to fetch pods 1) Assigned to a particular custom scheduler. 2) Should be able to fetch the state transition of the pods scheduled by a custom scheduler using the fields selector. Whats the correct way to use 'schedulerName' fields for fetching objects? Or is there any other way for the custom scheduler to determine the above information from the kubernetes? This is an urgent issue, any quick help/suggestion will be of great help. Environment: Kubernetes version : Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-07-14T01:48:01Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Install tools: kubeadm
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
/cc
cc
SIG API Machinery doesn't own the contents of the API--we only need to be involved if the mechanism needs to be general.
Given the error message mentioned in the bug report at the top of this thread, do you have a suggestion of what we forgot to do when we the SchedulerName field to the PodSpec? i.e. to enable field selection on the schedulerName field?
/cc me

}, { in: &api.Pod{ <ins> Spec: api.PodSpec{SchedulerName: "scheduler1"}, }, fieldSelector: fields.ParseSelectorOrDie("spec.schedulerName=scheduler1"), expectMatch:   true, }, { in: &api.Pod{ Spec: api.PodSpec{SchedulerName: "scheduler1"}, }, fieldSelector: fields.ParseSelectorOrDie("spec.schedulerName=scheduler2"), expectMatch:   false, }, { in: &api.Pod{ </ins> Status: api.PodStatus{Phase: api.PodRunning}, }, fieldSelector: fields.ParseSelectorOrDie("status.phase=Running"),


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig storage What happened: Offshoot of . Let's change the tests to use the programmatic API instead of gcloud CLI.
Just confirming that this issue is referring only to the 3 gcloud CLI calls in
yup
PR:

return zones, nil } <ins> // ListInstanceNames returns a string of instance names seperated by spaces. func (gce *GCECloud) ListInstanceNames(project, zone string) (string, error) { res, err := gce.service.Instances.List(project, zone).Fields("items(name)").Do() if err != nil { return "", err } var output string for _, item := range res.Items { output += item.Name + " " } return output, nil } // DeleteInstance deletes an instance specified by project, zone, and name func (gce *GCECloud) DeleteInstance(project, zone, name string) (*compute.Operation, error) { return gce.service.Instances.Delete(project, zone, name).Do() } </ins> // Implementation of Instances.CurrentNodeName func (gce *GCECloud) CurrentNodeName(hostname string) (types.NodeName, error) { return types.NodeName(hostname), nil


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig storage What happened: Offshoot of . Let's change the tests to use the programmatic API instead of gcloud CLI.
Just confirming that this issue is referring only to the 3 gcloud CLI calls in
yup
PR:

import ( "fmt" mathrand "math/rand" <del> "os/exec" </del> "strings" "time"


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig storage What happened: Offshoot of . Let's change the tests to use the programmatic API instead of gcloud CLI.
Just confirming that this issue is referring only to the 3 gcloud CLI calls in
yup
PR:

// Verify that disk shows up in node 0's volumeInUse list framework.ExpectNoError(waitForPDInVolumesInUse(nodeClient, diskName, host0Name, nodeStatusTimeout, true /* should exist*/)) <del> output, err := exec.Command("gcloud", "compute", "instances", "list", "--project="+framework.TestContext.CloudConfig.ProjectID).CombinedOutput() </del> <ins> gceCloud, err := framework.GetGCECloud() framework.ExpectNoError(err, fmt.Sprintf("Unable to create gcloud client err=%v", err)) output, err := gceCloud.ListInstanceNames(framework.TestContext.CloudConfig.ProjectID, framework.TestContext.CloudConfig.Zone) </ins> framework.ExpectNoError(err, fmt.Sprintf("Unable to get list of node instances err=%v output=%s", err, output)) Expect(true, strings.Contains(string(output), string(host0Name))) By("deleting host0") <ins> resp, err := gceCloud.DeleteInstance(framework.TestContext.CloudConfig.ProjectID, framework.TestContext.CloudConfig.Zone, string(host0Name)) framework.ExpectNoError(err, fmt.Sprintf("Failed to delete host0pod: err=%v response=%#v", err, resp)) </ins> <del> output, err = exec.Command("gcloud", "compute", "instances", "delete", string(host0Name), "--project="+framework.TestContext.CloudConfig.ProjectID, "--zone="+framework.TestContext.CloudConfig.Zone).CombinedOutput() framework.ExpectNoError(err, fmt.Sprintf("Failed to delete host0pod: err=%v output=%s", err, output)) output, err = exec.Command("gcloud", "compute", "instances", "list", "--project="+framework.TestContext.CloudConfig.ProjectID).CombinedOutput() </del> <ins> output, err = gceCloud.ListInstanceNames(framework.TestContext.CloudConfig.ProjectID, framework.TestContext.CloudConfig.Zone) </ins> framework.ExpectNoError(err, fmt.Sprintf("Unable to get list of node instances err=%v output=%s", err, output)) Expect(false, strings.Contains(string(output), string(host0Name)))


Question:
Seeing flakes on master: https://k8s- https://k8s- and on unrelated PRs: https://k8s-
I was able to reproduce this locally (although it's very hard to trigger). Initial thoughts are that it's different from ; in this case, the actual sync is instant (millseconds) but the CRD created in the test fails to appear in discovery calls over the course of repeated attempts every 5 seconds for the remainder of the test. Normally, the type shows up in discovery almost immediately after creation.
Looks like PR might be hitting a related flaky test. I get 6 failures in 60 runs from the command I'm super new around here so thought it prudent to add the information I have found before attempting to dig deeper. I followed instructions in to build from within directory If the test failure logs are of any use to anyone just message and I'll post one of them.
Thanks. I've also been able to reproduce locally and am debugging it today. I did have to hack up the integration test setup to get apiserver logs in this context. I've also opened to explore making the integration test script more configurable with regards to logging.
I was wrong about the discovery aspect; the CRD is coming back from discovery, but the restMapper fails to map it. I also notice we're using DiscoveryInterface.ServerPreferredResources which is not actually implemented with a cache in memCacheClient.
okay, now I'm getting somewhere. I made a which exercises discovery and REST mapping without the GC in the picture at all, and have been able to reproduce the flake 1 time out of 40 runs so far with .
First problem is pretty clear in hindsight; we reset the restmapper, then get deletable resources using the discovery client directly, then sync based on the resources from discovery. The direct discovery call doesn't use the client's internal cache, and so the restmapper might be out of sync with the subsequent discovery call, causing us to sync with resources not yet represented in the restmapper's cache. That explains the isolated flake in the test I made, but doesn't yet explain why the restmapper would continue to fail on subsequent sync calls (since the nest restmapper reset would usually bring it up to date with the resources returned from the direct discovery call).
I'm out tomorrow, so a quick update here in case somebody else looks into it... can reproduce the issue where DeferredDiscoveryRESTMapper sometimes becomes unable to map the CRD resource, even after the type is available from discovery. I've also ruled out the memCacheClient by replacing it with a non-caching implementation like: And using that in the setup like: The issue can still be reproduced that way. Looks like more debugging of DeferredDiscoveryRESTMapper itself is required.
Finally tracked this down. Custom resource kinds are always given a plural suffix of "s". If the singular name ends in "s", the assumes the plural form is an "es" suffix, which will never match the generated kind. This explains the intermittent nature of the failure (the randomly generated names often happen to match the assumptions of the rest mapper).
is the root cause
Good effort

//NewRandomNameCustomResourceDefinition generates a CRD with random name to avoid name conflict in e2e tests func NewRandomNameCustomResourceDefinition(scope apiextensionsv1beta1.ResourceScope) *apiextensionsv1beta1.CustomResourceDefinition { <del> gName := names.SimpleNameGenerator.GenerateName("foo") </del> <ins> // ensure the singular doesn't end in an s for now gName := names.SimpleNameGenerator.GenerateName("foo") + "a" </ins> return &apiextensionsv1beta1.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: gName + "s.mygroup.example.com"}, Spec: apiextensionsv1beta1.CustomResourceDefinitionSpec{


Question:
In Cinder cloud provider openstack Inside of AttachDisk function, it issues DetachDisk if it find out the the volume is currently attached to a different instance. This is very dangerous because it is very likely the volume is still mounted to that instance. The correct behavior should be fail this AttachDisk call. K8s attachdetach controller should handle unmount and detach volume safely.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
cc

glog.V(4).Infof("Disk %s is already attached to instance %s", volumeID, instanceID) return volume.ID, nil } <del> glog.V(2).Infof("Disk %s is attached to a different instance (%s), detaching", volumeID, volume.AttachedServerId) err = os.DetachDisk(volume.AttachedServerId, volumeID) if err != nil { return "", err } </del> <ins> errmsg := fmt.Sprintf("Disk %s is attached to a different instance (%s)", volumeID, volume.AttachedServerId) glog.V(2).Infof(errmsg) return "", errors.New(errmsg) </ins> } startTime := time.Now()


Question:
Making an API request for table output and specifying the parameter causes a server error in parameter conversion, and an inaccurate error message in response: server log: E0804 09:41:14. ] apiserver received an error that is not an metav1.Status: converting (url.Values).[]string to (v1alpha1.TableOptions).v1alpha1.IncludeObjectPolicy: couldn't copy '[]string' into 'v1alpha1.IncludeObjectPolicy'; didn't understand types
Ugh, typed strings.

go_library( name = "go_default_library", srcs = [ <ins> "conversion.go", </ins> "deepcopy.go", "doc.go", "generated.pb.go",


Question:
Making an API request for table output and specifying the parameter causes a server error in parameter conversion, and an inaccurate error message in response: server log: E0804 09:41:14. ] apiserver received an error that is not an metav1.Status: converting (url.Values).[]string to (v1alpha1.TableOptions).v1alpha1.IncludeObjectPolicy: couldn't copy '[]string' into 'v1alpha1.IncludeObjectPolicy'; didn't understand types
Ugh, typed strings.

<ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package v1alpha1 import "k8s.io/apimachinery/pkg/conversion" // Convert_Slice_string_To_v1alpha1_IncludeObjectPolicy allows converting a URL query parameter value func Convert_Slice_string_To_v1alpha1_IncludeObjectPolicy(input *[]string, out *IncludeObjectPolicy, s conversion.Scope) error { if len(*input) > 0 { *out = IncludeObjectPolicy((*input)[0]) } return nil } </ins>


Question:
Making an API request for table output and specifying the parameter causes a server error in parameter conversion, and an inaccurate error message in response: server log: E0804 09:41:14. ] apiserver received an error that is not an metav1.Status: converting (url.Values).[]string to (v1alpha1.TableOptions).v1alpha1.IncludeObjectPolicy: couldn't copy '[]string' into 'v1alpha1.IncludeObjectPolicy'; didn't understand types
Ugh, typed strings.

&PartialObjectMetadataList{}, ) <ins> if err := scheme.AddConversionFuncs( Convert_Slice_string_To_v1alpha1_IncludeObjectPolicy, ); err != nil { panic(err) } </ins> // register manually. This usually goes through the SchemeBuilder, which we cannot use here. //scheme.AddGeneratedDeepCopyFuncs(GetGeneratedDeepCopyFuncs()...) }


Question:
Making an API request for table output and specifying the parameter causes a server error in parameter conversion, and an inaccurate error message in response: server log: E0804 09:41:14. ] apiserver received an error that is not an metav1.Status: converting (url.Values).[]string to (v1alpha1.TableOptions).v1alpha1.IncludeObjectPolicy: couldn't copy '[]string' into 'v1alpha1.IncludeObjectPolicy'; didn't understand types
Ugh, typed strings.

}, }, }, <ins> { accept: runtime.ContentTypeJSON + ";as=Table;v=v1alpha1;g=meta.k8s.io", params: url.Values{"includeObject": []string{"Metadata"}}, expected: &metav1alpha1.Table{ TypeMeta: metav1.TypeMeta{Kind: "Table", APIVersion: "meta.k8s.io/v1alpha1"}, ColumnDefinitions: []metav1alpha1.TableColumnDefinition{ {Name: "Name", Type: "string", Description: metaDoc["name"]}, {Name: "Created At", Type: "date", Description: metaDoc["creationTimestamp"]}, }, Rows: []metav1alpha1.TableRow{ {Cells: []interface{}{"foo1", now.Time.UTC().Format(time.RFC3339)}, Object: runtime.RawExtension{Raw: encodedBody}}, }, }, }, </ins> } for i, test := range tests { u, err := url.Parse(server.URL + "/" + prefix + "/" + testGroupVersion.Group + "/" + testGroupVersion.Version + "/namespaces/default/simple/id")


Question:
Making an API request for table output and specifying the parameter causes a server error in parameter conversion, and an inaccurate error message in response: server log: E0804 09:41:14. ] apiserver received an error that is not an metav1.Status: converting (url.Values).[]string to (v1alpha1.TableOptions).v1alpha1.IncludeObjectPolicy: couldn't copy '[]string' into 'v1alpha1.IncludeObjectPolicy'; didn't understand types
Ugh, typed strings.

} if test.statusCode != 0 { if resp.StatusCode != test.statusCode { <del> t.Errorf("%d: unexpected response: %#v", resp) </del> <ins> t.Errorf("%d: unexpected response: %#v", i, resp) </ins> } continue } if resp.StatusCode != http.StatusOK { <del> t.Errorf("%d: unexpected response: %#v", resp) </del> <ins> t.Errorf("%d: unexpected response: %#v", i, resp) </ins> } var itemOut metav1alpha1.Table <del> if _, err = extractBody(resp, &itemOut); err != nil { </del> <ins> body, err := extractBody(resp, &itemOut) if err != nil { </ins> t.Fatal(err) } if !reflect.DeepEqual(test.expected, &itemOut) { <ins> t.Log(body) </ins> t.Errorf("%d: did not match: %s", i, diff.ObjectReflectDiff(test.expected, &itemOut)) } }


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Unit test error message match the conventional Go style: For example: if got, want := set.Status.Replicas, 3; got != want { t.Errorf("set.Status.Replicas = %v; want %v", got, want) }
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing
/assign
/reopen

t.Errorf("unexpected error: %v", err) } if len(times) != 0 { <del> t.Errorf("expected 0 start times, got: , got: %v", times) </del> <ins> t.Errorf("expected 0 start times, got: %v", times) </ins> } } {


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Unit test error message match the conventional Go style: For example: if got, want := set.Status.Replicas, 3; got != want { t.Errorf("set.Status.Replicas = %v; want %v", got, want) }
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing
/assign
/reopen

t.Errorf("unexpected error: %v", err) } if len(times) != 1 { <del> t.Errorf("expected 2 start times, got: , got: %v", times) </del> <ins> t.Errorf("expected 1 start times, got: %v", times) </ins> } else if !times[0].Equal(T2) { t.Errorf("expected: %v, got: %v", T1, times[0]) }


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Unit test error message match the conventional Go style: For example: if got, want := set.Status.Replicas, 3; got != want { t.Errorf("set.Status.Replicas = %v; want %v", got, want) }
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing
/assign
/reopen

t.Errorf("unexpected error: %v", err) } if len(times) != 2 { <del> t.Errorf("expected 2 start times, got: , got: %v", times) </del> <ins> t.Errorf("expected 2 start times, got: %v", times) </ins> } else { if !times[0].Equal(T1) { t.Errorf("expected: %v, got: %v", T1, times[0])


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Unit test error message match the conventional Go style: For example: if got, want := set.Status.Replicas, 3; got != want { t.Errorf("set.Status.Replicas = %v; want %v", got, want) }
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing
/assign
/reopen

set = obj.(*apps.StatefulSet) } if set.Status.Replicas != 3 { <del> t.Error("Falied to scale statefulset to 3 replicas") </del> <ins> t.Errorf("set.Status.Replicas = %v; want 3", set.Status.Replicas) </ins> } }


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Unit test error message match the conventional Go style: For example: if got, want := set.Status.Replicas, 3; got != want { t.Errorf("set.Status.Replicas = %v; want %v", got, want) }
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing
/assign
/reopen

set = obj.(*apps.StatefulSet) } if set.Status.Replicas != 3 { <del> t.Error("Falied to scale statefulset to 3 replicas") </del> <ins> t.Errorf("set.Status.Replicas = %v; want 3", set.Status.Replicas) </ins> } *set.Spec.Replicas = 0 if err := scaleDownStatefulSetController(set, ssc, spc); err != nil {


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Unit test error message match the conventional Go style: For example: if got, want := set.Status.Replicas, 3; got != want { t.Errorf("set.Status.Replicas = %v; want %v", got, want) }
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing
/assign
/reopen

set = obj.(*apps.StatefulSet) } if set.Status.Replicas != 3 { <del> t.Error("Falied to scale statefulset to 3 replicas") </del> <ins> t.Errorf("set.Status.Replicas = %v; want 3", set.Status.Replicas) </ins> } pods, err := spc.addTerminatingPod(set, 3) if err != nil {


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Unit test error message match the conventional Go style: For example: if got, want := set.Status.Replicas, 3; got != want { t.Errorf("set.Status.Replicas = %v; want %v", got, want) }
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing
/assign
/reopen

set = obj.(*apps.StatefulSet) } if set.Status.Replicas != 0 { <del> t.Error("Falied to scale statefulset to 3 replicas") </del> <ins> t.Errorf("set.Status.Replicas = %v; want 0", set.Status.Replicas) </ins> } }


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Unit test error message match the conventional Go style: For example: if got, want := set.Status.Replicas, 3; got != want { t.Errorf("set.Status.Replicas = %v; want %v", got, want) }
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing
/assign
/reopen

set = obj.(*apps.StatefulSet) } if set.Status.Replicas != 3 { <del> t.Error("Falied to scale statefulset to 3 replicas") </del> <ins> t.Errorf("set.Status.Replicas = %v; want 3", set.Status.Replicas) </ins> } *set.Spec.Replicas = 5 fakeResourceVersion(set)


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: e2e test panic in test What you expected to happen: no panic How to reproduce it (as minimally and precisely as possible): start " [Feature:ManualPerformance] should be able to handle 3 pods per node" e2e test on a 3-node cluster Anything else we need to know?: panic info : ! After some digging find why. Because the cluster size is 3 that d is 0 in sleepUpTo func which causes rand.Int63n panic happy to fix this. Environment: Kubernetes version (use ):1.7.3 Cloud provider or hardware configuration*: bare-metal OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): 16.04 Install tools: kubeadm Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig Testing
just add a sanity check is fine I think
It seems that the count of pods is 0.
pod count is not 0, is 9(33)
If totalPods less than throughput, in our case, totalPods is 9, throughput is 10, the creatingTime will be 0, which makes rand.Int63n panic
or throughput is getting from env LOADTESTTHROUGHPUT, safe to add a sanity check
If totalPods less than throughput, in our case, totalPods is 9, throughput is 10, the creatingTime will be 0, which makes rand.Int63n panic agree.

} func sleepUpTo(d time.Duration) { <del> time.Sleep(time.Duration(rand.Int63n(d.Nanoseconds()))) </del> <ins> if d.Nanoseconds() > 0 { time.Sleep(time.Duration(rand.Int63n(d.Nanoseconds()))) } </ins> } func createAllResources(configs []testutils.RunObjectConfig, creatingTime time.Duration) {


Question:
seems to belong to sig node PTAL and fix ASAP
/assign as it seems like having something to do with

dockercontainer "github.com/docker/docker/api/types/container" dockerfilters "github.com/docker/docker/api/types/filters" "github.com/golang/glog" <del> "k8s.io/api/core/v1" </del> runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/v1alpha1/runtime" )


Question:
seems to belong to sig node PTAL and fix ASAP
/assign as it seems like having something to do with

return 0 } <del> func (ds *dockerService) getSecurityOpts(containerName string, sandboxConfig *runtimeapi.PodSandboxConfig, separator rune) ([]string, error) { hasSeccompSetting := false annotations := sandboxConfig.GetAnnotations() if _, ok := annotations[v1.SeccompContainerAnnotationKeyPrefix+containerName]; !ok { _, hasSeccompSetting = annotations[v1.SeccompPodAnnotationKey] } else { hasSeccompSetting = true </del> <ins> func (ds *dockerService) getSecurityOpts(seccompProfile string, separator rune) ([]string, error) { if seccompProfile != "" { glog.Warningf("seccomp annotations are not supported on windows") </ins> } <del> if hasSeccompSetting { glog.Warningf("seccomp annotations found, but it is not supported on windows") } </del> return nil, nil }


Question:
We allow users to specify in the Service object. If the user later removes this from the Service Spec, the controller will fail to sync. For example: creates an LB for Service.Spec.LoadBalancerIP = "1.1.1.1" with a user-owned static IP address. updates the Service to remove the requested LoadBalancerIP tries to used by the existing forwarding rule (i.e., "1.1.1.1"). It encounters a conflict. Controller assumes it owns the IP and by name but fails. We can fix the controller to continue syncing and using the user-owned IP for the LB, or create a new address for the LB. The former seems less disruptive to me, but the downside is that if user will not be able to let other components use this IP (i.e., 1.1.1.1) unless they explicitly delete the Service. I think it's ok but would like more opinions. thoughts?
If a service is updated to remove a static address, then it seems like that would be a deliberate move on the part of the user. It seems like it should probably release the address and provision a dynamic one, even if this is disruptive. How do we handle the reverse? If a user has an already existing LoadBalancer service, and they add the after the fact? Would this release the dynamic address and recreate the forwarding rule with the reserved static address?
It seems like the creating a new address makes sense here... General comment: given the combinations (and the bugs we have been running into), I think this is a problem that all of the potential state transitions for the load balancer have not been fully spec'd out.
Creating a new address seems nice but the implementation is a nightmare. The service controller doesn't give you a previous Service spec, so you don't know what's changed (controller might not know either if spec was modified while offline). Given a desired service spec without a how do you decide to ditch your current IP and grab a new one? If we were to call to evaluate the situation, it's highly likely users will accidentally cause LBs to change IPs during typical usage. Imagine a user has a dynamically allocated IP for a service, but they want to promote it for safe keeping. The controller would think it needs to surrender the IP to the user's reservation. The only correct process would be to set before promoting the address; however, that will surely be a missed point in docs and result in unhappy users. I lean towards the LB continuing use of the IP. Besides the easy option of deleting & re-creating the service, they could change the to another static address then demote the address. I'm not too familiar with the ideological boundaries of controllers and spec, so let me know if this crosses the line. What if the service controller set to the created IP on first sync if the IP wasn't pre-specified? The controller could rely on loadBalancerIP to always indicate which IP to use. If the field is empty (blanked by user), then it's a signal to get a new IP.
By spec, I mean we should write down the careful reasoning that you just wrote in a permanent place :-)
would it then make sense for to be immutable on an already created service?
+1 +1 Was thinking the same. Unlike the many annotations we support, updates to this actual field could be validated and blocked at the apiserver... Changing IP is disruptive anyway. Could we simply ask the user to re-create the Service?
Of course, this could be an API-breaking change, and other cloud providers may actually be supporting this properly.
I just ran across this today, but in a more likely (IMO) case. 1) create the Service, type=LoadBalancer 2) let the IP be auto-allocated 3) manually promote that IP to static and give it a meaningful name 4) observe errors from service controller trying to allocate that IP Since my Service does not spec "loadBalancerIP", the question is, do we: a) use the IP, but leave it as static b) use the IP, but demote it back to ephemeral c) allocate and use a new IP C is completely hostile. B is subtly hostile. A seems correct. Can it be implemented? If we find that the IP exists but the name of it is not what we expect, we should treat it as user-provided and leave it alone. This is still nasty because it logs a failure in the GCP activities log for every loop, but I don't know how to avoid that without getting a list of IPs and scanning for the IP we want.
Filed for the looping.
+1 as long as we can rely on the "naming" of the address to decide whether the controller owns the address and can promote/demote it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Still relevant
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Still relevant
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
any update on this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.

existed = true } <ins> // If address exists, get it by IP, because name might be different. // This can specifically happen if the IP was changed from ephemeral to static, // which results in a new name for the IP. if existingIP != "" { addr, err := s.GetRegionAddressByIP(region, existingIP) if err != nil { return "", false, fmt.Errorf("error getting static IP address: %v", err) } return addr.Address, existed, nil } // Otherwise, get address by name </ins> addr, err := s.GetRegionAddress(name, region) if err != nil { return "", false, fmt.Errorf("error getting static IP address: %v", err)


Question:
We allow users to specify in the Service object. If the user later removes this from the Service Spec, the controller will fail to sync. For example: creates an LB for Service.Spec.LoadBalancerIP = "1.1.1.1" with a user-owned static IP address. updates the Service to remove the requested LoadBalancerIP tries to used by the existing forwarding rule (i.e., "1.1.1.1"). It encounters a conflict. Controller assumes it owns the IP and by name but fails. We can fix the controller to continue syncing and using the user-owned IP for the LB, or create a new address for the LB. The former seems less disruptive to me, but the downside is that if user will not be able to let other components use this IP (i.e., 1.1.1.1) unless they explicitly delete the Service. I think it's ok but would like more opinions. thoughts?
If a service is updated to remove a static address, then it seems like that would be a deliberate move on the part of the user. It seems like it should probably release the address and provision a dynamic one, even if this is disruptive. How do we handle the reverse? If a user has an already existing LoadBalancer service, and they add the after the fact? Would this release the dynamic address and recreate the forwarding rule with the reserved static address?
It seems like the creating a new address makes sense here... General comment: given the combinations (and the bugs we have been running into), I think this is a problem that all of the potential state transitions for the load balancer have not been fully spec'd out.
Creating a new address seems nice but the implementation is a nightmare. The service controller doesn't give you a previous Service spec, so you don't know what's changed (controller might not know either if spec was modified while offline). Given a desired service spec without a how do you decide to ditch your current IP and grab a new one? If we were to call to evaluate the situation, it's highly likely users will accidentally cause LBs to change IPs during typical usage. Imagine a user has a dynamically allocated IP for a service, but they want to promote it for safe keeping. The controller would think it needs to surrender the IP to the user's reservation. The only correct process would be to set before promoting the address; however, that will surely be a missed point in docs and result in unhappy users. I lean towards the LB continuing use of the IP. Besides the easy option of deleting & re-creating the service, they could change the to another static address then demote the address. I'm not too familiar with the ideological boundaries of controllers and spec, so let me know if this crosses the line. What if the service controller set to the created IP on first sync if the IP wasn't pre-specified? The controller could rely on loadBalancerIP to always indicate which IP to use. If the field is empty (blanked by user), then it's a signal to get a new IP.
By spec, I mean we should write down the careful reasoning that you just wrote in a permanent place :-)
would it then make sense for to be immutable on an already created service?
+1 +1 Was thinking the same. Unlike the many annotations we support, updates to this actual field could be validated and blocked at the apiserver... Changing IP is disruptive anyway. Could we simply ask the user to re-create the Service?
Of course, this could be an API-breaking change, and other cloud providers may actually be supporting this properly.
I just ran across this today, but in a more likely (IMO) case. 1) create the Service, type=LoadBalancer 2) let the IP be auto-allocated 3) manually promote that IP to static and give it a meaningful name 4) observe errors from service controller trying to allocate that IP Since my Service does not spec "loadBalancerIP", the question is, do we: a) use the IP, but leave it as static b) use the IP, but demote it back to ephemeral c) allocate and use a new IP C is completely hostile. B is subtly hostile. A seems correct. Can it be implemented? If we find that the IP exists but the name of it is not what we expect, we should treat it as user-provided and leave it alone. This is still nasty because it logs a failure in the GCP activities log for every loop, but I don't know how to avoid that without getting a list of IPs and scanning for the IP we want.
Filed for the looping.
+1 as long as we can rely on the "naming" of the address to decide whether the controller owns the address and can promote/demote it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Still relevant
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Still relevant
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
any update on this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.

if err != nil || !existed || ip != ipPrime { t.Fatalf(`ensureStaticIP(%v, %v, %v, %v, %v) = %v, %v, %v; want %v, true, nil`, gce, ipName, serviceName, gce.region, ip, ipPrime, existed, err, ip) } <ins> // Ensure call with different name ipName = "another-name-for-static-ip" ipPrime, existed, err = ensureStaticIP(gce, ipName, serviceName, gce.region, ip, cloud.NetworkTierDefault) if err != nil || !existed || ip != ipPrime { t.Fatalf(`ensureStaticIP(%v, %v, %v, %v, %v) = %v, %v, %v; want %v, true, nil`, gce, ipName, serviceName, gce.region, ip, ipPrime, existed, err, ip) } </ins> } func TestEnsureStaticIPWithTier(t *testing.T) {


Question:
When using IP aliases on GCE, use a secondary range rather than subnetwork to reserve cluster IPs.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals

--network ${NETWORK}  --region ${REGION}  --range ${NODE_IP_RANGE}  <del> --secondary-range "pods-default=${CLUSTER_IP_RANGE}" </del> <ins> --secondary-range "pods-default=${CLUSTER_IP_RANGE}"  --secondary-range "services-default=${SERVICE_CLUSTER_IP_RANGE}" </ins> echo "Created subnetwork ${IP_ALIAS_SUBNETWORK}" else if ! echo ${subnet} | grep --quiet secondaryIpRanges ${subnet}; then


Question:
When using IP aliases on GCE, use a secondary range rather than subnetwork to reserve cluster IPs.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals

exit 1 fi fi <del> # Services subnetwork. local subnet=$(gcloud beta compute networks subnets describe  --project "${PROJECT}"  --region ${REGION}  ${SERVICE_CLUSTER_IP_SUBNETWORK} 2>/dev/null) if [[ -z ${subnet} ]]; then if [[ ${SERVICE_CLUSTER_IP_SUBNETWORK} != ${INSTANCE_PREFIX}-subnet-services ]]; then echo "${color_red}Subnetwork ${NETWORK}:${SERVICE_CLUSTER_IP_SUBNETWORK} does not exist${color_norm}" exit 1 fi echo "Creating subnet for reserving service cluster IPs ${NETWORK}:${SERVICE_CLUSTER_IP_SUBNETWORK}" gcloud beta compute networks subnets create  ${SERVICE_CLUSTER_IP_SUBNETWORK}  --description "Automatically generated subnet for ${INSTANCE_PREFIX} cluster. This will be removed on cluster teardown."  --project "${PROJECT}"  --network ${NETWORK}  --region ${REGION}  --range ${SERVICE_CLUSTER_IP_RANGE} echo "Created subnetwork ${SERVICE_CLUSTER_IP_SUBNETWORK}" else echo "Subnet ${SERVICE_CLUSTER_IP_SUBNETWORK} already exists" fi </del> } function delete-firewall-rules() {


Question:
When using IP aliases on GCE, use a secondary range rather than subnetwork to reserve cluster IPs.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals

${IP_ALIAS_SUBNETWORK} fi fi <del> if [[ ${SERVICE_CLUSTER_IP_SUBNETWORK} == ${INSTANCE_PREFIX}-subnet-services ]]; then echo "Removing auto-created subnet ${NETWORK}:${SERVICE_CLUSTER_IP_SUBNETWORK}" if [[ -n $(gcloud beta compute networks subnets describe  --project "${PROJECT}"  --region ${REGION}  ${SERVICE_CLUSTER_IP_SUBNETWORK} 2>/dev/null) ]]; then gcloud --quiet beta compute networks subnets delete  --project "${PROJECT}"  --region ${REGION}  ${SERVICE_CLUSTER_IP_SUBNETWORK} fi fi </del> } # Generates SSL certificates for etcd cluster. Uses cfssl program.


Question:
Test Cases: [ ] Egress blocking based on cidr to a set of pods [ ] Egress blocking based on namespace to a set of pods [ ] Egress blocking based on a set of pods to another set of pods
Update: I'm working to try to enable these new features in a few CNI plug-ins so the tests can run locally.
Can we please move out of 1.8 milestone?
/label status/approved-for-milestone
/kind cleanup
/remove-kind bug
Those labels are not set on the issue: <detailsIn response to : /remove-kind bug Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind cleanup
/status approved-for-milestone /kind cleanup /remove-kind bug
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Adding tests, refactoring, fixing old bugs. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Can we please move out of 1.9 milestone? :)
this will be partially fixed with . I think it is safe to keep in the milestone.

testCannotConnect(f, f.Namespace, "client-b", service, 81) }) }) <ins> It("should allow egress access on one named port [Feature:NetworkPolicy]", func() { clientPodName := "client-a" protocolUDP := v1.ProtocolUDP policy := &networkingv1.NetworkPolicy{ ObjectMeta: metav1.ObjectMeta{ Name: "allow-client-a-via-named-port-egress-rule", }, Spec: networkingv1.NetworkPolicySpec{ // Apply this policy to client-a PodSelector: metav1.LabelSelector{ MatchLabels: map[string]string{ "pod-name": clientPodName, }, }, // Allow traffic to only one named port: "serve-80". Egress: []networkingv1.NetworkPolicyEgressRule{{ Ports: []networkingv1.NetworkPolicyPort{ { Port: &intstr.IntOrString{Type: intstr.String, StrVal: "serve-80"}, }, // Allow DNS look-ups { Protocol: &protocolUDP, Port:     &intstr.IntOrString{Type: intstr.Int, IntVal: 53}, }, }, }}, }, } policy, err := f.ClientSet.NetworkingV1().NetworkPolicies(f.Namespace.Name).Create(policy) Expect(err).NotTo(HaveOccurred()) defer cleanupNetworkPolicy(f, policy) By("Creating client-a which should be able to contact the server.", func() { testCanConnect(f, f.Namespace, clientPodName, service, 80) }) By("Creating client-a which should not be able to contact the server on port 81.", func() { testCannotConnect(f, f.Namespace, clientPodName, service, 81) }) }) </ins> }) })


Question:
bumped the vendored docker library and introduced a regression where pods with images that don't contain an explicit "" repo path result in failures. Results in the following error: Pod gets shows that the image has been pulled, though not by that name Changing the image to in the pod spec corrects the issue. I think the new code that is changing the behavior is somewhere in here is a new function brought in by the bump. If I checkout the commit before the bump, the problem does not occur. related:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
taking a look now
What docker version are you using?
Do you have any e2e tests that are failing? We run a lot of tests with image "busybox"
Didn't think about it before, but I guess 1.12 is the supported version for 1.8 eh? Let me try it on 1.12.
Hm, weird. We also have cluster e2e test coverage for 1.13. Many test are using directly from this function
I installed a completely fresh box with docker 1.12 and could still recreate. I have no explanation :confused: from kubelet built from upstream/release-1.8
This is one of our e2e test node log: You could find tons of . One thing I don't understand on your side is that, logs in our test are always: Why on your node it's ? Where does that image come from? What if you delete it?
does this mean that upgrades will fail if the local image names need to change?
Seems not. Based on the following code, it's fine to pull/inspect with new reference when we previously pulled with old reference. Could not reproduce this.
I could not reproduce with the code above either.
For any image which has no tag, kubelet will add a default tag , and default repo to it. So will change to and send to dockerd to pull. But dockerd knows and are the same thing, so should return right instead of not found. (as demo) Can you make sure your dockerd is the a version compatible with k8s 1.8?
Just ran into this. If you up the kubelet's log level to 4, you'll see a "not found" error from For example: If we look at the code in that helper function, we have this: Here is the result of inspecting the image on my system (only showing relevant fields): You'll notice that the RepoTag it has is for , not . I am guessing the fact that we're somewhere inserting "library" into the image's name is what's tripping this up.
If it wasn't clear, I believe dockershim is what's having the issue. Once the kubelet sends the protobuf message to the dockershim socket to pull the image, it goes from to to to and then into the helper function I listed above.
It appears to be this: to to to That's where the "library" is coming from.
Prior to the PR that bumped docker/distribution, used to have , which does not do any normalization (i.e. including "library").
OK, I see why when we bumped to the newer docker/distribution, we switch to using . It's because now requires that the input be in "canonical" format (in this case, has and , if applicable), which you get automatically by calling ...
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
could you give steps to reproduce this? We can't reproduce this and many of the node/cluster e2e tests pull images using unqualified name such as "buxybox"
What docker distribution are you using? I could not reproduce this myself. I don't understand how does the reference show up, we never pull image with it. Is this a problem specific to a docker distribution?
run the e2e tests on a centos/fedora/rhel image
Is there any signal in test-grid for centos/fedora/rhel? Would be nice to catch issues like this sooner than later.
run the e2e tests on a centos/fedora/rhel image there is not, and definitely should be, given that k8s releases rpms as part of each release. will be on the short list for the release retrospective.
I feel like this is a problem in docker distribution in centos/fedora/rhel. Docker should not return when we are using to pull image. Should we fix the corresponding docker distribution instead of blocking Kubernete release?
Since that would mean releasing a new version would regress existing users, we generally say that we'll do our best to tolerate it instead of releasing something known broken. We've done that with various versions of Docker, systemd, and iptables for years, and I'm sure we'll do our best to do that. I do agree that this looks like a bug in that docker distribution that should be fixed. I also agree that this should have been in the test grid and that the distro owner is responsible for ensuring the project has that signal.
Thanks! I agree that we should not break existing users. I'm fine with as long as we keep in mind we made this change, and clean it up in the future. I think it would be helpful if we could have test coverage in test-grid on more distros, so that we could have caught similar problems earlier.
Agree - I think anyone who causes a release slip is required to make sure they fix the problem so it can't happen in the future. And get better all around testing. On Sep 27, 2017, at 7:40 PM, Lantao Liu <: ! I agree that we should not break existing users. I'm fine with long as we keep in mind we made this change, and clean it up in the future. I think it would be helpful if we could have test coverage on more distros, so that we could have caught similar problems earlier. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
I have opened a bug for this
This has broken the CRI as well. Kubernetes should not normalize image names on behalf of the CRI runtime. It's up to implementors to decide how to handle an image name and since there's not a standard around Kube and Image names, I'd prefer not to break the CRI.
Are you saying it's currently broken, or the fix was ok? , Antonio Murdaca <:
The fix is ok but the original patch caused another bug
Which one?
caused a bug fixed by
Does this problem fixed?

// hostname or not, we only check for the suffix match. if strings.HasSuffix(image, tag) || strings.HasSuffix(tag, image) { return true <ins> } else { // TODO: We need to remove this hack when project atomic based // docker distro(s) like centos/fedora/rhel image fix problems on // their end. // Say the tag is "docker.io/busybox:latest" // and the image is "docker.io/library/busybox:latest" t, err := dockerref.ParseNormalizedNamed(tag) if err != nil { continue } // the parsed/normalized tag will look like // reference.taggedReference { // 	 namedRepository: reference.repository { // 	   domain: "docker.io", // 	   path: "library/busybox" //	}, // 	tag: "latest" // } // If it does not have tags then we bail out t2, ok := t.(dockerref.Tagged) if !ok { continue } // normalized tag would look like "docker.io/library/busybox:latest" // note the library get added in the string normalizedTag := t2.String() if normalizedTag == "" { continue } if strings.HasSuffix(image, normalizedTag) || strings.HasSuffix(normalizedTag, image) { return true } </ins> } } }