Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pods with volumeMount gitRepo volumes in 1.8.0 see the following error in kubelet logs (also in ) It is important to not that the git repo clone is done via ssh. I think what is happening is that hyperkube image now used to have command in it and now it does not have it. So the git clone command that kubelet runs fails with the above stacktrace since it cannot find . What you expected to happen: gitRepo is volume mounted. How to reproduce it (as minimally and precisely as possible): Reproducible in kubernetes 1.8.0+ hyperkube images. Anything else we need to know?: This probably is due to missing ssh binary in the new 1.8.0 hyperkube images. The commit might be an issue. Will comment further if I find otherwise. Environment: Kubernetes version (use ): =1.8.0 Cloud provider or hardware configuration**: Not Application. OS (e.g. from /etc/os-release): hyperkube base image Kernel (e.g. ): Install tools: Others:
How did you installed Kubernetes may I ask?
how does this help?
Normally, it is suggest to install kubelet as a daemon on host instead of using hyperkube container. cc you may be interested in this issue.
Need your help in identifying the fix here. I can send something for review. Suggestions are welcome.
The problem is that ssh isn't installed inside of the container. I don't know if it should be either, as the hyperkube image grows so large. If you install the openssh-client, it should work
Yes I did identify that as the problem if you see the description I mentioned. I pointed to a git commit that might have caused this. I am not sure what package installation has changed in the hyperkube container. "ssh" used to be available and now it is not. For the gitrepo feature to work it should be available.
Ah okay. If it was there before, we might want to add it back. SGTM.
Yeah, I switched the hyperkube image base from to our with some additional dependencies back. Apparently due to lack of volume + hyperkube test coverage, I've missed a number of dependencies needed. cc
we could use dropbear as a replacement for just the ssh client?
I would prefer to use the openssh client over the dropbear client. It has wider usage and a stronger security track record IMO. The dropbear client also has somewhat different usage and configuration, so arguably using it is still a regression over the previous state. Saving a couple MB isn't worth using worse utilities, however much fun "dockerfile-golfing" is.
to fix
cherrypick to release-1.8:
This should be fixed in v1.8.3.

ARCH?=amd64 HYPERKUBE_BIN?=_output/dockerized/bin/linux/$(ARCH)/hyperkube <del> BASEIMAGE=gcr.io/google-containers/debian-hyperkube-base-$(ARCH):0.5 </del> <ins> BASEIMAGE=gcr.io/google-containers/debian-hyperkube-base-$(ARCH):0.6 </ins> TEMP_DIR:=$(shell mktemp -d -t hyperkubeXXXXXX) all: build


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I'm running a Redis pod with a PV provisioned as an Azure Managed Disk. It's got an exec readiness probe that calls in the container. At some point it receives a SIGTERM. The kubelet then tries to sync the pod and restart the container, but it panics in (looking over the code, that's the only place where the error from is ignored) As a result, the pod hangs in a state, since the readiness probe cannot be ran in a non existing container. What you expected to happen: The container would successfully restart. How to reproduce it (as minimally and precisely as possible): It's the 2nd time it happens but I haven't yet figured out the conditions to reproduce. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: Azure w/ Managed Disks OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools:Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi could you also share the storageclass, pvc & pod yaml files? I would like to repro, thx
I may insert a nil check statment in this func: But I need to reproduce this bug anyway, so that I could find the root cause of this issue, thx.
Yes, an error check is what I'm hoping for :) I've used the chart. The PV/PVC were created manually, they're set to an empty storage class. I should mention that I've also noticed other pods being terminated/recreated at the time this one failed, however I didn't trace the cause of this yet.
thanks for the reporting, I may have found the root cause, just need a few more debuging time, thx.
GitHub didn't allow me to assign the following users: xiazhang. Note that only can be assigned. <detailsIn response to : /assign xiazhang Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign andyzhangx
- Thanks for the fix!

var _ volume.Mounter = &azureDiskMounter{} func (m *azureDiskMounter) GetAttributes() volume.Attributes { <del> volumeSource, _ := getVolumeSource(m.spec) </del> <ins> readOnly := false volumeSource, err := getVolumeSource(m.spec) if err != nil && volumeSource.ReadOnly != nil { readOnly = *volumeSource.ReadOnly } </ins> return volume.Attributes{ <del> ReadOnly:        *volumeSource.ReadOnly, Managed:         !*volumeSource.ReadOnly, </del> <ins> ReadOnly:        readOnly, Managed:         !readOnly, </ins> SupportsSELinux: true, } }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I'm running a Redis pod with a PV provisioned as an Azure Managed Disk. It's got an exec readiness probe that calls in the container. At some point it receives a SIGTERM. The kubelet then tries to sync the pod and restart the container, but it panics in (looking over the code, that's the only place where the error from is ignored) As a result, the pod hangs in a state, since the readiness probe cannot be ran in a non existing container. What you expected to happen: The container would successfully restart. How to reproduce it (as minimally and precisely as possible): It's the 2nd time it happens but I haven't yet figured out the conditions to reproduce. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: Azure w/ Managed Disks OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools:Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi could you also share the storageclass, pvc & pod yaml files? I would like to repro, thx
I may insert a nil check statment in this func: But I need to reproduce this bug anyway, so that I could find the root cause of this issue, thx.
Yes, an error check is what I'm hoping for :) I've used the chart. The PV/PVC were created manually, they're set to an empty storage class. I should mention that I've also noticed other pods being terminated/recreated at the time this one failed, however I didn't trace the cause of this yet.
thanks for the reporting, I may have found the root cause, just need a few more debuging time, thx.
GitHub didn't allow me to assign the following users: xiazhang. Note that only can be assigned. <detailsIn response to : /assign xiazhang Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign andyzhangx
- Thanks for the fix!

options := []string{"bind"} <del> if *volumeSource.ReadOnly { </del> <ins> if volumeSource.ReadOnly != nil && *volumeSource.ReadOnly { </ins> options = append(options, "ro") }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I'm running a Redis pod with a PV provisioned as an Azure Managed Disk. It's got an exec readiness probe that calls in the container. At some point it receives a SIGTERM. The kubelet then tries to sync the pod and restart the container, but it panics in (looking over the code, that's the only place where the error from is ignored) As a result, the pod hangs in a state, since the readiness probe cannot be ran in a non existing container. What you expected to happen: The container would successfully restart. How to reproduce it (as minimally and precisely as possible): It's the 2nd time it happens but I haven't yet figured out the conditions to reproduce. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: Azure w/ Managed Disks OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools:Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi could you also share the storageclass, pvc & pod yaml files? I would like to repro, thx
I may insert a nil check statment in this func: But I need to reproduce this bug anyway, so that I could find the root cause of this issue, thx.
Yes, an error check is what I'm hoping for :) I've used the chart. The PV/PVC were created manually, they're set to an empty storage class. I should mention that I've also noticed other pods being terminated/recreated at the time this one failed, however I didn't trace the cause of this yet.
thanks for the reporting, I may have found the root cause, just need a few more debuging time, thx.
GitHub didn't allow me to assign the following users: xiazhang. Note that only can be assigned. <detailsIn response to : /assign xiazhang Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign andyzhangx
- Thanks for the fix!

return mountErr } <del> if !*volumeSource.ReadOnly { </del> <ins> if volumeSource.ReadOnly == nil || !*volumeSource.ReadOnly { </ins> volume.SetVolumeOwnership(m, fsGroup) }


Question:
Our 5k-node cluster jobs recently started failing even before the e2e tests start. The reason is the step which checks that all kube-system pods are running is timing out: cc
From the scheduler logs I'm seeing that the pods are being scheduled quite slowly, getting worse with time: 2-3 pods/min looks quite bad. cc
Also finding such traces: Seems like most of the time is being spent in priority calculation.
From the pod spec, I'm seeing soft inter-pod anti-affinity at node-level topology: I'm almost sure this is the one causing the mischief. We've been having performance issues with anti-affinity traditionally :p
Ok... PR is the cause. It introduced anti-affinity to kube-dns. I think we should revert this change given known issues with anti-affinity. - Opinions?
FYI - it is taking 80m until the last pod is scheduled when I'm running 5k-node cluster locally.
That's quite unfortunate. It looks like we should revert the change until the anti-affinity change is scalable enough to be used more broadly, unless the scheduler folks have a target fix for this use case.
cc
We are very well aware of the performance issues in our affinity/anti-affinity functions, but I am not aware of a concrete solution at this point. This is partly due to the flexible design of affinity/anti-affinity concept. My humble opinion at this point is to drop some of the flexibility as they are rarely used, in order to streamline the algorithm and improve performance. Anyway, I think the proper solution to this problem needs more time than your requirements to address the issue in hand. So, you may want to change your configuration and drop the anti-affinity rule.
would it make sense to add a note to the anti-affinity user documentation to hint at performance problems at large scale, until this is fixed?
Re-opening this issue as this problem has struck again. The following runs of our 5k-node scalability jobs failed to bring up the cluster on time: https://k8s- https://k8s- https://k8s- https://k8s- And the reason is kube-dns pods pending to be scheduled for a long time:
These failures started happening about 4 days ago and I'm pretty sure it's this one: Add self anti-affinity to kube-dns pods Sadly, we'll have to revert it again for now :( cc - Seems like the improvements to anti-affinity performance made under weren't enough?
We need to find other improvements for the predicate. If we cannot find major improvement opportunities, we may have to change its design to be more scalable at the cost of the reducing its flexibility.
- my personal opinion is that we need to change the design anyway. Having it work both ways (i.e. existing pods everywhere impacting placing of our pod) will always be painful (especially for things like cluster autoscaler)
I agree. The symetrical application of anti-affinity is problematic. I also think we need to change it.
Great. - do you agree with that too?
, I'm wondering after some affinity/anti-affinity optimization in scheduler, do you think if it's fine to test coreDNS with anti-affinity policy in large cluster with 5000 nodes?
We probably should try to see where we are - but this requires testing before enabling that.
I agree with - Could we first try checking with a benchmark integration test () as a proxy for real cluster test?
I'm wondering how many real nodes are required for testing?
Since the integration tests fake out the nodes, I think it's safe to crank it up to 5000 (that's the current largest integration test benchmark size) I'm also hoping to refactor our benchmarks soon, so let me know if you have any thoughts about them.

annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: <del> affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: k8s-app operator: In values: ["kube-dns"] topologyKey: kubernetes.io/hostname </del> tolerations: - key: "CriticalAddonsOnly" operator: "Exists"


Question:
Our 5k-node cluster jobs recently started failing even before the e2e tests start. The reason is the step which checks that all kube-system pods are running is timing out: cc
From the scheduler logs I'm seeing that the pods are being scheduled quite slowly, getting worse with time: 2-3 pods/min looks quite bad. cc
Also finding such traces: Seems like most of the time is being spent in priority calculation.
From the pod spec, I'm seeing soft inter-pod anti-affinity at node-level topology: I'm almost sure this is the one causing the mischief. We've been having performance issues with anti-affinity traditionally :p
Ok... PR is the cause. It introduced anti-affinity to kube-dns. I think we should revert this change given known issues with anti-affinity. - Opinions?
FYI - it is taking 80m until the last pod is scheduled when I'm running 5k-node cluster locally.
That's quite unfortunate. It looks like we should revert the change until the anti-affinity change is scalable enough to be used more broadly, unless the scheduler folks have a target fix for this use case.
cc
We are very well aware of the performance issues in our affinity/anti-affinity functions, but I am not aware of a concrete solution at this point. This is partly due to the flexible design of affinity/anti-affinity concept. My humble opinion at this point is to drop some of the flexibility as they are rarely used, in order to streamline the algorithm and improve performance. Anyway, I think the proper solution to this problem needs more time than your requirements to address the issue in hand. So, you may want to change your configuration and drop the anti-affinity rule.
would it make sense to add a note to the anti-affinity user documentation to hint at performance problems at large scale, until this is fixed?
Re-opening this issue as this problem has struck again. The following runs of our 5k-node scalability jobs failed to bring up the cluster on time: https://k8s- https://k8s- https://k8s- https://k8s- And the reason is kube-dns pods pending to be scheduled for a long time:
These failures started happening about 4 days ago and I'm pretty sure it's this one: Add self anti-affinity to kube-dns pods Sadly, we'll have to revert it again for now :( cc - Seems like the improvements to anti-affinity performance made under weren't enough?
We need to find other improvements for the predicate. If we cannot find major improvement opportunities, we may have to change its design to be more scalable at the cost of the reducing its flexibility.
- my personal opinion is that we need to change the design anyway. Having it work both ways (i.e. existing pods everywhere impacting placing of our pod) will always be painful (especially for things like cluster autoscaler)
I agree. The symetrical application of anti-affinity is problematic. I also think we need to change it.
Great. - do you agree with that too?
, I'm wondering after some affinity/anti-affinity optimization in scheduler, do you think if it's fine to test coreDNS with anti-affinity policy in large cluster with 5000 nodes?
We probably should try to see where we are - but this requires testing before enabling that.
I agree with - Could we first try checking with a benchmark integration test () as a proxy for real cluster test?
I'm wondering how many real nodes are required for testing?
Since the integration tests fake out the nodes, I think it's safe to crank it up to 5000 (that's the current largest integration test benchmark size) I'm also hoping to refactor our benchmarks soon, so let me know if you have any thoughts about them.


Question:
BUG REPORT: /kind bug /sig aws What happened: Adding this annotation to an existing service: did not trigger an update on AWS ELB (no new tags). What you expected to happen: Expected to see those 2 new tags under the existing ELB: How to reproduce it (as minimally and precisely as possible): a service without annotation the annotation tags on AWS Anything else we need to know?: Creating a new service WITH the annotation will create the expected tags on AWS. Environment: Kubernetes version (use ): 1.7.9 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): Linux ip-xxxxx 4.4.78-k8s SMP Fri Jul 28 01:28:39 UTC 2017 x86_64 GNU/Linux Install tools: Others: Using Kops
+1
+1
+1
+1
+1
I've done some work on a fix for this, will try and get a PR submitted later today.

CreateLoadBalancer(*elb.CreateLoadBalancerInput) (*elb.CreateLoadBalancerOutput, error) DeleteLoadBalancer(*elb.DeleteLoadBalancerInput) (*elb.DeleteLoadBalancerOutput, error) DescribeLoadBalancers(*elb.DescribeLoadBalancersInput) (*elb.DescribeLoadBalancersOutput, error) <ins> AddTags(*elb.AddTagsInput) (*elb.AddTagsOutput, error) </ins> RegisterInstancesWithLoadBalancer(*elb.RegisterInstancesWithLoadBalancerInput) (*elb.RegisterInstancesWithLoadBalancerOutput, error) DeregisterInstancesFromLoadBalancer(*elb.DeregisterInstancesFromLoadBalancerInput) (*elb.DeregisterInstancesFromLoadBalancerOutput, error) CreateLoadBalancerPolicy(*elb.CreateLoadBalancerPolicyInput) (*elb.CreateLoadBalancerPolicyOutput, error)


Question:
BUG REPORT: /kind bug /sig aws What happened: Adding this annotation to an existing service: did not trigger an update on AWS ELB (no new tags). What you expected to happen: Expected to see those 2 new tags under the existing ELB: How to reproduce it (as minimally and precisely as possible): a service without annotation the annotation tags on AWS Anything else we need to know?: Creating a new service WITH the annotation will create the expected tags on AWS. Environment: Kubernetes version (use ): 1.7.9 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): Linux ip-xxxxx 4.4.78-k8s SMP Fri Jul 28 01:28:39 UTC 2017 x86_64 GNU/Linux Install tools: Others: Using Kops
+1
+1
+1
+1
+1
I've done some work on a fix for this, will try and get a PR submitted later today.

return ret, nil } <ins> func (c *Cloud) addLoadBalancerTags(loadBalancerName string, requested map[string]string) error { var tags []*elb.Tag for k, v := range requested { tag := &elb.Tag{ Key:   aws.String(k), Value: aws.String(v), } tags = append(tags, tag) } request := &elb.AddTagsInput{} request.LoadBalancerNames = []*string{&loadBalancerName} request.Tags = tags _, err := c.elb.AddTags(request) if err != nil { return fmt.Errorf("error adding tags to load balancer: %v", err) } return nil } </ins> // Retrieves instance's vpc id from metadata func (c *Cloud) findVPCID() (string, error) { macs, err := c.metadata.GetMetadata("network/interfaces/macs/")


Question:
BUG REPORT: /kind bug /sig aws What happened: Adding this annotation to an existing service: did not trigger an update on AWS ELB (no new tags). What you expected to happen: Expected to see those 2 new tags under the existing ELB: How to reproduce it (as minimally and precisely as possible): a service without annotation the annotation tags on AWS Anything else we need to know?: Creating a new service WITH the annotation will create the expected tags on AWS. Environment: Kubernetes version (use ): 1.7.9 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): Linux ip-xxxxx 4.4.78-k8s SMP Fri Jul 28 01:28:39 UTC 2017 x86_64 GNU/Linux Install tools: Others: Using Kops
+1
+1
+1
+1
+1
I've done some work on a fix for this, will try and get a PR submitted later today.

panic("Not implemented") } <ins> func (elb *FakeELB) AddTags(input *elb.AddTagsInput) (*elb.AddTagsOutput, error) { panic("Not implemented") } </ins> func (elb *FakeELB) RegisterInstancesWithLoadBalancer(*elb.RegisterInstancesWithLoadBalancerInput) (*elb.RegisterInstancesWithLoadBalancerOutput, error) { panic("Not implemented") }


Question:
BUG REPORT: /kind bug /sig aws What happened: Adding this annotation to an existing service: did not trigger an update on AWS ELB (no new tags). What you expected to happen: Expected to see those 2 new tags under the existing ELB: How to reproduce it (as minimally and precisely as possible): a service without annotation the annotation tags on AWS Anything else we need to know?: Creating a new service WITH the annotation will create the expected tags on AWS. Environment: Kubernetes version (use ): 1.7.9 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): Linux ip-xxxxx 4.4.78-k8s SMP Fri Jul 28 01:28:39 UTC 2017 x86_64 GNU/Linux Install tools: Others: Using Kops
+1
+1
+1
+1
+1
I've done some work on a fix for this, will try and get a PR submitted later today.

} } } <ins> { // Add additional tags glog.V(2).Infof("Creating additional load balancer tags for %s", loadBalancerName) tags := getLoadBalancerAdditionalTags(annotations) if len(tags) > 0 { err := c.addLoadBalancerTags(loadBalancerName, tags) if err != nil { return nil, fmt.Errorf("unable to create additional load balancer tags: %v", err) } } } </ins> } // Whether the ELB was new or existing, sync attributes regardless. This accounts for things


Question:
BUG REPORT: /kind bug /sig aws What happened: Adding this annotation to an existing service: did not trigger an update on AWS ELB (no new tags). What you expected to happen: Expected to see those 2 new tags under the existing ELB: How to reproduce it (as minimally and precisely as possible): a service without annotation the annotation tags on AWS Anything else we need to know?: Creating a new service WITH the annotation will create the expected tags on AWS. Environment: Kubernetes version (use ): 1.7.9 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): Linux ip-xxxxx 4.4.78-k8s SMP Fri Jul 28 01:28:39 UTC 2017 x86_64 GNU/Linux Install tools: Others: Using Kops
+1
+1
+1
+1
+1
I've done some work on a fix for this, will try and get a PR submitted later today.

}) } <ins> func (m *MockedFakeELB) AddTags(input *elb.AddTagsInput) (*elb.AddTagsOutput, error) { args := m.Called(input) return args.Get(0).(*elb.AddTagsOutput), nil } </ins> func TestReadAWSCloudConfig(t *testing.T) { tests := []struct { name string


Question:
BUG REPORT: /kind bug /sig aws What happened: Adding this annotation to an existing service: did not trigger an update on AWS ELB (no new tags). What you expected to happen: Expected to see those 2 new tags under the existing ELB: How to reproduce it (as minimally and precisely as possible): a service without annotation the annotation tags on AWS Anything else we need to know?: Creating a new service WITH the annotation will create the expected tags on AWS. Environment: Kubernetes version (use ): 1.7.9 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): Linux ip-xxxxx 4.4.78-k8s SMP Fri Jul 28 01:28:39 UTC 2017 x86_64 GNU/Linux Install tools: Others: Using Kops
+1
+1
+1
+1
+1
I've done some work on a fix for this, will try and get a PR submitted later today.

} } <ins> // Test that we can add a load balancer tag func TestAddLoadBalancerTags(t *testing.T) { loadBalancerName := "test-elb" awsServices := newMockedFakeAWSServices(TestClusterId) c, _ := newAWSCloud(strings.NewReader("[global]"), awsServices) want := make(map[string]string) want["tag1"] = "val1" expectedAddTagsRequest := &elb.AddTagsInput{ LoadBalancerNames: []*string{&loadBalancerName}, Tags: []*elb.Tag{ { Key:   aws.String("tag1"), Value: aws.String("val1"), }, }, } awsServices.elb.(*MockedFakeELB).On("AddTags", expectedAddTagsRequest).Return(&elb.AddTagsOutput{}) err := c.addLoadBalancerTags(loadBalancerName, want) assert.Nil(t, err, "Error adding load balancer tags: %v", err) awsServices.elb.(*MockedFakeELB).AssertExpectations(t) } </ins> func newMockedFakeAWSServices(id string) *FakeAWSServices { s := NewFakeAWSServices(id) s.ec2 = &MockedFakeEC2{FakeEC2Impl: s.ec2.(*FakeEC2Impl)}


Question:
On AWS, mutable API calls such as AttachVolume or DeleteVolume have separate API quota from API calls that do not modify the resource. Currently k8s makes a high number of AttachVolume and DeleteVolume API calls even though k8s can check if the operation is going to succeed.
/sig aws

ec2Device := "/dev/xvd" + string(mountDevice) if !alreadyAttached { <ins> available, err := c.checkIfAvailable(disk, "attaching", awsInstance.awsID) if !available { attachEnded = true return "", err } </ins> request := &ec2.AttachVolumeInput{ Device:     aws.String(ec2Device), InstanceId: aws.String(awsInstance.awsID),


Question:
On AWS, mutable API calls such as AttachVolume or DeleteVolume have separate API quota from API calls that do not modify the resource. Currently k8s makes a high number of AttachVolume and DeleteVolume API calls even though k8s can check if the operation is going to succeed.
/sig aws

if err != nil { return false, err } <ins> available, err := c.checkIfAvailable(awsDisk, "deleting", "") if !available { return false, err } </ins> return awsDisk.deleteVolume() } <ins> func (c *Cloud) checkIfAvailable(disk *awsDisk, opName string, instance string) (bool, error) { info, err := disk.describeVolume() if err != nil { glog.Errorf("Error describing volume %q: %q", disk.awsID, err) // if for some reason we can not describe volume we will return error return false, err } volumeState := aws.StringValue(info.State) opError := fmt.Sprintf("Error %s EBS volume %q", opName, disk.awsID) if len(instance) != 0 { opError = fmt.Sprintf("%q to instance %q", opError, instance) } // Only available volumes can be attached or deleted if volumeState != "available" { // Volume is attached somewhere else and we can not attach it here if len(info.Attachments) > 0 { attachment := info.Attachments[0] attachErr := fmt.Errorf("%s since volume is currently attached to %q", opError, aws.StringValue(attachment.InstanceId)) glog.Error(attachErr) return false, attachErr } attachErr := fmt.Errorf("%s since volume is in %q state", opError, volumeState) glog.Error(attachErr) return false, attachErr } return true, nil } </ins> func (c *Cloud) GetLabelsForVolume(pv *v1.PersistentVolume) (map[string]string, error) { // Ignore any volumes that are being provisioned if pv.Spec.AWSElasticBlockStore.VolumeID == volume.ProvisionedVolumeName {


Question:
We should make the CRI log parsing code in kubelet to be a library. So that container runtimes and external tools don't need to copy the whole code ( vendor it instead ). Referring
/assign
/sig node
I can help on it.
/assign
GitHub didn't allow me to assign the following users: yanxuean. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks. I have a branch ready for this. Will open a PR later and ping you when it is out.
ok

"//pkg/kubelet/container:go_default_library", "//pkg/kubelet/events:go_default_library", "//pkg/kubelet/images:go_default_library", <ins> "//pkg/kubelet/kuberuntime/logs:go_default_library", </ins> "//pkg/kubelet/lifecycle:go_default_library", "//pkg/kubelet/metrics:go_default_library", "//pkg/kubelet/prober/results:go_default_library",


Question:
We should make the CRI log parsing code in kubelet to be a library. So that container runtimes and external tools don't need to copy the whole code ( vendor it instead ). Referring
/assign
/sig node
I can help on it.
/assign
GitHub didn't allow me to assign the following users: yanxuean. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks. I have a branch ready for this. Will open a PR later and ping you when it is out.
ok

"//pkg/util/tail:go_default_library", "//pkg/util/version:go_default_library", "//vendor/github.com/armon/circbuf:go_default_library", <del> "//vendor/github.com/docker/docker/pkg/jsonlog:go_default_library", "//vendor/github.com/fsnotify/fsnotify:go_default_library", </del> "//vendor/github.com/golang/glog:go_default_library", "//vendor/github.com/google/cadvisor/info/v1:go_default_library", "//vendor/google.golang.org/grpc:go_default_library",


Question:
We should make the CRI log parsing code in kubelet to be a library. So that container runtimes and external tools don't need to copy the whole code ( vendor it instead ). Referring
/assign
/sig node
I can help on it.
/assign
GitHub didn't allow me to assign the following users: yanxuean. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks. I have a branch ready for this. Will open a PR later and ping you when it is out.
ok

"kuberuntime_container_test.go", "kuberuntime_gc_test.go", "kuberuntime_image_test.go", <del> "kuberuntime_logs_test.go", </del> "kuberuntime_manager_test.go", "kuberuntime_sandbox_test.go", "labels_test.go",


Question:
We should make the CRI log parsing code in kubelet to be a library. So that container runtimes and external tools don't need to copy the whole code ( vendor it instead ). Referring
/assign
/sig node
I can help on it.
/assign
GitHub didn't allow me to assign the following users: yanxuean. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks. I have a branch ready for this. Will open a PR later and ping you when it is out.
ok

filegroup( name = "all-srcs", <del> srcs = [":package-srcs"], </del> <ins> srcs = [ ":package-srcs", "//pkg/kubelet/kuberuntime/logs:all-srcs", ], </ins> tags = ["automanaged"], )


Question:
We should make the CRI log parsing code in kubelet to be a library. So that container runtimes and external tools don't need to copy the whole code ( vendor it instead ). Referring
/assign
/sig node
I can help on it.
/assign
GitHub didn't allow me to assign the following users: yanxuean. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks. I have a branch ready for this. Will open a PR later and ping you when it is out.
ok

package kuberuntime import ( <del> "bufio" "bytes" "encoding/json" "errors" "fmt" </del> "io" <del> "math" "os" </del> "time" <del> "github.com/docker/docker/pkg/jsonlog" "github.com/fsnotify/fsnotify" "github.com/golang/glog" </del> "k8s.io/api/core/v1" <del> runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/v1alpha1/runtime" "k8s.io/kubernetes/pkg/util/tail" </del> <ins> "k8s.io/kubernetes/pkg/kubelet/kuberuntime/logs" </ins> ) <del> // Notice that the current kuberuntime logs implementation doesn't handle // log rotation. // * It will not retrieve logs in rotated log file. // * If log rotation happens when following the log: //   * If the rotation is using create mode, we'll still follow the old file. //   * If the rotation is using copytruncate, we'll be reading at the original position and get nothing. // TODO(random-liu): Support log rotation. // streamType is the type of the stream. type streamType string const ( stderrType streamType = "stderr" stdoutType streamType = "stdout" // timeFormat is the time format used in the log. timeFormat = time.RFC3339Nano // blockSize is the block size used in tail. blockSize = 1024 // stateCheckPeriod is the period to check container state while following // the container log. Kubelet should not keep following the log when the // container is not running. stateCheckPeriod = 5 * time.Second ) var ( // eol is the end-of-line sign in the log. eol = []byte{'n'} // delimiter is the delimiter for timestamp and streamtype in log line. delimiter = []byte{' '} ) // logMessage is the internal log type. type logMessage struct { timestamp time.Time stream    streamType log       []byte } // reset resets the log to nil. func (l *logMessage) reset() { l.timestamp = time.Time{} l.stream = "" l.log = nil } // logOptions is the internal type of all log options. type logOptions struct { tail      int64 bytes     int64 since     time.Time follow    bool timestamp bool } // newLogOptions convert the v1.PodLogOptions to internal logOptions. func newLogOptions(apiOpts *v1.PodLogOptions, now time.Time) *logOptions { opts := &logOptions{ tail:      -1, // -1 by default which means read all logs. bytes:     -1, // -1 by default which means read all logs. follow:    apiOpts.Follow, timestamp: apiOpts.Timestamps, } if apiOpts.TailLines != nil { opts.tail = *apiOpts.TailLines } if apiOpts.LimitBytes != nil { opts.bytes = *apiOpts.LimitBytes } if apiOpts.SinceSeconds != nil { opts.since = now.Add(-time.Duration(*apiOpts.SinceSeconds) * time.Second) } if apiOpts.SinceTime != nil && apiOpts.SinceTime.After(opts.since) { opts.since = apiOpts.SinceTime.Time } return opts } </del> // ReadLogs read the container log and redirect into stdout and stderr. // Note that containerID is only needed when following the log, or else // just pass in empty string "". func (m *kubeGenericRuntimeManager) ReadLogs(path, containerID string, apiOpts *v1.PodLogOptions, stdout, stderr io.Writer) error { <del> f, err := os.Open(path) if err != nil { return fmt.Errorf("failed to open log file %q: %v", path, err) } defer f.Close() </del> // Convert v1.PodLogOptions into internal log options. <del> opts := newLogOptions(apiOpts, time.Now()) // Search start point based on tail line. start, err := tail.FindTailLineStartIndex(f, opts.tail) if err != nil { return fmt.Errorf("failed to tail %d lines of log file %q: %v", opts.tail, path, err) } if _, err := f.Seek(start, os.SEEK_SET); err != nil { return fmt.Errorf("failed to seek %d in log file %q: %v", start, path, err) } // Start parsing the logs. r := bufio.NewReader(f) // Do not create watcher here because it is not needed if `Follow` is false. var watcher *fsnotify.Watcher var parse parseFunc writer := newLogWriter(stdout, stderr, opts) msg := &logMessage{} for { l, err := r.ReadBytes(eol[0]) if err != nil { if err != io.EOF { // This is an real error return fmt.Errorf("failed to read log file %q: %v", path, err) } if !opts.follow { // Return directly when reading to the end if not follow. if len(l) > 0 { glog.Warningf("Incomplete line in log file %q: %q", path, l) } glog.V(2).Infof("Finish parsing log file %q", path) return nil } // Reset seek so that if this is an incomplete line, // it will be read again. if _, err := f.Seek(-int64(len(l)), os.SEEK_CUR); err != nil { return fmt.Errorf("failed to reset seek in log file %q: %v", path, err) } if watcher == nil { // Intialize the watcher if it has not been initialized yet. if watcher, err = fsnotify.NewWatcher(); err != nil { return fmt.Errorf("failed to create fsnotify watcher: %v", err) } defer watcher.Close() if err := watcher.Add(f.Name()); err != nil { return fmt.Errorf("failed to watch file %q: %v", f.Name(), err) } } // Wait until the next log change. if found, err := m.waitLogs(containerID, watcher); !found { return err } continue } if parse == nil { // Intialize the log parsing function. parse, err = getParseFunc(l) if err != nil { return fmt.Errorf("failed to get parse function: %v", err) } } // Parse the log line. msg.reset() if err := parse(l, msg); err != nil { glog.Errorf("Failed with err %v when parsing log for log file %q: %q", err, path, l) continue } // Write the log line into the stream. if err := writer.write(msg); err != nil { if err == errMaximumWrite { glog.V(2).Infof("Finish parsing log file %q, hit bytes limit %d(bytes)", path, opts.bytes) return nil } glog.Errorf("Failed with err %v when writing log for log file %q: %+v", err, path, msg) return err } } } // waitLogs wait for the next log write. It returns a boolean and an error. The boolean // indicates whether a new log is found; the error is error happens during waiting new logs. func (m *kubeGenericRuntimeManager) waitLogs(id string, w *fsnotify.Watcher) (bool, error) { errRetry := 5 for { select { case e := <-w.Events: switch e.Op { case fsnotify.Write: return true, nil default: glog.Errorf("Unexpected fsnotify event: %v, retrying...", e) } case err := <-w.Errors: glog.Errorf("Fsnotify watch error: %v, %d error retries remaining", err, errRetry) if errRetry == 0 { return false, err } errRetry-- case <-time.After(stateCheckPeriod): s, err := m.runtimeService.ContainerStatus(id) if err != nil { return false, err } // Only keep following container log when it is running. if s.State != runtimeapi.ContainerState_CONTAINER_RUNNING { glog.Errorf("Container %q is not running (state=%q)", id, s.State) // Do not return error because it's normal that the container stops // during waiting. return false, nil } } } } // parseFunc is a function parsing one log line to the internal log type. // Notice that the caller must make sure logMessage is not nil. type parseFunc func([]byte, *logMessage) error var parseFuncs []parseFunc = []parseFunc{ parseCRILog,        // CRI log format parse function parseDockerJSONLog, // Docker JSON log format parse function } // parseCRILog parses logs in CRI log format. CRI Log format example: //   2016-10-06T00:17:09.669794202Z stdout log content 1 //   2016-10-06T00:17:09.669794203Z stderr log content 2 func parseCRILog(log []byte, msg *logMessage) error { var err error // Parse timestamp idx := bytes.Index(log, delimiter) if idx < 0 { return fmt.Errorf("timestamp is not found") } msg.timestamp, err = time.Parse(timeFormat, string(log[:idx])) if err != nil { return fmt.Errorf("unexpected timestamp format %q: %v", timeFormat, err) } // Parse stream type log = log[idx+1:] idx = bytes.Index(log, delimiter) if idx < 0 { return fmt.Errorf("stream type is not found") } msg.stream = streamType(log[:idx]) if msg.stream != stdoutType && msg.stream != stderrType { return fmt.Errorf("unexpected stream type %q", msg.stream) } // Get log content msg.log = log[idx+1:] return nil } // parseDockerJSONLog parses logs in Docker JSON log format. Docker JSON log format // example: //   {"log":"content 1","stream":"stdout","time":"2016-10-20T18:39:20.57606443Z"} //   {"log":"content 2","stream":"stderr","time":"2016-10-20T18:39:20.57606444Z"} func parseDockerJSONLog(log []byte, msg *logMessage) error { var l = &jsonlog.JSONLog{} l.Reset() // TODO: JSON decoding is fairly expensive, we should evaluate this. if err := json.Unmarshal(log, l); err != nil { return fmt.Errorf("failed with %v to unmarshal log %q", err, l) } msg.timestamp = l.Created msg.stream = streamType(l.Stream) msg.log = []byte(l.Log) return nil } // getParseFunc returns proper parse function based on the sample log line passed in. func getParseFunc(log []byte) (parseFunc, error) { for _, p := range parseFuncs { if err := p(log, &logMessage{}); err == nil { return p, nil } } return nil, fmt.Errorf("unsupported log format: %q", log) } // logWriter controls the writing into the stream based on the log options. type logWriter struct { stdout io.Writer stderr io.Writer opts   *logOptions remain int64 } // errMaximumWrite is returned when all bytes have been written. var errMaximumWrite = errors.New("maximum write") // errShortWrite is returned when the message is not fully written. var errShortWrite = errors.New("short write") func newLogWriter(stdout io.Writer, stderr io.Writer, opts *logOptions) *logWriter { w := &logWriter{ stdout: stdout, stderr: stderr, opts:   opts, remain: math.MaxInt64, // initialize it as infinity } if opts.bytes >= 0 { w.remain = opts.bytes } return w } </del> <ins> opts := logs.NewLogOptions(apiOpts, time.Now()) </ins> <del> // writeLogs writes logs into stdout, stderr. func (w *logWriter) write(msg *logMessage) error { if msg.timestamp.Before(w.opts.since) { // Skip the line because it's older than since return nil } line := msg.log if w.opts.timestamp { prefix := append([]byte(msg.timestamp.Format(timeFormat)), delimiter[0]) line = append(prefix, line...) } // If the line is longer than the remaining bytes, cut it. if int64(len(line)) > w.remain { line = line[:w.remain] } // Get the proper stream to write to. var stream io.Writer switch msg.stream { case stdoutType: stream = w.stdout case stderrType: stream = w.stderr default: return fmt.Errorf("unexpected stream type %q", msg.stream) } n, err := stream.Write(line) w.remain -= int64(n) if err != nil { return err } // If the line has not been fully written, return errShortWrite if n < len(line) { return errShortWrite } // If there are no more bytes left, return errMaximumWrite if w.remain <= 0 { return errMaximumWrite } return nil </del> <ins> return logs.ReadLogs(path, containerID, opts, m.runtimeService, stdout, stderr) </ins> }


Question:
We should make the CRI log parsing code in kubelet to be a library. So that container runtimes and external tools don't need to copy the whole code ( vendor it instead ). Referring
/assign
/sig node
I can help on it.
/assign
GitHub didn't allow me to assign the following users: yanxuean. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks. I have a branch ready for this. Will open a PR later and ping you when it is out.
ok

<del> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package kuberuntime import ( "bytes" "testing" "time" "github.com/stretchr/testify/assert" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) func TestLogOptions(t *testing.T) { var ( line         = int64(8) bytes        = int64(64) timestamp    = metav1.Now() sinceseconds = int64(10) ) for c, test := range []struct { apiOpts *v1.PodLogOptions expect  *logOptions }{ { // empty options apiOpts: &v1.PodLogOptions{}, expect:  &logOptions{tail: -1, bytes: -1}, }, { // test tail lines apiOpts: &v1.PodLogOptions{TailLines: &line}, expect:  &logOptions{tail: line, bytes: -1}, }, { // test limit bytes apiOpts: &v1.PodLogOptions{LimitBytes: &bytes}, expect:  &logOptions{tail: -1, bytes: bytes}, }, { // test since timestamp apiOpts: &v1.PodLogOptions{SinceTime: &timestamp}, expect:  &logOptions{tail: -1, bytes: -1, since: timestamp.Time}, }, { // test since seconds apiOpts: &v1.PodLogOptions{SinceSeconds: &sinceseconds}, expect:  &logOptions{tail: -1, bytes: -1, since: timestamp.Add(-10 * time.Second)}, }, } { t.Logf("TestCase #%d: %+v", c, test) opts := newLogOptions(test.apiOpts, timestamp.Time) assert.Equal(t, test.expect, opts) } } func TestParseLog(t *testing.T) { timestamp, err := time.Parse(timeFormat, "2016-10-20T18:39:20.57606443Z") assert.NoError(t, err) msg := &logMessage{} for c, test := range []struct { line string msg  *logMessage err  bool }{ { // Docker log format stdout line: `{"log":"docker stdout test log","stream":"stdout","time":"2016-10-20T18:39:20.57606443Z"}` + "n", msg: &logMessage{ timestamp: timestamp, stream:    stdoutType, log:       []byte("docker stdout test log"), }, }, { // Docker log format stderr line: `{"log":"docker stderr test log","stream":"stderr","time":"2016-10-20T18:39:20.57606443Z"}` + "n", msg: &logMessage{ timestamp: timestamp, stream:    stderrType, log:       []byte("docker stderr test log"), }, }, { // CRI log format stdout line: "2016-10-20T18:39:20.57606443Z stdout cri stdout test logn", msg: &logMessage{ timestamp: timestamp, stream:    stdoutType, log:       []byte("cri stdout test logn"), }, }, { // CRI log format stderr line: "2016-10-20T18:39:20.57606443Z stderr cri stderr test logn", msg: &logMessage{ timestamp: timestamp, stream:    stderrType, log:       []byte("cri stderr test logn"), }, }, { // Unsupported Log format line: "unsupported log format test logn", msg:  &logMessage{}, err:  true, }, } { t.Logf("TestCase #%d: %+v", c, test) parse, err := getParseFunc([]byte(test.line)) if test.err { assert.Error(t, err) continue } assert.NoError(t, err) err = parse([]byte(test.line), msg) assert.NoError(t, err) assert.Equal(t, test.msg, msg) } } func TestWriteLogs(t *testing.T) { timestamp := time.Unix(1234, 4321) log := "abcdefgn" for c, test := range []struct { stream       streamType since        time.Time timestamp    bool expectStdout string expectStderr string }{ { // stderr log stream:       stderrType, expectStderr: log, }, { // stdout log stream:       stdoutType, expectStdout: log, }, { // since is after timestamp stream: stdoutType, since:  timestamp.Add(1 * time.Second), }, { // timestamp enabled stream:       stderrType, timestamp:    true, expectStderr: timestamp.Format(timeFormat) + " " + log, }, } { t.Logf("TestCase #%d: %+v", c, test) msg := &logMessage{ timestamp: timestamp, stream:    test.stream, log:       []byte(log), } stdoutBuf := bytes.NewBuffer(nil) stderrBuf := bytes.NewBuffer(nil) w := newLogWriter(stdoutBuf, stderrBuf, &logOptions{since: test.since, timestamp: test.timestamp, bytes: -1}) err := w.write(msg) assert.NoError(t, err) assert.Equal(t, test.expectStdout, stdoutBuf.String()) assert.Equal(t, test.expectStderr, stderrBuf.String()) } } func TestWriteLogsWithBytesLimit(t *testing.T) { timestamp := time.Unix(1234, 4321) timestampStr := timestamp.Format(timeFormat) log := "abcdefgn" for c, test := range []struct { stdoutLines  int stderrLines  int bytes        int timestamp    bool expectStdout string expectStderr string }{ { // limit bytes less than one line stdoutLines:  3, bytes:        3, expectStdout: "abc", }, { // limit bytes across lines stdoutLines:  3, bytes:        len(log) + 3, expectStdout: "abcdefgnabc", }, { // limit bytes more than all lines stdoutLines:  3, bytes:        3 * len(log), expectStdout: "abcdefgnabcdefgnabcdefgn", }, { // limit bytes for stderr stderrLines:  3, bytes:        len(log) + 3, expectStderr: "abcdefgnabc", }, { // limit bytes for both stdout and stderr, stdout first. stdoutLines:  1, stderrLines:  2, bytes:        len(log) + 3, expectStdout: "abcdefgn", expectStderr: "abc", }, { // limit bytes with timestamp stdoutLines:  3, timestamp:    true, bytes:        len(timestampStr) + 1 + len(log) + 2, expectStdout: timestampStr + " " + log + timestampStr[:2], }, } { t.Logf("TestCase #%d: %+v", c, test) msg := &logMessage{ timestamp: timestamp, log:       []byte(log), } stdoutBuf := bytes.NewBuffer(nil) stderrBuf := bytes.NewBuffer(nil) w := newLogWriter(stdoutBuf, stderrBuf, &logOptions{timestamp: test.timestamp, bytes: int64(test.bytes)}) for i := 0; i < test.stdoutLines; i++ { msg.stream = stdoutType if err := w.write(msg); err != nil { assert.EqualError(t, err, errMaximumWrite.Error()) } } for i := 0; i < test.stderrLines; i++ { msg.stream = stderrType if err := w.write(msg); err != nil { assert.EqualError(t, err, errMaximumWrite.Error()) } } assert.Equal(t, test.expectStdout, stdoutBuf.String()) assert.Equal(t, test.expectStderr, stderrBuf.String()) } } </del>


Question:
We should make the CRI log parsing code in kubelet to be a library. So that container runtimes and external tools don't need to copy the whole code ( vendor it instead ). Referring
/assign
/sig node
I can help on it.
/assign
GitHub didn't allow me to assign the following users: yanxuean. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks. I have a branch ready for this. Will open a PR later and ping you when it is out.
ok

<ins> load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test") go_library( name = "go_default_library", srcs = ["logs.go"], importpath = "k8s.io/kubernetes/pkg/kubelet/kuberuntime/logs", visibility = ["//visibility:public"], deps = [ "//pkg/kubelet/apis/cri:go_default_library", "//pkg/kubelet/apis/cri/v1alpha1/runtime:go_default_library", "//pkg/util/tail:go_default_library", "//vendor/github.com/docker/docker/pkg/jsonlog:go_default_library", "//vendor/github.com/fsnotify/fsnotify:go_default_library", "//vendor/github.com/golang/glog:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", ], ) go_test( name = "go_default_test", srcs = ["logs_test.go"], importpath = "k8s.io/kubernetes/pkg/kubelet/kuberuntime/logs", library = ":go_default_library", deps = [ "//vendor/github.com/stretchr/testify/assert:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", ], ) filegroup( name = "package-srcs", srcs = glob(["**"]), tags = ["automanaged"], visibility = ["//visibility:private"], ) filegroup( name = "all-srcs", srcs = [":package-srcs"], tags = ["automanaged"], visibility = ["//visibility:public"], ) </ins>


Question:
We should make the CRI log parsing code in kubelet to be a library. So that container runtimes and external tools don't need to copy the whole code ( vendor it instead ). Referring
/assign
/sig node
I can help on it.
/assign
GitHub didn't allow me to assign the following users: yanxuean. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks. I have a branch ready for this. Will open a PR later and ping you when it is out.
ok

<ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package logs import ( "bufio" "bytes" "encoding/json" "errors" "fmt" "io" "math" "os" "time" "github.com/docker/docker/pkg/jsonlog" "github.com/fsnotify/fsnotify" "github.com/golang/glog" "k8s.io/api/core/v1" internalapi "k8s.io/kubernetes/pkg/kubelet/apis/cri" runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/v1alpha1/runtime" "k8s.io/kubernetes/pkg/util/tail" ) // Notice that the current CRI logs implementation doesn't handle // log rotation. // * It will not retrieve logs in rotated log file. // * If log rotation happens when following the log: //   * If the rotation is using create mode, we'll still follow the old file. //   * If the rotation is using copytruncate, we'll be reading at the original position and get nothing. // TODO(random-liu): Support log rotation. // streamType is the type of the stream. type streamType string const ( stderrType streamType = "stderr" stdoutType streamType = "stdout" // timeFormat is the time format used in the log. timeFormat = time.RFC3339Nano // blockSize is the block size used in tail. blockSize = 1024 // stateCheckPeriod is the period to check container state while following // the container log. Kubelet should not keep following the log when the // container is not running. stateCheckPeriod = 5 * time.Second ) var ( // eol is the end-of-line sign in the log. eol = []byte{'n'} // delimiter is the delimiter for timestamp and streamtype in log line. delimiter = []byte{' '} ) // logMessage is the CRI internal log type. type logMessage struct { timestamp time.Time stream    streamType log       []byte } // reset resets the log to nil. func (l *logMessage) reset() { l.timestamp = time.Time{} l.stream = "" l.log = nil } // LogOptions is the CRI internal type of all log options. type LogOptions struct { tail      int64 bytes     int64 since     time.Time follow    bool timestamp bool } // NewLogOptions convert the v1.PodLogOptions to CRI internal LogOptions. func NewLogOptions(apiOpts *v1.PodLogOptions, now time.Time) *LogOptions { opts := &LogOptions{ tail:      -1, // -1 by default which means read all logs. bytes:     -1, // -1 by default which means read all logs. follow:    apiOpts.Follow, timestamp: apiOpts.Timestamps, } if apiOpts.TailLines != nil { opts.tail = *apiOpts.TailLines } if apiOpts.LimitBytes != nil { opts.bytes = *apiOpts.LimitBytes } if apiOpts.SinceSeconds != nil { opts.since = now.Add(-time.Duration(*apiOpts.SinceSeconds) * time.Second) } if apiOpts.SinceTime != nil && apiOpts.SinceTime.After(opts.since) { opts.since = apiOpts.SinceTime.Time } return opts } // parseFunc is a function parsing one log line to the internal log type. // Notice that the caller must make sure logMessage is not nil. type parseFunc func([]byte, *logMessage) error var parseFuncs = []parseFunc{ parseCRILog,        // CRI log format parse function parseDockerJSONLog, // Docker JSON log format parse function } // parseCRILog parses logs in CRI log format. CRI Log format example: //   2016-10-06T00:17:09.669794202Z stdout log content 1 //   2016-10-06T00:17:09.669794203Z stderr log content 2 func parseCRILog(log []byte, msg *logMessage) error { var err error // Parse timestamp idx := bytes.Index(log, delimiter) if idx < 0 { return fmt.Errorf("timestamp is not found") } msg.timestamp, err = time.Parse(timeFormat, string(log[:idx])) if err != nil { return fmt.Errorf("unexpected timestamp format %q: %v", timeFormat, err) } // Parse stream type log = log[idx+1:] idx = bytes.Index(log, delimiter) if idx < 0 { return fmt.Errorf("stream type is not found") } msg.stream = streamType(log[:idx]) if msg.stream != stdoutType && msg.stream != stderrType { return fmt.Errorf("unexpected stream type %q", msg.stream) } // Get log content msg.log = log[idx+1:] return nil } // parseDockerJSONLog parses logs in Docker JSON log format. Docker JSON log format // example: //   {"log":"content 1","stream":"stdout","time":"2016-10-20T18:39:20.57606443Z"} //   {"log":"content 2","stream":"stderr","time":"2016-10-20T18:39:20.57606444Z"} func parseDockerJSONLog(log []byte, msg *logMessage) error { var l = &jsonlog.JSONLog{} l.Reset() // TODO: JSON decoding is fairly expensive, we should evaluate this. if err := json.Unmarshal(log, l); err != nil { return fmt.Errorf("failed with %v to unmarshal log %q", err, l) } msg.timestamp = l.Created msg.stream = streamType(l.Stream) msg.log = []byte(l.Log) return nil } // getParseFunc returns proper parse function based on the sample log line passed in. func getParseFunc(log []byte) (parseFunc, error) { for _, p := range parseFuncs { if err := p(log, &logMessage{}); err == nil { return p, nil } } return nil, fmt.Errorf("unsupported log format: %q", log) } // logWriter controls the writing into the stream based on the log options. type logWriter struct { stdout io.Writer stderr io.Writer opts   *LogOptions remain int64 } // errMaximumWrite is returned when all bytes have been written. var errMaximumWrite = errors.New("maximum write") // errShortWrite is returned when the message is not fully written. var errShortWrite = errors.New("short write") func newLogWriter(stdout io.Writer, stderr io.Writer, opts *LogOptions) *logWriter { w := &logWriter{ stdout: stdout, stderr: stderr, opts:   opts, remain: math.MaxInt64, // initialize it as infinity } if opts.bytes >= 0 { w.remain = opts.bytes } return w } // writeLogs writes logs into stdout, stderr. func (w *logWriter) write(msg *logMessage) error { if msg.timestamp.Before(w.opts.since) { // Skip the line because it's older than since return nil } line := msg.log if w.opts.timestamp { prefix := append([]byte(msg.timestamp.Format(timeFormat)), delimiter[0]) line = append(prefix, line...) } // If the line is longer than the remaining bytes, cut it. if int64(len(line)) > w.remain { line = line[:w.remain] } // Get the proper stream to write to. var stream io.Writer switch msg.stream { case stdoutType: stream = w.stdout case stderrType: stream = w.stderr default: return fmt.Errorf("unexpected stream type %q", msg.stream) } n, err := stream.Write(line) w.remain -= int64(n) if err != nil { return err } // If the line has not been fully written, return errShortWrite if n < len(line) { return errShortWrite } // If there are no more bytes left, return errMaximumWrite if w.remain <= 0 { return errMaximumWrite } return nil } // ReadLogs read the container log and redirect into stdout and stderr. // Note that containerID is only needed when following the log, or else // just pass in empty string "". func ReadLogs(path, containerID string, opts *LogOptions, runtimeService internalapi.RuntimeService, stdout, stderr io.Writer) error { f, err := os.Open(path) if err != nil { return fmt.Errorf("failed to open log file %q: %v", path, err) } defer f.Close() // Search start point based on tail line. start, err := tail.FindTailLineStartIndex(f, opts.tail) if err != nil { return fmt.Errorf("failed to tail %d lines of log file %q: %v", opts.tail, path, err) } if _, err := f.Seek(start, os.SEEK_SET); err != nil { return fmt.Errorf("failed to seek %d in log file %q: %v", start, path, err) } // Start parsing the logs. r := bufio.NewReader(f) // Do not create watcher here because it is not needed if `Follow` is false. var watcher *fsnotify.Watcher var parse parseFunc writer := newLogWriter(stdout, stderr, opts) msg := &logMessage{} for { l, err := r.ReadBytes(eol[0]) if err != nil { if err != io.EOF { // This is an real error return fmt.Errorf("failed to read log file %q: %v", path, err) } if !opts.follow { // Return directly when reading to the end if not follow. if len(l) > 0 { glog.Warningf("Incomplete line in log file %q: %q", path, l) } glog.V(2).Infof("Finish parsing log file %q", path) return nil } // Reset seek so that if this is an incomplete line, // it will be read again. if _, err := f.Seek(-int64(len(l)), os.SEEK_CUR); err != nil { return fmt.Errorf("failed to reset seek in log file %q: %v", path, err) } if watcher == nil { // Intialize the watcher if it has not been initialized yet. if watcher, err = fsnotify.NewWatcher(); err != nil { return fmt.Errorf("failed to create fsnotify watcher: %v", err) } defer watcher.Close() if err := watcher.Add(f.Name()); err != nil { return fmt.Errorf("failed to watch file %q: %v", f.Name(), err) } } // Wait until the next log change. if found, err := waitLogs(containerID, watcher, runtimeService); !found { return err } continue } if parse == nil { // Intialize the log parsing function. parse, err = getParseFunc(l) if err != nil { return fmt.Errorf("failed to get parse function: %v", err) } } // Parse the log line. msg.reset() if err := parse(l, msg); err != nil { glog.Errorf("Failed with err %v when parsing log for log file %q: %q", err, path, l) continue } // Write the log line into the stream. if err := writer.write(msg); err != nil { if err == errMaximumWrite { glog.V(2).Infof("Finish parsing log file %q, hit bytes limit %d(bytes)", path, opts.bytes) return nil } glog.Errorf("Failed with err %v when writing log for log file %q: %+v", err, path, msg) return err } } } // waitLogs wait for the next log write. It returns a boolean and an error. The boolean // indicates whether a new log is found; the error is error happens during waiting new logs. func waitLogs(id string, w *fsnotify.Watcher, runtimeService internalapi.RuntimeService) (bool, error) { errRetry := 5 for { select { case e := <-w.Events: switch e.Op { case fsnotify.Write: return true, nil default: glog.Errorf("Unexpected fsnotify event: %v, retrying...", e) } case err := <-w.Errors: glog.Errorf("Fsnotify watch error: %v, %d error retries remaining", err, errRetry) if errRetry == 0 { return false, err } errRetry-- case <-time.After(stateCheckPeriod): s, err := runtimeService.ContainerStatus(id) if err != nil { return false, err } // Only keep following container log when it is running. if s.State != runtimeapi.ContainerState_CONTAINER_RUNNING { glog.Errorf("Container %q is not running (state=%q)", id, s.State) // Do not return error because it's normal that the container stops // during waiting. return false, nil } } } } </ins>


Question:
We should make the CRI log parsing code in kubelet to be a library. So that container runtimes and external tools don't need to copy the whole code ( vendor it instead ). Referring
/assign
/sig node
I can help on it.
/assign
GitHub didn't allow me to assign the following users: yanxuean. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks. I have a branch ready for this. Will open a PR later and ping you when it is out.
ok

<ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package logs import ( "bytes" "testing" "time" "github.com/stretchr/testify/assert" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) func TestLogOptions(t *testing.T) { var ( line         = int64(8) bytes        = int64(64) timestamp    = metav1.Now() sinceseconds = int64(10) ) for c, test := range []struct { apiOpts *v1.PodLogOptions expect  *LogOptions }{ { // empty options apiOpts: &v1.PodLogOptions{}, expect:  &LogOptions{tail: -1, bytes: -1}, }, { // test tail lines apiOpts: &v1.PodLogOptions{TailLines: &line}, expect:  &LogOptions{tail: line, bytes: -1}, }, { // test limit bytes apiOpts: &v1.PodLogOptions{LimitBytes: &bytes}, expect:  &LogOptions{tail: -1, bytes: bytes}, }, { // test since timestamp apiOpts: &v1.PodLogOptions{SinceTime: &timestamp}, expect:  &LogOptions{tail: -1, bytes: -1, since: timestamp.Time}, }, { // test since seconds apiOpts: &v1.PodLogOptions{SinceSeconds: &sinceseconds}, expect:  &LogOptions{tail: -1, bytes: -1, since: timestamp.Add(-10 * time.Second)}, }, } { t.Logf("TestCase #%d: %+v", c, test) opts := NewLogOptions(test.apiOpts, timestamp.Time) assert.Equal(t, test.expect, opts) } } func TestParseLog(t *testing.T) { timestamp, err := time.Parse(timeFormat, "2016-10-20T18:39:20.57606443Z") assert.NoError(t, err) msg := &logMessage{} for c, test := range []struct { line string msg  *logMessage err  bool }{ { // Docker log format stdout line: `{"log":"docker stdout test log","stream":"stdout","time":"2016-10-20T18:39:20.57606443Z"}` + "n", msg: &logMessage{ timestamp: timestamp, stream:    stdoutType, log:       []byte("docker stdout test log"), }, }, { // Docker log format stderr line: `{"log":"docker stderr test log","stream":"stderr","time":"2016-10-20T18:39:20.57606443Z"}` + "n", msg: &logMessage{ timestamp: timestamp, stream:    stderrType, log:       []byte("docker stderr test log"), }, }, { // CRI log format stdout line: "2016-10-20T18:39:20.57606443Z stdout cri stdout test logn", msg: &logMessage{ timestamp: timestamp, stream:    stdoutType, log:       []byte("cri stdout test logn"), }, }, { // CRI log format stderr line: "2016-10-20T18:39:20.57606443Z stderr cri stderr test logn", msg: &logMessage{ timestamp: timestamp, stream:    stderrType, log:       []byte("cri stderr test logn"), }, }, { // Unsupported Log format line: "unsupported log format test logn", msg:  &logMessage{}, err:  true, }, } { t.Logf("TestCase #%d: %+v", c, test) parse, err := getParseFunc([]byte(test.line)) if test.err { assert.Error(t, err) continue } assert.NoError(t, err) err = parse([]byte(test.line), msg) assert.NoError(t, err) assert.Equal(t, test.msg, msg) } } func TestWriteLogs(t *testing.T) { timestamp := time.Unix(1234, 4321) log := "abcdefgn" for c, test := range []struct { stream       streamType since        time.Time timestamp    bool expectStdout string expectStderr string }{ { // stderr log stream:       stderrType, expectStderr: log, }, { // stdout log stream:       stdoutType, expectStdout: log, }, { // since is after timestamp stream: stdoutType, since:  timestamp.Add(1 * time.Second), }, { // timestamp enabled stream:       stderrType, timestamp:    true, expectStderr: timestamp.Format(timeFormat) + " " + log, }, } { t.Logf("TestCase #%d: %+v", c, test) msg := &logMessage{ timestamp: timestamp, stream:    test.stream, log:       []byte(log), } stdoutBuf := bytes.NewBuffer(nil) stderrBuf := bytes.NewBuffer(nil) w := newLogWriter(stdoutBuf, stderrBuf, &LogOptions{since: test.since, timestamp: test.timestamp, bytes: -1}) err := w.write(msg) assert.NoError(t, err) assert.Equal(t, test.expectStdout, stdoutBuf.String()) assert.Equal(t, test.expectStderr, stderrBuf.String()) } } func TestWriteLogsWithBytesLimit(t *testing.T) { timestamp := time.Unix(1234, 4321) timestampStr := timestamp.Format(timeFormat) log := "abcdefgn" for c, test := range []struct { stdoutLines  int stderrLines  int bytes        int timestamp    bool expectStdout string expectStderr string }{ { // limit bytes less than one line stdoutLines:  3, bytes:        3, expectStdout: "abc", }, { // limit bytes across lines stdoutLines:  3, bytes:        len(log) + 3, expectStdout: "abcdefgnabc", }, { // limit bytes more than all lines stdoutLines:  3, bytes:        3 * len(log), expectStdout: "abcdefgnabcdefgnabcdefgn", }, { // limit bytes for stderr stderrLines:  3, bytes:        len(log) + 3, expectStderr: "abcdefgnabc", }, { // limit bytes for both stdout and stderr, stdout first. stdoutLines:  1, stderrLines:  2, bytes:        len(log) + 3, expectStdout: "abcdefgn", expectStderr: "abc", }, { // limit bytes with timestamp stdoutLines:  3, timestamp:    true, bytes:        len(timestampStr) + 1 + len(log) + 2, expectStdout: timestampStr + " " + log + timestampStr[:2], }, } { t.Logf("TestCase #%d: %+v", c, test) msg := &logMessage{ timestamp: timestamp, log:       []byte(log), } stdoutBuf := bytes.NewBuffer(nil) stderrBuf := bytes.NewBuffer(nil) w := newLogWriter(stdoutBuf, stderrBuf, &LogOptions{timestamp: test.timestamp, bytes: int64(test.bytes)}) for i := 0; i < test.stdoutLines; i++ { msg.stream = stdoutType if err := w.write(msg); err != nil { assert.EqualError(t, err, errMaximumWrite.Error()) } } for i := 0; i < test.stderrLines; i++ { msg.stream = stderrType if err := w.write(msg); err != nil { assert.EqualError(t, err, errMaximumWrite.Error()) } } assert.Equal(t, test.expectStdout, stdoutBuf.String()) assert.Equal(t, test.expectStderr, stderrBuf.String()) } } </ins>


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT: What happened: When a statefulset object is created, a lot of controllerrevisions are created at the same time. the controllerrevisions have differenct hash value but the same Data.Raw. How to fix it: two controllerrevisions are seen as the same , when [ControllerRevisionHashLabel] , Controllerrevision.Data.Raw , Controllerrevision.Data.Object are strictly the same // this line shoud be to fix the bug like below: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.8 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I would help fix it
/kind bug /sig apps
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
StatefulSet uses the following code to construct its history object (controllerrevision): However, collisionCount shouldn't be blindly increased when there's name collision. Because controller uses a cache store, the name collision could be due to a controllerrevision created but not yet observed by the controller. The right way to handle it is to first check if the controllerrevision with the same name has the same content as the one we're going to create, like how DaemonSet controller handles it:

hash := HashControllerRevision(revision, collisionCount) // Update the revisions name and labels clone.Name = ControllerRevisionName(parent.GetName(), hash) <del> created, err := rh.client.AppsV1().ControllerRevisions(parent.GetNamespace()).Create(clone) </del> <ins> ns := parent.GetNamespace() created, err := rh.client.AppsV1().ControllerRevisions(ns).Create(clone) </ins> if errors.IsAlreadyExists(err) { <ins> exists, err := rh.client.AppsV1().ControllerRevisions(ns).Get(clone.Name, metav1.GetOptions{}) if err != nil { return nil, err } if bytes.Equal(exists.Data.Raw, clone.Data.Raw) { return exists, nil } </ins> *collisionCount++ continue }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT: What happened: When a statefulset object is created, a lot of controllerrevisions are created at the same time. the controllerrevisions have differenct hash value but the same Data.Raw. How to fix it: two controllerrevisions are seen as the same , when [ControllerRevisionHashLabel] , Controllerrevision.Data.Raw , Controllerrevision.Data.Object are strictly the same // this line shoud be to fix the bug like below: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.8 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I would help fix it
/kind bug /sig apps
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
StatefulSet uses the following code to construct its history object (controllerrevision): However, collisionCount shouldn't be blindly increased when there's name collision. Because controller uses a cache store, the name collision could be due to a controllerrevision created but not yet observed by the controller. The right way to handle it is to first check if the controllerrevision with the same name has the same content as the one we're going to create, like how DaemonSet controller handles it:

history := NewHistory(client, informer.Lister()) var collisionCount int32 <del> for i := range test.existing { _, err := history.CreateControllerRevision(test.existing[i].parent, test.existing[i].revision, &collisionCount) </del> <ins> for _, item := range test.existing { _, err := client.AppsV1().ControllerRevisions(item.parent.GetNamespace()).Create(item.revision) </ins> if err != nil { t.Fatal(err) }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT: What happened: When a statefulset object is created, a lot of controllerrevisions are created at the same time. the controllerrevisions have differenct hash value but the same Data.Raw. How to fix it: two controllerrevisions are seen as the same , when [ControllerRevisionHashLabel] , Controllerrevision.Data.Raw , Controllerrevision.Data.Object are strictly the same // this line shoud be to fix the bug like below: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.8 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I would help fix it
/kind bug /sig apps
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
StatefulSet uses the following code to construct its history object (controllerrevision): However, collisionCount shouldn't be blindly increased when there's name collision. Because controller uses a cache store, the name collision could be due to a controllerrevision created but not yet observed by the controller. The right way to handle it is to first check if the controllerrevision with the same name has the same content as the one we're going to create, like how DaemonSet controller handles it:

t.Errorf("%s: on name collision wanted new name %s got %s", test.name, expectedName, created.Name) } <del> // Second name collision should have incremented collisionCount to 2 </del> <ins> // Second name collision will be caused by an identical revision, so no need to do anything </ins> _, err = history.CreateControllerRevision(test.parent, test.revision, &collisionCount) if err != nil { t.Errorf("%s: %s", test.name, err) } <del> if collisionCount != 2 { </del> <ins> if collisionCount != 1 { </ins> t.Errorf("%s: on second name collision wanted collisionCount 1 got %d", test.name, collisionCount) } }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT: What happened: When a statefulset object is created, a lot of controllerrevisions are created at the same time. the controllerrevisions have differenct hash value but the same Data.Raw. How to fix it: two controllerrevisions are seen as the same , when [ControllerRevisionHashLabel] , Controllerrevision.Data.Raw , Controllerrevision.Data.Object are strictly the same // this line shoud be to fix the bug like below: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.8 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I would help fix it
/kind bug /sig apps
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
StatefulSet uses the following code to construct its history object (controllerrevision): However, collisionCount shouldn't be blindly increased when there's name collision. Because controller uses a cache store, the name collision could be due to a controllerrevision created but not yet observed by the controller. The right way to handle it is to first check if the controllerrevision with the same name has the same content as the one we're going to create, like how DaemonSet controller handles it:

t.Fatal(err) } ss1Rev1.Namespace = ss1.Namespace <del> ss1Rev2, err := NewControllerRevision(ss1, parentKind, ss1.Spec.Template.Labels, rawTemplate(&ss1.Spec.Template), 2, ss1.Status.CollisionCount) </del> <ins> // Create a new revision with the same name and hash label as an existing revision, but with // a different template. This could happen as a result of a hash collision, but in this test // this situation is created by setting name and hash label to values known to be in use by // an existing revision. modTemplate := ss1.Spec.Template.DeepCopy() modTemplate.Labels["foo"] = "not_bar" ss1Rev2, err := NewControllerRevision(ss1, parentKind, ss1.Spec.Template.Labels, rawTemplate(modTemplate), 2, ss1.Status.CollisionCount) ss1Rev2.Name = ss1Rev1.Name ss1Rev2.Labels[ControllerRevisionHashLabel] = ss1Rev1.Labels[ControllerRevisionHashLabel] </ins> if err != nil { t.Fatal(err) }


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT: What happened: When a statefulset object is created, a lot of controllerrevisions are created at the same time. the controllerrevisions have differenct hash value but the same Data.Raw. How to fix it: two controllerrevisions are seen as the same , when [ControllerRevisionHashLabel] , Controllerrevision.Data.Raw , Controllerrevision.Data.Object are strictly the same // this line shoud be to fix the bug like below: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.8 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I would help fix it
/kind bug /sig apps
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
StatefulSet uses the following code to construct its history object (controllerrevision): However, collisionCount shouldn't be blindly increased when there's name collision. Because controller uses a cache store, the name collision could be due to a controllerrevision created but not yet observed by the controller. The right way to handle it is to first check if the controllerrevision with the same name has the same content as the one we're going to create, like how DaemonSet controller handles it:

}{ { parent:   ss1, <del> revision: ss1Rev1, </del> <ins> revision: ss1Rev2, </ins> }, }, rename: true,


Question:
/kind bug /sig cli What happened: Tried to list the age of pods in the namespace in an otherwise empty cluster $ kubectl --namespace kube-system plugin aging The Magnificent Aging Plugin. No pods What you expected to happen: That the pods in the namespace would have been listed How to reproduce it (as minimally and precisely as possible): the and from into . the command: Anything else we need to know?: No Environment: Kubernetes version (use ): v1.8.2 Cloud provider or hardware configuration: minikube & GKE OS (e.g. from /etc/os-release): Mac OSX El Capitan Kernel (e.g. ): Darwin Kernel Version 15.6.0 Install tools: Google Cloud SDK
hi for cli issue, could you please file it in this repo next time ? thanks !!
Will the k8s github bots be able to resolve references in commit messages made in kubernetes/kubernetes PRs be able to find the corresponding issue ticket in that other project? Semi-OT but related: I know there's an overall intention to refactoring k8s proper into many distinct projects, so it would be helpful if either the contributor guide contained a mapping of each and every module making up a k8s release with its corresponding github project and protocol governing the component. Or if each directory (module) in kubernetes/kubernetes contains its own explaining the contribution protocol. Right now documentation and information about ownership, contribution and protocols of various project modules are a sprawl and a bit of a mess, creating a significant barrier to entry (contribution). What's the proper place and form to discuss these sort of things?
thanks, I think those are some good suggestions, as far as I know you could send an email to our mailling-list(google-groups) or create an new issue then cc

end end <del> pods_json = `kubectl get pods -o json` </del> <ins> namespace = ENV['KUBECTL_PLUGINS_CURRENT_NAMESPACE'] || 'default' pods_json = `kubectl --namespace #{namespace} get pods -o json` </ins> pods_parsed = JSON.parse(pods_json) puts "The Magnificent Aging Plugin."


Question:
/kind bug /sig cli What happened: Tried to list the age of pods in the namespace in an otherwise empty cluster $ kubectl --namespace kube-system plugin aging The Magnificent Aging Plugin. No pods What you expected to happen: That the pods in the namespace would have been listed How to reproduce it (as minimally and precisely as possible): the and from into . the command: Anything else we need to know?: No Environment: Kubernetes version (use ): v1.8.2 Cloud provider or hardware configuration: minikube & GKE OS (e.g. from /etc/os-release): Mac OSX El Capitan Kernel (e.g. ): Darwin Kernel Version 15.6.0 Install tools: Google Cloud SDK
hi for cli issue, could you please file it in this repo next time ? thanks !!
Will the k8s github bots be able to resolve references in commit messages made in kubernetes/kubernetes PRs be able to find the corresponding issue ticket in that other project? Semi-OT but related: I know there's an overall intention to refactoring k8s proper into many distinct projects, so it would be helpful if either the contributor guide contained a mapping of each and every module making up a k8s release with its corresponding github project and protocol governing the component. Or if each directory (module) in kubernetes/kubernetes contains its own explaining the contribution protocol. Right now documentation and information about ownership, contribution and protocols of various project modules are a sprawl and a bit of a mess, creating a significant barrier to entry (contribution). What's the proper place and form to discuss these sort of things?
thanks, I think those are some good suggestions, as far as I know you could send an email to our mailling-list(google-groups) or create an new issue then cc

shortDesc: "Aging shows pods by age" longDesc: > Aging shows pods from the current namespace by age. <del> Once we have plugin support for global flags through env vars (planned for V1) we'll be able to switch between namespaces using the --namespace flag. </del> command: ./aging.rb


"sync" "time" <ins> "github.com/aws/aws-sdk-go/aws" </ins> "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/request" "github.com/golang/glog"


if delay > 0 { glog.Warningf("Inserting delay before AWS request (%s) to avoid RequestLimitExceeded: %s", describeRequest(r), delay.String()) <del> r.Config.SleepDelay(delay) </del> <ins> if sleepFn := r.Config.SleepDelay; sleepFn != nil { // Support SleepDelay for backwards compatibility sleepFn(delay) } else if err := aws.SleepWithContext(r.Context(), delay); err != nil { r.Error = awserr.New(request.CanceledErrorCode, "request context canceled", err) r.Retryable = aws.Bool(false) return } </ins> // Avoid clock skew problems r.Time = now


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --BUG REPORT /kind bug What happened: I tried to submit a Service Replace () to the kube-apiserver and this results the kube-apiserver container crashing and restarting. PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenameWhat you expected to happen: I expected the server not to crash and to get validation errors if the request was invalid. How to reproduce it (as minimally and precisely as possible): Create a service, get the resourceVersion and clusterIP from the created resource GET localhost:8001/api/v1/namespaces/<namespace/services/<servicenameSubmit a replace command to PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenamewith body like If the same request is submitted with the nodePort fields in the ports array then the request succeeds. I get that I may be sending an invalid request without a node port (though shouldn't the merge deal with that?) but it shouldn't take down the API server. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T23:29:08Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Bare metal cluster OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-98-generic Install tools: Others:
thanks for reporting this! Seems like this is similar to and should be fixed by . can you PTAL that PR to see if it's good to merge?
/sig api-machinery
Duplicate of

proxyTransport   http.RoundTripper } <ins> // ServiceNodePort includes protocol and port number of a service NodePort. type ServiceNodePort struct { // The IP protocol for this port. Supports "TCP" and "UDP". Protocol api.Protocol // The port on each node on which this service is exposed. // Default is to auto-allocate a port if the ServiceType of this Service requires one. NodePort int32 } </ins> // NewStorage returns a new REST. func NewStorage(registry Registry, endpoints endpoint.Registry, serviceIPs ipallocator.Interface, serviceNodePorts portallocator.Interface, proxyTransport http.RoundTripper) *ServiceRest {


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --BUG REPORT /kind bug What happened: I tried to submit a Service Replace () to the kube-apiserver and this results the kube-apiserver container crashing and restarting. PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenameWhat you expected to happen: I expected the server not to crash and to get validation errors if the request was invalid. How to reproduce it (as minimally and precisely as possible): Create a service, get the resourceVersion and clusterIP from the created resource GET localhost:8001/api/v1/namespaces/<namespace/services/<servicenameSubmit a replace command to PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenamewith body like If the same request is submitted with the nodePort fields in the ports array then the request succeeds. I get that I may be sending an invalid request without a node port (though shouldn't the merge deal with that?) but it shouldn't take down the API server. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T23:29:08Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Bare metal cluster OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-98-generic Install tools: Others:
thanks for reporting this! Seems like this is similar to and should be fixed by . can you PTAL that PR to see if it's good to merge?
/sig api-machinery
Duplicate of

// This is O(N), but we expect haystack to be small; // so small that we expect a linear search to be faster <del> func contains(haystack []int, needle int) bool { </del> <ins> func containsNumber(haystack []int, needle int) bool { </ins> for _, v := range haystack { if v == needle { return true


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --BUG REPORT /kind bug What happened: I tried to submit a Service Replace () to the kube-apiserver and this results the kube-apiserver container crashing and restarting. PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenameWhat you expected to happen: I expected the server not to crash and to get validation errors if the request was invalid. How to reproduce it (as minimally and precisely as possible): Create a service, get the resourceVersion and clusterIP from the created resource GET localhost:8001/api/v1/namespaces/<namespace/services/<servicenameSubmit a replace command to PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenamewith body like If the same request is submitted with the nodePort fields in the ports array then the request succeeds. I get that I may be sending an invalid request without a node port (though shouldn't the merge deal with that?) but it shouldn't take down the API server. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T23:29:08Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Bare metal cluster OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-98-generic Install tools: Others:
thanks for reporting this! Seems like this is similar to and should be fixed by . can you PTAL that PR to see if it's good to merge?
/sig api-machinery
Duplicate of

return false } <ins> // This is O(N), but we expect serviceNodePorts to be small; // so small that we expect a linear search to be faster func containsNodePort(serviceNodePorts []ServiceNodePort, serviceNodePort ServiceNodePort) bool { for _, snp := range serviceNodePorts { if snp == serviceNodePort { return true } } return false } </ins> func CollectServiceNodePorts(service *api.Service) []int { servicePorts := []int{} for i := range service.Spec.Ports {


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --BUG REPORT /kind bug What happened: I tried to submit a Service Replace () to the kube-apiserver and this results the kube-apiserver container crashing and restarting. PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenameWhat you expected to happen: I expected the server not to crash and to get validation errors if the request was invalid. How to reproduce it (as minimally and precisely as possible): Create a service, get the resourceVersion and clusterIP from the created resource GET localhost:8001/api/v1/namespaces/<namespace/services/<servicenameSubmit a replace command to PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenamewith body like If the same request is submitted with the nodePort fields in the ports array then the request succeeds. I get that I may be sending an invalid request without a node port (though shouldn't the merge deal with that?) but it shouldn't take down the API server. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T23:29:08Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Bare metal cluster OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-98-generic Install tools: Others:
thanks for reporting this! Seems like this is similar to and should be fixed by . can you PTAL that PR to see if it's good to merge?
/sig api-machinery
Duplicate of

} func (rs *REST) updateNodePorts(oldService, newService *api.Service, nodePortOp *portallocator.PortAllocationOperation) error { <del> oldNodePorts := CollectServiceNodePorts(oldService) </del> <ins> oldNodePortsNumbers := CollectServiceNodePorts(oldService) newNodePorts := []ServiceNodePort{} portAllocated := map[int]bool{} </ins> <del> newNodePorts := []int{} </del> for i := range newService.Spec.Ports { servicePort := &newService.Spec.Ports[i] <del> nodePort := int(servicePort.NodePort) if nodePort != 0 { if !contains(oldNodePorts, nodePort) { err := nodePortOp.Allocate(nodePort) </del> <ins> nodePort := ServiceNodePort{Protocol: servicePort.Protocol, NodePort: servicePort.NodePort} if nodePort.NodePort != 0 { if !containsNumber(oldNodePortsNumbers, int(nodePort.NodePort)) && !portAllocated[int(nodePort.NodePort)] { err := nodePortOp.Allocate(int(nodePort.NodePort)) </ins> if err != nil { <del> el := field.ErrorList{field.Invalid(field.NewPath("spec", "ports").Index(i).Child("nodePort"), nodePort, err.Error())} </del> <ins> el := field.ErrorList{field.Invalid(field.NewPath("spec", "ports").Index(i).Child("nodePort"), nodePort.NodePort, err.Error())} </ins> return errors.NewInvalid(api.Kind("Service"), newService.Name, el) } <ins> portAllocated[int(nodePort.NodePort)] = true </ins> } } else { <del> nodePort, err := nodePortOp.AllocateNext() </del> <ins> nodePortNumber, err := nodePortOp.AllocateNext() </ins> if err != nil { // TODO: what error should be returned here?  It's not a // field-level validation failure (the field is valid), and it's // not really an internal error. return errors.NewInternalError(fmt.Errorf("failed to allocate a nodePort: %v", err)) } <del> servicePort.NodePort = int32(nodePort) </del> <ins> servicePort.NodePort = int32(nodePortNumber) nodePort.NodePort = servicePort.NodePort </ins> } <del> // Detect duplicate node ports; this should have been caught by validation, so we panic if contains(newNodePorts, nodePort) { panic("duplicate node port") </del> <ins> if containsNodePort(newNodePorts, nodePort) { return fmt.Errorf("duplicate nodePort: %v", nodePort) </ins> } newNodePorts = append(newNodePorts, nodePort) } <ins> newNodePortsNumbers := CollectServiceNodePorts(newService) </ins> // The comparison loops are O(N^2), but we don't expect N to be huge // (there's a hard-limit at 2^16, because they're ports; and even 4 ports would be a lot) <del> for _, oldNodePort := range oldNodePorts { if contains(newNodePorts, oldNodePort) { </del> <ins> for _, oldNodePortNumber := range oldNodePortsNumbers { if containsNumber(newNodePortsNumbers, oldNodePortNumber) { </ins> continue } <del> nodePortOp.ReleaseDeferred(oldNodePort) </del> <ins> nodePortOp.ReleaseDeferred(int(oldNodePortNumber)) </ins> } return nil