Question:
/kind bug /sig node When the is set to and a container that was granted exclusive CPUs terminates with a normal exit status, the CPU Manager never receives a call. Kuberuntime should be updated to invoke the container manager's hook on detecting normal termination for these containers. This method is already idempotent since it is currently invoked both from kuberuntime's and (the latter of which is invoked by GC.) How to reproduce it (as minimally and precisely as possible): Start singleton cluster with static CPU manager policy enabled. Start a guaranteed pod with a single container that consumes 1 CPU (to get an exclusive allocation) with restartPolicy never that simply exits . Try to start a similar container that won't fit on the node unless the first container's CPU is reaped. Observe the second container fails with insufficient CPUs, and this problem does not resolve even after an extended period of time. Environment: Kubernetes version (use ): v1.8 beta Cloud provider or hardware configuration**: all OS (e.g. from /etc/os-release): all cc
/assign
Yikes, that's a fun issue. I guess we need to listen for PLEG events?
There's a spot in the kuberuntime sync where we think we can make the call.
well, i knew i would miss something ;-) good catch.
All thanks to workload perf testing
By the way, where is of CPU manager called if is set to or ? invoked by GC?
Hi, Besides, I wonder do we have a mechanism to recover things in state of CPU manager when kubelet restarts? Here I have a trial, seems it doesn't work as expected, that is: a few logs before and after the action of addcontainer and removecontainer in policy, to see the content of state: a pod that declares an exclusive CPU, and saw logs: kubelet the pod, to compare the state with that , I think if recovered, the content of them should be the same. but saw logs: Seems that the content in state did not recovered, this way, the CPU manager would behavior far from what we expected if kubelet restart. Did I miss something important?
is called as a part of internal container lifecycle pre-stop and post-stop hooks (see and ). Regarding Kubelet restarts, it is a known issue and captured in
priority/important-soon
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Well, I understand business in , what I wonder is upper behavor, that is, the behavior of in containerRuntime, When a container with restartStrategy exits, It seems that the runtime try to restart new container directly, without calling things like . So I failed to see the place where remove related hooks called in this case, so here I have a trial: a pod whose container is in restartStrategy of and would exit after a perid exited after a peiod: try to restart it but failed: restart after quite a preiod: So I think the resource is not recycled synchronously, and the remove related hook is called in GC. Am I right? Does it works in the way we expected?
Yes, that's exactly correct. The linked PR with the fix for this issue also helps the situation you described to converge faster.

// check the status of containers. for idx, container := range pod.Spec.Containers { containerStatus := podStatus.FindContainerStatusByName(container.Name) <ins> // Call internal container post-stop lifecycle hook for any non-running container so that any // allocated cpus are released immediately. If the container is restarted, cpus will be re-allocated // to it. if containerStatus != nil && containerStatus.State != kubecontainer.ContainerStateRunning { if err := m.internalLifecycle.PostStopContainer(containerStatus.ID.ID); err != nil { glog.Errorf("internal container post-stop lifecycle hook failed for container %v in pod %v with error %v", container.Name, pod.Name, err) } } </ins> // If container does not exist, or is not running, check whether we // need to restart it. if containerStatus == nil || containerStatus.State != kubecontainer.ContainerStateRunning {


Question:
Follow-up of Docker 1.13 introduced a change into the logging mechanism (), that split the log lines longer than 16K into chunks of no more than that. Current default logging integrations (fluentd-gcp and fluentd-es addons) do not support that and this might result in broken ingestion, i.e. when JSON parsing is broken for entries longer than 16K. One possible solution is introducing a fluentd plugin to the configuration to concat such entries in the similar way does it (assuming there is one out there, cc for that) /cc
Presumably this should be done in the Docker fluentd logging driver... The discussion on seems to be leaning that way...
In fluentd logging driver side, journald driver approach is better: I'm not sure buffering partial message in logging driver is good or not... In fluentd side, implementing or is needed.
do we need to block 1.8 release? can we please move this out of 1.8 if it's not?
Moved to 1.9, since it's not specific for 1.8
Is there any way this can be fixed by running a custom fluentd? Or is there a fluentd docker driver github issue I can follow for updates on when they expect to have this fixed? Txh
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Problem is that there's no fluentd plugin to fix the problem, so running a custom fluentd won't help, there's nothing to add to the configuration to make it work (unless you want to write your own solution) They don't think it's a docker problem and don't plan to fix it. However, it will be fixed in GKE
That seems fair from Docker's perspective. I'd rather have stable containers as well :) So you're thinking of switching from the current log drive - at least that's what my Google Container Optimized image in GKE 1.8.1 uses - to the journald driver to circumvent the problem? And then use a plugin like to recombine the partial messages?
No, that's not the case. Solution is TBD, but if you export your logs to Stackdriver (that has a limit of 100KB per entry) you won't see the difference
I think there's still a difference in that the 100KB stackdriver logging limit is larger than the 16KB docker limit at which the partial messaging starts, so without proper recombination of those chunked messages you'll lose the ability to send log lines larger than 16KB into stackdriver. However we're also running our own fluentd that sends messages straight into pub/sub, which has a limit of 10MB per message. And although we're trying to keep the messages much smaller than 16KB it would be nice to be able to send bigger messages without instrumenting our applications to push the messages into pub/sub themselves. I hope you find a nice solution that will make this possible :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten Still relevant, hasn't been fixed
I've been reading up on this issue, it's really too bad docker would not allow the configurable log-max-size option, this would be an easy temporary workaround to the issue until fully resolved. Regardless, I also wanted to point out a 2nd issue that the log splitting causes which might help increase the priority of this issue. If you are using json logging and have MCD or Structured Arguements attributes in your log messages, what ends up happening unfortunately is the attributes are lost on the split messages. This is a huge drawback right now. In my case I am using spring cloud sleuth which adds span and traceid's to all log messages allowing you to easily filter and correlate all related logging messages. When the messages are split, the traceid's are not part of the message in elasticsearch making filtering by traceid impossible. My application logging is setup as follows: is it still the plan to have fluentd put the logs back together into a single message? can anyone update if any progress is being made? thanks for your help.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten Still relevant, hasn't been fixed
Would using the "concat" fluentd plugin be relevant? It seems to have explicit support for our specific use case. Edit: For EFK, Adding fluentd-plugin-concat to fluentd-es image (, ) and adding to the Works well. Would a PR with those changes be acceptable?
trying to solve this as well. Having the same issue as where long lines are being truncated and MDC gone.
Please see The big problem here is concurrency. Just concatenating all partial log messages as they arrive may be wrong. Let's say your app logs two events, called and here. Then the Docker log messages may be written to the JSON log file (or whatever Docker logging driver is used, e.g. in plain Docker or Docker Swarm) in this order: The fluentd concat plugin cannot decide what to do. What records belong together? The good message is: This can be solved (similar to ) as Docker logs contain all the needed information here: So you just have to extend these methods and the struct by the partial log meta data: Then you have to fix Mind the ID and ordinal of the partial log metadata so that you concatenate the right records in the correct order.
I just edited my above comment because I am not sure how the case of linebreaks (i.e. those written by the application) in log messages relate to this issue. However, often you can control the logging of your application and let it write JSON to STDOUT (e.g. for Java / Spring Boot applications) so you do not have linebreaks. Back to the original issue: I have serious doubts that will do it under load with a lot of concurrent partial messages. I observed this in a Docker Swarm setup with fluentd and the concat plugin configured analogously.
Could you please reopen this issue? The concurrency is the problem here: Let your container simultaneously produce two log messages of 16KB (let's say 20 KB). Then fluentd will concatenate all four messages to one large (16+16+4+4 KB = 40 KB) message. The fluentd (or whatever you use) needs to know the ID of the partial log metadata to concatenate only those messages with identical IDs. So this needs to be implemented in the (default) local JSON file log driver in Docker (moby).
Problem is that ID comes with Docker 18.06 if I'm not wrong, this is why I didn't use it: do we want to require a docker version which is 2 months old?
is no longer much involved with the project. if you would like to open this back up to alter the implementation, please file a new issue
If you do so, please ping me as well.
I have the concat plugin but still it is not concatenating, Any Idea?
No idea why it does not work. For up-to-date information, I have inserted: Just after the kubernetes_metadate injection.

kind: ConfigMap apiVersion: v1 metadata: <del> name: fluentd-es-config-v0.1.5 </del> <ins> name: fluentd-es-config-v0.1.6 </ins> namespace: kube-system labels: addonmanager.kubernetes.io/mode: Reconcile


Question:
Follow-up of Docker 1.13 introduced a change into the logging mechanism (), that split the log lines longer than 16K into chunks of no more than that. Current default logging integrations (fluentd-gcp and fluentd-es addons) do not support that and this might result in broken ingestion, i.e. when JSON parsing is broken for entries longer than 16K. One possible solution is introducing a fluentd plugin to the configuration to concat such entries in the similar way does it (assuming there is one out there, cc for that) /cc
Presumably this should be done in the Docker fluentd logging driver... The discussion on seems to be leaning that way...
In fluentd logging driver side, journald driver approach is better: I'm not sure buffering partial message in logging driver is good or not... In fluentd side, implementing or is needed.
do we need to block 1.8 release? can we please move this out of 1.8 if it's not?
Moved to 1.9, since it's not specific for 1.8
Is there any way this can be fixed by running a custom fluentd? Or is there a fluentd docker driver github issue I can follow for updates on when they expect to have this fixed? Txh
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Problem is that there's no fluentd plugin to fix the problem, so running a custom fluentd won't help, there's nothing to add to the configuration to make it work (unless you want to write your own solution) They don't think it's a docker problem and don't plan to fix it. However, it will be fixed in GKE
That seems fair from Docker's perspective. I'd rather have stable containers as well :) So you're thinking of switching from the current log drive - at least that's what my Google Container Optimized image in GKE 1.8.1 uses - to the journald driver to circumvent the problem? And then use a plugin like to recombine the partial messages?
No, that's not the case. Solution is TBD, but if you export your logs to Stackdriver (that has a limit of 100KB per entry) you won't see the difference
I think there's still a difference in that the 100KB stackdriver logging limit is larger than the 16KB docker limit at which the partial messaging starts, so without proper recombination of those chunked messages you'll lose the ability to send log lines larger than 16KB into stackdriver. However we're also running our own fluentd that sends messages straight into pub/sub, which has a limit of 10MB per message. And although we're trying to keep the messages much smaller than 16KB it would be nice to be able to send bigger messages without instrumenting our applications to push the messages into pub/sub themselves. I hope you find a nice solution that will make this possible :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten Still relevant, hasn't been fixed
I've been reading up on this issue, it's really too bad docker would not allow the configurable log-max-size option, this would be an easy temporary workaround to the issue until fully resolved. Regardless, I also wanted to point out a 2nd issue that the log splitting causes which might help increase the priority of this issue. If you are using json logging and have MCD or Structured Arguements attributes in your log messages, what ends up happening unfortunately is the attributes are lost on the split messages. This is a huge drawback right now. In my case I am using spring cloud sleuth which adds span and traceid's to all log messages allowing you to easily filter and correlate all related logging messages. When the messages are split, the traceid's are not part of the message in elasticsearch making filtering by traceid impossible. My application logging is setup as follows: is it still the plan to have fluentd put the logs back together into a single message? can anyone update if any progress is being made? thanks for your help.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten Still relevant, hasn't been fixed
Would using the "concat" fluentd plugin be relevant? It seems to have explicit support for our specific use case. Edit: For EFK, Adding fluentd-plugin-concat to fluentd-es image (, ) and adding to the Works well. Would a PR with those changes be acceptable?
trying to solve this as well. Having the same issue as where long lines are being truncated and MDC gone.
Please see The big problem here is concurrency. Just concatenating all partial log messages as they arrive may be wrong. Let's say your app logs two events, called and here. Then the Docker log messages may be written to the JSON log file (or whatever Docker logging driver is used, e.g. in plain Docker or Docker Swarm) in this order: The fluentd concat plugin cannot decide what to do. What records belong together? The good message is: This can be solved (similar to ) as Docker logs contain all the needed information here: So you just have to extend these methods and the struct by the partial log meta data: Then you have to fix Mind the ID and ordinal of the partial log metadata so that you concatenate the right records in the correct order.
I just edited my above comment because I am not sure how the case of linebreaks (i.e. those written by the application) in log messages relate to this issue. However, often you can control the logging of your application and let it write JSON to STDOUT (e.g. for Java / Spring Boot applications) so you do not have linebreaks. Back to the original issue: I have serious doubts that will do it under load with a lot of concurrent partial messages. I observed this in a Docker Swarm setup with fluentd and the concat plugin configured analogously.
Could you please reopen this issue? The concurrency is the problem here: Let your container simultaneously produce two log messages of 16KB (let's say 20 KB). Then fluentd will concatenate all four messages to one large (16+16+4+4 KB = 40 KB) message. The fluentd (or whatever you use) needs to know the ID of the partial log metadata to concatenate only those messages with identical IDs. So this needs to be implemented in the (default) local JSON file log driver in Docker (moby).
Problem is that ID comes with Docker 18.06 if I'm not wrong, this is why I didn't use it: do we want to require a docker version which is 2 months old?
is no longer much involved with the project. if you would like to open this back up to alter the implementation, please file a new issue
If you do so, please ping me as well.
I have the concat plugin but still it is not concatenating, Any Idea?
No idea why it does not work. For up-to-date information, I have inserted: Just after the kubernetes_metadate injection.

@type kubernetes_metadata </filter> <ins> # Concatenate multi-line logs <filter **> @type concat key message multiline_end_regexp /n$/ separator "" </filter> </ins> <match **> @id elasticsearch @type elasticsearch


Question:
Follow-up of Docker 1.13 introduced a change into the logging mechanism (), that split the log lines longer than 16K into chunks of no more than that. Current default logging integrations (fluentd-gcp and fluentd-es addons) do not support that and this might result in broken ingestion, i.e. when JSON parsing is broken for entries longer than 16K. One possible solution is introducing a fluentd plugin to the configuration to concat such entries in the similar way does it (assuming there is one out there, cc for that) /cc
Presumably this should be done in the Docker fluentd logging driver... The discussion on seems to be leaning that way...
In fluentd logging driver side, journald driver approach is better: I'm not sure buffering partial message in logging driver is good or not... In fluentd side, implementing or is needed.
do we need to block 1.8 release? can we please move this out of 1.8 if it's not?
Moved to 1.9, since it's not specific for 1.8
Is there any way this can be fixed by running a custom fluentd? Or is there a fluentd docker driver github issue I can follow for updates on when they expect to have this fixed? Txh
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Problem is that there's no fluentd plugin to fix the problem, so running a custom fluentd won't help, there's nothing to add to the configuration to make it work (unless you want to write your own solution) They don't think it's a docker problem and don't plan to fix it. However, it will be fixed in GKE
That seems fair from Docker's perspective. I'd rather have stable containers as well :) So you're thinking of switching from the current log drive - at least that's what my Google Container Optimized image in GKE 1.8.1 uses - to the journald driver to circumvent the problem? And then use a plugin like to recombine the partial messages?
No, that's not the case. Solution is TBD, but if you export your logs to Stackdriver (that has a limit of 100KB per entry) you won't see the difference
I think there's still a difference in that the 100KB stackdriver logging limit is larger than the 16KB docker limit at which the partial messaging starts, so without proper recombination of those chunked messages you'll lose the ability to send log lines larger than 16KB into stackdriver. However we're also running our own fluentd that sends messages straight into pub/sub, which has a limit of 10MB per message. And although we're trying to keep the messages much smaller than 16KB it would be nice to be able to send bigger messages without instrumenting our applications to push the messages into pub/sub themselves. I hope you find a nice solution that will make this possible :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten Still relevant, hasn't been fixed
I've been reading up on this issue, it's really too bad docker would not allow the configurable log-max-size option, this would be an easy temporary workaround to the issue until fully resolved. Regardless, I also wanted to point out a 2nd issue that the log splitting causes which might help increase the priority of this issue. If you are using json logging and have MCD or Structured Arguements attributes in your log messages, what ends up happening unfortunately is the attributes are lost on the split messages. This is a huge drawback right now. In my case I am using spring cloud sleuth which adds span and traceid's to all log messages allowing you to easily filter and correlate all related logging messages. When the messages are split, the traceid's are not part of the message in elasticsearch making filtering by traceid impossible. My application logging is setup as follows: is it still the plan to have fluentd put the logs back together into a single message? can anyone update if any progress is being made? thanks for your help.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten Still relevant, hasn't been fixed
Would using the "concat" fluentd plugin be relevant? It seems to have explicit support for our specific use case. Edit: For EFK, Adding fluentd-plugin-concat to fluentd-es image (, ) and adding to the Works well. Would a PR with those changes be acceptable?
trying to solve this as well. Having the same issue as where long lines are being truncated and MDC gone.
Please see The big problem here is concurrency. Just concatenating all partial log messages as they arrive may be wrong. Let's say your app logs two events, called and here. Then the Docker log messages may be written to the JSON log file (or whatever Docker logging driver is used, e.g. in plain Docker or Docker Swarm) in this order: The fluentd concat plugin cannot decide what to do. What records belong together? The good message is: This can be solved (similar to ) as Docker logs contain all the needed information here: So you just have to extend these methods and the struct by the partial log meta data: Then you have to fix Mind the ID and ordinal of the partial log metadata so that you concatenate the right records in the correct order.
I just edited my above comment because I am not sure how the case of linebreaks (i.e. those written by the application) in log messages relate to this issue. However, often you can control the logging of your application and let it write JSON to STDOUT (e.g. for Java / Spring Boot applications) so you do not have linebreaks. Back to the original issue: I have serious doubts that will do it under load with a lot of concurrent partial messages. I observed this in a Docker Swarm setup with fluentd and the concat plugin configured analogously.
Could you please reopen this issue? The concurrency is the problem here: Let your container simultaneously produce two log messages of 16KB (let's say 20 KB). Then fluentd will concatenate all four messages to one large (16+16+4+4 KB = 40 KB) message. The fluentd (or whatever you use) needs to know the ID of the partial log metadata to concatenate only those messages with identical IDs. So this needs to be implemented in the (default) local JSON file log driver in Docker (moby).
Problem is that ID comes with Docker 18.06 if I'm not wrong, this is why I didn't use it: do we want to require a docker version which is 2 months old?
is no longer much involved with the project. if you would like to open this back up to alter the implementation, please file a new issue
If you do so, please ping me as well.
I have the concat plugin but still it is not concatenating, Any Idea?
No idea why it does not work. For up-to-date information, I have inserted: Just after the kubernetes_metadate injection.

apiVersion: apps/v1 kind: DaemonSet metadata: <del> name: fluentd-es-v2.2.0 </del> <ins> name: fluentd-es-v2.2.1 </ins> namespace: kube-system labels: k8s-app: fluentd-es <del> version: v2.2.0 </del> <ins> version: v2.2.1 </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile spec: selector: matchLabels: k8s-app: fluentd-es <del> version: v2.2.0 </del> <ins> version: v2.2.1 </ins> template: metadata: labels: k8s-app: fluentd-es kubernetes.io/cluster-service: "true" <del> version: v2.2.0 </del> <ins> version: v2.2.1 </ins> # This annotation ensures that fluentd does not get evicted if the node # supports critical pod annotation based priority scheme. # Note that this does not guarantee admission on the nodes (#40573).


Question:
Follow-up of Docker 1.13 introduced a change into the logging mechanism (), that split the log lines longer than 16K into chunks of no more than that. Current default logging integrations (fluentd-gcp and fluentd-es addons) do not support that and this might result in broken ingestion, i.e. when JSON parsing is broken for entries longer than 16K. One possible solution is introducing a fluentd plugin to the configuration to concat such entries in the similar way does it (assuming there is one out there, cc for that) /cc
Presumably this should be done in the Docker fluentd logging driver... The discussion on seems to be leaning that way...
In fluentd logging driver side, journald driver approach is better: I'm not sure buffering partial message in logging driver is good or not... In fluentd side, implementing or is needed.
do we need to block 1.8 release? can we please move this out of 1.8 if it's not?
Moved to 1.9, since it's not specific for 1.8
Is there any way this can be fixed by running a custom fluentd? Or is there a fluentd docker driver github issue I can follow for updates on when they expect to have this fixed? Txh
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Problem is that there's no fluentd plugin to fix the problem, so running a custom fluentd won't help, there's nothing to add to the configuration to make it work (unless you want to write your own solution) They don't think it's a docker problem and don't plan to fix it. However, it will be fixed in GKE
That seems fair from Docker's perspective. I'd rather have stable containers as well :) So you're thinking of switching from the current log drive - at least that's what my Google Container Optimized image in GKE 1.8.1 uses - to the journald driver to circumvent the problem? And then use a plugin like to recombine the partial messages?
No, that's not the case. Solution is TBD, but if you export your logs to Stackdriver (that has a limit of 100KB per entry) you won't see the difference
I think there's still a difference in that the 100KB stackdriver logging limit is larger than the 16KB docker limit at which the partial messaging starts, so without proper recombination of those chunked messages you'll lose the ability to send log lines larger than 16KB into stackdriver. However we're also running our own fluentd that sends messages straight into pub/sub, which has a limit of 10MB per message. And although we're trying to keep the messages much smaller than 16KB it would be nice to be able to send bigger messages without instrumenting our applications to push the messages into pub/sub themselves. I hope you find a nice solution that will make this possible :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten Still relevant, hasn't been fixed
I've been reading up on this issue, it's really too bad docker would not allow the configurable log-max-size option, this would be an easy temporary workaround to the issue until fully resolved. Regardless, I also wanted to point out a 2nd issue that the log splitting causes which might help increase the priority of this issue. If you are using json logging and have MCD or Structured Arguements attributes in your log messages, what ends up happening unfortunately is the attributes are lost on the split messages. This is a huge drawback right now. In my case I am using spring cloud sleuth which adds span and traceid's to all log messages allowing you to easily filter and correlate all related logging messages. When the messages are split, the traceid's are not part of the message in elasticsearch making filtering by traceid impossible. My application logging is setup as follows: is it still the plan to have fluentd put the logs back together into a single message? can anyone update if any progress is being made? thanks for your help.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten Still relevant, hasn't been fixed
Would using the "concat" fluentd plugin be relevant? It seems to have explicit support for our specific use case. Edit: For EFK, Adding fluentd-plugin-concat to fluentd-es image (, ) and adding to the Works well. Would a PR with those changes be acceptable?
trying to solve this as well. Having the same issue as where long lines are being truncated and MDC gone.
Please see The big problem here is concurrency. Just concatenating all partial log messages as they arrive may be wrong. Let's say your app logs two events, called and here. Then the Docker log messages may be written to the JSON log file (or whatever Docker logging driver is used, e.g. in plain Docker or Docker Swarm) in this order: The fluentd concat plugin cannot decide what to do. What records belong together? The good message is: This can be solved (similar to ) as Docker logs contain all the needed information here: So you just have to extend these methods and the struct by the partial log meta data: Then you have to fix Mind the ID and ordinal of the partial log metadata so that you concatenate the right records in the correct order.
I just edited my above comment because I am not sure how the case of linebreaks (i.e. those written by the application) in log messages relate to this issue. However, often you can control the logging of your application and let it write JSON to STDOUT (e.g. for Java / Spring Boot applications) so you do not have linebreaks. Back to the original issue: I have serious doubts that will do it under load with a lot of concurrent partial messages. I observed this in a Docker Swarm setup with fluentd and the concat plugin configured analogously.
Could you please reopen this issue? The concurrency is the problem here: Let your container simultaneously produce two log messages of 16KB (let's say 20 KB). Then fluentd will concatenate all four messages to one large (16+16+4+4 KB = 40 KB) message. The fluentd (or whatever you use) needs to know the ID of the partial log metadata to concatenate only those messages with identical IDs. So this needs to be implemented in the (default) local JSON file log driver in Docker (moby).
Problem is that ID comes with Docker 18.06 if I'm not wrong, this is why I didn't use it: do we want to require a docker version which is 2 months old?
is no longer much involved with the project. if you would like to open this back up to alter the implementation, please file a new issue
If you do so, please ping me as well.
I have the concat plugin but still it is not concatenating, Any Idea?
No idea why it does not work. For up-to-date information, I have inserted: Just after the kubernetes_metadate injection.

path: /var/lib/docker/containers - name: config-volume configMap: <del> name: fluentd-es-config-v0.1.5 </del> <ins> name: fluentd-es-config-v0.1.6 </ins>


Question:
Follow-up of Docker 1.13 introduced a change into the logging mechanism (), that split the log lines longer than 16K into chunks of no more than that. Current default logging integrations (fluentd-gcp and fluentd-es addons) do not support that and this might result in broken ingestion, i.e. when JSON parsing is broken for entries longer than 16K. One possible solution is introducing a fluentd plugin to the configuration to concat such entries in the similar way does it (assuming there is one out there, cc for that) /cc
Presumably this should be done in the Docker fluentd logging driver... The discussion on seems to be leaning that way...
In fluentd logging driver side, journald driver approach is better: I'm not sure buffering partial message in logging driver is good or not... In fluentd side, implementing or is needed.
do we need to block 1.8 release? can we please move this out of 1.8 if it's not?
Moved to 1.9, since it's not specific for 1.8
Is there any way this can be fixed by running a custom fluentd? Or is there a fluentd docker driver github issue I can follow for updates on when they expect to have this fixed? Txh
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Problem is that there's no fluentd plugin to fix the problem, so running a custom fluentd won't help, there's nothing to add to the configuration to make it work (unless you want to write your own solution) They don't think it's a docker problem and don't plan to fix it. However, it will be fixed in GKE
That seems fair from Docker's perspective. I'd rather have stable containers as well :) So you're thinking of switching from the current log drive - at least that's what my Google Container Optimized image in GKE 1.8.1 uses - to the journald driver to circumvent the problem? And then use a plugin like to recombine the partial messages?
No, that's not the case. Solution is TBD, but if you export your logs to Stackdriver (that has a limit of 100KB per entry) you won't see the difference
I think there's still a difference in that the 100KB stackdriver logging limit is larger than the 16KB docker limit at which the partial messaging starts, so without proper recombination of those chunked messages you'll lose the ability to send log lines larger than 16KB into stackdriver. However we're also running our own fluentd that sends messages straight into pub/sub, which has a limit of 10MB per message. And although we're trying to keep the messages much smaller than 16KB it would be nice to be able to send bigger messages without instrumenting our applications to push the messages into pub/sub themselves. I hope you find a nice solution that will make this possible :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten Still relevant, hasn't been fixed
I've been reading up on this issue, it's really too bad docker would not allow the configurable log-max-size option, this would be an easy temporary workaround to the issue until fully resolved. Regardless, I also wanted to point out a 2nd issue that the log splitting causes which might help increase the priority of this issue. If you are using json logging and have MCD or Structured Arguements attributes in your log messages, what ends up happening unfortunately is the attributes are lost on the split messages. This is a huge drawback right now. In my case I am using spring cloud sleuth which adds span and traceid's to all log messages allowing you to easily filter and correlate all related logging messages. When the messages are split, the traceid's are not part of the message in elasticsearch making filtering by traceid impossible. My application logging is setup as follows: is it still the plan to have fluentd put the logs back together into a single message? can anyone update if any progress is being made? thanks for your help.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten Still relevant, hasn't been fixed
Would using the "concat" fluentd plugin be relevant? It seems to have explicit support for our specific use case. Edit: For EFK, Adding fluentd-plugin-concat to fluentd-es image (, ) and adding to the Works well. Would a PR with those changes be acceptable?
trying to solve this as well. Having the same issue as where long lines are being truncated and MDC gone.
Please see The big problem here is concurrency. Just concatenating all partial log messages as they arrive may be wrong. Let's say your app logs two events, called and here. Then the Docker log messages may be written to the JSON log file (or whatever Docker logging driver is used, e.g. in plain Docker or Docker Swarm) in this order: The fluentd concat plugin cannot decide what to do. What records belong together? The good message is: This can be solved (similar to ) as Docker logs contain all the needed information here: So you just have to extend these methods and the struct by the partial log meta data: Then you have to fix Mind the ID and ordinal of the partial log metadata so that you concatenate the right records in the correct order.
I just edited my above comment because I am not sure how the case of linebreaks (i.e. those written by the application) in log messages relate to this issue. However, often you can control the logging of your application and let it write JSON to STDOUT (e.g. for Java / Spring Boot applications) so you do not have linebreaks. Back to the original issue: I have serious doubts that will do it under load with a lot of concurrent partial messages. I observed this in a Docker Swarm setup with fluentd and the concat plugin configured analogously.
Could you please reopen this issue? The concurrency is the problem here: Let your container simultaneously produce two log messages of 16KB (let's say 20 KB). Then fluentd will concatenate all four messages to one large (16+16+4+4 KB = 40 KB) message. The fluentd (or whatever you use) needs to know the ID of the partial log metadata to concatenate only those messages with identical IDs. So this needs to be implemented in the (default) local JSON file log driver in Docker (moby).
Problem is that ID comes with Docker 18.06 if I'm not wrong, this is why I didn't use it: do we want to require a docker version which is 2 months old?
is no longer much involved with the project. if you would like to open this back up to alter the implementation, please file a new issue
If you do so, please ping me as well.
I have the concat plugin but still it is not concatenating, Any Idea?
No idea why it does not work. For up-to-date information, I have inserted: Just after the kubernetes_metadate injection.

gem 'fluentd', '<=1.2.4' gem 'activesupport', '~>5.2.1' <del> gem 'fluent-plugin-kubernetes_metadata_filter', '~>2.0.0' gem 'fluent-plugin-elasticsearch', '~>2.11.5' gem 'fluent-plugin-systemd', '~>1.0.1' </del> <ins> gem 'fluent-plugin-concat', '~>2.3.0' </ins> gem 'fluent-plugin-detect-exceptions', '~>0.0.11' <del> gem 'fluent-plugin-prometheus', '~>1.0.1' </del> <ins> gem 'fluent-plugin-elasticsearch', '~>2.11.5' gem 'fluent-plugin-kubernetes_metadata_filter', '~>2.0.0' </ins> gem 'fluent-plugin-multi-format-parser', '~>1.0.0' <ins> gem 'fluent-plugin-prometheus', '~>1.0.1' gem 'fluent-plugin-systemd', '~>1.0.1' </ins> gem 'oj', '~>3.6.5'


Question:
The following that have been failing on our 5k-node tests (https://k8s-): [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] ReplicationController light Should scale from 1 pod to 2 pods [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] ReplicationController light Should scale from 2 pods to 1 pod This is part of the correctness suite and large cluster tests are release-blocking. cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/autoscaling sig/scalability: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Could you PTAL into this or redirect as apt? It's a release-blocker.
xRef
The tests seem to be passing on the normal correctness job - https://k8s- (except the latest run) but they're failing on large cluster and the reason seems to be timeout waiting for scaling to happen.
If i have to guess it is heapster who is failing to provide metrics. HPA itself has no direct dependency on the number of nodes in the cluster. Assigning to .
is going to take a look.
From the logs it seems like Heapster is unable to get metrics from nodes. It appears to be a separate issue from addon resizer restarts.
Thanks for checking it. I have got the same logs for smaller cluster (10 nodes), so at least we know that the problem is not related to big number of nodes/pods.
- I'm seeing the test pass on our regular (small cluster) gce job - https://k8s- And seems like it was passing before and then started failing on our 2k-node job - https://k8s-
thanks for pointing on it. The main difference between runs on small and big cluster - special node that we are creating for heapster container. And my local experiment have shown that without this node everything works, but with it - fails. So I assume that it's more network related. I'll try to figure out what's going on.
Removing this as release-blocker and downing the priority as it's a setup issue specific to use of heapster node. Would continue to work on it (thanks!).

local network=$(make-gcloud-network-argument  "${NETWORK_PROJECT}"  "${REGION}"  <del> "${NETWORK}" </del> <ins> "${NETWORK}"  </ins> "${SUBNETWORK:-}"  ""  "${ENABLE_IP_ALIASES:-}"


Question:
When creating a service passing the annotation , two IPs are created in the service, a private and a public: Having a private IP is not a desired scenario for service discovery in federated global environments. If we have, for example, a cluster in GKE and join a local OpenStack cluster, this private IP is written in the DNS zone but the GKE cluster won't be able to access it. We could add an annotation to say we don't want the private IP to be registered in service's LoadBalancer Ingress field, like: or And when it's false, only the public IP would be written in the service, therefore in the DNS zone /kind feature
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack /sig federation
'' sounds better.
I don't like adding annotations just to make things work. I counter-propose just not publishing the private IP whenever there is also a floating-ip (without a configuration option to control this behaviour).

glog.V(2).Infof("Deleted obsolete listener: %s", listener.ID) } <del> status := &v1.LoadBalancerStatus{} status.Ingress = []v1.LoadBalancerIngress{{IP: loadbalancer.VipAddress}} </del> portID := loadbalancer.VipPortID floatIP, err := getFloatingIPByPortID(lbaas.network, portID) if err != nil && err != ErrNotFound {


Question:
When creating a service passing the annotation , two IPs are created in the service, a private and a public: Having a private IP is not a desired scenario for service discovery in federated global environments. If we have, for example, a cluster in GKE and join a local OpenStack cluster, this private IP is written in the DNS zone but the GKE cluster won't be able to access it. We could add an annotation to say we don't want the private IP to be registered in service's LoadBalancer Ingress field, like: or And when it's false, only the public IP would be written in the service, therefore in the DNS zone /kind feature
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack /sig federation
'' sounds better.
I don't like adding annotations just to make things work. I counter-propose just not publishing the private IP whenever there is also a floating-ip (without a configuration option to control this behaviour).

return nil, fmt.Errorf("Error creating LB floatingip %+v: %v", floatIPOpts, err) } } <ins> status := &v1.LoadBalancerStatus{} </ins> if floatIP != nil { <del> status.Ingress = append(status.Ingress, v1.LoadBalancerIngress{IP: floatIP.FloatingIP}) </del> <ins> status.Ingress = []v1.LoadBalancerIngress{{IP: floatIP.FloatingIP}} } else { status.Ingress = []v1.LoadBalancerIngress{{IP: loadbalancer.VipAddress}} </ins> } if lbaas.opts.ManageSecurityGroups {


Question:
When creating a service passing the annotation , two IPs are created in the service, a private and a public: Having a private IP is not a desired scenario for service discovery in federated global environments. If we have, for example, a cluster in GKE and join a local OpenStack cluster, this private IP is written in the DNS zone but the GKE cluster won't be able to access it. We could add an annotation to say we don't want the private IP to be registered in service's LoadBalancer Ingress field, like: or And when it's false, only the public IP would be written in the service, therefore in the DNS zone /kind feature
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack /sig federation
'' sounds better.
I don't like adding annotations just to make things work. I counter-propose just not publishing the private IP whenever there is also a floating-ip (without a configuration option to control this behaviour).

} status := &v1.LoadBalancerStatus{} <del> status.Ingress = []v1.LoadBalancerIngress{{IP: vip.Address}} </del> if floatingPool != "" && !internalAnnotation { floatIPOpts := floatingips.CreateOpts{ FloatingNetworkID: floatingPool,


Question:
When creating a service passing the annotation , two IPs are created in the service, a private and a public: Having a private IP is not a desired scenario for service discovery in federated global environments. If we have, for example, a cluster in GKE and join a local OpenStack cluster, this private IP is written in the DNS zone but the GKE cluster won't be able to access it. We could add an annotation to say we don't want the private IP to be registered in service's LoadBalancer Ingress field, like: or And when it's false, only the public IP would be written in the service, therefore in the DNS zone /kind feature
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack /sig federation
'' sounds better.
I don't like adding annotations just to make things work. I counter-propose just not publishing the private IP whenever there is also a floating-ip (without a configuration option to control this behaviour).

return nil, fmt.Errorf("Error creating floatingip for openstack load balancer %s: %v", name, err) } <del> status.Ingress = append(status.Ingress, v1.LoadBalancerIngress{IP: floatIP.FloatingIP}) </del> <ins> status.Ingress = []v1.LoadBalancerIngress{{IP: floatIP.FloatingIP}} } else { status.Ingress = []v1.LoadBalancerIngress{{IP: vip.Address}} </ins> } return status, nil


Question:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: FC plugin returns TargetWWNs at GetVolumeName(), but TargetWWNs are not unique identifier if user has multiple volumes from same target wwn. As a result, after introducing attach/detach functions, FC plugin always fails to attach volumes if a pod requests 2+ luns. The GetVolumeName() is called from GetUniqueVolumeNameFromSpec() during Pod creation. FC plugin should return unique identifier in the GetVolumeName(). If user uses WWID to specify FC volume, this problem doesn't happen because the WWID is unique identifier. What you expected to happen: All FC volumes are attached to pod. How to reproduce it (as minimally and precisely as possible): Create two PVs and PVCs which uses same target wwn. Create Pod with these two PVCs Anything else we need to know?: Environment: Kubernetes version (use ):Install tools: local-up- /cc
/sig storage
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals

return "", err } <del> if len(volumeSource.TargetWWNs) != 0 { </del> <ins> // API server validates these parameters beforehand but attach/detach // controller creates volumespec without validation. They may be nil // or zero length. We should check again to avoid unexpected conditions. if len(volumeSource.TargetWWNs) != 0 && volumeSource.Lun != nil { </ins> // TargetWWNs are the FibreChannel target worldwide names <del> return fmt.Sprintf("%v", volumeSource.TargetWWNs), nil </del> <ins> return fmt.Sprintf("%v:%v", volumeSource.TargetWWNs, *volumeSource.Lun), nil </ins> } else if len(volumeSource.WWIDs) != 0 { // WWIDs are the FibreChannel World Wide Identifiers return fmt.Sprintf("%v", volumeSource.WWIDs), nil


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If a pod is already running in the default namespace with the expected name (i.e. ), the PV recycle method just assumes it's a recycler pod. However, if it is not a recycler, it could result in the volume being reused without the data being cleared. What you expected to happen: The recycled volume should not be reused until the data has been verifiably deleted. The easiest approach would be to make it an error to not run the recycler pod successfully, i.e. delete these lines: I think that should be safe, since the recycler will retry. Other errors in that method should also be considered fatal, e.g.:
/cc
How about generating a unique name for each recycler pod so that it doesn't matter what existing pods are named?
Could someone from your team pick this up?
Ack
OpenShift runs the recycler pod in a dedicated namespace, not visible to users. Maybe it could be a good idea to do so also in Kubernetes. Anyway, killing the old recycler is probably the best option - it could be forged by user to block recycling and cheat quota. /assign
Agree with running in a dedicated namespace, but it sounds like this feature is just being deprecated, so it might not be worth the effort. Thanks for picking this up.

// attempted before returning. // // In case there is a pod with the same namespace+name already running, this <del> // function assumes it's an older instance of the recycler pod and watches // this old pod instead of starting a new one. </del> <ins> // function deletes it as it is not able to judge if it is an old recycler // or user has forged a fake recycler to block Kubernetes from recycling.// </ins> // //  pod - the pod designed by a volume plugin to recycle the volume. pod.Name //        will be overwritten with unique name based on PV.Name.


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If a pod is already running in the default namespace with the expected name (i.e. ), the PV recycle method just assumes it's a recycler pod. However, if it is not a recycler, it could result in the volume being reused without the data being cleared. What you expected to happen: The recycled volume should not be reused until the data has been verifiably deleted. The easiest approach would be to make it an error to not run the recycler pod successfully, i.e. delete these lines: I think that should be safe, since the recycler will retry. Other errors in that method should also be considered fatal, e.g.:
/cc
How about generating a unique name for each recycler pod so that it doesn't matter what existing pods are named?
Could someone from your team pick this up?
Ack
OpenShift runs the recycler pod in a dedicated namespace, not visible to users. Maybe it could be a good idea to do so also in Kubernetes. Anyway, killing the old recycler is probably the best option - it could be forged by user to block recycling and cheat quota. /assign
Agree with running in a dedicated namespace, but it sounds like this feature is just being deprecated, so it might not be worth the effort. Thanks for picking this up.

_, err = recyclerClient.CreatePod(pod) if err != nil { if errors.IsAlreadyExists(err) { <del> glog.V(5).Infof("old recycler pod %q found for volume", pod.Name) </del> <ins> deleteErr := recyclerClient.DeletePod(pod.Name, pod.Namespace) if deleteErr != nil { return fmt.Errorf("failed to delete old recycler pod %s/%s: %s", pod.Namespace, pod.Name, deleteErr) } // Recycler will try again and the old pod will be hopefuly deleted // at that time. return fmt.Errorf("old recycler pod found, will retry later") </ins> } else { return fmt.Errorf("unexpected error creating recycler pod:  %+vn", err) } } <del> defer func(pod *v1.Pod) { glog.V(2).Infof("deleting recycler pod %s/%s", pod.Namespace, pod.Name) if err := recyclerClient.DeletePod(pod.Name, pod.Namespace); err != nil { glog.Errorf("failed to delete recycler pod %s/%s: %v", pod.Namespace, pod.Name, err) } }(pod) </del> <ins> err = waitForPod(pod, recyclerClient, podCh) // In all cases delete the recycler pod and log its result. glog.V(2).Infof("deleting recycler pod %s/%s", pod.Namespace, pod.Name) deleteErr := recyclerClient.DeletePod(pod.Name, pod.Namespace) if deleteErr != nil { glog.Errorf("failed to delete recycler pod %s/%s: %v", pod.Namespace, pod.Name, err) } // Returning recycler error is preferred, the pod will be deleted again on // the next retry. if err != nil { return fmt.Errorf("failed to recycle volume: %s", err) } // Recycle succeeded but we failed to delete the recycler pod. Report it, // the controller will re-try recycling the PV again shortly. if deleteErr != nil { return fmt.Errorf("failed to delete recycler pod: %s", deleteErr) } return nil } </ins> <del> // Now only the old pod or the new pod run. Watch it until it finishes // and send all events on the pod to the PV </del> <ins> // waitForPod watches the pod it until it finishes and send all events on the // pod to the PV. func waitForPod(pod *v1.Pod, recyclerClient recyclerClient, podCh <-chan watch.Event) error { </ins> for { event, ok := <-podCh if !ok {


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If a pod is already running in the default namespace with the expected name (i.e. ), the PV recycle method just assumes it's a recycler pod. However, if it is not a recycler, it could result in the volume being reused without the data being cleared. What you expected to happen: The recycled volume should not be reused until the data has been verifiably deleted. The easiest approach would be to make it an error to not run the recycler pod successfully, i.e. delete these lines: I think that should be safe, since the recycler will retry. Other errors in that method should also be considered fatal, e.g.:
/cc
How about generating a unique name for each recycler pod so that it doesn't matter what existing pods are named?
Could someone from your team pick this up?
Ack
OpenShift runs the recycler pod in a dedicated namespace, not visible to users. Maybe it could be a good idea to do so also in Kubernetes. Anyway, killing the old recycler is probably the best option - it could be forged by user to block recycling and cheat quota. /assign
Agree with running in a dedicated namespace, but it sounds like this feature is just being deprecated, so it might not be worth the effort. Thanks for picking this up.

{v1.EventTypeWarning, "Unable to mount volumes for pod "recycler-for-podRecyclerFailure_default(3c9809e5-347c-11e6-a79b-3c970e965218)": timeout expired waiting for volumes to attach/mount"}, {v1.EventTypeWarning, "Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"recycler-for-podRecyclerFailure". list of unattached/unmounted"}, }, <del> expectedError: "Pod was active on the node longer than specified deadline", </del> <ins> expectedError: "failed to recycle volume: Pod was active on the node longer than specified deadline", </ins> }, { // Recycler pod gets deleted


Question:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If a pod is already running in the default namespace with the expected name (i.e. ), the PV recycle method just assumes it's a recycler pod. However, if it is not a recycler, it could result in the volume being reused without the data being cleared. What you expected to happen: The recycled volume should not be reused until the data has been verifiably deleted. The easiest approach would be to make it an error to not run the recycler pod successfully, i.e. delete these lines: I think that should be safe, since the recycler will retry. Other errors in that method should also be considered fatal, e.g.:
/cc
How about generating a unique name for each recycler pod so that it doesn't matter what existing pods are named?
Could someone from your team pick this up?
Ack
OpenShift runs the recycler pod in a dedicated namespace, not visible to users. Maybe it could be a good idea to do so also in Kubernetes. Anyway, killing the old recycler is probably the best option - it could be forged by user to block recycling and cheat quota. /assign
Agree with running in a dedicated namespace, but it sounds like this feature is just being deprecated, so it might not be worth the effort. Thanks for picking this up.

expectedEvents: []mockEvent{ {v1.EventTypeNormal, "Successfully assigned recycler-for-podRecyclerDeleted to 127.0.0.1"}, }, <del> expectedError: "recycler pod was deleted", </del> <ins> expectedError: "failed to recycle volume: recycler pod was deleted", </ins> }, { // Another recycler pod is already running <del> name:        "RecyclerRunning", existingPod: newPod("podOldRecycler", v1.PodRunning, ""), createPod:   newPod("podNewRecycler", v1.PodFailed, "mock message"), eventSequence: []watch.Event{ // Old pod succeeds newPodEvent(watch.Modified, "podOldRecycler", v1.PodSucceeded, ""), }, // No error = old pod succeeded. If the new pod was used, there // would be error with "mock message". expectedError: "", }, { // Another recycler pod is already running and fails name:        "FailedRecyclerRunning", existingPod: newPod("podOldRecycler", v1.PodRunning, ""), createPod:   newPod("podNewRecycler", v1.PodFailed, "mock message"), eventSequence: []watch.Event{ // Old pod failure newPodEvent(watch.Modified, "podOldRecycler", v1.PodFailed, "Pod was active on the node longer than specified deadline"), }, // If the new pod was used, there would be error with "mock message". expectedError: "Pod was active on the node longer than specified deadline", </del> <ins> name:          "RecyclerRunning", existingPod:   newPod("podOldRecycler", v1.PodRunning, ""), createPod:     newPod("podNewRecycler", v1.PodFailed, "mock message"), eventSequence: []watch.Event{}, expectedError: "old recycler pod found, will retry later", </ins> }, }


Question:
Is this a BUG REPORT or FEATURE REQUEST?: What happened: Kubeadm E2E test is currently broken due to the commit I report the same issue on my local cluster. If I revert PR , everything works fine. We believe the current weave-net plugin doesn't work with the CNI version change. Right now all kubeadm related development is blocked on this. Could we consider reverting this change from head until we figure out what's wrong in weave-net/CNI? More context in
/assign
Please reopen this in kubernetes/kubeadm. There is an issue as e2e tests & PR job are failing, but locally the CNI bit seem to work for me, so I think it has something with building the debs to do. Not technically. We just don't have any coverage right now. Not IMO. We just need to forward-fix the issue. Closing this; you can reopen in kubernetes/kubeadm
kubeadm e2e test still red. Do you have any progress or hint in terms how to fix it forward? Just to clarify, If I install k8s and kubeadm locally using deb, I can repro the issue on my test box. I could believe it's sth wrong with debs. In the meantime, CNI and plug-in like weave-net change is a big deal. We clearly didn't test it thoroughly for this PR. Reverting the change seems to be the clean solution.
FYI kubernetes/release bumps new CNI releases for deb/rpm packages.
Thanks.
Is this the fix needed to fix these tests?
I actually think that will take us in the other direction. During the kubeadm e2e tests, we use the deb packages built by bazel. During our official release process, we use the debs built by anago (in the release repo). Bumping CNI in our bazel build broken our e2e tests. Bumping CNI in the release repo scripts will similarly break our official release builds as well. If I understand the problem correctly, the only way to fix both of them is for the CNI providers like weave-net to be updated to handle the changes of upgrading CNI to 0.6.0.
I think this issue worth some more discussion. Re-open the case as it might not be affecting kubeadm only
sorry, i cannot reopen the issue as it's closed by But let's continue to discuss the option here.
Yes, I think so
This will be fixed by ; this was due to the CNI binaries being extracted in the wrong place. It is kind of a kubeadm packaging issue. (Reopening just to make the PR close this)
nice find!

pkg_tar( name = "kubernetes-cni-data", <del> package_dir = "/opt/cni", </del> <ins> package_dir = "/opt/cni/bin", </ins> deps = ["@kubernetes_cni//file"], )


Question:
/kind bug Right now we do not validate PV updates so admin can update at any time. Kubernetes has no facility to reflect this update - it won't magically re-mount an existing volume to e.g. a different NFS server. IMO the PersistentVolumeSource to be read-only and set to stone once created.
/assign
I think it is a good idea to make the VolumeSource read-only but I am afraid ship has sailed in v1 and existing code/tooling that relies on the ability to modify PVs will break as a result of this change.
- do you have any advice on how we should proceed on fixing this issue?

func ValidatePersistentVolumeUpdate(newPv, oldPv *api.PersistentVolume) field.ErrorList { allErrs := field.ErrorList{} allErrs = ValidatePersistentVolume(newPv) <ins> // PersistentVolumeSource should be immutable after creation. if !apiequality.Semantic.DeepEqual(newPv.Spec.PersistentVolumeSource, oldPv.Spec.PersistentVolumeSource) { allErrs = append(allErrs, field.Forbidden(field.NewPath("spec", "persistentvolumesource"), "is immutable after creation")) } </ins> newPv.Status = oldPv.Status return allErrs }


Question:
/kind bug Right now we do not validate PV updates so admin can update at any time. Kubernetes has no facility to reflect this update - it won't magically re-mount an existing volume to e.g. a different NFS server. IMO the PersistentVolumeSource to be read-only and set to stone once created.
/assign
I think it is a good idea to make the VolumeSource read-only but I am afraid ship has sailed in v1 and existing code/tooling that relies on the ability to modify PVs will break as a result of this change.
- do you have any advice on how we should proceed on fixing this issue?

} <ins> func TestValidatePersistentVolumeSourceUpdate(t *testing.T) { validVolume := testVolume("foo", "", api.PersistentVolumeSpec{ Capacity: api.ResourceList{ api.ResourceName(api.ResourceStorage): resource.MustParse("1G"), }, AccessModes: []api.PersistentVolumeAccessMode{api.ReadWriteOnce}, PersistentVolumeSource: api.PersistentVolumeSource{ HostPath: &api.HostPathVolumeSource{ Path: "/foo", Type: newHostPathType(string(api.HostPathDirectory)), }, }, StorageClassName: "valid", }) validPvSourceNoUpdate := validVolume.DeepCopy() invalidPvSourceUpdateType := validVolume.DeepCopy() invalidPvSourceUpdateType.Spec.PersistentVolumeSource = api.PersistentVolumeSource{ FlexVolume: &api.FlexVolumeSource{ Driver: "kubernetes.io/blue", FSType: "ext4", }, } invalidPvSourceUpdateDeep := validVolume.DeepCopy() invalidPvSourceUpdateDeep.Spec.PersistentVolumeSource = api.PersistentVolumeSource{ HostPath: &api.HostPathVolumeSource{ Path: "/updated", Type: newHostPathType(string(api.HostPathDirectory)), }, } scenarios := map[string]struct { isExpectedFailure bool oldVolume         *api.PersistentVolume newVolume         *api.PersistentVolume }{ "condition-no-update": { isExpectedFailure: false, oldVolume:         validVolume, newVolume:         validPvSourceNoUpdate, }, "condition-update-source-type": { isExpectedFailure: true, oldVolume:         validVolume, newVolume:         invalidPvSourceUpdateType, }, "condition-update-source-deep": { isExpectedFailure: true, oldVolume:         validVolume, newVolume:         invalidPvSourceUpdateDeep, }, } for name, scenario := range scenarios { errs := ValidatePersistentVolumeUpdate(scenario.newVolume, scenario.oldVolume) if len(errs) == 0 && scenario.isExpectedFailure { t.Errorf("Unexpected success for scenario: %s", name) } if len(errs) > 0 && !scenario.isExpectedFailure { t.Errorf("Unexpected failure for scenario: %s - %+v", name, errs) } } } </ins> func TestValidateLocalVolumes(t *testing.T) { scenarios := map[string]struct { isExpectedFailure bool


Question:
I also hit when running cri-containerd as the container runtime. The reason is that we don't populate stats like in cri-containerd integration now. is adding the containerd support in cadvisor, after that we should have full summary metrics. However, heapster should not panic in this case anyway. Can we cut a heapster patch release and update heapster in the main repo?
+1. Many new CRI integration would run into this issue.
In fact, as far as I am aware, not a single CRI implementation currently implements any of the stats calls.
I beg to differ: That's the reason why filed the issue.
Yeah does implement the CRI stats. Currently it only support core metrics, which is only part of summary api. We are working on integration with cadvisor, before that some fields in summary api such as will be nil. Current heapster version will crash when is nil, which shouldn't happen. :)
/cc
My bad, thanks for the pointer :slightlysmilingface:. I'll correct my statement: I believe no CRI implementation except for cri-containerd implements the stats calls. The ones I explicitly checked are: dockershim, cri-o, rktlet.
Something I'm wondering about though: The intention of the stats calls is in order to be able to get rid of cAdvisor. Making the stats calls depend on cAdvisor seems to cover up the issues it has. Re:
Here is the reason that we still have to integrate with cadvisor for now.
/cc
If I understand correctly this is meant as a migration mean, so that both are possible for some time, and then we drop cAdvisor once a reasonable migration period has been granted.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I will take a look at Heapster and try to fix this. Can you advise how to reproduce the issue? This will help a lot with debugging and verifying a fix once it's ready.
this is already fixed at head and 1.4.3 version of heapster The problem that is using kubernetes version from head and we have there 1.5.0-beta.0: I see two possible solutions: version of heapster to 1.4.3 at head of kubernetes. Piotr mentioned that 1.5.0 has still a lot of problems, so it maybe makes sense. a new release of heapster (1.5.0-beta.1) and after that bump version at head.
And thanks for taking this issue.
Not directly related to this issue: a more general discussion is at We'd like to understand which fields are required and optional in heapster, so that we won't crash heapster with partial result.
We've merged a work around for now Hopefully it works. But anyway, we should fix this in 1.9.
what needs to be done for 1.9 here?
we'll need an assignee as well
We just need to update heapster version.

- /heapster - --source=kubernetes.summary_api:'' - --sink=gcm <del> - image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.0 </del> <ins> - image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.1 </ins> name: eventer command: - /eventer


Question:
I also hit when running cri-containerd as the container runtime. The reason is that we don't populate stats like in cri-containerd integration now. is adding the containerd support in cadvisor, after that we should have full summary metrics. However, heapster should not panic in this case anyway. Can we cut a heapster patch release and update heapster in the main repo?
+1. Many new CRI integration would run into this issue.
In fact, as far as I am aware, not a single CRI implementation currently implements any of the stats calls.
I beg to differ: That's the reason why filed the issue.
Yeah does implement the CRI stats. Currently it only support core metrics, which is only part of summary api. We are working on integration with cadvisor, before that some fields in summary api such as will be nil. Current heapster version will crash when is nil, which shouldn't happen. :)
/cc
My bad, thanks for the pointer :slightlysmilingface:. I'll correct my statement: I believe no CRI implementation except for cri-containerd implements the stats calls. The ones I explicitly checked are: dockershim, cri-o, rktlet.
Something I'm wondering about though: The intention of the stats calls is in order to be able to get rid of cAdvisor. Making the stats calls depend on cAdvisor seems to cover up the issues it has. Re:
Here is the reason that we still have to integrate with cadvisor for now.
/cc
If I understand correctly this is meant as a migration mean, so that both are possible for some time, and then we drop cAdvisor once a reasonable migration period has been granted.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I will take a look at Heapster and try to fix this. Can you advise how to reproduce the issue? This will help a lot with debugging and verifying a fix once it's ready.
this is already fixed at head and 1.4.3 version of heapster The problem that is using kubernetes version from head and we have there 1.5.0-beta.0: I see two possible solutions: version of heapster to 1.4.3 at head of kubernetes. Piotr mentioned that 1.5.0 has still a lot of problems, so it maybe makes sense. a new release of heapster (1.5.0-beta.1) and after that bump version at head.
And thanks for taking this issue.
Not directly related to this issue: a more general discussion is at We'd like to understand which fields are required and optional in heapster, so that we won't crash heapster with partial result.
We've merged a work around for now Hopefully it works. But anyway, we should fix this in 1.9.
what needs to be done for 1.9 here?
we'll need an assignee as well
We just need to update heapster version.

- --memory={{base_eventer_memory}} - --extra-memory={{eventer_memory_per_node}}Ki - --threshold=5 <del> - --deployment=heapster-v1.5.0-beta.0 </del> <ins> - --deployment=heapster-v1.5.0-beta.1 </ins> - --container=eventer - --poll-period=300000 - --estimator=exponential


Question:
I also hit when running cri-containerd as the container runtime. The reason is that we don't populate stats like in cri-containerd integration now. is adding the containerd support in cadvisor, after that we should have full summary metrics. However, heapster should not panic in this case anyway. Can we cut a heapster patch release and update heapster in the main repo?
+1. Many new CRI integration would run into this issue.
In fact, as far as I am aware, not a single CRI implementation currently implements any of the stats calls.
I beg to differ: That's the reason why filed the issue.
Yeah does implement the CRI stats. Currently it only support core metrics, which is only part of summary api. We are working on integration with cadvisor, before that some fields in summary api such as will be nil. Current heapster version will crash when is nil, which shouldn't happen. :)
/cc
My bad, thanks for the pointer :slightlysmilingface:. I'll correct my statement: I believe no CRI implementation except for cri-containerd implements the stats calls. The ones I explicitly checked are: dockershim, cri-o, rktlet.
Something I'm wondering about though: The intention of the stats calls is in order to be able to get rid of cAdvisor. Making the stats calls depend on cAdvisor seems to cover up the issues it has. Re:
Here is the reason that we still have to integrate with cadvisor for now.
/cc
If I understand correctly this is meant as a migration mean, so that both are possible for some time, and then we drop cAdvisor once a reasonable migration period has been granted.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I will take a look at Heapster and try to fix this. Can you advise how to reproduce the issue? This will help a lot with debugging and verifying a fix once it's ready.
this is already fixed at head and 1.4.3 version of heapster The problem that is using kubernetes version from head and we have there 1.5.0-beta.0: I see two possible solutions: version of heapster to 1.4.3 at head of kubernetes. Piotr mentioned that 1.5.0 has still a lot of problems, so it maybe makes sense. a new release of heapster (1.5.0-beta.1) and after that bump version at head.
And thanks for taking this issue.
Not directly related to this issue: a more general discussion is at We'd like to understand which fields are required and optional in heapster, so that we won't crash heapster with partial result.
We've merged a work around for now Hopefully it works. But anyway, we should fix this in 1.9.
what needs to be done for 1.9 here?
we'll need an assignee as well
We just need to update heapster version.

- --source=kubernetes.summary_api:'' - --sink=influxdb:http://monitoring-influxdb:8086 - --sink=gcm:?metrics=autoscaling <del> - image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.0 </del> <ins> - image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.1 </ins> name: eventer command: - /eventer


Question:
I also hit when running cri-containerd as the container runtime. The reason is that we don't populate stats like in cri-containerd integration now. is adding the containerd support in cadvisor, after that we should have full summary metrics. However, heapster should not panic in this case anyway. Can we cut a heapster patch release and update heapster in the main repo?
+1. Many new CRI integration would run into this issue.
In fact, as far as I am aware, not a single CRI implementation currently implements any of the stats calls.
I beg to differ: That's the reason why filed the issue.
Yeah does implement the CRI stats. Currently it only support core metrics, which is only part of summary api. We are working on integration with cadvisor, before that some fields in summary api such as will be nil. Current heapster version will crash when is nil, which shouldn't happen. :)
/cc
My bad, thanks for the pointer :slightlysmilingface:. I'll correct my statement: I believe no CRI implementation except for cri-containerd implements the stats calls. The ones I explicitly checked are: dockershim, cri-o, rktlet.
Something I'm wondering about though: The intention of the stats calls is in order to be able to get rid of cAdvisor. Making the stats calls depend on cAdvisor seems to cover up the issues it has. Re:
Here is the reason that we still have to integrate with cadvisor for now.
/cc
If I understand correctly this is meant as a migration mean, so that both are possible for some time, and then we drop cAdvisor once a reasonable migration period has been granted.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I will take a look at Heapster and try to fix this. Can you advise how to reproduce the issue? This will help a lot with debugging and verifying a fix once it's ready.
this is already fixed at head and 1.4.3 version of heapster The problem that is using kubernetes version from head and we have there 1.5.0-beta.0: I see two possible solutions: version of heapster to 1.4.3 at head of kubernetes. Piotr mentioned that 1.5.0 has still a lot of problems, so it maybe makes sense. a new release of heapster (1.5.0-beta.1) and after that bump version at head.
And thanks for taking this issue.
Not directly related to this issue: a more general discussion is at We'd like to understand which fields are required and optional in heapster, so that we won't crash heapster with partial result.
We've merged a work around for now Hopefully it works. But anyway, we should fix this in 1.9.
what needs to be done for 1.9 here?
we'll need an assignee as well
We just need to update heapster version.

- /heapster - --source=kubernetes.summary_api:'' - --sink=influxdb:http://monitoring-influxdb:8086 <del> - image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.0 </del> <ins> - image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.1 </ins> name: eventer command: - /eventer


Question:
I also hit when running cri-containerd as the container runtime. The reason is that we don't populate stats like in cri-containerd integration now. is adding the containerd support in cadvisor, after that we should have full summary metrics. However, heapster should not panic in this case anyway. Can we cut a heapster patch release and update heapster in the main repo?
+1. Many new CRI integration would run into this issue.
In fact, as far as I am aware, not a single CRI implementation currently implements any of the stats calls.
I beg to differ: That's the reason why filed the issue.
Yeah does implement the CRI stats. Currently it only support core metrics, which is only part of summary api. We are working on integration with cadvisor, before that some fields in summary api such as will be nil. Current heapster version will crash when is nil, which shouldn't happen. :)
/cc
My bad, thanks for the pointer :slightlysmilingface:. I'll correct my statement: I believe no CRI implementation except for cri-containerd implements the stats calls. The ones I explicitly checked are: dockershim, cri-o, rktlet.
Something I'm wondering about though: The intention of the stats calls is in order to be able to get rid of cAdvisor. Making the stats calls depend on cAdvisor seems to cover up the issues it has. Re:
Here is the reason that we still have to integrate with cadvisor for now.
/cc
If I understand correctly this is meant as a migration mean, so that both are possible for some time, and then we drop cAdvisor once a reasonable migration period has been granted.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I will take a look at Heapster and try to fix this. Can you advise how to reproduce the issue? This will help a lot with debugging and verifying a fix once it's ready.
this is already fixed at head and 1.4.3 version of heapster The problem that is using kubernetes version from head and we have there 1.5.0-beta.0: I see two possible solutions: version of heapster to 1.4.3 at head of kubernetes. Piotr mentioned that 1.5.0 has still a lot of problems, so it maybe makes sense. a new release of heapster (1.5.0-beta.1) and after that bump version at head.
And thanks for taking this issue.
Not directly related to this issue: a more general discussion is at We'd like to understand which fields are required and optional in heapster, so that we won't crash heapster with partial result.
We've merged a work around for now Hopefully it works. But anyway, we should fix this in 1.9.
what needs to be done for 1.9 here?
we'll need an assignee as well
We just need to update heapster version.

- --memory={{ base_eventer_memory }} - --extra-memory={{ eventer_memory_per_node }}Ki - --threshold=5 <del> - --deployment=heapster-v1.5.0-beta.0 </del> <ins> - --deployment=heapster-v1.5.0-beta.1 </ins> - --container=eventer - --poll-period=300000 - --estimator=exponential


Question:
I also hit when running cri-containerd as the container runtime. The reason is that we don't populate stats like in cri-containerd integration now. is adding the containerd support in cadvisor, after that we should have full summary metrics. However, heapster should not panic in this case anyway. Can we cut a heapster patch release and update heapster in the main repo?
+1. Many new CRI integration would run into this issue.
In fact, as far as I am aware, not a single CRI implementation currently implements any of the stats calls.
I beg to differ: That's the reason why filed the issue.
Yeah does implement the CRI stats. Currently it only support core metrics, which is only part of summary api. We are working on integration with cadvisor, before that some fields in summary api such as will be nil. Current heapster version will crash when is nil, which shouldn't happen. :)
/cc
My bad, thanks for the pointer :slightlysmilingface:. I'll correct my statement: I believe no CRI implementation except for cri-containerd implements the stats calls. The ones I explicitly checked are: dockershim, cri-o, rktlet.
Something I'm wondering about though: The intention of the stats calls is in order to be able to get rid of cAdvisor. Making the stats calls depend on cAdvisor seems to cover up the issues it has. Re:
Here is the reason that we still have to integrate with cadvisor for now.
/cc
If I understand correctly this is meant as a migration mean, so that both are possible for some time, and then we drop cAdvisor once a reasonable migration period has been granted.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I will take a look at Heapster and try to fix this. Can you advise how to reproduce the issue? This will help a lot with debugging and verifying a fix once it's ready.
this is already fixed at head and 1.4.3 version of heapster The problem that is using kubernetes version from head and we have there 1.5.0-beta.0: I see two possible solutions: version of heapster to 1.4.3 at head of kubernetes. Piotr mentioned that 1.5.0 has still a lot of problems, so it maybe makes sense. a new release of heapster (1.5.0-beta.1) and after that bump version at head.
And thanks for taking this issue.
Not directly related to this issue: a more general discussion is at We'd like to understand which fields are required and optional in heapster, so that we won't crash heapster with partial result.
We've merged a work around for now Hopefully it works. But anyway, we should fix this in 1.9.
what needs to be done for 1.9 here?
we'll need an assignee as well
We just need to update heapster version.

- --memory={{ base_metrics_memory }} - --extra-memory={{metrics_memory_per_node}}Mi - --threshold=5 <del> - --deployment=heapster-v1.5.0-beta.0 </del> <ins> - --deployment=heapster-v1.5.0-beta.1 </ins> - --container=heapster - --poll-period=300000 - --estimator=exponential


Question:
I also hit when running cri-containerd as the container runtime. The reason is that we don't populate stats like in cri-containerd integration now. is adding the containerd support in cadvisor, after that we should have full summary metrics. However, heapster should not panic in this case anyway. Can we cut a heapster patch release and update heapster in the main repo?
+1. Many new CRI integration would run into this issue.
In fact, as far as I am aware, not a single CRI implementation currently implements any of the stats calls.
I beg to differ: That's the reason why filed the issue.
Yeah does implement the CRI stats. Currently it only support core metrics, which is only part of summary api. We are working on integration with cadvisor, before that some fields in summary api such as will be nil. Current heapster version will crash when is nil, which shouldn't happen. :)
/cc
My bad, thanks for the pointer :slightlysmilingface:. I'll correct my statement: I believe no CRI implementation except for cri-containerd implements the stats calls. The ones I explicitly checked are: dockershim, cri-o, rktlet.
Something I'm wondering about though: The intention of the stats calls is in order to be able to get rid of cAdvisor. Making the stats calls depend on cAdvisor seems to cover up the issues it has. Re:
Here is the reason that we still have to integrate with cadvisor for now.
/cc
If I understand correctly this is meant as a migration mean, so that both are possible for some time, and then we drop cAdvisor once a reasonable migration period has been granted.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I will take a look at Heapster and try to fix this. Can you advise how to reproduce the issue? This will help a lot with debugging and verifying a fix once it's ready.
this is already fixed at head and 1.4.3 version of heapster The problem that is using kubernetes version from head and we have there 1.5.0-beta.0: I see two possible solutions: version of heapster to 1.4.3 at head of kubernetes. Piotr mentioned that 1.5.0 has still a lot of problems, so it maybe makes sense. a new release of heapster (1.5.0-beta.1) and after that bump version at head.
And thanks for taking this issue.
Not directly related to this issue: a more general discussion is at We'd like to understand which fields are required and optional in heapster, so that we won't crash heapster with partial result.
We've merged a work around for now Hopefully it works. But anyway, we should fix this in 1.9.
what needs to be done for 1.9 here?
we'll need an assignee as well
We just need to update heapster version.

apiVersion: extensions/v1beta1 kind: Deployment metadata: <del> name: heapster-v1.5.0-beta.0 </del> <ins> name: heapster-v1.5.0-beta.1 </ins> namespace: kube-system labels: k8s-app: heapster kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile <del> version: v1.5.0-beta.0 </del> <ins> version: v1.5.0-beta.1 </ins> spec: replicas: 1 selector: matchLabels: k8s-app: heapster <del> version: v1.5.0-beta.0 </del> <ins> version: v1.5.0-beta.1 </ins> template: metadata: labels: k8s-app: heapster <del> version: v1.5.0-beta.0 </del> <ins> version: v1.5.0-beta.1 </ins> annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: containers: <del> - image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.0 </del> <ins> - image: gcr.io/google_containers/heapster-amd64:v1.5.0-beta.1 </ins> name: heapster livenessProbe: httpGet: