Update description

app.py

@@ -64,7 +64,8 @@ def calculate(p, m, f, b, w, c, mem):
   return [baseline_time, baseline_bubble, baseline_acceleration, baseline_image, zb_time, zb_bubble, zb_acceleration, zb_image, zbv_time, zbv_bubble, zbv_acceleration, zbv_image]
 
 with gr.Blocks() as demo:
-  gr.Markdown("Zero bubble pipeline parallel bubble calculator")
+  gr.Markdown(open("description1.md").read())
+  gr.Markdown("# Pipeline Scheduler Playground")
   with gr.Row():
     with gr.Column(scale=1):
       with gr.Group():
@@ -123,4 +124,5 @@ with gr.Blocks() as demo:
       with gr.Column(scale=4):
         zbv_image=gr.Image(None, interactive=False, label="Schedule Image")
     button.click(calculate, inputs=[p, m, f, b, w, c, mem], outputs=[baseline_time, baseline_bubble, baseline_acceleration, baseline_image, zb_time, zb_bubble, zb_acceleration, zb_image, zbv_time, zbv_bubble, zbv_acceleration, zbv_image])
+  gr.Markdown(open("description2.md").read())
 demo.launch()

description1.md

@@ -0,0 +1,34 @@
+# Zero Bubble Pipeline Parallelism
+
+Zero Bubble Pipeline Parallelism is a novel pipeline parallelism algorithm able to reduce the bubble of pipeline parallelism to almost zero while preserving synchronous semantics.
+
+Our paper is coming soon.
+
+Try out our implementation based on Megatron on [https://github.com/sail-sg/zero-bubble-pipeline-parallelism](https://github.com/sail-sg/zero-bubble-pipeline-parallelism)
+
+Experiments shows zero bubble pipeline parallelism can accelerate training up to 30% with a similar memory comsumption. A detailed table of experiments is coming soon.
+
+## Zero Bubble Schedules
+The key of achieving zero bubble is to breaking a backward pass into a B pass and W pass. B on one stage will only depend on the B on its next stage, compared to depending on both B and W of in 1F1B.
+
+![image](https://hackmd.io/_uploads/Bkc7CL7N6.png)
+
+### Comparision of Schedules
+* 1F1B
+![image](https://hackmd.io/_uploads/Hkq-gD7N6.png)
+* ZB1P
+![image](https://hackmd.io/_uploads/Hy2GxwmEa.png)
+* ZB2P
+![image](https://hackmd.io/_uploads/S10QgvmV6.png)
+* ZBV - Each device is assigned to exactly 2 chunks (virtual stages), where white text colors represent the first chunk and black text colors represent the second chunk. The sequence of dependencies among model chunks follows a ”V” shape pattern for both the forward and backward passes.
+![image](https://hackmd.io/_uploads/Sk9uyY4ra.png)
+
+
+    
+
+| Comparison assuming T_F=T_B=T_W                                                      | 1F1B    | ZB1P     | ZB2P | ZBV (Recommended) |
+| ----------------------------------------------------- | ------- | -------- | ---- | --- |
+| Bubble Rate                                           | (p-1)/m | (p-1)/3m | 0    | 0   |
+| Activation Memory <br> (Compared to 1F1B)             | 1x       | 1x        | 2x    | 1x   |
+| Pipeline Communication Volume <br> (Compared to 1F1B) | 1x       | 1x        | 1x    | 2x   |
+

description2.md

@@ -0,0 +1,10 @@
+
+
+## Optimizer Post Validation
+
+In most practices of PP there's an all-reduce cross all pipeline stages for numerical robustness, e.g. global gradient norm for gradient clipping. INF/NAN check for mixed precision training, etc. This all-reduce breaks parallelogram and makes zero bubble impossible.
+Under the observation that during a stable training both the gradient clipping and INF/NAN rarely triggers, we replace the before-hand synchronizations with a post update validation.
+
+![image](https://hackmd.io/_uploads/B16R3q4N6.png)
+
+We eagerly step the optimizers assuming the grad cliping, INF/NAN conditions are not triggered. In case an amendment to the gradient is required, a rollback will be issued and then we redo the optimizer step based on the fully reduced global state.