Update README.md

README.md

@@ -10,6 +10,8 @@ library_name: transformers
 ---
 # **NanoTranslator-Experimental**
 
+## Models
+
 | Arch. | Act. |  V.  |  H.  |  I.  |  L.  | A. | K. | Tie |
 | :--: | :--: | :--: | :-----: | :---: | :------: | :--: | :--: | :--: |
 |  LLaMA  |  SwiGLU  | 2K | 256 | 768 |  2  |  8  |  4   | True |
@@ -24,13 +26,13 @@ library_name: transformers
 | StableLM | SwiGLU | 2K | 256 | 768 | 2 | 8 | 4 | True |
 | GPT2 | GeGLU | 2K | 256 | 1024 |  2  |  8  |  8   | True |
 | GPT-J | GeGLU | 2K | 256 | 1024 | 2 | 4 | 4 | True |
-| GPT-Neo | GeGLU | 2K | 256 | 1024 | 2 | 8 | 8 | True |
 |  GPT-NeoX  |  GeGLU  | 2K | 256 | 1024 |  2  |  8  |  8  | True |
 |  Bloom  | GeGLU | 2K | 256 | 1024 |  2  |  8  |  8   | True |
 | MPT | GeGLU | 2K | 256 | 1024 | 2 | 8 | 8 | True |
 | RWKV | - | 2K | 256 | 1024 | 2 | - | - | True |
 
 
+## Experimental Setup
 
 |                |           Value            |
 | :------------: | :------------------------: |
@@ -51,6 +53,9 @@ library_name: transformers
 |      Seed      |           3407           |
 
 
+## Results
+
+### Trapezoidal v.s. Cosine
 
 <table border="1" cellpadding="10" cellspacing="0" style="margin: 0 auto; border-collapse: collapse; text-align: center;">
   <thead>
@@ -205,4 +210,60 @@ library_name: transformers
       <td></td>
     </tr>
   </tbody>
-</table>
+</table>
+
+
+### BF16 & FP16
+
+<table border="1" cellpadding="10" cellspacing="0" style="margin: 0 auto; border-collapse: collapse; text-align: center;">
+  <thead>
+    <tr>
+      <th rowspan="2">Arch.</th>
+      <th colspan="2">Total Loss</th>
+      <th colspan="2">Final Loss (Last 10 steps Avg.)</th>
+    </tr>
+    <tr>
+      <th>FP16</th>
+      <th>BF16</th>
+      <th>FP16</th>
+      <th>BF16</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td>LLaMA</td>
+      <td>1.5734</td>
+      <td>1.5714</td>
+      <td>1.2784</td>
+      <td>1.2758</td>
+    </tr>
+    <tr>
+      <td>Qwen2</td>
+      <td>1.5735</td>
+      <td>1.5675</td>
+      <td>1.2760</td>
+      <td>1.2764</td>
+    </tr>
+    <tr>
+      <td>Mistral</td>
+      <td>1.5756</td>
+      <td>1.5694</td>
+      <td>1.2787</td>
+      <td>1.2740</td>
+    </tr>
+    <tr>
+      <td>OLMo</td>
+      <td>1.6011</td>
+      <td>1.6059</td>
+      <td>1.2857</td>
+      <td>1.2901</td>
+    </tr>
+    <tr>
+      <td>Cohere</td>
+      <td>2.1327</td>
+      <td>2.1112</td>
+      <td>1.6244</td>
+      <td>1.6346</td>
+    </tr>
+  </tbody>
+</table>