TRI-ML
/

mamba-7b-rw

Text Generation

Eval Results (legacy)

Model card Files Files and versions

sedrickkeh commited on Apr 22, 2024

Commit

bca90b5

·

verified ·

1 Parent(s): f945251

Update README.md

Files changed (1) hide show

README.md +0 -1

README.md CHANGED Viewed

@@ -100,7 +100,6 @@ We follow their training recipe and release our version of Mamba-7B.
 | Optimizer          | AdamW      |
 | Learning rate      | 3e-4       |
 | LR cooldown end    | 1e-5       |
-| QK-norm            | False      |
 | Warmup steps       | 2000       |
 | Z-loss             | 1e-4       |
 | Batch size         | 2M         |

 | Optimizer          | AdamW      |
 | Learning rate      | 3e-4       |
 | LR cooldown end    | 1e-5       |
 | Warmup steps       | 2000       |
 | Z-loss             | 1e-4       |
 | Batch size         | 2M         |