Update README.md
Browse files
README.md
CHANGED
|
@@ -9,4 +9,5 @@ I used the Mega source code (namely the `MegaEncoderLayer` class) and created wr
|
|
| 9 |
was pretrained for 5 epochs (11.3k gradient steps) on wikitext-103, which took roughly 5 hours on a single T4 (in Colab's free tier).
|
| 10 |
|
| 11 |
See [the Colab notebook](https://colab.research.google.com/drive/1qfUO6o5HRdxBblWlw058HVyvaEPhPpH8?usp=sharing)
|
| 12 |
-
for further training details
|
|
|
|
|
|
| 9 |
was pretrained for 5 epochs (11.3k gradient steps) on wikitext-103, which took roughly 5 hours on a single T4 (in Colab's free tier).
|
| 10 |
|
| 11 |
See [the Colab notebook](https://colab.research.google.com/drive/1qfUO6o5HRdxBblWlw058HVyvaEPhPpH8?usp=sharing)
|
| 12 |
+
for further training details. In order to load the pretrained weights for this model, you'll need to use the
|
| 13 |
+
[Mega repo](https://github.com/facebookresearch/mega) along with the example code at the end of the Colab notebook.
|