Update README.md

README.md

@@ -1,3 +1,83 @@
 ---
 license: apache-2.0
+language: 
+  - ind
+datasets:
+  - uonlp/CulturaX
+tags:
+  - t5
 ---
+
+## IndoNanoT5 Base
+
+IndoNanoT5 Base is an Indonesian sequence-to-sequence language model based on the [T5](https://arxiv.org/abs/1910.10683) architecture. We conducted pre-training on an open-source Indonesian corpus of [uonlp/CulturaX](https://huggingface.co/datasets/uonlp/CulturaX). On a held-out subset of the corpus, our model achieved an evaluation loss of 2.082 or a perplexity of about 8.02.
+
+This model was trained using the [nanoT5](https://github.com/PiotrNawrot/nanoT5) PyTorch framework. All training was done on an Nvidia H100 GPU. [LazarusNLP/IndoNanoT5-base](https://huggingface.co/LazarusNLP/IndoNanoT5-base) is released under Apache 2.0 license.
+
+## Model Detail
+
+- **Developed by**: [LazarusNLP](https://lazarusnlp.github.io/)
+- **Model type**: Encoder-decoder T5 transformer language model
+- **Language(s)**: Indonesian
+- **License**: [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0.html)
+- **Contact**: [Wilson Wongso](https://wilsonwongso.dev/)
+
+## Use in 🤗Transformers
+
+```python
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+
+model_checkpoint = "LazarusNLP/IndoNanoT5-base"
+
+tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
+model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)
+```
+
+## Training Datasets
+
+Around 4B tokens from the following corpora were used during pre-training.
+
+- [Cleaned, Enormous, and Public: The Multilingual Fuel to Democratize Large Language Models for 167 Languages](https://huggingface.co/datasets/uonlp/CulturaX)
+
+## Training Hyperparameters
+
+The following hyperparameters were used during training:
+
+- `total_steps`: 65536
+- `input_length`: 512
+- `batch_size`: 128
+- `grad_acc`: 1
+- `base_lr`: 5e-3
+- `optimizer`: AdamWScaled with `betas=(0.9,0.999)` and `epsilon=1e-08`
+- `weight_decay`: 0.0
+- `lr_scheduler`: cosine
+- `warmup_steps`: 10000
+- `final_cosine`: 1e-5
+- `grad_clip`: 1.0
+- `precision`: `bf16`
+
+## Acknowledgements
+
+We would like to acknowledge [nanoT5](https://github.com/PiotrNawrot/nanoT5) for inspiring this project.
+
+## Credits
+
+BhinnekaLM is developed with love by:
+
+<div style="display: flex;">
+<a href="https://github.com/anantoj">
+    <img src="https://github.com/anantoj.png" alt="GitHub Profile" style="border-radius: 50%;width: 64px;margin:0 4px;">
+</a>
+
+<a href="https://github.com/DavidSamuell">
+    <img src="https://github.com/DavidSamuell.png" alt="GitHub Profile" style="border-radius: 50%;width: 64px;margin:0 4px;">
+</a>
+
+<a href="https://github.com/stevenlimcorn">
+    <img src="https://github.com/stevenlimcorn.png" alt="GitHub Profile" style="border-radius: 50%;width: 64px;margin:0 4px;">
+</a>
+
+<a href="https://github.com/w11wo">
+    <img src="https://github.com/w11wo.png" alt="GitHub Profile" style="border-radius: 50%;width: 64px;margin:0 4px;">
+</a>
+</div>