Update README.md

README.md

@@ -1,10 +1,36 @@
 ---
-tags:
-- model_hub_mixin
-- pytorch_model_hub_mixin
+license: cc-by-4.0
+language:
+- en
 ---
 
-This model has been pushed to the Hub using the [PytorchModelHubMixin](https://huggingface.co/docs/huggingface_hub/package_reference/mixins#huggingface_hub.PyTorchModelHubMixin) integration:
-- Code: [More Information Needed]
-- Paper: [More Information Needed]
-- Docs: [More Information Needed]
+ # ARC-Encoder models
+
+ This page houses three different versions of pretrained ARC-Encoders, architectures and methods to train them are described in [PAPER](https://kyutai.org). A code to reproduce the pretraining, further fine-tune the encoders or even evaluate them on dowstream tasks is available at [ARC-Encoder repository](https://github.com/kyutai-labs/ARC-Encoder/tree/main).
+
+ ## Models Details 
+
+ All the encoders released here are trained on web crawl filtered using [Dactory](https://github.com/kyutai-labs/dactory) based on a [Llama3.2-3B](https://github.com/meta-llama/llama-cookbook) base backbone. It consists in two ARC-Encoder specifically trained for one decoder and one for two decoders in the same time:
+- ARC8-Encoder_Llama, trained on 6.5B tokens on [Llama3.1-8B](https://github.com/meta-llama/llama-cookbook) base specifically with a pooling factor of 8.    
+- ARC8-Encoder_Mistral, trained on 6.5B tokens on [Mistral-7B](https://github.com/mistralai/mistral-finetune?tab=readme-ov-file) base specifically with a pooling factor of 8.    
+- ARC8-Encoder_multi, trained by sampling among the two decoders above using 6.5B tokens for each one with a pooling factor of 8. 
+
+ ### Uses
+
+ As described in [PAPER](https://kyutai.org), the pretrained ARC-Encoders can be fine-tuned to perform various downstream tasks.
+You can also adapt an ARC-Encoder to a new pooling factor (PF) by fine-tuning it on the desired PF.
+For optimal results, we recommend fine-tuning toward a lower PF than the one used during pretraining.
+To reproduce the results presented in the paper, you can use our released fine-tuning dataset, [ARC_finetuning](https://huggingface.co/datasets/kyutai/ARC_finetuning).
+
+ ### Licensing
+
+ ARC-Encoders are licensed under the CC-BY 4.0 license.
+
+
+ ## Citations
+
+ If you use one of these models, please cite:
+
+ ```
+blabla
+```