license: cc-by-4.0
language:
- en
ARC-Encoder models
This page houses three different versions of pretrained ARC-Encoders, architectures and methods to train them are described in PAPER. A code to reproduce the pretraining, further fine-tune the encoders or even evaluate them on dowstream tasks is available at ARC-Encoder repository.
Models Details
All the encoders released here are trained on web crawl filtered using Dactory based on a Llama3.2-3B base backbone. It consists in two ARC-Encoder specifically trained for one decoder and one for two decoders in the same time:
- ARC8-Encoder_Llama, trained on 6.5B tokens on Llama3.1-8B base specifically with a pooling factor of 8.
- ARC8-Encoder_Mistral, trained on 6.5B tokens on Mistral-7B base specifically with a pooling factor of 8.
- ARC8-Encoder_multi, trained by sampling among the two decoders above using 6.5B tokens for each one with a pooling factor of 8.
Uses
As described in PAPER, the pretrained ARC-Encoders can be fine-tuned to perform various downstream tasks. You can also adapt an ARC-Encoder to a new pooling factor (PF) by fine-tuning it on the desired PF. For optimal results, we recommend fine-tuning toward a lower PF than the one used during pretraining. To reproduce the results presented in the paper, you can use our released fine-tuning dataset, ARC_finetuning.
Licensing
ARC-Encoders are licensed under the CC-BY 4.0 license.
Terms of use: As the released models are pretrained from Llama3.2 3B backbone, ARC-Encoders are subject to the Llama Terms of Use found at Llama license.
Citations
If you use one of these models, please cite:
blabla