| --- |
| license: llama3.2 |
| datasets: |
| - UMCU/DutchMedicalText |
| language: |
| - nl |
| base_model: |
| - meta-llama/Llama-3.2-1B-Instruct |
| tags: |
| - medical |
| - cardiology |
| --- |
| |
| Llama-3.2-1B-Instruct, with domain adapted pretraining (DAPT), also called Continuous Pre-training (CPT) on a Dutch medical corpus, slightly biased towards cardiology. |
|
|
| Training for one full epoch, with a 256 batch size, maximally 768 sequence length and a linear-cosine schedule (details follow..). |
|
|
| This model will be further pre-trained on 5 million cardiology records from the UMCU. |
|
|
| The perplexity was around 5 on the validation set. |
|
|
| Note: this is not instruction tuned, and does not generate an EOS token. Update coming. |
