File size: 679 Bytes
f29ace4 c55e87a 4af54be c55e87a a389e24 a3a130e 4af54be |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
---
license: llama3.2
datasets:
- UMCU/DutchMedicalText
language:
- nl
base_model:
- meta-llama/Llama-3.2-1B-Instruct
tags:
- medical
- cardiology
---
Llama-3.2-1B-Instruct, with domain adapted pretraining (DAPT), also called Continuous Pre-training (CPT) on a Dutch medical corpus, slightly biased towards cardiology.
Training for one full epoch, with a 256 batch size, maximally 768 sequence length and a linear-cosine schedule (details follow..).
This model will be further pre-trained on 5 million cardiology records from the UMCU.
The perplexity was around 5 on the validation set.
Note: this is not instruction tuned, and does not generate an EOS token. Update coming. |