essential-text-bert-dapt
bert-base-uncased (revision 86b5e0934494bd15c9632b12f734a8a67f723594) further pretrained with masked language
modelling on 42M tokens of 10-K text, keeping BERT's own vocabulary. It is the controlled domain-adaptive
pretraining (DAPT) arm of the course project
Evaluating Domain Adaptation in Financial Sentiment Analysis
(Extensions §4): it separates the effect of financial text (vocabulary fixed) from the effect of FinBERT-MLM's own
vocabulary and larger corpus.
This checkpoint is a masked-language model, not a sentiment classifier. In the project it is fine-tuned with the same
recipe as the other encoders (scripts/run_extra_external.py --dapt-model <this checkpoint>).
Pretraining
| Item | Value |
|---|---|
| Corpus | eloukas/edgar-corpus (revision 7e90f0f342569b35213445f809cfaf3b91f9964f), 10-K filings 2017-2020, sections 1 and 7 |
| Documents / sentences kept | 5,007 / 1,263,792 |
| Tokens | 41.7M (one pass, 164,103 blocks of 256 tokens) |
| Leakage filter | 218 sentences dropped that matched Financial PhraseBank, SEntFiN, FiQA or Twitter Financial News evaluation sentences (exact or 8-gram overlap) |
| Corpus SHA-256 | bc60b28396fba3cb001529174e5c24dd034984466ffee93a937f72f5eb0dbca0 |
| Objective | MLM, 15% masking |
| Steps / batch / sequence length | 5,128 / 32 / 256 |
| Optimiser | AdamW, lr 5e-5, warmup 6%, weight decay 0.01, seed 42 |
| Hardware | Kaggle T4 (fp16), 44 min |
model.safetensors SHA-256: 300399b902d1f80188c5bf683c5c534d51919f0cdf453d18bea911e3d7221b6c (the value recorded in
every bert_dapt run of the project).
Result in the project
Fine-tuned on Financial PhraseBank, this model moves BERT by at most about 0.03 Macro-F1 with no consistent sign; see the project README, Extensions §4.
Licence
Apache-2.0, as the base model. The 10-K filings are public SEC documents.
- Downloads last month
- 15
Model tree for necatifcolak/essential-text-bert-dapt
Base model
google-bert/bert-base-uncased