essential-text-bert-dapt

bert-base-uncased (revision 86b5e0934494bd15c9632b12f734a8a67f723594) further pretrained with masked language modelling on 42M tokens of 10-K text, keeping BERT's own vocabulary. It is the controlled domain-adaptive pretraining (DAPT) arm of the course project Evaluating Domain Adaptation in Financial Sentiment Analysis (Extensions §4): it separates the effect of financial text (vocabulary fixed) from the effect of FinBERT-MLM's own vocabulary and larger corpus.

This checkpoint is a masked-language model, not a sentiment classifier. In the project it is fine-tuned with the same recipe as the other encoders (scripts/run_extra_external.py --dapt-model <this checkpoint>).

Pretraining

Item Value
Corpus eloukas/edgar-corpus (revision 7e90f0f342569b35213445f809cfaf3b91f9964f), 10-K filings 2017-2020, sections 1 and 7
Documents / sentences kept 5,007 / 1,263,792
Tokens 41.7M (one pass, 164,103 blocks of 256 tokens)
Leakage filter 218 sentences dropped that matched Financial PhraseBank, SEntFiN, FiQA or Twitter Financial News evaluation sentences (exact or 8-gram overlap)
Corpus SHA-256 bc60b28396fba3cb001529174e5c24dd034984466ffee93a937f72f5eb0dbca0
Objective MLM, 15% masking
Steps / batch / sequence length 5,128 / 32 / 256
Optimiser AdamW, lr 5e-5, warmup 6%, weight decay 0.01, seed 42
Hardware Kaggle T4 (fp16), 44 min

model.safetensors SHA-256: 300399b902d1f80188c5bf683c5c534d51919f0cdf453d18bea911e3d7221b6c (the value recorded in every bert_dapt run of the project).

Result in the project

Fine-tuned on Financial PhraseBank, this model moves BERT by at most about 0.03 Macro-F1 with no consistent sign; see the project README, Extensions §4.

Licence

Apache-2.0, as the base model. The 10-K filings are public SEC documents.

Downloads last month
15
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for necatifcolak/essential-text-bert-dapt

Finetuned
(7,262)
this model

Dataset used to train necatifcolak/essential-text-bert-dapt