Update README.md
Browse files
README.md
CHANGED
|
@@ -2,4 +2,29 @@
|
|
| 2 |
license: mit
|
| 3 |
language:
|
| 4 |
- en
|
| 5 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
license: mit
|
| 3 |
language:
|
| 4 |
- en
|
| 5 |
+
---
|
| 6 |
+
|
| 7 |
+
Update README.md
|
| 8 |
+
## Exploring the Data Efficiency of Cross-Lingual Post-Training in Pretrained Language Models
|
| 9 |
+
Paper: https://arxiv.org/abs/2310.13312
|
| 10 |
+
|
| 11 |
+
Github: https://github.com/deep-over/FiLM
|
| 12 |
+
|
| 13 |
+
**FiLM**(**Fi**nancial **L**anguage **M**odel) Models 🌟
|
| 14 |
+
|
| 15 |
+
FiLM is a Pre-trained Language Model (PLM) optimized for the Financial domain, built upon a diverse range of Financial domain corpora. Initialized with the RoBERTa-base model, FiLM undergoes further training to achieve performance that surpasses RoBERTa-base in the economics sector for the first time.
|
| 16 |
+
|
| 17 |
+
To train FiLM, we have categorized our Financial Corpus into specific groups and gathered a diverse range of corpora to ensure optimal performance.
|
| 18 |
+
|
| 19 |
+
We offer two versions of the FiLM model, each tailored for specific use-cases in the Financial domain:
|
| 20 |
+
|
| 21 |
+
**FiLM (2.4B): Our Base Model**
|
| 22 |
+
|
| 23 |
+
This is our foundational model, trained on the entire range of corpora as outlined in the above Corpus table. Ideal for a wide array of financial applications. 📊
|
| 24 |
+
|
| 25 |
+
**FiLM (5.5B): Optimized for SEC Filings**
|
| 26 |
+
|
| 27 |
+
This model is specialized for handling SEC filings. We expanded the training set by adding 3.1 billion tokens from the SEC filings corpus dataset. The dataset is sourced from EDGAR-CORPUS: Billions of Tokens Make The World Go Round (Loukas et al., ECONLP 2021) and can be downloaded from Zenodo. 📑
|
| 28 |
+
|
| 29 |
+
**Types of Training Corpora 📚**
|
| 30 |
+
|