ojhfklsjhl commited on
Commit
90342dd
·
verified ·
1 Parent(s): 192f883

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -4
README.md CHANGED
@@ -17,14 +17,13 @@ Another key feature of our model is that it also avoids lookback bias. In partic
17
 
18
  Our model is trained on 1 billion words (1-2 billion tokens) from Parliament Q&As, TV show conversations, music lyrics, patents, FOMC documents, public access books, newspapers, election campaign documents, and research papers. The model is based on the base-size DeBERTa model architecture and a custom ByteLevelBPETokenizer trained using the same training data.
19
 
20
- Our model achieves nearly state-of-the-art performance with less than 1% of training data and the smallest model size.
21
 
22
  | Model | Vocabulary (K) | Backbone #Params (M) | COLA | SST2 | QQP|MNLI|QNLI
23
  |------------------------|:--------------:|:--------------------:|:-----------------:|:---------------:|:--------------------:|:-----------------:|:---------------:|
24
- | ChronoBERT_1999 | 50 | 149 | 0.57|0.92|0.89|0.86|0.91|
25
  | FinBERT | 30 | 110 | 0.29|0.89|0.87|0.79|0.86|
26
- | StoriesLM | 30 | 110 | 0.47|0.90|0.87|0.80|0.87|
27
- | NolBERT | 30 | 109 | 0.43|0.91|0.91|0.82|0.89
28
 
29
  ## Usage Examples
30
 
 
17
 
18
  Our model is trained on 1 billion words (1-2 billion tokens) from Parliament Q&As, TV show conversations, music lyrics, patents, FOMC documents, public access books, newspapers, election campaign documents, and research papers. The model is based on the base-size DeBERTa model architecture and a custom ByteLevelBPETokenizer trained using the same training data.
19
 
20
+ Our model achieves state-of-the-art performance with less than 10% of training data.
21
 
22
  | Model | Vocabulary (K) | Backbone #Params (M) | COLA | SST2 | QQP|MNLI|QNLI
23
  |------------------------|:--------------:|:--------------------:|:-----------------:|:---------------:|:--------------------:|:-----------------:|:---------------:|
 
24
  | FinBERT | 30 | 110 | 0.29|0.89|0.87|0.79|0.86|
25
+ | StoriesLM | 30 | 110 | **0.47**|0.90|0.87|0.80|0.87|
26
+ | NolBERT | 30 | 109 | 0.43|**0.91**|**0.91**|**0.82**|**0.89**
27
 
28
  ## Usage Examples
29