Update README.md

README.md

@@ -5,38 +5,53 @@ language:
 ---
 # German FinBERT (Further Pre-trained Version)
 
-This model card details the further pre-trained version of German FinBERT, a language model focusing on the financial domain within the German language.
+German FinBERT is a BERT language model focusing on the financial domain within the German language. In my [paper](https://arxiv.org/pdf/2010.10906.pdf) (UPDATE!!), I describe in more detail the steps taken to train the model and show that it outperforms its generic benchmarks for finance specific downstream tasks. 
+This version of German FinBERT starts with the [gbert-base](https://huggingface.co/deepset/gbert-base) model and continues pre-training on finance specific textual data.
 
 ## Overview  
-**Author:
-** Moritz Scherrmann  
-**Framework:
-** BERT-base 
-**Language:
-** German  
-**Specialization:
-** Financial textual data  
-**Original Model:
-** gbert of deepset  
-
-## Pre-training Corpus
-The pre-training corpus consists of German financial textual data. It comprises a comprehensive collection that includes financial reports, ad-hoc announcements, and news related to German companies. The corpus size is on par with those used for standard BERT models, indicating substantial coverage and depth.
+**Author:** [here](https://arxiv.org/pdf/2010.10906.pdf)  (UPDATE!)
+**Archticture:** BERT base 
+**Language:** German  
+**Specialization:** Financial textual data  
+**Original Model:** [gbert-base (deepset)](https://huggingface.co/deepset/gbert-base)
+**Framework:** [MosaicML](https://github.com/mosaicml/examples/tree/main/examples/benchmarks/bert)
+
+## Pre-training
+German FinBERT's pre-training corpus includes a diverse range of financial documents, such as Bundesanzeiger reports, Handelsblatt articles, MarketScreener data, and additional sources including FAZ, ad-hoc announcements, LexisNexis & Event Registry content, Zeit Online articles, Wikipedia entries, and Gabler Wirtschaftslexikon. In total, the corpus spans from 1996 to 2023, consisting of 12.15 million documents with 10.12 billion tokens over 53.19 GB.
+
+I further pre-train the model for 10,400 steps with a batch size of 4096, which is one epoch. I use an Adam optimizer with decoupled weight decay regularization, with Adam parameters 0.9, 0.98, 1e − 6,a weight
+decay of 1e − 5 and a maximal learning of 1e − 4. . I train the model using a Nvidia DGX A100 node consisting of 8 A100 GPUs with 80 GB of memory each.
+
 ## Performance  
 ### Fine-tune Datasets
-German FinBERT has been evaluated on three finance-specific tasks against generic German language models, showing improved performance in:
-- Sentiment prediction
-- Topic recognition
-- Question answering
-The model effectively captures domain-specific nuances, outperforming standard models on finance-related texts.
+To fine-tune the model, I use several datasets, including:
+- A manually labeled [multi-label database of German ad-hoc announcements](https://arxiv.org/pdf/2010.10906.pdf) (UPDATE!!!) containing 31,771 sentences, each associated with up to 20 possible topics.
+- An extractive question-answering dataset based on the SQuAD format, which was created using 3,044 ad-hoc announcements processed by OpenAI's ChatGPT to generate and answer questions.
+- The [financial phrase bank](https://arxiv.org/abs/1307.5336) of Malo et al. (2013) for sentiment classification, translated to German using [DeepL](https://www.deepl.com/translator)
 
 ### Benchmark Results
-- *The precise benchmark results and comparisons are not provided in the accessed part of the document.*
+The further pre-trained German FinBERT model demonstrated the following performances on finance-specific downstream tasks:
+
+Ad-Hoc Multi-Label Database:
+- Macro F1: 86.08%
+- Micro F1: 85.65%
+
+Ad-Hoc QuAD (Question Answering):
+- Exact Match (EM): 52.50%
+- F1 Score: 74.61%
+
+Translated Financial Phrase Bank:
+- Accuracy: 95.41%
+- Macro F1: 91.49%
   
 ## Authors  
-**Moritz Scherrmann
-**  Institute for Finance & Banking 
-Ludwig-Maximilians-Universität München  
-Ludwigstr. 28, RB 80539 Munich, Germany  
-Email: scherrmann@lmu.de
+Moritz Scherrmann: `scherrmann [at] lmu.de`
+
+
+For additional details regarding the performance on fine-tune datasets and benchmark results, please refer to the full documentation provided in the study.
 
-For additional details regarding the performance on fine-tune datasets and benchmark results, please refer to the full documentation provided in the study. German FinBERT represents an innovative development in the field of financial NLP, offering enhanced capabilities for analyzing German financial texts.
+See also:  
+scherrmann/GermanFinBERT_SC
+scherrmann/GermanFinBERT_FP_Topic
+scherrmann/GermanFinBERT_FP_QuAD
+scherrmann/GermanFinBERT_SC_Sentiment