Create README.md

README.md

@@ -0,0 +1,46 @@
+---
+language:
+- en
+---
+# Introduction
+This is the implementation of the BERT model using the LongNet structure (paper: https://arxiv.org/pdf/2307.02486.pdf). The model is pre-trained with 10-K/Q filings of US firms from 1994 to 2018.
+
+# Training code
+https://github.com/minhtriphan/LongFinBERT-base/tree/main
+
+# Training configuration
+* The model is trained with 4 epochs using the Masked Language Modeling (MLM) task;
+* The masking probability is 15%;
+* Details about the training configuration are given in the log file named `train_v1a_0803_1144_seed_1.log`;
+
+# Instruction to load the pre-trained model
+* Clone the git repo
+```
+git clone https://github.com/minhtriphan/LongFinBERT-base.git
+cd LongBERT
+```
+or
+```
+!git clone https://github.com/minhtriphan/LongFinBERT-base.git
+import sys
+sys.path.append('/LongFinBERT-base')
+```
+
+* Load the pre-trained tokenizer, model configuration, and model weights
+```
+from model import LongBERT
+from custom_config import LongBERTConfig
+from tokenizer import LongBERTTokenizer
+
+backbone = 'minhtriphan/LongFinBERT'
+
+tokenizer = LongBERTTokenizer.from_pretrained(backbone)
+config = LongBERTConfig.from_pretrained(backbone)
+model = LongBERT.from_pretrained(backbone)
+```
+
+# Contact
+For any comments, questions, feedback, please get in touch with us via phanminhtri2611@gmail.com or triminh.phan@unisg.ch.
+
+# Paper
+(updating)