Update README.md

README.md

@@ -42,6 +42,25 @@ language:
 datasets:
 - oscar-corpus/colossal-oscar-1.0
 - HuggingFaceFW/fineweb-edu
+- joelniklaus/eurlex_resources
+- joelniklaus/legal-mc4
+- projecte-aina/CATalog
+- UFRGS/brwac
+- community-datasets/hrwac
+- danish-foundation-models/danish-gigaword
+- HiTZ/euscrawl
+- PleIAs/French-PD-Newspapers
+- PleIAs/French-PD-Books
+- AI-team-UoA/greek_legal_code
+- HiTZ/latxa-corpus-v1.1
+- allenai/peS2o
+- pile-of-law/pile-of-law
+- PORTULAN/parlamento-pt
+- hoskinson-center/proof-pile
+- togethercomputer/RedPajama-Data-1T
+- bigcode/starcoderdata
+- bjoernp/tagesschau-2018-2023
+- EleutherAI/the_pile_deduplicated
 ---
 
 ![](./images/logo_alia_2.png)
@@ -51,7 +70,7 @@ datasets:
 > 
 > The weights will be promptly updated as soon as the training process is complete. 
 
-# Salmandra ALIA-40b Model Card
+# ALIA-40b Model Card
 
 ALIA-40b is a highly multilingual model pre-trained from scratch that will come with its respective base and instruction-tuned variants. This model card corresponds to the 40B base version.
 
@@ -288,13 +307,13 @@ The pre-training corpus comprises data from 35 European languages and 92 program
 The initial 1.5 training epochs used 2.4 trillion tokens, obtained by manually adjusting data proportion to balance the representation 
 and give more importance to Spain’s co-official (Spanish, Catalan, Galician, and Basque). This way, we downsampled code and English data to half, 
 Spanish co-official languages were oversampled by 2x, and the remaining languages were kept in their original proportions.
-Following, during the following epochs (still training), the Colossal OSCAR dataset was replaced with the FineWebEdu dataset. 
+During the following epochs (still training), the Colossal OSCAR dataset was replaced with the FineWeb-Edu dataset. 
 This adjustment resulted in a total of 2.68 trillion tokens, distributed as outlined below:
 
 ![lang distrib](./images/corpus_languages.png)
 
 The pretraining corpus is predominantly composed of data from Colossal OSCAR, which contributes a significant 53,05% of the total tokens. 
-Following this, Starcoder provides 13,67%, and FineWebEdu (350B tokens subset) adds 10,24%. The next largest sources are HPLT at 4,21% and French-PD at 3,59%. 
+Following this, Starcoder provides 13,67%, and FineWeb-Edu (350B tokens subset) adds 10,24%. The next largest sources are HPLT at 4,21% and French-PD at 3,59%. 
 Other notable contributions include MaCoCu, Legal-ES, and EurLex, each contributing around 1.72% to 1.41%. 
 These major sources collectively form the bulk of the corpus, ensuring a rich and diverse dataset for training the language model. 
 The remaining 10% comes from smaller sources in various languages.