Update models files for readability-es-benchmark-bertin-es-paragraphs-3class

README.md

@@ -1,3 +1,71 @@
 ---
 license: cc-by-4.0
 ---
+
+## Readability benchmark (ES): bertin-es-paragraphs-2class
+
+This project is part of a series of models from the paper "A Benchmark for Neural Readability Assessment of Texts in Spanish". 
+You can find more details about the project in our [GitHub](https://github.com/lmvasque/readability-es-benchmark).
+
+## Models
+
+Our models were fine-tuned in multiple settings, including readability assessment in 2-class (simple/complex) and 3-class (basic/intermediate/advanced) for sentences and paragraph datasets.
+You can find more details in our [paper](https://drive.google.com/file/d/1KdwvqrjX8MWYRDGBKeHmiR1NCzDcVizo/view?usp=share_link). 
+These are the available models you can use (current model page in bold):
+
+| Model                                                                                                   | Granularity    | # classes |
+|---------------------------------------------------------------------------------------------------------|----------------|:---------:|
+| **[BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-paragraphs-2class)** | **paragraphs** |   **2**   |
+| [BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-paragraphs-3class)     | paragraphs     |     3     |
+| [BERTIN (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-2class)   | paragraphs     |     2     |
+| [BERTIN (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class)   | paragraphs     |     3     |
+| [mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-2class)       | paragraphs     |     2     |
+| [mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class)       | paragraphs     |     3     |
+| [mBERT (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class)    | paragraphs     |     3     |
+| [BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-paragraphs-2class)     | sentences      |     2     |
+| [BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-paragraphs-3class)     | sentences      |     3     |
+| [BERTIN (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-2class)   | sentences      |     2     |
+| [BERTIN (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class)   | sentences      |     3     |
+| [mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-2class)       | sentences      |     2     |
+| [mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class)       | sentences      |     3     |
+| [mBERT (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class)    | sentences      |     3     |
+
+For the zero-shot setting, we used the original models [BERTIN](bertin-project/bertin-roberta-base-spanish) and [mBERT](https://huggingface.co/bert-base-multilingual-uncased) with no further training.
+## Results
+
+These are our results for all the readability models in different settings. Please select your model based on the desired performance:
+
+| Granularity | Model         | F1 Score (2-class)  |  Precision (2-class)  |  Recall (2-class)  |  F1 Score (3-class)  |  Precision (3-class)  |  Recall (3-class)  |
+|-------------|---------------|:-------------------:|:---------------------:|:------------------:|:--------------------:|:---------------------:|:------------------:|
+| Paragraph   | Baseline (TF-IDF+LR) |        0.829        |         0.832         |       0.827        |        0.556         |         0.563         |       0.550        |
+| Paragraph   | BERTIN (Zero) |        0.308        |         0.222         |       0.500        |        0.227         |         0.284         |       0.338        |
+| Paragraph   | BERTIN (ES)   |        0.924        |         0.923         |       0.925        |        0.772         |         0.776         |       0.768        |
+| Paragraph   | mBERT (Zero)  |        0.308        |         0.222         |       0.500        |        0.253         |         0.312         |       0.368        |
+| Paragraph   | mBERT (EN)    |          -          |           -           |         -          |        0.505         |         0.560         |       0.552        |
+| Paragraph   | mBERT (ES)    |      **0.933**      |       **0.932**       |     **0.936**      |        0.776         |         0.777         |       0.778        |
+| Paragraph   | mBERT (EN+ES) |          -          |           -           |         -          |      **0.779**       |       **0.783**       |     **0.779**      |
+| Sentence | Baseline (TF-IDF+LR) |        0.811        |         0.814         |       0.808        |        0.525         |         0.531         |       0.521        |
+| Sentence | BERTIN (Zero) |        0.367        |         0.290         |       0.500        |        0.188         |         0.232         |       0.335        |
+| Sentence | BERTIN (ES)   |      **0.900**      |       **0.900**       |     **0.900**      |      **0.699**       |       **0.701**       |     **0.698**      |
+| Sentence | mBERT (Zero)  |        0.367        |         0.290         |       0.500        |        0.278         |         0.329         |       0.351        |
+| Sentence | mBERT (EN)    |          -          |           -           |         -          |        0.521         |         0.565         |       0.539        |
+| Sentence | mBERT (ES)    |        0.893        |         0.891         |       0.896        |        0.688         |         0.686         |       0.691        |
+| Sentence | mBERT (EN+ES) |          -          |           -           |         -          |        0.679         |         0.676         |       0.682        |
+
+
+## Citation
+
+If you use our results and scripts in your research, please cite our work: "[A Benchmark for Neural Readability Assessment of Texts in Spanish](https://drive.google.com/file/d/1KdwvqrjX8MWYRDGBKeHmiR1NCzDcVizo/view?usp=share_link)" (to be published) 
+
+```
+@inproceedings{vasquez-rodriguez-etal-2022-benchmarking,
+    title = "The Role of Text Simplification Operations in Evaluation",
+    author = "V{\'a}squez-Rodr{\'\i}guez, Laura  and
+      Cuenca-Jim{\'\e}nez, Pedro-Manuel and
+      Morales-Esquivel, Sergio Esteban and
+      Alva-Manchego, Fernando",
+    booktitle = "Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), EMNLP 2022",
+    month = dec,
+    year = "2022",
+}
+```

config.json

@@ -0,0 +1,39 @@
+{
+  "_name_or_path": "bertin-project/bertin-roberta-base-spanish",
+  "architectures": [
+    "RobertaForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.0,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.0,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1",
+    "2": "LABEL_2"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1,
+    "LABEL_2": 2
+  },
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "problem_type": "single_label_classification",
+  "torch_dtype": "float32",
+  "transformers_version": "4.19.2",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 50262
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d786c747e1fbbc7e5e196d0747af823897bad349b40ebf103980401da1956cb
+size 498654189

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "sep_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "cls_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true}}

tokenizer_config.json

@@ -0,0 +1 @@
+{"errors": "replace", "bos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "sep_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "cls_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "add_prefix_space": false, "trim_offsets": true, "max_len": 512, "special_tokens_map_file": null, "name_or_path": "bertin-project/bertin-roberta-base-spanish", "tokenizer_class": "RobertaTokenizer"}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4cf3b269ef79b5657206db2e950d0834050f332d33c91301ccf5f8c9fa55bf6c
+size 3311