Add model for readability-es-benchmark-mbert-es-paragraphs-3class

README.md

@@ -1,3 +1,71 @@
 ---
 license: cc-by-4.0
 ---
+
+## Readability benchmark (ES): mbert-es-paragraphs-3class
+
+This project is part of a series of models from the paper "A Benchmark for Neural Readability Assessment of Texts in Spanish". 
+You can find more details about the project in our [GitHub](https://github.com/lmvasque/readability-es-benchmark).
+
+## Models
+
+Our models were fine-tuned in multiple settings, including readability assessment in 2-class (simple/complex) and 3-class (basic/intermediate/advanced) for sentences and paragraph datasets.
+You can find more details in our [paper](https://drive.google.com/file/d/1KdwvqrjX8MWYRDGBKeHmiR1NCzDcVizo/view?usp=share_link). 
+These are the available models you can use (current model page in bold):
+
+| Model                                                                                                 | Granularity    | # classes |
+|-------------------------------------------------------------------------------------------------------|----------------|:---------:|
+| [BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-paragraphs-2class) | paragraphs     |     2     |
+| [BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-paragraphs-3class)   | paragraphs     |     3     |
+| [BERTIN (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-2class) | paragraphs     |     2     |
+| [BERTIN (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class) | paragraphs     |     3     |
+| [mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-2class)     | paragraphs     |     2     |
+| **[mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class)** | **paragraphs** |   **3**   |
+| [mBERT (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class)  | paragraphs     |     3     |
+| [BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-paragraphs-2class)   | sentences      |     2     |
+| [BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-paragraphs-3class)   | sentences      |     3     |
+| [BERTIN (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-2class) | sentences      |     2     |
+| [BERTIN (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class) | sentences      |     3     |
+| [mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-2class)     | sentences      |     2     |
+| [mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class)     | sentences      |     3     |
+| [mBERT (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class)  | sentences      |     3     |
+
+For the zero-shot setting, we used the original models [BERTIN](bertin-project/bertin-roberta-base-spanish) and [mBERT](https://huggingface.co/bert-base-multilingual-uncased) with no further training.
+## Results
+
+These are our results for all the readability models in different settings. Please select your model based on the desired performance:
+
+| Granularity | Model         | F1 Score (2-class)  |  Precision (2-class)  |  Recall (2-class)  |  F1 Score (3-class)  |  Precision (3-class)  |  Recall (3-class)  |
+|-------------|---------------|:-------------------:|:---------------------:|:------------------:|:--------------------:|:---------------------:|:------------------:|
+| Paragraph   | Baseline (TF-IDF+LR) |        0.829        |         0.832         |       0.827        |        0.556         |         0.563         |       0.550        |
+| Paragraph   | BERTIN (Zero) |        0.308        |         0.222         |       0.500        |        0.227         |         0.284         |       0.338        |
+| Paragraph   | BERTIN (ES)   |        0.924        |         0.923         |       0.925        |        0.772         |         0.776         |       0.768        |
+| Paragraph   | mBERT (Zero)  |        0.308        |         0.222         |       0.500        |        0.253         |         0.312         |       0.368        |
+| Paragraph   | mBERT (EN)    |          -          |           -           |         -          |        0.505         |         0.560         |       0.552        |
+| Paragraph   | mBERT (ES)    |      **0.933**      |       **0.932**       |     **0.936**      |        0.776         |         0.777         |       0.778        |
+| Paragraph   | mBERT (EN+ES) |          -          |           -           |         -          |      **0.779**       |       **0.783**       |     **0.779**      |
+| Sentence | Baseline (TF-IDF+LR) |        0.811        |         0.814         |       0.808        |        0.525         |         0.531         |       0.521        |
+| Sentence | BERTIN (Zero) |        0.367        |         0.290         |       0.500        |        0.188         |         0.232         |       0.335        |
+| Sentence | BERTIN (ES)   |      **0.900**      |       **0.900**       |     **0.900**      |      **0.699**       |       **0.701**       |     **0.698**      |
+| Sentence | mBERT (Zero)  |        0.367        |         0.290         |       0.500        |        0.278         |         0.329         |       0.351        |
+| Sentence | mBERT (EN)    |          -          |           -           |         -          |        0.521         |         0.565         |       0.539        |
+| Sentence | mBERT (ES)    |        0.893        |         0.891         |       0.896        |        0.688         |         0.686         |       0.691        |
+| Sentence | mBERT (EN+ES) |          -          |           -           |         -          |        0.679         |         0.676         |       0.682        |
+
+
+## Citation
+
+If you use our results and scripts in your research, please cite our work: "[A Benchmark for Neural Readability Assessment of Texts in Spanish](https://drive.google.com/file/d/1KdwvqrjX8MWYRDGBKeHmiR1NCzDcVizo/view?usp=share_link)" (to be published) 
+
+```
+@inproceedings{vasquez-rodriguez-etal-2022-benchmarking,
+    title = "The Role of Text Simplification Operations in Evaluation",
+    author = "V{\'a}squez-Rodr{\'\i}guez, Laura  and
+      Cuenca-Jim{\'\e}nez, Pedro-Manuel and
+      Morales-Esquivel, Sergio Esteban and
+      Alva-Manchego, Fernando",
+    booktitle = "Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), EMNLP 2022",
+    month = dec,
+    year = "2022",
+}
+```

config.json

@@ -0,0 +1,42 @@
+{
+  "_name_or_path": "bert-base-multilingual-uncased",
+  "architectures": [
+    "BertForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "classifier_dropout": null,
+  "directionality": "bidi",
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "id2label": {
+    "0": "LABEL_0",
+    "1": "LABEL_1",
+    "2": "LABEL_2"
+  },
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "label2id": {
+    "LABEL_0": 0,
+    "LABEL_1": 1,
+    "LABEL_2": 2
+  },
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "pooler_fc_size": 768,
+  "pooler_num_attention_heads": 12,
+  "pooler_num_fc_layers": 3,
+  "pooler_size_per_head": 128,
+  "pooler_type": "first_token_transform",
+  "position_embedding_type": "absolute",
+  "problem_type": "single_label_classification",
+  "torch_dtype": "float32",
+  "transformers_version": "4.19.2",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 105879
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc2adf4c11b65a1eab1578cef3177ce06bdff39310343c500627b89b7186f002
+size 669505901

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 512, "special_tokens_map_file": null, "name_or_path": "bert-base-multilingual-uncased", "tokenizer_class": "BertTokenizer"}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e27b4858fae398d7c30d6398722d191610cd8c3907df9d5516ff9a0504e418bb
+size 3247