Commit
·
aba60a4
1
Parent(s):
ee45ad0
Update README.md
Browse files
README.md
CHANGED
|
@@ -30,7 +30,7 @@ co2_eq_emissions: 100
|
|
| 30 |
|
| 31 |
## Model Description
|
| 32 |
|
| 33 |
-
We present **QAmemBERT**, which is a [CamemBERT base](https://huggingface.co/camembert-base) fine-tuned for the Question-Answering task for the French language on four French Q&A datasets composed of contexts and questions with their answers inside the context (= SQuAD
|
| 34 |
All these datasets were concatenated into a single dataset that we called [frenchQA](https://huggingface.co/datasets/CATIE-AQ/frenchQA).
|
| 35 |
This represents a total of over **221,348 questions/answers pairs used to finetune this model and 6,376 to test it**.
|
| 36 |
|
|
@@ -39,14 +39,14 @@ This represents a total of over **221,348 questions/answers pairs used to finetu
|
|
| 39 |
|
| 40 |
| Dataset | Format | Train split | Dev split | Test split |
|
| 41 |
| ----------- | ----------- | ----------- | ----------- | ----------- |
|
| 42 |
-
| [piaf](https://www.data.gouv.fr/en/datasets/piaf-le-dataset-francophone-de-questions-reponses/)| SQuAD
|
| 43 |
-
| piaf_v2| SQuAD
|
| 44 |
-
| [fquad](https://fquad.illuin.tech/)| SQuAD
|
| 45 |
-
| fquad_v2 | SQuAD
|
| 46 |
-
| [lincoln/newsquadfr](https://huggingface.co/datasets/lincoln/newsquadfr) | SQuAD
|
| 47 |
-
| lincoln/newsquadfr_v2 | SQuAD
|
| 48 |
-
| [pragnakalp/squad_v2_french_translated](https://huggingface.co/datasets/pragnakalp/squad_v2_french_translated)| SQuAD
|
| 49 |
-
| pragnakalp/squad_v2_french_translated_v2| SQuAD
|
| 50 |
|
| 51 |
All these datasets were concatenated into a single dataset that we called [frenchQA](https://huggingface.co/datasets/CATIE-AQ/frenchQA).
|
| 52 |
|
|
@@ -58,44 +58,38 @@ The evaluation was carried out using the [**evaluate**](https://pypi.org/project
|
|
| 58 |
|
| 59 |
### FQuaD 1.0 (validation)
|
| 60 |
|
| 61 |
-
The metric used is
|
| 62 |
|
| 63 |
| Model | Exact_match | F1-score |
|
| 64 |
| ----------- | ----------- | ----------- |
|
| 65 |
| [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 53.60 | 78.09 |
|
| 66 |
| QAmembert (previous version) | 54.26 | 77.87 |
|
| 67 |
| QAmembert (**this version**) | 53.98 | 78.00 |
|
| 68 |
-
| QAmembert-large
|
| 69 |
-
| [fT0](https://huggingface.co/CATIE-AQ/frenchT0) | 41.15 | 65.79 |
|
| 70 |
|
| 71 |
-
♪ this model is available on request only
|
| 72 |
|
| 73 |
### qwant/squad_fr (validation)
|
| 74 |
|
| 75 |
-
The metric used is
|
| 76 |
|
| 77 |
| Model | Exact_match | F1-score |
|
| 78 |
| ----------- | ----------- | ----------- |
|
| 79 |
| [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 60.17 | 78.27 |
|
| 80 |
| QAmembert (previous version) | 60.40 | 77.27 |
|
| 81 |
| QAmembert (**this version**) | 60.95 | 77.30 |
|
| 82 |
-
| QAmembert-large
|
| 83 |
-
| [fT0](https://huggingface.co/CATIE-AQ/frenchT0) | 41.05 | 56.14 |
|
| 84 |
|
| 85 |
-
♪ this model is available on request only.
|
| 86 |
|
| 87 |
### frenchQA
|
| 88 |
|
| 89 |
-
This dataset includes question with no answers in the context. The metric used is
|
| 90 |
|
| 91 |
| Model | Exact_match | F1-score | Answer_f1 | NoAnswer_f1 |
|
| 92 |
| ----------- | ----------- | ----------- | ----------- | ----------- |
|
| 93 |
| [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | n/a | n/a | n/a | n/a |
|
| 94 |
| QAmembert (previous version) | 60.28 | 71.29 | 75.92 | 66.65
|
| 95 |
| QAmembert (**this version**) | **77.14** | 86.88 | 75.66 | 98.11
|
| 96 |
-
| QAmembert-large
|
| 97 |
-
|
| 98 |
-
♪ this model is available on request only.
|
| 99 |
|
| 100 |
|
| 101 |
|
|
|
|
| 30 |
|
| 31 |
## Model Description
|
| 32 |
|
| 33 |
+
We present **QAmemBERT**, which is a [CamemBERT base](https://huggingface.co/camembert-base) fine-tuned for the Question-Answering task for the French language on four French Q&A datasets composed of contexts and questions with their answers inside the context (= SQuAD 1.0 format) but also contexts and questions with their answers not inside the context (= SQuAD 2.0 format).
|
| 34 |
All these datasets were concatenated into a single dataset that we called [frenchQA](https://huggingface.co/datasets/CATIE-AQ/frenchQA).
|
| 35 |
This represents a total of over **221,348 questions/answers pairs used to finetune this model and 6,376 to test it**.
|
| 36 |
|
|
|
|
| 39 |
|
| 40 |
| Dataset | Format | Train split | Dev split | Test split |
|
| 41 |
| ----------- | ----------- | ----------- | ----------- | ----------- |
|
| 42 |
+
| [piaf](https://www.data.gouv.fr/en/datasets/piaf-le-dataset-francophone-de-questions-reponses/)| SQuAD 1.0 | 9 224 Q & A | X | X |
|
| 43 |
+
| piaf_v2| SQuAD 2.0 | 9 224 Q & A | X | X |
|
| 44 |
+
| [fquad](https://fquad.illuin.tech/)| SQuAD 1.0 | 20 731 Q & A | 3 188 Q & A (not used in training because it serves as a test dataset) | 2 189 Q & A (not used in our work because not freely available)|
|
| 45 |
+
| fquad_v2 | SQuAD 2.0 | 20 731 Q & A | 3 188 Q & A (not used in training because it serves as a test dataset) | X |
|
| 46 |
+
| [lincoln/newsquadfr](https://huggingface.co/datasets/lincoln/newsquadfr) | SQuAD 1.0 | 1 650 Q & A | 455 Q & A (not used in our work) | X |
|
| 47 |
+
| lincoln/newsquadfr_v2 | SQuAD 2.0 | 1 650 Q & A | 455 Q & A (not used in our work) | X |
|
| 48 |
+
| [pragnakalp/squad_v2_french_translated](https://huggingface.co/datasets/pragnakalp/squad_v2_french_translated)| SQuAD 2.0 | 79 069 Q & A | X | X |
|
| 49 |
+
| pragnakalp/squad_v2_french_translated_v2| SQuAD 2.0 | 79 069 Q & A | X | X |
|
| 50 |
|
| 51 |
All these datasets were concatenated into a single dataset that we called [frenchQA](https://huggingface.co/datasets/CATIE-AQ/frenchQA).
|
| 52 |
|
|
|
|
| 58 |
|
| 59 |
### FQuaD 1.0 (validation)
|
| 60 |
|
| 61 |
+
The metric used is SQuAD 1.0.
|
| 62 |
|
| 63 |
| Model | Exact_match | F1-score |
|
| 64 |
| ----------- | ----------- | ----------- |
|
| 65 |
| [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 53.60 | 78.09 |
|
| 66 |
| QAmembert (previous version) | 54.26 | 77.87 |
|
| 67 |
| QAmembert (**this version**) | 53.98 | 78.00 |
|
| 68 |
+
| [QAmembert-large](https://huggingface.co/CATIE-AQ/QAmembert-large) | **55.95** | **81.05** |
|
|
|
|
| 69 |
|
|
|
|
| 70 |
|
| 71 |
### qwant/squad_fr (validation)
|
| 72 |
|
| 73 |
+
The metric used is SQuAD 1.0.
|
| 74 |
|
| 75 |
| Model | Exact_match | F1-score |
|
| 76 |
| ----------- | ----------- | ----------- |
|
| 77 |
| [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | 60.17 | 78.27 |
|
| 78 |
| QAmembert (previous version) | 60.40 | 77.27 |
|
| 79 |
| QAmembert (**this version**) | 60.95 | 77.30 |
|
| 80 |
+
| [QAmembert-large](https://huggingface.co/CATIE-AQ/QAmembert-large) | **65.58** | **81.74** |
|
|
|
|
| 81 |
|
|
|
|
| 82 |
|
| 83 |
### frenchQA
|
| 84 |
|
| 85 |
+
This dataset includes question with no answers in the context. The metric used is SQuAD 2.0.
|
| 86 |
|
| 87 |
| Model | Exact_match | F1-score | Answer_f1 | NoAnswer_f1 |
|
| 88 |
| ----------- | ----------- | ----------- | ----------- | ----------- |
|
| 89 |
| [etalab-ia/camembert-base-squadFR-fquad-piaf](https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf) | n/a | n/a | n/a | n/a |
|
| 90 |
| QAmembert (previous version) | 60.28 | 71.29 | 75.92 | 66.65
|
| 91 |
| QAmembert (**this version**) | **77.14** | 86.88 | 75.66 | 98.11
|
| 92 |
+
| [QAmembert-large](https://huggingface.co/CATIE-AQ/QAmembert-large) | **77.14** | **88.74** | **78.83** | **98.65**
|
|
|
|
|
|
|
| 93 |
|
| 94 |
|
| 95 |
|