| | --- |
| | language: fr |
| | license: mit |
| | datasets: |
| | - oscar |
| | --- |
| | |
| | ## Model description |
| |
|
| | CamemBERT is a state-of-the-art language model for French based on the RoBERTa model. It is now available on Hugging Face in 6 different versions with varying number of parameters, amount of pretraining data and pretraining data source domains. |
| |
|
| | ## Evaluation |
| |
|
| | The model developers evaluated CamemBERT using four different downstream tasks for French: part-of-speech (POS) tagging, dependency parsing, named entity recognition (NER) and natural language inference (NLI). |
| |
|
| | ## Limitations and bias |
| |
|
| | Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). |
| |
|
| | This model was pretrinaed on a subcorpus of OSCAR multilingual corpus. Some of the limitations and risks associated with the OSCAR dataset, which are further detailed in the [OSCAR dataset card](https://huggingface.co/datasets/oscar), include the following: |
| |
|
| | > The quality of some OSCAR sub-corpora might be lower than expected, specifically for the lowest-resource languages. |
| |
|
| | > Constructed from Common Crawl, Personal and sensitive information might be present. |
| |
|
| | ## Training data |
| |
|
| | OSCAR or Open Super-large Crawled Aggregated coRpus is a multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the Ungoliant architecture. |
| |
|
| | ## How to use |
| |
|
| | -**Filling masks using pipeline** |
| |
|
| | ```python |
| | >>> from transformers import pipeline |
| | >>> camembert_fill_mask = pipeline("fill-mask", model="camembert-base") |
| | >>> results = camembert_fill_mask("Le camembert est <mask> :)") |
| | |
| | >>> result |
| | [{'score': 0.49091097712516785, |
| | 'token': 7200, |
| | 'token_str': 'délicieux', |
| | 'sequence': 'Le camembert est délicieux :)'}, |
| | {'score': 0.1055697426199913, |
| | 'token': 2183, |
| | 'token_str': 'excellent', |
| | 'sequence': 'Le camembert est excellent :)'}, |
| | {'score': 0.03453319892287254, |
| | 'token': 26202, |
| | 'token_str': 'succulent', |
| | 'sequence': 'Le camembert est succulent :)'}, |
| | {'score': 0.03303128108382225, |
| | 'token': 528, |
| | 'token_str': 'meilleur', |
| | 'sequence': 'Le camembert est meilleur :)'}, |
| | {'score': 0.030076386407017708, |
| | 'token': 1654, |
| | 'token_str': 'parfait', |
| | 'sequence': 'Le camembert est parfait :)'}] |
| | ``` |
| |
|
| | -**Extract contextual embedding features from Camembert output** |
| |
|
| | ```python |
| | import torch |
| | |
| | >>> tokenized_sentence = tokenizer.tokenize("J'aime le camembert !") |
| | >>> encoded_sentence = tokenizer.encode(tokenized_sentence) |
| | # Can be done in one step : tokenize.encode("J'aime le camembert !") |
| | |
| | >>> tokenized_sentence |
| | ['▁J', "'", 'aime', '▁le', '▁ca', 'member', 't', '▁!'] |
| | >>> encoded_sentence |
| | [5, 121, 11, 660, 16, 730, 25543, 110, 83, 6] |
| | |
| | encoded_sentence = torch.tensor(encoded_sentence).unsqueeze(0) |
| | embeddings, _ = camembert(encoded_sentence) |
| | # embeddings.detach() |
| | # embeddings.size torch.Size([1, 10, 768]) |
| | # tensor([[[-0.0254, 0.0235, 0.1027, ..., -0.1459, -0.0205, -0.0116], |
| | # [ 0.0606, -0.1811, -0.0418, ..., -0.1815, 0.0880, -0.0766], |
| | # [-0.1561, -0.1127, 0.2687, ..., -0.0648, 0.0249, 0.0446], |
| | # ..., |
| | ``` |
