README.md

---
language:
- sv
base_model:
- KB/bert-base-swedish-cased
pipeline_tag: text-classification
---

<!--
TODO:
 - Add URL to the repo
   - Clean and comment thesis repo
   - Publish thesis repo
 - Edit _How to Get Started_
 - Edit _Training Details_
   - Edit _Training Data_
   - Edit _Training Procedure_
     - Edit _Preprocessing_
     - Edit _Training Hyperparameters_
 - Edit _Evaluation_
   - Edit _Testing Data, Factors & Metrics_
     - Edit _Testing Data_
     - Edit _Factors_
     - Edit _Metrics_
   - Edit _Results_
     - Edit _Summary_
 - Edit _Model Examination_
 - Edit _Environmental Impact_
-->


# Model Card for Model ID

This model was developed as part of the [Computational SLA](https://spraakbanken.gu.se/en/projects/computational-sla) working group at Språkbanken Text.
It takes essays written in Swedish by second language learners and assigns them one of the [CEFR levels](https://en.wikipedia.org/wiki/Common_European_Framework_of_Reference_for_Languages).
Of note is that it only uses the first five levels of the scale (A1 to C1), ignoring level C2 due to both lack of training data and it measuring things differently than the other levels do.

Most of the information contained in this Model Card comes from [the paper that introduced the present model](https://aclanthology.org/2024.nlp4call-1.11/).
Feel free to check it out for more in-depth information.

## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

- **Developed by:** [Språkbanken Text](https://spraakbanken.gu.se/en), as part of the [Computational SLA group](https://spraakbanken.gu.se/en/projects/computational-sla)
- **Shared by:** Ricardo Muñoz Sánchez ([rimusa](https://huggingface.co/rimusa))
- **Model type:** BERT for text classification
- **Language(s):** Swedish
- **License:** GPL-3.0
- **Finetuned from model:** [KB/bert-base-swedish-cased](https://huggingface.co/KB/bert-base-swedish-cased)

### Model Sources [optional]

<!-- Provide the basic links for the model. -->

<!-- - **Repository:** _coming soon_ <!-- Need to update and publish thesis repos -->
- **Paper:** Jingle BERT, Jingle BERT, Frozen All the Way: Freezing Layers to Identify CEFR Levels of Second Language Learners Using BERT ([link](https://aclanthology.org/2024.nlp4call-1.11/))

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

This model is meant for building demos and tools that display an approximate CEFR level of language learner texts.
It is important to note that the predictions from this model should be taken as illustrative rather than as autorithative.

### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

This model should not be deployed on high-stakes situations, such as actual language assessment or decision-making regarding migration, work, education, etc.
It has relatively low performance compared to what would be needed for such situations, not to speak of potential issues regarding accountability.

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

The model has been trained in a heterogeneous dataset of Swedish language learner essays.
While this exposed the model to a variety of contexts, it also means that there might be biases in terms of topics and format.

We are currently studying the impact that the essay authors' first language(s) has on these models.
This model card will be updated once we have more results on this regard.


## Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

> ```
@inproceedings{sanchez-etal-2024-jingle,
    title = "Jingle {BERT}, Jingle {BERT}, Frozen All the Way: Freezing Layers to Identify {CEFR} Levels of Second Language Learners Using {BERT}",
    author = "Mu{\~n}oz S{\'a}nchez, Ricardo  and
      Alfter, David  and
      Dobnik, Simon  and
      Szawerna, Maria Irena  and
      Volodina, Elena",
    editor = {Gaillat, Thomas  and
      Mallart, Cyriel  and
      Moreau, Fabienne  and
      Li, Jen-Yu  and
      Drouet, Griselda  and
      Alfter, David  and
      Volodina, Elena  and
      J{\"o}nsson, Arne},
    booktitle = "Proceedings of the 13th Workshop on Natural Language Processing for Computer Assisted Language Learning",
    month = oct,
    year = "2024",
    address = "Rennes, France",
    publisher = "LiU Electronic Press",
    url = "https://aclanthology.org/2024.nlp4call-1.11/",
    pages = "137--152"
}

**APA:**

> Ricardo Muñoz Sánchez, David Alfter, Simon Dobnik, Maria Irena Szawerna, and Elena Volodina. 2024. Jingle BERT, Jingle BERT, Frozen All the Way: Freezing Layers to Identify CEFR Levels of Second Language Learners Using BERT. In Proceedings of the 13th Workshop on Natural Language Processing for Computer Assisted Language Learning, pages 137–152, Rennes, France. LiU Electronic Press.

## Model Card Authors

Ricardo Muñoz Sánchez ([rimusa](https://huggingface.co/rimusa))

## Model Card Contact

For more information about the model or the present Model Card, you can reach out to:
- Ricardo Muñoz Sánchez ([mailto:ricardo.munoz.sanchez@gu.se])
- Elena Volodina ([mailto:elena.volodina@gu.se])