README.md

---

tags:
- cefr
- swe-cefr-sp
- text-classification
- swedish
- prototype-based
- sentence-level
- language-assessment
language:
- sv
license: mit
library_name: transformers
pipeline_tag: text-classification
widget:
- text: Jag heter Anna.
  example_title: Simple sentence
- text: Det är viktigt att tänka på miljön när man planerar en resa.
  example_title: Complex sentence
---


# CEFR Prototype-based Model (k=3)

This is a prototype-based classifier for Swedish text CEFR level estimation with AutoConfig/AutoModel support, compatible with Hugging Face Transformers.

## Model Details

### Architecture
- **Base Model**: [KB/bert-base-swedish-cased](https://huggingface.co/KB/bert-base-swedish-cased)
- **Prototypes**: 3 prototypes per CEFR level
- **Total Prototypes**: 18 (6 levels × 3 prototypes)
- **Classification**: Cosine similarity with temperature scaling

### Key Features
- Mean pooling on BERT layer -2 (11th layer for BERT-base)
- Temperature scaling: 10.0
- L2-normalized embeddings and prototypes
- Prototypes averaged per class during inference
- SafeTensors format for efficient loading

### CEFR Levels
- 0: A1 (Beginner)
- 1: A2 (Elementary)
- 2: B1 (Intermediate)
- 3: B2 (Upper Intermediate)
- 4: C1 (Advanced)
- 5: C2 (Proficient)

## Usage

### Installation

```bash

pip install torch transformers

```

### Quick Start

```python

import torch

from transformers import AutoTokenizer



# Load model and tokenizer

model_name = "fffffwl/swe-cefr-sp"



# If you have the model class locally:

from convert_proto_model_to_hf import CEFRPrototypeModel

model = CEFRPrototypeModel.from_pretrained(model_name)

tokenizer = AutoTokenizer.from_pretrained(model_name)



# Example text

text = "Jag heter Anna och jag kommer från Sverige."



# Tokenize and predict

inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)

outputs = model(**inputs)



# Get predictions

probs = torch.nn.functional.softmax(outputs.logits, dim=-1)

predicted_class = torch.argmax(probs, dim=-1).item()



# Map to CEFR level

cefr_labels = ["A1", "A2", "B1", "B2", "C1", "C2"]

print(f"Text: {text}")

print(f"Predicted CEFR level: {cefr_labels[predicted_class]}")

print(f"Confidence: {probs[0][predicted_class].item():.3f}")

```

## Model Implementation

### Custom Classes

```python

class CEFRProtoConfig(PretrainedConfig):

    model_type = "cefr_prototype"



    def __init__(

        self,

        encoder_name: str = "KB/bert-base-swedish-cased",

        num_labels: int = 6,

        prototypes_per_class: int = 3,

        temperature: float = 10.0,

        layer_index: int = -2,

        hidden_size: int = 768,

        **kwargs

    ):

```

```python

class CEFRPrototypeModel(PreTrainedModel):

    def encode(self, input_ids, attention_mask, token_type_ids=None) -> torch.Tensor:

        # Mean pooling on BERT layer -2

        # L2 normalization

        pass



    def forward(self, input_ids, attention_mask, token_type_ids=None, labels=None):

        # Cosine similarity with prototypes

        # Temperature scaling

        pass

```

## Performance

On the Swedish CEFR sentence dataset (10k sentences from COCTAILL, 8 Sidor, and SUC3):

- **Macro-F1**: 84.1%
- **Quadratic Weighted Kappa (QWK)**: 94.6%
- **Accuracy**: Significantly outperforms BERT baseline by 12.1% in macro-F1

## Training Details

### Dataset
- Swedish CEFR-annotated sentences
- Multi-level annotations (low/high boundaries)
- Sentence-level classification

### Training Configuration
- **Optimizer**: AdamW
- **Loss**: Cross-entropy with class weighting
- **Prototypes initialization**: Mean of class embeddings + orthogonalization
- **Temperature**: 10.0 (trainable during fine-tuning)
- **Layer**: -2 (11th BERT layer)

## Model Files

- `model.safetensors` - Model weights (476MB)
- `config.json` - Model configuration
- `tokenizer.json` - Tokenizer vocabulary
- `tokenizer_config.json` - Tokenizer configuration

## Limitations

- Model is trained specifically for Swedish text
- Sentence-level classification (not document-level)
- Requires sentences with reasonable length (recommended: 8-128 tokens)

## Citations

If you use this model in your research, please cite:

```bibtex

@misc{fan2024swedish,

  title={Swedish Sentence-Level CEFR Classification with LLM Annotations},

  author={Fan, Wenlin},

  year={2024},

  howpublished={\url{https://huggingface.co/fffffwl/swe-cefr-sp}}

}

```

Or as part of the broader project:

```bibtex

@misc{fan2024swecefrsp,

  title={Swedish CEFR Sentence-level Assessment using Large Language Models},

  author={Fan, Wenlin},

  year={2024},

  publisher={GitHub},

  howpublished={\url{https://github.com/fanwenlin/swe-cefr-sp}},

  note={Dataset, LLM annotating codes and sentence-level assessment codes available}

}

```

## Project Links

- **GitHub Repository**: https://github.com/fanwenlin/swe-cefr-sp
- **Hugging Face Space**: Available with interactive demo
- **Dataset**: 10k Swedish sentences annotated from COCTAILL, 8 Sidor, and SUC3
- **Main Model**: This prototype-based model (k=3) with Swedish BERT

## Related Work

This work builds upon:
- Yoshioka et al. (2022): CEFR-based Sentence Profile (CEFR-SP) and prototype-based metric learning
- Volodina et al. (2016): Swedish passage readability assessment
- Scarton et al. (2018): Controllable text simplification

## License

This model is released under the MIT License. See LICENSE file for details.

## Related Models

This repository also contains:
- Original k=1 checkpoint: `metric-proto-k1.pt`
- Original k=3 checkpoint: `metric-proto-k3.pt` (this model)
- Original k=5 checkpoint: `metric-proto-k5.pt`
- BERT baseline: `bert-baseline.pt`
- Megatron version: `metric-proto-megatron-k3.pt`
- Traditional ML models: `linear_regression.joblib`, `logreg.joblib`, `svm.joblib`, `mlp.joblib`, `tree.joblib`

For more details, visit the [project repository](https://github.com/fanwenlin/swe-cefr-sp).