File size: 3,698 Bytes
9029278 fc9bbe6 9029278 45ff303 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 | ---
language:
- gsw
- en
- fr
- it
license: mit
library_name: transformers
pipeline_tag: token-classification
base_model: ZurichNLP/swissbert
tags:
- code-switching
- language-identification
- child-speech
- multilingual
---
# SwissBERT for Token-Level Language Identification in Multilingual Child Speech
This repository contains a fine-tuned version of **SwissBERT** for word-level language identification in multilingual child–caregiver interactions.
The model predicts a language label for each word and supports downstream analyses such as:
- Inter-sentential code-switching
- Intra-sentential code-switching
- Cross-speaker switching
- Switch-point detection
- Multilingual child speech profiling
The model was trained on manually annotated child speech transcripts containing Swiss German, English, French, Italian, and an “other” category.
Because Swiss German child speech data is limited, the model was partially trained on the **SwissDial dataset** to improve Swiss German coverage.
---
## Model Description
- **Base model:** `ZurichNLP/swissbert` (XLM-RoBERTa architecture)
- **Task:** Token classification (word-level language ID)
- **Labels:** Swiss German, English, French, Italian, Other
- **Tokenizer:** SentencePiece (slow tokenizer), extended with:
- `<medium>`
- `<year>`
- `<month>`
The model is designed for:
- Child multilingualism research
- Code-switching analysis
- Annotation pipelines
- Automatic language tagging in naturalistic child speech
---
## Training Data
The training dataset is a tab-separated file with the following structure:
| sentence_id | token | label |
|-------------|-------|-------|
| 12 | Das | gsw |
| 12 | isch | gsw |
| 12 | good | eng |
| 12 | gäll | gsw |
Tokens are grouped by `sentence_id` to form sequences for token-level classification.
---
## Training Pipeline
The model was trained using the Hugging Face **Trainer API**.
### 1. Load labeled data
- Read TSV file with `(sentence_id, token, label)`
- Remove empty tokens and labels
- Normalize labels to lowercase
---
### 2. Group tokens into sentences
Tokens and labels are grouped by `sentence_id` to form input sequences.
---
### 3. Build label mappings
```python
label2id = {
"gsw": 0,
"deu": 1,
"eng": 2,
"fra": 3,
"ita": 4,
"other": 5
}
id2label = {v: k for k, v in label2id.items()}
```
### 4. Tokenization and label alignment
Because SwissBERT uses SentencePiece, each token may split into multiple subword units.
Manual alignment was implemented:
- First subword receives the label
- Remaining subwords receive `-100` (ignored in loss)
- CLS and SEP tokens also receive `-100`
- Sequences padded/truncated to `MAX_LENGTH = 128`
---
## 5. Training Configuration
- **Epochs:** 5
- **Batch size:** 8
- **Learning rate:** 5e‑5
- **Weight decay:** 0.01
- **Evaluation:** every epoch
- **Metric:** F1 (seqeval)
- **Best model selection:** enabled
- **Tokenizer:** slow SentencePiece tokenizer
---
## 6. Model Setup
```python
AutoModelForTokenClassification.from_pretrained(
MODEL_NAME,
num_labels=len(label2id),
id2label=id2label,
label2id=label2id,
)
```
## 7. Metrics
Evaluation uses seqeval, reporting:
- token‑level F1
- per‑label precision and recall
- full classification report printed during training
## References
Agnese D'Angelo, Sina Ahmadi, Moritz M. Daum, and Stephanie Wermelinger. 2026. Code-Switching Detection in Multilingual Child Speech with SwissBERT. In *Proceedings of the 11th Swiss Text Analytics Conference (SwissText 2026)*, Zurich, Switzerland. |