File size: 1,249 Bytes
5115436 0dcc8f9 518fb7d 0dcc8f9 518fb7d 5115436 0dcc8f9 518fb7d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
---
license: apache-2.0
pipeline_tag: text-classification
language:
- yue
widget:
- text: 係唔係去食飯？
example_title: Cantonese
- text: 台灣真美!
example_title: Traditional Chinese
datasets:
- raptorkwok/cantonese-traditional-chinese-parallel-corpus
---
## Model Description
A BERT-based model trained to classify text as either Cantonese or Traditional Chinese.
## Intended Use
- **Primary Application**: Language classification for Cantonese and Traditional Chinese texts.
- **Users**: NLP researchers, developers working with Chinese language data.
## Training Data
Utilizes the "raptorkwok/cantonese-traditional-chinese-parallel-corpus" from Hugging Face Datasets.
## Training Procedure
- **Base Model**: `bert-base-chinese`
- **Epochs**: 3
- **Learning Rate**: 2e-5
## How to Use
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("ming030890/chinese-langid")
model = AutoModelForSequenceClassification.from_pretrained("ming030890/chinese-langid")
text = "係唔係廣東話？"
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
# 0 for Cantonese, 1 for Traditional Chinese
prediction = outputs.logits.argmax(-1).item()
``` |