Upload folder using huggingface_hub

README.md

@@ -0,0 +1,139 @@
+---
+language:
+- vi
+license: mit
+tags:
+- bert
+- roberta
+- vietnamese
+- fill-mask
+- feature-extraction
+datasets:
+- cc100
+pipeline_tag: fill-mask
+model-index:
+- name: viBERT-base
+  results:
+  - task:
+      type: token-classification
+      name: Named Entity Recognition
+    dataset:
+      name: PhoNER_COVID19
+      type: VinAIResearch/PhoNER_COVID19
+    metrics:
+    - type: f1
+      value: 89.38
+      name: F1 (micro)
+---
+
+# viBERT-base
+
+A Vietnamese RoBERTa-based language model pre-trained on CC-100 Vietnamese and custom Vietnamese corpus.
+
+## Model Description
+
+**viBERT-base** is a BERT-base architecture model trained with RoBERTa-style pre-training on Vietnamese text data. It can be used for various Vietnamese NLP downstream tasks such as Named Entity Recognition, Text Classification, Question Answering, and more.
+
+### Model Architecture
+
+| Parameter | Value |
+|-----------|-------|
+| Architecture | BERT-base |
+| Hidden size | 768 |
+| Attention heads | 12 |
+| Hidden layers | 12 |
+| Vocab size | 41,035 |
+| Max sequence length | 512 |
+| Parameters | ~110M |
+
+## Training Data
+
+- **CC-100 Vietnamese**: Large-scale web crawl data
+- **Custom Vietnamese corpus**: Additional curated Vietnamese text
+
+## Usage
+
+### Feature Extraction
+
+```python
+from transformers import AutoModel, AutoTokenizer
+
+# Load model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained("mainguyen9/viBERT-base")
+model = AutoModel.from_pretrained("mainguyen9/viBERT-base")
+
+# Encode text
+text = "Xin chào Việt Nam"
+inputs = tokenizer(text, return_tensors="pt")
+outputs = model(**inputs)
+
+# Get embeddings
+last_hidden_state = outputs.last_hidden_state
+```
+
+### Masked Language Modeling
+
+```python
+from transformers import pipeline
+
+fill_mask = pipeline("fill-mask", model="mainguyen9/viBERT-base")
+result = fill_mask("Hà Nội là [MASK] đô của Việt Nam.")
+print(result)
+```
+
+### Fine-tuning for NER
+
+```python
+from transformers import AutoModelForTokenClassification, AutoTokenizer
+
+model = AutoModelForTokenClassification.from_pretrained(
+    "mainguyen9/viBERT-base",
+    num_labels=num_labels
+)
+tokenizer = AutoTokenizer.from_pretrained("mainguyen9/viBERT-base")
+```
+
+## Benchmark Results
+
+| Task | Dataset | Metric | Score |
+|------|---------|--------|-------|
+| NER | PhoNER_COVID19 | F1 | **89.38** |
+
+### NER Performance Details (PhoNER_COVID19)
+
+Fine-tuned with 5 epochs, batch size 32, learning rate 2e-5.
+
+| Entity Type | Precision | Recall | F1-Score | Support |
+|-------------|-----------|--------|----------|---------|
+| AGE | 90.91 | 97.27 | 93.98 | 586 |
+| DATE | 98.20 | 99.17 | 98.68 | 3,026 |
+| GENDER | 89.96 | 92.23 | 91.08 | 476 |
+| JOB | 66.59 | 51.75 | 58.24 | 570 |
+| LOCATION | 88.52 | 91.33 | 89.90 | 10,845 |
+| NAME | 94.09 | 90.56 | 92.29 | 1,388 |
+| ORGANIZATION | 77.02 | 78.05 | 77.53 | 1,640 |
+| PATIENT_ID | 95.61 | 98.54 | 97.05 | 2,120 |
+| SYMPTOM_AND_DISEASE | 82.84 | 74.70 | 78.56 | 2,158 |
+| TRANSPORTATION | 85.63 | 91.41 | 88.43 | 489 |
+| **Micro Average** | **89.09** | **89.69** | **89.38** | **23,298** |
+
+## Limitations
+
+- Primarily trained on Vietnamese text; performance may vary for code-mixed text
+- 512 token maximum sequence length
+
+## Citation
+
+```bibtex
+@misc{vibert-base,
+  author = {Mai Nguyen},
+  title = {viBERT-base: A Vietnamese RoBERTa Model},
+  year = {2024},
+  publisher = {Hugging Face},
+  url = {https://huggingface.co/mainguyen9/viBERT-base}
+}
+```
+
+## License
+
+MIT License

config.json

@@ -0,0 +1,30 @@
+{
+  "add_cross_attention": false,
+  "architectures": [
+    "BertModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": null,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": null,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "is_decoder": false,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 512,
+  "model_type": "bert",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "tie_word_embeddings": true,
+  "transformers_version": "5.3.0",
+  "type_vocab_size": 2,
+  "use_cache": true,
+  "vocab_size": 41035
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3479edd7b96e3a081918b04005163d02c638720450d38a34970fcbdeffad0dc5
+size 470247248

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:51d97adbfa1cb0a84bbde0e1dffca6a86cf13093a917f90bc287cd46b501eaad
+size 470466795

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "name_or_path": "/home/hieu.tran5/models/moberta_mlm_base",
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "special_tokens_map_file": "/home/hieu.tran5/models/moberta_mlm_base/special_tokens_map.json",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}