HiImHa
/

phobert-cross-encoder

@@ -1,189 +1,170 @@
 ---
 library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Shared by [optional]:** Hong Ha
-- **Language(s) (NLP):** Vietnamese
-- **Finetuned from model [optional]:** vinai-phobert-base-v2
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
 ## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
+tags:
+- transformers
+- text-classification
+- reranking
+- cross-encoder
+- vietnamese
+- phobert
+- rag
+- generated_from_trainer
+base_model: vinai/phobert-base-v2
+pipeline_tag: text-classification
 library_name: transformers
+metrics:
+- accuracy
+- f1
+model-index:
+- name: PhoBERT Cross-Encoder for Reranking
+  results:
+  - task:
+      type: text-classification
+      name: Relevance Classification
+    dataset:
+      name: cross_eval
+      type: cross_eval
+    metrics:
+    - type: accuracy
+      value: 0.995473
+      name: Accuracy
+    - type: f1
+      value: 0.990951
+      name: F1 Score
 ---
+# PhoBERT Cross-Encoder for Vietnamese Reranking
+This model is a cross-encoder fine-tuned from `vinai/phobert-base-v2` for binary relevance classification between a query and a document.
+Unlike bi-encoders, this model jointly encodes (query, context) pairs, enabling high-accuracy reranking in retrieval systems.
+## Model Overview
+*   **Architecture:** Cross-Encoder (Sequence Classification)
+*   **Base Model:** `vinai/phobert-base-v2`
+*   **Task:** Binary classification (relevant / not relevant)
+*   **Input Format:** `[CLS] query [SEP] context [SEP]`
+*   **Max Sequence Length:** 256 tokens
+## Intended Use
+This model is designed for:
+*   Reranking top-k results from a bi-encoder
+*   Improving semantic search precision
+*   Vietnamese legal QA systems
+*   Second-stage ranking in RAG pipelines
 ## Training Details
+### Dataset
+**Format:**
+*   query
+*   context
+*   label (0 = irrelevant, 1 = relevant)
+### Training Configuration
+*   **Epochs:** 5
+*   **Learning rate:** 2e-5
+*   **Batch size:**
+    *   Train: 16
+    *   Eval: 32
+*   **Warmup:** 0.1
+*   **Weight decay:** 0.01
+*   **Mixed precision:** FP16 (if GPU available)
+## Evaluation Results
+| Epoch | Validation Loss | Accuracy | F1 Score |
+| :---: | :---: | :---: | :---: |
+| 1 | 0.0820 | 0.9934 | 0.9869 |
+| 2 | 0.0675 | 0.9936 | 0.9871 |
+| 3 | 0.0793 | 0.9934 | 0.9869 |
+| 4 | 0.0572 | 0.9955 | 0.9910 |
+| 5 | 0.0711 | 0.9955 | 0.9910 |
+*Best model selected based on F1 score = 0.9909*
+## Model Architecture
+PhoBERT (RoBERTa-based encoder) -> Classification Head (dense + output layer)
+## Usage
+### Load model
+```python
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+model_name = "HiImHa/phobert-cross-encoder"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForSequenceClassification.from_pretrained(model_name)
+```
+### Inference Example
+```python
+query = "Tôi lái xe không giữ khoảng cách an toàn thì bị phạt như thế nào?"
+context = "Phạt tiền từ 2.000.000 đến 3.000.000 đồng nếu không giữ khoảng cách an toàn."
+inputs = tokenizer(
+    query,
+    context,
+    return_tensors="pt",
+    truncation="only_second",
+    max_length=256
+)
+outputs = model(**inputs)
+score = outputs.logits.softmax(dim=-1)[0][1].item()
+print(score)  # relevance score
+```
+## How to Use in RAG
+Typical pipeline:
+1.  Use bi-encoder -> retrieve top-k documents
+2.  Use this cross-encoder -> rerank candidates
+3.  Select top results for downstream tasks
+## Notes on Initialization
+*   Classification head was randomly initialized and trained during fine-tuning
+*   Some PhoBERT pretraining weights (e.g., `lm_head`) are unused -> expected behavior
+*   LayerNorm naming differences (beta/gamma vs weight/bias) are automatically handled
+## Limitations
+*   Slower than bi-encoder (pairwise inference)
+*   Limited to 256 tokens -> long contexts are truncated
+*   Binary classification may not capture nuanced ranking differences
+## Future Improvements
+*   Pairwise / listwise ranking loss
+*   Hard negative mining
+*   Knowledge distillation from cross -> bi encoder
+*   Larger and more diverse dataset
+## Training Configuration (Summary)
+*   **Epochs:** 5
+*   **Learning rate:** 2e-5
+*   **Loss:** Cross-entropy
+*   **Metric:** F1 (primary)
+## Acknowledgements
+*   PhoBERT by VinAI
+*   Hugging Face Transformers
+## License
+Specify your license here (e.g., MIT / Apache 2.0)
+## Citation
+```bibtex
+@inproceedings{reimers-2019-sentence-bert,
+  title={Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
+  author={Reimers, Nils and Gurevych, Iryna},
+  year={2019}
+}
+```