|
|
--- |
|
|
library_name: transformers |
|
|
tags: [quantization, qwen3, qlora, causal-lm, low-rank-adapters, 4bit, bitsandbytes, peft, efficient-finetuning] |
|
|
--- |
|
|
|
|
|
# Qwen3-0.6B Quantized with QLoRA for Reasoning Tasks |
|
|
|
|
|
This is a 4-bit quantized version of `Qwen/Qwen3-0.6B-Base`, fine-tuned using LoRA adapters on multiple MCQA-style reasoning datasets. The model was optimized using QLoRA, a parameter-efficient tuning method with minimal memory footprint and minimal accuracy loss. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
### Model Description |
|
|
|
|
|
This model is: |
|
|
- A quantized version of `Qwen/Qwen3-0.6B-Base` using `bitsandbytes` 4-bit NormalFloat (nf4) |
|
|
- Fine-tuned using Low-Rank Adaptation (LoRA) with rank 8 |
|
|
- Adapted to multiple-choice reasoning datasets like AQuA-RAT and TheoremQA |
|
|
- Fully compatible with Hugging Face Transformers |
|
|
|
|
|
- **Developed by:** Ahmed Abdelmalek (EPFL CS-552 Project) |
|
|
- **Model type:** Causal Language Model |
|
|
- **Language(s):** English |
|
|
- **License:** Apache 2.0 |
|
|
- **Fine-tuned from model:** `Qwen/Qwen3-0.6B-Base` |
|
|
|
|
|
### Model Sources |
|
|
|
|
|
- [Repository](https://huggingface.co/Qwen/Qwen3-0.6B-Base) |
|
|
|
|
|
## Uses |
|
|
|
|
|
### Direct Use |
|
|
|
|
|
You can directly use this model for MCQA-style question-answering tasks using generation. |
|
|
|
|
|
### Out-of-Scope Use |
|
|
|
|
|
- Not intended for open-ended generation or safety-critical applications |
|
|
- Not intended for real-time or commercial deployment without evaluation |
|
|
|
|
|
## Bias, Risks, and Limitations |
|
|
|
|
|
- Inherits biases from its base model and training data (e.g., reasoning datasets) |
|
|
- May fail on adversarial or out-of-distribution logic tasks |
|
|
|
|
|
### Recommendations |
|
|
|
|
|
Evaluate the model against your specific reasoning task before production use. |
|
|
|
|
|
## How to Get Started with the Model |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
|
|
model_id = "your-username/MNLP_M2_quantized_model" |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) |
|
|
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True) |
|
|
|
|
|
prompt = "Question: What is 3 + 5? |
|
|
Options: |
|
|
A) 6 |
|
|
B) 8 |
|
|
C) 9 |
|
|
D) 10 |
|
|
Answer:" |
|
|
inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
|
|
outputs = model.generate(**inputs, max_new_tokens=50) |
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
``` |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Training Data |
|
|
|
|
|
- Processed versions of AQuA-RAT, TheoremQA, and custom MCQA datasets |
|
|
- Unified into a single format with rationale-enhanced prompts |
|
|
|
|
|
### Training Procedure |
|
|
|
|
|
- **Precision:** fp16 |
|
|
- **Quantization:** 4-bit nf4 + double quant + float16 compute |
|
|
- **Adapter Type:** LoRA (r=8, α=16, dropout=0.05) |
|
|
- **Base model frozen** |
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
|
|
- **Epochs:** 3 |
|
|
- **Batch size:** 4 |
|
|
- **Grad accum steps:** 2 |
|
|
- **Optimizer:** paged_adamw_8bit |
|
|
|
|
|
## Evaluation |
|
|
|
|
|
### Testing Data |
|
|
|
|
|
Validation set with 1000 samples held out from the unified dataset. |
|
|
|
|
|
### Metrics |
|
|
|
|
|
- Accuracy / F1 (to be reported in evaluation phase) |
|
|
|
|
|
## Environmental Impact |
|
|
|
|
|
- **Hardware:** Google Colab Pro, GPU A100 |
|
|
- **Hours used:** ~6–7 hours |
|
|
- **Carbon Emitted:** Estimated with [MLCO2](https://mlco2.github.io/impact#compute) |
|
|
|
|
|
## Technical Specifications |
|
|
|
|
|
### Architecture |
|
|
|
|
|
- Qwen3-0.6B base |
|
|
- 28-layer transformer with rotary positional encoding and 16 heads |
|
|
|
|
|
### Compute Infrastructure |
|
|
|
|
|
- **Hardware:** Colab A100 GPU, High RAM |
|
|
- **Software:** Python 3.10, PyTorch 2.2.2, Transformers 4.51.3 |
|
|
|
|
|
## Contact |
|
|
|
|
|
- **Author:** Ahmed Abdelmalek |
|
|
- **Email:** ahmed.abdelmalek@epfl.ch |
