|
|
--- |
|
|
base_model: sentence-transformers/all-MiniLM-L6-v2 |
|
|
library_name: peft |
|
|
license: mit |
|
|
tags: |
|
|
- lora |
|
|
- peft |
|
|
- code |
|
|
- programming |
|
|
- software |
|
|
- domain-adaptation |
|
|
- sentence-embeddings |
|
|
language: |
|
|
- en |
|
|
--- |
|
|
|
|
|
# Code LoRA Adapter for DomainEmbedder-v2.6 |
|
|
|
|
|
Domain-specific LoRA adapter for code/programming text embeddings. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
| Property | Value | |
|
|
|----------|-------| |
|
|
| **Base Model** | sentence-transformers/all-MiniLM-L6-v2 | |
|
|
| **Parent System** | DomainEmbedder-v2.6 | |
|
|
| **Domain** | Code / Programming | |
|
|
| **LoRA Rank** | 16 | |
|
|
| **LoRA Alpha** | 32 | |
|
|
| **Target Modules** | query, value | |
|
|
| **Trainable Params** | 147,456 (0.645%) | |
|
|
|
|
|
## Training Data |
|
|
|
|
|
Trained on 40,000 code-related text pairs from: |
|
|
- Code Alpaca |
|
|
- MBPP (Mostly Basic Python Problems) |
|
|
- Code Contests |
|
|
- Python Instructions |
|
|
|
|
|
## Training Configuration |
|
|
|
|
|
| Parameter | Value | |
|
|
|-----------|-------| |
|
|
| Epochs | 3 | |
|
|
| Batch Size | 32 | |
|
|
| Learning Rate | 2e-4 | |
|
|
| Loss | Contrastive (InfoNCE) | |
|
|
| Best Val Loss | 0.0039 | |
|
|
|
|
|
## Usage |
|
|
|
|
|
This adapter is part of the DomainEmbedder-v2.6 system. It is selected automatically by the RL policy when code-related content is detected. |
|
|
|
|
|
```python |
|
|
from peft import PeftModel |
|
|
from transformers import AutoModel |
|
|
|
|
|
# Load base encoder |
|
|
base_encoder = AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2') |
|
|
|
|
|
# Apply code LoRA |
|
|
code_model = PeftModel.from_pretrained(base_encoder, 'path/to/code_lora') |
|
|
``` |
|
|
|
|
|
## Author |
|
|
|
|
|
**Zain Asad** |
|
|
|
|
|
## License |
|
|
|
|
|
MIT License |
|
|
|
|
|
## Framework Versions |
|
|
|
|
|
- PEFT 0.18.1 |
|
|
- Transformers 4.x |
|
|
- PyTorch 2.x |
|
|
|
