File size: 4,224 Bytes
981d02b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 |
---
base_model: Qwen/Qwen3-8B
library_name: peft
pipeline_tag: text-generation
language: en
license: mit
tags:
- lora
- sft
- transformers
- trl
- unsloth
- fine-tuned
datasets:
- theprint/Advocate-9.4k
---
# DevilsAdvocate-8B
A fine-tuned Qwen 3 8B model, fine tuned for more engaging conversation, encouraging the user to think about different aspects.
## Model Details
This model is a fine-tuned version of Qwen/Qwen3-8B using the Unsloth framework with LoRA (Low-Rank Adaptation) for efficient training.
- **Developed by:** theprint
- **Model type:** Causal Language Model (Fine-tuned with LoRA)
- **Language:** en
- **License:** mit
- **Base model:** Qwen/Qwen3-8B
- **Fine-tuning method:** LoRA with rank 128
## Intended Use
General conversation, project feedback and brainstorming.
## GGUF Quantized Versions
Quantized GGUF versions are available in the [theprint/DevilsAdvocate-8B-GGUF](https://huggingface.co/theprint/DevilsAdvocate-8B-GGUF) repo.
- `DevilsAdvocate-8B-f16.gguf` (15628.9 MB) - 16-bit float (original precision, largest file)
- `DevilsAdvocate-8B-q3_k_m.gguf` (3933.1 MB) - 3-bit quantization (medium quality)
- `DevilsAdvocate-8B-q4_k_m.gguf` (4794.9 MB) - 4-bit quantization (medium, recommended for most use cases)
- `DevilsAdvocate-8B-q5_k_m.gguf` (5580.1 MB) - 5-bit quantization (medium, good quality)
- `DevilsAdvocate-8B-q6_k.gguf` (6414.3 MB) - 6-bit quantization (high quality)
- `DevilsAdvocate-8B-q8_0.gguf` (8306.0 MB) - 8-bit quantization (very high quality)
## Training Details
### Training Data
The data set used is [theprint/Advocate-9.4k](https://huggingface.co/datasets/theprint/Advocate-9.4k).
- **Dataset:** theprint/Advocate-9.4k
- **Format:** alpaca
### Training Procedure
- **Training epochs:** 2
- **LoRA rank:** 128
- **Learning rate:** 5e-05
- **Batch size:** 2
- **Framework:** Unsloth + transformers + PEFT
- **Hardware:** NVIDIA RTX 5090
## Usage
```python
from unsloth import FastLanguageModel
import torch
# Load model and tokenizer
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="theprint/DevilsAdvocate-8B",
max_seq_length=4096,
dtype=None,
load_in_4bit=True,
)
# Enable inference mode
FastLanguageModel.for_inference(model)
# Example usage
inputs = tokenizer(["Your prompt here"], return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```
### Alternative Usage (Standard Transformers)
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained(
"theprint/DevilsAdvocate-8B",
torch_dtype=torch.float16,
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("theprint/DevilsAdvocate-8B")
# Example usage
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Your question here"}
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
outputs = model.generate(inputs, max_new_tokens=256, temperature=0.7, do_sample=True)
response = tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True)
print(response)
```
### Using with llama.cpp
```bash
# Download a quantized version (q4_k_m recommended for most use cases)
wget https://huggingface.co/theprint/DevilsAdvocate-8B/resolve/main/gguf/DevilsAdvocate-8B-q4_k_m.gguf
# Run with llama.cpp
./llama.cpp/main -m DevilsAdvocate-8B-q4_k_m.gguf -p "Your prompt here" -n 256
```
## Limitations
May provide incorrect information.
## Citation
If you use this model, please cite:
```bibtex
@misc{devilsadvocate_8b,
title={DevilsAdvocate-8B: Fine-tuned Qwen/Qwen3-8B},
author={theprint},
year={2025},
publisher={Hugging Face},
url={https://huggingface.co/theprint/DevilsAdvocate-8B}
}
```
## Acknowledgments
- Base model: [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B)
- Training dataset: [theprint/Advocate-9.4k](https://huggingface.co/datasets/theprint/Advocate-9.4k)
- Fine-tuning framework: [Unsloth](https://github.com/unslothai/unsloth)
- Quantization: [llama.cpp](https://github.com/ggerganov/llama.cpp)
|