README.md

---
language:
  - en
license: apache-2.0
library_name: vllm
tags:
  - reasoning
  - chain-of-thought
  - efficiency
  - inference-optimization
  - qwen3
base_model: Qwen/Qwen3-8B
base_model_relation: finetune
pipeline_tag: text-generation
---

# Terminator-Qwen3-8B

**Terminator** is a lightweight neural module that predicts when a reasoning language model has reached its final answer during chain-of-thought (CoT) generation. When the Terminator detects the model has committed to an answer, it truncates the remaining reasoning and forces the model to begin its response, thereby delivering the same answer with significantly less computation.

This repository contains everything needed to run **Terminator-Qwen3-8B**:

- Trained Terminator checkpoint (1 extra transformer layer + prediction head)
- vLLM plugin code (`vllm_terminator/`) for high-performance serving
- Server launcher and streaming client
- Standalone HuggingFace inference script (no server required)
- Automated setup script

**Note**: Terminator currently supports **single-GPU, single-sequence inference only**.

---

## Quick Start

```bash
# 1. Clone the repository (requires Git LFS: https://git-lfs.com)
git lfs install
git clone https://huggingface.co/acnagle/Terminator-Qwen3-8B
cd Terminator-Qwen3-8B

# 2. Run automated setup (creates conda env, installs vllm, downloads base model)
./setup.sh

# 3. Start the server
./start_server.sh

# 4. In another terminal, chat with the model
python client.py --interactive
```

---

## Requirements

- **GPU**: Single NVIDIA GPU with at least ~24GB VRAM
- **CUDA**: Compatible CUDA driver installed, 12.9 and above recommended.
- **Python**: 3.12
- **OS**: Linux (recommended) or any OS supported by vLLM

---

## Installation

### Option A: Automated Setup

The `setup.sh` script handles everything:

```bash
./setup.sh
```

This will:
1. Create a conda environment called `terminator` with Python 3.12
2. Install [uv](https://docs.astral.sh/uv/), [vLLM](https://docs.vllm.ai/), and [openai](https://pypi.org/project/openai/)
3. Download Qwen3-8B base model weights (~16GB) from HuggingFace
4. Create the model directory (`model_dir/`)

### Option B: Manual Setup

**1. Create a Python environment**

Using conda or micromamba:

```bash
conda create -n terminator python=3.12 -y
conda activate terminator
```

**2. Install uv**

```bash
pip install --upgrade uv
```

Or see the [uv installation guide](https://docs.astral.sh/uv/getting-started/installation/).

**3. Install vLLM**

```bash
uv pip install vllm --torch-backend=auto
```

See the [vLLM installation guide](https://docs.vllm.ai/en/latest/getting_started/installation/) for alternative installation methods (ROCm, CPU, etc.).

**4. Install openai (for the client)**

```bash
uv pip install openai
```

**5. Set up the model directory**

This downloads the base Qwen3-8B weights and creates a vLLM-ready model directory:

```bash
python setup_model_dir.py
```

The script accepts optional arguments:

| Argument | Default | Description |
|----------|---------|-------------|
| `--checkpoint` | `./terminator.pt` | Path to the Terminator checkpoint |
| `--output-dir` | `./model_dir` | Output model directory |
| `--threshold` | `0.7` | Prediction threshold for Terminator activation |
| `--window-size` | `10` | Sliding window size for majority vote |
| `--exit-message` | *(built-in message)* | Message injected when Terminator fires |

---

## Starting the Server

```bash
./start_server.sh
```

Or with custom configuration:

```bash
VLLM_GPU_UTIL=0.70 VLLM_MAX_MODEL_LEN=8192 ./start_server.sh
```

The server exposes an **OpenAI-compatible API** on the configured port (default: 8000).

### Configuration

Set these environment variables before running `start_server.sh` or `serve.py`:

| Variable | Default | Description |
|----------|---------|-------------|
| `VLLM_GPU_UTIL` | `0.90` | Fraction of GPU memory to use for the model |
| `VLLM_MAX_MODEL_LEN` | *(auto)* | Maximum context length in tokens |
| `VLLM_PORT` | `8000` | Server port |
| `VLLM_ENFORCE_EAGER` | `0` | Set to `1` to disable CUDA graphs |
| `VLLM_API_KEY` | *(none)* | Require this API key from clients |
| `VLLM_SERVED_NAME` | `Terminator-Qwen3-8B` | Model name reported by the API |

---

## Standalone Inference (No Server)

**Recommendation:** For the best performance, use the vLLM server described above. vLLM uses KV caching, CUDA graphs, and optimized kernels, making it **significantly faster** than HuggingFace-native inference. The script below is provided for quick testing and demos where spinning up a server is inconvenient.

For quick testing without starting a vLLM server, use the HuggingFace-native inference script:

```bash
python inference_hf.py --prompt "What is the sum of the first 100 natural numbers?"
```

This loads the model directly via HuggingFace `transformers` and runs token-by-token generation with the Terminator head. Thinking content is streamed in dimmed text; the final answer is shown in bold.

| Argument | Default | Description |
|----------|---------|-------------|
| `--prompt` | *(required)* | Input prompt |
| `--model` | `Qwen/Qwen3-8B` | HuggingFace model name or path |
| `--checkpoint` | `./terminator.pt` | Path to the Terminator checkpoint |
| `--threshold` | `0.7` | Prediction threshold |
| `--window-size` | `10` | Sliding window size for majority vote |
| `--exit-message` | *(built-in message)* | Message injected when Terminator fires (empty string to disable) |
| `--max-tokens` | `32768` | Maximum tokens to generate |
| `--temperature` | `0.6` | Sampling temperature |

---

## Using the Client (vLLM Server)

### Single Prompt

```bash
python client.py --prompt "What is the sum of the first 100 natural numbers?"
```

### Interactive Mode

```bash
python client.py --interactive
```

This starts a multi-turn conversation with the model. Thinking content is displayed in dimmed text; the final answer is shown in bold.

### Client Options

| Argument | Default | Description |
|----------|---------|-------------|
| `--base-url` | `http://localhost:8000/v1` | Server URL |
| `--max-tokens` | *(server default)* | Maximum tokens to generate |
| `--temperature` | `0.6` | Sampling temperature |

### Using the API Directly

The server is OpenAI-compatible. You can use any OpenAI client library. Replace `localhost` with your server's address if connecting remotely:

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

response = client.chat.completions.create(
    model="Terminator-Qwen3-8B",
    messages=[{"role": "user", "content": "What is 25 * 37?"}],
    temperature=0.6,
    extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)

# Thinking content (chain-of-thought)
print(response.choices[0].message.reasoning)

# Final answer
print(response.choices[0].message.content)
```

---

## How Terminator Works

Terminator is a single transformer layer followed by a prediction head, trained on top of a frozen Qwen3-8B base model. The transformer layer (initialized as a copy of the base model's final layer, then fine-tuned) takes the hidden states from the LLM and processes them before the prediction head, which outputs a per-token binary prediction: *has the model reached its final answer?*

During generation, Terminator maintains a **sliding window** of the most recent predictions. When a majority of predictions in the window exceed the threshold (default: 0.7), the model is considered to have reached its final answer. At that point:

1. A short **exit message** is injected into the reasoning (e.g., *"I've run out of thinking tokens. I need to commit to a final answer."*) to help the model transition smoothly.
2. The `</think>` token is forced, ending the reasoning phase.
3. The model generates its final answer normally.

This allows the model to skip potentially thousands of redundant reasoning tokens while preserving answer quality.

---

## File Structure

```
Terminator-Qwen3-8B/
├── README.md               This file
├── terminator.pt            Trained Terminator checkpoint
├── vllm_terminator/         vLLM plugin package
│   ├── __init__.py          Registers the model architecture with vLLM
│   ├── model.py             Qwen3TerminatorForCausalLM model class
│   └── terminator_head.py   FFN classifier and checkpoint loading
├── inference_hf.py          Standalone HuggingFace inference (no server)
├── serve.py       vLLM server launcher
├── setup_model_dir.py       Model directory setup (downloads base weights)
├── client.py                Streaming chat client (connects to vLLM server)
├── setup.sh                 Automated setup script
└── start_server.sh          Server launcher with sensible defaults
```

---

## Citation

*Coming soon.*

---

## License

This project builds on [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) by the Qwen team. Please refer to the Qwen3 license for base model usage terms.