---
language:
- en
license: mit
library_name: pytorch
tags:
- text-generation
- causal-lm
- pretrained
- qarvexium
pipeline_tag: text-generation
---

# QED-Base-v2

QED-Base-v2 is a ~107M parameter causal language model pretrained from scratch by [Qarvexium](https://huggingface.co/Qarvexium). It is a **base model** — it has not been instruction-tuned or aligned for chat, and will continue text rather than follow instructions or hold a conversation.

## Model Details

- **Developed by:** Qarvexium
- **Model type:** Decoder-only causal language base model
- **Language:** English
- **License:** MIT
- **Tokenizer:** QED-B1 tokenizer (SentencePiece BPE, 48,000 vocabulary)

### Architecture

| Component | Value |
| --------------------- | -----------------------: |
| Tokenizer             |     QED-B1 tokenizer |
| Vocabulary size       |               48,000 |
| Model type            | Decoder-only Transformer |
| Parameters            |                    ~107M |
| Hidden size           |                      768 |
| Layers                |                       12 |
| Attention heads       |                       12 |
| KV heads              |                        4 |
| Attention             |                      GQA |
| Intermediate FFN size |                     1792 |
| Activation            |                   SwiGLU |
| Normalization         |                  RMSNorm |
| Position encoding     |                     RoPE |
| Context length        |                     2048 |
| RoPE theta            |                    10000 |

Weight-tied embeddings/LM head.

## Uses

### Direct Use

As a base model, QED-Base-v2 is intended for:
- Text completion / continuation
- Research on small-scale language model pretraining
- A starting checkpoint for further fine-tuning (instruction tuning, domain adaptation, etc.)

### Out-of-Scope Use

This model has **not** been instruction-tuned, RLHF'd, or safety-aligned. It should not be deployed directly in a chat or assistant product, or in any application where reliable instruction-following or content moderation is required, without further fine-tuning and evaluation.

## Bias, Risks, and Limitations

QED-Base-v2 was trained on a large web-crawled corpus and will reflect the biases, inaccuracies, and occasionally toxic content present in that data. As a base model it has no built-in refusal behavior or safety tuning — outputs should be filtered/evaluated before use in any user-facing setting. At ~107M parameters, factual accuracy and reasoning ability are limited compared to larger models.

## How to Get Started

```python
from infer import load_model, load_tokenizer, run

model = load_model("QED-Base-v2.pt")
tokenizer = load_tokenizer("tok.model")

text = run("Once upon a time", model, tokenizer, max_new_tokens=100)
print(text)
```

## License

MIT License