File size: 2,850 Bytes
e46d69f a78624e e46d69f a78624e e46d69f 7cef8a8 a78624e 821ac96 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 | ---
language:
- en
license: mit
library_name: pytorch
tags:
- text-generation
- causal-lm
- pretrained
- qarvexium
pipeline_tag: text-generation
---
# QED-Base-v2
QED-Base-v2 is a ~107M parameter causal language model pretrained from scratch by [Qarvexium](https://huggingface.co/Qarvexium). It is a **base model** — it has not been instruction-tuned or aligned for chat, and will continue text rather than follow instructions or hold a conversation.
## Model Details
- **Developed by:** Qarvexium
- **Model type:** Decoder-only causal language base model
- **Language:** English
- **License:** MIT
- **Tokenizer:** QED-B1 tokenizer (SentencePiece BPE, 48,000 vocabulary)
### Architecture
| Component | Value |
| --------------------- | -----------------------: |
| Tokenizer | QED-B1 tokenizer |
| Vocabulary size | 48,000 |
| Model type | Decoder-only Transformer |
| Parameters | ~107M |
| Hidden size | 768 |
| Layers | 12 |
| Attention heads | 12 |
| KV heads | 4 |
| Attention | GQA |
| Intermediate FFN size | 1792 |
| Activation | SwiGLU |
| Normalization | RMSNorm |
| Position encoding | RoPE |
| Context length | 2048 |
| RoPE theta | 10000 |
Weight-tied embeddings/LM head.
## Uses
### Direct Use
As a base model, QED-Base-v2 is intended for:
- Text completion / continuation
- Research on small-scale language model pretraining
- A starting checkpoint for further fine-tuning (instruction tuning, domain adaptation, etc.)
### Out-of-Scope Use
This model has **not** been instruction-tuned, RLHF'd, or safety-aligned. It should not be deployed directly in a chat or assistant product, or in any application where reliable instruction-following or content moderation is required, without further fine-tuning and evaluation.
## Bias, Risks, and Limitations
QED-Base-v2 was trained on a large web-crawled corpus and will reflect the biases, inaccuracies, and occasionally toxic content present in that data. As a base model it has no built-in refusal behavior or safety tuning — outputs should be filtered/evaluated before use in any user-facing setting. At ~107M parameters, factual accuracy and reasoning ability are limited compared to larger models.
## How to Get Started
```python
from infer import load_model, load_tokenizer, run
model = load_model("QED-Base-v2.pt")
tokenizer = load_tokenizer("tok.model")
text = run("Once upon a time", model, tokenizer, max_new_tokens=100)
print(text)
```
## License
MIT License |