| --- |
| license: mit |
| language: |
| - en |
| library_name: pytorch |
| pipeline_tag: text-generation |
| tags: |
| - text-generation |
| - causal-lm |
| - language-model |
| - base-model |
| - pretrained |
| - transformer |
| - decoder-only |
| - english |
| - pytorch |
| - sentencepiece |
| - gqa |
| - swiglu |
| - rmsnorm |
| - rope |
| - 154m |
| - qarvexium |
| - qed |
| - qed-base |
| - qed-base-v3 |
| - foundation-model |
| - foundation |
| model-index: |
| - name: QED-Base-v3 |
| results: [] |
| --- |
| |
| # QED-Base-v3 |
|
|
| QED-Base-v3 is a **~111M parameter causal language model** pretrained from scratch by **Qarvexium**. It is a base model — it has not been instruction-tuned or aligned for chat, and is designed to continue text rather than follow instructions or hold a conversation. |
|
|
| ## Model Details |
|
|
| * **Developed by:** Qarvexium |
| * **Model type:** Decoder-only causal language base model |
| * **Language:** English |
| * **License:** MIT |
| * **Tokenizer:** QED-B3 tokenizer (SentencePiece, 56,000 vocabulary) |
|
|
| ### Architecture |
|
|
| | Component | Value | |
| | --------------------- | ------------------------ | |
| | Tokenizer | QED-B3 tokenizer | |
| | Vocabulary size | 56,000 | |
| | Model type | Decoder-only Transformer | |
| | Parameters | ~111M | |
| | Hidden size | 768 | |
| | Layers | 12 | |
| | Attention heads | 12 | |
| | KV heads | 4 | |
| | Attention | GQA | |
| | Intermediate FFN size | 1,792 | |
| | Activation | SwiGLU | |
| | Normalization | RMSNorm | |
| | Position encoding | RoPE | |
| | Context length | 2,048 | |
| | RoPE theta | 10,000 | |
|
|
| Weight-tied embeddings/LM head. |
|
|
| ## Uses |
|
|
| ### Direct Use |
|
|
| As a base model, QED-Base-v3 is intended for: |
|
|
| * Text completion / continuation |
| * Research on small-scale language model pretraining |
| * Experimenting with the QED architecture |
| * Studying tokenizer and language-model behavior |
| * A starting checkpoint for further fine-tuning |
| * Instruction tuning and downstream model development |
|
|
| ### Out-of-Scope Use |
|
|
| This model has not been instruction-tuned, RLHF'd, or safety-aligned. |
|
|
| It should not be deployed directly as a chat or assistant model, or in applications requiring reliable instruction-following or content moderation, without additional fine-tuning and evaluation. |
|
|
| ## Bias, Risks, and Limitations |
|
|
| QED-Base-v3 is an experimental base language model and may produce incorrect, nonsensical, repetitive, biased, or otherwise undesirable text. |
|
|
| Because it is a base model, it does not have built-in instruction-following or refusal behavior. |
|
|
| Its relatively small parameter count also means that its factual knowledge, reasoning ability, and generalization capabilities are limited compared with substantially larger language models. |
|
|
| Outputs should be evaluated and filtered before use in user-facing applications. |
|
|
| ## How to Get Started |
|
|
| The repository includes a lightweight inference implementation in `infer.py`. |
|
|
| ```python |
| from infer import load_model, load_tokenizer, run |
| |
| model = load_model("QED-Base-v3.pt") |
| tokenizer = load_tokenizer("qed-b3-tok.model") |
| |
| text = run( |
| "Once upon a time", |
| model, |
| tokenizer, |
| max_new_tokens=100 |
| ) |
| |
| print(text) |
| ``` |
|
|
| For generation, the included inference implementation supports temperature, top-k, top-p, repetition penalty, and seeded generation. |
|
|
| ## Tokenizer |
|
|
| QED-Base-v3 introduces a **new tokenizer trained specifically for this model generation**, rather than recycling the tokenizer used by QED-Base-v1. |
|
|
| The tokenizer uses a **56,000-token vocabulary** and is provided in the repository as: |
|
|
| `qed-b3-tok.model` |
|
|
| ## QED Family |
|
|
| QED-Base-v3 is part of the QED family of language models. |
|
|
| | Model | Description | |
| | ------------------ | ---------------------------------------------------- | |
| | QED-Base-v1 | First-generation QED base model | |
| | QED-Base-v2 | Second-generation QED base model | |
| | **QED-Base-v3** | Third-generation QED base model with a new tokenizer | |
| | QED-B1/B2-Instruction | Instruction-tuned QED variants | |
|
|
| QED-Base-v3 is intended to serve as a foundation for future QED experiments and fine-tuned models. |
|
|
| ## License |
|
|
| MIT License |
