| --- |
| language: |
| - en |
| license: mit |
| library_name: pytorch |
| tags: |
| - text-generation |
| - causal-lm |
| - pretrained |
| - qarvexium |
| pipeline_tag: text-generation |
| --- |
| |
| # QED-Base-v2 |
|
|
| QED-Base-v2 is a ~107M parameter causal language model pretrained from scratch by [Qarvexium](https://huggingface.co/Qarvexium). It is a **base model** — it has not been instruction-tuned or aligned for chat, and will continue text rather than follow instructions or hold a conversation. |
|
|
| ## Model Details |
|
|
| - **Developed by:** Qarvexium |
| - **Model type:** Decoder-only causal language base model |
| - **Language:** English |
| - **License:** MIT |
| - **Tokenizer:** QED-B1 tokenizer (SentencePiece BPE, 48,000 vocabulary) |
|
|
| ### Architecture |
|
|
| | Component | Value | |
| | --------------------- | -----------------------: | |
| | Tokenizer | QED-B1 tokenizer | |
| | Vocabulary size | 48,000 | |
| | Model type | Decoder-only Transformer | |
| | Parameters | ~107M | |
| | Hidden size | 768 | |
| | Layers | 12 | |
| | Attention heads | 12 | |
| | KV heads | 4 | |
| | Attention | GQA | |
| | Intermediate FFN size | 1792 | |
| | Activation | SwiGLU | |
| | Normalization | RMSNorm | |
| | Position encoding | RoPE | |
| | Context length | 2048 | |
| | RoPE theta | 10000 | |
|
|
| Weight-tied embeddings/LM head. |
|
|
| ## Uses |
|
|
| ### Direct Use |
|
|
| As a base model, QED-Base-v2 is intended for: |
| - Text completion / continuation |
| - Research on small-scale language model pretraining |
| - A starting checkpoint for further fine-tuning (instruction tuning, domain adaptation, etc.) |
|
|
| ### Out-of-Scope Use |
|
|
| This model has **not** been instruction-tuned, RLHF'd, or safety-aligned. It should not be deployed directly in a chat or assistant product, or in any application where reliable instruction-following or content moderation is required, without further fine-tuning and evaluation. |
|
|
| ## Bias, Risks, and Limitations |
|
|
| QED-Base-v2 was trained on a large web-crawled corpus and will reflect the biases, inaccuracies, and occasionally toxic content present in that data. As a base model it has no built-in refusal behavior or safety tuning — outputs should be filtered/evaluated before use in any user-facing setting. At ~107M parameters, factual accuracy and reasoning ability are limited compared to larger models. |
|
|
| ## How to Get Started |
|
|
| ```python |
| from infer import load_model, load_tokenizer, run |
| |
| model = load_model("QED-Base-v2.pt") |
| tokenizer = load_tokenizer("tok.model") |
| |
| text = run("Once upon a time", model, tokenizer, max_new_tokens=100) |
| print(text) |
| ``` |
|
|
| ## License |
|
|
| MIT License |
