| --- |
| license: mit |
| language: |
| - en |
| tags: |
| - QED |
| - QED-Base-v1 |
| - language-model |
| - text-generation |
| - causal-lm |
| - transformer |
| - decoder-only |
| - pytorch |
| - sentencepiece |
| - swiglu |
| - rope |
| - rmsnorm |
| - 300M-parameters |
| - foundation-model |
| - research |
| --- |
| |
| # QED-Base-v1-300M |
|
|
| ## Model Description |
|
|
| QED-Base-v1-300M is a decoder-only Transformer language model trained entirely from scratch. |
|
|
| Unlike fine-tuned or continually pretrained models, QED-Base-v1 was not initialized from or adapted from any existing pretrained model. All model weights were learned through original autoregressive pretraining using an independently created dataset. |
|
|
| The model serves as the foundation for future instruction-tuned and task-specific QED models while remaining suitable for language modeling research and experimentation. |
|
|
| --- |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| |---|---| |
| | Model Name | QED-Base-v1-300M | |
| | Model Type | Decoder-only Transformer | |
| | Parameters | 299.82M | |
| | Hidden Dimension | 1024 | |
| | Layers | 12 | |
| | Attention Heads | 16 | |
| | Activation Function | SwiGLU | |
| | Normalization | RMSNorm | |
| | Position Encoding | RoPE | |
| | Vocabulary Size | 96,000 | |
| | Tokenizer | SentencePiece | |
| | Training Method | Pretraining from Scratch | |
| | Base Model | None | |
|
|
| --- |
|
|
| ## Architecture |
|
|
| QED-Base-v1 retains a modern decoder-only Transformer architecture featuring: |
|
|
| - Rotary Positional Embeddings (RoPE) |
| - RMSNorm |
| - SwiGLU feed-forward networks |
| - Multi-head causal self-attention |
| - Weight-tied input and output embeddings |
|
|
| The architecture is designed as a general-purpose language foundation model suitable for pretraining, research, and downstream fine-tuning. |
|
|
| --- |
|
|
| ## Training Data |
|
|
| QED-Base-v1 was pretrained using a fully original dataset created specifically for this project. |
|
|
| The training corpus was independently authored and prepared for large-scale autoregressive language modeling. |
|
|
| The data preparation process included: |
|
|
| - Original content creation |
| - Custom formatting |
| - Custom preprocessing |
| - SentencePiece tokenization |
| - Dataset preparation for autoregressive pretraining |
|
|
| No existing pretrained model weights were used during training. |
|
|
| --- |
|
|
| ## Intended Use |
|
|
| QED-Base-v1 is intended for: |
|
|
| - Language model research |
| - Transformer architecture experimentation |
| - Fine-tuning into instruction-following models |
| - Domain adaptation |
| - Educational projects |
| - Creating derivative language models |
|
|
| This is a **foundation model** and has **not** been instruction-tuned. |
|
|
| --- |
|
|
| ## Limitations |
|
|
| As a pretrained foundation model: |
|
|
| - It is not optimized for instruction following. |
| - It may generate incomplete or unexpected responses. |
| - It has not been aligned with human preferences. |
| - It may require instruction tuning for assistant-style applications. |
| - Outputs should be reviewed before use in production or high-risk scenarios. |
|
|
| --- |
|
|
| ## License |
|
|
| MIT License. |
