--- license: apache-2.0 language: - en tags: - qwen3 - gast - causal-lm - base-model - text-generation - gguf - small-language-model library_name: transformers pipeline_tag: text-generation datasets: - HuggingFaceFW/fineweb-edu - Salesforce/wikitext ---

Gast-Mini

Parameters Context Tokens

Base causal language model with 7.07M parameters, trained from scratch.

--- ## Architecture | Field | Value | |---|---| | Parameters | 7.07M | | Layers | 16 | | Hidden size | 192 | | Attention | GQA (3 Query / 1 KV head, head_dim 64) | | Intermediate size | 512 SwiGLU | | Norm | RMSNorm (eps 1e-6) + per-head QK-Norm | | RoPE theta | 10000.0 | | Context length | 512 | | Vocab size | 4000 | | Tokenizer | Custom BPE (byte-level) | | Embeddings | Tied (`lm_head` == `embed_tokens`) | | HF Layout | `Qwen3ForCausalLM` | ## Training & Corpus | Parameter | Value | |---|---| | Optimizer | AdamW (beta1=0.9, beta2=0.95, wd=0.1) | | LR | 2e-3 (flat, 500-step warmup) | | Batch | 196,608 tokens/step | | Precision | fp16 | | Tokens seen | ~9B | ### Pretraining Data | Source | Tokens | License | |---|---:|---| | FineWeb-Edu (sample/10BT) | ~8.85B | ODC-By 1.0 | | WikiText-103-raw-v1 | ~0.17B | CC BY-SA 4.0 | | **Total** | **~9B** | | ## Samples Generated with `llama.cpp`, `temperature=0.85, top_p=0.92, top_k=50, repeat_penalty=1.15`. > **Prompt:** Artificial intelligence is > > Artificial intelligence is the most efficient, and it is difficult to obtain data from human beings. It has no clearly defined or explicit identity, but only a firm assumption of a common law. The Digital Literature Society, Digital Literature Society, ABA, and Reference Library Society have been compiled to address the data collection needs of the digital literacy system. They provide advanced technical assistance in a variety of applications, including on-line communication with computers, and are now referred to as Dig > **Prompt:** Photosynthesis is the process by which > > Photosynthesis is the process by which DNA can transfer a protein to a cell with the same function of its function. The most common function of a cell is to transfer it from one cell to another, and then to use it for a given cell or genome. The key step in creating a healthy organism is to create a bridge between living organisms that can survive. This process involves the formation of new cells in which the cells are made up of new cells. The cell contains only two cells, each one being formed into two sections, with an additional two cells having a different cell shape > **Prompt:** The theory of relativity states that > > The theory of relativity states that the stars are all uniform in shape. For example, the star can be measured by a gravitational force, which is a small amount of mass that can be measured in terms of grams per kilogram. Similarly, the star is used to calculate its mass: This theory states that the grams of the stars are equal to 1,000,000,000 and their masses are equal to 2,500,000,000 and the masses are equal to 12,000,000. ## Usage ### llama.cpp ```bash ./llama-cli -m gast-7m-f16.gguf -p "The history of the Roman Empire" -n 128 --temp 0.85 --top-k 50 --top-p 0.92 --repeat-penalty 1.15 ``` ### llama-cpp-python ```python from llama_cpp import Llama llm = Llama(model_path="gast-7m-f16.gguf", n_ctx=512) out = llm( "The history of the Roman Empire", max_tokens=120, temperature=0.85, top_p=0.92, top_k=50, repeat_penalty=1.15, ) print(out["choices"][0]["text"]) ``` ## Limitations - English only. - Base model, not instruction tuned, no chat template. - Trained on ~9B tokens; not factual, barely coherent beyond a few words. ## License Apache 2.0. ## Credits - FineWeb-Edu - Hugging Face (ODC-By 1.0) - WikiText-103 - Salesforce (CC BY-SA 4.0) - llama.cpp - GGUF tooling - Hugging Face ## Citation ```bibtex @misc{gastmini, title = {Gast-Mini}, author = {Meridian-MRM}, howpublished = {https://huggingface.co/Meridian-MRM/Gast-Mini} } ```