--- license: apache-2.0 datasets: - Ricky1509/finephrase - nampdn-ai/tiny-strange-textbooks - noanabeshima/TinyStoriesV2 - Pageshift-Entertainment/LongPage language: - en tags: - fromzero - synthetic - qwen3 - tiny - tlm - slm - harley-ml - lyjohnny --- # **Syn-2.6M** ## Summary ```text Task: Text-Generation Total training time: 1.8 hours Inputs: text Outputs: text Params: 2,604,210 Final Loss: 2.37 Important Benchmark Scores: 1. ARC Easy - 32.11% 2. BLiMP - 65.33% 3. HellaSwag - 27.03% Framework: PyTorch, transformers Authors: Paul Courneya, Jonathan LY ``` ## Description **Syn** is a **Tiny Language Model (TLM)** trained on 2.7 billion tokens of **synthetic data**. The name **Syn** is an abbreviation of **synthetic**, reflecting the type of data the model was trained on. ## Model Details - Architecture: Qwen3.5 - Hidden Size: 140 - Number of Layers: 9 - Intermediate Size: 392 (a 2.8x expansion) - Number of Attention Heads: 4 - Number of KV Heads: 1 - Head Dim: 35 - Vocab Size: 3584 - Max Position Embeddings: 640 - Total Parameters: 2,604,210 ## Training ### Training Details - Maximum Learning Rate: 3e-3 - Minimum Learning Rate: 0 - Number of Epochs: 1 - Sequence Length: 256 (yes, we accidentally trained at 256 instead of 640) - Batch Size: 428 - Eval Split Ratio: 0.006 - Gradient Accumulation Steps: 2 - Gradient Checkpointing: True - Gradient Clipping: 1.0 - Torch Compile: True - Torch Compile Mode: `max-autotune-no-cudagraphs` - AdamW Betas: `(0.9, 0.95)` - WSD Warmup Ratio: 0.02 - WSD Stable Ratio: 0.78 - WSD Decay Ratio: 0.20 - DType: `float16` ### Dataset | Dataset | Bytes | Size | Share | | ---------------------- | ------------: | -------: | -----: | | FinePhrase | 8,000,000,000 | 8.000 GB | 61.61% | | Tiny-Strange-Textbooks | 4,000,000,000 | 4.000 GB | 30.81% | | TinyStoriesv2 | 700,000,000 | 0.700 GB | 5.39% | | LongPage | 284,000,000 | 0.284 GB | 2.19% | Note: the byte counts are rounded. ### Final Eval and Train Loss - Train: 2.37 - Val: 2.357 ### Hardware - GPU: NVIDIA RTX 2060 (used for training) - CPU: AMD Ryzen 5 2600 (used for tokenization) ## Benchmarks | Task | Value | |---------------|------:| | BLiMP | 65.33% | | ARC Easy | 32.11% | | ARC Challenge | 20.39% | | HellaSwag | 27.03% | | SWAG | 33.38% | | PiQA | 53.48% | ArithMark-2.0: | Ops = 1 | Ops = 2 | Ops = 3 | Avg | |---------|---------|---------|------:| | 25.04% | 30.13% | 24.60% | 26.48% | For a comparison with other small language models like this one, go [here](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard). ## Generation Sample ```text Prompt : 'Artificial intelligence is' ------------------------------------------------------------ Generated: a form of artificial intelligence that involves creating an attractive and reliable source of information. This involves using advanced technology to create interactive, user-friendly platforms for users who are looking to use them in their daily lives. ## II. Why Use Applications? To enhance the user experience, it's important to have the opportunity to learn how to write and understand your content properly. To ensure that you are using new tools, take some time to read, then let go of yourself or others through conversations with the user as they look at them. Additionally, by practicing self-assessment, users can gain more control over their own ideas. ### IV. Conclusion In this lesson, we learned about the different types of apps used for frames, their importance, and its applications. We also discussed the practical aspects of the application and practical applications of Frameworks. By understanding these concepts, readers can apply these skills to various scenarios in their careers. ``` ## Use Cases 1. Educational work and research 2. Fine-tuning for downstream use 3. Deployment on edge devices 4. Or just for fun. ## Limitations 1. Cannot chat, reason, code, or answer questions 2. Almost always unfactual 3. No long-context handling ## License Before using, distributing, selling, or modifying this software, you must read the license [here](https://huggingface.co/fromziro/Syn-2.6M/blob/main/LICENSE.txt). ## Inference ```python #!/usr/bin/env python3 MODEL_DIR = "fromziro/Syn-2.6M" TOKENIZER_PATH = MODEL_DIR PROMPT = "Artificial intelligence is" MAX_NEW_TOKENS = 256 TEMPERATURE = 0.7 TOP_P = 0.95 TOP_K = 30 REPETITION_PENALTY = 1.2 DO_SAMPLE = True import torch from pathlib import Path from transformers import AutoModelForCausalLM, AutoTokenizer, PreTrainedTokenizerFast device = ( "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu" ) print(f"Device : {device}") def load_tokenizer(path_or_repo: str): p = Path(path_or_repo) if p.exists() and p.is_file() and p.suffix.lower() == ".json": tok = PreTrainedTokenizerFast(tokenizer_file=str(p.resolve())) else: tok = AutoTokenizer.from_pretrained(path_or_repo, use_fast=True) if tok.bos_token is None: tok.add_special_tokens({"bos_token": "<|bos|>"}) if tok.eos_token is None: tok.add_special_tokens({"eos_token": "<|eos|>"}) if tok.unk_token is None: tok.add_special_tokens({"unk_token": "<|unk|>"}) if tok.pad_token is None: tok.pad_token = tok.eos_token if tok.eos_token is not None else "<|pad|>" tok.padding_side = "left" return tok print("Loading tokenizer...") tokenizer = load_tokenizer(TOKENIZER_PATH) print(f" Vocab size : {len(tokenizer)}") print(f" BOS : {tokenizer.bos_token!r}") print(f" EOS : {tokenizer.eos_token!r}") print(f" PAD : {tokenizer.pad_token!r} (id={tokenizer.pad_token_id})") print(f"\nLoading model from {MODEL_DIR} ...") model = AutoModelForCausalLM.from_pretrained( MODEL_DIR, torch_dtype=torch.float16 if device == "cuda" else torch.float32, low_cpu_mem_usage=True, ) model.eval() model.to(device) model.config.use_cache = False if hasattr(model, "generation_config") and model.generation_config is not None: model.generation_config.use_cache = False total_params = sum(p.numel() for p in model.parameters()) print(f" Parameters : {total_params:,}") def generate( prompt: str = PROMPT, max_new_tokens: int = MAX_NEW_TOKENS, temperature: float = TEMPERATURE, top_p: float = TOP_P, top_k: int = TOP_K, repetition_penalty: float = REPETITION_PENALTY, do_sample: bool = DO_SAMPLE, ) -> str: bos = tokenizer.bos_token or "" full_prompt = bos + prompt inputs = tokenizer( full_prompt, return_tensors="pt", add_special_tokens=False, ).to(device) inputs.pop("token_type_ids", None) gen_kwargs = dict( max_new_tokens=max_new_tokens, do_sample=do_sample, repetition_penalty=repetition_penalty, eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id, use_cache=False, ) if do_sample: gen_kwargs["temperature"] = temperature gen_kwargs["top_p"] = top_p gen_kwargs["top_k"] = top_k with torch.inference_mode(): output_ids = model.generate(**inputs, **gen_kwargs) prompt_len = inputs["input_ids"].shape[-1] new_ids = output_ids[0][prompt_len:] return tokenizer.decode(new_ids, skip_special_tokens=True) if __name__ == "__main__": print(f"\nPrompt : {PROMPT!r}") print("-" * 60) output = generate(PROMPT) print("Generated:") print(output) ``` ## Copyright ``` Copyright (c) 2026 FromZero Copyright (c) 2026 Paul Courneya Copyright (c) 2026 Jonathan LY ``` ## Citation ```bibtex @misc{syn2.6m, title = {Syn-2.6M: A Tiny Languagte Model trained on 2.7B tokens of Synthetic Data}, organization = [FromZero], authors = {Paul Courneya, Jonathon LY}, year = {2026}, url = {https://huggingface.co/fromziro/Syn-2.6M} } ```