--- license: mit datasets: - openbmb/Ultra-FineWeb-L3 language: - en pipeline_tag: text-generation library_name: transformers tags: - deepseek_v41 - text-generation - custom_code --- # Penny Alpha A 59M-parameter base language model using the DeepSeek V4.1 Flash architecture, trained from scratch by [LocalWisdom](https://huggingface.co/LocalWisdom). > We believe the future of LLM is SLM and local-first, and that AI will be an integral part of everyday life. > That needs a local-friendly suite. We are LocalWisdom, a suite of next-gen, local-friendly AI devkits and userkits. > We believe the future relies on diversified, commoditized LLMs, and LocalWisdom is here to push that narrative. ## Model details | Key | Details | |---|---| | Parameters | 59M | | Architecture | DeepSeek V4.1 Flash (MoE, sparse attention, Engram, hyper-connections) | | Training data | 1B tokens of [Ultra-FineWeb-L3](https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L3) (English) | | Training compute | 1x RTX 5060 Ti, 1 day 12 hours, about $5.64 | | Context window | 8,192 tokens (trained length) | | Type | Base model: no chat template, no instruction tuning | | License | MIT | The architecture supports a native context of 1M tokens, but Penny Alpha was only trained on sequences up to 8,192 tokens. Do not expect reliable behavior beyond that. ## Other formats - GGUF: [LocalWisdom/Penny-Alpha-GGUF](https://huggingface.co/LocalWisdom/Penny-Alpha-GGUF). Works directly with the latest llama.cpp. ## How to use Penny Alpha is a **base model**: it continues text and does not follow instructions. For the safetensors weights, use Transformers with `trust_remote_code=True` (the modeling code ships in this repo as `model.py`). ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer repo = "LocalWisdom/Penny-Alpha" tok = AutoTokenizer.from_pretrained(repo) model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval() # Training documents were separated by EOS, so start the prompt with it. prompt = tok.eos_token + "The history of the bicycle begins" inputs = tok(prompt, add_special_tokens=False, return_tensors="pt") with torch.no_grad(): out = model.generate( **inputs, max_new_tokens=200, do_sample=True, temperature=0.8, top_k=50, top_p=0.95, repetition_penalty=1.1, use_cache=False, # required: this implementation has no KV cache ) print(tok.decode(out[0], skip_special_tokens=True)) ``` ### Notes - **Prepend EOS.** The tokenizer does not add it automatically. Without it the model starts mid-document and quality drops. - **`use_cache=False` is required.** There is no KV cache, so every new token re-runs the whole sequence. This is fine at 59M parameters. - **Batching.** The attention mask is ignored, so use batch size 1 or equal-length prompts without padding. - **Precision.** `float32` is the safest choice. `bfloat16` also loads, but results may differ slightly. - **GPU is optional.** The model is small enough to run on CPU. Add `.to("cuda")` to the model and inputs if you have a GPU. ## Limitations - A 59M-parameter model trained on 1B tokens: expect fluent-looking but frequently repetitive or factually wrong text. - English only, base model only, and no safety tuning. - Multi-token prediction (DSpark/MTP) and vision components of the full architecture are not included. ## Training Pretrained from scratch on 1B tokens of Ultra-FineWeb-L3 (English), packed into 8,192-token windows, on a single RTX 5060 Ti.