Penny-Alpha / README.md
hanzceo's picture
Update README.md
14c7810 verified
|
Raw History Blame Contribute Delete
3.59 kB
---
license: mit
datasets:
- openbmb/Ultra-FineWeb-L3
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- deepseek_v41
- text-generation
- custom_code
---
# Penny Alpha
A 59M-parameter base language model using the DeepSeek V4.1 Flash architecture, trained from scratch by [LocalWisdom](https://huggingface.co/LocalWisdom).
> We believe the future of LLM is SLM and local-first, and that AI will be an integral part of everyday life.
> That needs a local-friendly suite. We are LocalWisdom, a suite of next-gen, local-friendly AI devkits and userkits.
> We believe the future relies on diversified, commoditized LLMs, and LocalWisdom is here to push that narrative.
## Model details
| Key | Details |
|---|---|
| Parameters | 59M |
| Architecture | DeepSeek V4.1 Flash (MoE, sparse attention, Engram, hyper-connections) |
| Training data | 1B tokens of [Ultra-FineWeb-L3](https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L3) (English) |
| Training compute | 1x RTX 5060 Ti, 1 day 12 hours, about $5.64 |
| Context window | 8,192 tokens (trained length) |
| Type | Base model: no chat template, no instruction tuning |
| License | MIT |
The architecture supports a native context of 1M tokens, but Penny Alpha was only trained on sequences up to 8,192 tokens. Do not expect reliable behavior beyond that.
## Other formats
- GGUF: [LocalWisdom/Penny-Alpha-GGUF](https://huggingface.co/LocalWisdom/Penny-Alpha-GGUF). Works directly with the latest llama.cpp.
## How to use
Penny Alpha is a **base model**: it continues text and does not follow instructions. For the safetensors weights, use Transformers with `trust_remote_code=True` (the modeling code ships in this repo as `model.py`).
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "LocalWisdom/Penny-Alpha"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval()
# Training documents were separated by EOS, so start the prompt with it.
prompt = tok.eos_token + "The history of the bicycle begins"
inputs = tok(prompt, add_special_tokens=False, return_tensors="pt")
with torch.no_grad():
out = model.generate(
**inputs,
max_new_tokens=200,
do_sample=True,
temperature=0.8,
top_k=50,
top_p=0.95,
repetition_penalty=1.1,
use_cache=False, # required: this implementation has no KV cache
)
print(tok.decode(out[0], skip_special_tokens=True))
```
### Notes
- **Prepend EOS.** The tokenizer does not add it automatically. Without it the model starts mid-document and quality drops.
- **`use_cache=False` is required.** There is no KV cache, so every new token re-runs the whole sequence. This is fine at 59M parameters.
- **Batching.** The attention mask is ignored, so use batch size 1 or equal-length prompts without padding.
- **Precision.** `float32` is the safest choice. `bfloat16` also loads, but results may differ slightly.
- **GPU is optional.** The model is small enough to run on CPU. Add `.to("cuda")` to the model and inputs if you have a GPU.
## Limitations
- A 59M-parameter model trained on 1B tokens: expect fluent-looking but frequently repetitive or factually wrong text.
- English only, base model only, and no safety tuning.
- Multi-token prediction (DSpark/MTP) and vision components of the full architecture are not included.
## Training
Pretrained from scratch on 1B tokens of Ultra-FineWeb-L3 (English), packed into 8,192-token windows, on a single RTX 5060 Ti.