File size: 3,590 Bytes
2923e16
 
 
 
 
 
 
14c7810
2923e16
 
14c7810
 
2923e16
 
 
83abb1e
14c7810
860706e
14c7810
 
 
be30ecf
14c7810
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
---
license: mit
datasets:
- openbmb/Ultra-FineWeb-L3
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- deepseek_v41
- text-generation
- custom_code
---

# Penny Alpha

A 59M-parameter base language model using the DeepSeek V4.1 Flash architecture, trained from scratch by [LocalWisdom](https://huggingface.co/LocalWisdom).

> We believe the future of LLM is SLM and local-first, and that AI will be an integral part of everyday life.
> That needs a local-friendly suite. We are LocalWisdom, a suite of next-gen, local-friendly AI devkits and userkits.
> We believe the future relies on diversified, commoditized LLMs, and LocalWisdom is here to push that narrative.

## Model details

| Key | Details |
|---|---|
| Parameters | 59M |
| Architecture | DeepSeek V4.1 Flash (MoE, sparse attention, Engram, hyper-connections) |
| Training data | 1B tokens of [Ultra-FineWeb-L3](https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L3) (English) |
| Training compute | 1x RTX 5060 Ti, 1 day 12 hours, about $5.64 |
| Context window | 8,192 tokens (trained length) |
| Type | Base model: no chat template, no instruction tuning |
| License | MIT |

The architecture supports a native context of 1M tokens, but Penny Alpha was only trained on sequences up to 8,192 tokens. Do not expect reliable behavior beyond that.

## Other formats

- GGUF: [LocalWisdom/Penny-Alpha-GGUF](https://huggingface.co/LocalWisdom/Penny-Alpha-GGUF). Works directly with the latest llama.cpp.

## How to use

Penny Alpha is a **base model**: it continues text and does not follow instructions. For the safetensors weights, use Transformers with `trust_remote_code=True` (the modeling code ships in this repo as `model.py`).

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "LocalWisdom/Penny-Alpha"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval()

# Training documents were separated by EOS, so start the prompt with it.
prompt = tok.eos_token + "The history of the bicycle begins"
inputs = tok(prompt, add_special_tokens=False, return_tensors="pt")

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=200,
        do_sample=True,
        temperature=0.8,
        top_k=50,
        top_p=0.95,
        repetition_penalty=1.1,
        use_cache=False,  # required: this implementation has no KV cache
    )
print(tok.decode(out[0], skip_special_tokens=True))
```

### Notes

- **Prepend EOS.** The tokenizer does not add it automatically. Without it the model starts mid-document and quality drops.
- **`use_cache=False` is required.** There is no KV cache, so every new token re-runs the whole sequence. This is fine at 59M parameters.
- **Batching.** The attention mask is ignored, so use batch size 1 or equal-length prompts without padding.
- **Precision.** `float32` is the safest choice. `bfloat16` also loads, but results may differ slightly.
- **GPU is optional.** The model is small enough to run on CPU. Add `.to("cuda")` to the model and inputs if you have a GPU.

## Limitations

- A 59M-parameter model trained on 1B tokens: expect fluent-looking but frequently repetitive or factually wrong text.
- English only, base model only, and no safety tuning.
- Multi-token prediction (DSpark/MTP) and vision components of the full architecture are not included.

## Training

Pretrained from scratch on 1B tokens of Ultra-FineWeb-L3 (English), packed into 8,192-token windows, on a single RTX 5060 Ti.