hanzceo commited on
Commit
14c7810
·
verified ·
1 Parent(s): 10877c1

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +72 -5
README.md CHANGED
@@ -5,16 +5,83 @@ datasets:
5
  language:
6
  - en
7
  pipeline_tag: text-generation
 
8
  tags:
9
  - deepseek_v41
 
 
10
  ---
11
 
12
  # Penny Alpha
13
- A 59M parameters model with Deepseek V4.1 Flash architechture
14
 
15
- Other formats:
16
- - GGUF: https://huggingface.co/LocalWisdom/Penny-Alpha-GGUF
17
 
18
- Trained on 1B tokens of Ultra-FineWeb-L3 over 1 day 12 hours using 1x RTX 5060 Ti costing us $5.64.
 
 
19
 
20
- Official context window is 8192 tokens. However, the model's native context window is 1M tokens. Tokens above 8192 has never been trained.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5
  language:
6
  - en
7
  pipeline_tag: text-generation
8
+ library_name: transformers
9
  tags:
10
  - deepseek_v41
11
+ - text-generation
12
+ - custom_code
13
  ---
14
 
15
  # Penny Alpha
 
16
 
17
+ A 59M-parameter base language model using the DeepSeek V4.1 Flash architecture, trained from scratch by [LocalWisdom](https://huggingface.co/LocalWisdom).
 
18
 
19
+ > We believe the future of LLM is SLM and local-first, and that AI will be an integral part of everyday life.
20
+ > That needs a local-friendly suite. We are LocalWisdom, a suite of next-gen, local-friendly AI devkits and userkits.
21
+ > We believe the future relies on diversified, commoditized LLMs, and LocalWisdom is here to push that narrative.
22
 
23
+ ## Model details
24
+
25
+ | Key | Details |
26
+ |---|---|
27
+ | Parameters | 59M |
28
+ | Architecture | DeepSeek V4.1 Flash (MoE, sparse attention, Engram, hyper-connections) |
29
+ | Training data | 1B tokens of [Ultra-FineWeb-L3](https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L3) (English) |
30
+ | Training compute | 1x RTX 5060 Ti, 1 day 12 hours, about $5.64 |
31
+ | Context window | 8,192 tokens (trained length) |
32
+ | Type | Base model: no chat template, no instruction tuning |
33
+ | License | MIT |
34
+
35
+ The architecture supports a native context of 1M tokens, but Penny Alpha was only trained on sequences up to 8,192 tokens. Do not expect reliable behavior beyond that.
36
+
37
+ ## Other formats
38
+
39
+ - GGUF: [LocalWisdom/Penny-Alpha-GGUF](https://huggingface.co/LocalWisdom/Penny-Alpha-GGUF). Works directly with the latest llama.cpp.
40
+
41
+ ## How to use
42
+
43
+ Penny Alpha is a **base model**: it continues text and does not follow instructions. For the safetensors weights, use Transformers with `trust_remote_code=True` (the modeling code ships in this repo as `model.py`).
44
+
45
+ ```python
46
+ import torch
47
+ from transformers import AutoModelForCausalLM, AutoTokenizer
48
+
49
+ repo = "LocalWisdom/Penny-Alpha"
50
+ tok = AutoTokenizer.from_pretrained(repo)
51
+ model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.float32).eval()
52
+
53
+ # Training documents were separated by EOS, so start the prompt with it.
54
+ prompt = tok.eos_token + "The history of the bicycle begins"
55
+ inputs = tok(prompt, add_special_tokens=False, return_tensors="pt")
56
+
57
+ with torch.no_grad():
58
+ out = model.generate(
59
+ **inputs,
60
+ max_new_tokens=200,
61
+ do_sample=True,
62
+ temperature=0.8,
63
+ top_k=50,
64
+ top_p=0.95,
65
+ repetition_penalty=1.1,
66
+ use_cache=False, # required: this implementation has no KV cache
67
+ )
68
+ print(tok.decode(out[0], skip_special_tokens=True))
69
+ ```
70
+
71
+ ### Notes
72
+
73
+ - **Prepend EOS.** The tokenizer does not add it automatically. Without it the model starts mid-document and quality drops.
74
+ - **`use_cache=False` is required.** There is no KV cache, so every new token re-runs the whole sequence. This is fine at 59M parameters.
75
+ - **Batching.** The attention mask is ignored, so use batch size 1 or equal-length prompts without padding.
76
+ - **Precision.** `float32` is the safest choice. `bfloat16` also loads, but results may differ slightly.
77
+ - **GPU is optional.** The model is small enough to run on CPU. Add `.to("cuda")` to the model and inputs if you have a GPU.
78
+
79
+ ## Limitations
80
+
81
+ - A 59M-parameter model trained on 1B tokens: expect fluent-looking but frequently repetitive or factually wrong text.
82
+ - English only, base model only, and no safety tuning.
83
+ - Multi-token prediction (DSpark/MTP) and vision components of the full architecture are not included.
84
+
85
+ ## Training
86
+
87
+ Pretrained from scratch on 1B tokens of Ultra-FineWeb-L3 (English), packed into 8,192-token windows, on a single RTX 5060 Ti.