File size: 4,964 Bytes
76dbbc1 a44827c 76dbbc1 9cb7f4c 76dbbc1 9cb7f4c 76dbbc1 a5139f7 76dbbc1 9cb7f4c 76dbbc1 861cb6a 76dbbc1 5eca61f 76dbbc1 a5139f7 76dbbc1 a44827c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 | ---
language:
- en
license: mit
tags:
- fleck
- causal-language-model
- custom-architecture
- research
datasets:
- HuggingFaceFW/fineweb
- HuggingFaceFW/fineweb-edu
---
# [Fleck-S-100K-Base](https://huggingface.co/ML-is-Fun/Fleck-S-100K-Base)
**[Fleck-S-100K-Base](https://huggingface.co/ML-is-Fun/Fleck-S-100K-Base)** — a compact decoder-only language model trained from scratch on 100M real tokenizer tokens.
- **109,384 parameters**
- **BF16 SafeTensors weights**
- Factorized tied embedding with a rank-32 embedding projection
- Designed as a small research model for local inference on Apple Silicon
## Model Details
| | |
| --- | --- |
| Architecture | Decoder-only Transformer |
| Parameters | 109,384 |
| Hidden size | 64 |
| FFN size | 128 |
| Physical blocks | 2 |
| Effective depth | 4 (`A → B → A → B`) |
| Attention | GQA — 4 query heads, 2 KV heads, head dimension 16 |
| Normalization | RMSNorm |
| Embedding | Factorized tied embedding, rank 32 |
| Vocabulary | 1,024 |
| Context length | 2,048 tokens |
| Canonical dtype | BF16 |
## Training
| | |
| --- | --- |
| Initialization | Fresh initialization |
| Dataset | FineWeb-Edu / FineWeb |
| Data mixture | 70% FineWeb-Edu + 30% FineWeb |
| Real tokenizer tokens | 100,000,000 exactly |
| Instruction tuning | No |
| Hardware | Apple M2 (10-core GPU) |
This is the immutable Base-100M parent used to produce the independent [`Fleck-S-100K`](https://huggingface.co/ML-is-Fun/Fleck-S-100K) instruction-tuned model.
## Benchmark Results
The Base-100M checkpoint was evaluated with the corrected zero-shot aggregation protocol. This minimal public bundle does not include the evaluation artifact; the table below is a reference result for the released Base model.
Evaluation conditions: zero-shot, no chat template, FP32 evaluation, Apple Silicon MPS.
| Task | Metric | Shots | Base |
| --- | --- | ---: | ---: |
| HellaSwag | `acc_norm` | 0 | 26.28% |
| PIQA | `acc_norm` | 0 | 51.14% |
| ARC-Easy | `acc_norm` | 0 | 26.05% |
| ARC-Challenge | `acc_norm` | 0 | 26.19% |
| LAMBADA OpenAI | `acc` | 0 | 0.02% |
| WinoGrande | `acc` | 0 | 49.41% |
| BoolQ | `acc` | 0 | 37.83% |
| MMLU (57-subject macro) | `acc` | 0 | 23.09% |
| **Eight-task mean** | — | 0 | **30.00%** |
## Tokenizer
- `Fleck-Tokenizer-1024`
- Byte-level BPE
- Vocabulary size: 1,024
### Special Tokens
| Token | ID | Role |
| --- | ---: | --- |
| `<bos>` | 0 | sequence start |
| `<eos>` | 1 | sequence end |
| `<pad>` | 2 | padding |
| `<unk>` | 3 | unknown token |
| `<\|system\|>` | 4 | system turn |
| `<\|user\|>` | 5 | user turn |
| `<\|assistant\|>` | 6 | assistant turn |
| `<\|eot\|>` | 7 | end of turn |
## Usage
The bundle includes a self-contained `inference.py`; it does not import the Fleck-LM checkout. The accompanying `config.json`, `generation_config.json`, and `tokenizer_config.json` describe the custom architecture and generation/tokenizer defaults; standard `transformers.AutoModel` loading is not supported. Install the three runtime dependencies and run a single greedy continuation:
```bash
python -m pip install torch safetensors tokenizers
python inference.py \
--ckpt model.safetensors \
--tokenizer tokenizer.json \
--prompt "Hello, world" \
--max-tokens 32 \
--device cpu
```
The script reads and runs the BF16 checkpoint without an FP32 model copy, validates every SafeTensors key, shape, and dtype, and uses FP32 only for attention score/softmax and tied-logit accumulation. The model implements its factorized tied embedding/logits, effective-depth execution `A → B → A → B`, half-split RoPE, GQA, physical KV caches, RMSNorms, and greedy generation without any repository-local imports.
The default filenames are `model.safetensors` and `tokenizer.json`, so the shorter command also works:
```bash
python inference.py --prompt "Hello" --max-tokens 16 --device cpu
```
For tokenizer-only use:
```python
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")
print(tokenizer.encode("Hello, world!", add_special_tokens=False).ids)
```
## Limitations
This model is extremely small and is intended for research and local experimentation rather than general-purpose language generation. It may produce repetitions, malformed text, weak factual answers, or incoherent continuations. Benchmark scores should be interpreted in the context of the 109K parameter count.
## License
MIT License.
## Files
The public bundle contains these files:
- `README.md` — model card and usage documentation
- `inference.py` — standalone strict loader and greedy inference CLI
- `model.safetensors` — BF16 model weights
- `tokenizer.json` — standalone tokenizer
- `config.json` — custom architecture configuration
- `generation_config.json` — greedy generation defaults
- `tokenizer_config.json` — tokenizer defaults and special-token mapping
No training data, optimizer state, or other training outputs are included. |