README.md

---
language:
- en
license: mit
tags:
- fleck
- causal-language-model
- custom-architecture
- research
datasets:
- HuggingFaceFW/fineweb
- HuggingFaceFW/fineweb-edu
---

# [Fleck-S-100K-Base](https://huggingface.co/ML-is-Fun/Fleck-S-100K-Base)

**[Fleck-S-100K-Base](https://huggingface.co/ML-is-Fun/Fleck-S-100K-Base)** — a compact decoder-only language model trained from scratch on 100M real tokenizer tokens.

- **109,384 parameters**
- **BF16 SafeTensors weights**
- Factorized tied embedding with a rank-32 embedding projection
- Designed as a small research model for local inference on Apple Silicon

## Model Details

|  |  |
| --- | --- |
| Architecture | Decoder-only Transformer |
| Parameters | 109,384 |
| Hidden size | 64 |
| FFN size | 128 |
| Physical blocks | 2 |
| Effective depth | 4 (`A → B → A → B`) |
| Attention | GQA — 4 query heads, 2 KV heads, head dimension 16 |
| Normalization | RMSNorm |
| Embedding | Factorized tied embedding, rank 32 |
| Vocabulary | 1,024 |
| Context length | 2,048 tokens |
| Canonical dtype | BF16 |

## Training

|  |  |
| --- | --- |
| Initialization | Fresh initialization |
| Dataset | FineWeb-Edu / FineWeb |
| Data mixture | 70% FineWeb-Edu + 30% FineWeb |
| Real tokenizer tokens | 100,000,000 exactly |
| Instruction tuning | No |
| Hardware | Apple M2 (10-core GPU) |

This is the immutable Base-100M parent used to produce the independent [`Fleck-S-100K`](https://huggingface.co/ML-is-Fun/Fleck-S-100K) instruction-tuned model.

## Benchmark Results

The Base-100M checkpoint was evaluated with the corrected zero-shot aggregation protocol. This minimal public bundle does not include the evaluation artifact; the table below is a reference result for the released Base model.

Evaluation conditions: zero-shot, no chat template, FP32 evaluation, Apple Silicon MPS.

| Task | Metric | Shots | Base |
| --- | --- | ---: | ---: |
| HellaSwag | `acc_norm` | 0 | 26.28% |
| PIQA | `acc_norm` | 0 | 51.14% |
| ARC-Easy | `acc_norm` | 0 | 26.05% |
| ARC-Challenge | `acc_norm` | 0 | 26.19% |
| LAMBADA OpenAI | `acc` | 0 | 0.02% |
| WinoGrande | `acc` | 0 | 49.41% |
| BoolQ | `acc` | 0 | 37.83% |
| MMLU (57-subject macro) | `acc` | 0 | 23.09% |
| **Eight-task mean** | — | 0 | **30.00%** |

## Tokenizer

- `Fleck-Tokenizer-1024`
- Byte-level BPE
- Vocabulary size: 1,024

### Special Tokens

| Token | ID | Role |
| --- | ---: | --- |
| `<bos>` | 0 | sequence start |
| `<eos>` | 1 | sequence end |
| `<pad>` | 2 | padding |
| `<unk>` | 3 | unknown token |
| `<\|system\|>` | 4 | system turn |
| `<\|user\|>` | 5 | user turn |
| `<\|assistant\|>` | 6 | assistant turn |
| `<\|eot\|>` | 7 | end of turn |

## Usage

The bundle includes a self-contained `inference.py`; it does not import the Fleck-LM checkout. The accompanying `config.json`, `generation_config.json`, and `tokenizer_config.json` describe the custom architecture and generation/tokenizer defaults; standard `transformers.AutoModel` loading is not supported. Install the three runtime dependencies and run a single greedy continuation:

```bash
python -m pip install torch safetensors tokenizers
python inference.py \
  --ckpt model.safetensors \
  --tokenizer tokenizer.json \
  --prompt "Hello, world" \
  --max-tokens 32 \
  --device cpu
```

The script reads and runs the BF16 checkpoint without an FP32 model copy, validates every SafeTensors key, shape, and dtype, and uses FP32 only for attention score/softmax and tied-logit accumulation. The model implements its factorized tied embedding/logits, effective-depth execution `A → B → A → B`, half-split RoPE, GQA, physical KV caches, RMSNorms, and greedy generation without any repository-local imports.

The default filenames are `model.safetensors` and `tokenizer.json`, so the shorter command also works:

```bash
python inference.py --prompt "Hello" --max-tokens 16 --device cpu
```

For tokenizer-only use:

```python
from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")
print(tokenizer.encode("Hello, world!", add_special_tokens=False).ids)
```

## Limitations

This model is extremely small and is intended for research and local experimentation rather than general-purpose language generation. It may produce repetitions, malformed text, weak factual answers, or incoherent continuations. Benchmark scores should be interpreted in the context of the 109K parameter count.

## License

MIT License.

## Files

The public bundle contains these files:

- `README.md` — model card and usage documentation
- `inference.py` — standalone strict loader and greedy inference CLI
- `model.safetensors` — BF16 model weights
- `tokenizer.json` — standalone tokenizer
- `config.json` — custom architecture configuration
- `generation_config.json` — greedy generation defaults
- `tokenizer_config.json` — tokenizer defaults and special-token mapping
No training data, optimizer state, or other training outputs are included.