Text Generation
Transformers
English
Thai
faber-genesis
gqa
rope
swiglu
weight-sharing
cass
bf16
a100
conversational
custom_code
Instructions to use bbkdevops/FABER-Genesis-A100-Max with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use bbkdevops/FABER-Genesis-A100-Max with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="bbkdevops/FABER-Genesis-A100-Max", trust_remote_code=True) messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("bbkdevops/FABER-Genesis-A100-Max", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use bbkdevops/FABER-Genesis-A100-Max with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "bbkdevops/FABER-Genesis-A100-Max" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "bbkdevops/FABER-Genesis-A100-Max", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/bbkdevops/FABER-Genesis-A100-Max
- SGLang
How to use bbkdevops/FABER-Genesis-A100-Max with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "bbkdevops/FABER-Genesis-A100-Max" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "bbkdevops/FABER-Genesis-A100-Max", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "bbkdevops/FABER-Genesis-A100-Max" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "bbkdevops/FABER-Genesis-A100-Max", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use bbkdevops/FABER-Genesis-A100-Max with Docker Model Runner:
docker model run hf.co/bbkdevops/FABER-Genesis-A100-Max
FABER-Genesis-A100-Max 13.86B architecture: config + HF modeling + profiles + benchmark harness
Browse files- EVIDENCE_GENESIS_A100_MAX.md +92 -0
- README.md +59 -0
- __pycache__/modeling_faber_genesis.cpython-313.pyc +0 -0
- benchmark_genesis_a100.py +168 -0
- config.json +1 -0
- genesis_profiles.json +54 -0
- modeling_faber_genesis.py +137 -0
- tokenizer_config.json +1 -0
EVIDENCE_GENESIS_A100_MAX.md
ADDED
|
@@ -0,0 +1,92 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# FABER-Genesis A100-Max — Model + Ceiling Benchmark (v0.9.x)
|
| 2 |
+
|
| 3 |
+
## 1. What was built (real code, not formula-only)
|
| 4 |
+
|
| 5 |
+
- `faber/genesis.py` — decoder-only transformer exactly per spec:
|
| 6 |
+
hidden 6144, heads 48, KV 8, head-dim 128, SwiGLU FFN 16384,
|
| 7 |
+
vocab 32768, tied embeddings, RMSNorm, RoPE, GQA,
|
| 8 |
+
activation checkpointing, BF16, CASS state/value/halt heads.
|
| 9 |
+
Weight sharing via `share_map`: 34 physical -> 42 logical.
|
| 10 |
+
- `benchmarks/benchmark_genesis_a100.py` — real ceiling harness:
|
| 11 |
+
BF16 + AdamW8bit (bitsandbytes, fallback warns) + ckpt,
|
| 12 |
+
microbatch=1, ctx=512, fwd+bwd+optim step, `torch.cuda` peak,
|
| 13 |
+
per-config OOM catch. Supports `--sweep`, `--profile`, `--estimate-only`.
|
| 14 |
+
- `tests/test_genesis.py` — 7 tests: sharing, param landmarks,
|
| 15 |
+
boxed numbers, estimator-vs-reported, ordering, profiles, tiny CPU fwd+bwd.
|
| 16 |
+
- `benchmarks/results_genesis_a100_estimate.json` — estimator output (this machine).
|
| 17 |
+
|
| 18 |
+
## 2. Param accounting (exact counts from code)
|
| 19 |
+
|
| 20 |
+
`per_block = 390,082,560`; `emb = 201,326,592`; CASS = 6,303,746.
|
| 21 |
+
|
| 22 |
+
| n_phys | total (B) | label |
|
| 23 |
+
|---:|---:|:---|
|
| 24 |
+
| 25 | 9.9597 | ref-9.96B |
|
| 25 |
+
| 29 | 11.5200 | 11.52B |
|
| 26 |
+
| 32 | 12.6903 | 12.69B |
|
| 27 |
+
| 33 | 13.0804 | production-13.08B |
|
| 28 |
+
| 34 | 13.4704 | safe-13.47B |
|
| 29 |
+
| 35 | 13.8605 | **max-13.86B** (34 unique + 1 recurrent) |
|
| 30 |
+
| 36 | 14.2506 | **oom-14.25B** (backbone 14.2443B + CASS) |
|
| 31 |
+
| 37 | 14.6407 | 14.64B |
|
| 32 |
+
|
| 33 |
+
Boxed claims reproduced: `13.8605B passed-design`, `14.2443B backbone OOM-design`.
|
| 34 |
+
|
| 35 |
+
## 3. VRAM: estimator (local) vs required A100 measurement
|
| 36 |
+
|
| 37 |
+
Estimator = 6 bytes/param (BF16 w + BF16 g + 8-bit m+v) + ckpt activations
|
| 38 |
+
+ GQA KV transient + 0.45 GiB reserve. Local run (RTX 3090, estimate-only):
|
| 39 |
+
|
| 40 |
+
| n | params | estimate | reported A100 | delta |
|
| 41 |
+
|---:|---:|---:|---:|---:|
|
| 42 |
+
| 25 | 9.96B | 56.56 | 55.97 | +0.59 |
|
| 43 |
+
| 29 | 11.52B | 65.33 | 64.78 | +0.55 |
|
| 44 |
+
| 32 | 12.69B | 71.91 | 71.36 | +0.55 |
|
| 45 |
+
| 33 | 13.08B | 74.10 | 73.55 | +0.55 |
|
| 46 |
+
| 34 | 13.47B | 76.30 | 75.75 | +0.55 |
|
| 47 |
+
| 35 | 13.86B | 78.49 | 77.94 | +0.55 |
|
| 48 |
+
| 36 | 14.25B | 80.67 | ~78.72 pre-OOM | — |
|
| 49 |
+
| 37 | 14.64B | 82.86 | ~78.72 pre-OOM | — |
|
| 50 |
+
|
| 51 |
+
Ordering holds: 35 fits under Jobs-visible 79.25 GiB, 36 exceeds it.
|
| 52 |
+
The estimator is conservative by ~0.55 GiB; it is NOT a substitute for
|
| 53 |
+
the A100 run below. Reported 39.97 GiB / 7.05B row is a legacy smaller
|
| 54 |
+
config (same family at n=18 gives 7.23B with this vocab); all 9.9B+ rows
|
| 55 |
+
match exactly.
|
| 56 |
+
|
| 57 |
+
## 4. Locally verified (this machine, RTX 3090 24GB)
|
| 58 |
+
|
| 59 |
+
- `python -m pytest tests/test_genesis.py -q` — **7/7 pass**.
|
| 60 |
+
- `benchmark_genesis_a100.py --sweep --estimate-only` — table above, pass.
|
| 61 |
+
- Tiny real-CUDA path: hidden=256/n=2/ctx=512 BF16 fwd+bwd+AdamW step —
|
| 62 |
+
**pass, peak 0.035 GiB**. Proves the harness executes fwd+bwd+step on CUDA.
|
| 63 |
+
- Full 13B allocation NOT attempted here (needs ~78 GiB > 24 GiB); doing so
|
| 64 |
+
would OOM by design on this card.
|
| 65 |
+
|
| 66 |
+
## 5. Required A100 run (proves the ceiling, run on the A100 box)
|
| 67 |
+
|
| 68 |
+
```bash
|
| 69 |
+
pip install "torch==2.6.0" bitsandbytes
|
| 70 |
+
# single max profile, real measurement:
|
| 71 |
+
python benchmarks/benchmark_genesis_a100.py --profile max-13.86B --steps 1 \
|
| 72 |
+
--out benchmarks/results_genesis_max.json
|
| 73 |
+
# full ceiling sweep with real peaks + OOM capture:
|
| 74 |
+
python benchmarks/benchmark_genesis_a100.py --sweep \
|
| 75 |
+
--n-list 25,29,32,33,34,35,36,37 --steps 1 \
|
| 76 |
+
--out benchmarks/results_genesis_sweep.json
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
+
Pass criteria: each row prints `GENESIS_RESULT` with `measured_peak_gib`
|
| 80 |
+
and `status=pass`, except n=36,37 which must print `status=oom`.
|
| 81 |
+
Only `measured_peak_gib` from that run certifies 13.86B-pass / 14.25B-OOM.
|
| 82 |
+
|
| 83 |
+
## 6. Profiles (decision carried over)
|
| 84 |
+
|
| 85 |
+
- `production-13.08B` (n=33): ~73.5 GiB, ~5.7 GiB headroom — default for long runs.
|
| 86 |
+
- `safe-13.47B` (n=34): ~75.7 GiB, ~3.5 GiB headroom — squeeze with safety.
|
| 87 |
+
- `max-13.86B` (n=35): ~77.9 GiB — capacity experiments only, not default.
|
| 88 |
+
- `oom-14.25B` (n=36+): not a single-A100 resident AdamW8bit config.
|
| 89 |
+
|
| 90 |
+
Model size != intelligence: random-init step-pass says nothing about Thai /
|
| 91 |
+
code / reasoning quality; that needs curriculum + CASS + verifier + agent
|
| 92 |
+
training and SWE/APEX evals separately.
|
README.md
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language: [en, th]
|
| 3 |
+
library_name: transformers
|
| 4 |
+
license: apache-2.0
|
| 5 |
+
pipeline_tag: text-generation
|
| 6 |
+
tags: [faber-genesis, gqa, rope, swiglu, weight-sharing, cass, bf16, a100, th]
|
| 7 |
+
---
|
| 8 |
+
|
| 9 |
+
# FABER-Genesis-A100-Max (13.86B architecture)
|
| 10 |
+
|
| 11 |
+
Decoder-only transformer with weight sharing + CASS state/value/halt heads.
|
| 12 |
+
|
| 13 |
+
- hidden 6144, heads 48, KV 8, head-dim 128, SwiGLU FFN 16384
|
| 14 |
+
- vocab 32768, tied embeddings, RMSNorm, RoPE, GQA
|
| 15 |
+
- 35 physical blocks -> 42 logical transformations + 1 recurrent pass
|
| 16 |
+
- total params: **13.860526082B** (backbone + CASS)
|
| 17 |
+
- training recipe: BF16, AdamW8bit, activation checkpointing, microbatch=1, ctx=512
|
| 18 |
+
|
| 19 |
+
## Profiles
|
| 20 |
+
|
| 21 |
+
| profile | physical | params | status |
|
| 22 |
+
|---|---|---:|---|
|
| 23 |
+
| production-13.08B | 33 | 13.0804B | default for long runs (~73.55 GiB) |
|
| 24 |
+
| safe-13.47B | 34 | 13.4704B | squeeze with safety (~75.75 GiB) |
|
| 25 |
+
| max-13.86B | 35 | 13.8605B | capacity experiments (~77.94 GiB) |
|
| 26 |
+
| oom-14.25B | 36 | 14.2506B | OOM on single A100 AdamW8bit |
|
| 27 |
+
|
| 28 |
+
## What this repo contains
|
| 29 |
+
|
| 30 |
+
`config.json` (max profile), `modeling_faber_genesis.py` (self-contained HF
|
| 31 |
+
model code), `genesis_profiles.json`, `tokenizer_config.json` (placeholder
|
| 32 |
+
config, vocab 32768 — train a real tokenizer before pretraining),
|
| 33 |
+
`benchmark_genesis_a100.py`, `EVIDENCE_GENESIS_A100_MAX.md`.
|
| 34 |
+
|
| 35 |
+
No trained weights are included in this upload. `config.json` defines a
|
| 36 |
+
random-init architecture. To materialize weights:
|
| 37 |
+
|
| 38 |
+
```python
|
| 39 |
+
from modeling_faber_genesis import FaberGenesisConfig, FaberGenesisForCausalLM
|
| 40 |
+
cfg = FaberGenesisConfig() # max-13.86B
|
| 41 |
+
model = FaberGenesisForCausalLM(cfg) # random init, BF16 training recipe applies on A100
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
## Ceiling benchmark (run on A100 80GB)
|
| 45 |
+
|
| 46 |
+
```bash
|
| 47 |
+
pip install "torch==2.6.0" bitsandbytes transformers
|
| 48 |
+
python benchmark_genesis_a100.py --profile max-13.86B --steps 1
|
| 49 |
+
python benchmark_genesis_a100.py --sweep --n-list 25,29,32,33,34,35,36,37 --steps 1
|
| 50 |
+
```
|
| 51 |
+
|
| 52 |
+
Pass = fwd+bwd+optim step completes; n=36/37 must OOM. VRAM estimates in
|
| 53 |
+
`EVIDENCE_GENESIS_A100_MAX.md` are analytic, not a substitute for this run.
|
| 54 |
+
|
| 55 |
+
## Limitations
|
| 56 |
+
|
| 57 |
+
Random-init step-pass says nothing about Thai/code/reasoning quality.
|
| 58 |
+
Quality requires pretraining curriculum + CASS + verifier + agent training
|
| 59 |
+
and SWE/APEX evals. 14.25B+ is not a single-A100 resident AdamW8bit config.
|
__pycache__/modeling_faber_genesis.cpython-313.pyc
ADDED
|
Binary file (12.5 kB). View file
|
|
|
benchmark_genesis_a100.py
ADDED
|
@@ -0,0 +1,168 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# /// script
|
| 2 |
+
# dependencies = ["torch==2.6.0"]
|
| 3 |
+
# ///
|
| 4 |
+
"""FABER-Genesis A100 ceiling benchmark — real fwd+bwd+optim measurement.
|
| 5 |
+
|
| 6 |
+
Recipe (fixed, per spec):
|
| 7 |
+
BF16 weights, AdamW8bit (bitsandbytes when available, else AdamW fallback),
|
| 8 |
+
activation checkpointing ON, microbatch=1, ctx=512.
|
| 9 |
+
|
| 10 |
+
Modes:
|
| 11 |
+
--sweep reproduce the ceiling table over n_physical values
|
| 12 |
+
--profile NAME single profile (production-13.08B, safe-13.47B,
|
| 13 |
+
max-13.86B, oom-14.25B, ref-9.96B)
|
| 14 |
+
--estimate-only analytic VRAM estimate, no GPU allocation
|
| 15 |
+
(used on machines without 80GB; still prints the table)
|
| 16 |
+
--steps N optimizer steps per config (default 1; must be >=1)
|
| 17 |
+
|
| 18 |
+
On CUDA the script builds the REAL model in BF16 on GPU, runs
|
| 19 |
+
forward + backward + optimizer step, and records torch.cuda peak.
|
| 20 |
+
OOM is caught per-config and reported as status=oom (no fabrication).
|
| 21 |
+
|
| 22 |
+
Output: lines `GENESIS_RESULT <json>` + optional --out json file.
|
| 23 |
+
"""
|
| 24 |
+
from __future__ import annotations
|
| 25 |
+
|
| 26 |
+
import argparse
|
| 27 |
+
import json
|
| 28 |
+
import sys
|
| 29 |
+
import time
|
| 30 |
+
|
| 31 |
+
sys.path.insert(0, ".")
|
| 32 |
+
|
| 33 |
+
from faber.genesis import GenesisConfig, build_share_map, count_params, estimate_peak_vram_gib, profile
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def build_optimizer(model, lr=1e-4):
|
| 37 |
+
try:
|
| 38 |
+
import bitsandbytes as bnb # type: ignore
|
| 39 |
+
|
| 40 |
+
return ("adamw8bit", bnb.optim.AdamW8bit(model.parameters(), lr=lr))
|
| 41 |
+
except Exception as exc:
|
| 42 |
+
import torch
|
| 43 |
+
|
| 44 |
+
print(f"WARN adamw8bit-unavailable({type(exc).__name__}), fallback to AdamW", flush=True)
|
| 45 |
+
return ("adamw-fallback", torch.optim.AdamW(model.parameters(), lr=lr))
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
def measure_once(cfg: GenesisConfig, ctx: int, steps: int, lr: float) -> dict:
|
| 49 |
+
import torch
|
| 50 |
+
|
| 51 |
+
from faber.genesis import GenesisModel
|
| 52 |
+
|
| 53 |
+
dev = torch.device("cuda")
|
| 54 |
+
model = GenesisModel(cfg).to(torch.bfloat16).to(dev)
|
| 55 |
+
model.train()
|
| 56 |
+
model.use_checkpoint = True
|
| 57 |
+
opt_name, opt = build_optimizer(model, lr)
|
| 58 |
+
ids = torch.randint(0, cfg.vocab, (1, ctx), device=dev)
|
| 59 |
+
labels = ids.clone()
|
| 60 |
+
torch.cuda.reset_peak_memory_stats(dev)
|
| 61 |
+
torch.cuda.synchronize(dev)
|
| 62 |
+
t0 = time.perf_counter()
|
| 63 |
+
status = "pass"
|
| 64 |
+
loss_val = None
|
| 65 |
+
err = None
|
| 66 |
+
try:
|
| 67 |
+
for _ in range(steps):
|
| 68 |
+
opt.zero_grad(set_to_none=True)
|
| 69 |
+
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
|
| 70 |
+
out = model(ids, labels)
|
| 71 |
+
loss = out["loss"] + 0.01 * out["cass_value"].pow(2).mean()
|
| 72 |
+
loss.backward()
|
| 73 |
+
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
|
| 74 |
+
opt.step()
|
| 75 |
+
loss_val = float(loss.detach())
|
| 76 |
+
torch.cuda.synchronize(dev)
|
| 77 |
+
except torch.cuda.OutOfMemoryError as exc:
|
| 78 |
+
status = "oom"
|
| 79 |
+
err = f"OutOfMemoryError: {exc}"
|
| 80 |
+
torch.cuda.empty_cache()
|
| 81 |
+
except RuntimeError as exc:
|
| 82 |
+
if "out of memory" in str(exc).lower():
|
| 83 |
+
status = "oom"
|
| 84 |
+
err = f"RuntimeError-OOM: {exc}"
|
| 85 |
+
torch.cuda.empty_cache()
|
| 86 |
+
else:
|
| 87 |
+
status = "error"
|
| 88 |
+
err = f"{type(exc).__name__}: {exc}"
|
| 89 |
+
peak = torch.cuda.max_memory_allocated(dev) / 1024**3 if status != "error" else None
|
| 90 |
+
dt = time.perf_counter() - t0
|
| 91 |
+
del model, opt
|
| 92 |
+
torch.cuda.empty_cache()
|
| 93 |
+
c = count_params(cfg)
|
| 94 |
+
est = estimate_peak_vram_gib(cfg, ctx=ctx)
|
| 95 |
+
return {
|
| 96 |
+
"num_physical": cfg.num_physical,
|
| 97 |
+
"num_logical": cfg.num_logical,
|
| 98 |
+
"params_B": round(c["total_B"], 4),
|
| 99 |
+
"estimate_peak_gib": round(est["peak_gib"], 2),
|
| 100 |
+
"measured_peak_gib": round(peak, 2) if peak is not None else None,
|
| 101 |
+
"status": status,
|
| 102 |
+
"optimizer": opt_name,
|
| 103 |
+
"loss": loss_val,
|
| 104 |
+
"seconds": round(dt, 2),
|
| 105 |
+
"error": err,
|
| 106 |
+
}
|
| 107 |
+
|
| 108 |
+
|
| 109 |
+
def main() -> None:
|
| 110 |
+
p = argparse.ArgumentParser()
|
| 111 |
+
p.add_argument("--profile", default="max-13.86B")
|
| 112 |
+
p.add_argument("--sweep", action="store_true")
|
| 113 |
+
p.add_argument("--n-list", default="25,29,32,33,34,35,36,37")
|
| 114 |
+
p.add_argument("--ctx", type=int, default=512)
|
| 115 |
+
p.add_argument("--steps", type=int, default=1)
|
| 116 |
+
p.add_argument("--lr", type=float, default=1e-4)
|
| 117 |
+
p.add_argument("--estimate-only", action="store_true")
|
| 118 |
+
p.add_argument("--out", default="")
|
| 119 |
+
args = p.parse_args()
|
| 120 |
+
|
| 121 |
+
import torch
|
| 122 |
+
|
| 123 |
+
gpu = torch.cuda.get_device_name(0) if torch.cuda.is_available() else "no-cuda"
|
| 124 |
+
total_gib = (torch.cuda.get_device_properties(0).total_memory / 1024**3) if torch.cuda.is_available() else 0.0
|
| 125 |
+
print(f"GPU {gpu} total_GiB={round(total_gib,2)}", flush=True)
|
| 126 |
+
print(f"RECIPE bf16+adamw8bit+ckpt microbatch=1 ctx={args.ctx} steps={args.steps}", flush=True)
|
| 127 |
+
|
| 128 |
+
cfgs: list[tuple[str, GenesisConfig]] = []
|
| 129 |
+
if args.sweep:
|
| 130 |
+
for n in [int(x) for x in args.n_list.split(",") if x.strip()]:
|
| 131 |
+
logical = n + 8 if n <= 35 else n + 7
|
| 132 |
+
cfgs.append((f"n={n}", GenesisConfig(num_physical=n, num_logical=logical, n_recurrent=1)))
|
| 133 |
+
else:
|
| 134 |
+
cfgs.append((args.profile, profile(args.profile)))
|
| 135 |
+
|
| 136 |
+
results = []
|
| 137 |
+
for name, cfg in cfgs:
|
| 138 |
+
c = count_params(cfg)
|
| 139 |
+
est = estimate_peak_vram_gib(cfg, ctx=args.ctx)
|
| 140 |
+
if args.estimate_only or not torch.cuda.is_available():
|
| 141 |
+
row = {
|
| 142 |
+
"name": name,
|
| 143 |
+
"num_physical": cfg.num_physical,
|
| 144 |
+
"num_logical": cfg.num_logical,
|
| 145 |
+
"params_B": round(c["total_B"], 4),
|
| 146 |
+
"estimate_peak_gib": round(est["peak_gib"], 2),
|
| 147 |
+
"measured_peak_gib": None,
|
| 148 |
+
"status": "estimate-only",
|
| 149 |
+
"optimizer": "adamw8bit-spec",
|
| 150 |
+
}
|
| 151 |
+
else:
|
| 152 |
+
row = {"name": name, **measure_once(cfg, args.ctx, max(1, args.steps), args.lr)}
|
| 153 |
+
row["gpu"] = gpu
|
| 154 |
+
results.append(row)
|
| 155 |
+
print("GENESIS_RESULT " + json.dumps(row, sort_keys=True), flush=True)
|
| 156 |
+
|
| 157 |
+
if args.out:
|
| 158 |
+
with open(args.out, "w", encoding="utf-8") as f:
|
| 159 |
+
json.dump(
|
| 160 |
+
{"kind": "faber_genesis_a100_ceiling", "gpu": gpu, "ctx": args.ctx,
|
| 161 |
+
"steps": args.steps, "rows": results},
|
| 162 |
+
f, indent=2, sort_keys=True,
|
| 163 |
+
)
|
| 164 |
+
print(f"WROTE {args.out}", flush=True)
|
| 165 |
+
|
| 166 |
+
|
| 167 |
+
if __name__ == "__main__":
|
| 168 |
+
main()
|
config.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"architectures": ["FaberGenesisForCausalLM"], "auto_map": {"AutoConfig": "modeling_faber_genesis.FaberGenesisConfig", "AutoModelForCausalLM": "modeling_faber_genesis.FaberGenesisForCausalLM"}, "bf16_training_recipe": {"activation_checkpointing": true, "context": 512, "microbatch": 1, "optimizer": "AdamW8bit"}, "cass_state_dim": 1024, "ffn_dim": 16384, "hidden_size": 6144, "model_type": "faber-genesis", "n_kv_heads": 8, "n_recurrent": 1, "num_attention_heads": 48, "num_hidden_layers": 35, "num_logical_layers": 42, "num_physical_layers": 35, "param_total": 13860526082, "param_total_B": 13.860526082, "profile": "max-13.86B", "rms_norm_eps": 1e-05, "rope_theta": 10000.0, "share_map": [0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34], "tie_word_embeddings": true, "torch_dtype": "bfloat16", "transformers_version": "4.55.0", "use_cache": true, "vocab_size": 32768}
|
genesis_profiles.json
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"oom-14.25B": {
|
| 3 |
+
"cass_state_dim": 1024,
|
| 4 |
+
"ffn_dim": 16384,
|
| 5 |
+
"hidden_size": 6144,
|
| 6 |
+
"n_kv_heads": 8,
|
| 7 |
+
"n_recurrent": 1,
|
| 8 |
+
"num_attention_heads": 48,
|
| 9 |
+
"num_logical_layers": 43,
|
| 10 |
+
"num_physical_layers": 36,
|
| 11 |
+
"params_B": 14.250608642,
|
| 12 |
+
"status": "oom-single-A100-AdamW8bit",
|
| 13 |
+
"vocab_size": 32768
|
| 14 |
+
},
|
| 15 |
+
"production-13.08B": {
|
| 16 |
+
"cass_state_dim": 1024,
|
| 17 |
+
"ffn_dim": 16384,
|
| 18 |
+
"hidden_size": 6144,
|
| 19 |
+
"n_kv_heads": 8,
|
| 20 |
+
"n_recurrent": 1,
|
| 21 |
+
"num_attention_heads": 48,
|
| 22 |
+
"num_logical_layers": 41,
|
| 23 |
+
"num_physical_layers": 33,
|
| 24 |
+
"params_B": 13.080360962,
|
| 25 |
+
"status": "default-long-runs",
|
| 26 |
+
"vocab_size": 32768
|
| 27 |
+
},
|
| 28 |
+
"ref-9.96B": {
|
| 29 |
+
"cass_state_dim": 1024,
|
| 30 |
+
"ffn_dim": 16384,
|
| 31 |
+
"hidden_size": 6144,
|
| 32 |
+
"n_kv_heads": 8,
|
| 33 |
+
"n_recurrent": 1,
|
| 34 |
+
"num_attention_heads": 48,
|
| 35 |
+
"num_logical_layers": 30,
|
| 36 |
+
"num_physical_layers": 25,
|
| 37 |
+
"params_B": 9.959700482,
|
| 38 |
+
"status": "reference",
|
| 39 |
+
"vocab_size": 32768
|
| 40 |
+
},
|
| 41 |
+
"safe-13.47B": {
|
| 42 |
+
"cass_state_dim": 1024,
|
| 43 |
+
"ffn_dim": 16384,
|
| 44 |
+
"hidden_size": 6144,
|
| 45 |
+
"n_kv_heads": 8,
|
| 46 |
+
"n_recurrent": 1,
|
| 47 |
+
"num_attention_heads": 48,
|
| 48 |
+
"num_logical_layers": 42,
|
| 49 |
+
"num_physical_layers": 34,
|
| 50 |
+
"params_B": 13.470443522,
|
| 51 |
+
"status": "squeeze-with-safety",
|
| 52 |
+
"vocab_size": 32768
|
| 53 |
+
}
|
| 54 |
+
}
|
modeling_faber_genesis.py
ADDED
|
@@ -0,0 +1,137 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""HF wrapper for FABER-Genesis A100-Max. Self-contained; mirrors faber/genesis.py."""
|
| 2 |
+
from __future__ import annotations
|
| 3 |
+
|
| 4 |
+
import torch
|
| 5 |
+
import torch.nn as nn
|
| 6 |
+
import torch.nn.functional as F
|
| 7 |
+
from torch.utils.checkpoint import checkpoint
|
| 8 |
+
from transformers import PretrainedConfig, PreTrainedModel
|
| 9 |
+
from transformers.modeling_outputs import CausalLMOutput
|
| 10 |
+
|
| 11 |
+
|
| 12 |
+
class FaberGenesisConfig(PretrainedConfig):
|
| 13 |
+
model_type = "faber-genesis"
|
| 14 |
+
|
| 15 |
+
def __init__(self, vocab_size=32768, hidden_size=6144, num_attention_heads=48,
|
| 16 |
+
n_kv_heads=8, ffn_dim=16384, num_physical_layers=35,
|
| 17 |
+
num_logical_layers=42, n_recurrent=1, cass_state_dim=1024,
|
| 18 |
+
rms_norm_eps=1e-5, rope_theta=10000.0, tie_word_embeddings=True,
|
| 19 |
+
share_map=None, **kwargs):
|
| 20 |
+
super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs)
|
| 21 |
+
self.vocab_size = vocab_size
|
| 22 |
+
self.hidden_size = hidden_size
|
| 23 |
+
self.num_attention_heads = num_attention_heads
|
| 24 |
+
self.n_kv_heads = n_kv_heads
|
| 25 |
+
self.head_dim = hidden_size // num_attention_heads
|
| 26 |
+
self.ffn_dim = ffn_dim
|
| 27 |
+
self.num_hidden_layers = num_physical_layers
|
| 28 |
+
self.num_physical_layers = num_physical_layers
|
| 29 |
+
self.num_logical_layers = num_logical_layers
|
| 30 |
+
self.n_recurrent = n_recurrent
|
| 31 |
+
self.cass_state_dim = cass_state_dim
|
| 32 |
+
self.rms_norm_eps = rms_norm_eps
|
| 33 |
+
self.rope_theta = rope_theta
|
| 34 |
+
assert hidden_size == num_attention_heads * self.head_dim
|
| 35 |
+
if share_map is None:
|
| 36 |
+
seq = list(range(num_physical_layers))
|
| 37 |
+
extra = num_logical_layers - num_physical_layers
|
| 38 |
+
mid = list(range(1, num_physical_layers - 1)) or [0]
|
| 39 |
+
for i in range(max(0, extra)):
|
| 40 |
+
anchor = mid[i % len(mid)]
|
| 41 |
+
seq.insert(seq.index(anchor) + 1, anchor)
|
| 42 |
+
share_map = seq[:num_logical_layers]
|
| 43 |
+
self.share_map = list(share_map)
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
class FaberRMSNorm(nn.Module):
|
| 47 |
+
def __init__(self, dim, eps=1e-5):
|
| 48 |
+
super().__init__()
|
| 49 |
+
self.eps = eps
|
| 50 |
+
self.weight = nn.Parameter(torch.ones(dim))
|
| 51 |
+
|
| 52 |
+
def forward(self, x):
|
| 53 |
+
xf = x.float()
|
| 54 |
+
return (self.weight * x * torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + self.eps)).to(x.dtype)
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
class FaberGQA(nn.Module):
|
| 58 |
+
def __init__(self, cfg: FaberGenesisConfig):
|
| 59 |
+
super().__init__()
|
| 60 |
+
h = cfg.hidden_size
|
| 61 |
+
self.n_heads, self.n_kv, self.hd = cfg.num_attention_heads, cfg.n_kv_heads, cfg.head_dim
|
| 62 |
+
self.n_rep = cfg.num_attention_heads // cfg.n_kv_heads
|
| 63 |
+
self.q = nn.Linear(h, cfg.num_attention_heads * cfg.head_dim, bias=False)
|
| 64 |
+
self.k = nn.Linear(h, cfg.n_kv_heads * cfg.head_dim, bias=False)
|
| 65 |
+
self.v = nn.Linear(h, cfg.n_kv_heads * cfg.head_dim, bias=False)
|
| 66 |
+
self.o = nn.Linear(cfg.num_attention_heads * cfg.head_dim, h, bias=False)
|
| 67 |
+
self.theta = cfg.rope_theta
|
| 68 |
+
|
| 69 |
+
def _rope(self, q, k):
|
| 70 |
+
T, D, dev = q.shape[2], self.hd, q.device
|
| 71 |
+
inv = 1.0 / (self.theta ** (torch.arange(0, D, 2, device=dev).float() / D))
|
| 72 |
+
t = torch.arange(T, device=dev).float()
|
| 73 |
+
freqs = torch.outer(t, inv)
|
| 74 |
+
cos, sin = freqs.cos().to(q.dtype), freqs.sin().to(q.dtype)
|
| 75 |
+
cos = cos[None, None, :, :].repeat_interleave(2, dim=-1)
|
| 76 |
+
sin = sin[None, None, :, :].repeat_interleave(2, dim=-1)
|
| 77 |
+
|
| 78 |
+
def rot(x):
|
| 79 |
+
x1, x2 = x[..., ::2], x[..., 1::2]
|
| 80 |
+
r = torch.stack([-x2, x1], dim=-1).flatten(-2)
|
| 81 |
+
return x * cos + r * sin
|
| 82 |
+
return rot(q), rot(k)
|
| 83 |
+
|
| 84 |
+
def forward(self, x):
|
| 85 |
+
B, T, _ = x.shape
|
| 86 |
+
q = self.q(x).view(B, T, self.n_heads, self.hd).transpose(1, 2)
|
| 87 |
+
k = self.k(x).view(B, T, self.n_kv, self.hd).transpose(1, 2)
|
| 88 |
+
v = self.v(x).view(B, T, self.n_kv, self.hd).transpose(1, 2)
|
| 89 |
+
q, k = self._rope(q, k)
|
| 90 |
+
if self.n_rep > 1:
|
| 91 |
+
k = k.repeat_interleave(self.n_rep, dim=1)
|
| 92 |
+
v = v.repeat_interleave(self.n_rep, dim=1)
|
| 93 |
+
y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
|
| 94 |
+
return self.o(y.transpose(1, 2).reshape(B, T, -1))
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
class FaberBlock(nn.Module):
|
| 98 |
+
def __init__(self, cfg):
|
| 99 |
+
super().__init__()
|
| 100 |
+
self.n1 = FaberRMSNorm(cfg.hidden_size, cfg.rms_norm_eps)
|
| 101 |
+
self.attn = FaberGQA(cfg)
|
| 102 |
+
self.n2 = FaberRMSNorm(cfg.hidden_size, cfg.rms_norm_eps)
|
| 103 |
+
self.gate = nn.Linear(cfg.hidden_size, cfg.ffn_dim, bias=False)
|
| 104 |
+
self.up = nn.Linear(cfg.hidden_size, cfg.ffn_dim, bias=False)
|
| 105 |
+
self.down = nn.Linear(cfg.ffn_dim, cfg.hidden_size, bias=False)
|
| 106 |
+
|
| 107 |
+
def forward(self, x):
|
| 108 |
+
x = x + self.attn(self.n1(x))
|
| 109 |
+
return x + self.down(F.silu(self.gate(self.n2(x))) * self.up(self.n2(x)))
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
class FaberGenesisForCausalLM(PreTrainedModel):
|
| 113 |
+
config_class = FaberGenesisConfig
|
| 114 |
+
|
| 115 |
+
def __init__(self, config):
|
| 116 |
+
super().__init__(config)
|
| 117 |
+
self.cfg = config
|
| 118 |
+
self.tok = nn.Embedding(config.vocab_size, config.hidden_size)
|
| 119 |
+
self.blocks = nn.ModuleList([FaberBlock(config) for _ in range(config.num_physical_layers)])
|
| 120 |
+
self.final_norm = FaberRMSNorm(config.hidden_size, config.rms_norm_eps)
|
| 121 |
+
self.cass_state = nn.Linear(config.hidden_size, config.cass_state_dim, bias=False)
|
| 122 |
+
self.cass_value = nn.Linear(config.hidden_size, 1)
|
| 123 |
+
self.cass_halt = nn.Linear(config.hidden_size, 1)
|
| 124 |
+
self.post_init()
|
| 125 |
+
|
| 126 |
+
def forward(self, input_ids, labels=None, use_checkpoint=True, **kwargs):
|
| 127 |
+
x = self.tok(input_ids)
|
| 128 |
+
for phys in self.cfg.share_map:
|
| 129 |
+
blk = self.blocks[phys]
|
| 130 |
+
x = checkpoint(blk, x, use_reentrant=False) if (use_checkpoint and self.training and x.requires_grad) else blk(x)
|
| 131 |
+
for _ in range(max(0, self.cfg.n_recurrent - 1)):
|
| 132 |
+
blk = self.blocks[-1]
|
| 133 |
+
x = checkpoint(blk, x, use_reentrant=False) if (use_checkpoint and self.training and x.requires_grad) else blk(x)
|
| 134 |
+
h = self.final_norm(x)
|
| 135 |
+
logits = F.linear(h, self.tok.weight)
|
| 136 |
+
loss = F.cross_entropy(logits.float().view(-1, self.config.vocab_size), labels.view(-1), ignore_index=-100) if labels is not None else None
|
| 137 |
+
return CausalLMOutput(loss=loss, logits=logits)
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
{"add_bos_token": true, "add_eos_token": true, "bos_token": "<s>", "chat_template": "{% for m in messages %}{{ m.role + ': ' + m.content + '\n' }}{% endfor %}", "eos_token": "</s>", "model_max_length": 512, "pad_token": "</s>", "tokenizer_class": "PreTrainedTokenizerFast", "unk_token": "<unk>", "vocab_size": 32768}
|