bbkdevops commited on
Commit
2ffbf1f
·
verified ·
1 Parent(s): 0b7def2

FABER-Genesis-A100-Max 13.86B architecture: config + HF modeling + profiles + benchmark harness

Browse files
EVIDENCE_GENESIS_A100_MAX.md ADDED
@@ -0,0 +1,92 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # FABER-Genesis A100-Max — Model + Ceiling Benchmark (v0.9.x)
2
+
3
+ ## 1. What was built (real code, not formula-only)
4
+
5
+ - `faber/genesis.py` — decoder-only transformer exactly per spec:
6
+ hidden 6144, heads 48, KV 8, head-dim 128, SwiGLU FFN 16384,
7
+ vocab 32768, tied embeddings, RMSNorm, RoPE, GQA,
8
+ activation checkpointing, BF16, CASS state/value/halt heads.
9
+ Weight sharing via `share_map`: 34 physical -> 42 logical.
10
+ - `benchmarks/benchmark_genesis_a100.py` — real ceiling harness:
11
+ BF16 + AdamW8bit (bitsandbytes, fallback warns) + ckpt,
12
+ microbatch=1, ctx=512, fwd+bwd+optim step, `torch.cuda` peak,
13
+ per-config OOM catch. Supports `--sweep`, `--profile`, `--estimate-only`.
14
+ - `tests/test_genesis.py` — 7 tests: sharing, param landmarks,
15
+ boxed numbers, estimator-vs-reported, ordering, profiles, tiny CPU fwd+bwd.
16
+ - `benchmarks/results_genesis_a100_estimate.json` — estimator output (this machine).
17
+
18
+ ## 2. Param accounting (exact counts from code)
19
+
20
+ `per_block = 390,082,560`; `emb = 201,326,592`; CASS = 6,303,746.
21
+
22
+ | n_phys | total (B) | label |
23
+ |---:|---:|:---|
24
+ | 25 | 9.9597 | ref-9.96B |
25
+ | 29 | 11.5200 | 11.52B |
26
+ | 32 | 12.6903 | 12.69B |
27
+ | 33 | 13.0804 | production-13.08B |
28
+ | 34 | 13.4704 | safe-13.47B |
29
+ | 35 | 13.8605 | **max-13.86B** (34 unique + 1 recurrent) |
30
+ | 36 | 14.2506 | **oom-14.25B** (backbone 14.2443B + CASS) |
31
+ | 37 | 14.6407 | 14.64B |
32
+
33
+ Boxed claims reproduced: `13.8605B passed-design`, `14.2443B backbone OOM-design`.
34
+
35
+ ## 3. VRAM: estimator (local) vs required A100 measurement
36
+
37
+ Estimator = 6 bytes/param (BF16 w + BF16 g + 8-bit m+v) + ckpt activations
38
+ + GQA KV transient + 0.45 GiB reserve. Local run (RTX 3090, estimate-only):
39
+
40
+ | n | params | estimate | reported A100 | delta |
41
+ |---:|---:|---:|---:|---:|
42
+ | 25 | 9.96B | 56.56 | 55.97 | +0.59 |
43
+ | 29 | 11.52B | 65.33 | 64.78 | +0.55 |
44
+ | 32 | 12.69B | 71.91 | 71.36 | +0.55 |
45
+ | 33 | 13.08B | 74.10 | 73.55 | +0.55 |
46
+ | 34 | 13.47B | 76.30 | 75.75 | +0.55 |
47
+ | 35 | 13.86B | 78.49 | 77.94 | +0.55 |
48
+ | 36 | 14.25B | 80.67 | ~78.72 pre-OOM | — |
49
+ | 37 | 14.64B | 82.86 | ~78.72 pre-OOM | — |
50
+
51
+ Ordering holds: 35 fits under Jobs-visible 79.25 GiB, 36 exceeds it.
52
+ The estimator is conservative by ~0.55 GiB; it is NOT a substitute for
53
+ the A100 run below. Reported 39.97 GiB / 7.05B row is a legacy smaller
54
+ config (same family at n=18 gives 7.23B with this vocab); all 9.9B+ rows
55
+ match exactly.
56
+
57
+ ## 4. Locally verified (this machine, RTX 3090 24GB)
58
+
59
+ - `python -m pytest tests/test_genesis.py -q` — **7/7 pass**.
60
+ - `benchmark_genesis_a100.py --sweep --estimate-only` — table above, pass.
61
+ - Tiny real-CUDA path: hidden=256/n=2/ctx=512 BF16 fwd+bwd+AdamW step —
62
+ **pass, peak 0.035 GiB**. Proves the harness executes fwd+bwd+step on CUDA.
63
+ - Full 13B allocation NOT attempted here (needs ~78 GiB > 24 GiB); doing so
64
+ would OOM by design on this card.
65
+
66
+ ## 5. Required A100 run (proves the ceiling, run on the A100 box)
67
+
68
+ ```bash
69
+ pip install "torch==2.6.0" bitsandbytes
70
+ # single max profile, real measurement:
71
+ python benchmarks/benchmark_genesis_a100.py --profile max-13.86B --steps 1 \
72
+ --out benchmarks/results_genesis_max.json
73
+ # full ceiling sweep with real peaks + OOM capture:
74
+ python benchmarks/benchmark_genesis_a100.py --sweep \
75
+ --n-list 25,29,32,33,34,35,36,37 --steps 1 \
76
+ --out benchmarks/results_genesis_sweep.json
77
+ ```
78
+
79
+ Pass criteria: each row prints `GENESIS_RESULT` with `measured_peak_gib`
80
+ and `status=pass`, except n=36,37 which must print `status=oom`.
81
+ Only `measured_peak_gib` from that run certifies 13.86B-pass / 14.25B-OOM.
82
+
83
+ ## 6. Profiles (decision carried over)
84
+
85
+ - `production-13.08B` (n=33): ~73.5 GiB, ~5.7 GiB headroom — default for long runs.
86
+ - `safe-13.47B` (n=34): ~75.7 GiB, ~3.5 GiB headroom — squeeze with safety.
87
+ - `max-13.86B` (n=35): ~77.9 GiB — capacity experiments only, not default.
88
+ - `oom-14.25B` (n=36+): not a single-A100 resident AdamW8bit config.
89
+
90
+ Model size != intelligence: random-init step-pass says nothing about Thai /
91
+ code / reasoning quality; that needs curriculum + CASS + verifier + agent
92
+ training and SWE/APEX evals separately.
README.md ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: [en, th]
3
+ library_name: transformers
4
+ license: apache-2.0
5
+ pipeline_tag: text-generation
6
+ tags: [faber-genesis, gqa, rope, swiglu, weight-sharing, cass, bf16, a100, th]
7
+ ---
8
+
9
+ # FABER-Genesis-A100-Max (13.86B architecture)
10
+
11
+ Decoder-only transformer with weight sharing + CASS state/value/halt heads.
12
+
13
+ - hidden 6144, heads 48, KV 8, head-dim 128, SwiGLU FFN 16384
14
+ - vocab 32768, tied embeddings, RMSNorm, RoPE, GQA
15
+ - 35 physical blocks -> 42 logical transformations + 1 recurrent pass
16
+ - total params: **13.860526082B** (backbone + CASS)
17
+ - training recipe: BF16, AdamW8bit, activation checkpointing, microbatch=1, ctx=512
18
+
19
+ ## Profiles
20
+
21
+ | profile | physical | params | status |
22
+ |---|---|---:|---|
23
+ | production-13.08B | 33 | 13.0804B | default for long runs (~73.55 GiB) |
24
+ | safe-13.47B | 34 | 13.4704B | squeeze with safety (~75.75 GiB) |
25
+ | max-13.86B | 35 | 13.8605B | capacity experiments (~77.94 GiB) |
26
+ | oom-14.25B | 36 | 14.2506B | OOM on single A100 AdamW8bit |
27
+
28
+ ## What this repo contains
29
+
30
+ `config.json` (max profile), `modeling_faber_genesis.py` (self-contained HF
31
+ model code), `genesis_profiles.json`, `tokenizer_config.json` (placeholder
32
+ config, vocab 32768 — train a real tokenizer before pretraining),
33
+ `benchmark_genesis_a100.py`, `EVIDENCE_GENESIS_A100_MAX.md`.
34
+
35
+ No trained weights are included in this upload. `config.json` defines a
36
+ random-init architecture. To materialize weights:
37
+
38
+ ```python
39
+ from modeling_faber_genesis import FaberGenesisConfig, FaberGenesisForCausalLM
40
+ cfg = FaberGenesisConfig() # max-13.86B
41
+ model = FaberGenesisForCausalLM(cfg) # random init, BF16 training recipe applies on A100
42
+ ```
43
+
44
+ ## Ceiling benchmark (run on A100 80GB)
45
+
46
+ ```bash
47
+ pip install "torch==2.6.0" bitsandbytes transformers
48
+ python benchmark_genesis_a100.py --profile max-13.86B --steps 1
49
+ python benchmark_genesis_a100.py --sweep --n-list 25,29,32,33,34,35,36,37 --steps 1
50
+ ```
51
+
52
+ Pass = fwd+bwd+optim step completes; n=36/37 must OOM. VRAM estimates in
53
+ `EVIDENCE_GENESIS_A100_MAX.md` are analytic, not a substitute for this run.
54
+
55
+ ## Limitations
56
+
57
+ Random-init step-pass says nothing about Thai/code/reasoning quality.
58
+ Quality requires pretraining curriculum + CASS + verifier + agent training
59
+ and SWE/APEX evals. 14.25B+ is not a single-A100 resident AdamW8bit config.
__pycache__/modeling_faber_genesis.cpython-313.pyc ADDED
Binary file (12.5 kB). View file
 
benchmark_genesis_a100.py ADDED
@@ -0,0 +1,168 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # /// script
2
+ # dependencies = ["torch==2.6.0"]
3
+ # ///
4
+ """FABER-Genesis A100 ceiling benchmark — real fwd+bwd+optim measurement.
5
+
6
+ Recipe (fixed, per spec):
7
+ BF16 weights, AdamW8bit (bitsandbytes when available, else AdamW fallback),
8
+ activation checkpointing ON, microbatch=1, ctx=512.
9
+
10
+ Modes:
11
+ --sweep reproduce the ceiling table over n_physical values
12
+ --profile NAME single profile (production-13.08B, safe-13.47B,
13
+ max-13.86B, oom-14.25B, ref-9.96B)
14
+ --estimate-only analytic VRAM estimate, no GPU allocation
15
+ (used on machines without 80GB; still prints the table)
16
+ --steps N optimizer steps per config (default 1; must be >=1)
17
+
18
+ On CUDA the script builds the REAL model in BF16 on GPU, runs
19
+ forward + backward + optimizer step, and records torch.cuda peak.
20
+ OOM is caught per-config and reported as status=oom (no fabrication).
21
+
22
+ Output: lines `GENESIS_RESULT <json>` + optional --out json file.
23
+ """
24
+ from __future__ import annotations
25
+
26
+ import argparse
27
+ import json
28
+ import sys
29
+ import time
30
+
31
+ sys.path.insert(0, ".")
32
+
33
+ from faber.genesis import GenesisConfig, build_share_map, count_params, estimate_peak_vram_gib, profile
34
+
35
+
36
+ def build_optimizer(model, lr=1e-4):
37
+ try:
38
+ import bitsandbytes as bnb # type: ignore
39
+
40
+ return ("adamw8bit", bnb.optim.AdamW8bit(model.parameters(), lr=lr))
41
+ except Exception as exc:
42
+ import torch
43
+
44
+ print(f"WARN adamw8bit-unavailable({type(exc).__name__}), fallback to AdamW", flush=True)
45
+ return ("adamw-fallback", torch.optim.AdamW(model.parameters(), lr=lr))
46
+
47
+
48
+ def measure_once(cfg: GenesisConfig, ctx: int, steps: int, lr: float) -> dict:
49
+ import torch
50
+
51
+ from faber.genesis import GenesisModel
52
+
53
+ dev = torch.device("cuda")
54
+ model = GenesisModel(cfg).to(torch.bfloat16).to(dev)
55
+ model.train()
56
+ model.use_checkpoint = True
57
+ opt_name, opt = build_optimizer(model, lr)
58
+ ids = torch.randint(0, cfg.vocab, (1, ctx), device=dev)
59
+ labels = ids.clone()
60
+ torch.cuda.reset_peak_memory_stats(dev)
61
+ torch.cuda.synchronize(dev)
62
+ t0 = time.perf_counter()
63
+ status = "pass"
64
+ loss_val = None
65
+ err = None
66
+ try:
67
+ for _ in range(steps):
68
+ opt.zero_grad(set_to_none=True)
69
+ with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
70
+ out = model(ids, labels)
71
+ loss = out["loss"] + 0.01 * out["cass_value"].pow(2).mean()
72
+ loss.backward()
73
+ torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
74
+ opt.step()
75
+ loss_val = float(loss.detach())
76
+ torch.cuda.synchronize(dev)
77
+ except torch.cuda.OutOfMemoryError as exc:
78
+ status = "oom"
79
+ err = f"OutOfMemoryError: {exc}"
80
+ torch.cuda.empty_cache()
81
+ except RuntimeError as exc:
82
+ if "out of memory" in str(exc).lower():
83
+ status = "oom"
84
+ err = f"RuntimeError-OOM: {exc}"
85
+ torch.cuda.empty_cache()
86
+ else:
87
+ status = "error"
88
+ err = f"{type(exc).__name__}: {exc}"
89
+ peak = torch.cuda.max_memory_allocated(dev) / 1024**3 if status != "error" else None
90
+ dt = time.perf_counter() - t0
91
+ del model, opt
92
+ torch.cuda.empty_cache()
93
+ c = count_params(cfg)
94
+ est = estimate_peak_vram_gib(cfg, ctx=ctx)
95
+ return {
96
+ "num_physical": cfg.num_physical,
97
+ "num_logical": cfg.num_logical,
98
+ "params_B": round(c["total_B"], 4),
99
+ "estimate_peak_gib": round(est["peak_gib"], 2),
100
+ "measured_peak_gib": round(peak, 2) if peak is not None else None,
101
+ "status": status,
102
+ "optimizer": opt_name,
103
+ "loss": loss_val,
104
+ "seconds": round(dt, 2),
105
+ "error": err,
106
+ }
107
+
108
+
109
+ def main() -> None:
110
+ p = argparse.ArgumentParser()
111
+ p.add_argument("--profile", default="max-13.86B")
112
+ p.add_argument("--sweep", action="store_true")
113
+ p.add_argument("--n-list", default="25,29,32,33,34,35,36,37")
114
+ p.add_argument("--ctx", type=int, default=512)
115
+ p.add_argument("--steps", type=int, default=1)
116
+ p.add_argument("--lr", type=float, default=1e-4)
117
+ p.add_argument("--estimate-only", action="store_true")
118
+ p.add_argument("--out", default="")
119
+ args = p.parse_args()
120
+
121
+ import torch
122
+
123
+ gpu = torch.cuda.get_device_name(0) if torch.cuda.is_available() else "no-cuda"
124
+ total_gib = (torch.cuda.get_device_properties(0).total_memory / 1024**3) if torch.cuda.is_available() else 0.0
125
+ print(f"GPU {gpu} total_GiB={round(total_gib,2)}", flush=True)
126
+ print(f"RECIPE bf16+adamw8bit+ckpt microbatch=1 ctx={args.ctx} steps={args.steps}", flush=True)
127
+
128
+ cfgs: list[tuple[str, GenesisConfig]] = []
129
+ if args.sweep:
130
+ for n in [int(x) for x in args.n_list.split(",") if x.strip()]:
131
+ logical = n + 8 if n <= 35 else n + 7
132
+ cfgs.append((f"n={n}", GenesisConfig(num_physical=n, num_logical=logical, n_recurrent=1)))
133
+ else:
134
+ cfgs.append((args.profile, profile(args.profile)))
135
+
136
+ results = []
137
+ for name, cfg in cfgs:
138
+ c = count_params(cfg)
139
+ est = estimate_peak_vram_gib(cfg, ctx=args.ctx)
140
+ if args.estimate_only or not torch.cuda.is_available():
141
+ row = {
142
+ "name": name,
143
+ "num_physical": cfg.num_physical,
144
+ "num_logical": cfg.num_logical,
145
+ "params_B": round(c["total_B"], 4),
146
+ "estimate_peak_gib": round(est["peak_gib"], 2),
147
+ "measured_peak_gib": None,
148
+ "status": "estimate-only",
149
+ "optimizer": "adamw8bit-spec",
150
+ }
151
+ else:
152
+ row = {"name": name, **measure_once(cfg, args.ctx, max(1, args.steps), args.lr)}
153
+ row["gpu"] = gpu
154
+ results.append(row)
155
+ print("GENESIS_RESULT " + json.dumps(row, sort_keys=True), flush=True)
156
+
157
+ if args.out:
158
+ with open(args.out, "w", encoding="utf-8") as f:
159
+ json.dump(
160
+ {"kind": "faber_genesis_a100_ceiling", "gpu": gpu, "ctx": args.ctx,
161
+ "steps": args.steps, "rows": results},
162
+ f, indent=2, sort_keys=True,
163
+ )
164
+ print(f"WROTE {args.out}", flush=True)
165
+
166
+
167
+ if __name__ == "__main__":
168
+ main()
config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"architectures": ["FaberGenesisForCausalLM"], "auto_map": {"AutoConfig": "modeling_faber_genesis.FaberGenesisConfig", "AutoModelForCausalLM": "modeling_faber_genesis.FaberGenesisForCausalLM"}, "bf16_training_recipe": {"activation_checkpointing": true, "context": 512, "microbatch": 1, "optimizer": "AdamW8bit"}, "cass_state_dim": 1024, "ffn_dim": 16384, "hidden_size": 6144, "model_type": "faber-genesis", "n_kv_heads": 8, "n_recurrent": 1, "num_attention_heads": 48, "num_hidden_layers": 35, "num_logical_layers": 42, "num_physical_layers": 35, "param_total": 13860526082, "param_total_B": 13.860526082, "profile": "max-13.86B", "rms_norm_eps": 1e-05, "rope_theta": 10000.0, "share_map": [0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34], "tie_word_embeddings": true, "torch_dtype": "bfloat16", "transformers_version": "4.55.0", "use_cache": true, "vocab_size": 32768}
genesis_profiles.json ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "oom-14.25B": {
3
+ "cass_state_dim": 1024,
4
+ "ffn_dim": 16384,
5
+ "hidden_size": 6144,
6
+ "n_kv_heads": 8,
7
+ "n_recurrent": 1,
8
+ "num_attention_heads": 48,
9
+ "num_logical_layers": 43,
10
+ "num_physical_layers": 36,
11
+ "params_B": 14.250608642,
12
+ "status": "oom-single-A100-AdamW8bit",
13
+ "vocab_size": 32768
14
+ },
15
+ "production-13.08B": {
16
+ "cass_state_dim": 1024,
17
+ "ffn_dim": 16384,
18
+ "hidden_size": 6144,
19
+ "n_kv_heads": 8,
20
+ "n_recurrent": 1,
21
+ "num_attention_heads": 48,
22
+ "num_logical_layers": 41,
23
+ "num_physical_layers": 33,
24
+ "params_B": 13.080360962,
25
+ "status": "default-long-runs",
26
+ "vocab_size": 32768
27
+ },
28
+ "ref-9.96B": {
29
+ "cass_state_dim": 1024,
30
+ "ffn_dim": 16384,
31
+ "hidden_size": 6144,
32
+ "n_kv_heads": 8,
33
+ "n_recurrent": 1,
34
+ "num_attention_heads": 48,
35
+ "num_logical_layers": 30,
36
+ "num_physical_layers": 25,
37
+ "params_B": 9.959700482,
38
+ "status": "reference",
39
+ "vocab_size": 32768
40
+ },
41
+ "safe-13.47B": {
42
+ "cass_state_dim": 1024,
43
+ "ffn_dim": 16384,
44
+ "hidden_size": 6144,
45
+ "n_kv_heads": 8,
46
+ "n_recurrent": 1,
47
+ "num_attention_heads": 48,
48
+ "num_logical_layers": 42,
49
+ "num_physical_layers": 34,
50
+ "params_B": 13.470443522,
51
+ "status": "squeeze-with-safety",
52
+ "vocab_size": 32768
53
+ }
54
+ }
modeling_faber_genesis.py ADDED
@@ -0,0 +1,137 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """HF wrapper for FABER-Genesis A100-Max. Self-contained; mirrors faber/genesis.py."""
2
+ from __future__ import annotations
3
+
4
+ import torch
5
+ import torch.nn as nn
6
+ import torch.nn.functional as F
7
+ from torch.utils.checkpoint import checkpoint
8
+ from transformers import PretrainedConfig, PreTrainedModel
9
+ from transformers.modeling_outputs import CausalLMOutput
10
+
11
+
12
+ class FaberGenesisConfig(PretrainedConfig):
13
+ model_type = "faber-genesis"
14
+
15
+ def __init__(self, vocab_size=32768, hidden_size=6144, num_attention_heads=48,
16
+ n_kv_heads=8, ffn_dim=16384, num_physical_layers=35,
17
+ num_logical_layers=42, n_recurrent=1, cass_state_dim=1024,
18
+ rms_norm_eps=1e-5, rope_theta=10000.0, tie_word_embeddings=True,
19
+ share_map=None, **kwargs):
20
+ super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs)
21
+ self.vocab_size = vocab_size
22
+ self.hidden_size = hidden_size
23
+ self.num_attention_heads = num_attention_heads
24
+ self.n_kv_heads = n_kv_heads
25
+ self.head_dim = hidden_size // num_attention_heads
26
+ self.ffn_dim = ffn_dim
27
+ self.num_hidden_layers = num_physical_layers
28
+ self.num_physical_layers = num_physical_layers
29
+ self.num_logical_layers = num_logical_layers
30
+ self.n_recurrent = n_recurrent
31
+ self.cass_state_dim = cass_state_dim
32
+ self.rms_norm_eps = rms_norm_eps
33
+ self.rope_theta = rope_theta
34
+ assert hidden_size == num_attention_heads * self.head_dim
35
+ if share_map is None:
36
+ seq = list(range(num_physical_layers))
37
+ extra = num_logical_layers - num_physical_layers
38
+ mid = list(range(1, num_physical_layers - 1)) or [0]
39
+ for i in range(max(0, extra)):
40
+ anchor = mid[i % len(mid)]
41
+ seq.insert(seq.index(anchor) + 1, anchor)
42
+ share_map = seq[:num_logical_layers]
43
+ self.share_map = list(share_map)
44
+
45
+
46
+ class FaberRMSNorm(nn.Module):
47
+ def __init__(self, dim, eps=1e-5):
48
+ super().__init__()
49
+ self.eps = eps
50
+ self.weight = nn.Parameter(torch.ones(dim))
51
+
52
+ def forward(self, x):
53
+ xf = x.float()
54
+ return (self.weight * x * torch.rsqrt(xf.pow(2).mean(-1, keepdim=True) + self.eps)).to(x.dtype)
55
+
56
+
57
+ class FaberGQA(nn.Module):
58
+ def __init__(self, cfg: FaberGenesisConfig):
59
+ super().__init__()
60
+ h = cfg.hidden_size
61
+ self.n_heads, self.n_kv, self.hd = cfg.num_attention_heads, cfg.n_kv_heads, cfg.head_dim
62
+ self.n_rep = cfg.num_attention_heads // cfg.n_kv_heads
63
+ self.q = nn.Linear(h, cfg.num_attention_heads * cfg.head_dim, bias=False)
64
+ self.k = nn.Linear(h, cfg.n_kv_heads * cfg.head_dim, bias=False)
65
+ self.v = nn.Linear(h, cfg.n_kv_heads * cfg.head_dim, bias=False)
66
+ self.o = nn.Linear(cfg.num_attention_heads * cfg.head_dim, h, bias=False)
67
+ self.theta = cfg.rope_theta
68
+
69
+ def _rope(self, q, k):
70
+ T, D, dev = q.shape[2], self.hd, q.device
71
+ inv = 1.0 / (self.theta ** (torch.arange(0, D, 2, device=dev).float() / D))
72
+ t = torch.arange(T, device=dev).float()
73
+ freqs = torch.outer(t, inv)
74
+ cos, sin = freqs.cos().to(q.dtype), freqs.sin().to(q.dtype)
75
+ cos = cos[None, None, :, :].repeat_interleave(2, dim=-1)
76
+ sin = sin[None, None, :, :].repeat_interleave(2, dim=-1)
77
+
78
+ def rot(x):
79
+ x1, x2 = x[..., ::2], x[..., 1::2]
80
+ r = torch.stack([-x2, x1], dim=-1).flatten(-2)
81
+ return x * cos + r * sin
82
+ return rot(q), rot(k)
83
+
84
+ def forward(self, x):
85
+ B, T, _ = x.shape
86
+ q = self.q(x).view(B, T, self.n_heads, self.hd).transpose(1, 2)
87
+ k = self.k(x).view(B, T, self.n_kv, self.hd).transpose(1, 2)
88
+ v = self.v(x).view(B, T, self.n_kv, self.hd).transpose(1, 2)
89
+ q, k = self._rope(q, k)
90
+ if self.n_rep > 1:
91
+ k = k.repeat_interleave(self.n_rep, dim=1)
92
+ v = v.repeat_interleave(self.n_rep, dim=1)
93
+ y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
94
+ return self.o(y.transpose(1, 2).reshape(B, T, -1))
95
+
96
+
97
+ class FaberBlock(nn.Module):
98
+ def __init__(self, cfg):
99
+ super().__init__()
100
+ self.n1 = FaberRMSNorm(cfg.hidden_size, cfg.rms_norm_eps)
101
+ self.attn = FaberGQA(cfg)
102
+ self.n2 = FaberRMSNorm(cfg.hidden_size, cfg.rms_norm_eps)
103
+ self.gate = nn.Linear(cfg.hidden_size, cfg.ffn_dim, bias=False)
104
+ self.up = nn.Linear(cfg.hidden_size, cfg.ffn_dim, bias=False)
105
+ self.down = nn.Linear(cfg.ffn_dim, cfg.hidden_size, bias=False)
106
+
107
+ def forward(self, x):
108
+ x = x + self.attn(self.n1(x))
109
+ return x + self.down(F.silu(self.gate(self.n2(x))) * self.up(self.n2(x)))
110
+
111
+
112
+ class FaberGenesisForCausalLM(PreTrainedModel):
113
+ config_class = FaberGenesisConfig
114
+
115
+ def __init__(self, config):
116
+ super().__init__(config)
117
+ self.cfg = config
118
+ self.tok = nn.Embedding(config.vocab_size, config.hidden_size)
119
+ self.blocks = nn.ModuleList([FaberBlock(config) for _ in range(config.num_physical_layers)])
120
+ self.final_norm = FaberRMSNorm(config.hidden_size, config.rms_norm_eps)
121
+ self.cass_state = nn.Linear(config.hidden_size, config.cass_state_dim, bias=False)
122
+ self.cass_value = nn.Linear(config.hidden_size, 1)
123
+ self.cass_halt = nn.Linear(config.hidden_size, 1)
124
+ self.post_init()
125
+
126
+ def forward(self, input_ids, labels=None, use_checkpoint=True, **kwargs):
127
+ x = self.tok(input_ids)
128
+ for phys in self.cfg.share_map:
129
+ blk = self.blocks[phys]
130
+ x = checkpoint(blk, x, use_reentrant=False) if (use_checkpoint and self.training and x.requires_grad) else blk(x)
131
+ for _ in range(max(0, self.cfg.n_recurrent - 1)):
132
+ blk = self.blocks[-1]
133
+ x = checkpoint(blk, x, use_reentrant=False) if (use_checkpoint and self.training and x.requires_grad) else blk(x)
134
+ h = self.final_norm(x)
135
+ logits = F.linear(h, self.tok.weight)
136
+ loss = F.cross_entropy(logits.float().view(-1, self.config.vocab_size), labels.view(-1), ignore_index=-100) if labels is not None else None
137
+ return CausalLMOutput(loss=loss, logits=logits)
tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"add_bos_token": true, "add_eos_token": true, "bos_token": "<s>", "chat_template": "{% for m in messages %}{{ m.role + ': ' + m.content + '\n' }}{% endfor %}", "eos_token": "</s>", "model_max_length": 512, "pad_token": "</s>", "tokenizer_class": "PreTrainedTokenizerFast", "unk_token": "<unk>", "vocab_size": 32768}