lucas-mella commited on
Commit
dcd9f8e
Β·
verified Β·
1 Parent(s): 18b943c

docs: update status - ablation running, paper on Academia.edu

Browse files
Files changed (1) hide show
  1. README.md +366 -389
README.md CHANGED
@@ -1,389 +1,366 @@
1
- ---
2
- license: other
3
- license_name: busl-1.1
4
- license_link: LICENSE
5
- language:
6
- - en
7
- - es
8
- tags:
9
- - code-generation
10
- - pytorch
11
- - transformer
12
- - from-scratch
13
- - bio-inspired
14
- - on-device
15
- - reasoning
16
- - rag
17
- library_name: pytorch
18
- pipeline_tag: text-generation
19
- model-index:
20
- - name: PAMPAr-Coder
21
- results: []
22
- ---
23
-
24
- <p align="center">
25
- <img src="PAMPAR-coder.png" alt="PAMPAr-Coder" width="200" />
26
- </p>
27
-
28
- <h1 align="center">PAMPAr-Coder</h1>
29
-
30
- <p align="center">
31
- <strong>Pure reasoning engine</strong> β€” 62.6M params, local-first, on-device RAG.
32
- </p>
33
-
34
- <p align="center">
35
- <a href="LICENSE"><img src="https://img.shields.io/badge/license-BUSL--1.1-blue" alt="License" /></a>
36
- <a href="https://doi.org/10.57967/hf/8329"><img src="https://img.shields.io/badge/DOI-10.57967%2Fhf%2F8329-blue" alt="DOI" /></a>
37
- <img src="https://img.shields.io/badge/params-62.6M-green" alt="Params" />
38
- <img src="https://img.shields.io/badge/python-3.11%2B-blue" alt="Python" />
39
- <img src="https://img.shields.io/badge/pytorch-2.x-orange" alt="PyTorch" />
40
- </p>
41
-
42
- ---
43
-
44
- ## What is PAMPAr-Coder
45
-
46
- PAMPAr-Coder is a 62.6M parameter language model that **reasons over reference information** rather than memorizing answers. It works like a physicist: it understands the fundamental axioms and can derive solutions for any domain using documentation available on the device.
47
-
48
- - **Weights**: reasoning capability (read docs, understand problems, derive solutions step-by-step)
49
- - **Device**: knowledge via local RAG (Python docs, MDN, man pages, user files)
50
- - **Hardware**: designed to run on consumer hardware (GTX 1650, 4 GB VRAM)
51
-
52
- **Current state**: `v3_train.pt` β€” 98K steps, Mixed Selectivity (FiLM). Classroom system with conversational mentor (Qwen-plus) + 5 bio-inspired mechanisms. Tree of 21 concepts with adaptive prerequisites.
53
-
54
- ---
55
-
56
- ## 2D Architecture (PamparV3)
57
-
58
- ```
59
- tok_emb [48K x 640]
60
- -> TalamoInicial (LLAVES 80% + attn_proj 20% + context_conv)
61
- -> terr_acts [B, L, 4] / zona_acts [B, L, 52]
62
- -> 4 parallel streams (dim=640)
63
-
64
- NivelProfundo x5:
65
- 1. Shared GQA Attention (8 Q heads / 2 KV heads, head_dim=80)
66
- 2. Lightweight Thalamus re-routing
67
- 3. 4 x independent StreamFFN SwiGLU
68
- 4. Lateral gates per stream (bottleneck=128)
69
-
70
- -> norm_f (RMSNorm) -> lm_head (weight-tied, vocab=48K)
71
- ```
72
-
73
- ### The 4 Streams
74
-
75
- | Stream | Brodmann Zones | Processes |
76
- | -------------- | -------------- | --------------------------------- |
77
- | **SYNTAX** | B01-B15 | Keywords, operators, punctuation |
78
- | **SEMANTICS** | B16-B30 | Types, variables, literals |
79
- | **LOGIC** | B31-B42 | Control flow, conditionals, loops |
80
- | **STRUCTURAL** | B43-B52 | Blocks, indentation, scope |
81
-
82
- ### Parameters
83
-
84
- | Parameter | Value |
85
- | ---------------- | ----------- |
86
- | `dim` | 640 |
87
- | `n_streams` | 4 |
88
- | `n_levels` | 5 |
89
- | `n_heads` | 8 |
90
- | `n_kv_heads` | 2 (GQA 4:1) |
91
- | `vocab_size` | 48,000 |
92
- | `max_seq_len` | 4096 |
93
- | **Total params** | **62.6M** |
94
-
95
- ---
96
-
97
- ## Key Innovations
98
-
99
- ### LLAVES System (TalamoInicial)
100
-
101
- - **80% explicit rules**: routing based on code patterns (INT8, pre-computed)
102
- - **20% learned attention**: fine-tuning for ambiguous cases
103
- - Produces `terr_acts` and `zona_acts` with zero inference overhead
104
-
105
- ### 2D Cortical Architecture
106
-
107
- - **4 streams Γ— 5 levels** = grid where rows specialize and columns refine
108
- - **GQA 4:1**: lower VRAM, same quality
109
- - **Lateral gates** (bottleneck 128): cross-stream communication like white-matter fibers
110
- - **Re-routing** per level: the Thalamus adapts which stream leads based on accumulated context
111
-
112
- ### On-Device RAG
113
-
114
- The model uses the machine where it's installed as its knowledge source:
115
-
116
- - Scanner detects OS, packages, available files
117
- - RAGResidual indexes local documentation (FAISS + sentence-transformers)
118
- - The model reasons over references, it doesn't memorize content
119
-
120
- ---
121
-
122
- ## Classroom β€” Conversational Mentor + Bio-Mechanisms
123
-
124
- A learning system where a mentor model (Qwen-plus via DashScope) teaches PamparV3 through dynamic conversations, like a tutor in a chat. The mentor generates unique explanations, examples, and exercises for each lesson β€” the student absorbs knowledge via gradient descent.
125
-
126
- ### Lesson Flow
127
-
128
- ```
129
- 1. StudentProfile selects adaptive concept (21 concepts with prerequisites)
130
- 2. Mentor generates lesson: explanation + example + exercise + solution
131
- 3. Phase A β€” Absorb: train on explanation + example (all tokens)
132
- 4. Phase B β€” Practice: student attempts the exercise
133
- 5. Phase C β€” Correct: mentor evaluates, train on correct solution + replay
134
- 6. Update student profile (mastery per concept)
135
- ```
136
-
137
- ### Concept Tree (CONCEPT_TREE)
138
-
139
- 21 concepts organized in 5 levels with prerequisites:
140
-
141
- | Level | Concepts |
142
- | ----- | --------------------------------------------------------------------- |
143
- | 1 | arithmetic β†’ variables_types β†’ conditionals, strings, functions_basic |
144
- | 2 | loops_for β†’ loops_while, lists β†’ tuples_sets, dicts |
145
- | 3 | recursion, higher_order, generators, error_handling |
146
- | 4 | classes_basic β†’ inheritance, dunder_methods |
147
- | 5 | decorators, context_managers, algorithms, file_io |
148
-
149
- `StudentProfile` tracks mastery per concept and selects adaptively:
150
-
151
- - Prioritizes concepts with attempts but not yet mastered (reinforcement)
152
- - Then new concepts whose prerequisites are met
153
- - Finally spaced review of mastered concepts
154
-
155
- ### Core Mechanisms
156
-
157
- | Mechanism | Purpose |
158
- | -------------------------------------- | ----------------------------------------------------------------- |
159
- | **EWC** (Elastic Weight Consolidation) | Protects important weights β€” penalizes changes to critical params |
160
- | **Replay Buffer** | Mixes new and previous examples (simulates sleep consolidation) |
161
- | **Differential LR** | LLAVES/Thalamus 0.01Γ—, attention 0.1Γ—, embedding 0.1Γ—, FFN 1.0Γ— |
162
- | **Conversational Absorption** | Trains on mentor explanations + examples (knowledge distillation) |
163
-
164
- ### Bio-Mechanisms (`bio_mechanisms.py`)
165
-
166
- 5 mechanisms based on real neuroscience, integrated as post-lesson hooks:
167
-
168
- | Mechanism | Biological Inspiration | Implementation |
169
- | ----------------------- | ------------------------- | -------------------------------------------------------------------------------------- |
170
- | **Neuromodulation** | Dopamine + Norepinephrine | Dynamically modulates LR based on success/error (Γ—0.3 to Γ—3.0) |
171
- | **LTP** | Long-term potentiation | Strengthens `LateralGate.scale` of streams with consistent high activation (Hebb rule) |
172
- | **Sleep Consolidation** | REM + SWS phases | Periodic replay (every 15 lessons): random (REM) + sorted by difficulty (SWS) |
173
- | **Neurogenesis** | New hippocampal neurons | Injects LoRA adapters (rank=8, ~10K params) into StreamFFN when loss > 4.0 |
174
- | **Synaptic Pruning** | Synaptic pruning (~50%) | Reduces `LateralGate.scale < 0.03` every 30 lessons (decay Γ—0.5) |
175
-
176
- All coordinated by `BioOrchestrator.after_lesson()`. Can be disabled with `--no-bio`.
177
-
178
- ### Mentor Pilot Results (5 lessons)
179
-
180
- - Absorption loss: ~7-8 (new content from mentor)
181
- - Exercise loss decreasing: 5.89 β†’ 5.44 β†’ 4.40 β†’ 3.94 β†’ 4.38
182
- - Brain score stable: 88.24% (prior knowledge preservation)
183
- - EWC penalty growing: 0.000002 β†’ 0.000044 (active regularization)
184
- - Each lesson is UNIQUE β€” mentor generates dynamically, no repetition
185
-
186
- ### Usage
187
-
188
- ```bash
189
- # Conversational mentor with Qwen-plus (recommended)
190
- python scripts/classroom_server.py \
191
- --checkpoint checkpoints/v3_train.pt \
192
- --checkpoint-out checkpoints/v3_classroom_mentor.pt \
193
- --teacher qwen --model qwen-plus \
194
- --max-lessons 200 --lr 1e-5 --ewc-lambda 50 --no-bio --no-ui
195
-
196
- # With bio-inspired mechanisms enabled
197
- python scripts/classroom_server.py \
198
- --checkpoint checkpoints/v3_train.pt \
199
- --teacher qwen --model qwen-plus \
200
- --max-lessons 200 --lr 1e-5
201
-
202
- # With web interface (SSE + dashboard)
203
- python scripts/classroom_server.py \
204
- --checkpoint checkpoints/v3_train.pt \
205
- --teacher qwen --port 8787
206
-
207
- # With GitHub Models API (alternative)
208
- python scripts/classroom_server.py \
209
- --checkpoint checkpoints/v3_train.pt \
210
- --teacher github --model gpt-4o-mini
211
-
212
- # Replay a recorded session
213
- # Open sessions/classroom_*.html in browser
214
- ```
215
-
216
- ---
217
-
218
- ## Subsystems
219
-
220
- | Module | Components | Purpose |
221
- | ------------- | --------------------------- | ----------------------------------------------------------------------------------------------- |
222
- | **Model** | `pampar/coder/v3/` | PamparV3: forward, generate, routing, blocks |
223
- | **Memory** | `pampar/memoria/` | ClasificadorPareto (L0-L3), RAGResidual (FAISS), ColaFinetune |
224
- | **Runtime** | `pampar/runtime/` | Agent (orchestrator), Scanner (device), BootProtocol |
225
- | **Skills** | `pampar/skills/` | LectorArchivos (30+ ext), EjecutorCodigo (subprocess) |
226
- | **Inference** | `pampar/inference.py` | JSON-lines stdin/stdout server for VS Code |
227
- | **Classroom** | `scripts/classroom*.py` | Conversational mentor: engine + teacher + curriculum + training + events + memory + persistence |
228
- | **Bio-Mech** | `scripts/bio_mechanisms.py` | 5 neuroscience mechanisms: Neuromod, LTP, Sleep, Neurogenesis, Pruning |
229
-
230
- ---
231
-
232
- ## Installation
233
-
234
- ```bash
235
- git clone https://github.com/lucasmella-stack/PAMPAr-Coder.git
236
- cd PAMPAr-Coder
237
- pip install -r requirements.txt
238
- ```
239
-
240
- ---
241
-
242
- ## Usage
243
-
244
- ### Instantiate the model
245
-
246
- ```python
247
- from pampar.coder.v3 import PamparV3, PRESET_V3
248
- import torch
249
-
250
- model = PamparV3(PRESET_V3)
251
- model.eval()
252
-
253
- # Forward pass
254
- ids = torch.randint(0, 48_000, (1, 64))
255
- with torch.no_grad():
256
- logits, loss, info = model(ids)
257
-
258
- # Autoregressive generation
259
- gen = model.generate(ids, max_tokens=100, temperature=0.8, top_k=50)
260
- ```
261
-
262
- ### Use the Agent (with RAG + Skills)
263
-
264
- ```python
265
- from pampar.runtime import Agente
266
-
267
- agent = Agente(
268
- checkpoint="checkpoints/v3_train.pt",
269
- workspace_root=".",
270
- )
271
- response = agent.responder("how to read a CSV with pandas?")
272
- ```
273
-
274
- ---
275
-
276
- ## Project Structure
277
-
278
- ```
279
- PAMPAr-Coder/
280
- β”œβ”€β”€ pampar/
281
- β”‚ β”œβ”€β”€ coder/v3/ # Active architecture (62.6M)
282
- β”‚ β”‚ β”œβ”€β”€ modelo.py # PamparV3 β€” forward, generate
283
- β”‚ β”‚ β”œβ”€β”€ config.py # ConfigV3 + presets
284
- β”‚ β”‚ β”œβ”€β”€ talamo.py # TalamoInicial β€” routing
285
- β”‚ β”‚ β”œβ”€β”€ bloques.py # GQA, SwiGLU, LateralGate, NivelProfundo
286
- β”‚ β”‚ β”œβ”€β”€ llaves.py # LlavesV2 β€” INT8 lookup
287
- β”‚ β”‚ β”œβ”€β”€ zonas.py # 52 Brodmann Zones
288
- β”‚ β”‚ β”œβ”€β”€ ghidra_probe.py # Read-only instrumentation
289
- β”‚ β”‚ └── engrama_stream.py # Activation memory
290
- β”‚ β”œβ”€β”€ memoria/
291
- β”‚ β”‚ β”œβ”€β”€ clasificador.py # ClasificadorPareto (L0-L3)
292
- β”‚ β”‚ β”œβ”€β”€ rag.py # RAGResidual (FAISS + TF-IDF fallback)
293
- β”‚ β”‚ └── cola_finetune.py # ColaFinetune (auto-SFT buffer)
294
- β”‚ β”œβ”€β”€ skills/
295
- β”‚ β”‚ β”œβ”€β”€ lector_archivos.py # File reader (sandboxed)
296
- β”‚ β”‚ └── ejecutar_codigo.py # Code executor (subprocess)
297
- β”‚ β”œβ”€β”€ runtime/
298
- β”‚ β”‚ β”œβ”€β”€ agente.py # Main orchestrator
299
- β”‚ β”‚ β”œβ”€β”€ scanner.py # Device inspection
300
- β”‚ β”‚ └── boot.py # Boot sequence
301
- β”‚ └── inference.py # JSON-lines server for VS Code
302
- β”œβ”€β”€ scripts/
303
- β”‚ β”œβ”€β”€ classroom.py # ClassroomEngine (~600 lines)
304
- β”‚ β”œβ”€β”€ classroom_curriculum.py# CONCEPT_TREE (21 concepts) + StudentProfile
305
- β”‚ β”œβ”€β”€ classroom_teacher.py # Mentor API (GitHub/OpenRouter/Qwen)
306
- β”‚ β”œβ”€β”€ classroom_training.py # Tokenization + differential LR + train_step
307
- β”‚ β”œβ”€β”€ classroom_events.py # Console event formatting
308
- β”‚ β”œβ”€β”€ classroom_memory.py # EWC + ReplayBuffer + compute_ewc_baseline
309
- β”‚ β”œβ”€β”€ classroom_persistence.py # Checkpoint + session + HTML recording save
310
- β”‚ β”œβ”€β”€ classroom_server.py # HTTP SSE server + CLI (entry point)
311
- β”‚ └── bio_mechanisms.py # 5 bio mechanisms
312
- β”œβ”€β”€ data/tokenizer/
313
- β”‚ └── pampar_48k.model # 48K bilingual vocab (active)
314
- β”œβ”€β”€ checkpoints/ # Model checkpoints (gitignored)
315
- β”œβ”€β”€ tests/ # pytest test suite
316
- └── _archive/ # Pre-refactoring backups
317
- ```
318
-
319
- ---
320
-
321
- ## Understanding the Loss
322
-
323
- | Loss | Meaning |
324
- | ----- | --------------------- |
325
- | ~10.7 | Untrained (log 48000) |
326
- | 7-8 | Random weights |
327
- | 5-7 | Beginning to learn |
328
- | 2-4 | Active learning |
329
- | 1.5-2 | Optimal zone |
330
- | < 1.5 | Topic well learned |
331
- | < 0.7 | Topic mastered |
332
-
333
- ---
334
-
335
- ## Tests
336
-
337
- ```bash
338
- python -m pytest tests/ -v
339
- ```
340
-
341
- 142 tests, all passing.
342
-
343
- ---
344
-
345
- ## Philosophy
346
-
347
- > _"You don't need 72 billion parameters. You need the right architecture and the right axioms."_
348
-
349
- 1. **Reasoning > memorization** β€” the model learns to use references, not to memorize
350
- 2. **The device is the knowledge base** β€” local RAG, not cloud
351
- 3. **Code is structured** β€” 4 specialized streams + LLAVES 80% rules
352
- 4. **Consumer hardware** β€” 1.4 GB VRAM for fp16 training
353
-
354
- ---
355
-
356
- ## Roadmap
357
-
358
- - [x] Territorial architecture (52 Brodmann zones, 4 streams Γ— 5 levels)
359
- - [x] LLAVES system (INT8 routing, 80% rules)
360
- - [x] BPE 48K bilingual tokenizer (ES + code)
361
- - [x] GQA 4:1, SwiGLU, lateral gates
362
- - [x] Memory module (ClasificadorPareto, RAG, ColaFinetune)
363
- - [x] Skills (LectorArchivos, EjecutorCodigo)
364
- - [x] Runtime.Agent (tool-use loop)
365
- - [x] GhidraProbe (read-only diagnostics)
366
- - [x] EngramaStream (activation memory)
367
- - [x] Bio-inspired Classroom (EWC, replay buffer, differential LR, curriculum)
368
- - [x] HTML session recording and replay
369
- - [x] GitHub Models API integration (gpt-4o-mini as teacher)
370
- - [x] Bio-mechanisms: Neuromodulation, LTP, Sleep Consolidation, Neurogenesis, Synaptic Pruning
371
- - [x] Conversational mentor: Qwen-plus generates dynamic lessons as tutor
372
- - [x] CONCEPT_TREE: 21 concepts with adaptive prerequisites
373
- - [x] StudentProfile: per-concept mastery tracking
374
- - [x] Loss masking: -100 on prompt tokens (train only on responses)
375
- - [x] Conversational absorption: train on mentor explanations + examples
376
- - [ ] Multimodal: image/diagram input support
377
- - [ ] Training data expansion (textbook + SFT multi-language)
378
- - [ ] KV cache in generate()
379
- - [ ] Multi-language execution (JS, Rust, Bash)
380
- - [ ] Benchmarks against reference models
381
- - [ ] VS Code extension
382
-
383
- ---
384
-
385
- ## License
386
-
387
- BUSL-1.1 β€” Copyright (c) 2024-2026 Lucas Ricardo Mella Chillemi
388
-
389
- Change Date: April 7, 2030 β€” License converts to Apache-2.0. See [LICENSE](LICENSE) for details.
 
1
+ <p align="center">
2
+ <img src="PAMPAR-coder.png" alt="PAMPAr-Coder" width="200" />
3
+ </p>
4
+
5
+ <h1 align="center">PAMPAr-Coder</h1>
6
+
7
+ <p align="center">
8
+ <strong>Pure reasoning engine</strong> β€” 62.6M params, local-first, on-device RAG.
9
+ </p>
10
+
11
+ <p align="center">
12
+ <a href="LICENSE"><img src="https://img.shields.io/badge/license-BUSL--1.1-blue" alt="License" /></a>
13
+ <a href="https://doi.org/10.57967/hf/8329"><img src="https://img.shields.io/badge/DOI-10.57967%2Fhf%2F8329-blue" alt="DOI" /></a>
14
+ <img src="https://img.shields.io/badge/params-62.6M-green" alt="Params" />
15
+ <img src="https://img.shields.io/badge/python-3.11%2B-blue" alt="Python" />
16
+ <img src="https://img.shields.io/badge/pytorch-2.x-orange" alt="PyTorch" />
17
+ </p>
18
+
19
+ ---
20
+
21
+ ## What is PAMPAr-Coder
22
+
23
+ PAMPAr-Coder is a 62.6M parameter language model that **reasons over reference information** rather than memorizing answers. It works like a physicist: it understands the fundamental axioms and can derive solutions for any domain using documentation available on the device.
24
+
25
+ - **Weights**: reasoning capability (read docs, understand problems, derive solutions step-by-step)
26
+ - **Device**: knowledge via local RAG (Python docs, MDN, man pages, user files)
27
+ - **Hardware**: designed to run on consumer hardware (GTX 1650, 4 GB VRAM)
28
+
29
+ **Current state**: `v3_train.pt` β€” 98K steps, Mixed Selectivity (FiLM). Ablation study running on RTX 3090 (4 experiments Γ— 30K steps). Paper published on [Academia.edu](https://www.academia.edu/works/165626856) β€” DOI: [10.57967/hf/8329](https://doi.org/10.57967/hf/8329).
30
+
31
+ ---
32
+
33
+ ## 2D Architecture (PamparV3)
34
+
35
+ ```
36
+ tok_emb [48K x 640]
37
+ -> TalamoInicial (LLAVES 80% + attn_proj 20% + context_conv)
38
+ -> terr_acts [B, L, 4] / zona_acts [B, L, 52]
39
+ -> 4 parallel streams (dim=640)
40
+
41
+ NivelProfundo x5:
42
+ 1. Shared GQA Attention (8 Q heads / 2 KV heads, head_dim=80)
43
+ 2. Lightweight Thalamus re-routing
44
+ 3. 4 x independent StreamFFN SwiGLU
45
+ 4. Lateral gates per stream (bottleneck=128)
46
+
47
+ -> norm_f (RMSNorm) -> lm_head (weight-tied, vocab=48K)
48
+ ```
49
+
50
+ ### The 4 Streams
51
+
52
+ | Stream | Brodmann Zones | Processes |
53
+ | -------------- | -------------- | --------------------------------- |
54
+ | **SYNTAX** | B01-B15 | Keywords, operators, punctuation |
55
+ | **SEMANTICS** | B16-B30 | Types, variables, literals |
56
+ | **LOGIC** | B31-B42 | Control flow, conditionals, loops |
57
+ | **STRUCTURAL** | B43-B52 | Blocks, indentation, scope |
58
+
59
+ ### Parameters
60
+
61
+ | Parameter | Value |
62
+ | ---------------- | ----------- |
63
+ | `dim` | 640 |
64
+ | `n_streams` | 4 |
65
+ | `n_levels` | 5 |
66
+ | `n_heads` | 8 |
67
+ | `n_kv_heads` | 2 (GQA 4:1) |
68
+ | `vocab_size` | 48,000 |
69
+ | `max_seq_len` | 4096 |
70
+ | **Total params** | **62.6M** |
71
+
72
+ ---
73
+
74
+ ## Key Innovations
75
+
76
+ ### LLAVES System (TalamoInicial)
77
+
78
+ - **80% explicit rules**: routing based on code patterns (INT8, pre-computed)
79
+ - **20% learned attention**: fine-tuning for ambiguous cases
80
+ - Produces `terr_acts` and `zona_acts` with zero inference overhead
81
+
82
+ ### 2D Cortical Architecture
83
+
84
+ - **4 streams Γ— 5 levels** = grid where rows specialize and columns refine
85
+ - **GQA 4:1**: lower VRAM, same quality
86
+ - **Lateral gates** (bottleneck 128): cross-stream communication like white-matter fibers
87
+ - **Re-routing** per level: the Thalamus adapts which stream leads based on accumulated context
88
+
89
+ ### On-Device RAG
90
+
91
+ The model uses the machine where it's installed as its knowledge source:
92
+
93
+ - Scanner detects OS, packages, available files
94
+ - RAGResidual indexes local documentation (FAISS + sentence-transformers)
95
+ - The model reasons over references, it doesn't memorize content
96
+
97
+ ---
98
+
99
+ ## Classroom β€” Conversational Mentor + Bio-Mechanisms
100
+
101
+ A learning system where a mentor model (Qwen-plus via DashScope) teaches PamparV3 through dynamic conversations, like a tutor in a chat. The mentor generates unique explanations, examples, and exercises for each lesson β€” the student absorbs knowledge via gradient descent.
102
+
103
+ ### Lesson Flow
104
+
105
+ ```
106
+ 1. StudentProfile selects adaptive concept (21 concepts with prerequisites)
107
+ 2. Mentor generates lesson: explanation + example + exercise + solution
108
+ 3. Phase A β€” Absorb: train on explanation + example (all tokens)
109
+ 4. Phase B β€” Practice: student attempts the exercise
110
+ 5. Phase C β€” Correct: mentor evaluates, train on correct solution + replay
111
+ 6. Update student profile (mastery per concept)
112
+ ```
113
+
114
+ ### Concept Tree (CONCEPT_TREE)
115
+
116
+ 21 concepts organized in 5 levels with prerequisites:
117
+
118
+ | Level | Concepts |
119
+ | ----- | --------------------------------------------------------------------- |
120
+ | 1 | arithmetic β†’ variables_types β†’ conditionals, strings, functions_basic |
121
+ | 2 | loops_for β†’ loops_while, lists β†’ tuples_sets, dicts |
122
+ | 3 | recursion, higher_order, generators, error_handling |
123
+ | 4 | classes_basic β†’ inheritance, dunder_methods |
124
+ | 5 | decorators, context_managers, algorithms, file_io |
125
+
126
+ `StudentProfile` tracks mastery per concept and selects adaptively:
127
+
128
+ - Prioritizes concepts with attempts but not yet mastered (reinforcement)
129
+ - Then new concepts whose prerequisites are met
130
+ - Finally spaced review of mastered concepts
131
+
132
+ ### Core Mechanisms
133
+
134
+ | Mechanism | Purpose |
135
+ | -------------------------------------- | ----------------------------------------------------------------- |
136
+ | **EWC** (Elastic Weight Consolidation) | Protects important weights β€” penalizes changes to critical params |
137
+ | **Replay Buffer** | Mixes new and previous examples (simulates sleep consolidation) |
138
+ | **Differential LR** | LLAVES/Thalamus 0.01Γ—, attention 0.1Γ—, embedding 0.1Γ—, FFN 1.0Γ— |
139
+ | **Conversational Absorption** | Trains on mentor explanations + examples (knowledge distillation) |
140
+
141
+ ### Bio-Mechanisms (`bio_mechanisms.py`)
142
+
143
+ 5 mechanisms based on real neuroscience, integrated as post-lesson hooks:
144
+
145
+ | Mechanism | Biological Inspiration | Implementation |
146
+ | ----------------------- | ------------------------- | -------------------------------------------------------------------------------------- |
147
+ | **Neuromodulation** | Dopamine + Norepinephrine | Dynamically modulates LR based on success/error (Γ—0.3 to Γ—3.0) |
148
+ | **LTP** | Long-term potentiation | Strengthens `LateralGate.scale` of streams with consistent high activation (Hebb rule) |
149
+ | **Sleep Consolidation** | REM + SWS phases | Periodic replay (every 15 lessons): random (REM) + sorted by difficulty (SWS) |
150
+ | **Neurogenesis** | New hippocampal neurons | Injects LoRA adapters (rank=8, ~10K params) into StreamFFN when loss > 4.0 |
151
+ | **Synaptic Pruning** | Synaptic pruning (~50%) | Reduces `LateralGate.scale < 0.03` every 30 lessons (decay Γ—0.5) |
152
+
153
+ All coordinated by `BioOrchestrator.after_lesson()`. Can be disabled with `--no-bio`.
154
+
155
+ ### Mentor Pilot Results (5 lessons)
156
+
157
+ - Absorption loss: ~7-8 (new content from mentor)
158
+ - Exercise loss decreasing: 5.89 β†’ 5.44 β†’ 4.40 β†’ 3.94 β†’ 4.38
159
+ - Brain score stable: 88.24% (prior knowledge preservation)
160
+ - EWC penalty growing: 0.000002 β†’ 0.000044 (active regularization)
161
+ - Each lesson is UNIQUE β€” mentor generates dynamically, no repetition
162
+
163
+ ### Usage
164
+
165
+ ```bash
166
+ # Conversational mentor with Qwen-plus (recommended)
167
+ python scripts/classroom_server.py \
168
+ --checkpoint checkpoints/v3_train.pt \
169
+ --checkpoint-out checkpoints/v3_classroom_mentor.pt \
170
+ --teacher qwen --model qwen-plus \
171
+ --max-lessons 200 --lr 1e-5 --ewc-lambda 50 --no-bio --no-ui
172
+
173
+ # With bio-inspired mechanisms enabled
174
+ python scripts/classroom_server.py \
175
+ --checkpoint checkpoints/v3_train.pt \
176
+ --teacher qwen --model qwen-plus \
177
+ --max-lessons 200 --lr 1e-5
178
+
179
+ # With web interface (SSE + dashboard)
180
+ python scripts/classroom_server.py \
181
+ --checkpoint checkpoints/v3_train.pt \
182
+ --teacher qwen --port 8787
183
+
184
+ # With GitHub Models API (alternative)
185
+ python scripts/classroom_server.py \
186
+ --checkpoint checkpoints/v3_train.pt \
187
+ --teacher github --model gpt-4o-mini
188
+
189
+ # Replay a recorded session
190
+ # Open sessions/classroom_*.html in browser
191
+ ```
192
+
193
+ ---
194
+
195
+ ## Subsystems
196
+
197
+ | Module | Components | Purpose |
198
+ | ------------- | --------------------------- | ----------------------------------------------------------------------------------------------- |
199
+ | **Model** | `pampar/coder/v3/` | PamparV3: forward, generate, routing, blocks |
200
+ | **Memory** | `pampar/memoria/` | ClasificadorPareto (L0-L3), RAGResidual (FAISS), ColaFinetune |
201
+ | **Runtime** | `pampar/runtime/` | Agent (orchestrator), Scanner (device), BootProtocol |
202
+ | **Skills** | `pampar/skills/` | LectorArchivos (30+ ext), EjecutorCodigo (subprocess) |
203
+ | **Inference** | `pampar/inference.py` | JSON-lines stdin/stdout server for VS Code |
204
+ | **Classroom** | `scripts/classroom*.py` | Conversational mentor: engine + teacher + curriculum + training + events + memory + persistence |
205
+ | **Bio-Mech** | `scripts/bio_mechanisms.py` | 5 neuroscience mechanisms: Neuromod, LTP, Sleep, Neurogenesis, Pruning |
206
+
207
+ ---
208
+
209
+ ## Installation
210
+
211
+ ```bash
212
+ git clone https://github.com/lucasmella-stack/PAMPAr-Coder.git
213
+ cd PAMPAr-Coder
214
+ pip install -r requirements.txt
215
+ ```
216
+
217
+ ---
218
+
219
+ ## Usage
220
+
221
+ ### Instantiate the model
222
+
223
+ ```python
224
+ from pampar.coder.v3 import PamparV3, PRESET_V3
225
+ import torch
226
+
227
+ model = PamparV3(PRESET_V3)
228
+ model.eval()
229
+
230
+ # Forward pass
231
+ ids = torch.randint(0, 48_000, (1, 64))
232
+ with torch.no_grad():
233
+ logits, loss, info = model(ids)
234
+
235
+ # Autoregressive generation
236
+ gen = model.generate(ids, max_tokens=100, temperature=0.8, top_k=50)
237
+ ```
238
+
239
+ ### Use the Agent (with RAG + Skills)
240
+
241
+ ```python
242
+ from pampar.runtime import Agente
243
+
244
+ agent = Agente(
245
+ checkpoint="checkpoints/v3_train.pt",
246
+ workspace_root=".",
247
+ )
248
+ response = agent.responder("how to read a CSV with pandas?")
249
+ ```
250
+
251
+ ---
252
+
253
+ ## Project Structure
254
+
255
+ ```
256
+ PAMPAr-Coder/
257
+ β”œβ”€β”€ pampar/
258
+ β”‚ β”œβ”€β”€ coder/v3/ # Active architecture (62.6M)
259
+ β”‚ β”‚ β”œβ”€β”€ modelo.py # PamparV3 β€” forward, generate
260
+ β”‚ β”‚ β”œβ”€β”€ config.py # ConfigV3 + presets
261
+ β”‚ β”‚ β”œβ”€β”€ talamo.py # TalamoInicial β€” routing
262
+ β”‚ β”‚ β”œβ”€β”€ bloques.py # GQA, SwiGLU, LateralGate, NivelProfundo
263
+ β”‚ β”‚ β”œβ”€β”€ llaves.py # LlavesV2 β€” INT8 lookup
264
+ β”‚ β”‚ β”œβ”€β”€ zonas.py # 52 Brodmann Zones
265
+ β”‚ β”‚ β”œβ”€β”€ ghidra_probe.py # Read-only instrumentation
266
+ β”‚ β”‚ └── engrama_stream.py # Activation memory
267
+ β”‚ β”œβ”€β”€ memoria/
268
+ β”‚ β”‚ β”œβ”€β”€ clasificador.py # ClasificadorPareto (L0-L3)
269
+ β”‚ β”‚ β”œβ”€β”€ rag.py # RAGResidual (FAISS + TF-IDF fallback)
270
+ β”‚ β”‚ └── cola_finetune.py # ColaFinetune (auto-SFT buffer)
271
+ β”‚ β”œβ”€β”€ skills/
272
+ β”‚ β”‚ β”œβ”€β”€ lector_archivos.py # File reader (sandboxed)
273
+ β”‚ β”‚ └── ejecutar_codigo.py # Code executor (subprocess)
274
+ β”‚ β”œβ”€β”€ runtime/
275
+ β”‚ β”‚ β”œβ”€β”€ agente.py # Main orchestrator
276
+ β”‚ β”‚ β”œβ”€β”€ scanner.py # Device inspection
277
+ β”‚ β”‚ └── boot.py # Boot sequence
278
+ β”‚ └── inference.py # JSON-lines server for VS Code
279
+ β”œβ”€β”€ scripts/
280
+ β”‚ β”œβ”€β”€ classroom.py # ClassroomEngine (~600 lines)
281
+ β”‚ β”œβ”€β”€ classroom_curriculum.py# CONCEPT_TREE (21 concepts) + StudentProfile
282
+ β”‚ β”œβ”€β”€ classroom_teacher.py # Mentor API (GitHub/OpenRouter/Qwen)
283
+ β”‚ β”œβ”€β”€ classroom_training.py # Tokenization + differential LR + train_step
284
+ β”‚ β”œβ”€β”€ classroom_events.py # Console event formatting
285
+ β”‚ β”œβ”€β”€ classroom_memory.py # EWC + ReplayBuffer + compute_ewc_baseline
286
+ β”‚ β”œβ”€β”€ classroom_persistence.py # Checkpoint + session + HTML recording save
287
+ β”‚ β”œβ”€β”€ classroom_server.py # HTTP SSE server + CLI (entry point)
288
+ β”‚ └── bio_mechanisms.py # 5 bio mechanisms
289
+ β”œβ”€β”€ data/tokenizer/
290
+ β”‚ └── pampar_48k.model # 48K bilingual vocab (active)
291
+ β”œβ”€β”€ checkpoints/ # Model checkpoints (gitignored)
292
+ β”œβ”€β”€ tests/ # pytest test suite
293
+ └── _archive/ # Pre-refactoring backups
294
+ ```
295
+
296
+ ---
297
+
298
+ ## Understanding the Loss
299
+
300
+ | Loss | Meaning |
301
+ | ----- | --------------------- |
302
+ | ~10.7 | Untrained (log 48000) |
303
+ | 7-8 | Random weights |
304
+ | 5-7 | Beginning to learn |
305
+ | 2-4 | Active learning |
306
+ | 1.5-2 | Optimal zone |
307
+ | < 1.5 | Topic well learned |
308
+ | < 0.7 | Topic mastered |
309
+
310
+ ---
311
+
312
+ ## Tests
313
+
314
+ ```bash
315
+ python -m pytest tests/ -v
316
+ ```
317
+
318
+ 142 tests, all passing.
319
+
320
+ ---
321
+
322
+ ## Philosophy
323
+
324
+ > _"You don't need 72 billion parameters. You need the right architecture and the right axioms."_
325
+
326
+ 1. **Reasoning > memorization** β€” the model learns to use references, not to memorize
327
+ 2. **The device is the knowledge base** β€” local RAG, not cloud
328
+ 3. **Code is structured** β€” 4 specialized streams + LLAVES 80% rules
329
+ 4. **Consumer hardware** β€” 1.4 GB VRAM for fp16 training
330
+
331
+ ---
332
+
333
+ ## Roadmap
334
+
335
+ - [x] Territorial architecture (52 Brodmann zones, 4 streams Γ— 5 levels)
336
+ - [x] LLAVES system (INT8 routing, 80% rules)
337
+ - [x] BPE 48K bilingual tokenizer (ES + code)
338
+ - [x] GQA 4:1, SwiGLU, lateral gates
339
+ - [x] Memory module (ClasificadorPareto, RAG, ColaFinetune)
340
+ - [x] Skills (LectorArchivos, EjecutorCodigo)
341
+ - [x] Runtime.Agent (tool-use loop)
342
+ - [x] GhidraProbe (read-only diagnostics)
343
+ - [x] EngramaStream (activation memory)
344
+ - [x] Bio-inspired Classroom (EWC, replay buffer, differential LR, curriculum)
345
+ - [x] HTML session recording and replay
346
+ - [x] GitHub Models API integration (gpt-4o-mini as teacher)
347
+ - [x] Bio-mechanisms: Neuromodulation, LTP, Sleep Consolidation, Neurogenesis, Synaptic Pruning
348
+ - [x] Conversational mentor: Qwen-plus generates dynamic lessons as tutor
349
+ - [x] CONCEPT_TREE: 21 concepts with adaptive prerequisites
350
+ - [x] StudentProfile: per-concept mastery tracking
351
+ - [x] Loss masking: -100 on prompt tokens (train only on responses)
352
+ - [x] Conversational absorption: train on mentor explanations + examples
353
+ - [ ] Multimodal: image/diagram input support
354
+ - [ ] Training data expansion (textbook + SFT multi-language)
355
+ - [ ] KV cache in generate()
356
+ - [ ] Multi-language execution (JS, Rust, Bash)
357
+ - [ ] Benchmarks against reference models
358
+ - [ ] VS Code extension
359
+
360
+ ---
361
+
362
+ ## License
363
+
364
+ BUSL-1.1 β€” Copyright (c) 2024-2026 Lucas Ricardo Mella Chillemi
365
+
366
+ Change Date: April 7, 2030 β€” License converts to Apache-2.0. See [LICENSE](LICENSE) for details.