| --- |
| library_name: transformers |
| base_model: Qwen/Qwen3-4B-Instruct-2507 |
| tags: |
| - program-as-weights |
| - compiler |
| - lora |
| - hypernetwork |
| pipeline_tag: text-generation |
| --- |
| |
| # paw-4b-gpt2 — ProgramAsWeights "Compact" compiler |
|
|
| This is the **Compact** compiler from **ProgramAsWeights (PAW)**. Given a natural-language **spec**, it emits a tiny per-task **program** — a LoRA adapter — that runs locally on a **GPT-2 (124M)** interpreter (small enough to run in the browser). |
|
|
| It is the model invoked by `paw.compile(spec, compiler="paw-4b-gpt2")`. |
|
|
| - Compiler base model: `Qwen/Qwen3-4B-Instruct-2507` |
| - Target interpreter: **a custom GPT-2 (124M)** whose positional embeddings are extended from 1024 → 2048 (`n_ctx=2048`); tokenizer is stock GPT-2 BPE. |
| - Snapshot: `20260406` (see git tag `20260406`) |
|
|
| ## Contents |
|
|
| - `compiler/` — a finetuned **Qwen3-4B-Instruct-2507** causal LM (the compiler). |
| - `lora_mapper.pt` — the mapper head (trunk + coefficient head + learnable LoRA basis matrices) that turns the compiler's hidden states into a LoRA program. |
| - `meta.json` — `lora_rank=64`, `lora_alpha=16`, `lora_num_bases=64`, `prefix_steps=64`, target modules `[c_attn, c_proj, c_fc]`. |
|
|
| ## How it works |
|
|
| 1. The 4B compiler generates a short "pseudo-program" (a task description plus a few I/O examples) from the spec. |
| 2. The text `chat_template(spec) + pseudo-program + 64 prefix tokens` is run through the compiler; the mapper reads the 64 prefix hidden states and emits per-layer LoRA `A`/`B` matrices as a learned mixture of basis matrices. |
| 3. The resulting LoRA (about 5 MB) is the **program**. It loads onto the GPT-2 interpreter and runs locally/offline (including in-browser). |
|
|
| ## Status |
|
|
| - Inference/runtime SDK (load + run a compiled program locally): https://github.com/programasweights/programasweights-python (browser SDK: https://github.com/programasweights/programasweights-js) |
| - The cleaned compile/runtime code and the arXiv preprint ("Program-as-Weights: A Programming Paradigm for Fuzzy Functions", AIware 2026) will be public by Jul 6, 2026. An uncleaned reference snapshot is at https://anonymous.4open.science/r/programasweights |
| - Live demo + program hub: https://programasweights.com |
|
|
