File size: 2,209 Bytes
9f723ea f893642 9f723ea | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | ---
library_name: transformers
base_model: Qwen/Qwen3-4B-Instruct-2507
tags:
- program-as-weights
- compiler
- lora
- hypernetwork
pipeline_tag: text-generation
---
# paw-4b-gpt2 — ProgramAsWeights "Compact" compiler
This is the **Compact** compiler from **ProgramAsWeights (PAW)**. Given a natural-language **spec**, it emits a tiny per-task **program** — a LoRA adapter — that runs locally on a **GPT-2 (124M)** interpreter (small enough to run in the browser).
It is the model invoked by `paw.compile(spec, compiler="paw-4b-gpt2")`.
- Compiler base model: `Qwen/Qwen3-4B-Instruct-2507`
- Target interpreter: **a custom GPT-2 (124M)** whose positional embeddings are extended from 1024 → 2048 (`n_ctx=2048`); tokenizer is stock GPT-2 BPE.
- Snapshot: `20260406` (see git tag `20260406`)
## Contents
- `compiler/` — a finetuned **Qwen3-4B-Instruct-2507** causal LM (the compiler).
- `lora_mapper.pt` — the mapper head (trunk + coefficient head + learnable LoRA basis matrices) that turns the compiler's hidden states into a LoRA program.
- `meta.json` — `lora_rank=64`, `lora_alpha=16`, `lora_num_bases=64`, `prefix_steps=64`, target modules `[c_attn, c_proj, c_fc]`.
## How it works
1. The 4B compiler generates a short "pseudo-program" (a task description plus a few I/O examples) from the spec.
2. The text `chat_template(spec) + pseudo-program + 64 prefix tokens` is run through the compiler; the mapper reads the 64 prefix hidden states and emits per-layer LoRA `A`/`B` matrices as a learned mixture of basis matrices.
3. The resulting LoRA (about 5 MB) is the **program**. It loads onto the GPT-2 interpreter and runs locally/offline (including in-browser).
## Status
- Inference/runtime SDK (load + run a compiled program locally): https://github.com/programasweights/programasweights-python (browser SDK: https://github.com/programasweights/programasweights-js)
- The cleaned compile/runtime code and the arXiv preprint ("Program-as-Weights: A Programming Paradigm for Fuzzy Functions", AIware 2026) will be public by Jul 6, 2026. An uncleaned reference snapshot is at https://anonymous.4open.science/r/programasweights
- Live demo + program hub: https://programasweights.com
|