README.md

---
license: apache-2.0
library_name: pin
tags:
  - weight-sharing
  - model-compression
  - parameter-efficient-fine-tuning
  - convolutional-network
  - cifar10
  - reproducible-research
---

# PIN: folded models, and what you can add to them

**0.9080 on CIFAR-10 from 270,277 stored values**, standing for
2,415,919,104 connections: a fold of 8,939 times. Code for the PIN papers,
[archived on Zenodo](https://doi.org/10.5281/zenodo.21980454) with a DOI.

Code for the PIN papers. Every script here is one file that runs on its own,
downloads or verifies its own data, checks itself before doing any real
work, prints a table, and writes results to disk as it goes. There is no
package to install and no configuration to write: paste one into a hosted
notebook and it runs.

That shape is deliberate. The work is done from a phone as often as from a
desk, so a script that needs four files and a setup step is a script that
does not get run.

---

## What a fold is, in one paragraph

An ordinary weight matrix stores one number per connection. A folded one
stores a much smaller set of VALUES and a PARTITION saying which connections
share which value:

    W[i, j] = v[ idx(i, j) ]

A convolution is one choice of `idx`. So are a patch layer, a butterfly, a
block-circulant matrix, and an arbitrary tying. The framework separates the
partition from the values, which makes the architecture a data structure
rather than a code path, and lets a model be reshaped by swapping the index
while keeping what it learned.

---

## Start here

| you want to | run |
|---|---|
| see a folded model reach 0.9080 on CIFAR-10 | `resnet.py` |
| see the fast path checked against the general one | `sliceconv.py` |
| build the saved bases everything else reads | `testbed_all.py` |
| see a member added to a frozen base | `injection.py` |
| see a model fail to notice its own ignorance | `confidence_member.py` |

`resnet.py` takes about half an hour on a modest GPU and prints its own
timing estimate after ten measured steps, so you can stop it early if that
is not what you want.

---

## Every figure and table in v5, and what produced it

| section | claim | script |
|---|---|---|
| §1 | 0.9080 on CIFAR-10 from 270,277 values | `resnet.py` |
| §2.1 | pooling, and the depth curve inverting | `pooling.py` |
| §2.2 | depth has a low optimum; reach does not predict it | `grow_depth.py`, `depth_law.py` |
| §2.3 | the stacks are chaotic and self-regulating | `criticality.py`, `homeostat.py` |
| §2.4 | blindness, visible in the gradients | `strain.py` |
| §3 | four curricula, four nulls | `grown.py`, `curriculum.py`, `scramble.py` |
| §4.1 | injection at exactly zero disturbance | `injection.py` |
| §4.2 | head capacity, and containment withdrawn | `head_capacity.py`, `provisioning.py`, `bodies.py` |
| §4.3 | a member from five examples, in 0.6 ms | `sample_efficiency.py` |
| §4.4 | the weight as a dial, and negative weights | `member_weight.py` |
| §4.5 | correction is bounded; extension is not | `gate.py`, `ignorance.py` |
| §4.6 | ignorance is invisible at the output | `confidence_member.py` |
| §4.7 | qualifying a member: concede or override | `qualify.py` |
| §4.9 | borrowing across bases | `borrow.py`, `calibration.py` |
| §4.10 | what a buyer is actually paying for | `marketplace.py` |
| §5 | the cascade, and confidence signals | `cascade.py`, `routing_signals.py` |
| §6 | determinism, and the 0.002 noise floor | `determinism.py` |

Supporting: `testbed_all.py` trains the saved bases; `sliceconv.py` holds the
fast convolution and its check; `switch.py` measures what reshaping costs;
`make_v5_figures.py` draws the six figures from the measured values.

---

## Reproducibility

Folded training was not reproducible until recently and now is. GPU atomic
scatter completes in hardware order and floating-point addition is not
associative, so the same seed produced models differing by about 0.002 in
final accuracy. A fixed-order reduction is exact, and it is cheap here
because the partition never changes during training, so the sort is paid
once for the whole run. `determinism.py` measures both.

Two protocol notes, learned the hard way and worth inheriting:

- **Compare paired within a seed.** An unpaired spread misled this project
  three times, once hiding a result at 2.6 deviations.
- **Read best epoch, not final, whenever an arm overfits.** A control with
  more epochs to overfit with once produced an apparent +0.0051 that was
  really +0.0001.

---

## What is not here

The papers withdraw things, and the code keeps the scripts that produced
the withdrawn results as well as the corrections. That is deliberate. A
project that never retracted anything is one nobody checked.

Superseded versions of a script are not included; where a fault mattered it
is described in the file that replaced it, in a comment at the place it
happened. Several of those comments are the most useful documentation in
the repository: an index sized on uniform test data that asked for 149 GB
against a real one, a member applied inside a suppression dead zone, a
storage figure that omitted the head it was compared against.

---

## Requirements

Python 3, NumPy, and CuPy if a GPU is present. Every script falls back to
NumPy on the CPU, more slowly. Datasets are fetched through Keras and
cached; `resnet.py` caches to a directory you choose so a restarted session
does not download CIFAR again.

Nothing else. There is no framework dependency, no custom kernel, and no
compiled extension. A folded convolution is expressed in operations any
array library already has, which is part of the claim.

---

## Citing

The papers carry the results and the reasoning; this repository carries what
produced them. Cite the paper for a finding and this for the method that
produced it.

## Licence

Apache 2.0. It permits commercial use and includes an explicit patent
grant, which matters more here than the licence text: the framework
describes mechanisms someone may want to build on, and a permissive licence
without a patent clause leaves that ambiguous.