--- license: apache-2.0 library_name: pin tags: - weight-sharing - model-compression - parameter-efficient-fine-tuning - convolutional-network - cifar10 - reproducible-research --- # PIN: folded models, and what you can add to them **0.9080 on CIFAR-10 from 270,277 stored values**, standing for 2,415,919,104 connections: a fold of 8,939 times. Code for the PIN papers, [archived on Zenodo](https://doi.org/10.5281/zenodo.21980454) with a DOI. Code for the PIN papers. Every script here is one file that runs on its own, downloads or verifies its own data, checks itself before doing any real work, prints a table, and writes results to disk as it goes. There is no package to install and no configuration to write: paste one into a hosted notebook and it runs. That shape is deliberate. The work is done from a phone as often as from a desk, so a script that needs four files and a setup step is a script that does not get run. --- ## What a fold is, in one paragraph An ordinary weight matrix stores one number per connection. A folded one stores a much smaller set of VALUES and a PARTITION saying which connections share which value: W[i, j] = v[ idx(i, j) ] A convolution is one choice of `idx`. So are a patch layer, a butterfly, a block-circulant matrix, and an arbitrary tying. The framework separates the partition from the values, which makes the architecture a data structure rather than a code path, and lets a model be reshaped by swapping the index while keeping what it learned. --- ## Start here | you want to | run | |---|---| | see a folded model reach 0.9080 on CIFAR-10 | `resnet.py` | | see the fast path checked against the general one | `sliceconv.py` | | build the saved bases everything else reads | `testbed_all.py` | | see a member added to a frozen base | `injection.py` | | see a model fail to notice its own ignorance | `confidence_member.py` | `resnet.py` takes about half an hour on a modest GPU and prints its own timing estimate after ten measured steps, so you can stop it early if that is not what you want. --- ## Every figure and table in v5, and what produced it | section | claim | script | |---|---|---| | §1 | 0.9080 on CIFAR-10 from 270,277 values | `resnet.py` | | §2.1 | pooling, and the depth curve inverting | `pooling.py` | | §2.2 | depth has a low optimum; reach does not predict it | `grow_depth.py`, `depth_law.py` | | §2.3 | the stacks are chaotic and self-regulating | `criticality.py`, `homeostat.py` | | §2.4 | blindness, visible in the gradients | `strain.py` | | §3 | four curricula, four nulls | `grown.py`, `curriculum.py`, `scramble.py` | | §4.1 | injection at exactly zero disturbance | `injection.py` | | §4.2 | head capacity, and containment withdrawn | `head_capacity.py`, `provisioning.py`, `bodies.py` | | §4.3 | a member from five examples, in 0.6 ms | `sample_efficiency.py` | | §4.4 | the weight as a dial, and negative weights | `member_weight.py` | | §4.5 | correction is bounded; extension is not | `gate.py`, `ignorance.py` | | §4.6 | ignorance is invisible at the output | `confidence_member.py` | | §4.7 | qualifying a member: concede or override | `qualify.py` | | §4.9 | borrowing across bases | `borrow.py`, `calibration.py` | | §4.10 | what a buyer is actually paying for | `marketplace.py` | | §5 | the cascade, and confidence signals | `cascade.py`, `routing_signals.py` | | §6 | determinism, and the 0.002 noise floor | `determinism.py` | Supporting: `testbed_all.py` trains the saved bases; `sliceconv.py` holds the fast convolution and its check; `switch.py` measures what reshaping costs; `make_v5_figures.py` draws the six figures from the measured values. --- ## Reproducibility Folded training was not reproducible until recently and now is. GPU atomic scatter completes in hardware order and floating-point addition is not associative, so the same seed produced models differing by about 0.002 in final accuracy. A fixed-order reduction is exact, and it is cheap here because the partition never changes during training, so the sort is paid once for the whole run. `determinism.py` measures both. Two protocol notes, learned the hard way and worth inheriting: - **Compare paired within a seed.** An unpaired spread misled this project three times, once hiding a result at 2.6 deviations. - **Read best epoch, not final, whenever an arm overfits.** A control with more epochs to overfit with once produced an apparent +0.0051 that was really +0.0001. --- ## What is not here The papers withdraw things, and the code keeps the scripts that produced the withdrawn results as well as the corrections. That is deliberate. A project that never retracted anything is one nobody checked. Superseded versions of a script are not included; where a fault mattered it is described in the file that replaced it, in a comment at the place it happened. Several of those comments are the most useful documentation in the repository: an index sized on uniform test data that asked for 149 GB against a real one, a member applied inside a suppression dead zone, a storage figure that omitted the head it was compared against. --- ## Requirements Python 3, NumPy, and CuPy if a GPU is present. Every script falls back to NumPy on the CPU, more slowly. Datasets are fetched through Keras and cached; `resnet.py` caches to a directory you choose so a restarted session does not download CIFAR again. Nothing else. There is no framework dependency, no custom kernel, and no compiled extension. A folded convolution is expressed in operations any array library already has, which is part of the claim. --- ## Citing The papers carry the results and the reasoning; this repository carries what produced them. Cite the paper for a finding and this for the method that produced it. ## Licence Apache 2.0. It permits commercial use and includes an explicit patent grant, which matters more here than the licence text: the framework describes mechanisms someone may want to build on, and a permissive licence without a patent clause leaves that ambiguous.