ALGORITHM.md

# HVCE v4 Algorithm

## Core name

**Proof-Carrying Omni-State Compression**

HVCE v4 is not one compressor. It is a verified portfolio machine. For every block or microgroup it searches a finite menu of reversible descriptions and stores the smallest exact proof-carrying representation.

A representation is accepted only when it can be decoded deterministically and the decoded bytes match the stored SHA-256 digest.

## Archive-level pipeline

```text
input tree
  -> safe path normalization
  -> metadata capture
  -> small-file solid micro-pack
  -> content-defined chunking for large files
  -> exact duplicate detection
  -> near-duplicate sparse XOR patch search
  -> per-unit representation search
  -> payload assembly
  -> compact compressed manifest
  -> optional whole-archive password envelope
  -> optional GF(256) recovery tail
```

## Compression branches

### 1. Entropy-respect classifier

HVCE samples each unit and estimates byte entropy. If the unit looks random, encrypted, or already compressed, v4 skips expensive symbolic modeling. This directly fixes the usual failure mode where a research compressor wastes time trying to compress entropy-saturated inputs.

For random/high-entropy blocks the strongest correct behavior is:

```text
store raw, dedupe if repeated, patch if near-duplicate, verify, optionally encrypt/recover
```

It is not mathematically legitimate to promise universal shrinkage on truly random data.

### 2. SPWSE / world-state recipes

Exact recipes include:

- `recipe_constant`: one repeated byte.
- `recipe_periodic`: one finite period repeated to length `n`.
- `recipe_sparse_zlib`: dominant background byte plus sparse defects.
- `recipe_rle_zlib`: run-length stream compressed by zlib.
- `recipe_polyword`: exact finite-difference polynomial streams over 8/16/32/64-bit little-endian words, degree ≤ 3.
- `recipe_rank1_2d8_zlib`: separable 2D byte field `row[y] + col[x] mod 256` plus sparse exact defects.

This is the main “world-state” idea: do not encode the observations when a compact deterministic state generator plus residuals is shorter.

### 3. Nanophotonic/vector transforms

Transforms are reversible bases that expose lower entropy to standard codecs:

- `delta8`
- `delta16le`
- `delta32le`
- `bitplane`
- `nibbleplane`
- `neural4`

Each transform is followed by a conventional backend such as zlib, bz2, or lzma. The decoder reverses the stack in the opposite order.

### 4. Causal neural residual branch

`neural4` is a deterministic integer predictor. It uses four prior bytes, bounded integer weights, and an online update rule. It stores residual bytes, not a neural model file. This preserves exact decoding and avoids hidden side information.

### 5. Cross-file archive-state branch

HVCE keeps a bounded cache of prior decoded chunks by length. A new chunk can be stored as:

```text
reference chunk id + sparse XOR defects
```

This is especially relevant for versioned media, model checkpoints, game assets, incremental backups, and repeated encrypted-looking payloads. The byte stream may look incompressible locally, yet the archive as a world-state object may contain obvious temporal redundancy.

### 6. Solid micro-pack branch

Many ordinary office folders contain hundreds or thousands of tiny files. Ordinary ZIP stores each member separately and pays repeated headers and dictionary resets. HVCE v4 packs low-entropy tiny files into solid microgroups before compression. Individual file boundaries are preserved by manifest offsets.

### 7. Header compression

The manifest is compressed with whichever of zlib-9 or lzma-extreme is smaller. Encrypted archives hide the whole manifest, including names and directory structure.

### 8. Password envelope

Password mode uses:

```text
PBKDF2-HMAC-SHA256(password, salt, iterations) -> 64 bytes
first 32 bytes  -> ChaCha20 encryption key
second 32 bytes -> HMAC-SHA256 authentication key
```

The whole private header and payload are encrypted and authenticated. Extraction verifies the HMAC before decryption.

### 9. Recovery records

The optional recovery tail uses two parity equations over GF(256):

```text
P0 = xor(shard_i)
P1 = xor((i+1) * shard_i)
```

Given shard hashes, HVCE can detect corrupted shards and recover up to two corrupted shards when the archive length is unchanged and the recovery tail is intact.

This is not a full replacement for external backups. It is a practical archive-integrity layer comparable in spirit to recovery records in mature archive tools.

## Complexity

Let `N` be input bytes and `B` be block size.

- Fast profile: near-linear in `N`, with early entropy gates.
- Balanced profile: linear plus portfolio trials on compressible blocks.
- Max profile: more exhaustive transform/codec search and should be used for release archives, not hot paths.

## Decode contract

The decoder requires only:

- Python standard library;
- the `.hvce` file;
- password if encrypted.

No hidden model, internet access, external dictionary, or hardware-specific side information is required.