README.md

---
language:
- en
license: mit
tags:
- text-generation
- transformer-decoder
- embeddings
- mpnet
- crypto
- pooled-embeddings
- social-media
library_name: pytorch
pipeline_tag: text-generation
base_model: sentence-transformers/all-mpnet-base-v2
---

# Aparecium v2 – Pooled MPNet Reverser (S1 Baseline)

## Summary

- **Task**: Reconstruct natural-language crypto social-media posts from a **single pooled MPNet embedding** (reverse embedding).
- **Focus**: Crypto domain (social-media posts / short-form content).
- **Checkpoint**: `aparecium_v2_s1.pt` — S1 supervised baseline, trained on synthetic crypto social-media posts.
- **Input contract**: a **pooled** `all-mpnet-base-v2` vector of shape `(768,)`, *not* a token-level `(seq_len, 768)` matrix.
- **Code**: this repo only hosts weights; loading & decoding are implemented in the Aparecium codebase
  (the v2 training repo and service are analogous in spirit to the v1 project  
  [`SentiChain/aparecium-seq2seq-reverser`](https://huggingface.co/SentiChain/aparecium-seq2seq-reverser)).

This is a **pooled-embedding variant** of Aparecium, distinct from the original token-level seq2seq reverser described in  
[`SentiChain/aparecium-seq2seq-reverser`](https://huggingface.co/SentiChain/aparecium-seq2seq-reverser).

---

## Intended use

- **Research / engineering**:
  - Study how much crypto-domain information is recoverable from a single pooled embedding.
  - Prototype tools around embedding interpretability, diagnostics, and “gist reconstruction” from vectors.
- **Not intended** for:
  - Reconstructing private, user-identifying, or sensitive content.
  - Any de‑anonymization of embedding corpora.

Reconstruction quality depends heavily on:

- The upstream encoder (`sentence-transformers/all-mpnet-base-v2`),
- Domain match (crypto social-media posts vs. your data),
- Decode settings (beam vs. sampling, constraints, reranking).

---

## Model architecture

On the encoder side, we assume a **pooled MPNet** encoder:

- Recommended: `sentence-transformers/all-mpnet-base-v2` (768‑D pooled output).

On the decoder side, v2 uses the Aparecium components:

- **EmbAdapter**:
  - Input: pooled vector `e ∈ R^768`.
  - Output: pseudo‑sequence memory `H ∈ R^{B × S × D}` suitable for a transformer decoder (multi‑scale).
- **Sketcher**:
  - Lightweight network producing a “plan” and simple control flags (e.g., URL presence) from `e`.
  - In the S1 baseline checkpoint, it is trained but only lightly used at inference.
- **RealizerDecoder**:
  - Transformer decoder (GPT‑style) with:
    - `d_model = 768`
    - `n_layer = 12`
    - `n_head = 8`
    - `d_ff = 3072`
    - Dropout ≈ 0.1
  - Consumes `H` as cross‑attention memory and generates text tokens.

Decoding:

- Deterministic beam search or sampling, with optional:
  - **Constraints** (e.g., require certain tickers/hashtags/amounts based on a plan).
  - **Surrogate similarity scorer `r(x, e)`** for reranking candidates.
  - **Final MPNet cosine rerank** across top‑K candidates.

The `aparecium_v2_s1.pt` checkpoint contains the adapter, sketcher, decoder, and tokenizer name, matching the training repo layout.

---

## Training data and provenance

- **Source**: synthetic crypto social-media posts generated via OpenAI models into a DB (e.g., `tweets.db`).
- **Domain**:
  - Crypto markets, DeFi, L2s, MEV, governance, NFTs, etc.
- **Preparation (v2 pipeline)**:
  1. Extract raw text from the DB into JSONL.
  2. Embed each tweet with `sentence-transformers/all-mpnet-base-v2`:
     - `embedding ∈ R^768` (pooled), L2‑normalized.
     - Optionally store a simple “plan” (tickers, hashtags, amounts, addresses).
  3. Split into train/val/test and shard into JSONL files.

No real social‑media content is used; all posts are synthetic, similar in spirit to the v1 project  
[`SentiChain/aparecium-seq2seq-reverser`](https://huggingface.co/SentiChain/aparecium-seq2seq-reverser).

---

## Training procedure (S1 baseline regimen)

This checkpoint corresponds to **S1 supervised training only** (no SCST/RL):

- Objective: teacher‑forcing cross‑entropy over the crypto tweet text, given the pooled embedding.
- Optimizer: AdamW
- Typical hyperparameters (baseline run):
  - Batch size: 64
  - Max length: 96 tokens (tweets)
  - Learning rate: 3e‑4 (cosine decay), warmup ~1k steps
  - Weight decay: 0.01
  - Grad clip: 1.0
  - Dropout: 0.1
- Data:
  - ~100k synthetic crypto tweets (train/val split).
  - Embeddings precomputed via `all-mpnet-base-v2` and normalized.
- Checkpointing:
  - Save final weights as `aparecium_v2_s1.pt` once training plateaus on validation cross‑entropy.

Future work (not in this checkpoint):

- SCST RL (S2) with a reward combining MPNet cosine, surrogate `r`, repetition penalty, and entity coverage.
- Stronger constraints and rerank policies as described in the training plan.

---

## Evaluation protocol (baseline qualitative)

This repo does **not** include a full eval harness. The S1 baseline was validated qualitatively:

- Sample 10–20 crypto sentences (held‑out).
- For each:
  1. Embed text with `all-mpnet-base-v2` (pooled, normalized).
  2. Invert with Aparecium v2 S1 (beam search + rerank).
  3. Re‑embed the generated text with MPNet and compute cosine with the original embedding.

For a v1‑style, large‑scale evaluation (crypto/equities split, cosine statistics, degeneracy rate, domain drift), refer to the v1 model card:  
[`SentiChain/aparecium-seq2seq-reverser`](https://huggingface.co/SentiChain/aparecium-seq2seq-reverser).

---

## Input contract and usage

**Input** (v2, S1 baseline):

- A **single pooled MPNet embedding** (crypto tweet) of shape `(768,)`, L2‑normalized.
- Recommended encoder: `sentence-transformers/all-mpnet-base-v2` from `sentence-transformers`.

Do **not** pass a token‑level `(seq_len, 768)` matrix – that is the contract for the v1 seq2seq model  
[`SentiChain/aparecium-seq2seq-reverser`](https://huggingface.co/SentiChain/aparecium-seq2seq-reverser), not this checkpoint.

**Usage pattern (high level, pseudocode)**:

```python
import torch, json
from sentence_transformers import SentenceTransformer

# 1) Pooled MPNet embedding
mpnet = SentenceTransformer("sentence-transformers/all-mpnet-base-v2",
                            device="cuda" if torch.cuda.is_available() else "cpu")
text = "Ethereum L2 blob fees spiked after EIP-4844; MEV still shapes order flow."
e = mpnet.encode([text], convert_to_numpy=True, normalize_embeddings=True)[0]  # (768,)

# 2) Load Aparecium v2 S1 checkpoint
ckpt = torch.load("aparecium_v2_s1.pt", map_location="cpu")

# 3) Recreate models from the Aparecium codebase (not included in this HF repo)
# from aparecium.aparecium.models.emb_adapter import EmbAdapter
# from aparecium.aparecium.models.decoder import RealizerDecoder
# from aparecium.aparecium.models.sketcher import Sketcher
# from aparecium.aparecium.utils.tokens import build_tokenizer
# and run the same decoding logic as in `aparecium/infer/service.py` or
# `aparecium/scripts/invert_once.py`.

# 4) Use beam search / constraints / reranking as in the training repo.
```

To actually use the model, you need the Aparecium codebase (training repo) where the `EmbAdapter`, `Sketcher`, `RealizerDecoder`, constraints, and decoding functions are defined.

---

## Limitations and responsible use

- Outputs are *approximations* of the original text under the MPNet embedding and LM prior:
  - They aim to preserve semantic gist and domain entities,
  - They are **not exact reconstructions**.
- The model can:
  - Produce generic phrasing,
  - Over‑use crypto buzzwords/hashtags,
  - Occasionally show noisy punctuation/emoji.
- Data are synthetic; domain semantics might differ from real social‑media distributions.
- Do **not** use this model to attempt to reconstruct sensitive or private user content from embeddings.

---

## Reproducibility (high‑level)

To reproduce or extend this checkpoint:

1. **Prepare data**:
   - Generate synthetic crypto tweets (or your own domain) into a DB (e.g., SQLite).
   - Extract raw text to `train/val/test` JSONL.
   - Embed with `all-mpnet-base-v2` (pooled 768‑D) and save as JSONL with `{"text","embedding","plan"}` fields.
2. **Train S1**:
   - Use the Aparecium v2 trainer (S1 supervised) with:
     - `batch_size ≈ 64`, `max_len ≈ 96`, `lr ≈ 3e-4`, cosine scheduler, warmup steps.
   - Train until validation cross‑entropy and cosine proxy metrics plateau.
3. **Optional**:
   - Train surrogate similarity scorer `r` for reranking.
   - Add SCST RL (S2) if you implement the safe reward/decoding policies.
4. **Evaluate**:
   - Build a small evaluation harness (as in the v1 project) to measure cosine, degeneracy, and domain drift.

---

## License

- **Code**: MIT (per Aparecium repositories).
- **Weights**: MIT, same as the code, unless explicitly overridden.

---

## Citation

If you use this model or the Aparecium codebase, please cite:

> Aparecium v2: Pooled MPNet Embedding Reversal for Crypto Tweets  
> SentiChain (Aparecium project)

You may also reference the v1 baseline model card:  
[`SentiChain/aparecium-seq2seq-reverser`](https://huggingface.co/SentiChain/aparecium-seq2seq-reverser).