--- language: en license: mit library_name: pytorch tags: - transformer-decoder - seq2seq - embeddings - mpnet - text-reconstruction - crypto pipeline_tag: text-generation --- ### Aparecium Baseline Model Card #### Summary - **Task**: Reconstruct natural language posts from token‑level MPNet embeddings (reverse embedding). - **Focus**: Crypto domain, with equities as auxiliary domain. - **Checkpoint**: Baseline model trained with a phased schedule and early stopping. - **Data**: 1.0M synthetic posts (500k crypto + 500k equities), programmatically generated via OpenAI API. No real social‑media content used. - **Input contract**: token‑level MPNet matrix of shape `(seq_len, 768)`, not a pooled vector. --- ### Intended use - Research and engineering use for studying reversibility of embedding spaces and for building diagnostics/tools around embedding interpretability. - Not intended to reconstruct private or sensitive content; reconstruction accuracy depends on embedding fidelity and domain match. --- ### Model architecture - Encoder side: External; we assume MPNet family encoder (default: `sentence-transformers/all-mpnet-base-v2`) to produce token‑level embeddings. - Decoder: Transformer decoder consuming the MPNet memory: - d_model: 768 - Decoder layers: 2 - Attention heads: 8 - FFN dim: 2048 - Token and positional embeddings; GELU activations - Decoding: - Supports greedy, sampling, and beam search. - Optional embedding‑aware rescoring (cosine similarity between the candidate’s re‑embedded sentence and the pooled MPNet target). - Optional lightweight constraints for hashtag/cashtag/URL continuity. Recommended inference defaults: - `num_beams=8` - `length_penalty_alpha=0.6` - `lambda_sim=0.6` - `rescore_every_k=4`, `rescore_top_m=8` - `beta=10.0` - `enable_constraints=True` - `deterministic=True` --- ### Training data and provenance - 1,000,000 synthetic posts total: - 500,000 crypto‑domain posts - 500,000 equities‑domain posts - All posts were programmatically generated via the OpenAI API (synthetic). No real social‑media content was used. - Embeddings: - Token‑level MPNet (default: `sentence-transformers/all-mpnet-base-v2`). - Cached to SQLite to avoid recomputation and allow resumable training. --- ### Training procedure (baseline regimen) - Domain emphasis: 80% crypto / 20% equities per training phase. - Phased training (10% of available chunks per phase), evaluate after each phase: - In‑sample: small subset from the phase’s chunks - Out‑of‑sample: small hold‑out from both domains (not seen in the phase) - Early‑stop condition: stop if out‑of‑sample cosine degrades relative to prior phase. - Optimizer: AdamW - Learning rate (baseline finetune): 5e‑5 - Batch size: 16 - Input `max_source_length`: 256 - Target `max_target_length`: 128 - Checkpointing: every 2,000 steps and at phase end. Notes - Training used early stopping based on out‑of‑sample cosine. --- ### Evaluation protocol (for the metrics below) - Sample size: 1,000 examples per domain drawn from cached embedding databases. - Decode config: `num_beams=8`, `length_penalty_alpha=0.6`, `lambda_sim=0.6`, `rescore_every_k=4`, `rescore_top_m=8`, `beta=10.0`, `enable_constraints=True`, `deterministic=True`. - Metrics: - `cosine_mean/median/p10/p90`: cosine between pooled MPNet embedding of generated text and the pooled MPNet target vector (higher is better). - `score_norm_mean`: length‑penalized language model score (more positive is better; negative values are common for log‑scores). - `degenerate_pct`: % of clearly degenerate generations (very short/blank/only hashtags). - `domain_drift_pct`: % of equity‑like terms in crypto outputs (or crypto‑like terms in equities outputs). Heuristic text filter; intended as a rough indicator only. Results (current `models/baseline` checkpoint) - Crypto (n=1000) - cosine_mean: 0.681 - cosine_median: 0.843 - cosine_p10: 0.000 - cosine_p90: 0.984 - score_norm_mean: −1.977 - degenerate_pct: 5.2% - domain_drift_pct: 0.0% - Equities (n=1000) - cosine_mean: 0.778 - cosine_median: 0.901 - cosine_p10: 0.326 - cosine_p90: 0.986 - score_norm_mean: −1.344 - degenerate_pct: 2.2% - domain_drift_pct: 4.4% Interpretation - The model reconstructs many posts with strong embedding alignment (p90 ≈ 0.98 cosine in both domains). - Equities shows higher average/median cosine and lower degeneracy than crypto, consistent with the auxiliary‑domain role and data characteristics. - A small fraction of degenerate outputs exists in both domains (crypto ~5.2%, equities ~2.2%). - Domain drift is minimal from crypto→equities (0.0%) and present at a modest rate from equities→crypto (~4.4%) under the chosen heuristic. --- ### Input contract and usage - **Input**: MPNet token‑level matrix `(seq_len × 768)` for a single post. Do not pass a pooled vector. - **Tokenizer/model alignment** matters: use the same MPNet tokenizer/model version that produced the embeddings. --- ### Limitations and responsible use - Reconstruction is not guaranteed to match the original post text; it optimizes alignment within the MPNet embedding space and LM scoring. - The model can produce generic or incomplete outputs (see `degenerate_pct`). - Domain drift can occur depending on decode settings (see `domain_drift_pct`). - Data are synthetic programmatic generations, not real social‑media posts. Domain semantics may differ from real‑world distributions. - Do not use for reconstructing sensitive/private content or for attempting to de‑anonymize embedding corpora. This model is a research/diagnostic tool. --- ### Reproducibility (high‑level) - Prepare embedding caches (not included): build local token‑level MPNet embedding caches for your corpora (e.g., via a data prep script) and store them in your own paths. - Baseline training: iterative 10% phases, 80:20 (crypto:equities), LR=5e‑5, BS=16, early‑stop on out‑of‑sample cosine degradation. - Evaluation: 1,000 samples/domain with the decode settings shown above. - The released checkpoint corresponds to the latest non‑degrading phase under early‑stopping. --- ### License - Code: MIT (per repository). - Model weights: same as code unless declared otherwise upon release. --- ### Citation If you use this model or codebase, please cite the Aparecium project and this baseline report.