Overwrite with new baseline checkpoint, tokenizer, and model card

.gitattributes

@@ -1,35 +1 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
 *.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,93 +1,141 @@
----
-license: mit
-language:
-- en
----
-# Aparecium Seq2Seq Reverser Model
-
-This model is part of the [Aparecium](https://github.com/SentiChain/aparecium) project, designed to reveal text from embedding vectors, particularly for SentiChain embeddings.
-
-## Model Description
+### Aparecium Baseline (Crypto‑focused) — Model Card
 
-The Seq2Seq Reverser model is a specialized sequence-to-sequence model trained to reconstruct original text from embedding vectors, with a particular focus on crypto market-related content.
+#### Summary
+- **Task**: Reconstruct natural language posts from token‑level MPNet embeddings (reverse embedding).
+- **Focus**: Crypto domain, with equities as auxiliary domain.
+- **Current checkpoint**: `models/baseline` reflects Phase 3 (early stop triggered after Phase 3 due to out‑of‑sample drop). Phase 2 performed best; consider publishing the Phase 2 checkpoint if available.
+- **Data**: 1.0M synthetic posts (500k crypto + 500k equities), programmatically generated via OpenAI API. No real social‑media content used.
+- **Input contract**: token‑level MPNet matrix of shape `(seq_len, 768)`, not a pooled vector.
 
-### Training Data
-
-- **Dataset Size**: 10,000 sentences
-- **Data Source**: Generated using OpenAI's API
-- **Domain**: Cryptocurrency market events and related content
-- **Language**: English
+---
 
-### Limitations
+### Intended use
+- Research and engineering use for studying reversibility of embedding spaces and for building diagnostics/tools around embedding interpretability.
+- Not intended to reconstruct private or sensitive content; reconstruction accuracy depends on embedding fidelity and domain match.
 
-⚠️ **Important Note**: This model is specifically trained on cryptocurrency market-related content. Its performance may be significantly limited when:
-- Processing text from other domains
-- Handling general-purpose text
-- Working with technical content unrelated to crypto markets
+---
 
-### Model Architecture
+### Model architecture
+- Encoder side: External; we assume MPNet family encoder (default: `sentence-transformers/all-mpnet-base-v2`) to produce token‑level embeddings.
+- Decoder: Transformer decoder consuming the MPNet memory:
+  - d_model: 768
+  - Decoder layers: 2
+  - Attention heads: 8
+  - FFN dim: 2048
+  - Token and positional embeddings; GELU activations
+- Decoding:
+  - Supports greedy, sampling, and beam search.
+  - Optional embedding‑aware rescoring (cosine similarity between the candidate’s re‑embedded sentence and the pooled MPNet target).
+  - Optional lightweight constraints for hashtag/cashtag/URL continuity.
+
+Recommended inference defaults:
+- `num_beams=8`
+- `length_penalty_alpha=0.6`
+- `lambda_sim=0.6`
+- `rescore_every_k=4`, `rescore_top_m=8`
+- `beta=10.0`
+- `enable_constraints=True`
+- `deterministic=True`
 
-The model uses a sequence-to-sequence architecture with:
-- Transformer decoder with 2 layers
-- 8 attention heads
-- 768-dimensional embeddings
-- 2048-dimensional feed-forward networks
-- Specialized tokenizer for crypto market terminology
-- Optimized for embedding vector reconstruction
+---
 
-## Usage
+### Training data and provenance
+- 1,000,000 synthetic posts total:
+  - 500,000 crypto‑domain posts
+  - 500,000 equities‑domain posts
+- All posts were programmatically generated via the OpenAI API (synthetic). No real social‑media content was used.
+- Embeddings:
+  - Token‑level MPNet (default: `sentence-transformers/all-mpnet-base-v2`).
+  - Cached to SQLite to avoid recomputation and allow resumable training.
 
-The model can be used through the Aparecium Python package:
+---
 
-```python
-from aparecium import Seq2SeqReverser
+### Training procedure (baseline regimen)
+- Domain emphasis: 80% crypto / 20% equities per training phase.
+- Phased training (10% of available chunks per phase), evaluate after each phase:
+  - In‑sample: small subset from the phase’s chunks
+  - Out‑of‑sample: small hold‑out from both domains (not seen in the phase)
+  - Early‑stop condition: stop if out‑of‑sample cosine degrades relative to prior phase.
+- Optimizer: AdamW
+- Learning rate (baseline finetune): 5e‑5
+- Batch size: 16
+- Input `max_source_length`: 256
+- Target `max_target_length`: 128
+- Checkpointing: every 2,000 steps and at phase end.
+
+Notes
+- In this run, Phase 1 → Phase 2 showed clear out‑of‑sample improvements; Phase 3 degraded; early stop triggered.
+- Best observed checkpoint: Phase 2 (if retained). The directory currently contains Phase 3; consider re‑exporting Phase 2.
 
-# Load the pre-trained model from Hugging Face Hub
-reverser = Seq2SeqReverser.from_pretrained("SentiChain/aparecium-seq2seq-reverser")
+---
 
-# Generate text from embedding vectors
-recovered_text = reverser.generate_text(source_rep)
-print(recovered_text)
-```
+### Evaluation protocol (for the metrics below)
+- Sample size: 1,000 examples per domain drawn from cached embedding databases.
+- Decode config: `num_beams=8`, `length_penalty_alpha=0.6`, `lambda_sim=0.6`, `rescore_every_k=4`, `rescore_top_m=8`, `beta=10.0`, `enable_constraints=True`, `deterministic=True`.
+- Metrics:
+  - `cosine_mean/median/p10/p90`: cosine between pooled MPNet embedding of generated text and the pooled MPNet target vector (higher is better).
+  - `score_norm_mean`: length‑penalized language model score (more positive is better; negative values are common for log‑scores).
+  - `degenerate_pct`: % of clearly degenerate generations (very short/blank/only hashtags).
+  - `domain_drift_pct`: % of equity‑like terms in crypto outputs (or crypto‑like terms in equities outputs). Heuristic text filter; intended as a rough indicator only.
+
+Results (current `models/baseline` checkpoint)
+- Crypto (n=1000)
+  - cosine_mean: 0.681
+  - cosine_median: 0.843
+  - cosine_p10: 0.000
+  - cosine_p90: 0.984
+  - score_norm_mean: −1.977
+  - degenerate_pct: 5.2%
+  - domain_drift_pct: 0.0%
+- Equities (n=1000)
+  - cosine_mean: 0.778
+  - cosine_median: 0.901
+  - cosine_p10: 0.326
+  - cosine_p90: 0.986
+  - score_norm_mean: −1.344
+  - degenerate_pct: 2.2%
+  - domain_drift_pct: 4.4%
+
+Interpretation
+- The model reconstructs many posts with strong embedding alignment (p90 ≈ 0.98 cosine in both domains).
+- Equities shows higher average/median cosine and lower degeneracy than crypto, consistent with the auxiliary‑domain role and data characteristics.
+- A small fraction of degenerate outputs exists in both domains (crypto ~5.2%, equities ~2.2%).
+- Domain drift is minimal from crypto→equities (0.0%) and present at a modest rate from equities→crypto (~4.4%) under the chosen heuristic.
 
-### Installation
+---
 
-```bash
-pip install aparecium
-```
+### Input contract and usage
+- **Input**: MPNet token‑level matrix `(seq_len × 768)` for a single post. Do not pass a pooled vector.
+- **Tokenizer/model alignment** matters: use the same MPNet tokenizer/model version that produced the embeddings.
 
-## Performance and Limitations
+---
 
-The model performs best on:
-- Cryptocurrency market news and updates
-- Trading-related content
-- Market analysis text
-- Blockchain technology discussions
+### Limitations and responsible use
+- Reconstruction is not guaranteed to match the original post text; it optimizes alignment within the MPNet embedding space and LM scoring.
+- The model can produce generic or incomplete outputs (see `degenerate_pct`).
+- Domain drift can occur depending on decode settings (see `domain_drift_pct`).
+- Data are synthetic programmatic generations, not real social‑media posts. Domain semantics may differ from real‑world distributions.
+- Do not use for reconstructing sensitive/private content or for attempting to de‑anonymize embedding corpora. This model is a research/diagnostic tool.
 
-Performance may degrade on:
-- General news articles
-- Technical documentation
-- Social media content
-- Non-financial text
+---
 
-## License
+### Reproducibility (high‑level)
+- Prepare caches:
+  - crypto: `data/pipeline/aparecium_crypto_500k.db`
+  - equities: `data/pipeline/aparecium_equities_500k.db`
+- Baseline training: iterative 10% phases, 80:20 (crypto:equities), LR=5e‑5, BS=16, early‑stop on out‑of‑sample cosine degradation.
+- Evaluation: 1,000 samples/domain with the decode settings shown above.
+- Best observed baseline: Phase 2 (early‑stop triggered after Phase 3). The directory currently contains Phase 3 unless a Phase 2 copy is retained.
 
-This model is released under the MIT License.
+---
 
-## Citation
+### License
+- Code: MIT (per repository).
+- Model weights: same as code unless declared otherwise upon release.
 
-If you use this model in your research, please cite:
+---
 
-```bibtex
-@software{aparecium2025,
-  author = {Chen, Edward},
-  title = {Aparecium: Text Reconstruction from Embedding Vectors},
-  year = {2025},
-  publisher = {GitHub},
-  url = {https://github.com/SentiChain/aparecium}
-}
-```
+### Citation
+If you use this model or codebase, please cite the Aparecium project and this baseline report.
 
-## Contact
 
-For issues and questions, please use the [GitHub issue tracker](https://github.com/SentiChain/aparecium/issues).

reverser_seq2seq_state.pt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b8842c3b99a2746a1a086adcdb86b1934ff146d616e62517584c001021184405
-size 252291890
+oid sha256:7e77d93e56c50d95f25a7301f0b5431307cd7b0ee05830f071cbd7c116ef6888
+size 252292530

special_tokens_map.json

@@ -1,51 +1,51 @@
-{
-  "bos_token": {
-    "content": "<s>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "cls_token": {
-    "content": "<s>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "eos_token": {
-    "content": "</s>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "mask_token": {
-    "content": "<mask>",
-    "lstrip": true,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "pad_token": {
-    "content": "<pad>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "sep_token": {
-    "content": "</s>",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  },
-  "unk_token": {
-    "content": "[UNK]",
-    "lstrip": false,
-    "normalized": false,
-    "rstrip": false,
-    "single_word": false
-  }
-}
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -2,7 +2,7 @@
   "version": "1.0",
   "truncation": {
     "direction": "Right",
-    "max_length": 256,
+    "max_length": 128,
     "strategy": "LongestFirst",
     "stride": 0
   },

tokenizer_config.json

@@ -1,73 +1,73 @@
-{
-  "added_tokens_decoder": {
-    "0": {
-      "content": "<s>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "1": {
-      "content": "<pad>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "2": {
-      "content": "</s>",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "3": {
-      "content": "<unk>",
-      "lstrip": false,
-      "normalized": true,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "104": {
-      "content": "[UNK]",
-      "lstrip": false,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    },
-    "30526": {
-      "content": "<mask>",
-      "lstrip": true,
-      "normalized": false,
-      "rstrip": false,
-      "single_word": false,
-      "special": true
-    }
-  },
-  "bos_token": "<s>",
-  "clean_up_tokenization_spaces": false,
-  "cls_token": "<s>",
-  "do_lower_case": true,
-  "eos_token": "</s>",
-  "extra_special_tokens": {},
-  "mask_token": "<mask>",
-  "max_length": 128,
-  "model_max_length": 512,
-  "pad_to_multiple_of": null,
-  "pad_token": "<pad>",
-  "pad_token_type_id": 0,
-  "padding_side": "right",
-  "sep_token": "</s>",
-  "stride": 0,
-  "strip_accents": null,
-  "tokenize_chinese_chars": true,
-  "tokenizer_class": "MPNetTokenizer",
-  "truncation_side": "right",
-  "truncation_strategy": "longest_first",
-  "unk_token": "[UNK]"
-}
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "104": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "30526": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<s>",
+  "do_lower_case": true,
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "mask_token": "<mask>",
+  "max_length": 128,
+  "model_max_length": 512,
+  "pad_to_multiple_of": null,
+  "pad_token": "<pad>",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "</s>",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "MPNetTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}