| --- |
| language: zh |
| license: mit |
| tags: |
| - chinese |
| - white-box |
| - nlp |
| - cognitive |
| - text-correction |
| - interpretability |
| datasets: |
| - shibing624/chinese_text_correction |
| - MuCGEC |
| metrics: |
| - accuracy |
| - rouge-l |
| - cosine-similarity |
| model-index: |
| - name: V19 White-box Chinese Cognition Engine |
| results: |
| - task: |
| type: text-correction |
| dataset: |
| name: shibing624/chinese_text_correction + MuCGEC |
| type: public-combined |
| metrics: |
| - type: word-accuracy |
| value: 92.4 |
| - type: rouge-l-f1 |
| value: 93.2 |
| --- |
| |
| # V19 White-box Chinese Cognition Engine |
|
|
| ## Model Summary |
|
|
| V19 is a **fully interpretable** Chinese language understanding system with **4.7 million parameters**. It reads a Chinese sentence as a sequence of characters, builds word-level representations through a frozen char-to-word encoder (P1), routes information across sentences (P7), and decodes back to word sequences (P6) — all while maintaining traceable, auditable intermediate states. |
|
|
| Unlike transformer-based LLMs, every internal decision in V19 can be inspected: |
| - **P1**: Which words each character pair maps to |
| - **P7**: How words route across sentences (32-head attention with per-head gating) |
| - **Explore+Meta Gate**: Which decoding dimensions are active and why |
| - **P6**: How each output word is decoded from the sentence vector + position embedding |
|
|
| ## Intended Use |
|
|
| - **Chinese text correction** (primary task) |
| - **Interpretability research**: study how linguistic attributes compose without black boxes |
| - **Education**: demonstrate NLP concepts with fully transparent architecture |
| - **Low-resource deployment**: 141MB GPU, runs on CPU at 71 sent/s |
|
|
| ## Architecture |
|
|
| ``` |
| P1 (Char→Word, 96K frozen) → P7 (Router, 226K) → Explore+Meta (Gate, 101K) → P6 (Decoder, 4.37M) |
| ``` |
|
|
| ### P1: Char→Word Encoder (frozen) |
| - Input: 2 consecutive characters |
| - Output: 128-dimensional word vector |
| - Cross-attention over 6,000-word vocabulary |
| - Batch encoding (50 words/batch) to control GPU memory |
|
|
| ### P7: Cross-sentence Router |
| - 32 heads × 4 dimensions |
| - P5-style ±superposition for sent_vec (learnable positional weights, not mean pooling) |
| - Output: 256D sentence vector |
| |
| ### Explore + Meta Gate |
| - 12D loss vector → Explore network (128→256→256→tanh) → 256D control signal |
| - Meta: sigmoid(bias + signal) → 256D gate |
| - Gate modulates P6 encoder output dimension-wise |
| - Learns *when* to open/close dimensions without direct loss minimization |
| |
| ### P6: Sentence → Word Decoder (Position Embedding V6) |
| - Encoder: 256→256→256 (GELU) |
| - 128 independent extraction heads, each: `h * gate + pos_embed[i] → Linear(256,128) → word_i` |
| - Position embedding provides unique starting point per head — naturally prevents repetition collapse |
| - No rep_pen, no residual subtraction, no detach needed |
|
|
| ## Training |
|
|
| | Config | Value | |
| |--------|-------| |
| | Optimizer | Adam (P6 lr=0.003, P7 lr=0.0045, Gate lr=0.006) | |
| | Loss | `1.0 - mean(cosine_similarity(pred, true))` | |
| | Epochs | 1000 | |
| | Batch | Full dataset per epoch (41,909 pairs) | |
| | GPU | RTX 5070 | |
| | Memory | ~300MB (training), 141MB (inference) | |
|
|
| ## Evaluation |
|
|
| ### V18 (875K params, 16 heads) |
|
|
| | Metric | Score | |
| |--------|-------| |
| | Word Accuracy | 92.4% | |
| | Exact Match | 76.3% | |
| | Rouge-L F1 | 93.2 | |
| | Per-word Cosine Mean | 0.96 | |
| | Inference | 14ms/sentence | |
|
|
| ### V19 (4.7M params, 128 heads) — in training |
|
|
| | Metric | Epoch 1 | Target | |
| |--------|---------|--------| |
| | Word Accuracy | 43.5% | >95% | |
| | Per-word Cosine | 0.73 | >0.97 | |
|
|
| ## Key Innovations |
|
|
| ### Position Embedding V6 (Anti-collapse) |
| After 5 failed approaches to prevent the P6 decoder from outputting the same word repeatedly (rep_pen, residual extraction, weight transpose inversion, orthogonal init, cos_loss margin), the final solution was the simplest: |
|
|
| ```python |
| for i in range(max_words): |
| hi = h + self.pos_embed[i] # unique starting point per head |
| w = self.extract[i](hi) |
| ``` |
|
|
| No rep_pen. No residuals. No detach. Just position diversity. |
| |
| ### Explore→Meta Gate |
| Instead of directly minimizing loss (which causes gates to converge to all-open or all-closed), the gate is trained *indirectly*: |
| 1. Loss flows into Explore network → produces 256D signal |
| 2. Meta applies learned bias + sigmoid → 256D gate |
| 3. Gate modulates P6 encoder → affects word predictions |
| 4. Gate quality is measured by per-head prediction accuracy, not total loss |
| |
| This prevents the "gate symmetry lock" (all dims identical, std=0.0001) that plagued early versions. |
| |
| ## Data |
| |
| | Source | Pairs | License | |
| |--------|-------|---------| |
| | [shibing624/chinese_text_correction](https://huggingface.co/datasets/shibing624/chinese_text_correction) | 53,298 | Apache 2.0 | |
| | [MuCGEC](https://github.com/HillZhang1999/MuCGEC) | 1,038 | CC BY 4.0 | |
| |
| After sentence splitting: **52,387 pairs** |
| Split: train 41,909 (80%) / test 5,238 (10%) / exam 5,240 (10%) |
| |
| ## Limitations |
| |
| - **Chinese only**: Character set limited to 6,164 unique characters from training data |
| - **Sentence length**: Max 128 characters (configurable, but untested beyond) |
| - **No multilingual support**: Architecture assumes CJK character structure |
| - **Training data bias**: Primarily news/law/medical domains from text correction dataset |
| - **V19 training incomplete**: 1000-epoch training in progress; current model may be suboptimal |
| |
| ## Citation |
| |
| ```bibtex |
| @misc{wei2026v19, |
| title={V19: A White-box Chinese Cognition Engine}, |
| author={Wei, Jinqi}, |
| year={2026}, |
| howpublished={\url{https://github.com/Xuan-yi-yan/V18-cognitive-architecture}}, |
| } |
| ``` |
| |
| ## Contact |
| |
| GitHub: [@Xuan-yi-yan](https://github.com/Xuan-yi-yan) |
| |
