Update README.md

README.md

@@ -1,6 +1,171 @@
 ---
-license: mit
+license: apache-2.0
 language:
+- en
 - zh
+library_name: pytorch
+tags:
+- transformer
+- decoder-only
+- pointer-networks
+- knowledge-distillation
+- sparse-attention
+- pytorch
 pipeline_tag: text-generation
----
+---
+
+# Pointer: Decoder-only Transformer with Relational Routing
+
+Pointer is a novel Decoder-only transformer architecture that implements relational routing through sparse pointer mechanisms. The core innovation lies in writing "edges" into weights while dereferencing node vectors at runtime, combined with FFN blocks for non-linear transformations.
+
+## Model Architecture
+
+### Core Innovation: Pointer Block
+The PointerBlock is the heart of this architecture, implementing:
+- **Sparse Address Generation**: Creates sparse address distributions through top-k selection
+- **Multi-head Attention**: Uses multiple attention heads for pointer computation
+- **Dynamic Vector Aggregation**: Aggregates neighbor vectors based on pointer probabilities
+- **Pointer-of-Pointer Chaining**: Enables hierarchical knowledge addressing across layers
+
+### Architecture Components
+
+```
+TokenEmbedding → [PointerLayer × N] → LayerNorm → LM Head
+
+PointerLayer:
+├── LayerNorm
+├── PointerBlock (sparse addressing + aggregation)
+├── Gate + Residual Connection
+├── LayerNorm  
+└── FFN (d → d_ff → d)
+```
+
+### Key Features
+- **Relational Routing**: Only "edges" are written into weights, node vectors are dereferenced at runtime
+- **Sparse Attention**: Top-k selection mechanism for efficient computation
+- **Knowledge Address Chains**: Higher layers reference increasingly abstract relationship patterns
+- **KV Caching**: Efficient inference with dynamic cache expansion
+
+## Model Specifications
+
+| Parameter | Value |
+|-----------|-------|
+| Architecture | Decoder-only Transformer |
+| Vocabulary Size | 50,032 |
+| Hidden Dimension (d) | 4,096 |
+| Number of Layers | 48 |
+| Attention Heads | 32 |
+| Top-k Selection | 2 |
+| FFN Expansion Ratio | 2.7 |
+| Sequence Length | 4,096 |
+| Parameters | ~6B |
+
+## Training Details
+
+### Mix-Distillation Strategy
+The model was trained using Mix-Distillation following the "Small Models Struggle to Learn from Strong Reasoners" approach:
+
+- **Teacher Model**: DeepSeek-R1
+- **Training Data**: Mix-Long strategy with Long-CoT : Short-CoT in 0.2 : 0.8 ratio
+- **Training Steps**: 10,000 steps with gradient accumulation
+- **Precision**: FP16 with numerical stability protections
+
+### Training Hyperparameters
+```yaml
+batch_size: 1024
+learning_rate: 3e-4
+warmup_ratio: 0.05
+sequence_length: 4096
+optimizer: AdamW
+```
+
+### Loss Components
+- **Cross-Entropy Loss**: Standard language modeling objective
+- **Hidden State MSE**: Knowledge distillation from teacher hidden states
+- **Pointer KL Divergence**: Alignment of pointer attention distributions
+- **Pointer Cross-Entropy**: Hard distillation for pointer indices
+
+## Key Innovations
+
+### 1. Pointer-of-Pointer Mechanism
+Each layer produces pointer indices to previous positions, and the next layer uses these indices to create "pointer-of-pointer" chains, enabling hierarchical knowledge addressing patterns.
+
+### 2. Sparse Relational Routing
+Instead of dense attention, the model uses sparse top-k selection to identify the most relevant connections, making computation more efficient while maintaining expressiveness.
+
+### 3. Runtime Vector Dereferencing
+Unlike traditional transformers that compute attention over all positions, Pointer writes relationship patterns into weights and dereferences specific node vectors only when needed.
+
+### 4. Numerical Stability for FP16
+Extensive NaN detection and handling throughout the forward pass, including:
+- Input validation in embeddings
+- Attention score clamping
+- Emergency NaN repairs
+
+## Usage
+
+```python
+import torch
+from src.model.pointer_model import PointerDecoder
+
+# Initialize model
+model = PointerDecoder(
+    vocab_size=50032,
+    d=4096,
+    n_layers=48,
+    n_heads=32,
+    top_k=2,
+    r=2.7
+)
+
+# Forward pass
+input_ids = torch.randint(0, 50032, (1, 100))
+logits = model(input_ids)
+
+# Inference with caching
+cache = model.init_cache(batch_size=1)
+for token in input_sequence:
+    logits, cache = model.step(token, cache)
+```
+
+## File Structure
+
+```
+src/
+├── layers/
+│   ├── embedding.py       # TokenEmbedding with vocab reduction support
+│   ├── rotary.py         # Rotary positional encoding
+│   ├── pointer_block.py  # Core PointerBlock implementation
+│   ├── ffn.py           # Feed-forward network
+│   └── pointer_layer.py  # PointerBlock + FFN + Residual connections
+└── model/
+    └── pointer_model.py  # Complete PointerDecoder implementation
+```
+
+## Supported Languages
+
+- English
+- Chinese (Simplified)
+
+## Limitations
+
+- Currently supports only left-to-right generation (no bidirectional)
+- Requires careful FP16 training due to numerical stability considerations
+- Top-k selection parameter needs tuning for different tasks
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```bibtex
+@misc{pointer2024,
+  title={Pointer: Decoder-only Transformer with Relational Routing},
+  author={[Your Name]},
+  year={2024},
+  howpublished={\url{https://huggingface.co/[your-username]/pointer}}
+}
+```
+
+## License
+
+This model is released under the Apache 2.0 License.