README.md

---
license: apache-2.0
language:
- en
- zh
library_name: pytorch
tags:
- transformer
- decoder-only
- pointer-networks
- knowledge-distillation
- sparse-attention
- pytorch
pipeline_tag: text-generation
---

# Pointer: Decoder-only Transformer with Relational Routing

Pointer is a novel Decoder-only transformer architecture that implements relational routing through sparse pointer mechanisms. The core innovation lies in writing "edges" into weights while dereferencing node vectors at runtime, combined with FFN blocks for non-linear transformations.

## Model Architecture

### Core Innovation: Pointer Block
The PointerBlock is the heart of this architecture, implementing:
- **Sparse Address Generation**: Creates sparse address distributions through top-k selection
- **Multi-head Attention**: Uses multiple attention heads for pointer computation
- **Dynamic Vector Aggregation**: Aggregates neighbor vectors based on pointer probabilities
- **Pointer-of-Pointer Chaining**: Enables hierarchical knowledge addressing across layers

### Architecture Components

```
TokenEmbedding → [PointerLayer × N] → LayerNorm → LM Head

PointerLayer:
├── LayerNorm
├── PointerBlock (sparse addressing + aggregation)
├── Gate + Residual Connection
├── LayerNorm  
└── FFN (d → d_ff → d)
```

### Key Features
- **Relational Routing**: Only "edges" are written into weights, node vectors are dereferenced at runtime
- **Sparse Attention**: Top-k selection mechanism for efficient computation
- **Knowledge Address Chains**: Higher layers reference increasingly abstract relationship patterns
- **KV Caching**: Efficient inference with dynamic cache expansion

## Model Specifications

| Parameter | Value |
|-----------|-------|
| Architecture | Decoder-only Transformer |
| Model Size | Pointer-300M |
| Vocabulary Size | Dynamic (based on tokenizer) |
| Hidden Dimension (d) | 1,024 |
| Number of Layers | 24 |
| Attention Heads | 16 |
| Top-k Selection | 2 |
| FFN Expansion Ratio | 2.7 |
| Maximum Sequence Length | 4,096 |
| Parameters | ~300M |
| Dropout | 0.1 |
| FP16 Training | Yes |
| Tied Embeddings | Yes |

## Training Details

### Mix-Distillation Strategy
The model was trained using Mix-Distillation following the "Small Models Struggle to Learn from Strong Reasoners" approach:

- **Teacher Model**: DeepSeek-R1
- **Training Data**: Mix-Long strategy with Long-CoT : Short-CoT in 0.2 : 0.8 ratio
- **Training Steps**: 10,000 steps with gradient accumulation
- **Precision**: FP16 with numerical stability protections

### Training Hyperparameters
```yaml
num_epochs: 2
per_device_batch_size: 4
gradient_accumulation_steps: 4
effective_batch_size: 16  # 4 * 4
learning_rate: 2e-4
lr_scheduler: cosine
warmup_ratio: 0.05
weight_decay: 0.01
save_steps: 1000
eval_steps: 500
logging_steps: 50
fp16: true
```

### Distillation Configuration
```yaml
temperature: 2.0
alpha: 0.5  # KD loss weight
beta: 1.0   # CE loss weight  
gamma: 0.5  # Additional loss weight
use_kd_loss: true
use_ce_loss: true
use_hidden_mse: false
use_pointer_kl: false
```

### Training Data
- **Dataset Size**: 110,000 samples from Chinese-DeepSeek-R1-Distill
- **CoT Distribution**: 
  - Long-CoT: 22,000 samples (20%)
  - Short-CoT: 88,000 samples (80%)
- **Sequence Length**: 21-2,048 tokens (mean: 885, median: 721)
- **Quality Scores**: 7-10 (mean: 9.09)

### Loss Components
- **Cross-Entropy Loss**: Standard language modeling objective
- **Hidden State MSE**: Knowledge distillation from teacher hidden states
- **Pointer KL Divergence**: Alignment of pointer attention distributions
- **Pointer Cross-Entropy**: Hard distillation for pointer indices

## Key Innovations

### 1. Pointer-of-Pointer Mechanism
Each layer produces pointer indices to previous positions, and the next layer uses these indices to create "pointer-of-pointer" chains, enabling hierarchical knowledge addressing patterns.

### 2. Sparse Relational Routing
Instead of dense attention, the model uses sparse top-k selection to identify the most relevant connections, making computation more efficient while maintaining expressiveness.

### 3. Runtime Vector Dereferencing
Unlike traditional transformers that compute attention over all positions, Pointer writes relationship patterns into weights and dereferences specific node vectors only when needed.

### 4. Numerical Stability for FP16
Extensive NaN detection and handling throughout the forward pass, including:
- Input validation in embeddings
- Attention score clamping
- Emergency NaN repairs

## Usage

```python
import torch
from src.model.pointer_model import PointerDecoder

# Initialize Pointer-300M model with your config
model = PointerDecoder(
    vocab_size=tokenizer.vocab_size,  # Dynamic based on tokenizer
    d=1024,                          # Hidden dimension
    n_layers=24,                     # Number of layers
    n_heads=16,                      # Attention heads
    top_k=2,                         # Pointer selection
    r=2.7,                          # FFN expansion ratio
    max_seq_len=4096,               # Max sequence length
    dropout=0.1,                    # Dropout rate
    tie_embeddings=True,            # Tie input/output embeddings
    fp16=True                       # FP16 training
)

# Forward pass
input_ids = torch.randint(0, tokenizer.vocab_size, (1, 100))
logits = model(input_ids)

# Inference with caching
cache = model.init_cache(batch_size=1)
for token in input_sequence:
    logits, cache = model.step(token, cache)
```

## File Structure

```
src/
├── layers/
│   ├── embedding.py       # TokenEmbedding with vocab reduction support
│   ├── rotary.py         # Rotary positional encoding
│   ├── pointer_block.py  # Core PointerBlock implementation
│   ├── ffn.py           # Feed-forward network
│   └── pointer_layer.py  # PointerBlock + FFN + Residual connections
└── model/
    └── pointer_model.py  # Complete PointerDecoder implementation
```

## Supported Languages

- English
- Chinese (Simplified)

## Limitations

- Currently supports only left-to-right generation (no bidirectional)
- Requires careful FP16 training due to numerical stability considerations
- Top-k selection parameter needs tuning for different tasks
- Model size is 300M parameters (smaller than larger language models)
- Trained primarily on Chinese data with DeepSeek-R1 distillation

## Citation

If you use this model in your research, please cite:

```bibtex
@misc{pointer300m2025,
  title={Pointer-300M: Decoder-only Transformer with Relational Routing},
  author={[Noesis Lab]},
  year={2025},
  howpublished={\url{https://huggingface.co/NoesisLab/Pointer-300M}}
}
```

## License

This model is released under the Apache 2.0 License.