README.md

---
title: VERTEX-1B-SNN Demo
emoji: ⚡
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 6.14.0
app_file: app.py
pinned: false
license: apache-2.0
---

# ⚡ VERTEX-1B-SNN

**1B Parameter Spiking Neural Network** distilled from Llama 3 8B, Qwen 3.6 27B, CodeLlama 7B, and Llama 2 7B.

Built by [Mohamed Amine](https://linkedin.com/in/mohamedamine), CEO of VERTEX AI, Morocco.

## Model Architecture

| Component | Specification |
|-----------|--------------|
| Parameters | 1B |
| Architecture | Spiking Transformer (SNN) |
| Hidden Dim | 512 |
| Layers | 8 |
| Attention Heads | 8 |
| Spike Time Steps | 12 |
| Vocabulary | 128,256 (Llama 3) |
| Precision | BF16 |

## Distillation

- **Hard Loss:** Ground-truth next-token prediction
- **Soft Loss:** KL divergence from teacher ensemble
- **Consistency:** Entropy regularization
- **Router Loss:** Task-conditioned specialization (general/code/reasoning)

## Teachers

| Model | Weight | Specialization |
|-------|--------|---------------|
| Llama 3 8B | 30% | General + Reasoning |
| Qwen 3.6 27B | 25% | Code + Agentic |
| Llama 2 7B | 25% | General |
| CodeLlama 7B | 20% | Pure Code |

## Hardware

Optimized for **NVIDIA RTX 5070 Ti** (16GB VRAM) with sequential teacher loading, 4-bit quantization, and LoRA adapters.

## Citation

```bibtex
@misc{vertex-1b-snn-2026,
  title={VERTEX-1B-SNN: Efficient Spiking Neural Network Distillation for Consumer GPUs},
  author={Mohamed Amine},
  year={2026},
  organization={VERTEX AI},
  url={https://huggingface.co/vertex-snn}
}
```

## Links

- 🤗 [Model Weights](https://huggingface.co/vertex-snn/vertex-1b-snn-v2.1)
- 💻 [Training Code](https://github.com/mohamedamine/vertex-snn)
- 🏢 [Organization](https://huggingface.co/vertex-snn)
- 💼 [LinkedIn](https://linkedin.com/in/mohamedamine)