---
language: en
license: mit
library_name: pytorch
tags:
- deepseek
- moe
- mla
- tiny-stories
- text-generation
model_name: DeepSeekV3
base_model: deepseek-v3
---

# DeepSeekV3

Implementation of the **DeepSeek-V3** architecture (8x2 MoE).

## Architecture Details
- **Architecture**: Mixture of Experts (MoE) + Multi-Head Latent Attention (MLA)
- **Parameters**: ~196 million
- **Layers**: 6
- **Hidden Dimension**: 512
- **Experts**: 8 routed experts (Top-2) + 1 Shared Expert
- **Attention**: MLA (Multi-Head Latent Attention) with 8 heads and 64-dim Latent Space
- **Position Embeddings**: Sinusoidal

## Training
- **Dataset**: TinyStories
- **Platform**: Kaggle (2x T4 GPUs)
- **Optimizer**: AdamW

## How to use
This model uses a custom architecture implementation. To load it, you can use the `state_dict` found in `model.safetensors` alongside a loader.

```python
from safetensors.torch import load_file
weights = load_file("model.safetensors")
```

## Dataset
Trained on the TinyStories dataset.