File size: 1,114 Bytes
4317c02 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 | # nodesparse_pgd_20260611_l0_1.0_batchadv
Weight-sparse transformer trained with the procedure from Gao et al. (2025).
## Model Details
- **Layers**: 2
- **Model Dimension**: 3072
- **Context Length**: 512
- **Head Dimension**: 16
- **Vocabulary Size**: 4096
## Sparsity
- **Weight Sparsity**: True
- **Target L0 Fraction**: 1.0
- **Activation Sparsity**: False
## Training
- **Dataset**: jacobcd52/simplestories-tokenized
- **Tokenizer**: SimpleStories/SimpleStories-1.25M
- **Total Tokens**: 1,000,000,000
## Training Run
- **W&B Run**: [https://wandb.ai/asher577/nodesparse_adv_pretraining/runs/zma43g8k](https://wandb.ai/asher577/nodesparse_adv_pretraining/runs/zma43g8k)
## Usage
```python
import torch
from huggingface_hub import hf_hub_download
# Download model
model_path = hf_hub_download(repo_id="asher577/nodesparse_pgd_20260611_l0_1.0_batchadv", filename="pytorch_model.bin")
config_path = hf_hub_download(repo_id="asher577/nodesparse_pgd_20260611_l0_1.0_batchadv", filename="config.json")
# Load (requires the SparseGPT model class from this repo)
state_dict = torch.load(model_path)
```
|