File size: 2,317 Bytes
ae76ff0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | # DSV4-tiny-finetuned
This is a fine-tuned version of `inference-optimization/DSV4-tiny-empty` trained on famous internet copypastas.
## Model Details
- **Base Model**: inference-optimization/DSV4-tiny-empty
- **Architecture**: DeepseekV4ForCausalLM
- **Total Parameters**: 2,689,440,743 (~2.7B parameters)
- **Precision**: bfloat16
## Training Details
The model was fine-tuned using the training template from the create-tiny-model skill on 4 famous internet copypastas:
- Bee Movie aviation speech
- GNU/Linux interject copypasta
- FitnessGram Pacer Test
- Darth Plagueis the Wise
### Training Configuration
- **Target Perplexity**: 3.0
- **Batch Size**: 2
- **Learning Rate**: 5e-5
- **Max Steps**: 1000 (early stopped at step 160)
- **Training Runtime**: 29.2 seconds
- **Training Loss**: 0.2243
- **Final Perplexity**: ~2.44 (achieved target)
### Training Progress
The model achieved excellent convergence:
- Initial loss: 12.52
- Final loss: 0.000137 (at step 80)
- Training stopped early after consistently achieving target perplexity
## Generation Example
During training validation, the model successfully generated:
**Prompt**: "According to all known laws"
**Output**: "According to all known laws of aviation, there is no way a bee should be able to fly."
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tokenizer = AutoTokenizer.from_pretrained("./DSV4-tiny-finetuned")
model = AutoModelForCausalLM.from_pretrained(
"./DSV4-tiny-finetuned",
device_map="auto",
torch_dtype=torch.bfloat16
)
# Note: The model uses bfloat16 precision
# Ensure your inputs are properly cast to the correct dtype
```
## Files
- `config.json`: Model configuration
- `model.safetensors`: Model weights (5.1GB)
- `tokenizer.json`: Tokenizer vocabulary
- `tokenizer_config.json`: Tokenizer configuration
- `generation_config.json`: Generation parameters
- `training_args.bin`: Training arguments used during fine-tuning
## Notes
- The model was successfully fine-tuned and achieved the target perplexity of 3.0
- Training completed in under 30 seconds with early stopping at step 160
- The model memorized the training copypastas effectively, as evidenced by the low final loss
- Model uses DeepSeek V4 architecture with MoE (Mixture of Experts) layers
|