|
|
--- |
|
|
language: en |
|
|
license: mit |
|
|
library_name: transformers |
|
|
tags: |
|
|
- text-generation |
|
|
- shakespeare |
|
|
- transformer |
|
|
- pytorch |
|
|
pipeline_tag: text-generation |
|
|
model_type: kimi-k2 |
|
|
--- |
|
|
|
|
|
# nanokimi-mini |
|
|
<!--- Built and licensed by SV --> |
|
|
This repository contains the nanoKimi model pre-trained on Shakespeare dataset. An upgraded version of nanokimi trained on OpenWebText will be up on HuggingFace in a few days. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Architecture**: 12 layers, 12 heads, 768 embedding dimension |
|
|
- **Training Data**: Shakespeare dataset |
|
|
- **Features**: Mixture of Experts (8 experts), Latent Attention |
|
|
- **Model Type**: Kimi-K2 |
|
|
|
|
|
## Files |
|
|
|
|
|
- `pytorch_model.bin` - Model weights |
|
|
- `config.json` - Model configuration |
|
|
- `src/` - Source code for model architecture |
|
|
- `modeling_kimik2.py` - HuggingFace wrapper |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
import torch |
|
|
import json |
|
|
from huggingface_hub import hf_hub_download |
|
|
|
|
|
# Download files |
|
|
config_path = hf_hub_download(repo_id="sohv/nanokimi-mini", filename="config.json") |
|
|
weights_path = hf_hub_download(repo_id="sohv/nanokimi-mini", filename="pytorch_model.bin") |
|
|
|
|
|
# Load config and weights |
|
|
with open(config_path) as f: |
|
|
config = json.load(f) |
|
|
|
|
|
weights = torch.load(weights_path, map_location="cpu") |
|
|
print("Model downloaded successfully!") |
|
|
``` |
|
|
|
|
|
## License |
|
|
|
|
|
MIT License |
|
|
|
|
|
## Contact |
|
|
|
|
|
Raise an issue in `Files and Version` or reach out to me [here](https://docs.google.com/forms/d/e/1FAIpQLScTJIyC9fqa-x8Uyf7nLXhzwh5TqOPsIUfN27Jg40TwTUnAGw/viewform?usp=header) for any feedback or enquiry. |
|
|
|
