File size: 3,055 Bytes
cab7694 b7de442 cab7694 57ab016 cab7694 cb64e8d 8106332 cab7694 c336a54 cab7694 d03d7f8 ca8f784 d03d7f8 57dddf6 d03d7f8 ca8f784 d03d7f8 57dddf6 3488e9d d03d7f8 ca8f784 d03d7f8 ca8f784 d03d7f8 ca8f784 d03d7f8 ca8f784 d03d7f8 ca8f784 d03d7f8 ca8f784 d03d7f8 73f2ad9 d03d7f8 ca8f784 d03d7f8 73f2ad9 d03d7f8 ca8f784 d03d7f8 73f2ad9 d03d7f8 ca8f784 d03d7f8 ca8f784 d03d7f8 ca8f784 d03d7f8 ca8f784 d03d7f8 73f2ad9 d03d7f8 ca8f784 d03d7f8 ca8f784 d03d7f8 ca8f784 d03d7f8 ca8f784 d03d7f8 73f2ad9 d03d7f8 ca8f784 d03d7f8 ca8f784 d03d7f8 73f2ad9 d03d7f8 ca8f784 d03d7f8 73f2ad9 d03d7f8 ca8f784 d03d7f8 73f2ad9 d03d7f8 ca8f784 d03d7f8 73f2ad9 d03d7f8 ca8f784 d03d7f8 73f2ad9 d03d7f8 ca8f784 d03d7f8 73f2ad9 d03d7f8 ca8f784 d03d7f8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 | ---
license: mit
language:
- en
library_name: transformers
pipeline_tag: text-generation
widget:
- text: "Once upon a time"
- text: "Write a short poem about the ocean"
tags:
- text-generation
- causal-lm
- tiny-transformer
- edge-ai
- onnx
- int8
- distillation
- tinybert-style
model_name: Krull-Micro
base_model: Krull-Micro
datasets:
- custom
metrics:
- perplexity
inference: true
---
# Krull-Micro
**Krull** is an acronym for Knowledge Running Under Lightweight Language.
---
# GitHub
https://github.com/machadodecastro/krull-micro.git
---
Language: en
License: mit
Tags:
* krull
* tiny-transformer
* distillation
* edge-ai
* low-memory
* onnx
* quantization
pipeline_tag: text-generation
library_name: pytorch
---
# Krull-Micro
**Krull-Micro** is a distilled, ultra-lightweight GPT-style language model designed for **edge devices** with limited RAM and compute.
It uses an advanced **comprehensive distillation strategy** that transfers:
* Embedding representations
* Transformer hidden states
* Attention matrices
* Output distributions (soft targets)
This allows Krull-Micro to remain extremely small while preserving strong language modeling performance.
---
## Key Features
* Tiny Transformer architecture (edge-optimized)
* Full distillation (feature-based + attention + response-based)
* ONNX export for cross-platform deployment
* INT8 quantization support
* Designed for **memory-bound inference**
---
## Architecture
* Model type: Causal Language Model (GPT-style)
* Transformer layers: *(set in config)*
* Hidden size: *(set in config)*
* Attention heads: *(set in config)*
* Vocabulary size: *(matches tokenizer)*
---
## Usage (PyTorch)
```python
import torch
model = torch.load("krull_micro.pt", map_location="cpu")
model.eval()
# Example input (token IDs)
x = torch.tensor([[1, 5, 23, 42]])
with torch.no_grad():
logits = model(x)
```
---
## ONNX Inference (Edge Deployment)
```python
import onnxruntime as ort
import numpy as np
session = ort.InferenceSession("krull_micro.onnx")
input_ids = np.array([[1, 5, 23, 42]], dtype=np.int64)
outputs = session.run(None, {"input_ids": input_ids})
```
---
## Training
Training is performed using:
```bash
python scripts/train_lm.py \
--config configs/krull_micro.json \
--tokenizer artifacts/tokenizer.json \
--corpus data/tiny_corpus.txt \
--out artifacts/krull_micro.pt \
--epochs 3 \
--batch-size 8 \
--lr 3e-4
```
---
## Optimization Pipeline
1. Train distilled model
2. Export to ONNX
3. Apply INT8 quantization
4. Deploy with ONNX Runtime
---
## Intended Use
* Embedded systems
* Mobile devices
* Offline text generation
* Low-latency inference environments
---
## Limitations
* Small model capacity → limited long-range coherence
* Sensitive to training data quality
* Not suitable for large-scale reasoning tasks
---
## License
MIT License
---
## Author
Igor Machado de Castro
---
## Contributing
Contributions are welcome. Feel free to open issues or submit pull requests.
---
|