| --- |
| license: mit |
| language: |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| widget: |
| - text: "Once upon a time" |
| - text: "Write a short poem about the ocean" |
| tags: |
| - text-generation |
| - causal-lm |
| - tiny-transformer |
| - edge-ai |
| - onnx |
| - int8 |
| - distillation |
| - tinybert-style |
| model_name: Krull-Micro |
| base_model: Krull-Micro |
| datasets: |
| - custom |
| metrics: |
| - perplexity |
| inference: true |
| --- |
| |
| # Krull-Micro |
|
|
| **Krull** is an acronym for Knowledge Running Under Lightweight Language. |
|
|
| --- |
|
|
| # GitHub |
| https://github.com/machadodecastro/krull-micro.git |
|
|
| --- |
|
|
| Language: en |
|
|
| License: mit |
|
|
| Tags: |
|
|
| * krull |
| * tiny-transformer |
| * distillation |
| * edge-ai |
| * low-memory |
| * onnx |
| * quantization |
| pipeline_tag: text-generation |
| library_name: pytorch |
|
|
| --- |
|
|
| # Krull-Micro |
|
|
| **Krull-Micro** is a distilled, ultra-lightweight GPT-style language model designed for **edge devices** with limited RAM and compute. |
|
|
| It uses an advanced **comprehensive distillation strategy** that transfers: |
|
|
| * Embedding representations |
| * Transformer hidden states |
| * Attention matrices |
| * Output distributions (soft targets) |
|
|
| This allows Krull-Micro to remain extremely small while preserving strong language modeling performance. |
|
|
| --- |
|
|
| ## Key Features |
|
|
| * Tiny Transformer architecture (edge-optimized) |
| * Full distillation (feature-based + attention + response-based) |
| * ONNX export for cross-platform deployment |
| * INT8 quantization support |
| * Designed for **memory-bound inference** |
|
|
| --- |
|
|
| ## Architecture |
|
|
| * Model type: Causal Language Model (GPT-style) |
| * Transformer layers: *(set in config)* |
| * Hidden size: *(set in config)* |
| * Attention heads: *(set in config)* |
| * Vocabulary size: *(matches tokenizer)* |
|
|
| --- |
|
|
| ## Usage (PyTorch) |
|
|
| ```python |
| import torch |
| |
| model = torch.load("krull_micro.pt", map_location="cpu") |
| model.eval() |
| |
| # Example input (token IDs) |
| x = torch.tensor([[1, 5, 23, 42]]) |
| |
| with torch.no_grad(): |
| logits = model(x) |
| ``` |
|
|
| --- |
|
|
| ## ONNX Inference (Edge Deployment) |
|
|
| ```python |
| import onnxruntime as ort |
| import numpy as np |
| |
| session = ort.InferenceSession("krull_micro.onnx") |
| |
| input_ids = np.array([[1, 5, 23, 42]], dtype=np.int64) |
| |
| outputs = session.run(None, {"input_ids": input_ids}) |
| ``` |
|
|
| --- |
|
|
| ## Training |
|
|
| Training is performed using: |
|
|
| ```bash |
| python scripts/train_lm.py \ |
| --config configs/krull_micro.json \ |
| --tokenizer artifacts/tokenizer.json \ |
| --corpus data/tiny_corpus.txt \ |
| --out artifacts/krull_micro.pt \ |
| --epochs 3 \ |
| --batch-size 8 \ |
| --lr 3e-4 |
| ``` |
|
|
| --- |
|
|
| ## Optimization Pipeline |
|
|
| 1. Train distilled model |
| 2. Export to ONNX |
| 3. Apply INT8 quantization |
| 4. Deploy with ONNX Runtime |
|
|
| --- |
|
|
| ## Intended Use |
|
|
| * Embedded systems |
| * Mobile devices |
| * Offline text generation |
| * Low-latency inference environments |
|
|
| --- |
|
|
| ## Limitations |
|
|
| * Small model capacity → limited long-range coherence |
| * Sensitive to training data quality |
| * Not suitable for large-scale reasoning tasks |
|
|
| --- |
|
|
| ## License |
|
|
| MIT License |
|
|
| --- |
|
|
| ## Author |
|
|
| Igor Machado de Castro |
|
|
| --- |
|
|
| ## Contributing |
|
|
| Contributions are welcome. Feel free to open issues or submit pull requests. |
|
|
| --- |
|
|
|
|
