File size: 3,231 Bytes
f1af3b8 78d592d f1af3b8 78d592d f1af3b8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 |
---
license: apache-2.0
tags:
- slm
- custom-code
- micro-model
- in-context-learning
pipeline_tag: text-generation
---
# ApproxDumb (560-Parameter Micro-SLM)
`ApproxDumb` is an ultra-micro language model built with just **560 parameters**.
It serves as a proof-of-concept for **In-Context Learning (ICL)** in micro-architectures, demonstrating how a model can infer underlying rules (such as dynamic addition patterns) on-the-fly from a single provided example.
---
## ⚠️ Input & Output Constraints
Due to the extreme parameter budget, this model **cannot** process natural language text or complex multi-digit arithmetic. Please strictly adhere to the following input and output specifications.
### 1. Input Constraints
- **Allowed Vocabulary**:
- **Single Digits**: `0`, `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9`
- **Separator**: `->`
- **Special Tokens**: `<bos>`, `<eos>`, `<pad>`
- *Note: Words, letters, multi-digit numbers (e.g., 10, 100), or unsupported punctuation are **NOT** allowed.*
- **Formatting**:
- Tokens **must be separated by half-width spaces** (e.g., `"1 3 -> 4"`).
- Recommended Prompt Structure: `[Example Input] [Example Output] -> [Query Input]` (Total of 4 tokens).
- **Max Sequence Length**: **8 tokens** (4 to 5 tokens recommended).
### 2. Output Constraints
- **Output Format**: The model predicts the probability distribution (logits) for the **next single digit token (`0`–`9`)**.
- **Limitations**:
- Cannot generate multi-digit numbers ($\ge 10$) or negative numbers.
- Not designed for multi-token free-form text generation (specialized for single next-token prediction).
---
## 💡 Prompt Examples
By providing a single "Example", the model infers the rule and predicts the answer for the "Query".
| Prompt (`input_text`) | Inferred Rule | Expected Prediction |
| :--- | :--- | :--- |
| `"1 2 -> 4"` | $+1$ rule | **`5`** |
| `"1 3 -> 4"` | $+2$ rule | **`6`** |
| `"1 4 -> 2"` | $+3$ rule | **`5`** |
| `"0 4 -> 1"` | $+4$ rule | **`5`** |
---
## 🚀 How to Use
Make sure to pass `trust_remote_code=True` when loading the model and tokenizer.
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load model and tokenizer
model_id = "56m/ApproxDumb" # Replace with your Hugging Face repository
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
# Prepare prompt: Example "1 -> 3" (+2 rule), Query "4 -> ?"
prompt = "1 3 -> 4"
inputs = tokenizer(prompt, return_tensors="pt")
# Inference
model.eval()
with torch.no_grad():
outputs = model(**inputs)
# Extract the most probable next digit token from the final position
next_token_logits = outputs.logits[0, -1, :]
predicted_token_id = torch.argmax(next_token_logits).item()
predicted_symbol = tokenizer.decode([predicted_token_id])
print(f"Input: '{prompt}'")
print(f"Predicted Next Digit: {predicted_symbol}")
```
---
## 🏗️ Model Architecture
- **Architecture**: Decoder-only Micro-Transformer
- **Total Parameters**: 560
- **Embedding Dimension ($d_{model}$)**: 6
- **Transformer Layers**: 1
- **Attention Heads**: 1
- **Vocabulary Size**: 14 |