|
|
| --- |
| license: apache-2.0 |
| tags: |
| - slm |
| - custom-code |
| - micro-model |
| - in-context-learning |
| pipeline_tag: text-generation |
| --- |
| |
| # ApproxDumb (560-Parameter Micro-SLM) |
|
|
| `ApproxDumb` is an ultra-micro language model built with just **560 parameters**. |
| It serves as a proof-of-concept for **In-Context Learning (ICL)** in micro-architectures, demonstrating how a model can infer underlying rules (such as dynamic addition patterns) on-the-fly from a single provided example. |
|
|
| --- |
|
|
| ## ⚠️ Input & Output Constraints |
|
|
| Due to the extreme parameter budget, this model **cannot** process natural language text or complex multi-digit arithmetic. Please strictly adhere to the following input and output specifications. |
|
|
| ### 1. Input Constraints |
| - **Allowed Vocabulary**: |
| - **Single Digits**: `0`, `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9` |
| - **Separator**: `->` |
| - **Special Tokens**: `<bos>`, `<eos>`, `<pad>` |
| - *Note: Words, letters, multi-digit numbers (e.g., 10, 100), or unsupported punctuation are **NOT** allowed.* |
| - **Formatting**: |
| - Tokens **must be separated by half-width spaces** (e.g., `"1 3 -> 4"`). |
| - Recommended Prompt Structure: `[Example Input] [Example Output] -> [Query Input]` (Total of 4 tokens). |
| - **Max Sequence Length**: **8 tokens** (4 to 5 tokens recommended). |
|
|
| ### 2. Output Constraints |
| - **Output Format**: The model predicts the probability distribution (logits) for the **next single digit token (`0`–`9`)**. |
| - **Limitations**: |
| - Cannot generate multi-digit numbers ($\ge 10$) or negative numbers. |
| - Not designed for multi-token free-form text generation (specialized for single next-token prediction). |
|
|
| --- |
|
|
| ## 💡 Prompt Examples |
|
|
| By providing a single "Example", the model infers the rule and predicts the answer for the "Query". |
|
|
| | Prompt (`input_text`) | Inferred Rule | Expected Prediction | |
| | :--- | :--- | :--- | |
| | `"1 2 -> 4"` | $+1$ rule | **`5`** | |
| | `"1 3 -> 4"` | $+2$ rule | **`6`** | |
| | `"1 4 -> 2"` | $+3$ rule | **`5`** | |
| | `"0 4 -> 1"` | $+4$ rule | **`5`** | |
|
|
| --- |
|
|
| ## 🚀 How to Use |
|
|
| Make sure to pass `trust_remote_code=True` when loading the model and tokenizer. |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| # Load model and tokenizer |
| model_id = "56m/ApproxDumb" # Replace with your Hugging Face repository |
| tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True) |
| |
| # Prepare prompt: Example "1 -> 3" (+2 rule), Query "4 -> ?" |
| prompt = "1 3 -> 4" |
| inputs = tokenizer(prompt, return_tensors="pt") |
| |
| # Inference |
| model.eval() |
| with torch.no_grad(): |
| outputs = model(**inputs) |
| |
| # Extract the most probable next digit token from the final position |
| next_token_logits = outputs.logits[0, -1, :] |
| predicted_token_id = torch.argmax(next_token_logits).item() |
| predicted_symbol = tokenizer.decode([predicted_token_id]) |
| |
| print(f"Input: '{prompt}'") |
| print(f"Predicted Next Digit: {predicted_symbol}") |
| ``` |
|
|
| --- |
|
|
| ## 🏗️ Model Architecture |
|
|
| - **Architecture**: Decoder-only Micro-Transformer |
| - **Total Parameters**: 560 |
| - **Embedding Dimension ($d_{model}$)**: 6 |
| - **Transformer Layers**: 1 |
| - **Attention Heads**: 1 |
| - **Vocabulary Size**: 14 |
