| --- |
| language: en |
| license: apache-2.0 |
| tags: |
| - text-generation |
| - domain-specific |
| - small-language-model |
| - TaskForgeSLM |
| --- |
| |
| # TaskForgeSLM |
|
|
| A GPT-style Small Language Model (~4.9M parameters) trained from scratch on a |
| credit card Q&A domain corpus, demonstrating the **TaskForgeSLM** framework for |
| building hyper-specialised, on-demand domain agents. |
|
|
| ## What is TaskForgeSLM? |
|
|
| TaskForgeSLM is an enterprise AI accelerator built around an army of small, |
| domain-specific models — each forged for a single task domain and coordinated |
| by a lightweight intent router. Rather than deploying one large general-purpose |
| model for every task, TaskForgeSLM trains a dedicated agent per domain: credit |
| card queries, balance enquiries, underwriting policy, dispute resolution, and so on. |
|
|
| ## Model Details |
|
|
| | Property | Value | |
| |---|---| |
| | Architecture | Decoder-only causal transformer (GPT-2 style) | |
| | Parameters | ~4.9M | |
| | Layers | 6 | |
| | Attention heads | 8 | |
| | Embedding dimensions | 256 | |
| | Context window | 128 tokens | |
| | Tokeniser | Character-level (vocab size 59) | |
| | Training stages | Pretrain on domain corpus + SFT with loss masking | |
| | Hardware | Apple M1 (MPS) | |
|
|
| ## Usage |
|
|
| This model requires the TaskForgeSLM inference code. Clone the repo and run: |
|
|
| ```bash |
| python3 main.py --mode infer \ |
| --model-size small \ |
| --checkpoint checkpoints/sft_model.pt \ |
| --tokenizer-type char \ |
| --prompt "Instruction: How can I avoid late fees?\nResponse: " \ |
| --temperature 0.2 --top-k 20 --top-p 0.95 \ |
| --max-new-tokens 500 |
| ``` |
|
|
| ## Training |
|
|
| - **Stage 1 — Pretrain**: 10,000+ iterations on a domain corpus of credit card |
| product guides, policy documents, and FAQs. |
| - **Stage 2 — SFT**: 300 iterations on 50 instruction/response pairs using loss |
| masking so only response tokens contribute gradients. |
|
|
| ## Limitations |
|
|
| - Small corpus (37 paragraphs) limits output fluency |
| - 128-token context window (~25 words) — coherence improves at `--max-new-tokens 500` |
| - Training from scratch; roadmap targets fine-tuning SmolLM-135M as a base model |
|
|
| ## Roadmap |
|
|
| See [TaskForgeSLM on GitHub](https://github.com/arsaha28/TaskForgeSLM) for the |
| full roadmap including intent router, LoRA fine-tuning of SmolLM-135M, and |
| FastAPI serving. |
|
|
| ## Citation |
|
|
| ``` |
| @misc{taskforgeslm2024, |
| author = {Arnab Saha}, |
| title = {TaskForgeSLM: Domain-Specific Small Language Model Framework}, |
| year = {2024}, |
| url = {https://huggingface.co/arsaha28/TaskForgeSLM} |
| } |
| ``` |
|
|
