---
language: en
license: apache-2.0
tags:
  - text-generation
  - domain-specific
  - small-language-model
  - TaskForgeSLM
---

# TaskForgeSLM

A GPT-style Small Language Model (~4.9M parameters) trained from scratch on a
credit card Q&A domain corpus, demonstrating the **TaskForgeSLM** framework for
building hyper-specialised, on-demand domain agents.

## What is TaskForgeSLM?

TaskForgeSLM is an enterprise AI accelerator built around an army of small,
domain-specific models — each forged for a single task domain and coordinated
by a lightweight intent router. Rather than deploying one large general-purpose
model for every task, TaskForgeSLM trains a dedicated agent per domain: credit
card queries, balance enquiries, underwriting policy, dispute resolution, and so on.

## Model Details

| Property | Value |
|---|---|
| Architecture | Decoder-only causal transformer (GPT-2 style) |
| Parameters | ~4.9M |
| Layers | 6 |
| Attention heads | 8 |
| Embedding dimensions | 256 |
| Context window | 128 tokens |
| Tokeniser | Character-level (vocab size 59) |
| Training stages | Pretrain on domain corpus + SFT with loss masking |
| Hardware | Apple M1 (MPS) |

## Usage

This model requires the TaskForgeSLM inference code. Clone the repo and run:

```bash
python3 main.py --mode infer \
  --model-size small \
  --checkpoint checkpoints/sft_model.pt \
  --tokenizer-type char \
  --prompt "Instruction: How can I avoid late fees?\nResponse: " \
  --temperature 0.2 --top-k 20 --top-p 0.95 \
  --max-new-tokens 500
```

## Training

- **Stage 1 — Pretrain**: 10,000+ iterations on a domain corpus of credit card
  product guides, policy documents, and FAQs.
- **Stage 2 — SFT**: 300 iterations on 50 instruction/response pairs using loss
  masking so only response tokens contribute gradients.

## Limitations

- Small corpus (37 paragraphs) limits output fluency
- 128-token context window (~25 words) — coherence improves at `--max-new-tokens 500`
- Training from scratch; roadmap targets fine-tuning SmolLM-135M as a base model

## Roadmap

See [TaskForgeSLM on GitHub](https://github.com/arsaha28/TaskForgeSLM) for the
full roadmap including intent router, LoRA fine-tuning of SmolLM-135M, and
FastAPI serving.

## Citation

```
@misc{taskforgeslm2024,
  author = {Arnab Saha},
  title  = {TaskForgeSLM: Domain-Specific Small Language Model Framework},
  year   = {2024},
  url    = {https://huggingface.co/arsaha28/TaskForgeSLM}
}
```