TaskForgeSLM

A GPT-style Small Language Model (~4.9M parameters) trained from scratch on a credit card Q&A domain corpus, demonstrating the TaskForgeSLM framework for building hyper-specialised, on-demand domain agents.

What is TaskForgeSLM?

TaskForgeSLM is an enterprise AI accelerator built around an army of small, domain-specific models — each forged for a single task domain and coordinated by a lightweight intent router. Rather than deploying one large general-purpose model for every task, TaskForgeSLM trains a dedicated agent per domain: credit card queries, balance enquiries, underwriting policy, dispute resolution, and so on.

Model Details

Property	Value
Architecture	Decoder-only causal transformer (GPT-2 style)
Parameters	~4.9M
Layers	6
Attention heads	8
Embedding dimensions	256
Context window	128 tokens
Tokeniser	Character-level (vocab size 59)
Training stages	Pretrain on domain corpus + SFT with loss masking
Hardware	Apple M1 (MPS)

Usage

This model requires the TaskForgeSLM inference code. Clone the repo and run:

python3 main.py --mode infer \
  --model-size small \
  --checkpoint checkpoints/sft_model.pt \
  --tokenizer-type char \
  --prompt "Instruction: How can I avoid late fees?\nResponse: " \
  --temperature 0.2 --top-k 20 --top-p 0.95 \
  --max-new-tokens 500

Training

• Stage 1 — Pretrain: 10,000+ iterations on a domain corpus of credit card product guides, policy documents, and FAQs.
• Stage 2 — SFT: 300 iterations on 50 instruction/response pairs using loss masking so only response tokens contribute gradients.

Limitations

• Small corpus (37 paragraphs) limits output fluency
• 128-token context window (~25 words) — coherence improves at --max-new-tokens 500
• Training from scratch; roadmap targets fine-tuning SmolLM-135M as a base model

Roadmap

See TaskForgeSLM on GitHub for the full roadmap including intent router, LoRA fine-tuning of SmolLM-135M, and FastAPI serving.

Citation

@misc{taskforgeslm2024,
  author = {Arnab Saha},
  title  = {TaskForgeSLM: Domain-Specific Small Language Model Framework},
  year   = {2024},
  url    = {https://huggingface.co/arsaha28/TaskForgeSLM}
}