README.md

---
license: apache-2.0
base_model: Qwen/Qwen2.5-1.5B-Instruct
datasets:
  - umarfarookm/UmarTransit-Instruct-3k
tags:
  - transit
  - gtfs
  - transportation
  - instruction-following
  - qwen2
  - qlora
  - unsloth
language:
  - en
pipeline_tag: text-generation
---

# UmarTransit-1B (v1.0)

A domain-specific language model for **public transit systems** and **GTFS (General Transit Feed Specification)** data, fine-tuned from Qwen2.5-1.5B-Instruct.

UmarTransit-1B specializes in:
- GTFS understanding and validation
- Transit route and schedule analysis
- Journey planning and transfer logic
- Stop/station information
- Transit operations concepts
- Transit network intelligence

> **Data Disclaimer:** This model was trained **exclusively on publicly available, open-source GTFS feeds** published by transit agencies for public use via the [Mobility Database](https://mobilitydatabase.org/). **No private, proprietary, or NDA-protected data** from any client, employer, or organization was used at any stage.

## Model Details

| Property | Value |
|----------|-------|
| **Base Model** | [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) |
| **Parameters** | 1.54B (1.31B non-embedding) |
| **Fine-tuning** | QLoRA (4-bit NF4, LoRA rank=16, alpha=32) |
| **Training Framework** | [Unsloth](https://unsloth.ai) + HuggingFace TRL |
| **Training Data** | 3,154 pairs from [UmarTransit-Instruct-3k](https://huggingface.co/datasets/umarfarookm/UmarTransit-Instruct-3k) |
| **Test Data** | 347 pairs (stratified 90/10 split) |
| **Categories** | 11 (agency, route, stop, schedule, transfer, network stats, GTFS knowledge, comparative, journey planning, GTFS validation, transit operations) |
| **Max Context** | 1,024 tokens |
| **License** | Apache 2.0 |
| **Developer** | [umarfarookm](https://github.com/umarfarookm) |

## What's New in v1.0

- **Expanded dataset:** 3,501 total pairs (up from 3,306 in v0.1)
- **3 new categories:** Journey planning (100 pairs), GTFS validation (20 pairs), Transit operations (20 pairs)
- **Expanded existing categories:** GTFS knowledge (22 → 53), network stats (30 → 45), comparative (14 → 23)
- **Journey planning fixed:** v0.1 scored below the base model on journey planning; v1.0 now beats it

## Evaluation Results

Evaluated on 193 benchmark questions across 6 categories:

| Metric | Base Model | v0.1 | v1.0 | v1.0 vs Base |
|--------|-----------|------|------|-------------|
| ROUGE-L | 0.129 | 0.375 | **0.409** | +217% |
| Keyword Match | 0.368 | 0.403 | **0.398** | +8% |
| Criteria Match | 0.020 | 0.072 | **0.098** | +385% |
| **Combined** | **0.168** | **0.293** | **0.313** | **+86%** |

### Per-Category (Combined Score)

| Category | Base | v0.1 | v1.0 |
|----------|------|------|------|
| GTFS Terminology | 0.342 | 0.351 | **0.361** |
| GTFS Validation | 0.267 | 0.314 | **0.323** |
| Route Analysis | 0.084 | 0.290 | **0.328** |
| Journey Planning | 0.297 | 0.243 | **0.311** |
| Schedule Reasoning | 0.121 | 0.253 | 0.224 |
| Transit Operations | 0.193 | 0.307 | **0.342** |

v1.0 beats the base model in **all 6 categories** (vs 5/6 in v0.1).

## Available Formats

| Format | Size | Use Case |
|--------|------|----------|
| **Safetensors** | ~3.1 GB | Python / Transformers |
| **GGUF Q4_K_M** | ~986 MB | Ollama / llama.cpp (recommended) |
| **GGUF Q8_0** | ~1.65 GB | Ollama / llama.cpp (higher quality) |

## Usage

### With Transformers (Python)

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("umarfarookm/UmarTransit-1B")
tokenizer = AutoTokenizer.from_pretrained("umarfarookm/UmarTransit-1B")

messages = [
    {"role": "system", "content": "You are UmarTransit-1B, a specialized AI assistant for public transit systems and GTFS data."},
    {"role": "user", "content": "What are the required files in a GTFS feed?"},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.1, top_p=0.9, do_sample=True)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
```

### With Ollama

```bash
ollama run hf.co/umarfarookm/UmarTransit-1B:Q4_K_M
```

## Training Details

| Parameter | Value |
|-----------|-------|
| Epochs | 3 |
| Batch size | 4 (x4 gradient accumulation = 16 effective) |
| Learning rate | 2e-4 (cosine schedule) |
| LoRA rank | 16 |
| LoRA alpha | 32 |
| LoRA dropout | 0 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Optimizer | AdamW 8-bit |
| Max sequence length | 1,024 tokens |
| Hardware | Google Colab T4 GPU (~30 min) |

## Training Data Coverage

15 GTFS feeds from 10 countries:

| Country | Agency |
|---------|--------|
| US | LA Metro, CTA (Chicago), MBTA (Boston), Valley Metro (Phoenix), Capital Metro (Austin), TriMet (Portland) |
| Canada | TTC (Toronto) |
| Germany | VBB (Berlin) |
| France | Ile-de-France Mobilites (Paris) |
| Netherlands | OVapi |
| Belgium | NMBS/SNCB |
| Finland | HSL (Helsinki) |
| Denmark | Rejseplanen |
| Australia | Transperth (Perth) |
| New Zealand | Auckland Transport |

## Limitations

- English only — no multilingual support
- Static schedule data only — no real-time predictions
- Not a trip planner — cannot compute optimal routes
- 1,024 token context — limited for very long queries
- Small training dataset (3,501 pairs) — may not generalize to all transit scenarios

## Links

- **GitHub:** [umarfarookm/transit-foundation-model](https://github.com/umarfarookm/transit-foundation-model)
- **Dataset:** [umarfarookm/UmarTransit-Instruct-3k](https://huggingface.co/datasets/umarfarookm/UmarTransit-Instruct-3k)
- **Web Demo:** [transit-foundation-model.vercel.app](https://transit-foundation-model.vercel.app)

## Citation

```bibtex
@model{umartransit_1b,
  author = {Umar Farook M},
  title = {UmarTransit-1B: Domain-Specific LLM for Public Transit and GTFS},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/umarfarookm/UmarTransit-1B}
}
```