README.md

---
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- causal-lm
- base-model
- non-instruction-tuned
- non-it
- llama
- from-scratch
- medical
- education
- slm-train-scratch
license: other
---

# blue-scrub-150M

Phase 5 Hugging Face deployment from slm-train-scratch.

## Model summary

- **Model type:** decoder-only causal language model (`LlamaForCausalLM`).
- **Model size:** 150M-class; measured checkpoint size is **154.42M parameters**.
- **Instruction tuning:** **Non-IT / non-instruction-tuned**. This is a base language model, not a chat model and not instruction aligned.
- **Tokenizer:** byte-level BPE-style tokenizer with vocabulary size 32000.
- **Precision:** bfloat16 checkpoint.

## Training data

The model was pretrained on a cleaned mixture of general educational text and medical/health-domain text.

- **General/education source:** HuggingFaceFW/FineWeb-Edu, cleaned and filtered subset: 2.00M rows.
- **Medical/health source:** TheBlueScrubs/the_blue_scrubs-v1, cleaned and filtered subset: 2.00M rows.
- **Packed train tokens:** 5.72B tokens.
- **Packed validation tokens:** 18.22M tokens. 


## Training run

- **Epochs configured:** 1. 
- **Approximate logged wall-clock training time:** 73.3 hours (3.1 days). This is based on recorded metrics and may include evaluation, resume, and pause gaps.

## Intended use

This checkpoint is intended for:

- research experiments with small/base language models;
- continued pretraining;
- domain adaptation experiments;
- evaluation of a compact medical-plus-education pretrained base model.

## Limitations

- This is **not** an instruction-tuned or chat-aligned model.
- Outputs may be incomplete, incorrect, or unsafe without downstream alignment and evaluation.
- The model must not be used as a source of medical advice.
- The training mixture contains web and domain text; users should evaluate bias, factuality, memorization, and domain safety before downstream use.

## Basic loading

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "mps/blue-scrub-150M"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype="auto")
```