README.md

metadata

language:
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - causal-lm
  - base-model
  - non-instruction-tuned
  - non-it
  - llama
  - from-scratch
  - medical
  - education
  - slm-train-scratch
license: other

blue-scrub-150M

Phase 5 Hugging Face deployment from slm-train-scratch.

Model summary

• Model type: decoder-only causal language model (LlamaForCausalLM).
• Model size: 150M-class; measured checkpoint size is 154.42M parameters.
• Instruction tuning: Non-IT / non-instruction-tuned. This is a base language model, not a chat model and not instruction aligned.
• Tokenizer: byte-level BPE-style tokenizer with vocabulary size 32000.
• Precision: bfloat16 checkpoint.

Training data

The model was pretrained on a cleaned mixture of general educational text and medical/health-domain text.

• General/education source: HuggingFaceFW/FineWeb-Edu, cleaned and filtered subset: 2.00M rows.
• Medical/health source: TheBlueScrubs/the_blue_scrubs-v1, cleaned and filtered subset: 2.00M rows.
• Packed train tokens: 5.72B tokens.
• Packed validation tokens: 18.22M tokens.

Training run

• Epochs configured: 1.
• Approximate logged wall-clock training time: 73.3 hours (3.1 days). This is based on recorded metrics and may include evaluation, resume, and pause gaps.

Intended use

This checkpoint is intended for:

• research experiments with small/base language models;
• continued pretraining;
• domain adaptation experiments;
• evaluation of a compact medical-plus-education pretrained base model.

Limitations

• This is not an instruction-tuned or chat-aligned model.
• Outputs may be incomplete, incorrect, or unsafe without downstream alignment and evaluation.
• The model must not be used as a source of medical advice.
• The training mixture contains web and domain text; users should evaluate bias, factuality, memorization, and domain safety before downstream use.

Basic loading

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "mps/blue-scrub-150M"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype="auto")