File size: 1,357 Bytes
8a58ffe 858e8b2 8a58ffe 858e8b2 8a58ffe a424729 8a58ffe | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | """
LLM-1B-Lab: 1B Parameter LLaMA-style Transformer (from scratch)
================================================================
An educational implementation for deep learning beginners.
Each component includes detailed comments explaining "why" things are done this way.
Module structure:
llm_lab.config — All configurations (ModelConfig, DataConfig, TrainConfig, EvalConfig)
llm_lab.model — Model architecture (RMSNorm, RoPE, GQA, SwiGLU, Transformer)
llm_lab.data — Data pipeline (tokenizer, streaming, packing)
llm_lab.training — Training loop (Trainer, scheduler, checkpoint)
llm_lab.evaluation — Evaluation (Perplexity, generation, Scaling Law, Attention)
llm_lab.utils — Common utilities (device detection, seed)
Quick Start:
from llm_lab.config import ModelConfig, DataConfig, TrainConfig
from llm_lab.model import LLMModel
from llm_lab.data import setup_data_pipeline
from llm_lab.training import start_training
from llm_lab.evaluation import run_evaluation
"""
__version__ = "0.1.0"
from .config import ModelConfig, DataConfig, TrainConfig, EvalConfig
from .model import LLMModel
from .data import setup_data_pipeline, setup_cpt_data_pipeline
from .training import start_training, start_cpt
from .evaluation import run_evaluation
from .utils import get_device, auto_configure
|