README_COMPONENT_4_MODEL_ARCHITECTURE.md

Component 4: Model Architecture (420M Starter)

What This Component Builds

• A decoder-only transformer language model for code generation.
• Configurable size through YAML config.
• Presets for small, medium (420M target), and large.
• Attention + rotary positional encoding + feed-forward blocks.

Main Files

• src/model_architecture/code_transformer.py
• configs/component4_model_config.yaml
• scripts/build_component4_model.py
• scripts/verify_component4_model.py

Commands (run from project root)

.\.venv\Scripts\Activate.ps1
python .\scripts\build_component4_model.py --config .\configs\component4_model_config.yaml
python .\scripts\verify_component4_model.py --config .\configs\component4_model_config.yaml --batch_size 1 --seq_len 256

What Success Looks Like

• Build script prints parameter count near the 420M target.
• Verify script prints:
  • VRAM usage at multiple stages
  • output tensor shape
  • Component 4 verification passed.