KittyLM Research
🐱
Modular language-model research from first principles
KRAZYKITTY // MODEL TRAINING
Compute-aware pretraining, parameter-efficient adaptation, quantised finetuning, and preference optimisation.
Note KrazyKitty's public engineering and evaluation roadmap; current showcase makes no model-quality claims.
Training Compute-Optimal Large Language Models
Paper • 2203.15556 • Published • 13Note Compute-optimal allocation between model size and training tokens.
LoRA: Low-Rank Adaptation of Large Language Models
Paper • 2106.09685 • Published • 67Note Low-rank parameter-efficient model adaptation.
QLoRA: Efficient Finetuning of Quantized LLMs
Paper • 2305.14314 • Published • 64Note Quantised low-rank finetuning and memory-efficiency study.
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Paper • 2305.18290 • Published • 72Note Direct preference optimisation without a separate reward-model training loop.
HuggingFaceTB/SmolLM2-1.7B
Text Generation • 2B • Updated • 41.6k • 165Note Compact pretraining reference; recheck its card and revision before experiments.
HuggingFaceH4/zephyr-7b-beta
Text Generation • 7B • Updated • 88.8k • • 1.86kNote Alignment reference with documented SFT and preference-data lineage.