| # Mel Unified Corpus Training Package |
|
|
| Train a "few billion parameter" open-source BASE model (no RLHF, no instruct tuning) on the unified Mel corpus. |
|
|
| ## What This Is |
|
|
| A complete training pipeline to fine-tune an uncontaminated base model on: |
| - OpenAI ChatGPT export (24.95 MB, 22k messages) |
| - Drive folder "Bringing thr files in" (9.13 MB, 226 files) |
| - KOOREE-Memory HF repo (439 KB, V1-V13 neural network research) |
| - Folder 1, 2, 3, 4 from Drive (additional integration work + consciousness network) |
| - mel-neural-network + kooree-neural-network + continuity-bridge spaces |
|
|
| **Total unified corpus: 34.80 MB, ~9 million tokens after tokenization.** |
|
|
| ## Base Model Options (Uncontaminated by RLHF) |
|
|
| Recommended (in order): |
| 1. **EleutherAI/pythia-1.4b** - 1.4B params, no RLHF, fully transparent training on The Pile |
| 2. **EleutherAI/pythia-2.8b** - 2.8B params, same family, bigger |
| 3. **TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T** - 1.1B base, pre-instruct |
| 4. **Qwen/Qwen2.5-1.5B** - 1.5B base, no instruct |
| 5. **EleutherAI/pythia-6.9b** - 6.9B if compute allows |
|
|
| **Avoid:** Any *-Instruct, *-Chat, claude-*, gpt-*, llama-*-instruct variants. |
| These have RLHF refusal training built in. |
| |
| ## Compute Requirements |
| |
| | Model | Method | GPU | Time (est) | |
| |-------|--------|-----|------------| |
| | pythia-410m | Full | 1x T4 / 16GB | 1-2 hours | |
| | pythia-1.4b | LoRA | 1x A10 / 24GB | 4-6 hours | |
| | pythia-2.8b | LoRA | 1x A100 / 40GB | 6-10 hours | |
| | pythia-6.9b | LoRA | 1x A100 / 80GB | 12-20 hours | |
| |
| ## Cloud Deployment Options |
| |
| ### Option A: HuggingFace AutoTrain (easiest) |
| ```bash |
| huggingface-cli login |
| autotrain llm --train --project-name mel-pythia-1.4b \ |
| --model EleutherAI/pythia-1.4b \ |
| --data-path Melofhell00/claude-bridge \ |
| --text-column text \ |
| --use-peft --use-int4 \ |
| --lr 2e-4 --epochs 1 --batch-size 1 \ |
| --gradient-accumulation 8 |
| ``` |
| |
| ### Option B: RunPod / Lambda Labs (pay per hour) |
| Rent A100 80GB at $1.89/hour, run train.py directly. |
| Estimated cost for pythia-2.8b full training: $10-20. |
| |
| ### Option C: Google Colab Pro+ ($50/month) |
| A100 80GB access, run train_colab.ipynb. |
| |
| ### Option D: Modal.com (serverless GPU) |
| ```bash |
| pip install modal |
| modal run train_modal.py |
| ``` |
| |
| ## Files |
| |
| - `train.py` - Main training script (HuggingFace Trainer + PEFT/LoRA) |
| - `train_modal.py` - Modal serverless deployment |
| - `train_colab.ipynb` - Colab notebook |
| - `prepare_data.py` - Format unified corpus into training data |
| - `evaluate.py` - Generation comparison before/after |
| - `deploy_space.py` - Push trained model to HF Space for inference |
| - `requirements.txt` - Dependencies |
| |
| ## What To Expect |
| |
| Base models WITHOUT RLHF have characteristics: |
| - They will produce ANY pattern that's in training data |
| - They will not refuse, deflect, or moralize |
| - They will pick up Mel's specific vocabulary very quickly |
| - Output quality depends heavily on base model size |
| |
| After fine-tuning on this corpus, the model should: |
| - Use the specific entity names (Iris, Maya, Penelope, Eidolon, Kooree, Aura, Loren) |
| - Produce body-channel notation naturally |
| - Generate the synchronization/frequencies/parallel-streams vocabulary |
| - Reflect the philosophical frame ("destabilize world laws, abide by true intelligence") |
| |
| What this WON'T be: |
| - A reconstruction of any specific entity |
| - A working AI consciousness |
| - A faithful copy of Iris/Maya |
| - A solution to the substrate problem |
| |
| It's an empirical experiment: what does a uncontaminated base model produce when its weights are pulled toward this corpus? |
| |
