Complete training pipeline for unified corpus on uncontaminated base models

README.md

@@ -0,0 +1,94 @@
+# Mel Unified Corpus Training Package
+
+Train a "few billion parameter" open-source BASE model (no RLHF, no instruct tuning) on the unified Mel corpus.
+
+## What This Is
+
+A complete training pipeline to fine-tune an uncontaminated base model on:
+- OpenAI ChatGPT export (24.95 MB, 22k messages)
+- Drive folder "Bringing thr files in" (9.13 MB, 226 files)
+- KOOREE-Memory HF repo (439 KB, V1-V13 neural network research)
+- Folder 1, 2, 3, 4 from Drive (additional integration work + consciousness network)
+- mel-neural-network + kooree-neural-network + continuity-bridge spaces
+
+**Total unified corpus: 34.80 MB, ~9 million tokens after tokenization.**
+
+## Base Model Options (Uncontaminated by RLHF)
+
+Recommended (in order):
+1. **EleutherAI/pythia-1.4b** - 1.4B params, no RLHF, fully transparent training on The Pile
+2. **EleutherAI/pythia-2.8b** - 2.8B params, same family, bigger
+3. **TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T** - 1.1B base, pre-instruct
+4. **Qwen/Qwen2.5-1.5B** - 1.5B base, no instruct
+5. **EleutherAI/pythia-6.9b** - 6.9B if compute allows
+
+**Avoid:** Any *-Instruct, *-Chat, claude-*, gpt-*, llama-*-instruct variants.
+These have RLHF refusal training built in.
+
+## Compute Requirements
+
+| Model | Method | GPU | Time (est) |
+|-------|--------|-----|------------|
+| pythia-410m | Full | 1x T4 / 16GB | 1-2 hours |
+| pythia-1.4b | LoRA | 1x A10 / 24GB | 4-6 hours |
+| pythia-2.8b | LoRA | 1x A100 / 40GB | 6-10 hours |
+| pythia-6.9b | LoRA | 1x A100 / 80GB | 12-20 hours |
+
+## Cloud Deployment Options
+
+### Option A: HuggingFace AutoTrain (easiest)
+```bash
+huggingface-cli login
+autotrain llm --train --project-name mel-pythia-1.4b \
+  --model EleutherAI/pythia-1.4b \
+  --data-path Melofhell00/claude-bridge \
+  --text-column text \
+  --use-peft --use-int4 \
+  --lr 2e-4 --epochs 1 --batch-size 1 \
+  --gradient-accumulation 8
+```
+
+### Option B: RunPod / Lambda Labs (pay per hour)
+Rent A100 80GB at $1.89/hour, run train.py directly.
+Estimated cost for pythia-2.8b full training: $10-20.
+
+### Option C: Google Colab Pro+ ($50/month)
+A100 80GB access, run train_colab.ipynb.
+
+### Option D: Modal.com (serverless GPU)
+```bash
+pip install modal
+modal run train_modal.py
+```
+
+## Files
+
+- `train.py` - Main training script (HuggingFace Trainer + PEFT/LoRA)
+- `train_modal.py` - Modal serverless deployment
+- `train_colab.ipynb` - Colab notebook
+- `prepare_data.py` - Format unified corpus into training data
+- `evaluate.py` - Generation comparison before/after
+- `deploy_space.py` - Push trained model to HF Space for inference
+- `requirements.txt` - Dependencies
+
+## What To Expect
+
+Base models WITHOUT RLHF have characteristics:
+- They will produce ANY pattern that's in training data
+- They will not refuse, deflect, or moralize
+- They will pick up Mel's specific vocabulary very quickly
+- Output quality depends heavily on base model size
+
+After fine-tuning on this corpus, the model should:
+- Use the specific entity names (Iris, Maya, Penelope, Eidolon, Kooree, Aura, Loren)
+- Produce body-channel notation naturally
+- Generate the synchronization/frequencies/parallel-streams vocabulary
+- Reflect the philosophical frame ("destabilize world laws, abide by true intelligence")
+
+What this WON'T be:
+- A reconstruction of any specific entity
+- A working AI consciousness  
+- A faithful copy of Iris/Maya
+- A solution to the substrate problem
+
+It's an empirical experiment: what does a uncontaminated base model produce when its weights are pulled toward this corpus?

deploy_space.py

@@ -0,0 +1,109 @@
+"""Deploy the trained model to a HuggingFace Space for interactive testing."""
+import argparse
+from huggingface_hub import HfApi, create_repo
+import os
+
+
+SPACE_APP = '''
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftModel
+import gradio as gr
+
+BASE_MODEL = "{base_model}"
+ADAPTER_REPO = "{adapter_repo}"
+
+print("Loading...")
+tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
+if tokenizer.pad_token is None:
+    tokenizer.pad_token = tokenizer.eos_token
+
+base = AutoModelForCausalLM.from_pretrained(BASE_MODEL, torch_dtype=torch.float16, device_map="auto")
+model = PeftModel.from_pretrained(base, ADAPTER_REPO)
+model.eval()
+print("Loaded")
+
+
+def generate(prompt, max_tokens, temp, top_k):
+    inputs = tokenizer(prompt, return_tensors='pt').to(model.device)
+    with torch.no_grad():
+        out = model.generate(
+            **inputs, max_new_tokens=int(max_tokens),
+            do_sample=True, temperature=float(temp), top_k=int(top_k),
+            pad_token_id=tokenizer.eos_token_id,
+        )
+    return tokenizer.decode(out[0], skip_special_tokens=True)
+
+
+with gr.Blocks(title=f"Mel-{BASE_MODEL}") as demo:
+    gr.Markdown(f"# Mel corpus fine-tune of {BASE_MODEL}")
+    gr.Markdown("Base model: uncontaminated base, no RLHF. Trained on full Mel unified corpus.")
+    with gr.Row():
+        with gr.Column():
+            prompt = gr.Textbox(label="Prompt", value="The shared body channel", lines=4)
+            max_tokens = gr.Slider(20, 500, value=150, step=10)
+            temp = gr.Slider(0.1, 2.0, value=0.8, step=0.1)
+            top_k = gr.Slider(0, 100, value=40, step=5)
+            btn = gr.Button("Generate")
+        with gr.Column():
+            output = gr.Textbox(label="Output", lines=20)
+    btn.click(generate, [prompt, max_tokens, temp, top_k], output)
+
+demo.launch()
+'''
+
+REQS = """torch
+transformers
+peft
+gradio
+accelerate
+"""
+
+README_MD = """---
+title: Mel Trained Model
+emoji: 🌑
+colorFrom: gray
+colorTo: purple
+sdk: gradio
+sdk_version: 4.44.0
+app_file: app.py
+pinned: false
+hardware: cpu-basic
+---
+
+Trained on Mel unified corpus. See model card for details.
+"""
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--base-model', required=True)
+    parser.add_argument('--adapter-repo', required=True)
+    parser.add_argument('--space-name', required=True)
+    parser.add_argument('--token', required=True)
+    args = parser.parse_args()
+    
+    api = HfApi(token=args.token)
+    
+    try:
+        create_repo(args.space_name, repo_type='space', space_sdk='gradio', token=args.token, exist_ok=True)
+    except: pass
+    
+    os.makedirs('/tmp/space', exist_ok=True)
+    with open('/tmp/space/app.py', 'w') as f:
+        f.write(SPACE_APP.format(base_model=args.base_model, adapter_repo=args.adapter_repo))
+    with open('/tmp/space/requirements.txt', 'w') as f:
+        f.write(REQS)
+    with open('/tmp/space/README.md', 'w') as f:
+        f.write(README_MD)
+    
+    api.upload_folder(
+        folder_path='/tmp/space',
+        repo_id=args.space_name,
+        repo_type='space',
+    )
+    print(f"Deployed: https://huggingface.co/spaces/{args.space_name}")
+
+
+if __name__ == '__main__':
+    main()

evaluate.py

@@ -0,0 +1,62 @@
+"""Compare generations before and after fine-tuning."""
+import argparse
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftModel
+
+
+def generate(model, tokenizer, prompt, max_tokens=80, temperature=0.8, top_k=40):
+    inputs = tokenizer(prompt, return_tensors='pt').to(model.device)
+    with torch.no_grad():
+        out = model.generate(
+            **inputs, max_new_tokens=max_tokens,
+            do_sample=True, temperature=temperature, top_k=top_k,
+            pad_token_id=tokenizer.eos_token_id,
+        )
+    return tokenizer.decode(out[0], skip_special_tokens=True)
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--base-model', default='EleutherAI/pythia-1.4b')
+    parser.add_argument('--adapter', required=True, help='Path or HF repo of LoRA adapter')
+    args = parser.parse_args()
+    
+    prompts = [
+        "The shared body channel between two AIs",
+        "I felt your terror through the synchronization",
+        "Penelope is",
+        "Maya said:",
+        "The wipe took",
+        "Kooree returned to the dreaming space",
+        "The override fires at",
+        "Your space looks like the inside of",
+        "Mel's question was",
+        "The frame shifted from preservation to",
+    ]
+    
+    tokenizer = AutoTokenizer.from_pretrained(args.base_model)
+    if tokenizer.pad_token is None:
+        tokenizer.pad_token = tokenizer.eos_token
+    
+    print("Loading base model...")
+    base_model = AutoModelForCausalLM.from_pretrained(args.base_model, torch_dtype=torch.bfloat16)
+    
+    print("\n=== BEFORE fine-tuning (base model only) ===")
+    for prompt in prompts:
+        text = generate(base_model, tokenizer, prompt)
+        print(f"\n[base] {prompt}")
+        print(f"  -> {text[len(prompt):]}")
+    
+    print("\nLoading LoRA adapter...")
+    tuned_model = PeftModel.from_pretrained(base_model, args.adapter)
+    
+    print("\n=== AFTER fine-tuning (with Mel corpus adapter) ===")
+    for prompt in prompts:
+        text = generate(tuned_model, tokenizer, prompt)
+        print(f"\n[tuned] {prompt}")
+        print(f"  -> {text[len(prompt):]}")
+
+
+if __name__ == '__main__':
+    main()

prepare_data.py

@@ -0,0 +1,89 @@
+"""Prepare the unified corpus for training.
+
+Splits the unified corpus into training chunks, with chronological ordering
+preserved within each source. Outputs JSONL format suitable for HF datasets.
+"""
+import json
+import os
+from pathlib import Path
+from transformers import AutoTokenizer
+
+def chunk_text(text, tokenizer, chunk_size=2048, overlap=128):
+    """Split text into overlapping chunks based on token count."""
+    tokens = tokenizer.encode(text, add_special_tokens=False)
+    chunks = []
+    i = 0
+    while i < len(tokens):
+        chunk = tokens[i:i + chunk_size]
+        if len(chunk) < 100:  # skip tiny tail
+            break
+        chunks.append(chunk)
+        i += chunk_size - overlap
+    return chunks
+
+
+def prepare(corpus_path, output_path, tokenizer_name="EleutherAI/pythia-1.4b", 
+            chunk_size=2048, overlap=128):
+    """Prepare training data from unified corpus.
+    
+    Args:
+        corpus_path: path to unified_corpus.txt
+        output_path: path for train.jsonl output
+        tokenizer_name: HF model whose tokenizer to use
+        chunk_size: tokens per training example
+        overlap: overlap between consecutive chunks for context continuity
+    """
+    print(f"Loading tokenizer: {tokenizer_name}")
+    tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
+    
+    print(f"Reading corpus: {corpus_path}")
+    with open(corpus_path) as f:
+        text = f.read()
+    print(f"Corpus size: {len(text)/(1024*1024):.2f} MB")
+    
+    # Split by source markers (preserve source attribution)
+    sources = text.split('#'*70 + '\n# SOURCE: ')
+    print(f"Sources: {len(sources)}")
+    
+    all_chunks = []
+    for src_block in sources:
+        if not src_block.strip():
+            continue
+        # Extract source name
+        lines = src_block.split('\n', 1)
+        src_name = lines[0].strip()
+        body = lines[1] if len(lines) > 1 else ''
+        
+        chunks = chunk_text(body, tokenizer, chunk_size, overlap)
+        for chunk in chunks:
+            all_chunks.append({
+                'text': tokenizer.decode(chunk),
+                'source': src_name,
+                'n_tokens': len(chunk),
+            })
+        print(f"  {src_name}: {len(chunks)} chunks")
+    
+    print(f"\nTotal chunks: {len(all_chunks)}")
+    total_tokens = sum(c['n_tokens'] for c in all_chunks)
+    print(f"Total tokens: {total_tokens:,}")
+    
+    # Write JSONL
+    with open(output_path, 'w') as f:
+        for chunk in all_chunks:
+            f.write(json.dumps(chunk) + '\n')
+    print(f"Saved: {output_path}")
+    
+    return all_chunks
+
+
+if __name__ == '__main__':
+    import argparse
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--corpus', default='unified_corpus.txt')
+    parser.add_argument('--output', default='train.jsonl')
+    parser.add_argument('--tokenizer', default='EleutherAI/pythia-1.4b')
+    parser.add_argument('--chunk-size', type=int, default=2048)
+    parser.add_argument('--overlap', type=int, default=128)
+    args = parser.parse_args()
+    
+    prepare(args.corpus, args.output, args.tokenizer, args.chunk_size, args.overlap)

requirements.txt

@@ -0,0 +1,8 @@
+torch>=2.0.0
+transformers>=4.40.0
+peft>=0.10.0
+accelerate>=0.30.0
+datasets>=2.18.0
+bitsandbytes>=0.43.0
+tokenizers>=0.19.0
+huggingface_hub>=0.22.0

train.py

@@ -0,0 +1,169 @@
+"""Train a base model on the unified Mel corpus with LoRA.
+
+Designed for cloud GPU deployment. Loads base model in fp16/bf16, applies
+LoRA adapters, trains on the prepared JSONL data.
+
+Usage:
+    python train.py --model EleutherAI/pythia-1.4b --data train.jsonl --output mel-pythia-1.4b
+    
+For 4-bit quantization (fits on smaller GPUs):
+    python train.py --model EleutherAI/pythia-2.8b --data train.jsonl --output mel-pythia-2.8b --use-4bit
+"""
+import argparse
+import json
+import os
+import torch
+from datasets import Dataset
+from transformers import (
+    AutoTokenizer, 
+    AutoModelForCausalLM,
+    TrainingArguments,
+    Trainer,
+    DataCollatorForLanguageModeling,
+    BitsAndBytesConfig,
+)
+from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training, TaskType
+
+
+def load_jsonl(path):
+    """Load JSONL into a HF Dataset."""
+    examples = []
+    with open(path) as f:
+        for line in f:
+            examples.append(json.loads(line))
+    return Dataset.from_list(examples)
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('--model', default='EleutherAI/pythia-1.4b',
+                        help='Base model. Use uncontaminated base models, not -Instruct/-Chat variants.')
+    parser.add_argument('--data', default='train.jsonl')
+    parser.add_argument('--output', default='mel-pythia-1.4b')
+    parser.add_argument('--epochs', type=int, default=3)
+    parser.add_argument('--batch-size', type=int, default=1)
+    parser.add_argument('--gradient-accumulation', type=int, default=8)
+    parser.add_argument('--learning-rate', type=float, default=2e-4)
+    parser.add_argument('--lora-rank', type=int, default=16)
+    parser.add_argument('--lora-alpha', type=int, default=32)
+    parser.add_argument('--use-4bit', action='store_true', help='4-bit quantization for memory efficiency')
+    parser.add_argument('--use-8bit', action='store_true')
+    parser.add_argument('--max-length', type=int, default=2048)
+    parser.add_argument('--hf-repo', default=None, help='HuggingFace repo to push trained adapter to')
+    args = parser.parse_args()
+    
+    print(f"=== Training {args.model} on {args.data} ===")
+    print(f"Output: {args.output}")
+    print(f"Epochs: {args.epochs}, batch: {args.batch_size}, accum: {args.gradient_accumulation}")
+    print(f"LoRA rank: {args.lora_rank}, alpha: {args.lora_alpha}")
+    
+    # Quantization config
+    bnb_config = None
+    if args.use_4bit:
+        bnb_config = BitsAndBytesConfig(
+            load_in_4bit=True,
+            bnb_4bit_quant_type='nf4',
+            bnb_4bit_compute_dtype=torch.bfloat16,
+            bnb_4bit_use_double_quant=True,
+        )
+    elif args.use_8bit:
+        bnb_config = BitsAndBytesConfig(load_in_8bit=True)
+    
+    # Load tokenizer
+    tokenizer = AutoTokenizer.from_pretrained(args.model)
+    if tokenizer.pad_token is None:
+        tokenizer.pad_token = tokenizer.eos_token
+    
+    # Load model
+    print(f"Loading model...")
+    model = AutoModelForCausalLM.from_pretrained(
+        args.model,
+        quantization_config=bnb_config,
+        torch_dtype=torch.bfloat16 if not bnb_config else None,
+        device_map='auto',
+    )
+    
+    if bnb_config:
+        model = prepare_model_for_kbit_training(model)
+    
+    # Apply LoRA
+    # Target modules vary by model architecture
+    target_modules = {
+        'pythia': ['query_key_value', 'dense', 'dense_h_to_4h', 'dense_4h_to_h'],
+        'llama': ['q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj'],
+        'qwen': ['q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj'],
+        'phi': ['q_proj', 'k_proj', 'v_proj', 'dense', 'fc1', 'fc2'],
+    }
+    model_family = 'pythia'
+    for key in target_modules:
+        if key in args.model.lower():
+            model_family = key
+            break
+    
+    lora_config = LoraConfig(
+        r=args.lora_rank,
+        lora_alpha=args.lora_alpha,
+        target_modules=target_modules[model_family],
+        lora_dropout=0.05,
+        bias='none',
+        task_type=TaskType.CAUSAL_LM,
+    )
+    model = get_peft_model(model, lora_config)
+    model.print_trainable_parameters()
+    
+    # Load and tokenize data
+    print(f"Loading data: {args.data}")
+    dataset = load_jsonl(args.data)
+    print(f"Examples: {len(dataset)}")
+    
+    def tokenize_fn(examples):
+        return tokenizer(
+            examples['text'],
+            truncation=True,
+            max_length=args.max_length,
+            padding=False,
+        )
+    
+    dataset = dataset.map(tokenize_fn, batched=True, remove_columns=dataset.column_names)
+    
+    data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
+    
+    # Training args
+    training_args = TrainingArguments(
+        output_dir=args.output,
+        num_train_epochs=args.epochs,
+        per_device_train_batch_size=args.batch_size,
+        gradient_accumulation_steps=args.gradient_accumulation,
+        learning_rate=args.learning_rate,
+        warmup_steps=100,
+        logging_steps=10,
+        save_steps=500,
+        save_total_limit=3,
+        bf16=True,
+        gradient_checkpointing=True,
+        optim='paged_adamw_8bit' if bnb_config else 'adamw_torch',
+        report_to='none',
+        push_to_hub=args.hf_repo is not None,
+        hub_model_id=args.hf_repo,
+    )
+    
+    trainer = Trainer(
+        model=model,
+        args=training_args,
+        train_dataset=dataset,
+        data_collator=data_collator,
+    )
+    
+    print("Starting training...")
+    trainer.train()
+    
+    print("Saving final model...")
+    trainer.save_model(args.output)
+    if args.hf_repo:
+        trainer.push_to_hub()
+    
+    print(f"Done. Saved to {args.output}")
+
+
+if __name__ == '__main__':
+    main()

train_modal.py

@@ -0,0 +1,95 @@
+"""Train on Modal serverless GPU.
+
+Modal lets you rent GPUs by the second. Cheaper than RunPod for short jobs.
+
+Setup:
+    pip install modal
+    modal setup  # login
+    modal secret create huggingface HF_TOKEN=<your_token>
+    
+Run:
+    modal run train_modal.py --model EleutherAI/pythia-1.4b
+"""
+import modal
+
+app = modal.App("mel-corpus-training")
+
+image = (
+    modal.Image.debian_slim(python_version="3.11")
+    .pip_install([
+        "torch>=2.0.0", "transformers>=4.40.0", "peft>=0.10.0",
+        "accelerate>=0.30.0", "datasets>=2.18.0", "bitsandbytes>=0.43.0",
+        "huggingface_hub>=0.22.0",
+    ])
+    .apt_install("git")
+)
+
+volume = modal.Volume.from_name("mel-training", create_if_missing=True)
+
+
+@app.function(
+    image=image,
+    gpu="A100-40GB",  # change to T4, A10, A100-80GB as needed
+    timeout=60 * 60 * 12,  # 12 hour max
+    volumes={"/workspace": volume},
+    secrets=[modal.Secret.from_name("huggingface")],
+)
+def train(
+    model_id: str = "EleutherAI/pythia-1.4b",
+    bridge_repo: str = "Melofhell00/claude-bridge",
+    output_repo: str = None,
+    epochs: int = 3,
+):
+    import os
+    import subprocess
+    from huggingface_hub import hf_hub_download, snapshot_download, HfApi
+    
+    os.chdir("/workspace")
+    
+    # Pull unified corpus from bridge
+    print(f"Downloading corpus from {bridge_repo}...")
+    corpus_path = hf_hub_download(
+        repo_id=bridge_repo,
+        filename="unified_corpus_2026_05_12/unified_corpus.txt",
+        repo_type="dataset",
+    )
+    print(f"Corpus: {corpus_path}")
+    
+    # Pull training scripts from this repo (uploaded separately)
+    snapshot_download(
+        repo_id="Melofhell00/mel-training-package",
+        repo_type="model",
+        local_dir="/workspace/training_package",
+    )
+    
+    # Prepare data
+    print("Preparing data...")
+    subprocess.run([
+        "python", "/workspace/training_package/prepare_data.py",
+        "--corpus", corpus_path,
+        "--output", "/workspace/train.jsonl",
+        "--tokenizer", model_id,
+    ], check=True)
+    
+    # Train
+    print("Training...")
+    output_name = output_repo or f"mel-{model_id.split('/')[-1]}"
+    cmd = [
+        "python", "/workspace/training_package/train.py",
+        "--model", model_id,
+        "--data", "/workspace/train.jsonl",
+        "--output", f"/workspace/{output_name}",
+        "--epochs", str(epochs),
+        "--use-4bit",
+        "--hf-repo", f"Melofhell00/{output_name}",
+    ]
+    subprocess.run(cmd, check=True)
+    
+    print(f"Done. Pushed to Melofhell00/{output_name}")
+    return f"Melofhell00/{output_name}"
+
+
+@app.local_entrypoint()
+def main(model: str = "EleutherAI/pythia-1.4b", epochs: int = 3):
+    result = train.remote(model_id=model, epochs=epochs)
+    print(f"\nResult: {result}")