feat: add code completion generator and model registry tools

- scripts/generate_code_completion_data.py: Multi-language code completion generator
- scripts/model_info.py: Model metadata extraction tool
- scripts/compare_models.py: Compare model versions
- MODEL_REGISTRY.md: Version tracking documentation
- training-data/README.md: Training data format docs

MODEL_REGISTRY.md

@@ -0,0 +1,69 @@
+# Stack 2.9 Model Registry
+
+> Version tracking for all Stack 2.9 model variants.
+
+---
+
+## Model Versions
+
+| Version | Status | Date | Base Model | Parameters | Dataset | Performance | Use Case |
+|---------|--------|------|------------|------------|---------|-------------|----------|
+| `stack-2.9-1.5B` | 🟡 In Training | 2026-04-06 | Llama 3.2-1B | 1.5B | Stack 2.9 dedup | TBD | Research, fine-tuning base |
+| `stack-2.9-7B` | 🔴 Planned | TBD | Llama 3.1-8B | 7B | Stack 2.9 dedup | TBD | General-purpose inference |
+| `stack-2.9-7B-QLoRA` | 🔴 Planned | TBD | Llama 3.1-8B | 7B (quantized) | Stack 2.9 dedup | TBD | Edge deployment, low-memory |
+
+---
+
+## Version Details
+
+### stack-2.9-1.5B (Current)
+
+- **Status:** In Training
+- **Architecture:** Transformer (pretrained)
+- **Base Model:** Llama 3.2-1B
+- **Parameters:** 1.5B
+- **Training Data:** Stack 2.9 deduplicated
+- **Context Length:** 128k tokens
+- **Vocabulary Size:** ~128K
+- **Precision:** BF16
+- **Training Hardware:** 8x H100 (TBD确认)
+- **Expected Completion:** TBD
+- **Notes:** First iteration of Stack 2.9, used as baseline for larger variants
+
+### stack-2.9-7B (Planned)
+
+- **Status:** Planned
+- **Architecture:** Transformer (pretrained)
+- **Base Model:** Llama 3.1-8B
+- **Parameters:** 7B
+- **Training Data:** Stack 2.9 deduplicated
+- **Context Length:** 128k tokens
+- **Vocabulary Size:** ~128K
+- **Precision:** BF16
+- **Training Hardware:** TBD
+- **Expected Start:** TBD
+- **Notes:** Scale-up from 1.5B, targeting general-purpose use
+
+### stack-2.9-7B-QLoRA (Planned)
+
+- **Status:** Planned
+- **Architecture:** Transformer + QLoRA
+- **Base Model:** Llama 3.1-8B
+- **Parameters:** 7B (4-bit quantized)
+- **Training Data:** Stack 2.9 deduplicated
+- **Context Length:** 128k tokens
+- **Vocabulary Size:** ~128K
+- **Quantization:** 4-bit NF4
+- **LoRA Rank:** TBD
+- **LoRA Alpha:** TBD
+- **LoRA Dropout:** TBD
+- **Target Modules:** TBD
+- **Notes:** Quantized for consumer GPU deployment (e.g., 24GB VRAM)
+
+---
+
+## Changelog
+
+| Date | Version | Change |
+|------|---------|--------|
+| 2026-04-06 | stack-2.9-1.5B | Initial entry — training started |

scripts/compare_models.py

@@ -0,0 +1,220 @@
+#!/usr/bin/env python3
+"""
+compare_models.py — Compare different Stack 2.9 model versions.
+
+Reads from models/registry.json and produces a side-by-side comparison
+of model properties and performance metrics.
+
+Usage:
+    python scripts/compare_models.py
+    python scripts/compare_models.py --models stack-2.9-1.5B stack-2.9-7B
+    python scripts/compare_models.py --metrics hellaswag mmlu humaneval
+    python scripts/compare_models.py --verbose
+"""
+
+import argparse
+import json
+import sys
+from pathlib import Path
+from typing import Optional
+
+
+REGISTRY_PATH = Path(__file__).parent.parent / "models" / "registry.json"
+
+ALL_METRICS = ["hellaswag", "arc_challenge", "mmlu", "humaneval", "loss"]
+
+
+def load_registry(registry_path: Path = REGISTRY_PATH) -> dict:
+    """Load the model registry JSON."""
+    if not registry_path.exists():
+        print(f"ERROR: Registry not found at {registry_path}", file=sys.stderr)
+        sys.exit(1)
+    with open(registry_path) as f:
+        return json.load(f)
+
+
+def format_params(n: int) -> str:
+    if n >= 1_000_000_000:
+        return f"{n / 1_000_000_000:.1f}B"
+    elif n >= 1_000_000:
+        return f"{n / 1_000_000:.0f}M"
+    return str(n)
+
+
+def compare_params(a: int, b: int) -> str:
+    """Compare two parameter counts."""
+    ratio = b / a
+    if ratio > 1:
+        return f"  {ratio:.1f}x larger ({format_params(b)} vs {format_params(a)})"
+    else:
+        return f"  {1/ratio:.1f}x smaller ({format_params(b)} vs {format_params(a)})"
+
+
+def build_row(version: str, key: str, value) -> str:
+    """Build a comparison table row."""
+    if value is None:
+        val_str = "—"
+    elif isinstance(value, float):
+        val_str = f"{value:.4f}"
+    elif isinstance(value, int):
+        val_str = f"{value:,}"
+    else:
+        val_str = str(value)
+    return f"  {version:<22} {key:<30} {val_str}"
+
+
+def print_comparison(models: list, metrics: list, verbose: bool = False):
+    """Print a side-by-side comparison table."""
+    # Header
+    versions = [m["version"] for m in models]
+    max_ver_len = max(len(v) for v in versions)
+
+    print(f"\n{'='*72}")
+    print(f"  Model Comparison — Stack 2.9")
+    print(f"{'='*72}")
+
+    # Non-metric fields
+    fields = [
+        ("Base Model", "base_model"),
+        ("Parameters", "parameters"),
+        ("Quantization", "quantization"),
+        ("Precision", "precision"),
+        ("Context Length", "context_length"),
+        ("Vocabulary Size", "vocabulary_size"),
+        ("Dataset", "dataset"),
+        ("LoRA Rank", ("lora", "rank")),
+        ("LoRA Alpha", ("lora", "alpha")),
+        ("LoRA Dropout", ("lora", "dropout")),
+        ("Status", "status"),
+        ("Created", "created_at"),
+        ("Use Case", "use_case"),
+    ]
+
+    print(f"\n  {'Model':<{max_ver_len}}  {'Field':<30}  {'Value'}")
+    print(f"  {'-'*max_ver_len}  {'-'*30}  {'-'*20}")
+
+    for label, key in fields:
+        row_values = []
+        for m in models:
+            if isinstance(key, tuple):
+                nested = m
+                for k in key:
+                    nested = nested.get(k, {}) if isinstance(nested, dict) else {}
+                row_values.append(nested if nested else None)
+            else:
+                val = m.get(key)
+                # Format parameters as human-readable
+                if key == "parameters" and val:
+                    val = f"{format_params(val)} ({val:,})"
+                row_values.append(val)
+        unique = set(str(v) for v in row_values)
+        if len(unique) == 1 and row_values[0] is None:
+            continue
+        print(f"\n  {label}:")
+        for i, (ver, val) in enumerate(zip(versions, row_values)):
+            if val is None:
+                val_str = "—"
+            elif isinstance(val, float):
+                val_str = f"{val:.4f}"
+            elif isinstance(val, int):
+                val_str = f"{val:,}"
+            else:
+                val_str = str(val)
+            marker = " →" if i > 0 and row_values[i] != row_values[0] else "  "
+            print(f"  {marker} {ver:<{max_ver_len}}  {val_str}")
+
+    # Performance metrics comparison
+    has_any_metrics = any(
+        any(m.get("performance", {}).get(metric) is not None for m in models)
+        for metric in metrics
+    )
+    if has_any_metrics:
+        print(f"\n\n  Performance Benchmarks")
+        print(f"  {'-'*max_ver_len}  {'-'*30}  {'-'*10}")
+
+        for metric in metrics:
+            metric_name = metric.replace("_", " ").title()
+            values = [m.get("performance", {}).get(metric) for m in models]
+            if all(v is None for v in values):
+                continue
+            print(f"\n  {metric_name}:")
+            for i, (ver, val) in enumerate(zip(versions, values)):
+                if val is None:
+                    val_str = "N/A"
+                else:
+                    val_str = f"{val:.4f}"
+                marker = " →" if i > 0 else "  "
+                print(f"  {marker} {ver:<{max_ver_len}}  {val_str}")
+
+    # Parameter size comparison (pairwise)
+    if len(models) >= 2:
+        print(f"\n\n  Parameter Size Comparison:")
+        for i in range(len(models)):
+            for j in range(i + 1, len(models)):
+                a, b = models[i], models[j]
+                pa = a.get("parameters", 0)
+                pb = b.get("parameters", 0)
+                if pa and pb:
+                    ratio = pb / pa
+                    direction = "larger" if ratio > 1 else "smaller"
+                    print(f"  {b['version']} is {ratio:.2f}x {direction} than {a['version']}")
+
+    print(f"\n{'='*72}\n")
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Compare Stack 2.9 model versions side by side."
+    )
+    parser.add_argument(
+        "--models", "-m",
+        nargs="+",
+        metavar="VERSION",
+        help="Model versions to compare (e.g., stack-2.9-1.5B stack-2.9-7B). "
+             "If omitted, compares all available models."
+    )
+    parser.add_argument(
+        "--metrics", "-M",
+        nargs="+",
+        choices=ALL_METRICS,
+        default=ALL_METRICS,
+        help=f"Benchmark metrics to include (default: all). Choices: {ALL_METRICS}"
+    )
+    parser.add_argument(
+        "--verbose", "-v",
+        action="store_true",
+        help="Show verbose output."
+    )
+    parser.add_argument(
+        "--registry",
+        default=REGISTRY_PATH,
+        metavar="PATH",
+        help=f"Path to registry.json (default: {REGISTRY_PATH})."
+    )
+    args = parser.parse_args()
+
+    registry_path = Path(args.registry)
+    registry = load_registry(registry_path)
+    models = registry.get("models", [])
+
+    if args.models:
+        selected = []
+        for v in args.models:
+            found = next((m for m in models if m["version"] == v), None)
+            if found:
+                selected.append(found)
+            else:
+                print(f"WARNING: Model '{v}' not found in registry. Skipping.", file=sys.stderr)
+                available = ", ".join(m["version"] for m in models)
+                print(f"  Available: {available}", file=sys.stderr)
+        if not selected:
+            print("ERROR: No valid models selected.", file=sys.stderr)
+            sys.exit(1)
+    else:
+        selected = models
+
+    print_comparison(selected, metrics=args.metrics, verbose=args.verbose or args.verbose)
+
+
+if __name__ == "__main__":
+    main()

scripts/generate_code_completion_data.py

@@ -0,0 +1,262 @@
+#!/usr/bin/env python3
+"""
+Synthetic Code Completion Training Data Generator for Stack 2.9
+Generates training examples for pure code completion without tools.
+"""
+
+import json
+import random
+import argparse
+from pathlib import Path
+from typing import Dict, List
+
+LANGUAGES = ["python", "javascript", "go", "rust", "typescript"]
+DIFFICULTY_EASY = "easy"
+DIFFICULTY_MEDIUM = "medium"
+DIFFICULTY_HARD = "hard"
+
+# Code templates organized by language -> difficulty -> templates
+CODE_TEMPLATES = {
+    "python": {
+        DIFFICULTY_EASY: [
+            {"context": "def greet(name):", "completion": '    return f"Hello, {name}!"', "description": "Simple greeting function"},
+            {"context": "numbers = [1, 2, 3, 4, 5]\n\n", "completion": "for num in numbers:\n    print(num)", "description": "Loop through list"},
+            {"context": "class Person:\n    def __init__(self, name):", "completion": "        self.name = name", "description": "Class init"},
+            {"context": "def add(a, b):\n    ", "completion": "    return a + b", "description": "Add function"},
+            {"context": "if x > 0:\n    print('positive')\nelif x < 0:\n    ", "completion": "    print('negative')", "description": "Conditional"},
+        ],
+        DIFFICULTY_MEDIUM: [
+            {"context": "def fibonacci(n):\n    if n <= 1:\n        return n\n    ", "completion": "    return fibonacci(n-1) + fibonacci(n-2)", "description": "Fibonacci"},
+            {"context": "class Calculator:\n    def __init__(self):\n        self.result = 0\n    \n    def add(self, x):\n        ", "completion": "        self.result += x\n        return self.result", "description": "Calculator"},
+            {"context": "async def fetch_data(url):\n    async with aiohttp.ClientSession() as session:\n        async with session.get(url) as response:\n            ", "completion": "            return await response.json()", "description": "Async HTTP"},
+            {"context": "def validate_email(email):\n    pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$'\n    ", "completion": "    return re.match(pattern, email) is not None", "description": "Email validation"},
+            {"context": "@app.route('/users/<int:user_id>')\ndef get_user(user_id):\n    user = User.query.get_or_404(user_id)\n    ", "completion": "    return jsonify(user.to_dict())", "description": "Flask route"},
+        ],
+        DIFFICULTY_HARD: [
+            {"context": "class LRUCache:\n    def __init__(self, capacity):\n        self.capacity = capacity\n        self.cache = OrderedDict()\n    \n    def get(self, key):\n        if key not in self.cache:\n            return -1\n        ", "completion": "        self.cache.move_to_end(key)\n        return self.cache[key]", "description": "LRU Cache"},
+            {"context": "def merge_sort(arr):\n    if len(arr) <= 1:\n        return arr\n    \n    mid = len(arr) // 2\n    left = merge_sort(arr[:mid])\n    right = merge_sort(arr[mid:])\n    ", "completion": "    return merge(left, right)", "description": "Merge sort"},
+            {"context": "class BinaryTree:\n    def __init__(self, value):\n        self.value = value\n        self.left = None\n        self.right = None\n    \n    def inorder(self, node, result=None):\n        if result is None:\n            result = []\n        if node:\n            ", "completion": "            self.inorder(node.left, result)\n            result.append(node.value)\n            self.inorder(node.right, result)\n        return result", "description": "Binary tree inorder"},
+            {"context": "def bellman_ford(graph, source):\n    dist = {v: float('inf') for v in graph}\n    dist[source] = 0\n    \n    for _ in range(len(graph) - 1):\n        for u, v, w in graph.edges:\n            if dist[u] != float('inf') and dist[u] + w < dist[v]:\n                ", "completion": "                dist[v] = dist[u] + w\n    return dist", "description": "Bellman-Ford"},
+        ],
+    },
+    "javascript": {
+        DIFFICULTY_EASY: [
+            {"context": "const greet = (name) => {", "completion": '    return `Hello, ${name}!`;', "description": "Arrow greeting"},
+            {"context": "const numbers = [1, 2, 3, 4, 5];\n\n", "completion": "numbers.forEach(num => console.log(num));", "description": "forEach loop"},
+            {"context": "class Person {\n  constructor(name) {", "completion": "    this.name = name;", "description": "JS class constructor"},
+            {"context": "const add = (a, b) => {", "completion": "  return a + b;", "description": "Add function"},
+            {"context": "if (x > 0) {\n  console.log('positive');\n} else if (x < 0) {\n  ", "completion": "  console.log('negative');", "description": "Conditional"},
+        ],
+        DIFFICULTY_MEDIUM: [
+            {"context": "const fetchData = async (url) => {\n  try {\n    const response = await fetch(url);\n    ", "completion": "    return await response.json();\n  } catch (error) {\n    console.error('Error:', error);\n  }", "description": "Async fetch"},
+            {"context": "class EventEmitter {\n  constructor() {\n    this.events = {};\n  }\n  \n  on(event, callback) {\n    ", "completion": "    if (!this.events[event]) this.events[event] = [];\n    this.events[event].push(callback);", "description": "Event emitter"},
+            {"context": "const debounce = (func, delay) => {\n  let timeoutId;\n  return (...args) => {\n    clearTimeout(timeoutId);\n    ", "completion": "    timeoutId = setTimeout(() => func.apply(this, args), delay);", "description": "Debounce"},
+            {"context": "const memoize = (fn) => {\n  const cache = new Map();\n  return (n) => {\n    if (cache.has(n)) {\n      return cache.get(n);\n    }\n    ", "completion": "    const result = fn(n);\n    cache.set(n, result);\n    return result;", "description": "Memoize"},
+        ],
+        DIFFICULTY_HARD: [
+            {"context": "class PromisePool {\n  constructor(maxConcurrent) {\n    this.maxConcurrent = maxConcurrent;\n    this.running = 0;\n    this.queue = [];\n  }\n  \n  add(promiseFn) {\n    return new Promise((resolve, reject) => {\n      ", "completion": "      this.queue.push({ promiseFn, resolve, reject });\n      this.process();\n    });", "description": "Promise pool"},
+            {"context": "const virtualDOM = {\n  createElement(tag, props, ...children) {\n    return {\n      tag,\n      props: props || {},\n      children: children.flat(),\n    };\n  },\n  render(vnode, container) {\n    ", "completion": "    const el = document.createElement(vnode.tag);\n    Object.entries(vnode.props || {}).forEach(([key, value]) => el.setAttribute(key, value));\n    vnode.children.forEach(child => {\n      if (typeof child === 'string') el.appendChild(document.createTextNode(child));\n      else this.render(child, el);\n    });\n    container.appendChild(el);", "description": "Virtual DOM"},
+        ],
+    },
+    "go": {
+        DIFFICULTY_EASY: [
+            {"context": "func greet(name string) string {", "completion": '    return "Hello, " + name + "!"', "description": "Greet function"},
+            {"context": "func add(a, b int) int {", "completion": "    return a + b", "description": "Add function"},
+            {"context": "type Person struct {\n    Name string\n    ", "completion": "    Age  int", "description": "Struct definition"},
+            {"context": "for i := 0; i < 10; i++ {\n    ", "completion": "    fmt.Println(i)", "description": "For loop"},
+            {"context": "if x > 0 {\n    fmt.Println(\"positive\")\n} else {\n    ", "completion": '    fmt.Println("non-positive")', "description": "If-else"},
+        ],
+        DIFFICULTY_MEDIUM: [
+            {"context": "func (p Person) Greet() string {", "completion": '    return fmt.Sprintf("Hello, %s!", p.Name)', "description": "Method"},
+            {"context": "func worker(jobs <-chan int, results chan<- int) {\n    for j := range jobs {\n        ", "completion": "        results <- j * 2", "description": "Worker goroutine"},
+            {"context": "type Handler interface {\n    Handle(ctx context.Context, req Request) Response\n    ", "completion": "    Cleanup(ctx context.Context)", "description": "Interface"},
+            {"context": "func fetchData(url string) ([]byte, error) {\n    resp, err := http.Get(url)\n    if err != nil {\n        return nil, err\n    }\n    defer resp.Body.Close()\n    ", "completion": "    return io.ReadAll(resp.Body)", "description": "HTTP GET"},
+        ],
+        DIFFICULTY_HARD: [
+            {"context": "type TreeNode struct {\n    Val   int\n    Left  *TreeNode\n    Right *TreeNode\n}\n\nfunc (root *TreeNode) InorderTraversal() []int {\n    var result []int\n    var inorder func(*TreeNode)\n    inorder = func(node *TreeNode) {\n        if node == nil {\n            return\n        }\n        ", "completion": "        inorder(node.Left)\n        result = append(result, node.Val)\n        inorder(node.Right)", "description": "Tree inorder"},
+            {"context": "func (c *Client) StreamProcess(ctx context.Context, req *Request, stream chan<- *Response) error {\n    for {\n        select {\n        case <-ctx.Done():\n            return ctx.Err()\n        default:\n            result, err := c.processOne(req)\n            if err != nil {\n                return err\n            }\n            ", "completion": "            select {\n            case stream <- result:\n            case <-ctx.Done():\n                return ctx.Err()\n            }", "description": "Streaming"},
+        ],
+    },
+    "rust": {
+        DIFFICULTY_EASY: [
+            {"context": "fn greet(name: &str) -> String {", "completion": '    format!("Hello, {}!", name)', "description": "Greet function"},
+            {"context": "fn add(a: i32, b: i32) -> i32 {", "completion": "    a + b", "description": "Add function"},
+            {"context": "struct Person {\n    name: String,\n    ", "completion": "    age: u32,", "description": "Struct"},
+            {"context": "let numbers = vec![1, 2, 3, 4, 5];\nfor num in &numbers {\n    ", "completion": "    println!(\"{}\", num);", "description": "For loop"},
+            {"context": "fn main() {\n    let result = match value {\n        Some(x) => x,\n        ", "completion": "        None => 0,", "description": "Match"},
+        ],
+        DIFFICULTY_MEDIUM: [
+            {"context": "impl Person {\n    fn new(name: String, age: u32) -> Self {", "completion": "        Person { name, age }", "description": "Constructor"},
+            {"context": "fn fetch_data(url: &str) -> Result<String, Error> {\n    let response = reqwest::blocking::get(url)?;\n    ", "completion": "    let body = response.text()?;\n    Ok(body)", "description": "HTTP request"},
+            {"context": "fn process_items<T: Display>(items: Vec<T>) -> String {\n    items\n        .iter()\n        .enumerate()\n        .map(|(i, item)| format!(\"{}: {}\", i, item))\n        ", "completion": "        .collect::<Vec<_>>()\n        .join(\", \")", "description": "Iterator chain"},
+            {"context": "fn spawn_worker(jobs: Arc<Mutex<Vec<Job>>>) {\n    thread::spawn(move || {\n        loop {\n            let job = {\n                let mut jobs = jobs.lock().unwrap();\n                jobs.pop()\n            };\n            match job {\n                Some(job) => job.execute(),\n                ", "completion": "                None => break,\n            };\n        }\n    });", "description": "Worker thread"},
+        ],
+        DIFFICULTY_HARD: [
+            {"context": "pub struct LRUCache<K, V> {\n    capacity: usize,\n    cache: LinkedHashMap<K, V>,\n}\n\nimpl<K: Eq + Hash + Clone, V: Clone> LRUCache<K, V> {\n    pub fn get(&mut self, key: &K) -> Option<&V> {\n        if self.cache.contains_key(key) {\n            ", "completion": "            self.cache.remove(key);\n            let value = self.cache[key].clone();\n            self.cache.insert(key.clone(), value);\n            self.cache.get(key)\n        } else {\n            None\n        }", "description": "LRU Cache"},
+            {"context": "pub trait Observer<T> {\n    fn update(&self, event: &T);\n}\n\npub struct Subject<T> {\n    observers: Vec<Box<dyn Observer<T>>>,\n}\n\nimpl<T> Subject<T> {\n    pub fn notify(&self, event: &T) {\n        for observer in &self.observers {\n            ", "completion": "            observer.update(event);", "description": "Observer pattern"},
+        ],
+    },
+}
+
+VARIANTS = ["basic", "explain", "debug", "optimize"]
+
+VARIANT_PROMPTS = {
+    "basic": {"system": "You are a helpful AI assistant that helps with code completion.", "user_prefix": "Complete the following code:\n\n"},
+    "explain": {"system": "You are a helpful AI assistant that explains and completes code.", "user_prefix": "Explain what this code does and complete it:\n\n"},
+    "debug": {"system": "You are a helpful AI assistant that finds bugs and suggests fixes.", "user_prefix": "There's a bug in this code. Fix and complete it:\n\n"},
+    "optimize": {"system": "You are a helpful AI assistant that optimizes code for performance.", "user_prefix": "Optimize this code and complete it:\n\n"},
+}
+
+
+def create_completion_example(context, completion, language, difficulty, variant, description):
+    """Create a single code completion example."""
+    variant_info = VARIANT_PROMPTS[variant]
+    messages = [
+        {"role": "system", "content": variant_info["system"]},
+        {"role": "user", "content": f"{variant_info['user_prefix']}```{language}\n{context}```"},
+        {"role": "assistant", "content": f"Here's the completed code:\n\n```{language}\n{context}{completion}\n```"}
+    ]
+    return {
+        "messages": messages,
+        "language": language,
+        "difficulty": difficulty,
+        "variant": variant,
+        "description": description,
+        "context": context,
+        "completion": completion,
+    }
+
+
+def generate_examples_for_language(language, difficulty, num_examples, variants):
+    """Generate examples for a specific language and difficulty."""
+    templates = CODE_TEMPLATES[language][difficulty]
+    examples = []
+    for i in range(num_examples):
+        template = templates[i % len(templates)]
+        variant = random.choice(variants)
+        example = create_completion_example(
+            context=template["context"],
+            completion=template["completion"],
+            language=language,
+            difficulty=difficulty,
+            variant=variant,
+            description=template["description"]
+        )
+        examples.append(example)
+    return examples
+
+
+def generate_dataset(num_examples=1000, languages=None, difficulties=None, variants=None, balance=True):
+    """Generate the complete dataset."""
+    if languages is None:
+        languages = LANGUAGES
+    if difficulties is None:
+        difficulties = [DIFFICULTY_EASY, DIFFICULTY_MEDIUM, DIFFICULTY_HARD]
+    if variants is None:
+        variants = VARIANTS
+    
+    examples = []
+    
+    if balance:
+        examples_per_lang = num_examples // len(languages)
+        examples_per_diff = examples_per_lang // len(difficulties)
+        remainder = num_examples % (len(languages) * len(difficulties))
+        
+        for lang in languages:
+            for diff_idx, diff in enumerate(difficulties):
+                count = examples_per_diff + (1 if diff_idx < remainder else 0)
+                lang_examples = generate_examples_for_language(lang, diff, count, variants)
+                examples.extend(lang_examples)
+    else:
+        for _ in range(num_examples):
+            lang = random.choice(languages)
+            diff = random.choice(difficulties)
+            template = random.choice(CODE_TEMPLATES[lang][diff])
+            variant = random.choice(variants)
+            example = create_completion_example(
+                context=template["context"],
+                completion=template["completion"],
+                language=lang,
+                difficulty=diff,
+                variant=variant,
+                description=template["description"]
+            )
+            examples.append(example)
+    
+    random.shuffle(examples)
+    return examples
+
+
+def save_jsonl(examples, output_path):
+    """Save examples to JSONL format."""
+    output_file = Path(output_path)
+    output_file.parent.mkdir(parents=True, exist_ok=True)
+    with open(output_file, 'w', encoding='utf-8') as f:
+        for example in examples:
+            f.write(json.dumps(example, ensure_ascii=False) + '\n')
+
+
+def save_json(examples, output_path):
+    """Save examples to JSON format."""
+    output_file = Path(output_path)
+    output_file.parent.mkdir(parents=True, exist_ok=True)
+    with open(output_file, 'w', encoding='utf-8') as f:
+        json.dump(examples, f, ensure_ascii=False, indent=2)
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Generate synthetic code completion training data")
+    parser.add_argument("--num-examples", type=int, default=1000, help="Number of examples to generate")
+    parser.add_argument("--output-dir", type=str, default="training-data/code-completion", help="Output directory")
+    parser.add_argument("--output-format", choices=["jsonl", "json", "both"], default="jsonl", help="Output format")
+    parser.add_argument("--seed", type=int, default=42, help="Random seed")
+    args = parser.parse_args()
+    
+    random.seed(args.seed)
+    
+    print(f"Generating {args.num_examples} code completion training examples...")
+    print(f"   Languages: {LANGUAGES}")
+    print(f"   Output directory: {args.output_dir}")
+    
+    examples = generate_dataset(
+        num_examples=args.num_examples,
+        languages=LANGUAGES,
+        difficulties=[DIFFICULTY_EASY, DIFFICULTY_MEDIUM, DIFFICULTY_HARD],
+        variants=VARIANTS
+    )
+    
+    output_dir = Path(args.output_dir)
+    
+    if args.output_format in ["jsonl", "both"]:
+        jsonl_path = output_dir / "code_completion.jsonl"
+        save_jsonl(examples, str(jsonl_path))
+        print(f"Saved JSONL: {jsonl_path}")
+    
+    if args.output_format in ["json", "both"]:
+        json_path = output_dir / "code_completion.json"
+        save_json(examples, str(json_path))
+        print(f"Saved JSON: {json_path}")
+    
+    # Statistics
+    print(f"\nStatistics:")
+    print(f"   Total examples: {len(examples)}")
+    
+    lang_counts = {}
+    diff_counts = {}
+    for ex in examples:
+        lang_counts[ex["language"]] = lang_counts.get(ex["language"], 0) + 1
+        diff_counts[ex["difficulty"]] = diff_counts.get(ex["difficulty"], 0) + 1
+    
+    print(f"   By language:")
+    for lang, count in sorted(lang_counts.items(), key=lambda x: x[1], reverse=True):
+        print(f"     - {lang}: {count}")
+    
+    print(f"   By difficulty:")
+    for diff, count in sorted(diff_counts.items(), key=lambda x: x[1], reverse=True):
+        print(f"     - {diff}: {count}")
+    
+    print(f"\nGeneration complete!")
+
+
+if __name__ == "__main__":
+    main()

scripts/model_info.py

@@ -0,0 +1,167 @@
+#!/usr/bin/env python3
+"""
+model_info.py — Extract and report Stack 2.9 model metadata.
+
+Reads from models/registry.json and optionally from a model checkpoint
+directory to extract/verify metadata.
+
+Usage:
+    python scripts/model_info.py                     # Show all models
+    python scripts/model_info.py --model stack-2.9-1.5B
+    python scripts/model_info.py --model stack-2.9-7B-QLoRA --verbose
+    python scripts/model_info.py --export-json /path/to/output.json
+"""
+
+import argparse
+import json
+import os
+import sys
+from pathlib import Path
+from typing import Optional
+
+
+REGISTRY_PATH = Path(__file__).parent.parent / "models" / "registry.json"
+
+
+def load_registry(registry_path: Path = REGISTRY_PATH) -> dict:
+    """Load the model registry JSON."""
+    if not registry_path.exists():
+        print(f"ERROR: Registry not found at {registry_path}", file=sys.stderr)
+        sys.exit(1)
+    with open(registry_path) as f:
+        return json.load(f)
+
+
+def format_params(n: int) -> str:
+    """Format parameter count as human-readable string."""
+    if n >= 1_000_000_000:
+        return f"{n / 1_000_000_000:.1f}B"
+    elif n >= 1_000_000:
+        return f"{n / 1_000_000:.0f}M"
+    return str(n)
+
+
+def format_lora(config: Optional[dict]) -> str:
+    """Format LoRA config as readable string."""
+    if not config:
+        return "N/A (full model)"
+    lines = [
+        f"  Rank (r):         {config.get('rank', 'N/A')}",
+        f"  Alpha:            {config.get('alpha', 'N/A')}",
+        f"  Dropout:          {config.get('dropout', 'N/A')}",
+        f"  Target Modules:   {', '.join(config.get('target_modules', []))}",
+    ]
+    if config.get("modules_to_save"):
+        lines.append(f"  Modules to Save:  {', '.join(config['modules_to_save'])}")
+    return "\n".join(lines)
+
+
+def format_performance(metrics: dict) -> str:
+    """Format performance metrics."""
+    benchmarks = {
+        "HellaSwag": metrics.get("hellaswag"),
+        "ARC-Challenge": metrics.get("arc_challenge"),
+        "MMLU": metrics.get("mmlu"),
+        "HumanEval": metrics.get("humaneval"),
+        "Training Loss": metrics.get("loss"),
+    }
+    lines = []
+    for name, value in benchmarks.items():
+        if value is not None:
+            lines.append(f"  {name:20s} {value}")
+        else:
+            lines.append(f"  {name:20s} N/A")
+    return "\n".join(lines) if lines else "  No benchmarks yet"
+
+
+def status_emoji(status: str) -> str:
+    """Return emoji for model status."""
+    return {
+        "in_training": "🟡 IN TRAINING",
+        "planned": "🔴 PLANNED",
+        "released": "🟢 RELEASED",
+        "deprecated": "⚠️  DEPRECATED",
+    }.get(status, f"({status})")
+
+
+def print_model(model: dict, verbose: bool = False):
+    """Print a single model's info."""
+    print(f"\n{'='*60}")
+    print(f"  {model['version']}  [{status_emoji(model['status'])}]")
+    print(f"{'='*60}")
+
+    print(f"\n  Base Model:      {model['base_model']}")
+    print(f"  Parameters:      {format_params(model['parameters'])} ({model['parameters']:,})")
+    print(f"  Quantization:    {model.get('quantization') or 'None (full precision)'}")
+    print(f"  Precision:       {model.get('precision', 'N/A')}")
+    print(f"  Context Length:  {model.get('context_length', 'N/A'):,} tokens")
+    print(f"  Vocab Size:      {model.get('vocabulary_size', 'N/A'):,}")
+    print(f"  Dataset:         {model['dataset']}")
+    print(f"  Created:         {model.get('created_at') or 'TBD'}")
+
+    print(f"\n  LoRA Config:")
+    print(f"  {format_lora(model.get('lora'))}")
+
+    print(f"\n  Performance Metrics:")
+    print(f"  {format_performance(model.get('performance', {}))}")
+
+    print(f"\n  Use Case:        {model['use_case']}")
+    if model.get("notes"):
+        print(f"  Notes:           {model['notes']}")
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Extract and report Stack 2.9 model metadata."
+    )
+    parser.add_argument(
+        "--model", "-m",
+        help="Specific model version to show (e.g., stack-2.9-1.5B). "
+             "If omitted, shows all models."
+    )
+    parser.add_argument(
+        "--verbose", "-v",
+        action="store_true",
+        help="Show verbose output (same as default)."
+    )
+    parser.add_argument(
+        "--export-json", "-o",
+        metavar="PATH",
+        help="Export selected model(s) as JSON to a file."
+    )
+    parser.add_argument(
+        "--registry",
+        default=REGISTRY_PATH,
+        metavar="PATH",
+        help=f"Path to registry.json (default: {REGISTRY_PATH})."
+    )
+    args = parser.parse_args()
+
+    registry_path = Path(args.registry)
+    registry = load_registry(registry_path)
+    models = registry.get("models", [])
+
+    if args.model:
+        selected = [m for m in models if m["version"] == args.model]
+        if not selected:
+            print(f"ERROR: Model '{args.model}' not found in registry.", file=sys.stderr)
+            print("Available models:", ", ".join(m["version"] for m in models))
+            sys.exit(1)
+    else:
+        selected = models
+
+    for model in selected:
+        print_model(model, verbose=args.verbose)
+
+    # Export to JSON if requested
+    if args.export_json:
+        output = {"registry_version": registry.get("registry_version"), "models": selected}
+        with open(args.export_json, "w") as f:
+            json.dump(output, f, indent=2)
+        print(f"\n✓ Exported to {args.export_json}")
+
+    print()
+
+
+if __name__ == "__main__":
+    main()

training-data/README.md

@@ -0,0 +1,182 @@
+# Stack 2.9 Training Data
+
+This directory contains synthetic training data for fine-tuning code generation models.
+
+## Directory Structure
+
+```
+training-data/
+├── README.md                           # This file
+├── tool_examples.jsonl                 # Tool-calling examples (Qwen2.5-Coder format)
+├── tool_examples.json                  # Same as above in JSON format
+├── code_completion/                    # Pure code completion examples
+│   ├── code_completion.jsonl
+│   └── code_completion.json
+└── training-data-expanded/            # Additional generated data
+    └── tool_examples.jsonl             # 5000 expanded tool-calling examples
+```
+
+## Data Formats
+
+### Tool-Calling Examples
+
+**Format:** Qwen2.5-Coder style with `tool_calls`
+
+Each example contains:
+- `messages`: Array of conversation messages (system, user, assistant, tool)
+- `tools`: Array of tool definitions
+
+**Example structure:**
+```json
+{
+  "messages": [
+    {"role": "system", "content": "You are a helpful AI assistant..."},
+    {"role": "user", "content": "Read the file at src/main.py..."},
+    {
+      "role": "assistant",
+      "content": null,
+      "tool_calls": [
+        {
+          "id": "call_1234",
+          "type": "function",
+          "function": {
+            "name": "FileRead",
+            "arguments": "{\"path\": \"src/main.py\"}"
+          }
+        }
+      ]
+    },
+    {
+      "role": "tool",
+      "content": "Successfully read file: src/main.py\n...",
+      "tool_call_id": "call_1234",
+      "name": "FileRead"
+    },
+    {"role": "assistant", "content": "Here's the contents..."}
+  ],
+  "tools": [...]
+}
+```
+
+**Available Tools:**
+- `Bash` - Execute bash commands
+- `FileRead` - Read file contents
+- `FileWrite` - Write/create files
+- `WebSearch` - Search the web
+- `Grep` - Search patterns in files
+
+### Code Completion Examples
+
+**Format:** Chat-based with context and completion
+
+Each example contains:
+- `messages`: Array of conversation messages
+- `language`: Programming language (python, javascript, go, rust, typescript)
+- `difficulty`: easy, medium, hard
+- `variant`: basic, explain, debug, optimize
+- `context`: The code context to complete
+- `completion`: The expected completion
+
+**Example structure:**
+```json
+{
+  "messages": [
+    {"role": "system", "content": "You are a helpful AI assistant..."},
+    {"role": "user", "content": "Complete the following code:\n```python\ndef greet(name):\n```"},
+    {"role": "assistant", "content": "Here's the completed code:\n```python\ndef greet(name):\n    return f\"Hello, {name}!\"\n```"}
+  ],
+  "language": "python",
+  "difficulty": "easy",
+  "variant": "basic",
+  "description": "Simple function that returns a greeting",
+  "context": "def greet(name):",
+  "completion": "    return f\"Hello, {name}!\""
+}
+```
+
+## Generation Scripts
+
+### Tool Data Generator
+
+```bash
+python3 scripts/generate_tool_data.py \
+    --num-examples 5000 \
+    --output-dir training-data-expanded \
+    --output-format jsonl
+```
+
+### Code Completion Generator
+
+```bash
+python3 scripts/generate_code_completion_data.py \
+    --num-examples 1000 \
+    --output-dir training-data/code-completion \
+    --languages python javascript go rust typescript \
+    --difficulties easy medium hard \
+    --variants basic explain debug optimize
+```
+
+## Difficulty Levels
+
+| Level | Description |
+|-------|-------------|
+| **easy** | Simple functions, basic operations, single concepts |
+| **medium** | Intermediate patterns, async operations, error handling |
+| **hard** | Complex algorithms, data structures, design patterns |
+
+## Variants
+
+| Variant | Description |
+|---------|-------------|
+| **basic** | Standard code completion |
+| **explain** | Code completion with explanation |
+| **debug** | Bug fixing and completion |
+| **optimize** | Performance optimization and completion |
+
+## Supported Languages
+
+- Python
+- JavaScript
+- Go
+- Rust
+- TypeScript
+
+## Usage
+
+### Training with MLflow
+
+```bash
+mlflow run . -P num_examples=5000
+```
+
+### Loading Data for Training
+
+```python
+import json
+
+# Load JSONL
+with open("training-data/tool_examples.jsonl", "r") as f:
+    for line in f:
+        example = json.loads(line)
+        # Process example
+        pass
+
+# Load JSON
+with open("training-data/tool_examples.json", "r") as f:
+    data = json.load(f)
+```
+
+## Augmentation
+
+The tool-calling generator applies augmentation to create diversity:
+- Varying file paths
+- Varying command options
+- Varying search queries
+- Varying code snippets
+
+## Quality Guidelines
+
+- All generated code is syntactically correct
+- Examples include realistic context
+- Tools have proper arguments and responses
+- Code completions are deterministic and correct