GDScript RAG assistant: app + corpus (index added later via Colab)

.gitattributes

@@ -1,35 +1,2 @@
-*.7z filter=lfs diff=lfs merge=lfs -text
-*.arrow filter=lfs diff=lfs merge=lfs -text
-*.bin filter=lfs diff=lfs merge=lfs -text
-*.bz2 filter=lfs diff=lfs merge=lfs -text
-*.ckpt filter=lfs diff=lfs merge=lfs -text
-*.ftz filter=lfs diff=lfs merge=lfs -text
-*.gz filter=lfs diff=lfs merge=lfs -text
-*.h5 filter=lfs diff=lfs merge=lfs -text
-*.joblib filter=lfs diff=lfs merge=lfs -text
-*.lfs.* filter=lfs diff=lfs merge=lfs -text
-*.mlmodel filter=lfs diff=lfs merge=lfs -text
-*.model filter=lfs diff=lfs merge=lfs -text
-*.msgpack filter=lfs diff=lfs merge=lfs -text
-*.npy filter=lfs diff=lfs merge=lfs -text
-*.npz filter=lfs diff=lfs merge=lfs -text
-*.onnx filter=lfs diff=lfs merge=lfs -text
-*.ot filter=lfs diff=lfs merge=lfs -text
-*.parquet filter=lfs diff=lfs merge=lfs -text
-*.pb filter=lfs diff=lfs merge=lfs -text
-*.pickle filter=lfs diff=lfs merge=lfs -text
-*.pkl filter=lfs diff=lfs merge=lfs -text
-*.pt filter=lfs diff=lfs merge=lfs -text
-*.pth filter=lfs diff=lfs merge=lfs -text
-*.rar filter=lfs diff=lfs merge=lfs -text
-*.safetensors filter=lfs diff=lfs merge=lfs -text
-saved_model/**/* filter=lfs diff=lfs merge=lfs -text
-*.tar.* filter=lfs diff=lfs merge=lfs -text
-*.tar filter=lfs diff=lfs merge=lfs -text
-*.tflite filter=lfs diff=lfs merge=lfs -text
-*.tgz filter=lfs diff=lfs merge=lfs -text
-*.wasm filter=lfs diff=lfs merge=lfs -text
-*.xz filter=lfs diff=lfs merge=lfs -text
-*.zip filter=lfs diff=lfs merge=lfs -text
-*.zst filter=lfs diff=lfs merge=lfs -text
-*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.faiss filter=lfs diff=lfs merge=lfs -text
+*.jsonl filter=lfs diff=lfs merge=lfs -text

DEPLOY.md

@@ -0,0 +1,57 @@
+# Deploying the GDScript Assistant (Colab-built jina index)
+
+The 280 MB jina index is built on a **free Colab GPU** and pushed straight to the
+Space, so it never moves over your local connection. You only push the app +
+`chunks.jsonl` (~90 MB) once.
+
+## 0. Prerequisites
+- HuggingFace account + **write token** (https://huggingface.co/settings/tokens).
+- `git`, `git-lfs`, `pip install huggingface_hub`.
+- `data/chunks.jsonl` is already staged in this folder.
+
+## Phase 1 — Push the app + corpus (your machine)
+The app tolerates a missing index (it answers without retrieval until the index
+is added), so deploy first:
+```bash
+huggingface-cli login            # write token
+huggingface-cli repo create gdscript-assistant --type space --space_sdk gradio
+cd hf-space/gdscript-assistant
+git init && git lfs install
+git add . && git commit -m "GDScript RAG assistant (app + corpus)"
+git remote add origin https://huggingface.co/spaces/<user>/gdscript-assistant
+git push -u origin main           # ~90MB: chunks.jsonl (LFS) + code
+```
+Then in **Space → Settings → Hardware → select "ZeroGPU"**.
+
+## Phase 2 — Build the jina index on Colab (free GPU, ~10 min)
+1. Open https://colab.research.google.com → new notebook →
+   **Runtime → Change runtime type → T4 GPU**.
+2. Cell 1 (install):
+   ```python
+   !pip install -q "transformers<5" sentence-transformers einops faiss-cpu huggingface_hub
+   ```
+3. Cell 2: paste the contents of **`colab_build_index.py`**, set at the top:
+   ```python
+   SPACE_REPO = "<user>/gdscript-assistant"
+   HF_TOKEN   = "hf_...your_write_token..."
+   ```
+   Run it. It pulls `chunks.jsonl` from the Space, embeds 91,720 chunks with
+   `jina-embeddings-v2-base-code` on the GPU, builds the FAISS index, and
+   **uploads `data/embeddings.faiss` + `data/id_map.json` back to the Space**.
+4. The Space auto-restarts and now answers with full RAG + sources.
+
+## Phase 3 — Verify on the Space
+- Ask *"Write a CharacterBody2D top-down movement script"* → GDScript answer, a
+  **✅ gdtoolkit validation** badge, and a **📚 Retrieved sources** list.
+- Force a mistake to see the **🔧 auto-correct** path.
+- Hitting ZeroGPU quota? HF **PRO** ($9/mo) gives much more GPU time.
+
+## Notes
+- Index format is built to match `rag.py` exactly (cosine `IndexIDMap2`,
+  `faiss_id == chunk id`; `id_map.json` keyed by `str(id)`).
+- `requirements.txt` pins `transformers~=4.45` so jina (query embedding) and
+  Qwen2.5-Coder both load with no patches.
+- Validation checks **syntax + style** (gdtoolkit), not runtime/scene semantics.
+- Fallback (local build): if you ever build the index locally
+  (`python crawl_gdscript.py embed`), run `bash stage_index.sh` then push — but
+  jina on this CPU is ~50h, so Colab is strongly preferred.

README.md

@@ -1,13 +1,56 @@
 ---
-title: Gdscript Assistant
-emoji: 🏃
+title: GDScript Coding Assistant
+emoji: 🤖
 colorFrom: purple
 colorTo: green
 sdk: gradio
-sdk_version: 6.15.2
-python_version: '3.13'
 app_file: app.py
 pinned: false
+license: mit
+short_description: RAG GDScript assistant with gdtoolkit validation
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# 🤖 GDScript Coding Assistant
+
+A Godot 4 / GDScript coding assistant that answers using **RAG** over a curated
+**91,720-chunk** corpus crawled from the official docs, demo repos, tutorial
+sites and YouTube descriptions. Generated GDScript is **syntax-validated with
+`gdtoolkit`** before it's shown.
+
+## How it works
+
+```
+question ─▶ jina query-embed (CPU) ─▶ FAISS top-k GDScript snippets
+         ─▶ Qwen2.5-Coder-7B-Instruct (ZeroGPU) ─▶ answer
+         ─▶ gdtoolkit parse + lint (CPU) ─▶ ✅/❌ + optional 1× self-fix
+```
+
+- **Retriever:** `jinaai/jina-embeddings-v2-base-code` (768-dim, code-tuned),
+  prebuilt FAISS cosine index bundled via Git LFS (`data/embeddings.faiss`,
+  `data/chunks.jsonl`).
+- **Generator:** `Qwen/Qwen2.5-Coder-7B-Instruct` on **ZeroGPU** (only the
+  generation call uses the GPU).
+- **Validation:** `gdtoolkit` (`gdparse` syntax + `gdlint` style). Note: this
+  checks *syntax and style*, not runtime/scene semantics.
+
+## Setup (hardware)
+
+In **Space → Settings → Hardware**, select **ZeroGPU**. The `spaces` package +
+`@spaces.GPU` decorator in `generate.py` do the rest.
+
+## Local dev
+
+```bash
+pip install -r requirements.txt
+# fast UI/flow test without downloading the 7B model:
+GDRAG_STUB_LLM=1 python app.py
+# real retrieval needs data/embeddings.faiss + data/chunks.jsonl present
+python rag.py "how do I use @export and signals"
+python validate.py
+```
+
+## Data provenance & licensing
+
+Snippets come from public Godot resources with **varying licenses** (docs CC-BY,
+repos MIT/Apache/GPL/…). Each retrieved snippet shows its source; respect the
+original licenses when reusing generated code.

app.py

@@ -0,0 +1,90 @@
+"""GDScript Coding Assistant — Gradio app (HF Space, ZeroGPU).
+
+Flow per question:  retrieve (CPU) -> generate (ZeroGPU) -> validate (CPU) ->
+optional 1x self-correct -> render answer + validation + sources.
+"""
+from __future__ import annotations
+
+import gradio as gr
+
+import rag
+import prompt as promptlib
+import generate as gen
+import validate as gdv
+
+
+def _sources_md(hits: list[rag.Hit]) -> str:
+    if not hits:
+        return ""
+    lines = ["\n\n<details><summary>📚 Retrieved sources</summary>\n"]
+    for i, h in enumerate(hits, 1):
+        loc = h.repo or "corpus"
+        url = h.origin_url or ""
+        link = f"[{loc}]({url})" if url.startswith("http") else loc
+        lines.append(f"{i}. {link} · `{h.file_path or h.kind}` · score {h.score:.2f}")
+    lines.append("\n</details>")
+    return "\n".join(lines)
+
+
+def respond(message: str, history, top_k: int, self_correct: bool):
+    message = (message or "").strip()
+    if not message:
+        return "Ask a GDScript or Godot question."
+
+    hits = rag.retrieve(message, k=int(top_k))
+    messages = promptlib.build_messages(message, hits)
+    answer = gen.generate(messages)
+
+    results = gdv.validate_answer(answer)
+
+    # One optional self-correction pass if a code block failed to parse.
+    if self_correct:
+        fail = gdv.first_syntax_error(results)
+        if fail is not None:
+            broken, err = fail
+            fixed = gen.generate(promptlib.build_fix_messages(broken, err))
+            fixed_results = gdv.validate_answer(fixed)
+            if fixed_results and all(r.ok for r in fixed_results):
+                answer = (answer
+                          + "\n\n---\n**🔧 Auto-corrected** (original had a syntax "
+                            "error):\n\n" + fixed)
+                results = fixed_results
+
+    report = gdv.render_report(results)
+    note = ("" if rag.index_available()
+            else "\n\n> ⏳ _Retrieval index not loaded yet — answering without "
+                 "corpus context. Build & push the index (see DEPLOY.md)._")
+    return f"{answer}\n\n---\n**Validation:** \n{report}{_sources_md(hits)}{note}"
+
+
+with gr.Blocks(title="GDScript Coding Assistant", fill_height=True) as demo:
+    gr.Markdown(
+        "# 🤖 GDScript Coding Assistant\n"
+        "RAG over a 91,720-chunk Godot/GDScript corpus · Qwen2.5-Coder-7B · "
+        "answers are **syntax-validated with gdtoolkit**."
+    )
+    with gr.Accordion("Settings", open=False):
+        top_k = gr.Slider(2, 10, value=6, step=1, label="Retrieved snippets (k)")
+        self_correct = gr.Checkbox(
+            value=True, label="Auto-correct one syntax error (extra GPU call)")
+
+    gr.ChatInterface(
+        fn=respond,
+        additional_inputs=[top_k, self_correct],
+        examples=[
+            ["Write a CharacterBody2D top-down movement script", 6, True],
+            ["How do I define and emit a custom signal?", 6, True],
+            ["Show a typed @export inventory array with @onready", 6, True],
+            ["Make an enemy follow the player using a NavigationAgent2D", 6, True],
+        ],
+        cache_examples=False,
+    )
+
+
+if __name__ == "__main__":
+    # Preload index/chunks/embedder (and the model unless stubbed) at startup.
+    try:
+        rag.warmup()
+    except Exception as e:
+        print(f"warmup (rag) skipped: {e}")
+    demo.queue(max_size=16).launch()

colab_build_index.py

@@ -0,0 +1,70 @@
+"""Build the jina FAISS index on a free Colab/Kaggle GPU and push it to the Space.
+
+Run this in a GPU Colab notebook (Runtime -> Change runtime type -> T4 GPU).
+It pulls chunks.jsonl from your Space repo, embeds all chunks with
+jina-embeddings-v2-base-code on the GPU (~minutes), builds the FAISS index in the
+exact format rag.py expects (cosine / IndexIDMap2, faiss_id == chunk id), and
+uploads embeddings.faiss + id_map.json back to the Space — so the ~280 MB index
+never touches your local machine.
+
+USAGE (paste into a Colab cell, or upload this file and `%run` it):
+    1) Set SPACE_REPO and HF_TOKEN below (token: https://huggingface.co/settings/tokens, write).
+    2) Run. When it finishes, the Space restarts with full RAG.
+
+Cell 0 (install):
+    !pip install -q "transformers<5" sentence-transformers einops faiss-cpu huggingface_hub
+"""
+import json
+import os
+
+import faiss
+import numpy as np
+from huggingface_hub import hf_hub_download, login, upload_file
+from sentence_transformers import SentenceTransformer
+
+# ─── CONFIG ────────────────────────────────────────────────────────────────
+SPACE_REPO = os.environ.get("SPACE_REPO", "<user>/gdscript-assistant")  # <-- set
+HF_TOKEN = os.environ.get("HF_TOKEN", "")                                # <-- set (write)
+MODEL = "jinaai/jina-embeddings-v2-base-code"
+BATCH = 256
+# ───────────────────────────────────────────────────────────────────────────
+
+login(token=HF_TOKEN)
+
+# 1. Pull chunks.jsonl from the Space repo (fast on Colab's connection).
+chunks_path = hf_hub_download(
+    repo_id=SPACE_REPO, repo_type="space", filename="data/chunks.jsonl")
+
+ids, texts, meta = [], [], {}
+with open(chunks_path, encoding="utf-8") as f:
+    for line in f:
+        if not line.strip():
+            continue
+        r = json.loads(line)
+        ids.append(int(r["id"]))
+        texts.append(r["text"])
+        meta[str(r["id"])] = {"origin_url": r.get("origin_url", ""),
+                              "repo": r.get("repo", "")}
+print(f"Loaded {len(ids)} chunks")
+
+# 2. Embed on GPU (normalized -> cosine via inner product).
+model = SentenceTransformer(MODEL, trust_remote_code=True, device="cuda")
+vecs = model.encode(texts, batch_size=BATCH, normalize_embeddings=True,
+                    convert_to_numpy=True, show_progress_bar=True)
+vecs = vecs.astype(np.float32)
+print("Embedded:", vecs.shape)
+
+# 3. Build FAISS index — IDMap2(FlatIP), faiss_id == chunk id (matches rag.py).
+index = faiss.IndexIDMap2(faiss.IndexFlatIP(vecs.shape[1]))
+index.add_with_ids(vecs, np.asarray(ids, dtype=np.int64))
+faiss.write_index(index, "embeddings.faiss")
+with open("id_map.json", "w", encoding="utf-8") as f:
+    json.dump(meta, f)
+print("Index built:", index.ntotal, "vectors")
+
+# 4. Push the index back to the Space repo (Colab -> HF; not your machine).
+for fn in ("embeddings.faiss", "id_map.json"):
+    upload_file(path_or_fileobj=fn, path_in_repo=f"data/{fn}",
+                repo_id=SPACE_REPO, repo_type="space",
+                commit_message="Add jina FAISS index (built on GPU)")
+print("Done — Space will restart with full RAG.")

data/chunks.jsonl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:106af2e9e00069642dc25312710d8a48eb7501947e8c7c5437a1e001d4858914
+size 88978917

generate.py

@@ -0,0 +1,78 @@
+"""Generation with Qwen2.5-Coder-7B-Instruct on ZeroGPU.
+
+Only this module touches the GPU: the decorated ``generate`` runs under
+``@spaces.GPU`` so ZeroGPU allocates an A100 slice on demand; retrieval and
+validation stay on CPU.
+
+Local testing: set GDRAG_STUB_LLM=1 to return a canned answer without loading the
+model (so rag/validate/app can be exercised without a GPU or a 15 GB download).
+"""
+from __future__ import annotations
+
+import os
+from functools import lru_cache
+
+MODEL_ID = os.environ.get("GDRAG_LLM", "Qwen/Qwen2.5-Coder-7B-Instruct")
+STUB = os.environ.get("GDRAG_STUB_LLM") == "1"
+
+# Optional ZeroGPU decorator — degrade to a no-op when running locally.
+try:
+    import spaces
+    GPU = spaces.GPU
+except Exception:                                  # not on a Space
+    def GPU(*dargs, **dkwargs):
+        def deco(fn):
+            return fn
+        # support both @GPU and @GPU(duration=...)
+        if dargs and callable(dargs[0]):
+            return dargs[0]
+        return deco
+
+
+@lru_cache(maxsize=1)
+def _model_and_tokenizer():
+    import torch
+    from transformers import AutoModelForCausalLM, AutoTokenizer
+    tok = AutoTokenizer.from_pretrained(MODEL_ID)
+    model = AutoModelForCausalLM.from_pretrained(
+        MODEL_ID, torch_dtype=torch.bfloat16, device_map="auto",
+    )
+    model.eval()
+    return model, tok
+
+
+def _render(messages, tok) -> str:
+    return tok.apply_chat_template(
+        messages, tokenize=False, add_generation_prompt=True)
+
+
+@GPU(duration=120)
+def generate(messages: list[dict], max_new_tokens: int = 512,
+             temperature: float = 0.2) -> str:
+    """Generate an assistant reply for chat-format ``messages``."""
+    if STUB:
+        return (
+            "Here is a Godot 4 movement script:\n\n```gdscript\n"
+            "extends CharacterBody2D\n\n@export var speed: float = 200.0\n\n"
+            "func _physics_process(delta: float) -> void:\n"
+            "\tvar dir := Input.get_vector(\"ui_left\", \"ui_right\", "
+            "\"ui_up\", \"ui_down\")\n\tvelocity = dir * speed\n"
+            "\tmove_and_slide()\n```\n"
+        )
+    import torch
+    model, tok = _model_and_tokenizer()
+    text = _render(messages, tok)
+    inputs = tok([text], return_tensors="pt").to(model.device)
+    with torch.no_grad():
+        out = model.generate(
+            **inputs, max_new_tokens=max_new_tokens,
+            do_sample=temperature > 0, temperature=max(temperature, 1e-4),
+            top_p=0.95, pad_token_id=tok.eos_token_id,
+        )
+    gen = out[0][inputs["input_ids"].shape[1]:]
+    return tok.decode(gen, skip_special_tokens=True).strip()
+
+
+def warmup() -> None:
+    if not STUB:
+        _model_and_tokenizer()

prompt.py

@@ -0,0 +1,56 @@
+"""Prompt assembly: system instruction + retrieved context -> chat messages."""
+from __future__ import annotations
+
+from rag import Hit
+
+SYSTEM_PROMPT = (
+    "You are an expert Godot 4 GDScript assistant. Answer using the reference "
+    "snippets provided below when they are relevant. Always write GDScript that "
+    "targets Godot 4 (GDScript 2.0). Put runnable code in ```gdscript fenced "
+    "blocks. Prefer static typing and @export/@onready annotations where natural. "
+    "If the snippets don't cover the question, answer from general Godot knowledge "
+    "and say so briefly. Be concise."
+)
+
+# Keep the context budget modest so generation stays fast on ZeroGPU.
+MAX_CONTEXT_CHARS = 6000
+
+
+def _format_context(hits: list[Hit]) -> str:
+    blocks, used = [], 0
+    for i, h in enumerate(hits, 1):
+        src = h.repo or h.origin_url or "corpus"
+        snippet = h.text.strip()
+        block = f"# Snippet {i} (source: {src})\n{snippet}"
+        if used + len(block) > MAX_CONTEXT_CHARS:
+            break
+        blocks.append(block)
+        used += len(block)
+    return "\n\n".join(blocks)
+
+
+def build_messages(question: str, hits: list[Hit],
+                   history: list[dict] | None = None) -> list[dict]:
+    """Build chat-template messages for the generator."""
+    context = _format_context(hits)
+    messages: list[dict] = [{"role": "system", "content": SYSTEM_PROMPT}]
+    if history:
+        messages.extend(history)
+    user = question if not context else (
+        f"Reference GDScript snippets from a curated Godot corpus:\n\n"
+        f"{context}\n\n---\n\nQuestion: {question}"
+    )
+    messages.append({"role": "user", "content": user})
+    return messages
+
+
+def build_fix_messages(broken_code: str, error: str) -> list[dict]:
+    """Messages asking the model to fix a GDScript snippet that failed to parse."""
+    return [
+        {"role": "system", "content": SYSTEM_PROMPT},
+        {"role": "user", "content": (
+            "The following GDScript failed to parse with this error:\n"
+            f"{error}\n\nFix it and return ONLY the corrected GDScript in a "
+            f"```gdscript block:\n\n```gdscript\n{broken_code}\n```"
+        )},
+    ]

rag.py

@@ -0,0 +1,119 @@
+"""Retrieval over the GDScript corpus.
+
+Loads the prebuilt FAISS index (cosine / IndexIDMap2, faiss_id == chunk id) and
+chunks.jsonl, embeds the query with the same jina code model used to build the
+index, and returns the top-k chunk records. Runs on CPU (query embedding is one
+text at a time, fast).
+"""
+from __future__ import annotations
+
+import json
+import os
+from dataclasses import dataclass
+from functools import lru_cache
+from pathlib import Path
+
+import faiss
+import numpy as np
+
+DATA_DIR = Path(os.environ.get("GDRAG_SPACE_DATA", Path(__file__).parent / "data"))
+FAISS_PATH = DATA_DIR / "embeddings.faiss"
+CHUNKS_PATH = DATA_DIR / "chunks.jsonl"
+EMBED_MODEL = "jinaai/jina-embeddings-v2-base-code"
+
+
+@dataclass
+class Hit:
+    score: float
+    text: str
+    repo: str
+    origin_url: str
+    file_path: str
+    kind: str
+
+
+# ---------------------------------------------------------------------------
+# Lazy singletons (loaded once per process)
+# ---------------------------------------------------------------------------
+@lru_cache(maxsize=1)
+def _index() -> faiss.Index:
+    return faiss.read_index(str(FAISS_PATH))
+
+
+@lru_cache(maxsize=1)
+def _chunks() -> dict[int, dict]:
+    by_id: dict[int, dict] = {}
+    with open(CHUNKS_PATH, "r", encoding="utf-8") as f:
+        for line in f:
+            if not line.strip():
+                continue
+            try:
+                r = json.loads(line)
+            except json.JSONDecodeError:
+                continue
+            by_id[r["id"]] = r
+    return by_id
+
+
+@lru_cache(maxsize=1)
+def _embedder():
+    # transformers ~=4.45 (pinned) loads jina's remote code without shims.
+    from sentence_transformers import SentenceTransformer
+    return SentenceTransformer(EMBED_MODEL, trust_remote_code=True)
+
+
+def _embed_query(query: str) -> np.ndarray:
+    vec = _embedder().encode([query], normalize_embeddings=True,
+                             show_progress_bar=False)
+    return np.asarray(vec, dtype=np.float32)
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+def index_available() -> bool:
+    return FAISS_PATH.exists() and CHUNKS_PATH.exists()
+
+
+def retrieve(query: str, k: int = 6) -> list[Hit]:
+    """Return the top-k GDScript chunks most relevant to the query.
+
+    Returns [] if the index hasn't been built/uploaded yet, so the Space still
+    runs (answers without retrieval) until the Colab build pushes the index.
+    """
+    if not query.strip() or not index_available():
+        return []
+    qv = _embed_query(query)
+    scores, ids = _index().search(qv, k)
+    chunks = _chunks()
+    hits: list[Hit] = []
+    for score, cid in zip(scores[0], ids[0]):
+        if cid < 0:
+            continue
+        rec = chunks.get(int(cid))
+        if not rec:
+            continue
+        hits.append(Hit(
+            score=float(score),
+            text=rec.get("text", ""),
+            repo=rec.get("repo", ""),
+            origin_url=rec.get("origin_url", ""),
+            file_path=rec.get("file_path", ""),
+            kind=rec.get("kind", ""),
+        ))
+    return hits
+
+
+def warmup() -> None:
+    """Preload index, chunks and embedder (call at Space startup)."""
+    if index_available():
+        _index(); _chunks(); _embedder()
+
+
+if __name__ == "__main__":
+    import sys
+    q = " ".join(sys.argv[1:]) or "how do I use @export and signals in GDScript"
+    print(f"Query: {q}\n")
+    for i, h in enumerate(retrieve(q, k=6), 1):
+        print(f"[{i}] score={h.score:.3f}  {h.repo}  {h.file_path}")
+        print("    " + h.text[:160].replace("\n", " ") + "...\n")

requirements.txt

@@ -0,0 +1,10 @@
+gradio>=4.44
+spaces>=0.30
+torch
+transformers~=4.45        # satisfies BOTH jina remote code (4.x) AND Qwen2.5-Coder
+sentence-transformers~=2.7
+einops                    # required by jina remote code
+accelerate                # device_map model loading
+faiss-cpu>=1.8
+numpy
+gdtoolkit>=4.0            # GDScript syntax parse + lint

stage_index.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/env bash
+# Copy the finished jina FAISS index into the Space's data/ dir.
+# Run this from the project root (v:\Coding_RAG) AFTER the embed build completes
+# (status shows embedded == 91720).
+set -e
+SRC=data/index
+DST=hf-space/gdscript-assistant/data
+
+cp "$SRC/embeddings.faiss" "$DST/embeddings.faiss"
+cp "$SRC/id_map.json"      "$DST/id_map.json"
+# chunks.jsonl is already staged; refresh in case it changed:
+cp data/chunks.jsonl       "$DST/chunks.jsonl"
+
+echo "Staged into $DST:"
+ls -lh "$DST"

validate.py

@@ -0,0 +1,134 @@
+"""Validate GDScript produced by the model using gdtoolkit (Scony's parser).
+
+Pure-Python, CPU-only, Godot-4 (GDScript 2.0). Checks SYNTAX (gdparse) and STYLE
+(gdlint); it does NOT check runtime/scene semantics (node paths, types against a
+real project) — that needs the Godot engine.
+"""
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass, field
+
+_FENCE_RE = re.compile(r"```(?:gdscript|gd|godot)?\s*\n(.*?)```", re.S | re.I)
+
+
+@dataclass
+class BlockResult:
+    code: str
+    ok: bool                      # parses (valid syntax)
+    error: str = ""               # syntax error message (if any)
+    lint: list[str] = field(default_factory=list)   # style/lint warnings
+    formatted: str = ""           # gdformat output (if available)
+
+
+def extract_gdscript_blocks(text: str) -> list[str]:
+    """Pull fenced GDScript blocks from a model answer."""
+    blocks = [m.group(1).strip() for m in _FENCE_RE.finditer(text or "")]
+    return [b for b in blocks if b]
+
+
+# ---------------------------------------------------------------------------
+# gdtoolkit wrappers (imported lazily so the module loads even if absent)
+# ---------------------------------------------------------------------------
+def _parse(code: str) -> tuple[bool, str]:
+    try:
+        from gdtoolkit.parser import parser
+    except Exception as e:                       # gdtoolkit not installed
+        return True, f"(parser unavailable: {e})"
+    try:
+        parser.parse(code, gather_metadata=False)
+        return True, ""
+    except TypeError:
+        # older/newer signature without gather_metadata
+        try:
+            parser.parse(code)
+            return True, ""
+        except Exception as e:
+            return False, _fmt_err(e)
+    except Exception as e:
+        return False, _fmt_err(e)
+
+
+def _fmt_err(e: Exception) -> str:
+    line = getattr(e, "line", None)
+    col = getattr(e, "column", None)
+    msg = str(e).strip().splitlines()[0] if str(e).strip() else type(e).__name__
+    if line is not None:
+        return f"line {line}:{col or 0}: {msg}"
+    return msg
+
+
+def _lint(code: str) -> list[str]:
+    try:
+        from gdtoolkit.linter import lint_code
+    except Exception:
+        return []
+    try:
+        problems = lint_code(code)
+    except Exception:
+        return []
+    out = []
+    for p in problems:
+        line = getattr(p, "line", "?")
+        name = getattr(p, "name", "")
+        desc = getattr(p, "description", str(p))
+        out.append(f"line {line}: {desc}" + (f" [{name}]" if name else ""))
+    return out
+
+
+def _format(code: str) -> str:
+    try:
+        from gdtoolkit.formatter import format_code
+        return format_code(code, max_line_length=100)
+    except Exception:
+        return ""
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+def validate_code(code: str) -> BlockResult:
+    ok, err = _parse(code)
+    return BlockResult(
+        code=code, ok=ok, error=err,
+        lint=_lint(code) if ok else [],
+        formatted=_format(code) if ok else "",
+    )
+
+
+def validate_answer(answer: str) -> list[BlockResult]:
+    return [validate_code(b) for b in extract_gdscript_blocks(answer)]
+
+
+def render_report(results: list[BlockResult]) -> str:
+    """Markdown summary for the UI."""
+    if not results:
+        return "_No GDScript code blocks detected to validate._"
+    lines = []
+    for i, r in enumerate(results, 1):
+        if r.ok:
+            badge = "✅ **valid GDScript** (syntax OK)"
+            if r.lint:
+                badge += f" · {len(r.lint)} lint note(s)"
+        else:
+            badge = f"❌ **syntax error** — {r.error}"
+        lines.append(f"**Block {i}:** {badge}")
+        for w in r.lint[:5]:
+            lines.append(f"- ⚠ {w}")
+    return "\n".join(lines)
+
+
+def first_syntax_error(results: list[BlockResult]) -> tuple[str, str] | None:
+    """Return (code, error) of the first block that failed to parse, else None."""
+    for r in results:
+        if not r.ok:
+            return r.code, r.error
+    return None
+
+
+if __name__ == "__main__":
+    good = "extends Node\n\n@export var speed: float = 5.0\n\nfunc _ready() -> void:\n\tprint(speed)\n"
+    bad = "extends Node\n\nfunc _ready(\n\tprint('oops')\n"
+    for label, code in (("GOOD", good), ("BAD", bad)):
+        r = validate_code(code)
+        print(f"== {label} == ok={r.ok} error={r.error!r} lint={r.lint}")