Fine-Tuned-Blender (V11 β€” Least-Path Curvature Pruning + Long Training)

A Qwen2.5-Coder-1.5B-Instruct LoRA adapter (r=4, lora_alpha=8 on q_proj and v_proj) paired with repeated blockwise Optimal Brain Surgeon (OBS) curvature-pruned FP16 query projections (pruned_q_proj.safetensors) for headless Blender 5.1 bpy scripting.

V11 warm-starts from the V10 adapter (benchmark/v10/) and runs a 3-stage, 228-step (3 epochs over 76 training examples) Least-Path pruning + masked LoRA recovery schedule combining:

  • Least-Path repeated activation-Hessian blockwise column pruning & least-squares repair (Zenodo 24115016) across 4 evenly spaced attention query projections (model.layers.{0,9,18,27}.self_attn.q_proj) to 50.0% exact column sparsity (768 / 1536 surviving input columns per pruned projection).
  • Least-Change weight-displacement regularization (Zenodo 24074524) with strength=50000.0, keeping final adapter squared displacement at 5.120e-08 while enforcing exact zero columns on lora_A[:, ~mask] after every step.

On the Local Blender 5.1 Workflow Suite (28 held-out tasks in headless Blender 5.1.0), V11 improves from V10's 21 / 28 (75.00%) to 25 / 28 (89.29%), while cutting 50% of input columns from the 4 targeted q_proj layers. Across all 40 combined held-out Blender 5.1 tasks (28 workflow + 12 legacy single-object tasks), V11 passes 33 / 40 (82.50%) vs V10's 30 / 40 (75.00%).

Storage & Architecture Caveat (pruned_q_proj.safetensors)

  • Dense FP16 storage (no claimed inference speedup): The 4 pruned projections (layers[0, 9, 18, 27].self_attn.q_proj) are stored as dense torch.float16 [1536, 1536] matrices alongside [1536] FP16 biases and [1536] uint8 column masks in pruned_q_proj.safetensors. Exact 50.0% structural column sparsity and 0.0 effective contribution from pruned columns (base + LoRA) are verified in benchmark/v11/effective_sparsity_check.json, without claiming sparse-kernel throughput or latency gains.
  • Loading pruned_q_proj.safetensors vs. unpruned NF4 base: The 25 / 28 and 8 / 12 Blender execution scores were generated with pruned_q_proj.safetensors applied in FP16 and remaining layers quantized to 4-bit NF4. Evaluating the V11 LoRA adapter alone on the unpruned 4-bit NF4 base yields closely matching teacher-forced NLL (1.1985 OOD validation / 0.0155 test vs 1.2050 / 0.0161 with pruned FP16 q_proj; see benchmark/v11/unpruned_base_check.json), though separate unpruned Blender generations were not run.

External Benchmark Coverage

External public Blender/3D benchmarks (BlenderGym, BlenderWorld, 3DCodeBench, 3DHarnessBench, 3DCodeEdit, and BlenderBench) were NOT run on this model. All reported pass rates come exclusively from the local headless Blender 5.1.0 fixture suites committed under benchmark/v11/.

Suite Status in this repo Notes
Local Blender 5.1 Workflow Suite Executed (28 tasks) Scene-fixture multi-step & new API tasks (cases 1–28); reference scripts verified 28 / 28.
Local Legacy Single-Object Suite (v3) Executed (12 tasks) Original 8 test + 4 held-out single-object tasks; reference scripts verified 12 / 12.
BlenderBench / BlenderGym / BlenderWorld / 3DCodeBench / 3DHarnessBench / 3DCodeEdit Not executed No external benchmark scores are claimed for V11 or prior releases.

Local Blender 5.1 Workflow Suite (V11 vs V10)

All completions were generated with greedy decoding (do_sample=False, max_new_tokens=220, batch_size=1, enable_thinking=False) on an NVIDIA GeForce GTX 1650 and graded identically in headless Blender 5.1.0 (--factory-startup) using benchmark/v11/check_workflows.py and benchmark/v11/check_scenes.py.

Suite V10 Baseline V11 Least-Path (50% Pruned) V10 Mean Answer Tokens V11 Mean Answer Tokens
Local Blender 5.1 Workflow Suite (28 tasks) 21 / 28 (75.00%) 25 / 28 (89.29%) 79.54 81.46
  β†³ Prior workflows (cases 1–20) 20 / 20 (100.00%) 19 / 20 (95.00%) β€” β€”
  β†³ New V11 workflows (cases 21–28) 1 / 8 (12.50%) 6 / 8 (75.00%) β€” β€”
Legacy Single-Object Suite (v3, 12 tasks) 9 / 12 (75.00%) 8 / 12 (66.67%) 49.83 50.50
Combined Held-Out Blender 5.1 Tasks (40) 30 / 40 (75.00%) 33 / 40 (82.50%) β€” β€”

New Workflow Breakdown (cases 21–28) & Error Analysis

  • Spot lights (cases 21–22): V10 0 / 2 (invalid enum "LIGHT_SPOT") β†’ V11 2 / 2 (StageSpotA, RimSpotB).
  • 3D Poly curves (cases 23–24): V10 0 / 2 ('Curve' object has no attribute 'points' / AssertionError) β†’ V11 2 / 2 (CableA, CableB).
  • Multi-material polygon slots (cases 27–28): V10 1 / 2 (case 27 mistyped "CraneA" instead of "CrateA") β†’ V11 2 / 2 (CrateA, CrateB).
  • Vertex groups (cases 25–26): V10 0 / 2, V11 0 / 2. Both models look up lowercase "sleeveA" / "sleeveB" instead of exact-case "SleeveA" / "SleeveB" (while V11 emits valid vertex_groups.new / vg.add calls).
  • Remaining regressions in V11: Workflow case 18 (ShellB subdivision + smooth shading) raised bpy_prop_collection: attribute "load" not found, and v3 task 5 (EvalTriangle) failed linking a Mesh datablock instead of an Object. Legacy v3 tasks 4, 6, and 12 failed in both V10 and V11 (v3_regression_scores.json).

Training, Pruning & Honest NLL Report

Training & Repeated Pruning Setup

  • Base model & starting weights: Qwen/Qwen2.5-Coder-1.5B-Instruct in 4-bit NF4 (bnb_4bit_compute_dtype=torch.float16, double quantization), warm-started from blender-assistant-v10-workflows/least_change/adapter_model.safetensors (seed=42).
  • Training set (76 disjoint examples, max_length=320): 12 single-object replays + 24 prior workflow replays + 40 new Spot light, 3D Poly curve, Vertex group, and Multi-Material slot examples. All 76 scripts passed ast.parse and were constructed disjoint from the 28 test tasks and 2 untouched OOD validation tasks (ico_sphere mesh primitive and MIRROR modifier).
  • Schedule (228 steps total, 3 epochs):
    1. Cut 1 (0% β†’ 25%): Dequantize layers[0, 9, 18, 27].self_attn.q_proj to FP16, collect input second-moment Hessians over all 76 sequences (13,666 calibration tokens), prune 4 lowest-saliency columns per block_size=16 (1,152 / 1,536 surviving), apply exact local least-squares repair ($\Delta_S = W_D H_{DS} H_{SS}^{-1}$), zero pruned lora_A[:, ~mask], then run Steps 1–76 masked LoRA recovery (lr=2e-4, strength=50000.0, logit chunk 4).
    2. Cut 2 (25% β†’ 50%): Re-calibrate activations on the Cut 1 recovered model (13,666 tokens), prune another 4 surviving columns per block (768 / 1,536 surviving), repair surviving columns, zero lora_A[:, ~mask], then run Steps 77–152 masked recovery.
    3. Long Continuation (Steps 153–228): Continue column-masked Least-Change LoRA training for a full 3rd epoch to Step 228.

Stage-by-Stage Teacher-Forced NLL & Displacement Trajectory

NLL Caveat (Prompt-Template Overlap): Very low test_28 NLL (0.0161 across 2,248 tokens on 28 cases) occurs because those 28 workflow prompts reuse training-seen structural API families and prompt phrasing with held-out names/parameters, whereas the 2 untouched out-of-distribution (OOD) validation templates (ico_sphere, MIRROR, 50 tokens total) stay at 1.2050 NLL. Pair both numbers rather than interpreting test_28 NLL as general bpy perplexity.

Stage Surviving Cols (q_proj) OOD Val NLL (2 ex / 50 tok) Workflow Test NLL (28 ex / 2,248 tok) Legacy v3 NLL (12 ex / 546 tok) Squared Weight Change (drift)
V10 starting baseline 1536 / 1536 (0% cut) 1.1180 0.1759 0.4868 β€”
Cut 1 unrecovered 1152 / 1536 (25% cut) 1.2022 0.1754 β€” β€”
Cut 1 recovered (step 76) 1152 / 1536 (25% cut) 1.0766 0.0198 0.4579 5.511e-08
Cut 2 unrecovered 768 / 1536 (50% cut) 1.0038 0.0186 β€” β€”
Cut 2 recovered (step 152) 768 / 1536 (50% cut) 1.1485 0.0324 0.4405 5.464e-08
Final V11 (step 228, pruned FP16 q_proj) 768 / 1536 (50% cut) 1.2050 0.0161 0.5037 5.120e-08
Unpruned NF4 base + V11 LoRA (no pruned_q_proj) 1536 / 1536 (unpruned) 1.1985 0.0155 β€” 5.120e-08

Verified Sparsity Per Pruned Layer (effective_sparsity_check.json)

Module Surviving Columns Base Zero Fraction Effective Dead-Column Max Abs (W_eff[:, ~mask])
model.layers.0.self_attn.q_proj 768 / 1536 0.500014 0.0
model.layers.9.self_attn.q_proj 768 / 1536 0.500000 0.0
model.layers.18.self_attn.q_proj 768 / 1536 0.500001 0.0
model.layers.27.self_attn.q_proj 768 / 1536 0.500003 0.0

Reproducibility & Decoding Logs

Every published completion file has a matching generation script and per-case decoding log with exact rendered_chat_prompt, generated_token_ids, stopped_on_eos, timings, environment versions (torch 2.1.2+cu118, transformers 4.46.3, peft 0.13.2, bitsandbytes 0.45.5), and SHA256 hashes:

Exact Prompt Template

Write a Blender 5.1 Python script using bpy. {task}
Keep it concise. Output code only, with no comments or Markdown.
Return only the Python solution.

Rendered via tokenizer.apply_chat_template([{"role": "user", "content": content}], tokenize=True, add_generation_prompt=True, enable_thinking=False).

Reproduce Completions & Blender 5.1 Grading

# 1. Regenerate greedy completions with pruned_q_proj.safetensors applied
python benchmark/v11/generate_eval.py \
  --base-model Qwen/Qwen2.5-Coder-1.5B-Instruct \
  --adapter . \
  --cases benchmark/v11/cases.jsonl \
  --output v11-repro.jsonl

# 2. Grade workflow suite (28 tasks) in headless Blender 5.1
blender -b --factory-startup --python benchmark/v11/check_workflows.py -- \
  --cases benchmark/v11/cases.jsonl benchmark/v11/v11-28_completions.jsonl

# 3. Grade legacy single-object suite (12 tasks) in headless Blender 5.1
blender -b --factory-startup --python benchmark/v11/check_scenes.py -- \
  --extended benchmark/v11/v11-v3_completions.jsonl

Quick Start

To reproduce V11's exact evaluated behavior, load pruned_q_proj.safetensors onto layers[0, 9, 18, 27].self_attn.q_proj before attaching the PEFT LoRA adapter:

import torch
from huggingface_hub import hf_hub_download
from peft import PeftModel
from safetensors.torch import load_file
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

base_id = "Qwen/Qwen2.5-Coder-1.5B-Instruct"
repo_id = "Chaman1234/Fine-Tuned-Blender"

tokenizer = AutoTokenizer.from_pretrained(base_id)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.float16,
)
base = AutoModelForCausalLM.from_pretrained(
    base_id,
    quantization_config=bnb_config,
    torch_dtype=torch.float16,
    device_map={"": 0},
)

pruned_path = hf_hub_download(repo_id, "pruned_q_proj.safetensors")
tensors = load_file(pruned_path, device="cpu")
for idx in (0, 9, 18, 27):
    attn = base.model.layers[idx].self_attn
    old = attn.q_proj
    weight = tensors[f"layer_{idx}.weight_fp16"]
    bias = tensors[f"layer_{idx}.bias_fp16"]
    linear = torch.nn.Linear(weight.shape[1], weight.shape[0], bias=True, dtype=torch.float16)
    with torch.no_grad():
        linear.weight.copy_(weight)
        linear.bias.copy_(bias)
    linear.requires_grad_(False)
    attn.q_proj = linear.to(old.weight.device)

model = PeftModel.from_pretrained(base, repo_id).eval()

task = "Create a spot light named StageSpotA at (3, -4, 5) with 750 watts, color (1.0, 0.9, 0.7), and spot angle 45 degrees."
prompt = (
    f"Write a Blender 5.1 Python script using bpy. {task}\n"
    "Keep it concise. Output code only, with no comments or Markdown.\n"
    "Return only the Python solution."
)
ids = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
)
input_ids = torch.tensor([ids], device=next(model.parameters()).device)
attention_mask = torch.ones_like(input_ids)
with torch.inference_mode():
    out = model.generate(
        input_ids=input_ids,
        attention_mask=attention_mask,
        pad_token_id=tokenizer.pad_token_id or tokenizer.eos_token_id,
        max_new_tokens=220,
        do_sample=False,
        temperature=None,
        top_p=None,
        top_k=None,
    )
print(tokenizer.decode(out[0][len(ids):], skip_special_tokens=True))

Related Zenodo records

Downloads last month
97
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Chaman1234/Fine-Tuned-Blender

Adapter
(237)
this model

Evaluation results

  • Headless Blender 5.1 Execution Pass Rate (25/28) on Local Blender 5.1 Workflow Suite (28 Tasks)
    self-reported
    89.290
  • Headless Blender 5.1 Execution Pass Rate (8/12) on Local Legacy Single-Object Suite (v3, 12 Tasks)
    self-reported
    66.670