Spaces:

VEFX-Reward
/

VEFX-Code

Running

File size: 8,469 Bytes

982d249
fdde4c8
 
 
 
befd6f9
fdde4c8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
befd6f9
fdde4c8
 
 
 
 
 
 
 
 
 
 
 
982d249
befd6f9
fdde4c8
 
 
 
befd6f9
 
 
fdde4c8
 
 
 
 
 
 
 
 
 
 
befd6f9
 
 
 
 
 
 
 
 
 
 
 
 
fdde4c8
befd6f9
fdde4c8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
befd6f9
fdde4c8

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>VEFX-Code · Reference code for VEFX-Bench v1.0</title>
<style>
  :root { --fg:#1a1a1a; --muted:#666; --bg:#fff; --card:#f7f7f9; --accent:#5b6cff; --border:#e5e7eb; }
  * { box-sizing:border-box; }
  body { font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif;
         max-width:880px; margin:0 auto; padding:2.5em 1.25em; color:var(--fg); background:var(--bg); line-height:1.6; }
  h1 { font-size:2.2em; margin:0 0 .2em; }
  h2 { margin-top:2em; padding-bottom:.3em; border-bottom:1px solid var(--border); }
  .tagline { color:var(--muted); font-size:1.1em; margin-bottom:1.5em; }
  .pill-row { display:flex; flex-wrap:wrap; gap:.6em; margin:1.5em 0; }
  .pill { display:inline-flex; align-items:center; gap:.4em; padding:.5em 1em; background:var(--card);
          border:1px solid var(--border); border-radius:999px; text-decoration:none; color:var(--fg);
          font-size:.95em; transition:all .15s; }
  .pill:hover { border-color:var(--accent); color:var(--accent); }
  table { width:100%; border-collapse:collapse; margin:1em 0; font-size:.92em; }
  th, td { padding:.5em .7em; border-bottom:1px solid var(--border); text-align:left; }
  th { background:var(--card); font-weight:600; }
  td.num, th.num { text-align:center; }
  code { background:var(--card); padding:.15em .4em; border-radius:4px; font-size:.9em; }
  td code { overflow-wrap:anywhere; word-break:break-all; }
  pre { background:var(--card); padding:1em; border-radius:6px; overflow-x:auto; }
  pre code { background:none; padding:0; }
  .grid { display:grid; grid-template-columns:1fr 1fr; gap:1em; margin:1em 0; }
  .card { background:var(--card); border:1px solid var(--border); border-radius:8px; padding:1em; }
  .card h3 { margin:0 0 .3em; font-size:1.05em; }
  .card p { margin:0; color:var(--muted); font-size:.9em; }
  footer { margin-top:3em; padding-top:1.5em; border-top:1px solid var(--border); color:var(--muted); font-size:.9em; }
  a { color:var(--accent); }
</style>
</head>
<body>

<h1>🎬 VEFX-Code</h1>
<p class="tagline">Reference code &amp; inference utilities for <strong>VEFX-Bench v1.0</strong> — frozen on 2026-08-03.</p>

<div class="pill-row">
  <a class="pill" href="https://huggingface.co/spaces/VEFX-Reward/VEFX-Code/tree/main" target="_blank">📂 Browse Files</a>
  <a class="pill" href="https://huggingface.co/spaces/VEFX-Reward/VEFX-Code/blob/main/README.md" target="_blank">📖 Full README</a>
  <a class="pill" href="https://huggingface.co/datasets/xiangbog/VEFX-Bench/tree/3bf997e7eb4fa0d0c2d56cce5ddfccc1dfbda235" target="_blank">📦 Frozen v1.0 Dataset</a>
  <a class="pill" href="https://huggingface.co/VEFX-Reward/VEFX-Reward-4B/tree/a15a8dbe1b3eb07ee0919e8de059f170436ec9ff" target="_blank">🤖 Frozen Reward Model</a>
  <a class="pill" href="https://vefx-leaderboard.com/" target="_blank">🏆 Live VEFX-Leaderboard</a>
</div>

<h2>📊 What's in VEFX-Bench</h2>
<p>5,049 annotated examples spanning 9 categories &amp; 32 subcategories, evaluated by <strong>VEFX-Reward</strong> — a VLM-based reward model that scores edits across three dimensions on a 1–4 scale:</p>

<div class="grid">
  <div class="card"><h3>Instructional Following (IF)</h3><p>Does the edit accurately reflect the editing instruction?</p></div>
  <div class="card"><h3>Render Quality (RQ)</h3><p>Visual clarity, temporal consistency, physical plausibility.</p></div>
  <div class="card"><h3>Edit Exclusivity (EE)</h3><p>Were only the intended regions modified, without side-effects?</p></div>
</div>

<h2>🗂️ Frozen v1.0 Task Taxonomy</h2>
<p>Attribute Editing · Camera Angle Editing · Camera Motion Editing · Creative Edit · Instance Editing · Instance Motion Editing · Quantity Editing · Style Editing · Visual Effect Editing</p>

<h2>🔒 Reproducibility</h2>
<table>
  <tbody>
    <tr><th>Dataset revision</th><td><code>3bf997e7eb4fa0d0c2d56cce5ddfccc1dfbda235</code></td></tr>
    <tr><th>benchmark_meta.json SHA-256</th><td><code>277d89f5cf23af4163fe6ee120654210f51d19c6dee2d3aa76a3248471d1f333</code></td></tr>
    <tr><th>Reward model revision</th><td><code>a15a8dbe1b3eb07ee0919e8de059f170436ec9ff</code></td></tr>
    <tr><th>model.safetensors SHA-256</th><td><code>c3c0d03f770f0a73631206821922213de75413bb517f7d7a2fd9ab1f2c38f59d</code></td></tr>
  </tbody>
</table>

<h2>🏆 Model Leaderboard</h2>
<p><strong>Overall is the primary GeoAgg metric, not the arithmetic average.</strong> For each item <code>j</code>, first compute <code>i=(IF-1)/3</code>, <code>r=(RQ-1)/3</code>, and <code>e=(EE-1)/3</code>, then <code>g_j=1+3·(i²·r·e)^(1/4)</code>. The leaderboard uses <code>Overall=(1/300)·Σg_j</code>: item-level GeoAgg first, then the mean over 300 items. Missing or failed items receive scores <code>(1,1,1)</code>. The arithmetic Mean is diagnostic only.</p>
<table>
  <thead><tr><th class="num">Rank</th><th>Model</th><th>Type</th><th class="num">IF ↑</th><th class="num">RQ ↑</th><th class="num">EE ↑</th><th class="num">GeoAgg ↑</th></tr></thead>
  <tbody>
    <tr><td class="num">🥇</td><td><strong>Kling o3 Omni</strong></td><td>Commercial</td><td class="num">3.033</td><td class="num"><strong>3.588</strong></td><td class="num">3.043</td><td class="num"><strong>3.057</strong></td></tr>
    <tr><td class="num">🥈</td><td><strong>Kling o1</strong></td><td>Commercial</td><td class="num"><strong>3.040</strong></td><td class="num">3.534</td><td class="num">2.976</td><td class="num">2.985</td></tr>
    <tr><td class="num">🥉</td><td><strong>Runway Gen-4.5</strong></td><td>Commercial</td><td class="num">2.817</td><td class="num">3.319</td><td class="num">2.923</td><td class="num">2.912</td></tr>
    <tr><td class="num">4</td><td>Seedance 2.0</td><td>Commercial</td><td class="num">2.811</td><td class="num">3.421</td><td class="num">3.088</td><td class="num">2.766</td></tr>
    <tr><td class="num">5</td><td>Grok Imagine</td><td>Commercial</td><td class="num">2.606</td><td class="num">3.346</td><td class="num"><strong>3.376</strong></td><td class="num">2.723</td></tr>
    <tr><td class="num">6</td><td>Luma Ray 3</td><td>Commercial</td><td class="num">2.702</td><td class="num">3.403</td><td class="num">2.705</td><td class="num">2.717</td></tr>
    <tr><td class="num">7</td><td>UniVideo</td><td>Open-source</td><td class="num">2.294</td><td class="num">3.266</td><td class="num">3.091</td><td class="num">2.516</td></tr>
    <tr><td class="num">8</td><td>Wan 2.6</td><td>Commercial</td><td class="num">2.012</td><td class="num">3.317</td><td class="num">2.446</td><td class="num">2.146</td></tr>
    <tr><td class="num">9</td><td>Luma Ray 2</td><td>Commercial</td><td class="num">2.038</td><td class="num">2.532</td><td class="num">1.363</td><td class="num">1.804</td></tr>
    <tr><td class="num">10</td><td>VACE</td><td>Open-source</td><td class="num">2.027</td><td class="num">3.172</td><td class="num">1.180</td><td class="num">1.775</td></tr>
  </tbody>
</table>

<h2>🎥 Sample Videos</h2>
<p>An example pair from <code>examples/sample_videos/</code> — original input on the left, edited output on the right.</p>
<div class="grid">
  <div><p style="margin:0 0 .3em;font-size:.9em;color:var(--muted);"><strong>Original</strong></p>
    <video controls style="width:100%;border-radius:6px;" src="examples/sample_videos/original.mp4"></video></div>
  <div><p style="margin:0 0 .3em;font-size:.9em;color:var(--muted);"><strong>Edited</strong></p>
    <video controls style="width:100%;border-radius:6px;" src="examples/sample_videos/edited.mp4"></video></div>
</div>

<h2>🚀 Quick Start</h2>
<pre><code>pip install -r requirements.txt
python examples/quick_start.py \
    --original examples/sample_videos/original.mp4 \
    --edited   examples/sample_videos/edited.mp4 \
    --instruction "Change the color of the trailer to bright yellow"</code></pre>
<p>See <a href="https://huggingface.co/spaces/VEFX-Reward/VEFX-Code/tree/main/examples" target="_blank">examples/</a> for batch &amp; multi-GPU scoring scripts.</p>

<footer>
  Released under Apache-2.0. Code lives in the <a href="https://huggingface.co/spaces/VEFX-Reward/VEFX-Code/tree/main" target="_blank">Files tab</a>; results and submissions are on the <a href="https://vefx-leaderboard.com/" target="_blank">live leaderboard</a>.
</footer>

</body>
</html>