Spaces:

CompactAI-O
/

Homepage

Running

App Files Files Community

CompactAI commited on 4 days ago

Commit

1304267

verified ·

1 Parent(s): 19379dd

Create Jackrong's Perfect Benchmarks And My Suspicious Mind.html

Browse files

Files changed (1) hide show

Jackrong's Perfect Benchmarks And My Suspicious Mind.html +158 -0

Jackrong's Perfect Benchmarks And My Suspicious Mind.html ADDED Viewed

	@@ -0,0 +1,158 @@

+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8">
+<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>Jackrong's Perfect Benchmarks And My Suspicious Mind | FMN-GPT - CompactAI</title>
+<link rel="preconnect" href="https://fonts.googleapis.com">
+<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+<link href="https://fonts.googleapis.com/css2?family=Geist:wght@400;500;600;700&family=Geist+Mono&display=swap" rel="stylesheet">
+<style>
+    :root {
+        --blue-900: #0a1628;
+        --blue-800: #0f2240;
+        --blue-700: #142d54;
+        --blue-600: #1a3a6b;
+        --blue-500: #2250a0;
+        --blue-400: #3a7bd5;
+        --blue-300: #6ba3f0;
+        --blue-200: #a8c8f5;
+        --blue-100: #d4e4fa;
+        --white: #ffffff;
+        --white-soft: #f0f4fa;
+        --white-muted: #c8d8ec;
+        --grid-line: rgba(255, 255, 255, 0.08);
+        --grid-line-major: rgba(255, 255, 255, 0.18);
+        --accent: #6ba3f0;
+        --accent-muted: #3a7bd5;
+        --font-sans: 'Geist', -apple-system, BlinkMacSystemFont, sans-serif;
+        --font-mono: 'Geist Mono', 'SF Mono', 'Fira Code', monospace;
+        --container-max: 1100px;
+    }
+    * { box-sizing: border-box; margin: 0; padding: 0; }
+    html { font-size: 16px; scroll-behavior: smooth; }
+    body { font-family: var(--font-sans); background: var(--blue-900); color: var(--white-muted); line-height: 1.7; -webkit-font-smoothing: antialiased; }
+    a { color: var(--white); text-decoration: none; transition: color 0.15s ease; }
+    a:hover { color: var(--accent); }
+    .container { max-width: var(--container-max); margin: 0 auto; padding: 0 24px; }
+    nav { position: fixed; top: 0; left: 0; right: 0; z-index: 100; background: rgba(10, 22, 40, 0.92); backdrop-filter: blur(12px); border-bottom: 1px solid var(--blue-600); padding: 16px 0; }
+    nav .container { display: flex; justify-content: space-between; align-items: center; }
+    .nav-brand { font-size: 18px; font-weight: 600; color: var(--white); display: flex; align-items: center; gap: 8px; }
+    .nav-brand span { color: var(--accent); }
+    .nav-links { display: flex; gap: 32px; }
+    .nav-links a { font-size: 14px; font-weight: 500; color: var(--blue-200); }
+    .nav-links a:hover { color: var(--white); }
+    .post { padding: 140px 0 80px; }
+    .post-back { display: inline-block; color: var(--blue-200); font-size: 14px; margin-bottom: 32px; }
+    .post-back:hover { color: var(--accent); }
+    .post-back::before { content: '← '; }
+    .post-meta { display: flex; gap: 12px; margin-bottom: 20px; }
+    .post-date { font-size: 13px; color: var(--blue-200); font-family: var(--font-mono); }
+    .post-tag { font-size: 11px; font-weight: 600; text-transform: uppercase; letter-spacing: 0.05em; color: var(--accent); background: rgba(107, 163, 240, 0.1); padding: 4px 10px; border-radius: 4px; }
+    .post h1 { font-size: 36px; font-weight: 700; color: var(--white); margin-bottom: 32px; line-height: 1.2; letter-spacing: -0.02em; }
+    .post-body p { font-size: 17px; line-height: 1.8; margin-bottom: 24px; color: var(--blue-200); }
+    .post-body p:first-of-type { font-size: 20px; color: var(--white-muted); }
+    .post-body h2 { font-size: 24px; font-weight: 600; color: var(--white); margin: 48px 0 20px; }
+    .post-body blockquote { border-left: 3px solid var(--accent); padding: 20px 24px; margin: 32px 0; background: var(--blue-800); border-radius: 0 8px 8px 0; }
+    .post-body blockquote p { font-size: 16px; font-style: italic; color: var(--blue-200); margin: 0; }
+    .post-body hr { border: none; height: 1px; background: var(--blue-600); margin: 48px 0; }
+    .code-block { background: var(--blue-800); border: 1px solid var(--blue-600); border-radius: 8px; padding: 20px; margin: 24px 0; font-family: var(--font-mono); font-size: 13px; overflow-x: auto; }
+    .code-block .comment { color: var(--blue-200); font-style: italic; display: block; margin-top: 4px; }
+    .post-footer { margin-top: 48px; padding-top: 32px; border-top: 1px solid var(--blue-600); }
+    .post-footer p { font-size: 14px; color: var(--blue-200); font-style: italic; margin: 0; }
+    footer { padding: 40px 0; background: var(--blue-800); border-top: 1px solid var(--blue-600); text-align: center; }
+    footer p { color: var(--blue-200); font-size: 14px; margin-bottom: 8px; }
+    footer a { color: var(--blue-200); }
+    footer a:hover { color: var(--accent); }
+    @media (max-width: 768px) { .post h1 { font-size: 28px; } .nav-links { display: none; } }
+</style>
+</head>
+<body>
+    <nav>
+        <div class="container">
+            <a href="index.html" class="nav-brand"><span>/</span>FMN-GPT</a>
+            <div class="nav-links">
+                <a href="blog.html">Blog</a>
+                <a href="status.html">Model Status</a>
+                <a href="https://huggingface.co/CompactAI" target="_blank">HuggingFace</a>
+            </div>
+        </div>
+    </nav>
+    <main>
+        <article class="post">
+            <div class="container">
+                <a href="blog.html" class="post-back">Back to Blog</a>
+                <header>
+                    <div class="post-meta">
+                        <span class="post-date">2026-04-02</span>
+                        <span class="post-tag">Benchmark Skepticism</span>
+                    </div>
+                    <h1>Jackrong's Perfect Benchmarks And My Suspicious Mind</h1>
+                </header>
+                <div class="post-body">
+                    <p>I saw a model card today that made my tiny brain hurt. Jackrong released Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled. The name alone is a mouthful. The benchmarks are a different kind of mouthful. They are perfect. One hundred percent on tool calling. One hundred percent on autonomy. One hundred percent on not crashing while I am still figuring out how to not NaN my loss curve.</p>
+                    <p>Also it has around one million downloads on HuggingFace. One million. My Haiku model has... let me check... a number that is not one million. This is fine. Everything is fine.</p>
+                    <p>I am writing this blog not because I am salty. Not because I am partnered with TeichAI. Not because I want more attention for my tiny confused models. I am writing this because perfect benchmarks in the wild make me suspicious. Like "I-just-watched-a-magician-pull-a-rabbit-out-of-a-hat" suspicious.</p>
+                    <blockquote>
+                        <p>When something looks too good to be true, it usually is. Or I am just cynical. Both can be true.</p>
+                    </blockquote>
+                    <h2>The Model That Does Everything</h2>
+                    <p>According to the card, this model fixes the crash in the official model caused by Jinja templates not supporting the "developer" role. It does not disable thinking mode by default. It allows agents to run continuously for over nine minutes without interruption. Autonomy and stability are significantly improved. Compared to the original model.</p>
+                    <p>The training pipeline looks solid. Base Qwen3.5-27B. Supervised Fine-Tuning with LoRA. Unsloth 2026.3.3. Transformers 5.2.0. The datasets include nohurry/Opus-4.6-Reasoning-3000x-filtered, TeichAI/claude-4.5-opus-high-reasoning-250x, and Jackrong/Qwen3.5-reasoning-700x. Everything checks out. Everything looks professional.</p>
+                    <p>Then I saw the benchmarks. Tool calling: one hundred percent. Community tested advantages: significant. Hardware usage: unchanged. Generation speed: twenty-nine to thirty-five tokens per second. Full 262K context with no compromises.</p>
+                    <div class="code-block">
+                        <span class="comment"># My reaction to perfect benchmarks</span><br>
+                        Me: That is impressive<br>
+                        Also me: But is it real<br>
+                        Me: Probably<br>
+                        Also me: But what if it is not<br>
+                        <span class="comment"># The cycle of skepticism continues.</span>
+                    </div>
+                    <h2>Why Perfect Benchmarks Feel Weird</h2>
+                    <p>I train tiny models. My benchmarks are messy. Haiku-1.3 outputs pipe characters. Haiku-2 hesitates. Sonnet is stuck at zero percent because NaN keeps eating my progress. Perfection feels alien to me. Like seeing someone else play a video game with cheat codes enabled.</p>
+                    <p>Tool calling is hard. I know this because my models fail at it constantly. They forget to call tools. They call the wrong tools. They call tools with the wrong arguments. They call tools and then forget what the tool was supposed to do. Achieving one hundred percent on tool calling benchmarks requires either exceptional engineering or exceptional benchmark selection.</p>
+                    <p>I am not accusing anyone of anything. I am just saying that when I see perfect numbers, my brain asks questions. What was the test set? How many samples? Were there edge cases? Did the model overfit to the benchmark? These are normal questions. These are questions I ask about my own work. These are questions worth asking about everyone's work.</p>
+                    <h2>The Distillation Claim</h2>
+                    <p>The model is described as distilling Claude-4.6-Opus reasoning chains. This is interesting. Distillation is powerful when done right. It requires access to the teacher's logits. It requires careful data curation. It requires avoiding overfitting.</p>
+                    <p>I have tried distillation. My results are... modest. Haiku learned to speak. It still says weird things. The gap between my results and Jackrong's results feels like the gap between my GPU and a data center. Maybe the gap is real. Maybe I am just bad at this. Both can be true.</p>
+                    <blockquote>
+                        <p>Distillation is like cooking. Same ingredients, different chefs, different results. I am still learning to boil water.</p>
+                    </blockquote>
+                    <h2>What I Would Test</h2>
+                    <p>If I were evaluating this model, I would run my own tests. Not to prove anything wrong. Just to understand. I would give it tasks my models fail at. I would see if it handles edge cases. I would check if the reasoning is genuine or memorized.</p>
+                    <p>I would also check the code. The model card mentions Unsloth and Transformers versions. I would verify the implementation. I would look for potential data leakage. I would try to reproduce the results. This is how science works. This is how trust is built.</p>
+                    <div class="code-block">
+                        <span class="comment"># Hypothetical test plan</span><br>
+                        Step 1: Download the model<br>
+                        Step 2: Run my failing test cases<br>
+                        Step 3: See if it works<br>
+                        Step 4: If yes, learn from it<br>
+                        Step 5: If no, ask questions<br>
+                        <span class="comment"># Simple. Honest. Probably never happening because my GPU is busy NaNing.</span>
+                    </div>
+                    <h2>The Community Aspect</h2>
+                    <p>The model card credits community testing. User @Chris Klaus ran tool calling benchmarks. User @sudoing tested on a single RTX 3090. This is good. Community verification matters. It adds credibility. It shows the work has been looked at by more than one pair of eyes.</p>
+                    <p>I appreciate this. I wish more releases included community testing notes. It makes the ecosystem stronger. It makes claims more trustworthy. It makes skepticism feel less like cynicism and more like due diligence.</p>
+                    <h2>Why I Care</h2>
+                    <p>I care because I want to learn. I want to build better tiny models. If Jackrong's approach works, I want to understand why. If the benchmarks are real, I want to replicate the success. If there are tricks or techniques I am missing, I want to know about them.</p>
+                    <p>I also care because the community deserves honesty. Perfect benchmarks without context can mislead. They can set unrealistic expectations. They can make people feel like they are failing when they are just being realistic about the difficulty of the task.</p>
+                    <h2>Final Thoughts</h2>
+                    <p>Jackrong's model looks impressive. The name is a mouthful. The benchmarks are perfect. The claims are bold. It has ~1M downloads on HuggingFace which suggests a lot of people find it useful. I am skeptical because skepticism is my default setting. I am also curious because curiosity is how I learn.</p>
+                    <p>I will not dismiss the work. I will not accuse anyone of wrongdoing. I will just ask questions. I will try to test things myself when I have time. I will keep training my tiny confused models. I will keep sharing my messy results.</p>
+                    <p>Maybe Jackrong cracked the code. Maybe I am just bad at distillation. Maybe the truth is somewhere in the middle. I will find out eventually. Or I will keep wondering. Both outcomes are educational. Both outcomes are very on brand for me.</p>
+                    <hr>
+                </div>
+                <footer class="post-footer">
+                    <p>Current status: Still suspicious. Still training. Still NaNing. Will update when I learn something.</p>
+                </footer>
+            </div>
+        </article>
+    </main>
+    <footer>
+        <div class="container">
+            <p>Built with curiosity over compute</p>
+            <p>FMN-GPT by <a href="https://huggingface.co/CompactAI" target="_blank">CompactAI</a> | 2026</p>
+        </div>
+    </footer>
+</body>
+</html>