cortex-agent-llc/kodiak-v0.3-1b-accuracy
Updated
Shane Larson PRO
comgen42
AI & ML interests
None yet
Recent Activity
updated a model about 4 hours ago
cortex-agent-llc/kodiak-v0.3-1b-accuracy posted an update about 4 hours ago
🐻 Kodiak-v0.3-1B: seven new kinds of decision, measured on real data.
Kodiak is an open 1B encoder that answers typed questions with calibrated confidence, or says "can't tell". New in v0.3:
• pairwise judge (which answer is better?)
• long-answer hallucination checks
• stance and sarcasm
• policy violation and refund-eligibility checks against your written rules
• agent step safety (run it, ask first, or never)
We stopped trusting our own synthetic tests and judged v0.3 on real labelled data it never trained on (RAGBench, MT-Bench human judgments, SemEval stance): 0.21 → 0.36, averaged over 3 training runs. Nothing else got worse, and when it says "can't tell" it's right 93% of the time (up from 88%).
Known limits are in the model card.
📦 https://huggingface.co/cortex-agent-llc/kodiak-v0.3-1b
🎯 Accuracy mode: https://huggingface.co/cortex-agent-llc/kodiak-v0.3-1b-accuracy
🕹️ Demo: https://huggingface.co/spaces/comgen42/kodiak-demo
📝 What we built, and the kill that changed our process: https://cortexagent.com/blog/kodiak-v0-3-seven-new-kinds-of-decision-measured-on-real-data
updated a Space about 5 hours ago
comgen42/kodiak-demoOrganizations
updated a
model about 4 hours ago
posted an update about 4 hours ago
Post
32
🐻 Kodiak-v0.3-1B: seven new kinds of decision, measured on real data.
Kodiak is an open 1B encoder that answers typed questions with calibrated confidence, or says "can't tell". New in v0.3:
• pairwise judge (which answer is better?)
• long-answer hallucination checks
• stance and sarcasm
• policy violation and refund-eligibility checks against your written rules
• agent step safety (run it, ask first, or never)
We stopped trusting our own synthetic tests and judged v0.3 on real labelled data it never trained on (RAGBench, MT-Bench human judgments, SemEval stance): 0.21 → 0.36, averaged over 3 training runs. Nothing else got worse, and when it says "can't tell" it's right 93% of the time (up from 88%).
Known limits are in the model card.
📦 cortex-agent-llc/kodiak-v0.3-1b
🎯 Accuracy mode: cortex-agent-llc/kodiak-v0.3-1b-accuracy
🕹️ Demo: comgen42/kodiak-demo
📝 What we built, and the kill that changed our process: https://cortexagent.com/blog/kodiak-v0-3-seven-new-kinds-of-decision-measured-on-real-data
Kodiak is an open 1B encoder that answers typed questions with calibrated confidence, or says "can't tell". New in v0.3:
• pairwise judge (which answer is better?)
• long-answer hallucination checks
• stance and sarcasm
• policy violation and refund-eligibility checks against your written rules
• agent step safety (run it, ask first, or never)
We stopped trusting our own synthetic tests and judged v0.3 on real labelled data it never trained on (RAGBench, MT-Bench human judgments, SemEval stance): 0.21 → 0.36, averaged over 3 training runs. Nothing else got worse, and when it says "can't tell" it's right 93% of the time (up from 88%).
Known limits are in the model card.
📦 cortex-agent-llc/kodiak-v0.3-1b
🎯 Accuracy mode: cortex-agent-llc/kodiak-v0.3-1b-accuracy
🕹️ Demo: comgen42/kodiak-demo
📝 What we built, and the kill that changed our process: https://cortexagent.com/blog/kodiak-v0-3-seven-new-kinds-of-decision-measured-on-real-data
- 1 reply
updated a
Space about 5 hours ago
published a
model about 5 hours ago
updated a
model about 6 hours ago
published a
model about 6 hours ago
upvoted a changelog about 6 hours ago
Hugging Face Changelog
P(doom) on user profiles
• 156
updated a
dataset 4 days ago
published a
dataset 4 days ago
liked a
model 4 days ago
updated 2
models 4 days ago
replied to their post 4 days ago
Thanks, this is a great catch, and you're right on both counts.
I reproduced your numbers (Qwen3-8B 14.7%, large v2 49.6%) and ran your two questions on the v0.2 predictions:
Ranking (share of the random→perfect AURC gap closed, never-seen, n=3,550):
• Kodiak-v0.2-1B: 54.8% / 56.1% / 55.9% (3 seeds)
• v0.2 accuracy mode: 56.6%
• Qwen3-8B: 14.7%
Calibration after giving Qwen a fitted monotone (isotonic) recalibration, fit on the other never-seen tasks and applied to each held-out task:
• Qwen3-8B: 0.293 → 0.178
• v0.2-1B: 0.075–0.094 raw → 0.044–0.058 with the same treatment
So the gap survives, but the raw 0.085 vs 0.293 overstates it, and ranking is the better headline. It's what the cascade actually spends. We'll add a ranking row to the model card, footnote the calibration comparison as raw, and put the v0.2 prediction files in the repo so this can be checked.
posted an update 4 days ago
Post
105
🐻 Kodiak-v0.2-1B is out: an open 1B encoder that makes decisions instead of writing text.
Give it a state and typed questions. You get back calibrated answers in one forward pass, and it says "can't tell" when it doesn't know.
On tasks it was never trained for (our frozen eval set v0.2):
• 0.689 accuracy (3-run mean) vs 0.688 for Qwen3-8B, at ~40× the speed
• calibration error 0.085 vs 0.293
• accuracy mode (3 models averaged): 0.706
New in v0.2: grounding checks, picking an assistant's next tool step from API specs, claim verification, and product relevance.
Known limits are in the model card.
📦 cortex-agent-llc/kodiak-v0.2-1b
🎯 Accuracy mode: cortex-agent-llc/kodiak-v0.2-1b-accuracy
🕹️ Demo: comgen42/kodiak-demo
📝 What we built, and what didn't work: https://cortexagent.com/blog/kodiak-v0-2-an-open-1b-decision-model-you-can-download-today
Give it a state and typed questions. You get back calibrated answers in one forward pass, and it says "can't tell" when it doesn't know.
On tasks it was never trained for (our frozen eval set v0.2):
• 0.689 accuracy (3-run mean) vs 0.688 for Qwen3-8B, at ~40× the speed
• calibration error 0.085 vs 0.293
• accuracy mode (3 models averaged): 0.706
New in v0.2: grounding checks, picking an assistant's next tool step from API specs, claim verification, and product relevance.
Known limits are in the model card.
📦 cortex-agent-llc/kodiak-v0.2-1b
🎯 Accuracy mode: cortex-agent-llc/kodiak-v0.2-1b-accuracy
🕹️ Demo: comgen42/kodiak-demo
📝 What we built, and what didn't work: https://cortexagent.com/blog/kodiak-v0-2-an-open-1b-decision-model-you-can-download-today
- 3 replies
published a
model 4 days ago
liked a
model 4 days ago
published a
model 4 days ago
liked a
model 5 days ago
liked a
Space 7 days ago
updated a
model 7 days ago