FineVision: Open Data Is All You Need
Paper • 2510.17269 • Published • 81
AI & ML interests
None defined yet.
andito
authored a
paper 7 months ago
Post
2668
Finally, our new paper is out! "𝗙𝗶𝗻𝗲𝗩𝗶𝘀𝗶𝗼𝗻: 𝗢𝗽𝗲𝗻 𝗗𝗮𝘁𝗮 𝗜𝘀 𝗔𝗹𝗹 𝗬𝗼𝘂 𝗡𝗲𝗲𝗱"! 🥳
FineVision: Open Data Is All You Need (2510.17269)
If you've ever trained a VLM, you know this problem: nobody shares their data mixtures. It's a black box, making replicating SOTA work impossible.
We wanted to change that.
FineVision unifies 200 sources into 24 million samples. With 17.3 million images and 9.5 billion answer tokens, it's the largest open resource of its kind.
In the paper, we share how we built it:
🔍 finding and cleaning data at scale
🧹 removing excessive duplicates across sources
🤗 decontaminating against 66 public benchmarks
My favorite part is Figure 6 (in the video!). It's our visual diversity analysis. It shows that FineVision isn't just bigger; it's more balanced and conceptually richer than other open datasets.
NVIDIA's Eagle 2 paper highlighted just how critical this visual diversity is, and our results confirm it: models trained on FineVision consistently outperform those trained on any other open dataset on 11 benchmarks!
🎉 To celebrate the paper, I’m also releasing a concatenated and shuffled version of the full dataset! 👉
It’s ready to stream, so you can start training your own models right away:
from datasets import load_dataset
d = load_dataset("HuggingFaceM4/FineVision_full_shuffled", split="train", streaming=True)
print(next(iter(d)))
A big shoutout to the first authors: Luis Wiedmann and Orr Zohar. They are rockstars!
FineVision: Open Data Is All You Need (2510.17269)
If you've ever trained a VLM, you know this problem: nobody shares their data mixtures. It's a black box, making replicating SOTA work impossible.
We wanted to change that.
FineVision unifies 200 sources into 24 million samples. With 17.3 million images and 9.5 billion answer tokens, it's the largest open resource of its kind.
In the paper, we share how we built it:
🔍 finding and cleaning data at scale
🧹 removing excessive duplicates across sources
🤗 decontaminating against 66 public benchmarks
My favorite part is Figure 6 (in the video!). It's our visual diversity analysis. It shows that FineVision isn't just bigger; it's more balanced and conceptually richer than other open datasets.
NVIDIA's Eagle 2 paper highlighted just how critical this visual diversity is, and our results confirm it: models trained on FineVision consistently outperform those trained on any other open dataset on 11 benchmarks!
🎉 To celebrate the paper, I’m also releasing a concatenated and shuffled version of the full dataset! 👉
HuggingFaceM4/FineVision_full_shuffled It’s ready to stream, so you can start training your own models right away:
from datasets import load_dataset
d = load_dataset("HuggingFaceM4/FineVision_full_shuffled", split="train", streaming=True)
print(next(iter(d)))
A big shoutout to the first authors: Luis Wiedmann and Orr Zohar. They are rockstars!
burtenshaw
authored a
paper 8 months ago
burtenshaw
posted an update 9 months ago
Post
8389
Smol course has a distinctive approach to teaching post-training, so I'm posting about how it’s different to other post-training courses, including the llm course that’s already available.
In short, the smol course is just more direct that any of the other course, and intended for semi-pro post trainers.
- It’s a minimal set of instructions on the core parts.
- It’s intended to bootstrap real projects you're working on.
- The material handsover to existing documentation for details
- Likewise, it handsover to the LLM course for basics.
- Assessment is based on a leaderboard, without reading all the material.
To start the smol course, follow here:
smol-course
In short, the smol course is just more direct that any of the other course, and intended for semi-pro post trainers.
- It’s a minimal set of instructions on the core parts.
- It’s intended to bootstrap real projects you're working on.
- The material handsover to existing documentation for details
- Likewise, it handsover to the LLM course for basics.
- Assessment is based on a leaderboard, without reading all the material.
To start the smol course, follow here:
burtenshaw
posted an update 9 months ago
Post
5538
new smol course
If you’re building with or learning about post training AI models right now, we have a new FREE and CERTIFIED course.
🔗 Follow the org to join in smol-course
The course builds on smol course v1 which was the fastest way to learn to train your custom AI models. It now has:
- A leaderboard for students to submit models to
- Certification based on exams and leaderboards
- Prizes based on Leaderboards
- Up to date content on TRL and SmolLM3
- Deep integration with the Hub’s compute for model training and evaluation
We will release chapters every few weeks, so you can follow the org to stay updated.
If you’re building with or learning about post training AI models right now, we have a new FREE and CERTIFIED course.
🔗 Follow the org to join in
The course builds on smol course v1 which was the fastest way to learn to train your custom AI models. It now has:
- A leaderboard for students to submit models to
- Certification based on exams and leaderboards
- Prizes based on Leaderboards
- Up to date content on TRL and SmolLM3
- Deep integration with the Hub’s compute for model training and evaluation
We will release chapters every few weeks, so you can follow the org to stay updated.
- 2 replies
burtenshaw
posted an update 9 months ago
Post
3182
The open source AI community is just made of people who are passionate and care about their work. So we thought it would be cool to share our favourite icons of the community with a fun award.
Winners get free Hugging Face Pro Subscriptions, Merchandise, or compute credits for the hub.
🔗 Follow and nominate here:
community-spotlight
This is a new initiative to recognise and celebrate the incredible work being done by community members. It's all about inspiring more collaboration and innovation in the world of machine learning and AI.
They're highlighting contributors in four key areas:
- model creators: building and sharing innovative and state-of-the-art models.
- educators: sharing knowledge through posts, articles, demos, and events.
- tool builders: creating the libraries, frameworks, and applications that we all use.
- community champions: supporting and mentoring others in forums.
Know someone who deserves recognition? Nominate them by opening a post in the Hugging Face community forum.
Winners get free Hugging Face Pro Subscriptions, Merchandise, or compute credits for the hub.
🔗 Follow and nominate here:
This is a new initiative to recognise and celebrate the incredible work being done by community members. It's all about inspiring more collaboration and innovation in the world of machine learning and AI.
They're highlighting contributors in four key areas:
- model creators: building and sharing innovative and state-of-the-art models.
- educators: sharing knowledge through posts, articles, demos, and events.
- tool builders: creating the libraries, frameworks, and applications that we all use.
- community champions: supporting and mentoring others in forums.
Know someone who deserves recognition? Nominate them by opening a post in the Hugging Face community forum.
- 1 reply
Post
3098
Many VLMs claim to process hours of video. But can they follow the story?🤔
Today, we introduce TimeScope: The benchmark that separates true temporal understanding from marketing hype. Let's see how much VLMs really understand!⏳
We test three skills that matter for real-world use:
🔎 Localized Retrieval: Find a specific action.
🧩 Information Synthesis: Piece together scattered clues.
🏃 Fine-Grained Perception: Analyze detailed motion (e.g., count how many times a person swings an axe).
The results are in, and they're revealing. Only Gemini 2.5 pro handles 1-hour-long videos.
Performance drops sharply with duration, proving that long video understanding is still challenging. We've found the breaking points—now the community can start fixing them.📈
Want to learn more? TimeScope is 100% open-source. Benchmark your model and help us build the next generation of video AI.
📖 Blog:
https://huggingface.co/blog/timescope-video-lmm-benchmark
👩💻 Leaderboard & Demo: Apollo-LMMs/TimeScope
📊 Dataset: Apollo-LMMs/TimeScope
⚙️ Eval Code: https://github.com/EvolvingLMMs-Lab/lmms-eval
Today, we introduce TimeScope: The benchmark that separates true temporal understanding from marketing hype. Let's see how much VLMs really understand!⏳
We test three skills that matter for real-world use:
🔎 Localized Retrieval: Find a specific action.
🧩 Information Synthesis: Piece together scattered clues.
🏃 Fine-Grained Perception: Analyze detailed motion (e.g., count how many times a person swings an axe).
The results are in, and they're revealing. Only Gemini 2.5 pro handles 1-hour-long videos.
Performance drops sharply with duration, proving that long video understanding is still challenging. We've found the breaking points—now the community can start fixing them.📈
Want to learn more? TimeScope is 100% open-source. Benchmark your model and help us build the next generation of video AI.
📖 Blog:
https://huggingface.co/blog/timescope-video-lmm-benchmark
👩💻 Leaderboard & Demo: Apollo-LMMs/TimeScope
📊 Dataset: Apollo-LMMs/TimeScope
⚙️ Eval Code: https://github.com/EvolvingLMMs-Lab/lmms-eval
burtenshaw
posted an update 11 months ago
Post
1646
Kimi-K2 is ready for general use! In these notebooks I walk you through use cases like function calling and structured outputs.
🔗 burtenshaw/Kimi-K2-notebooks
You can swap it into any OpenAI compatible application via Inference Providers and get to work with an open source model.
🔗 burtenshaw/Kimi-K2-notebooks
You can swap it into any OpenAI compatible application via Inference Providers and get to work with an open source model.
- 1 reply
Post
4106
🧠👁️ Can AI visualize solutions?
Humans often solve visual problems by sketching ideas in our minds. What if Vision-Language Models (VLMs) could do something similar, not by generating full images, but by using internal “mental sketches”?
That’s the idea behind Mirage, a new framework that empowers VLMs to reason using latent visual tokens. Instead of just thinking in words, Mirage mixes in abstract visual representations that help the model solve complex tasks.
These aren't photorealistic images. They're compact, internal representations optimized purely to support reasoning.
🔧 Mirage is trained in two phases:
1) Grounding: It learns to produce latent tokens anchored in real images.
2) Refinement: The model drops the images and learns to generate visual tokens on its own.
📈 And yes, it works!
On challenging benchmarks like Visual Spatial Planning, Jigsaw puzzles, and Spatial Attention Tasks, Mirage clearly outperforms GPT-4o and other strong baselines.
Smart sketches > empty words.
By mimicking the way humans visualize solutions, Mirage gives AI a new kind of imagination, one that’s faster, more efficient, and more human-like.
Kudos to the teams at UMass Amherst and MIT behind this exciting work.
Check the paper: Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens (2506.17218)
Humans often solve visual problems by sketching ideas in our minds. What if Vision-Language Models (VLMs) could do something similar, not by generating full images, but by using internal “mental sketches”?
That’s the idea behind Mirage, a new framework that empowers VLMs to reason using latent visual tokens. Instead of just thinking in words, Mirage mixes in abstract visual representations that help the model solve complex tasks.
These aren't photorealistic images. They're compact, internal representations optimized purely to support reasoning.
🔧 Mirage is trained in two phases:
1) Grounding: It learns to produce latent tokens anchored in real images.
2) Refinement: The model drops the images and learns to generate visual tokens on its own.
📈 And yes, it works!
On challenging benchmarks like Visual Spatial Planning, Jigsaw puzzles, and Spatial Attention Tasks, Mirage clearly outperforms GPT-4o and other strong baselines.
Smart sketches > empty words.
By mimicking the way humans visualize solutions, Mirage gives AI a new kind of imagination, one that’s faster, more efficient, and more human-like.
Kudos to the teams at UMass Amherst and MIT behind this exciting work.
Check the paper: Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens (2506.17218)
- 4 replies
burtenshaw
posted an update 11 months ago
Post
3179
Inference for generative ai models looks like a mine field, but there’s a simple protocol for picking the best inference:
🌍 95% of users >> If you’re using open (large) models and need fast online inference, then use Inference providers on auto mode, and let it choose the best provider for the model. https://huggingface.co/docs/inference-providers/index
👷 fine-tuners/ bespoke >> If you’ve got custom setups, use Inference Endpoints to define a configuration from AWS, Azure, GCP. https://endpoints.huggingface.co/
🦫 Locals >> If you’re trying to stretch everything you can out of a server or local machine, use Llama.cpp, Jan, LMStudio or vLLM. https://huggingface.co/settings/local-apps#local-apps
🪟 Browsers >> If you need open models running right here in the browser, use transformers.js. https://github.com/huggingface/transformers.js
Let me know what you’re using, and if you think it’s more complex than this.
🌍 95% of users >> If you’re using open (large) models and need fast online inference, then use Inference providers on auto mode, and let it choose the best provider for the model. https://huggingface.co/docs/inference-providers/index
👷 fine-tuners/ bespoke >> If you’ve got custom setups, use Inference Endpoints to define a configuration from AWS, Azure, GCP. https://endpoints.huggingface.co/
🦫 Locals >> If you’re trying to stretch everything you can out of a server or local machine, use Llama.cpp, Jan, LMStudio or vLLM. https://huggingface.co/settings/local-apps#local-apps
🪟 Browsers >> If you need open models running right here in the browser, use transformers.js. https://github.com/huggingface/transformers.js
Let me know what you’re using, and if you think it’s more complex than this.
burtenshaw
posted an update 11 months ago
Post
1197
You don't need remote APIs for a coding copliot, or the MCP Course! Set up a fully local IDE with MCP integration using Continue. In this tutorial Continue guides you through setting it up.
This is what you need to do to take control of your copilot:
1. Get the Continue extension from the [VS Code marketplace](https://marketplace.visualstudio.com/items?itemName=Continue.continue) to serve as the AI coding assistant.
2. Serve the model with an OpenAI compatible server in Llama.cpp / LmStudio/ etc.
3. Create a
4. Create a
Check out the full tutorial in the [the MCP course](https://huggingface.co/learn/mcp-course/unit2/continue-client)
This is what you need to do to take control of your copilot:
1. Get the Continue extension from the [VS Code marketplace](https://marketplace.visualstudio.com/items?itemName=Continue.continue) to serve as the AI coding assistant.
2. Serve the model with an OpenAI compatible server in Llama.cpp / LmStudio/ etc.
llama-server -hf unsloth/Devstral-Small-2505-GGUF:Q4_K_M3. Create a
.continue/models/llama-max.yaml file in your project to tell Continue how to use the local Ollama model.name: Llama.cpp model
version: 0.0.1
schema: v1
models:
- provider: llama.cpp
model: unsloth/Devstral-Small-2505-GGUF
apiBase: http://localhost:8080
defaultCompletionOptions:
contextLength: 8192
# Adjust based on the model
name: Llama.cpp Devstral-Small
roles:
- chat
- edit4. Create a
.continue/mcpServers/playwright-mcp.yaml file to integrate a tool, like the Playwright browser automation tool, with your assistant.name: Playwright mcpServer
version: 0.0.1
schema: v1
mcpServers:
- name: Browser search
command: npx
args:
- "@playwright/mcp@latest"Check out the full tutorial in the [the MCP course](https://huggingface.co/learn/mcp-course/unit2/continue-client)
- 1 reply
burtenshaw
posted an update 12 months ago
Post
1817
Brand new MCP Course has units are out, and now it's getting REAL! We've collaborated with Anthropic to dive deep into production ready and autonomous agents using MCP
🔗 mcp-course
This is what the new material covers and includes:
- Use Claude Code to build an autonomous PR agent
- Integrate your agent with Slack and Github to integrate it with you Team
- Get certified on your use case and share with the community
- Build an autonomous PR cleanup agent on the Hugging Face hub and deploy it with spaces
The material goes deep into these problems and helps you to build applications that work. We’re super excited to see what you build with it.
🔗 mcp-course
This is what the new material covers and includes:
- Use Claude Code to build an autonomous PR agent
- Integrate your agent with Slack and Github to integrate it with you Team
- Get certified on your use case and share with the community
- Build an autonomous PR cleanup agent on the Hugging Face hub and deploy it with spaces
The material goes deep into these problems and helps you to build applications that work. We’re super excited to see what you build with it.
burtenshaw
posted an update 12 months ago
Post
1682
Super excited to release Autotrain MCP. This is an MCP server for training AI models, so you can use your AI tools to train your AI models 🤯.
🔗 burtenshaw/autotrain-mcp
Use this MCP server with tools like Claude Desktop, Cursor, VSCode, or Continue to do this:
- Define an ML problem like Image Classification, LLM fine-tuning, Text Classification, etc.
- The AI can retrieve models and datasets from the hub using the hub MCP.
- Training happens on a Hugging Face space, so no worries about hardware restraints.
- Models are pushed to the hub to be used inference tools like Llama.cpp, vLLM, MLX, etc.
- Built on top of the AutoTrain library, so it has full integration with transformers and other libraries.
Everything is still under active development, but I’m super excited to hear what people build, and I’m open to contributions!
🔗 burtenshaw/autotrain-mcp
Use this MCP server with tools like Claude Desktop, Cursor, VSCode, or Continue to do this:
- Define an ML problem like Image Classification, LLM fine-tuning, Text Classification, etc.
- The AI can retrieve models and datasets from the hub using the hub MCP.
- Training happens on a Hugging Face space, so no worries about hardware restraints.
- Models are pushed to the hub to be used inference tools like Llama.cpp, vLLM, MLX, etc.
- Built on top of the AutoTrain library, so it has full integration with transformers and other libraries.
Everything is still under active development, but I’m super excited to hear what people build, and I’m open to contributions!
- 1 reply
dvilasuero
posted an update 12 months ago
Post
3438
Super excited to launch Hugging Face Sheets: Spreadsheets meet AI and unstructured data.
A few months ago, we started imagining new ways to build and transform datasets with the latest open-source models.
Today, I'm thrilled to introduce our first step in this direction.
In a nutshell:
📁 Effortlessly run prompts and models over your data.
🌐 Agentic search for accuracy and real-time information.
🖼️ Familiar, minimalistic interface for interacting with data.
🎯 Human feedback 2.0: Your input directly improves generated data.
💯 Access hundreds of open models and leading inference providers.
Go to this space to try it out!
aisheets/sheets
Leave your questions below, we're just getting started!
A few months ago, we started imagining new ways to build and transform datasets with the latest open-source models.
Today, I'm thrilled to introduce our first step in this direction.
In a nutshell:
📁 Effortlessly run prompts and models over your data.
🌐 Agentic search for accuracy and real-time information.
🖼️ Familiar, minimalistic interface for interacting with data.
🎯 Human feedback 2.0: Your input directly improves generated data.
💯 Access hundreds of open models and leading inference providers.
Go to this space to try it out!
aisheets/sheets
Leave your questions below, we're just getting started!
- 3 replies
loubnabnl
authored a
paper 12 months ago
andito
authored a
paper 12 months ago
burtenshaw
posted an update about 1 year ago
Post
2755
MCP course is now LIVE! We just dropped quizzes, videos, and live streams to make it a fully interactive course:
🔗 join in now: mcp-course
- It’s still free!
- Video 1 walks you through onboarding to the course
- The first live session is next week!
- You can now get a certificate via exam app
- We improved and written material with interactive quizzes
If you’re studying MCP and want a live, interactive, visual, certified course, then join us on the hub!
🔗 join in now: mcp-course
- It’s still free!
- Video 1 walks you through onboarding to the course
- The first live session is next week!
- You can now get a certificate via exam app
- We improved and written material with interactive quizzes
If you’re studying MCP and want a live, interactive, visual, certified course, then join us on the hub!
Post
7407
SmolVLM is now available on PocketPal — you can run it offline on your smartphone to interpret the world around you. 🌍📱
And check out this real-time camera demo by @ngxson , powered by llama.cpp:
https://github.com/ngxson/smolvlm-realtime-webcam
https://x.com/pocketpal_ai
And check out this real-time camera demo by @ngxson , powered by llama.cpp:
https://github.com/ngxson/smolvlm-realtime-webcam
https://x.com/pocketpal_ai
- 5 replies
burtenshaw
posted an update about 1 year ago
Post
3340
We're thrilled to announce the launch of our comprehensive Model Context Protocol (MCP) Course! This free program is designed to take learners from foundational understanding to practical application of MCP in AI.
Follow the course on the hub: mcp-course
In this course, you will:
📖 Study Model Context Protocol in theory, design, and practice.
🧑💻 Learn to use established MCP SDKs and frameworks.
💾 Share your projects and explore applications created by the community.
🏆 Participate in challenges and evaluate your MCP implementations.
🎓 Earn a certificate of completion.
At the end of this course, you'll understand how MCP works and how to build your own AI applications that leverage external data and tools using the latest MCP standards.
Follow the course on the hub: mcp-course
In this course, you will:
📖 Study Model Context Protocol in theory, design, and practice.
🧑💻 Learn to use established MCP SDKs and frameworks.
💾 Share your projects and explore applications created by the community.
🏆 Participate in challenges and evaluate your MCP implementations.
🎓 Earn a certificate of completion.
At the end of this course, you'll understand how MCP works and how to build your own AI applications that leverage external data and tools using the latest MCP standards.
- 1 reply
burtenshaw
posted an update about 1 year ago
Post
2585
Qwen 3 Fine tuning >> MoE. Update the experiment thread to include config and script for fine-tuning the Qwen3-30B-A3B model.
The goal is to make a low latency non-thinking model for a daily driver coding, so 3 billion parameters active should be perfect.
✔️ training running
✔️ evals running
⏭️ improve dataset
The moe isn't going to fit into colab's A100 even with quantization (🙏 @UnslothAI ). So I've been working on HF spaces' H100s for this. Everything is available in the tread and I'll share more tomorrow.
burtenshaw/Qwen3-Code-Lite#1
The goal is to make a low latency non-thinking model for a daily driver coding, so 3 billion parameters active should be perfect.
✔️ training running
✔️ evals running
⏭️ improve dataset
The moe isn't going to fit into colab's A100 even with quantization (🙏 @UnslothAI ). So I've been working on HF spaces' H100s for this. Everything is available in the tread and I'll share more tomorrow.
burtenshaw/Qwen3-Code-Lite#1