Hi Flash! I apologize for the delay in the response, but I'm still answering if anyone has this exact question. According to the GAIA paper, the evaluation is done by matching the answers your agent gives with a ground truth. Being only one correct answer per question. The responses are evaluated doing a quasi exact match between them. Hence, adding a reasoning trace would not improve the score. As the benchmark advices to send only the minimal answer to do this evaluation easier.
Carlos Ochoa PRO
hetline
AI & ML interests
None yet
Recent Activity
commentedon their article about 2 months ago
Practical advice when building agents: A fast view on GAIA benchmark challenge published a dataset 5 months ago
hetline/tiny-coop-es updated a dataset 5 months ago
hetline/tiny-coop-esOrganizations
commented on Practical advice when building agents: A fast view on GAIA benchmark challenge about 2 months ago
published a
dataset 5 months ago
updated a
dataset 5 months ago
reacted to qgallouedec's post with 🔥 5 months ago
Post
3098
@CohereLabs just released 🌿 Tiny Aya: a fully open-source 3B parameter model that speaks 70+ languages 🌍! But there’s a catch:
Tiny Aya is just a language model. It doesn’t support tool calling, the key capability that turns frontier models into powerful *agents*.
So the real question is:
How hard is it to turn Tiny Aya into an agent?
Turns out… it’s simple, thanks to Hugging Face TRL.
We’re sharing a hands-on example showing how to train Tiny Aya to turn it into a tool-calling agent using TRL, unlocking what could become the first *massively multilingual open agent*.
Small model. Global reach. Agent capabilities.
👉 https://github.com/huggingface/trl/blob/main/examples/notebooks/sft_tool_calling.ipynb
Tiny Aya is just a language model. It doesn’t support tool calling, the key capability that turns frontier models into powerful *agents*.
So the real question is:
How hard is it to turn Tiny Aya into an agent?
Turns out… it’s simple, thanks to Hugging Face TRL.
We’re sharing a hands-on example showing how to train Tiny Aya to turn it into a tool-calling agent using TRL, unlocking what could become the first *massively multilingual open agent*.
Small model. Global reach. Agent capabilities.
👉 https://github.com/huggingface/trl/blob/main/examples/notebooks/sft_tool_calling.ipynb
- 1 reply
published an article 9 months ago
Article
Practical advice when building agents: A fast view on GAIA benchmark challenge
hetline
• upvoted a paper 11 months ago
Dataset?
4
#3 opened about 1 year ago
by 
trentmkelly
reacted to Jaward's post with 🔥 about 1 year ago
Post
3140
Finally my first solo preprint is here:) a love letter to the field. Nothing much lol, this is just me trying to finetune my understanding of research behind the recent breakthroughs in reasoning models. It’s a preprint targeting beginners in the field - will eventually make necessary changes later. In the meantime have fun with it:)
Download: https://github.com/Jaykef/Jaykef/blob/main/papers/The-Dawn-of-Thinking-Machines.pdf
Download: https://github.com/Jaykef/Jaykef/blob/main/papers/The-Dawn-of-Thinking-Machines.pdf
upvoted a paper almost 2 years ago
updated a
dataset about 3 years ago