How to Build a Benchmark with a Private Test Set on Hugging Face
•
5
AI & ML interests
None defined yet.
Recent Activity
Articles
Post
2635
🚀 I am thrilled to announce the release of a new Konkani LLM!
We've seen some fantastic results for both translation and transliteration tasks, and I'm excited to share this progress with the community.
📖 Read the launch article and see the results: https://huggingface.co/blog/Reubencf/konkani-llm
🤖 Explore the model and collection:
konkani
I would love to hear your feedback or see what you build with it! #Konkani #LLM #NLP #HuggingFace #IndicNLP #Konkani
We've seen some fantastic results for both translation and transliteration tasks, and I'm excited to share this progress with the community.
📖 Read the launch article and see the results: https://huggingface.co/blog/Reubencf/konkani-llm
🤖 Explore the model and collection:
I would love to hear your feedback or see what you build with it! #Konkani #LLM #NLP #HuggingFace #IndicNLP #Konkani
gabboud
updated a Space 8 days ago
gabboud
updated 3 Spaces 9 days ago
gabboud
updated a Space 10 days ago
gabboud
published a Space 10 days ago
hist0613
authored 10 papers 11 days ago
FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmark
Paper
• 2409.19014 • Published
Towards Fully-Automated Materials Discovery via Large-Scale Synthesis Dataset and Expert-Level LLM-as-a-Judge
Paper
• 2502.16457 • Published
• 12
TextME: Bridging Unseen Modalities Through Text Descriptions
Paper
• 2602.03098 • Published
Overthinking Loops in Agents: A Structural Risk via MCP Tools
Paper
• 2602.14798 • Published
• 1
ScoreCL: Augmentation-Adaptive Contrastive Learning via Score-Matching Function
Paper
• 2306.04175 • Published
Interventional Speech Noise Injection for ASR Generalizable Spoken Language Understanding
Paper
• 2410.15609 • Published
Multi-Architecture Multi-Expert Diffusion Models
Paper
• 2306.04990 • Published
Addressing Cold Start Problem for End-to-end Automatic Speech Scoring
Paper
• 2306.14310 • Published
Cross Encoding as Augmentation: Towards Effective Educational Text Classification
Paper
• 2305.18977 • Published
Evaluation of Question Generation Needs More References
Paper
• 2305.16626 • Published
hannayukhymenko
submitted a paper to Daily Papers 13 days ago
hannayukhymenko
posted an
update 13 days ago
Post
1917
Do you translate your benchmarks from English correctly? 🤔
Turns out, for many languages it is much harder than you can imagine!
Introducing Recovered in Translation 🌍 together with @aalexandrov
https://ritranslation.insait.ai
Translating benchmarks is a painful process, requiring a lot of manual inspection and adjustments. You start from setting up the whole pipeline and adapting to every format type, including task specifics. There already exist some massive benchmarks, but they still have some simple (and sometimes silly) bugs, which can hurt the evaluations :( We present a novel automated translation framework to help with that!
Eastern and Southern European languages introduce richer linguistic structures compared to English and for benchmarks which heavily rely on grammatical coherence machine translation presents a risk of harming evaluations. We discover potential answer leakage or misleading through grammatical structure of the questions. Some benchmarks are also just outdated and need to be retranslated with newer and better models.
We present a framework with novel test-time scaling methods which allow to control time and cost investments, while at the same time mitigate the need for human-in-the-loop verification. While working on Ukrainian-focused MamayLM models, we had to translate 10+ benchmarks in a short span of time. Finding human evaluators is costly and time-consuming, same goes for using professional translators. With our pipeline we were able to do it in 3 days🏎️
We hope our findings will help enable stronger multilingual evaluations and developments. We release all produced benchmarks on Hugging Face together with the source code and Arxiv paper 🤗
Paper: Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets (2602.22207)
Code: https://github.com/insait-institute/ritranslation
Benchmarks: https://huggingface.co/collections/INSAIT-Institute/multilingual-benchmarks
Turns out, for many languages it is much harder than you can imagine!
Introducing Recovered in Translation 🌍 together with @aalexandrov
https://ritranslation.insait.ai
Translating benchmarks is a painful process, requiring a lot of manual inspection and adjustments. You start from setting up the whole pipeline and adapting to every format type, including task specifics. There already exist some massive benchmarks, but they still have some simple (and sometimes silly) bugs, which can hurt the evaluations :( We present a novel automated translation framework to help with that!
Eastern and Southern European languages introduce richer linguistic structures compared to English and for benchmarks which heavily rely on grammatical coherence machine translation presents a risk of harming evaluations. We discover potential answer leakage or misleading through grammatical structure of the questions. Some benchmarks are also just outdated and need to be retranslated with newer and better models.
We present a framework with novel test-time scaling methods which allow to control time and cost investments, while at the same time mitigate the need for human-in-the-loop verification. While working on Ukrainian-focused MamayLM models, we had to translate 10+ benchmarks in a short span of time. Finding human evaluators is costly and time-consuming, same goes for using professional translators. With our pipeline we were able to do it in 3 days🏎️
We hope our findings will help enable stronger multilingual evaluations and developments. We release all produced benchmarks on Hugging Face together with the source code and Arxiv paper 🤗
Paper: Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets (2602.22207)
Code: https://github.com/insait-institute/ritranslation
Benchmarks: https://huggingface.co/collections/INSAIT-Institute/multilingual-benchmarks
- 1 reply
gabboud
published a Space 15 days ago