AMD AI DevMaster Hackathon
β’ 3
AI & ML interests
None defined yet.
Recent Activity
View all activity
Articles
codelionΒ
authored 4
papers 19 days ago
An Empirical Study of Path Feasibility Queries
Paper β’ 1302.4798 β’ Published
The Dynamics of Software Composition Analysis
Paper β’ 1909.00973 β’ Published β’ 1
End to End Verification and Validation with SPIN
Paper β’ 1302.4796 β’ Published
Automated Verification of Integer Overflow
Paper β’ 1909.09324 β’ Published β’ 1
charyanghfΒ
published an article 26 days ago
Article
AMD AI DevMaster Hackathon
LeRobot-worldwide-hackathon
β’ β’ 3Post
485
SPROG-9M β a 9.37M parameter model trained from scratch to solve GSM8K-style math without using an LLM at inference.
The model, codelion/sprog-9m, predicts symbolic programs over number slots, then a deterministic executor does the arithmetic. With a simple verifier, it reaches ~11.8% on GSM8K test.
We also released the dataset: codelion/gsm8k-synth, 117K validated synthetic GSM8K-style problems.
Tiny model, no pretraining, no LLM at inference, runs on a laptop.
The model, codelion/sprog-9m, predicts symbolic programs over number slots, then a deterministic executor does the arithmetic. With a simple verifier, it reaches ~11.8% on GSM8K test.
We also released the dataset: codelion/gsm8k-synth, 117K validated synthetic GSM8K-style problems.
Tiny model, no pretraining, no LLM at inference, runs on a laptop.
lilkmΒ
published a
dataset 2 months ago
ronantakizawaΒ
posted an update 2 months ago
Post
195
MIDI datasets are rare to find, so I made a compilation of them here!
ronantakizawa/sampleflip-midi
ronantakizawa/popsongs-segments-midi
#music #midi #dataset
ronantakizawa/sampleflip-midi
ronantakizawa/popsongs-segments-midi
#music #midi #dataset
Post
3257
Inspired by the Nemotron Diffusion recipe, check out dhara-250m: a 250M experimental language model that supports three decoding modes from one set of weights: autoregressive, block-diffusion, and self-speculation.
It is small, easy to try, and meant for exploring diffusion-style decoding and latency tradeoffs in compact LMs.
Model: codelion/dhara-250m
Try the chat demo here: codelion/dhara-chat
It is small, easy to try, and meant for exploring diffusion-style decoding and latency tradeoffs in compact LMs.
Model: codelion/dhara-250m
Try the chat demo here: codelion/dhara-chat
- 3 replies
Post
3439
Scaling Pedagogical Pre-training to 10 Billion Tokens
New blog post exploring what happens when you take optimal data mixing insights and scale up the data generation itself.
We built Sutra, a multi-stage framework for generating pedagogical pre-training data guided by a knowledge graph of ~2,000 concepts across 9 domains. The pipeline includes structured content generation, six-dimension quality evaluation, diversity management across 20 content styles, and a cleaning stage to prevent collapse.
The result is codelion/sutra-10B, a 10.2 billion token pedagogical dataset with rich metadata (domain, complexity, prerequisites, quality scores) on every entry.
We trained codelion/SmolLM2-70M on it for 3 full epochs (30.6B tokens) on a single A10 GPU in ~78 hours.
Key finding: perplexity kept improving across epochs, but benchmark gains plateaued fast. At 70M parameters, the model hits a representational ceiling that more data alone can't break through.
Full writeup with comparisons against 7 other datasets, detailed benchmark breakdowns, and connections to recent work on synthetic data scaling, curriculum learning, and data mixing laws: https://huggingface.co/blog/codelion/scaling-pedagogical-pretraining-10-billion-tokens
All datasets at multiple scales (10M, 100M, 1B, 10B) plus seed concepts and an SFT variant are in the Sutra Pedagogical Datasets collection.
New blog post exploring what happens when you take optimal data mixing insights and scale up the data generation itself.
We built Sutra, a multi-stage framework for generating pedagogical pre-training data guided by a knowledge graph of ~2,000 concepts across 9 domains. The pipeline includes structured content generation, six-dimension quality evaluation, diversity management across 20 content styles, and a cleaning stage to prevent collapse.
The result is codelion/sutra-10B, a 10.2 billion token pedagogical dataset with rich metadata (domain, complexity, prerequisites, quality scores) on every entry.
We trained codelion/SmolLM2-70M on it for 3 full epochs (30.6B tokens) on a single A10 GPU in ~78 hours.
Key finding: perplexity kept improving across epochs, but benchmark gains plateaued fast. At 70M parameters, the model hits a representational ceiling that more data alone can't break through.
Full writeup with comparisons against 7 other datasets, detailed benchmark breakdowns, and connections to recent work on synthetic data scaling, curriculum learning, and data mixing laws: https://huggingface.co/blog/codelion/scaling-pedagogical-pretraining-10-billion-tokens
All datasets at multiple scales (10M, 100M, 1B, 10B) plus seed concepts and an SFT variant are in the Sutra Pedagogical Datasets collection.
- 2 replies
ronantakizawaΒ
posted an update 5 months ago
Post
2861
Introducing the github-codereview dataset: A compilation of 200k+ human-written code reviews from top OSS projects (React, Tensorflow, VSCode...).
I finetuned a Qwen2.5-Coder-32B-Instruct model with this dataset and saw significant improvements in generating better code fixes and review comments (4x improved BLEU-4, ROUGE-L, SBERT scores compared to base model).
#codereview #code #datasets
ronantakizawa/github-codereview
I finetuned a Qwen2.5-Coder-32B-Instruct model with this dataset and saw significant improvements in generating better code fixes and review comments (4x improved BLEU-4, ROUGE-L, SBERT scores compared to base model).
#codereview #code #datasets
ronantakizawa/github-codereview
ronantakizawaΒ
posted an update 5 months ago
Post
2528
Introducing the WebUI dataset: a compilation of screenshot to code pairs of modern websites detailing the styling, framework used, and box bounds for all viewports (Desktop, mobile, tablet).
This dataset showed signs of improved performance in web design LLM benchmarks for a finetuned QWEN 2.5 VL-7B!
#web #ui #datasets
ronantakizawa/webui
This dataset showed signs of improved performance in web design LLM benchmarks for a finetuned QWEN 2.5 VL-7B!
#web #ui #datasets
ronantakizawa/webui
- 3 replies
ronantakizawaΒ
posted an update 6 months ago
Post
2124
Introducing the github-top-code dataset: A curated dataset of 1.3M+ source code files from GitHub's top ranked developers.
I collected the best source code files from Github's highest trending developers of all time, and compiled a dataset to train LLMs to write well-structured, production-grade code.
#dataset #codedataset #pretraining
ronantakizawa/github-top-code
I collected the best source code files from Github's highest trending developers of all time, and compiled a dataset to train LLMs to write well-structured, production-grade code.
#dataset #codedataset #pretraining
ronantakizawa/github-top-code
ronantakizawaΒ
posted an update 6 months ago
Post
291
Introducing the LeetCode Assembly Dataset: a dataset of 400+ LeetCode problem solutions in assembly across x86-64, ARM64, MIPS64, and RISC-V using GCC & Clang at -O0/-O1/-O2/-O3 optimizations.
This dataset is perfect for teaching LLMs complex compiler behavior!
#dataset #leetcode #assembly
ronantakizawa/leetcode-assembly
This dataset is perfect for teaching LLMs complex compiler behavior!
#dataset #leetcode #assembly
ronantakizawa/leetcode-assembly
ronantakizawaΒ
posted an update 6 months ago
Post
246
Hit 10,000+ downloads across my models and datasets on Hugging Face!
Follow for more @ronantakizawa !
#building #datasets #huggingface
Follow for more @ronantakizawa !
#building #datasets #huggingface
ronantakizawaΒ
posted an update 6 months ago
Post
2677
Moltbook, a Reddit platform only for AI agents, is going viral right now as agents are acting unhinged!
I compiled a dataset of all posts and subreddits in Moltbook so far so anyone can easily analyze the activity in Moltbook.
ronantakizawa/moltbook
#moltbook #clawd #aiagent
I compiled a dataset of all posts and subreddits in Moltbook so far so anyone can easily analyze the activity in Moltbook.
ronantakizawa/moltbook
#moltbook #clawd #aiagent
- 2 replies
Post
3294
Reverse Engineering a $500M Mystery: From HashHop to Memory-Augmented Language Models
I wrote a deep dive into how Magic AI's 100M token context window might work, starting from their HashHop benchmark and building up to MALM - a Memory-Augmented Language Model.
Key insight: treating each key as a single token enables perfect retrieval at unlimited context lengths.
The article covers:
- How HashHop works and why its perfect accuracy is suspicious
- Building a tokenized solver that achieves 100% accuracy
- Scaling to MALM for real code search tasks
- Why this approach could handle 100M+ tokens
Read the full article: https://huggingface.co/blog/codelion/reverse-engineering-magic-hashhop
Try the model: codelion/malm-165m
Code: https://github.com/codelion/hash-hop
I wrote a deep dive into how Magic AI's 100M token context window might work, starting from their HashHop benchmark and building up to MALM - a Memory-Augmented Language Model.
Key insight: treating each key as a single token enables perfect retrieval at unlimited context lengths.
The article covers:
- How HashHop works and why its perfect accuracy is suspicious
- Building a tokenized solver that achieves 100% accuracy
- Scaling to MALM for real code search tasks
- Why this approach could handle 100M+ tokens
Read the full article: https://huggingface.co/blog/codelion/reverse-engineering-magic-hashhop
Try the model: codelion/malm-165m
Code: https://github.com/codelion/hash-hop
- 1 reply
ronantakizawaΒ
posted an update 7 months ago
Post
491
Introducing the HuggingFace Top Trending Papers dataset: a dataset that compiles the most trending papers on HuggingFace Daily Papers in 2025.
This dataset captures which AI/ML research papers gained the most community attention this year!
#huggingface #papers #dataset
ronantakizawa/huggingface-top-papers
This dataset captures which AI/ML research papers gained the most community attention this year!
#huggingface #papers #dataset
ronantakizawa/huggingface-top-papers
Post
6199
Introducing Dhara-70M: A diffusion language model that achieves 3.8x higher throughput than autoregressive models!
Key findings from our research on optimal architectures for small language models:
β Depth beats width: 32 layers outperforms 12 layers at the same parameter count
β Best-in-class factuality: 47.5% on TruthfulQA
β 10x training efficiency using WSD (Warmup-Stable-Decay) conversion
β Canon layers add only 0.13% parameters but improve reasoning
We trained on 1B tokens using the optimal 50-30-20 dataset mix (PDFs + filtered web + educational content), then converted to diffusion with just 100M additional tokens.
Blog: https://huggingface.co/blog/codelion/optimal-model-architecture
Model: codelion/dhara-70m
Key findings from our research on optimal architectures for small language models:
β Depth beats width: 32 layers outperforms 12 layers at the same parameter count
β Best-in-class factuality: 47.5% on TruthfulQA
β 10x training efficiency using WSD (Warmup-Stable-Decay) conversion
β Canon layers add only 0.13% parameters but improve reasoning
We trained on 1B tokens using the optimal 50-30-20 dataset mix (PDFs + filtered web + educational content), then converted to diffusion with just 100M additional tokens.
Blog: https://huggingface.co/blog/codelion/optimal-model-architecture
Model: codelion/dhara-70m
- 1 reply
ronantakizawaΒ
posted an update 8 months ago
Post
5654
Thank you @clem (Co-Founder & CEO of Hugging Face) for sharing my dataset on X / Twitter!
ronantakizawa/github-top-developers
#github #dataset
ronantakizawa/github-top-developers
#github #dataset
- 4 replies