Teaching an LLM to Decompile the Nintendo 64
Fine-tuning language models to turn N64 MIPS assembly back into compilable C, and what the home lab vs. cloud reality actually looked like
A writeup of the N64LLMDecompile project (HuggingFace Build Small hackathon, ≤32B params).1. The premise
When a video game ships, its source code is compiled down into raw machine instructions and that original code is very rarely released to the public. For retro games it is possible that the code can be lost entirely as companies change. Decompilation is the painstaking work of reversing the compiled code from the finished project and reconstructing readable source code that recompiles into a byte-for-byte identical copy of the original game. This allows for better preservation, reveals programming design and decisions made for the game, and allows people to modify the game in more ways than simple mods could.
Decompilation is an extremely difficult task that is done by trial and error, and usually takes thousands of hours from individual contributors to reverse engineer even very small parts of the original code. If even a small part of this process can be automated it can dramatically speed up decompilation projects across almost any platform.
My goal was to make an LLM that could take assembly from compiled Nintendo 64 games, and return the reverse engineered C that was compiled to match that assembly perfectly. This C code could then be used to fully reverse engineer the game for any purpose. The model would then be published as a general purpose decompiler for all Nintendo 64 games.
2. The data
The sources of data I used came from the Decomp.me community. Decomp.me is a hub for decompilation projects hosting many examples of decompiled assembly code, called scratches as a single unit, with various levels of matching. I received this data directly from their public discord as a PostgreSQL database. I narrowed the data down to about ~23,400 "perfect" scratches. A scratch is perfect when it creates a binary that matches the original binary exactly after being compiled. Scores for a match are golf style, 0 is a byte-exact binary match, with the score going up for every difference. This score is derived from asm-differ which is publicly available tool made by Simon Lindholm on github.
Non-perfect scratches had to be removed, even if close to a good result, since they would teach the model wrong answers during the fine tuning process. The data had a 95/5 split between training and evaluation data. ~22,246 items in the training data and ~1,171 items in the validation set.
One of the complications with compiling the code is that different compilers give different results, so to ensure a match, you need to use the exact compiler and the same compiler flags as the original programmers used. Fortunately in game development the compiler list can get limited. In the case of the N64 there were 4 compilers that covered 93% of the data and so I focused my tool on those examples for the sake of completing the project in time for the hackathon. The compilers and their flags were noted with every scratch so I could always match them when trying to test the model. The 4 compilers I used were ido7.1, ido5.3, gcc2.7.2kmc, and gcc2.8.1pm which were sourced from decompals and the Paper Mario decompilation team on github respectively.
While the compilers and flags are critical data, the model does not actually see them during inference, and they are only used when recompiling the generated code. I considered making a model that might be able to guess the compiler and flags separately but this would be beyond the scope of the project. So as is, the user must supply this information, which is often the first piece of information found from decompilation projects anyway.
3. The eval philosophy (decided up front)
The metric I was aiming for was for maximum compilation, so that I could measure a difference in compiled code afterwards. Non-compiling code might be useful and close to the compiling code that matches the assembly, but I cannot feasibly fix thousands of lines of LLM generated code to measure how close the model was. The predicted C from the model would go into the matching compiler and diffed against the original ROM binary to produce a measurable score.
The reported metrics from my evaluations were compile-success rate, perfect-match rate, score distribution buckets, and % better than a naive empty-stub baseline.
4. Infrastructure set up and challenges
My first set of finetuning and evaluation runs were on my personal NVIDIA DGX Spark using vLLM in Docker. While this was convenient at first to run my inference it actually became an issue at the time of evaluation. The compilers for the Nintendo 64 only ran on x86 infrastructure and the Spark is on ARM, so I actually had to split my whole pipeline between my DGX Spark and my personal laptop. This part of the project was a nice highlight for how long inference takes and how fast evaluation can be. It took my Spark several hours to generate all the code from the evaluation set, but took my laptop less than a minute to compile it all.
The bigger problem was speed. Fine-tuning and inference runs on my personal setup could take more than 20 hours, and in a hackathon the iteration loop is everything. That was ok early on when we had a week, but it quickly became the main bottleneck on how many ideas I could actually try. So I moved fine-tuning to Modal on a rented H200. A training run that took roughly 22 hours on the Spark finished in about 2.5 hours on Modal, and it cost me around $12 in credits. That is roughly 8 times faster, and that single comparison reset how I thought about the whole workflow. From that point on I routed all serious training and time-pressured generation sweeps through Modal, often running jobs in parallel so I could try far more models than the Spark would have ever allowed.
This left me with a clear mental model for the two pieces of hardware. The Spark is my local sandbox. It handles always-on inference and agent hosting, dataset and eval pipeline work, anything privacy-sensitive, and it's where I learn ARM64 and vLLM. Modal is for serious training compute. The short version is rent for training, own for inference.
A few Modal-specific lessons were worth the pain of learning them. Always run long jobs with -d or --detach, because a plain modal run dies the moment your laptop disconnects. Git Bash mangles /data/... volume paths on Windows, so I learned to run those commands from PowerShell instead. Checkpointing every N steps with resume support meant that an OOM in the middle of a run cost me nothing. And trading micro-batch size for gradient accumulation killed the activation-spike OOMs without actually changing the optimization.
5. Baseline results (untuned 6.7b)
Before I started fine-tuning anything, I ran the stock LLM4Decompile-6.7b model, which was direct from assembly to C in its original training and operation, against my evaluation set to see where it stood. The results were rough. Out of 1,171 functions it only compiled about 45 of them, roughly 3.8%, with 5 perfect byte-exact matches. It did manage to beat the empty-stub baseline about 31.8% of the time, so it was doing something, just not much.
This established the floor for the project. The stock model was trained on x86, not N64 MIPS, so it was working on an architecture it had never really seen. The common failure modes were what you might expect from a model out of its depth. It hallucinated struct fields, referenced undefined globals, produced endless lists of variable declarations, and frequently ran away until it hit the token cap without ever finishing the function.
6. Fine-tuning v1 and the limits of the Spark
For my first fine-tune I used LoRA with rank 16 and alpha 32, running through Unsloth and TRL's SFTTrainer in bf16. One detail that mattered was appending an explicit EOS token to the training examples so the model could learn where to stop, which would become a recurring theme throughout the project.
Getting this running on the Spark was a fight. The python3.12-dev headers weren't available on the Spark's arm64 mirror, which led to Triton kernel failures. To get around it I had to move training into the NVIDIA PyTorch container, and then deal with a string of version conflicts between unsloth, trl, and transformers. This was my first real taste of how much friction the ARM64 environment would add to the whole project.
When the v1 fine-tune finished, the train loss had dropped to around 0.14, which looked great on paper. The problem was that the model barely produced compilable C, only about 16 out of 1,168 functions.
Compilation is all-or-nothing and a single wrong token breaks the entire code. A low finetune loss might mean the model gets the wrong token only 20% of the time, but 20% can be spread out through a whole file and thus make it very difficult to fix. When I went back to check my saved runs, the untuned baseline actually appears to have compiled more rows than this v1 fine-tune, which was a clear sign that the raw assembly setup was not going to work. That result is what pushed me toward the Ghidra pivot later on.
7. End vs. Ref: benchmarking both LLM4Decompile families
LLM4Decompile actually ships two different model families that share an identical prompt wrapper, and I made a point of running both to understand the space I was working in. The End family, which is v1.5, takes raw assembly and turns it into C. The Ref family, which is v2, refines Ghidra pseudo-code into C and never reads the raw assembly at all.
I started with the End models and then moved to the Ghidra-based Ref model specifically to measure the difference between the pseudo-code path and the raw assembly baseline. The head-to-head comparison was the whole point here, not a course correction after something failed.
Moving to Ref meant building out an entire Ghidra pipeline. I had to headless-decompile the N64 objects into pseudo-C, and then feed that pseudo-C to the model as input instead of the assembly.
For the model size I went with 9b-v2 over 22b-v2. On the published x86_64 benchmark the 9b model actually edges out the 22b, scoring 64.9% against 63.6%, and on top of that it's cheaper and faster to train. It was a good reminder that bigger isn't automatically better.
There was also a practical reason Ghidra is more tractable for a fixed context budget. The pseudo-code is compact, with a median around 450 tokens, while the raw assembly has a median closer to 1,000 tokens and a heavy tail of monster functions running past 40k tokens.
8. The Ghidra/v2 run, the better run
(LLM4Decompile-9b-v2, Ghidra pseudo-code input, fine-tuned on Modal)
The final training run for this model went 2,607 steps in about 2 hours and 34 minutes, landing at a train loss of 0.342 and 91.7% mean token accuracy across roughly 18.6 million training tokens.
On the eval, it compiled and scored 53 functions, 10 of which were byte-exact, and it beat the baseline 61.5% of the time. The way I like to summarize it is that the model has about a 4.6% compile rate, but of the functions that do compile, 18.9% are byte-exact and 61.5% beat the baseline. It lands rarely, but when it lands, it lands well.
The score distribution was bimodal, with one cluster at score 0 and another cluster at 500 or higher, and very little in between. When this model misses, it misses big. I actually consider that a more fixable failure mode than a model that produces consistent mush in the middle.
It's worth pointing out that this run had a higher loss than v1, 0.342 against 0.14, and yet it's clearly the better model. It makes the same point the v1 run did: loss isn't comparable across different bases and inputs, and only the shared compile eval can actually rank them.
9. A clean negative result: output budget was not the bottleneck
At this point I had a clean hypothesis to test. Around 246 of my outputs were getting truncated, and I assumed they were failing only because they hit the 4096 token cap. The obvious fix was to double the cap to 8192 and let the long functions finish.
The reality was a clean negative result. Doubling the cap dropped the truncations from 246 to just 237, and not a single one of the newly-finished functions compiled. The eval came out byte-identical to the 4096 run. Doubling the budget bought me nothing.
When I actually eyeballed the outputs that were still getting truncated, the real cause was obvious. The model wasn't running out of room on legitimate long functions, it was generating runaway, degenerate output. A common pattern was emitting struct fields at incrementing offsets forever and never emitting an EOS token to stop.
The reason for this comes back to how the model was trained. The v2 model was trained at 4096 tokens total, counting the prompt and the completion together. Generating anything past that point is out of distribution, so the model never actually learned how to land a long output. The takeaway is that the lever for long functions is a longer training context, or a base model built for long context, and not a bigger inference budget. I reverted the cap back to 4096.
10. Capability beat output budget: Nemotron-32B
For the next experiment I held almost everything constant. Same task, same validation set, same eval harness, same 4096 cap, and the same LoRA recipe. The only thing I changed was the base model, swapping in OpenCodeReasoning-Nemotron-1.1-32B, which is a general reasoning model rather than a purpose-built decompiler. I fine-tuned it on Modal's H200 just like the others.
| metric | 9B-v2 (Ghidra) | Nemotron-32B (Ghidra) |
|---|---|---|
| compiled & scored | 53 | 58 |
| perfect (byte-exact) | 10 | 14 |
| beat baseline | 61.5% | 68.4% |
| perfect / compiled | 18.9% | 24.1% |
Put next to the 8192 experiment, the technical story tells itself. Doubling the output budget bought me zero, while swapping in a stronger base bought me 4 more perfect matches and 5 more compiles. A general 32B reasoning model, once fine-tuned, out-decompiled the purpose-built 9B decompiler at its own task.
The catch is in the size. It performed great for a model that wasn't specialized for this at all, but despite the name, OpenCodeReasoning-Nemotron-1.1-32B actually comes in a little above 32B parameters. The hackathon had a hard 32B ceiling, so even being slightly over the line was enough to disqualify it as the practical deliverable, no matter how well it scored.
11. The Cascade-2 finding, and the model I couldn't fine-tune
Before committing to any more fine-tuning, I ran a zero-shot scout of the raw base models against the same eval, and it told two very different stories. The raw OCR-Nemotron-32B only compiled 3 functions with 0 perfect matches, and fine-tuning took it all the way to 58 compiles and 14 perfect. That's a huge relative jump, but it's a jump from a base that essentially couldn't do the task at all.
The raw Nemotron-Cascade-2-30B-A3B told the more interesting story. This is a mixture-of-experts model with only 3B active parameters, and with zero task training it compiled 45 functions with 5 perfect matches. That nearly matches my fine-tuned 9B-v2, straight out of the box.
Naturally I wanted to fine-tune Cascade-2 next, but its architecture got in the way. It's a hybrid Mamba-2 and MoE model. Instead of relying purely on attention, Mamba uses a state-space model that processes a sequence in linear time, carrying a compressed running state forward rather than paying attention's quadratic all-pairs cost. Those state-space layers are then interleaved with mixture-of-experts and transformer blocks.
That hybrid design breaks the standard LoRA recipe in a few ways. The custom Mamba CUDA kernels have to build first, and they failed to compile on me twice. The LoRA target-module names don't match the usual attention q/k/v/o and FFN list that the tooling expects, and on top of that you have to be careful not to adapt the MoE router. Between that learning curve and the hackathon clock running down, I never got the chance to figure out how to fine-tune it properly. It's a high-upside stretch goal that I had to leave on the table.
12. Lessons worth keeping
Looking back across the whole project, a handful of lessons stand out that I'll carry into future work.
- Training loss is not definitive. Training loss can be encouraging during the fine-tuning phase but the actual performance of the model may not be captured in it depending on the task.
- Filter your training targets, don't truncate them. Cutting an over-length example down to fit just teaches the model to emit half a function and then stop, which causes unnecessary failures.
- Input and output share one context budget. That 4096 number is the prompt and the completion together, not each of them separately. Managing your context requires awareness of the input as much as it does on output.
- ARM64 is real production-adjacent experience, and a real source of friction. Native IDO, the x86-only GCC, the cross-machine eval split, and the missing arm64 Python headers all cost me time.
- Rent for training, own for inference. The Spark earns its keep on capacity-bound, privacy-sensitive, and always-on work, while the cloud wins anything that's bound by speed or scale.
- Measure before choosing a context size instead of guessing. If I had checked the token distribution of my outputs first, the 8192 experiment would have told me upfront that it wasn't going to help.
13. Conclusion
My fine-tuned models did not work well for solving the problem of decompilation, and fine-tuning, Ghidra or no Ghidra, honestly only made a small difference in practice. The perfect matches the model would get were usually on very small functions, empty functions, or functions that called lots of other functions. The imperfect matches are hard to measure: small differences in code can be significant changes in the binary that the matching score is focused on. Non-compiling code might be close to the final result, but it isn't feasible for me to search through my evaluation data for easily fixable code to see what might end up close to the original binary when compiled.
Ultimately I was very optimistic in regards to how much LLMs can do to tackle this very niche and very difficult problem. The dataset from decomp.me is definitely a wealth of information, but harnessing it will require more research and a much deeper understanding of the decompilation process. The results from the untuned Cascade model were very promising. Despite not having any training on the Ghidra and C pairs, it managed to make almost as many compilable outputs as the fine-tuned model. I am doubtful that my simple SFT run would be enough to turn that model into a breakthrough tool for the field, but I think it shows that more intelligent, more powerful models likely will eventually make breakthroughs in this problem.
So while my own naive SFT approach landed with lackluster results, the jump I saw from the stronger bases tells me the potential is real. The direction I would take next is reinforcement learning rather than plain supervised fine-tuning. The model would generate C, feed it straight into the compiler, and learn from the compiler's feedback as it tries to land a match, getting rewarded for output that compiles and moves closer to a byte-exact diff. That loop is much closer to how a human actually does decompilation: write something, compile it, study the diff, and adjust. Paired with a newer and more capable base model, I think that kind of feedback-driven training has a real shot at the results that supervised fine-tuning alone could not reach.
Stack: vLLM, Unsloth, TRL, PEFT/LoRA, Ghidra, Modal, asyncio/aiohttp, asm-differ.
Special thanks
To every team whose work this project stands on:
- decomp.me, for the community database of hand-matched N64 scratches that made the whole thing possible, and for packaging the compiler toolchains.
- The LLM4Decompile authors (albertan017), for the End and Ref model families this project was built on, and for publishing the benchmarks that guided my model choices.
- The Paper Mario decompilation team (pmret), for the GCC 2.8.1 compiler I leaned on in the eval pipeline.
- decompals, for the IDO static recompilation and MIPS GCC/binutils builds the rest of the eval pipeline runs on.
- The Modal team, for the cloud GPU infrastructure (and credits) that turned 22-hour training runs into ~2.5-hour ones.
- NVIDIA, for providing the Nemotron models (and the prize category that pushed me to try them).
- OpenAI, for some Codex use along the way.
- HuggingFace, for hosting the Build Small hackathon and providing ZeroGPU access.