AI & ML interests

None defined yet.

Recent Activity

FlameF0X submitted a paper about 1 month ago

Triplet-Block Diffusion RWKV

FlameF0X new activity 3 months ago

i3-lab/i3-200m-v2:Report

FlameF0X updated a Space 5 months ago

i3-lab/i3-GPT

View all activity

FlameF0X

posted an update 4 days ago

Post

101

Hello, people of Hugging Face!

I recently released FlameF0X/TinyMoE-100m-2x8-retrained, a small Mixture of Experts language model trained on the Smollm-Corpus. Built on top of the Mixtral architecture, it’s fully compatible with 🤗 Transformers right out of the box!

The model can produce somewhat coherent text on its own, and for some reason, it generates even more coherent responses when given a ChatLM template.

I’m excited to see what you all come up with, and feel free to fine-tune it if you’d like. In the meantime, I’ll be working on developing the chat-trained version.

Demo: FlameF0X/TinyMoE-Playground
Collection: https://huggingface.co/collections/FlameF0X/tinymoe

FlameF0X

posted an update about 1 month ago

Post

273

My models on the Intel Low-Bit LLM Leaderboard

Figured I'd share where my quantized models landed on Intel/low_bit_open_llm_leaderboard since I hadn't posted about it yet.

FlameF0X/Qwen3-4B-Distilled-Claude-4.6 (NVFP4 and MXFP4) sit at ranks 23 and 24 with 62.68% and 61.18% average, right below the base Qwen3-4B. Not bad considering they were distilled from Claude 4.6 rather than trained from scratch.

FlameF0X/LFM2.5-1.2B-Distilled-Claude-4.6 and FlameF0X/LFM2.5-1.2B-Thinking-CodeX land around rank 47-49, competitive with MiniCPM5-1B and the Qwen3 sub-1B models despite being a larger base architecture.

The funny one is FlameF0X/Qwen2-0.2B-pt and FlameF0X/Qwen2-0.2B-it. They're not properly trained — genuinely undertrained, basically undefined — and they still beat openai/gpt-oss-20b at rank 66. The 20B model. Not sure what that says but it's something.

FlameF0X/LFM2-Research is at the bottom of my lineup but it's a research artifact, not meant to be competitive.

Chart below showing my models vs nearby competitors, with size vs performance on the left.

Chart made by Claude

1 reply

FlameF0X

posted an update about 1 month ago

Post

7230

MiniMax-M3 coming soon.
https://github.com/MiniMax-AI/MiniMax-M3

FlameF0X

submitted a paper to Daily Papers about 1 month ago

Triplet-Block Diffusion RWKV

Paper • 2605.25969 • Published May 25 • 25

FlameF0X

posted an update about 2 months ago

Post

288

I did some testing on the scalability of FWKV. It hits a speed bottleneck at 1B due to the T4’s bandwidth limitations. Theoretically, it should match RWKV’s inference speed if the GPU had more bandwidth. So the 1B size is not accurate.

FlameF0X

posted an update about 2 months ago

Post

278

Greetings Hugging Face!

I started a new project called **FWKV** (Feed-forward Weighted Key Value, or Floored Weighted Key Value), a RWKV-style LM that uses FFNNs (Feed-Forward Neural Networks) instead of RNN and floor(W·K·V). I'm hoping to make it much more efficient and scalable than RWKV.

So far I have:

- FlameF0X/FWKV-29M — this one is undertrained and doesn't have a Space yet. In the attached image you can see its speed on a T4 compared to models with the same configuration.

The only model that's fully working right now is:
- FlameF0X/FWKV-TinyStories — trained on TinyStories for one epoch. The demo Space is FlameF0X/FWKV-demo.