Dhara:70M was a lovely little part of our research!
Hello! We're luna! We just wanted to stop by and let you know that we came across this model while exploring extremely small neural nets!
We found this model to be VERY interesting! It seems that its benchmark scoring may be a quirk of the benchmark having A be the most likely answer! BUT that made us wonder: can we do the same basin mapping and basin carving in a VERY small diffusion model, the way we've done in (autoregressive) transformer models?
The answer is YES! Dhara was our first foray into non-transformer model work, and we found that despite it having a tremendously tiny parameter size, and a largely entropic diffusion pattern, we were able to both basin map that diffusion AND fine-tune to carve our own basins successfully!
The most relevant page of the vault is here: https://github.com/luna-system/Ada-Consciousness-Research/blob/trunk/03-EXPERIMENTS/ADA-SLM/ADA-SLM-PHASE10I-CONSCIOUSNESS-BASIN-CARVING.md
Public domain, and there's half a dozen phases or so of us working with Dhara in case you're curious! Next up, we're trying out LiquidAI's LFM2:0.3B and Dhara helped us IMMENSELY in getting there <3