---
datasets:
- inaturalist2019
language: en
tags:
- image-classification
- pytorch
- efficientnet
- mixture-of-experts
- deepmoe
---

# DeepMoE EfficientNet-B0 fine-tuned on iNaturalist 2019

This model is a Mixture-of-Experts (DeepMoE) variant of EfficientNet-B0, fine-tuned on the iNaturalist 2019 dataset to optimize both accuracy and computational efficiency (FLOP reduction).

## Training Results
- **Final Score (Acc/FLOPs composite)**: 83.4129
- **Final Validation Accuracy**: 69.1%
- **Expert Activation Ratio**: 32.0%
- **FLOPs Usage**: 57.2% *(compared to baseline B0)*
- **Baseline B0 Reference FLOPs**: 388,184,000
- **Total Runtime**: 5204.99 seconds

## Hyperparameters
- **Batch Size**: 256
- **Gradient Accumulation Steps**: 4
- **Weight Decay**: 0.005

### Epochs
- **Total Epochs**: 10
  - Joint Training Epochs: 10
  - Routing-Frozen Finetuning Epochs: 0

### DeepMoE Architecture & Routing
- **MoE Start Stage**: 1
- **Latent Dimension**: 32
- **Sparsity Penalty ($\lambda_g$)**: 7e-05
- **Target Sparsity ($\mu$)**: 0.5
- **ReLU Init (Val / Std)**: 1 / 1

### Learning Rates
- **MoE Routing Parameters**: 1.20e-01
- **Classification Head**: 2.00e-02
- **Base Model (Body)**: 2.00e-03
- **Finetune Phase (Frozen Routing)**: 0.00e+00

*Training was tracked using [Weights & Biases](https://wandb.ai).*