--- license: apache-2.0 tags: - pytorch - efficientformer - multitask - wip - initialization --- # Efficientformer for Multitask ## Overview This is an experimental **Efficientformer** codebase for **Multitask**. It keeps the **tiny** setup intentionally manageable so architecture changes can be inspected before a full training run. --- ## Architecture - **Architecture**: Efficientformer - **Scale**: tiny - **Attention**: standard - **Fusion**: cross attention - **Activation**: relu - **Normalization**: batchnorm --- ## Training details - Weights initialized with a orthogonal scheme. - Runs on a single consumer GPU. - Full smoke test takes minutes, not hours. - No distributed setup required. --- ## Limitations The initialization checkpoint has not been trained or audited for robustness, fairness, or domain transfer. The implementation should be treated as an experimental starting point. Results from a future trained checkpoint must be documented separately from the defaults shipped here. --- ## Default experiment recipe The included configuration uses **lion** with a **linear warmup** schedule. These are starting values in the script, not evidence of a completed run. For a meaningful evaluation, train all baselines with the same data exposure, tuning budget, and random seeds. --- ## Quick check Run `python model.py --help` for the built-in smoke test. Because this is a custom implementation, generic automatic loading APIs require an explicit adapter before use. --- ## Roadmap Next steps, in no particular order: a cleaner entry point, input-shape docs, and a small evaluation script if time permits. --- ## Repository status One Python module contains the model and a runnable example. Generated configuration lives in `config.json`; the recipe in `training_args.json`. The checkpoint is an initialization artifact and should not be read as a trained model. Scores are deliberately absent. --- ## Files - `model.py` — primary artifact - `README.md` — this documentation - `config.json` — architecture configuration - `training_args.json` — default experiment settings - `model.safetensors` — initialization checkpoint --- ## License Released under **apache-2.0**. Review the source-data terms separately when this repository is used with external datasets.