KatieHg74's picture
Add model.py, README.md, config.json, training_args.json, model.safetensors
2cb2e8e verified
|
Raw History Blame Contribute Delete
2.41 kB
---
license: apache-2.0
tags:
- pytorch
- efficientformer
- multitask
- wip
- initialization
---
# Efficientformer for Multitask
## Overview
This is an experimental **Efficientformer** codebase for **Multitask**. It keeps the **tiny** setup intentionally manageable so architecture changes can be inspected before a full training run.
---
## Architecture
- **Architecture**: Efficientformer
- **Scale**: tiny
- **Attention**: standard
- **Fusion**: cross attention
- **Activation**: relu
- **Normalization**: batchnorm
---
## Training details
- Weights initialized with a orthogonal scheme.
- Runs on a single consumer GPU.
- Full smoke test takes minutes, not hours.
- No distributed setup required.
---
## Limitations
The initialization checkpoint has not been trained or audited for robustness, fairness, or domain transfer. The implementation should be treated as an experimental starting point. Results from a future trained checkpoint must be documented separately from the defaults shipped here.
---
## Default experiment recipe
The included configuration uses **lion** with a **linear warmup** schedule. These are starting values in the script, not evidence of a completed run. For a meaningful evaluation, train all baselines with the same data exposure, tuning budget, and random seeds.
---
## Quick check
Run `python model.py --help` for the built-in smoke test. Because this is a custom implementation, generic automatic loading APIs require an explicit adapter before use.
---
## Roadmap
Next steps, in no particular order: a cleaner entry point, input-shape docs, and a small evaluation script if time permits.
---
## Repository status
One Python module contains the model and a runnable example.
Generated configuration lives in `config.json`; the recipe in `training_args.json`.
The checkpoint is an initialization artifact and should not be read as a trained model.
Scores are deliberately absent.
---
## Files
- `model.py` — primary artifact
- `README.md` — this documentation
- `config.json` — architecture configuration
- `training_args.json` — default experiment settings
- `model.safetensors` — initialization checkpoint
---
## License
Released under **apache-2.0**. Review the source-data terms separately when this repository is used with external datasets.