| ---
|
| license: cc-by-4.0
|
| tags:
|
| - pytorch
|
| - albef
|
| - multitask
|
| ---
|
|
|
| # Working Albef implementation for Multitask
|
|
|
| ## Overview
|
|
|
| A small **Albef** implementation for **Multitask**, packaged with an explicit configuration and an initialization checkpoint. The **base** variant is a reproducible starting point, not a trained model release.
|
|
|
| ---
|
|
|
| ## Quick Check
|
|
|
| Run `python pipeline.py --help` for the built-in smoke test. Because this is a custom implementation, generic automatic loading APIs require an explicit adapter before use.
|
|
|
| ---
|
|
|
| ## Contact
|
|
|
| Questions or corrections: open a discussion in this repository.
|
|
|
| ---
|
|
|
| ## Limitations
|
|
|
| The checkpoint is untrained and unaudited. Treat the repository as an experimental starting point; anything learned from a future trained version must be documented separately.
|
|
|
| ---
|
|
|
| ## Repository Status
|
|
|
| 1. Single-file implementation; the `__main__` block runs a smoke test.
|
| 2. `config.json` and `training_args.json` hold the generated settings.
|
| 3. `model.safetensors` is freshly initialized — plumbing, not results.
|
| 4. No benchmark numbers are reported on purpose.
|
|
|
| ---
|
|
|
| ## Training Details
|
|
|
| 1. **Framework**: PyTorch
|
| 2. **Precision**: fp32
|
| 3. **Init scheme**: xavier
|
| 4. **Hardware**: a single consumer GPU is enough
|
| 5. **Typical run**: minutes, not hours
|
|
|
| ---
|
|
|
| ## Architecture
|
|
|
| 1. **Architecture**: Albef
|
| 2. **Scale**: base
|
| 3. **Attention**: linear
|
| 4. **Fusion**: tucker
|
| 5. **Activation**: gelu tanh
|
| 6. **Normalization**: groupnorm
|
|
|
| ---
|
|
|
| ## Default Experiment Recipe
|
|
|
| The generated recipe uses lion with a constant warmup schedule.
|
|
|
| These values exist so the script runs end-to-end. A meaningful experiment needs its own tuning budget and at least three seeds.
|
|
|
| ---
|
|
|
| ## Files
|
|
|
| 1. `pipeline.py` — primary artifact
|
| 2. `README.md` — this documentation
|
| 3. `config.json` — architecture configuration
|
| 4. `training_args.json` — default experiment settings
|
| 5. `model.safetensors` — initialization checkpoint
|
|
|
| ---
|
|
|
| ## License
|
|
|
| Distributed under the **cc-by-4.0** license. External datasets keep their own terms — check them separately.
|
|
|
