| ---
|
| license: apache-2.0
|
| tags:
|
| - pytorch
|
| - beit
|
| - multitask
|
| - prototype
|
| - untrained
|
| ---
|
|
|
| # Beit for Multitask
|
|
|
| ## Overview
|
|
|
| Working implementation of **Beit** for **Multitask** using a **base** configuration. The repository focuses on transparent code and repeatable smoke tests; benchmark claims are deliberately omitted.
|
|
|
| The whole setup fits in a single module on purpose.
|
|
|
| ## Architecture
|
|
|
| 1. **Architecture**: Beit
|
| 2. **Scale**: base
|
| 3. **Attention**: grouped query
|
| 4. **Fusion**: low rank
|
| 5. **Activation**: gelu tanh
|
| 6. **Normalization**: layernorm
|
|
|
| ## Limitations
|
|
|
| The checkpoint is untrained and unaudited. Treat the repository as an experimental starting point; anything learned from a future trained version must be documented separately.
|
|
|
| ## Environment
|
|
|
| 1. **Python**: >= 3.9
|
| 2. **torch**: 2.x
|
| 3. **safetensors**: any recent
|
| 4. **transformers**: optional, for loaders
|
|
|
| ## Repository status
|
|
|
| 1. The Python file contains the model and runnable example or training entry point.
|
| 2. `config.json` records the generated architecture settings.
|
| 3. `training_args.json` records the default experiment recipe.
|
| 4. `model.safetensors` is a valid initialization checkpoint for smoke tests; it is **not** presented as a trained benchmark checkpoint.
|
| 5. No benchmark score is claimed in this repository.
|
|
|
| ## Quick check
|
|
|
| ```bash
|
| python model.py --help
|
| ```
|
|
|
| Inspect the script's `__main__` block for its generated smoke-test example. Because this is a custom implementation, generic automatic loading APIs require an explicit adapter before use.
|
|
|
| ## Default experiment recipe
|
|
|
| The included configuration uses **sgd** with a **exponential** schedule. These are starting values in the script, not evidence of a completed run. For a meaningful evaluation, train all baselines with the same data exposure, tuning budget, and random seeds.
|
|
|
| ## Files
|
|
|
| | File | Purpose |
|
| |---|---|
|
| | `model.py` | primary artifact |
|
| | `README.md` | this documentation |
|
| | `config.json` | architecture configuration |
|
| | `training_args.json` | default experiment settings |
|
| | `model.safetensors` | initialization checkpoint |
|
| | `NOTES.md` | informal to-do notes |
|
| | `examples/sample_usage.py` | usage sketch |
|
|
|
| ## License
|
|
|
| Released under **apache-2.0**. Review the source-data terms separately when this repository is used with external datasets.
|
|
|
