File size: 2,390 Bytes
84ffcf8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 | ---
license: apache-2.0
tags:
- pytorch
- beit
- multitask
- prototype
- untrained
---
# Beit for Multitask
## Overview
Working implementation of **Beit** for **Multitask** using a **base** configuration. The repository focuses on transparent code and repeatable smoke tests; benchmark claims are deliberately omitted.
The whole setup fits in a single module on purpose.
## Architecture
1. **Architecture**: Beit
2. **Scale**: base
3. **Attention**: grouped query
4. **Fusion**: low rank
5. **Activation**: gelu tanh
6. **Normalization**: layernorm
## Limitations
The checkpoint is untrained and unaudited. Treat the repository as an experimental starting point; anything learned from a future trained version must be documented separately.
## Environment
1. **Python**: >= 3.9
2. **torch**: 2.x
3. **safetensors**: any recent
4. **transformers**: optional, for loaders
## Repository status
1. The Python file contains the model and runnable example or training entry point.
2. `config.json` records the generated architecture settings.
3. `training_args.json` records the default experiment recipe.
4. `model.safetensors` is a valid initialization checkpoint for smoke tests; it is **not** presented as a trained benchmark checkpoint.
5. No benchmark score is claimed in this repository.
## Quick check
```bash
python model.py --help
```
Inspect the script's `__main__` block for its generated smoke-test example. Because this is a custom implementation, generic automatic loading APIs require an explicit adapter before use.
## Default experiment recipe
The included configuration uses **sgd** with a **exponential** schedule. These are starting values in the script, not evidence of a completed run. For a meaningful evaluation, train all baselines with the same data exposure, tuning budget, and random seeds.
## Files
| File | Purpose |
|---|---|
| `model.py` | primary artifact |
| `README.md` | this documentation |
| `config.json` | architecture configuration |
| `training_args.json` | default experiment settings |
| `model.safetensors` | initialization checkpoint |
| `NOTES.md` | informal to-do notes |
| `examples/sample_usage.py` | usage sketch |
## License
Released under **apache-2.0**. Review the source-data terms separately when this repository is used with external datasets.
|