Add eval.py, README.md, config.json, training_args.json, model.safetensors, sample_usage.py
a32891a verified | license: bsd-2-clause | |
| tags: | |
| - pytorch | |
| - vit | |
| - generation | |
| - wip | |
| # ViT experiments: Generation | |
| ## Overview | |
| Working implementation of **ViT** for **Generation** using a **small** configuration. The repository focuses on transparent code and repeatable smoke tests; benchmark claims are deliberately omitted. | |
| --- | |
| ## Architecture | |
| 1. **Architecture**: ViT | |
| 2. **Scale**: small | |
| 3. **Attention**: multi query | |
| 4. **Fusion**: gated fusion | |
| 5. **Activation**: swish | |
| 6. **Normalization**: rmsnorm | |
| --- | |
| ## Repository Status | |
| 1. The Python file contains the model and runnable example or training entry point. | |
| 2. `config.json` records the generated architecture settings. | |
| 3. `training_args.json` records the default experiment recipe. | |
| 4. `model.safetensors` is a valid initialization checkpoint for smoke tests; it is **not** presented as a trained benchmark checkpoint. | |
| 5. No benchmark score is claimed in this repository. | |
| --- | |
| ## Quick Check | |
| Run `python eval.py --help` for the built-in smoke test. Because this is a custom implementation, generic automatic loading APIs require an explicit adapter before use. | |
| --- | |
| ## Limitations | |
| The initialization checkpoint has not been trained or audited for robustness, fairness, or domain transfer. The implementation should be treated as an experimental starting point. Results from a future trained checkpoint must be documented separately from the defaults shipped here. | |
| --- | |
| ## Default Experiment Recipe | |
| Defaults: **rmsprop** optimizer, **onecycle** schedule. Treat these as starting points; a real run should sweep learning rate and batch size before drawing conclusions. | |
| --- | |
| ## Training Details | |
| 1. Weights initialized with a kaiming scheme. | |
| 2. Runs on a single consumer GPU. | |
| 3. Full smoke test takes minutes, not hours. | |
| 4. No distributed setup required. | |
| --- | |
| ## Files | |
| | File | Purpose | | |
| |---|---| | |
| | `eval.py` | primary artifact | | |
| | `README.md` | this documentation | | |
| | `config.json` | architecture configuration | | |
| | `training_args.json` | default experiment settings | | |
| | `model.safetensors` | initialization checkpoint | | |
| | `examples/sample_usage.py` | usage sketch | | |
| --- | |
| ## License | |
| Released under **bsd-2-clause**. Review the source-data terms separately when this repository is used with external datasets. | |