| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - mistral |
| - fp32 |
| - adamw |
| - transformer |
| - monte-carlo |
| - dit |
| - ernie |
| pipeline_tag: text-to-image |
| --- |
| |
| # **Model Card** |
|
|
| # **Overview** |
|
|
| This repository documents two separate large language model training methodologies and precision strategies: |
|
|
| --- |
|
|
| # **Mistral LLM Training** |
|
|
| - **Fully trained in native FP32 precision** |
| - Optimization performed using standard **AdamW** |
| - **No Adam8bit**, quantized optimizer states, or reduced-precision optimizer approximations were used during training |
| - Intended to preserve **numerical stability** and **high-fidelity gradient accumulation** throughout all training phases |
|
|
| --- |
|
|
| # **DIT Ernie Model** |
|
|
| - Uses a **Monte Carlo estimation** approach to approximate **FP32 behavior** |
|
|
| --- |
|
|
| # **Training Details** |
|
|
| # **Mistral LLM** |
|
|
| ## **Precision** |
|
|
| - **Full FP32 training** |
| - **FP32 activations** |
| - **FP32 optimizer states** |
| - **FP32 gradients** |
|
|
| ## **Optimizer** |
|
|
| - **AdamW** |
| - Weight decay enabled |
| - **No 8-bit optimizer compression** |
| - **No low-rank optimizer approximation** |
|
|
| ## **Notes** |
|
|
| The Mistral configuration prioritizes: |
|
|
| - **numerical consistency** |
| - **deterministic convergence behavior** |
| - **stable long-context optimization** |
| - **reduced quantization-induced gradient noise** |
|
|
| This setup is computationally expensive but provides **high-fidelity optimization dynamics** during pretraining and finetuning. |
|
|
| --- |
|
|
| # **DIT Ernie** |
|
|
| ## **Precision Strategy** |
|
|
| The DIT Ernie architecture utilizes: |
|
|
| - **Monte Carlo estimation techniques** |
| - **probabilistic FP32 approximation** |
| - **stochastic numerical reconstruction** |
|
|
| Rather than maintaining strict FP32 execution across the entire training stack, the model estimates FP32-equivalent statistical behavior through sampling-based computation. |
|
|
| ## **Goals** |
|
|
| - reduce memory bandwidth requirements |
| - improve throughput efficiency |
| - retain approximate FP32 convergence characteristics |
| - balance numerical quality with hardware scalability |
|
|
| ## **Notes** |
|
|
| This methodology may introduce: |
|
|
| - **stochastic variance between runs** |
| - **approximation noise** |
| - **non-deterministic optimization characteristics** |
|
|
| However, it can significantly reduce training cost relative to native FP32 execution. |
|
|
| --- |
|
|
| # **Intended Use** |
|
|
| This repository is intended for: |
|
|
| - research documentation |
| - training methodology comparison |
| - optimizer precision analysis |
| - numerical stability benchmarking |
| - transformer architecture experimentation |
|
|
| --- |
|
|
| # **Limitations** |
|
|
| Results can vary depending on: |
|
|
| - sampling strategy |
| - hardware backend |
| - distributed training topology |
| - random seed initialization |
|
|
| --- |
|
|
| # **License** |
|
|
| **Apache License 2.0** |
