File size: 2,710 Bytes
094aa44 59e3509 702d29b 9846ce9 094aa44 702d29b 59e3509 702d29b 59e3509 702d29b 59ae0d6 702d29b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 | ---
license: cc0-1.0
datasets:
- HuggingFaceFW/fineweb-edu
- TIGER-Lab/Fineweb-Instruct
language:
- en
base_model:
- VDC-team/VdrontV1
tags:
- gpt2
- gpt
- llm
- 60m
- 0.06b
- lm
- ai
- dront
---
# cc0-1.0
[](https://huggingface.co/)
[](https://huggingface.co/VDC-team/DrontGPTQ)
[]()
**Small description**
*VDrontV1.5-Flash* is a *60M-parameter GPT-like model*. Train in 1.5b tokens.
The model was trained on *FineWeb-Edu + FineWeb-Edu-Instruct*.
Stored in *Hugging Face* format and *DrontQ(.dq) quantized* format.
VDrontV1.5-Flash/ - HF format 32 bit
VDrontV1.5-Flash_quant/ .dq format, the folder contains *DQuse.py* for using .dq.
**Formats**
| format | bit | Path | Use | Size | Generation quality |
|---------|--------|--------|--------|--------|--------|
| Hugging Face .safetensors| 32 bit | VDrontV1.5-Flash/ | use.py | 250mb | Full |
| .dq (dront quant)| 16 bit | VDrontV1.5-Flash_quant/VDrontV1.5-Flash_Q16.dq| VDrontV1.5_quant/DQuse.py | 175mb | slightly worse |
| .dq (dront quant) | 12 bit | VDrontV1.5-Flash_quant/VDrontV1.5-Flash_Q12.dq | VDrontV1.5_quant/DQuse.py | 130mb | between 8-bit and 16-bit |
| .dq (dront quant) | 8 bit| VDrontV1.5-Flash_quant/VDrontV1.5-Flash_Q8.dq | VDrontV1.5_quant/DQuse.py | 90mb | worse |
**Details**
Context length: *512*
Tokenizer: *gpt2*
Base model: *VdrontV1* (ours)
Fine-tuning on GPU using the *Lion* optimizer
Ratio of *FW-Edu to FW-Edu-instruct training* = *3:1*
Use of *Drontal Quant*.
License : *cc0-1.0*.
**Guide**
To use the HF model, go to the files, download the `VDrontV1.5-Flash/` folder and `use.py`, and place them together. `use.py` includes prompt formatting with `<|user|>` and `<|assistant|>` and is ready for immediate use.
To use Dront Quant models, go to the VDrontV1.5-Flash_quant/ folder. It contains 3 .dq models (8-bit, 12-bit, and 16-bit) as well as DQuse.py. Download the model you are interested in and place DQuse.py and the .dq model next to each other. At the end of the DQuse.py file, change the model name to the one you want to use. Note that the prompt formatting uses <|user|> and <|assistant|> tokens. (388 line: chat_with_dq_model("[path].dq", 256) )
**Fine-tune**
If you fine-tune the VDrontV1.5-Flash model, we will most likely add it to the VDrontV collection.
*Python libraries*:
DQuse.py : torch json os struct numpy collections transformers
use.py : torch transformers threading
DrontQuant : https://huggingface.co/VDC-team/DrontGPTQ
Hugging Face : VDC-team/ |