---
license: cc0-1.0
datasets:
- HuggingFaceFW/fineweb-edu
- TIGER-Lab/Fineweb-Instruct
language:
- en
base_model:
- VDC-team/VdrontV1
tags:
- gpt2
- gpt
- llm
- 60m
- 0.06b
- lm
- ai
- dront
---

# cc0-1.0


[![Hugging Face](https://img.shields.io/badge/🤗%20Hugging%20Face-Model-blue)](https://huggingface.co/)
[![Dront Quant](https://img.shields.io/badge/🦤%20Dront%20Quant-Model-red)](https://huggingface.co/VDC-team/DrontGPTQ)
[![Parameters](https://img.shields.io/badge/Parameters-60M-green)]()

**Small description**

*VDrontV1.5-Flash* is a *60M-parameter GPT-like model*. Train in 1.5b tokens.

The model was trained on *FineWeb-Edu + FineWeb-Edu-Instruct*.

Stored in *Hugging Face* format and *DrontQ(.dq) quantized* format.

VDrontV1.5-Flash/ - HF format 32 bit

VDrontV1.5-Flash_quant/ .dq format, the folder contains *DQuse.py* for using .dq.

**Formats**

| format | bit | Path | Use |  Size | Generation quality |
|---------|--------|--------|--------|--------|--------|
| Hugging Face .safetensors| 32 bit | VDrontV1.5-Flash/ | use.py | 250mb | Full |
| .dq (dront quant)| 16 bit | VDrontV1.5-Flash_quant/VDrontV1.5-Flash_Q16.dq| VDrontV1.5_quant/DQuse.py | 175mb |  slightly worse |
| .dq (dront quant) | 12 bit | VDrontV1.5-Flash_quant/VDrontV1.5-Flash_Q12.dq | VDrontV1.5_quant/DQuse.py | 130mb | between 8-bit and 16-bit |
| .dq (dront quant) | 8 bit| VDrontV1.5-Flash_quant/VDrontV1.5-Flash_Q8.dq | VDrontV1.5_quant/DQuse.py | 90mb |  worse |

**Details**

Context length: *512*

Tokenizer: *gpt2*

Base model: *VdrontV1* (ours)

Fine-tuning on GPU using the *Lion* optimizer

Ratio of *FW-Edu to FW-Edu-instruct training* = *3:1*

Use of *Drontal Quant*. 

License : *cc0-1.0*.

**Guide**

To use the HF model, go to the files, download the `VDrontV1.5-Flash/` folder and `use.py`, and place them together. `use.py` includes prompt formatting with `<|user|>` and `<|assistant|>` and is ready for immediate use.

To use Dront Quant models, go to the VDrontV1.5-Flash_quant/ folder. It contains 3 .dq models (8-bit, 12-bit, and 16-bit) as well as DQuse.py. Download the model you are interested in and place DQuse.py and the .dq model next to each other. At the end of the DQuse.py file, change the model name to the one you want to use. Note that the prompt formatting uses <|user|> and <|assistant|> tokens. (388 line: chat_with_dq_model("[path].dq", 256) )


**Fine-tune**

If you fine-tune the VDrontV1.5-Flash model, we will most likely add it to the VDrontV collection.

*Python libraries*:

  DQuse.py : torch json os struct numpy collections transformers

  use.py : torch transformers threading

DrontQuant : https://huggingface.co/VDC-team/DrontGPTQ

Hugging Face : VDC-team/