README.md

---
license: cc0-1.0
datasets:
- HuggingFaceFW/fineweb-edu
- TIGER-Lab/Fineweb-Instruct
language:
- en
base_model:
- VDC-team/VdrontV1
tags:
- gpt2
- gpt
- llm
- 150m
- 0.15b
- lm
- ai
- dront
---


# cc0-1.0


[![Hugging Face](https://img.shields.io/badge/🤗%20Hugging%20Face-Model-blue)](https://huggingface.co/)
[![Dront Quant](https://img.shields.io/badge/🦤%20Dront%20Quant-Model-red)](https://huggingface.co/VDC-team/DrontGPTQ)
[![Parameters](https://img.shields.io/badge/Parameters-150M-green)]()

**Small description**

*VDrontV1.5* is a *150M-parameter GPT-like model*. It is an enlarged version of V1, trained on approximately *3.5 billion tokens*.

The model was trained on *FineWeb-Edu + FineWeb-Edu-Instruct*.

Stored in *Hugging Face* format and *DrontQ(.dq) quantized* format.

VDrontV1.5/ - HF format 32 bit

VDrontV1.5_quant/ .dq format, the folder contains *DQuse.py* for using .dq.

**Formats**

| format | bit | Path | Use |  Size | Generation quality |
|---------|--------|--------|--------|--------|--------|
| Hugging Face .safetensors| 32 bit | VDrontV1.5/ | use.py | 600mb | Full |
| .dq (dront quant)| 4 bit | VDrontV1.5_quant/VDrontV1.5_Q4.dq| VDrontV1.5_quant/DQuse.py | 95mb | Almost terrible |
| .dq (dront quant) | 8 bit | VDrontV1.5_quant/VDrontV1.5_Q8.dq | VDrontV1.5_quant/DQuse.py | 190mb | Noticeably worse |
| .dq (dront quant) | 16 bit| VDrontV1.5_quant/VDrontV1.5_Q16.dq | VDrontV1.5_quant/DQuse.py | 370mb |  A little worse |

**Dront Model Tester**

| Dront Model Tester | VDrontV1.5 | VDrontV1.5_Q16.dq| VDrontV1.5_Q8.dq |  VDrontV1.5_Q4.dq |
|---------|--------|--------|--------|--------|
| talk | 30% | 30%  | 25% | 8% |
| info | 9.5% | 9.5% |  8% | 1% |
| size | 600mb | 370mb | 190mb | 95mb |
| bits per param | F32 | Q16 |  Q8 | Q4 | 


**Examples**

*The model is more akin to an instruction-tuned model than a conversational one; it supports the `<|user|>` – `<|assistant|>` format but won't respond to a greeting with a greeting of its own. However, it answers questions like "What is Python?" or "How many people are there in Asia?" quite well.*

VDrontV1.5 HF (t 0.4)
```text
Q: What is a Python? 

A: Python language is a language that is used to communicate with a person or group of people. It is used in many languages, but is not a standard language. The most common programming language used is Python.

The Python language is a programming language that is used by many people today. It is used by many people today as a means of communication between people and the world. It is used by many people today as a way to communicate with people, and it is used by many people today.

Some of the most popular programming languages include:

• Python - Python is a programming language. It is used by many people today, especially in the UK. It is used by many people today.

• Python - Python is used by many people today. It is used by many people today. It is used by many people today.
```

VDrontV1.5_Q16.dq (t 0.4)
```text
Q: What is a Python? 

A: A Python is an extension of the Python language. The following are some of the common Python commands in Python:

• Python is a Python language. It provides a code for creating code to be used by Python libraries and libraries. This code can be used for creating Python libraries or libraries.

• Python is a Python programming language. Python can be used for many other languages, including Python, Web Services, etc. The Python language is used for many different languages.

• Python is used for many types of applications like web hosting, website hosting, Web hosting, Web services, etc. Python is used for many things. Python is also used for other programming languages such as Web Services and Web Services.
```

VDrontV1.5_Q8.dq (t 0.4)
```text
Q: What is a Python? 

A: A Python is a type of code that allows you to write the code for a particular topic or topic. It can be used to create a code for a specific topic or topic, such as a topic or topic. The code will then be sent back to you for further processing. 

Some examples of Python are:

- PHP - This is a Python code that allows you to write code for a specific topic or topic. It can be used to create a code for a specific topic or topic.

- Python - This code allows you to create a code for a specific topic or topic. It also allows you to create a code for a particular topic or topic.

- Python - This code allows you to create a code for a specific topic or topic in a specific subject area. You can also create code for an existing topic or topic.
```


VDrontV1.5_Q4.dq
```text
Q: What is a Python? 

A:The Python world is not only useful, but it also allows the best thing to be found in the real world. The world also serves as the most time for your next post! The best time for you are in the time of your next post time in time (we have no big time, they are they? We run time, they are they? We have no longer, they are they, they are they, they are they? Maybe they? The end time, they are they? The end time, they are they? The end time, they are they, they are they. The end time, they are they? They are they are they? We are they? The end time, they are they? Maybe they? We're they? The end time, they're they? They are they?
```

**Details**

Context length: *512*

Tokenizer: *gpt2*

Base model: *VdrontV1* (ours)

Fine-tuning on GPU using the *Lion* optimizer

The number of parameters has been increased *by creating new layers*.

Ratio of *FW-Edu to FW-Edu-instruct training* = *3:1*

Use of *Drontal Quant*. 

License : *cc0-1.0*.

**Guide**

To use the HF model, go to the files, download the `VDrontV1.5/` folder and `use.py`, and place them together. `use.py` includes prompt formatting with `<|user|>` and `<|assistant|>` and is ready for immediate use.

To use Dront Quant models, go to the VDrontV1.5_quant/ folder. It contains 3 .dq models (4-bit, 8-bit, and 16-bit) as well as DQuse.py. Download the model you are interested in and place DQuse.py and the .dq model next to each other. At the end of the DQuse.py file, change the model name to the one you want to use. Note that the prompt formatting uses <|user|> and <|assistant|> tokens. (388 line: chat_with_dq_model("[path].dq", 256) )


**Fine-tune**

If you fine-tune the VDrontV1.5 model, we will most likely add it to the VDrontV collection.

*Python libraries*:

  DQuse.py : torch json os struct numpy collections transformers

  use.py : torch transformers threading

DrontQuant : https://huggingface.co/VDC-team/DrontGPTQ

Hugging Face : VDC-team/

P.S The next model is VDrontV1.5-Flash (60M parameters).