| --- |
| license: cc0-1.0 |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| tags: |
| - MoE |
| - VDront |
| - AI |
| - LMM |
| - LM |
| - 200m |
| - 100m |
| --- |
|  |
|
|
| # VDrontV3-Mini - custom MoE model. (gpt 2 tokenizer) |
|
|
| You can find `model.py` in the repository; it describes the model architecture. |
|
|
| The repository also contains `use.py` and `GUIvdront.py`—example scripts for running the model. To use them, download the model files and place them in the `VDrontV3-Mini/` folder, then place `model.py`, `use.py`, and `GUIvdront.py` alongside that folder and run the script. |
|
|
| The model has two modes: "Base" and "Quality" In "Base" mode, responses are shorter and more varied, whereas in "Quality" mode, they are more similar in structure but longer than those in "Base". |
|
|
| # GUIvdront.py |
|  |
|
|
| # 1024 ctx |
|
|
| # Format user, assistant, endoftext |
|
|
| # cc0-1.0 |
|
|
|
|
| # model architecture |
|  |
|
|
|
|
| *Base model VDrontV2-0.1b* |
|
|
| VDrontV3 is a transformer-based architecture that implements a hybrid Mixture of Experts (MoE) design with two specialized expert blocks. |
|
|
| Embedding & Routing: The input passes through standard token and position embeddings. A "Router top-1" (a linear layer) processes the embedding to select a single expert (top-1 routing). |
| |
| Expert Block: A sequence of transformer layers is replicated into multiple parallel "Expert Blocks" (purple boxes). The router dynamically activates exactly one of these expert versions during each forward pass. |
| |
| Base Block: After the expert block, the hidden states pass through a shared, fixed "Base Block" (green box) consisting of standard transformer layers. This provides a common foundation for all routing paths. |
| |
| Output Block & Mode Block: The final layers form an "Output Block" (also multi-version) that feeds into the LM head. Crucially, the architecture includes a "ModeBlock" (yellow box), which suggests the model can switch between different inference modes by selecting different versions of the output block. |
| |
|
|
|
|
| # Chat Quality example |
|
|
| ```text |
| |
| You: Hello, what is a Python? |
| AI: Python is an interpreter that provides the code and its high-level programming language. It was widely used to perform several popular |
| programming languages like Ruby on your programming language known for its high-level programming language's high-level, easy-to-read syntax. |
| Its powerful, easy-to-use, easy-to-read syntax and easy-to-read syntax. It has an excellent choice for beginners. Python is one of the most |
| popular programming languages in Python, and has been popular among developers. It is one of the standard programming languages in the world today. |
| It is one of the most popular programming languages in the world. It has a high-level, easy-to-read syntax and easy-to-read syntax but can be as easy and accessible. |
| |
| You: Who are you? |
| AI: I am an artificial intelligence assistant, a language model designed to help people with tasks. I don't have a |
| physical body and generate human consciousness. I'm trained on a large dataset of data, which allows me to understand patterns in |
| language and information. I don't produce human-like responses. My primary purpose is to make your life easier and more enjoyable. I'm |
| trained on a vast range of tasks without having a physical body or emotions like humans do my job is to assist with writing, and more interesting |
| conversation. I don't have a physical body or emotions like humans do. I'm here to help with everything from writing, and more interesting. So, |
| what can I do today? I'm here for you? I'm here today? I'm here to help you with today? |
| |
| You: What is a PC? |
| AI: A computer, or computer's computer system that runs on the internet and manages the Internet. It acts as an |
| operating system to access and control over the computer and the computer and all computers in the computer and software |
| that uses a network of computers, including both on your computer and on your computer and the computer and all over the internet. |
| The OSI, which handles tasks without having a physical computer, and users to communicate with each other. A computer and the internet. |
| It handles servers, and manages and controls the internet through a single computer and the internet. There are three main types: the internet, |
| and the internet has several key components: the internet and the internet. It operates on the internet through a network of computers and networks |
| throughout various computers and devices. It provides a vast amount of data; and the most important aspect of modern computing, enabling everything from |
| email communication, and the internet. |
| |
| ``` |
|
|
| # Chat Base example |
|
|
| ```text |
| |
| You: Hello! |
| AI: Hello, I'm here to help you with your questions. |
| |
| You: What is a Python? |
| AI: A.py is a program that uses Python to create and execute code. It can be used for programming, and data analysis. Key aspects: |
| 1. Python-based |
| 2. Python-based |
| 3. Data analysis |
| 4. Data analysis |
| 5. Data analysis |
| |
| You: Who are you? |
| AI: I am a professional golfer who is passionate about golf. I believe in the importance of golfing and the benefits of playing with golfers. |
| * The first time you played this game |
| 2. Golfers enjoy golfing and playing with golf |
| 3. You can play golf with golfers |
| |
| ``` |
|
|
| VDrontV3-Mini is a test version of the VDrontV3 architecture; the "Base" and "Quality" modes merely make the response format resemble those versions but do not significantly improve quality, as Mini is based on the 0.1b base model(VDrontV2-0.1b). |
