README.md

---
license: cc0-1.0
datasets:
- HuggingFaceFW/fineweb-edu
- openbmb/Ultra-FineWeb-L3
language:
- en
tags:
- LLM
- LM
- Dront
- GPT
- GPT2
- 200m
---

![](pre.png)

# Dront 200m is a GPT-like model trained from scratch on approximately 5 billion tokens.

Context: 1024 tokens

Tokenizer: GPT2 50k

License: cc0-1.0

# Perplexity

| Model | openbmb/Ultra-FineWeb-L3 | FineWeb-Edu-L4-5 |
|---------|--------|--------|
| Dront-200m | 11.33 | 13.23 |
| VDrontV2-0.1b | 17.52 | 11.01 |
| GPT-2 120m | 10.53 | 8.54 |

# Generate Example

Prompt: The Python is a...

```text

The Python is a powerful tool for building complex code, particularly in scenarios involving complex data structures.
It supports the development of complex queries, such as identifying a specific value or identifying a specific
attribute. This approach supports efficient data handling and high-level analysis. The core of Python is the Visual
 Basic Language (VML), which is widely used across various programming languages to build complex algorithms.

Python is a powerful tool for generating complex code by manipulating data structures. It is especially useful
for handling large datasets and complex data types. For example, a Python script can be written to handle a wide
 range of data types, including tables, graphs, and functions, making it suitable for both data analysis and modeling.

In summary, Python is a powerful tool for creating complex systems with complex data structures. Its ability
to process large datasets, generate complex models, and automate tasks makes it a valuable tool for developers
working on complex problems. By combining Python and VML, you can significantly improve your programming skills and improve your overall coding experience.

For those interested in learning more about Python, additional resources are available through dedicated
tutorials. These include detailed tutorials on the basics, practical tips for developing Python, and guidance
on how to use Python for code development.

Python is a powerful tool for building complex programs, especially in environments where data manipulation
is critical. It provides a clear overview of the fundamentals of Python, including its syntax, function, and syntax.
While Python is not without limitations, it offers a solid foundation for building complex applications.

For those seeking to learn Python, several online platforms offer access to tutorials, tutorials, and tutorials
on topics like object-oriented programming. These platforms provide a rich collection of tutorials, tutorials, and
tutorials designed to help users build and test their skills.

For those interested in learning Python, resources such as tutorials and tutorials can further support learning.
These resources are available in both English and Spanish, offering guidance on how to use them effectively.

In summary, Python is a powerful tool for building complex code, improving performance, and improving code quality.
 With consistent effort and careful planning, it can become a powerful tool for building complex applications.
The benefits of learning Python are well documented, making it a versatile tool that supports both beginners and experienced developers.

To begin learning Python, individuals should explore the official Python website at http://www.python.org/learn-to-learn-
Python-learn-with-Python.html. This resource offers a comprehensive overview of Python’s features and benefits, along with
step-by-step guidance on how to integrate it into their own projects.

The Python community has developed a range of tutorials and tutorials designed to help users learn Python from beginners.
These include tutorials on using Python to write and debug code, examples of advanced Python libraries, and a guide on using
Python for creating and managing complex code. Additionally, the platform includes tutorials on Python for creating and testing
 code, which helps learners develop strong coding abilities and improve their coding skills.

Overall, learning Python is a rewarding and effective method for building complex code. Whether you're new to Python or
seeking to expand your understanding, learning Python is a rewarding journey. The benefits of learning Python are vast
 and varied, making it a valuable addition to your training regimen.

For those interested in exploring Python’s potential in education, resources like The Python Tutorial Guide
 provide extensive information on Python, including tutorials and tutorials on Python, and additional tutorials on Python.

If you have questions about Python, feel free to contact us directly. We welcome your questions and
suggestions, and we encourage you to share your thoughts and concerns.

The Python Tutorial Guide offers a comprehensive overview of Python, covering its core components, functionality,
 and integration with other programming languages. It covers essential topics such as the fundamentals of Python, the
 fundamentals of programming, and the advantages of using Python in real-world projects. The guide also explores the
principles of Python, including its use cases, features, and advantages in Python, helping users understand the language’s purpose and benefits.

The Python Tutorial Guide is a comprehensive resource that covers the fundamentals of Python, including its structure,
implementation, and core concepts. It covers core concepts such as loops, loops, and loops, providing a structured way
to learn Python quickly and efficiently.

Users can also explore Python’s capabilities through a tutorial or online course, which provides detailed explanations
and examples. The guide explains the basics of Python, including its syntax, use cases, and the use of tools like Python.
Additionally, the guide explains how to create and deploy Python programs using Python, including the use of Python libraries,
libraries, and tools for creating and deploying programs.


```


# Easy Start ( ! Local !)

```Python

import torch
import sys
import warnings
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer

warnings.filterwarnings('ignore')

torch.backends.cudnn.benchmark = True
torch.backends.cuda.matmul.allow_tf32 = True
torch.set_float32_matmul_precision('high')

MODEL_NAME = "Dront-200m" # LOCAL FOLDER
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

print("Loading model...")
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, local_files_only=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    local_files_only=True,
    torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
    device_map="auto"
)
model.eval()
print("Model loaded")


def generate_text_streaming(prompt):
    inputs = tokenizer(prompt, return_tensors="pt").to(DEVICE)

    # Создаем стример для потокового вывода
    streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)

    print("Output: ", end="", flush=True)

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=1024,
            temperature=0.45,
            do_sample=True,
            top_p=0.95,
            repetition_penalty=1.1,
            pad_token_id=tokenizer.pad_token_id,
            eos_token_id=tokenizer.eos_token_id,
            streamer=streamer,  # Добавляем стример
            use_cache=True
        )

    print()  # Новая строка после завершения генерации
    return outputs


print("Text generation started! Type 'exit' to quit.")
while True:
    try:
        user_input = input("Input: ").strip()
        if user_input.lower() in ['exit', 'quit', 'q']:
            print("Goodbye!")
            break
        if not user_input:
            continue

        generate_text_streaming(user_input)

    except KeyboardInterrupt:
        print("\nGoodbye!")
        break
    except Exception as e:
        print(f"Error: {e}")
        continue

```

Bye!