Instructions to use KBlueLeaf/TIPO-200M-ft with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use KBlueLeaf/TIPO-200M-ft with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="KBlueLeaf/TIPO-200M-ft")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("KBlueLeaf/TIPO-200M-ft")
model = AutoModelForCausalLM.from_pretrained("KBlueLeaf/TIPO-200M-ft")

llama-cpp-python

How to use KBlueLeaf/TIPO-200M-ft with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="KBlueLeaf/TIPO-200M-ft",
	filename="TIPO-200M-ft-F16.gguf",
)

output = llm(
	"Once upon a time,",
	max_tokens=512,
	echo=True
)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use KBlueLeaf/TIPO-200M-ft with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf KBlueLeaf/TIPO-200M-ft:F16
# Run inference directly in the terminal:
llama-cli -hf KBlueLeaf/TIPO-200M-ft:F16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf KBlueLeaf/TIPO-200M-ft:F16
# Run inference directly in the terminal:
llama-cli -hf KBlueLeaf/TIPO-200M-ft:F16

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf KBlueLeaf/TIPO-200M-ft:F16
# Run inference directly in the terminal:
./llama-cli -hf KBlueLeaf/TIPO-200M-ft:F16

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf KBlueLeaf/TIPO-200M-ft:F16
# Run inference directly in the terminal:
./build/bin/llama-cli -hf KBlueLeaf/TIPO-200M-ft:F16

Use Docker

docker model run hf.co/KBlueLeaf/TIPO-200M-ft:F16

LM Studio
Jan

vLLM

How to use KBlueLeaf/TIPO-200M-ft with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "KBlueLeaf/TIPO-200M-ft"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KBlueLeaf/TIPO-200M-ft",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/KBlueLeaf/TIPO-200M-ft:F16

SGLang

How to use KBlueLeaf/TIPO-200M-ft with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "KBlueLeaf/TIPO-200M-ft" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KBlueLeaf/TIPO-200M-ft",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "KBlueLeaf/TIPO-200M-ft" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KBlueLeaf/TIPO-200M-ft",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use KBlueLeaf/TIPO-200M-ft with Ollama:
```
ollama run hf.co/KBlueLeaf/TIPO-200M-ft:F16
```

Unsloth Studio new

How to use KBlueLeaf/TIPO-200M-ft with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for KBlueLeaf/TIPO-200M-ft to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for KBlueLeaf/TIPO-200M-ft to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for KBlueLeaf/TIPO-200M-ft to start chatting

Docker Model Runner
How to use KBlueLeaf/TIPO-200M-ft with Docker Model Runner:
```
docker model run hf.co/KBlueLeaf/TIPO-200M-ft:F16
```

Lemonade

How to use KBlueLeaf/TIPO-200M-ft with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull KBlueLeaf/TIPO-200M-ft:F16

Run and chat with the model

lemonade run user.TIPO-200M-ft-F16

List all available models

lemonade list

TIPO: Text to Image with text presampling for Prompt Optimization

200M LLaMA arch model trained for TIPO.
Tech Report: https://arxiv.org/abs/2411.08127

Introduction

In this project, we introduce "TIPO" (Text to Image with text presampling for Prompt Optimization), an innovative framework designed to significantly enhance the quality and usability of Text-to-Image (T2I) generative models. TIPO utilizes the Large Language Models (LLMs) to perform "Text Presampling" within the inference pipeline of text-to-image generative modeling. By refining and extending user input prompts, TIPO enables generative models to produce superior results with minimal user effort, making T2I systems more accessible and effective for a wider range of users.

Usage

Use updated version of DTG extension (renamed to z-tipo-extension), current version of z-tipo-extension support stable-diffusion-webui, stable-diffusion-webui-forge and ComfyUI. SD-Next haven't been tested. https://github.com/KohakuBlueleaf/z-tipo-extension

Model arch and Training

This model is LLaMA arch with 200M parameters, the training data is combined version of Danbooru2023, Coyo-HD-11M.
The total token seen is around 50B tokens.
For more information please refer to the tech report and following table.

	TIPO-200M	TIPO-200M-ft	TIPO-500M
Arch	LLaMA	LLaMA	LLaMA
Max ctx length	1024	1024	1024
Batch Size	2048	2048	3584
Training dataset	Danbooru, GBC10M, 5epoch Danbooru, GBC10M, Coyo11M, 3epoch	Danbooru(pixtral), Coyo11M, 2epoch	Danbooru, GBC10M, Coyo11M, 5epoch
Real Token Seen*	40B token	50B (10B more from TIPO-200M)	30B token
Training Hardware	RTX 3090 x 4	RTX 3090 x 4	H100 x 8
Training Time	420 hour`	120 hour`	100 hour`
Huggingface	KBlueLeaf/TIPO-200M · Hugging Face	You Are HERE	KBlueLeaf/TIPO-500M · Hugging Face

*: We only count "non-padding token" in the token seen, since all the training data have very large length range.
`: Since the training data is pretty short, it cost more time to reach same token seen than general LLM pretraining.
As reference, with 4096 as max ctx length and almost all the data have reach that length, you may only need 2days to reach 10B token seen on RTX 3090 x 4 with 200M model.

Evaluation

Evaluation are done on TIPO-200M model
We have tested TIPO compared to other Model in several test and metrics:

Scenery tag test

In this test we use single "scenery" tag as input. (With some certain meta)
To test each prompt gen method to see if they can obtain the desired distribution of outputs while maintain the quality of images.

Scenery Tag Test	Original	GPT4o-mini	Prompt DB	Promptis	TIPO(ours)
FDD ↓	0.3558	0.5414	0.3247	0.2350	0.2282
Aesthetic ↑	5.0569	6.3676	6.1609	5.9468	6.2571
AI Corrupt ↑	0.4257	0.7490	0.5024	0.5669	0.9195

Short/Truncated Long test

In this test we use short caption or manually truncated caption from GBC10M and CoyoHD11M.
This test examine the ability of prompt gen method on handling almostly completed prompts.

Short	Original	GPT4o-mini	Prompt DB	Promptis	TIPO(ours)
FDD ↓	0.0957	0.1668	0.0980	0.1783	0.1168
Aesthetic ↑	5.8370	6.0589	5.8213	5.7963	5.8531
AI Corrupt ↑	0.7113	0.6985	0.7064	0.6314	0.7131

Truncated Long	Original	GPT4o-mini	Prompt DB	Promptis	TIPO(ours)
FDD ↓	0.0955	0.1683	0.1247	0.2096	0.1210
Aesthetic ↑	5.7497	6.0168	5.8191	5.7759	5.8364
AI Corrupt ↑	0.6868	0.6712	0.6741	0.5925	0.7130

LICENSE

This model is released under Kohaku License 1.0
You can check the above provided URL or check the LICENSE file in this repo.

Citation

@misc{yeh2024tipotextimagetext,
      title={TIPO: Text to Image with Text Presampling for Prompt Optimization}, 
      author={Shih-Ying Yeh and Sang-Hyun Park and Giyeong Oh and Min Song and Youngjae Yu},
      year={2024},
      eprint={2411.08127},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.08127}, 
}

Downloads last month: 542

Safetensors

Model size

0.2B params

Tensor type

F32

Datasets used to train KBlueLeaf/TIPO-200M-ft

Collection including KBlueLeaf/TIPO-200M-ft

TIPO

Collection

Text to Image with text presampling for Prompt Optimization • 6 items • Updated Jan 22, 2025 • 6

Paper for KBlueLeaf/TIPO-200M-ft

TIPO: Text to Image with Text Presampling for Prompt Optimization

Paper • 2411.08127 • Published Nov 12, 2024 • 4