Instructions to use Floobin/TSwifty-SN6 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Floobin/TSwifty-SN6 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Floobin/TSwifty-SN6")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Floobin/TSwifty-SN6")
model = AutoModelForCausalLM.from_pretrained("Floobin/TSwifty-SN6", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Floobin/TSwifty-SN6 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Floobin/TSwifty-SN6"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Floobin/TSwifty-SN6",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Floobin/TSwifty-SN6

SGLang

How to use Floobin/TSwifty-SN6 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Floobin/TSwifty-SN6" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Floobin/TSwifty-SN6",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Floobin/TSwifty-SN6" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Floobin/TSwifty-SN6",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Floobin/TSwifty-SN6 with Docker Model Runner:
```
docker model run hf.co/Floobin/TSwifty-SN6
```

TSwifty-SN6 / venv /Lib /site-packages /setuptools /distutils_patch.py

Floobin

Upload folder using huggingface_hub

0a594a8 verified over 2 years ago

Raw

History Blame Contribute Delete

1.75 kB

	"""
	Ensure that the local copy of distutils is preferred over stdlib.

	See https://github.com/pypa/setuptools/issues/417#issuecomment-392298401
	for more motivation.
	"""

	import sys
	import re
	import os
	import importlib
	import warnings


	is_pypy = '__pypy__' in sys.builtin_module_names


	def warn_distutils_present():
	if 'distutils' not in sys.modules:
	return
	if is_pypy and sys.version_info < (3, 7):
	# PyPy for 3.6 unconditionally imports distutils, so bypass the warning
	# https://foss.heptapod.net/pypy/pypy/-/blob/be829135bc0d758997b3566062999ee8b23872b4/lib-python/3/site.py#L250
	return
	warnings.warn(
	"Distutils was imported before Setuptools. This usage is discouraged "
	"and may exhibit undesirable behaviors or errors. Please use "
	"Setuptools' objects directly or at least import Setuptools first.")


	def clear_distutils():
	if 'distutils' not in sys.modules:
	return
	warnings.warn("Setuptools is replacing distutils.")
	mods = [name for name in sys.modules if re.match(r'distutils\b', name)]
	for name in mods:
	del sys.modules[name]


	def enabled():
	"""
	Allow selection of distutils by environment variable.
	"""
	which = os.environ.get('SETUPTOOLS_USE_DISTUTILS', 'stdlib')
	return which == 'local'


	def ensure_local_distutils():
	clear_distutils()
	distutils = importlib.import_module('setuptools._distutils')
	distutils.__name__ = 'distutils'
	sys.modules['distutils'] = distutils

	# sanity check that submodules load as expected
	core = importlib.import_module('distutils.core')
	assert '_distutils' in core.__file__, core.__file__


	warn_distutils_present()
	if enabled():
	ensure_local_distutils()