Instructions to use RecursiveSI/flint with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RecursiveSI/flint with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RecursiveSI/flint")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RecursiveSI/flint")
model = AutoModelForCausalLM.from_pretrained("RecursiveSI/flint", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RecursiveSI/flint with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RecursiveSI/flint"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RecursiveSI/flint",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/RecursiveSI/flint

SGLang

How to use RecursiveSI/flint with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RecursiveSI/flint" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RecursiveSI/flint",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RecursiveSI/flint" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RecursiveSI/flint",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use RecursiveSI/flint with Docker Model Runner:
```
docker model run hf.co/RecursiveSI/flint
```

flint / sglang /README.md

lvmopt

Initial release

2a247be about 17 hours ago

preview code

Raw History Blame Contribute Delete

13.7 kB

Serving Flint with SGLang

This directory is Flint's serving kit for SGLang 0.5.16: the flint_sglang model package (a wheel), a Dockerfile, install.sh for existing containers, the FA4 learned-sink patch, serve scripts and two example clients. The commands below pin a commit of this repository.

Flint reasons at length on hard problems (see max_tokens below).

Quick start (Docker, one GPU)

Requirements:

One GPU with ≥141 GB per replica: B300/B200 (Blackwell) or H200 (Hopper). 80 GB GPUs are not supported.
NVIDIA driver ≥ 580 (CUDA 13). Older datacenter drivers (e.g. R570) work through the CUDA forward-compatibility libraries in the image (/usr/local/cuda/compat), which recent NVIDIA Container Toolkit and enroot setups pick up automatically. Tested: B300 with 580.126.20, H200 with 570.133.20.
Building the image needs access to Docker Hub, pypi.org/files.pythonhosted.org, flashinfer.ai and github.com (FlashInfer release assets).
About 80 GB of disk for the weights plus about 50 GB for the image.

pip install -U huggingface_hub openai
REV=$(python3 -c "from huggingface_hub import HfApi; print(HfApi().model_info('RecursiveSI/flint').sha)")
hf download RecursiveSI/flint --revision "$REV" --local-dir "flint-$REV"
cd "flint-$REV" && hf cache verify RecursiveSI/flint --revision "$REV" --local-dir .   # verify the download
docker build -t "flint-sglang:${REV:0:12}" sglang
docker run -d --name flint --gpus '"device=0"' --ipc=host -p 127.0.0.1:30000:30000 \
  -v "$PWD:/model" -e MODEL=/model "flint-sglang:${REV:0:12}" \
  bash /model/sglang/serve_b300.sh               # on H200: serve_h200.sh
until curl -sf localhost:30000/health_generate >/dev/null; do sleep 15; done   # progress: docker logs -f flint
python3 sglang/example_client.py --base-url http://localhost:30000/v1
python3 sglang/example_tools.py --base-url http://localhost:30000/v1    # nested tool arguments, parallel calls

The first start takes about 4 minutes on a B300 with the weights on fast local storage (longer when many replicas read the weights from network storage at once; budget up to 15 minutes). The log shows Flint weights loaded: 585 parameters matched the checkpoint. and then The server is fired up and ready to roll!. example_client.py prints the reasoning, a get_weather call for Paris, and a final answer that uses the tool result (sunny, 21C). example_tools.py books a meeting and fills a cart through tools with nested argument schemas (enums, patterns, bounds, arrays of objects) and makes three parallel weather calls. It answers each call with a canned result, which Flint's final answers report.

Serve scripts

Requests use the model name flint. Arguments after the script name are appended to sglang.launch_server, so a repeated flag overrides the script's value, e.g. bash /model/sglang/serve_b300.sh --mem-fraction-static 0.85.

Variable	Default	Notes
`MODEL`	`RecursiveSI/flint`	Use the local download (`/model` in Docker). With the Hub id the server downloads 79.6 GB at start-up.
`REVISION`	unset	Hub commit when `MODEL` is a Hub id (ignored for local paths).
`PORT`	`30000`	Replica port.
`CUDA_VISIBLE_DEVICES`	all GPUs	GPU of a single replica (the server uses the first visible GPU).
`FLINT_TRITON_CACHE_BASE`	`$HOME/.triton/cache`	Kernel cache root; each replica uses its own `flint-<host>-<port>` subdirectory. Mount a volume here to keep compiled kernels across container restarts.

The servers listen on 0.0.0.0 without authentication: publish ports only on localhost or a private network, or append --api-key <key> and give clients the same key.

Validated configurations

One GPU per replica:

B300 (x86_64): serve_b300.sh (--attention-backend fa4 --moe-runner-backend flashinfer_trtllm_routed). B200 and GB200/GB300 (arm64) use the same flags but are not validated yet.
H200: serve_h200.sh (--attention-backend fa3 --moe-runner-backend triton).

Both use --enable-fp32-lm-head --context-length 262144. The other settings are those Flint was evaluated with on each GPU type (B300: --mem-fraction-static 0.8 --swa-full-tokens-ratio 0.05; H200: 0.9 / 0.4).

The model refuses to start with a MoE runner other than flashinfer_trtllm_routed, triton or auto (flashinfer_trtllm and triton_kernel bypass Flint's sigmoid routing), with any --moe-a2a-backend, with an unpatched FA4 kernel, or if FlashInfer does not use its CUDA RMSNorm kernels (FLASHINFER_USE_CUDA_NORM=1, which the image and flint_sglang set). --attention-backend flashinfer does not support attention sinks. Tensor, expert or pipeline parallelism, DP attention, quantized weights or KV cache (e.g. FP8) and speculative decoding are not validated.

Several replicas behind a router

Replicas are independent single-GPU servers; the router (sglang_router 0.3.2, included in the image) spreads requests over them.

# 8 replicas on one 8-GPU node, ports 30000-30007 (Blackwell; on H200 add -e SERVE_SCRIPT=serve_h200.sh)
docker run -d --name flint-8x --gpus all --ipc=host -p 30000-30007:30000-30007 \
  -v "$PWD:/model" -e MODEL=/model "flint-sglang:${REV:0:12}" bash /model/sglang/serve_node_8x.sh
# router on any host that can reach every replica
docker run -d --name flint-router --network host "flint-sglang:${REV:0:12}" \
  python3 -m sglang_router.launch_router --host 0.0.0.0 --port 8000 --policy cache_aware \
  --worker-urls $(for n in node0 node1; do for p in $(seq 30000 30007); do printf 'http://%s:%d ' $n $p; done; done)

serve_node_8x.sh stops all 8 replicas when one exits; for independent restarts run one container per GPU instead (--gpus '"device=N"', -e PORT=3000N).

Existing container (no image build), e.g. enroot/pyxis

Inside a container of the base image pinned in sglang/Dockerfile (lmsysorg/sglang:v0.5.16-cu130), with network access:

bash flint-$REV/sglang/install.sh flint-$REV/sglang   # ends with: Flint support installed (flint_sglang-0.1.4-py3-none-any.whl).
MODEL=$PWD/flint-$REV PORT=30000 bash flint-$REV/sglang/serve_h200.sh   # Blackwell: serve_b300.sh

install.sh changes the container in place (FlashInfer 0.6.16.post1, CUTLASS DSL 4.6.2, the patched FA4 kernel, flint_sglang), can be re-run, and works without root if the container's system Python is writable (enroot/pyxis containers are). Keep or save that container (pyxis: --container-save=flint-sglang.sqsh); a fresh copy of the base image has no Flint support. On H200 the FA4 patch is applied but unused.

Capacity and tuning

KV cache per token: 28 KiB on the 14 full-attention layers (≈7.5 GB for one 262,144-token sequence) plus 70 KiB on the 35 sliding-window layers, which keep only the last 513 tokens of each running request.
--mem-fraction-static is the share of GPU memory for the weights (79.6 GB) plus the KV pools.
--swa-full-tokens-ratio sizes the sliding-window pool relative to the full-attention pool: raise it for many concurrent short requests, lower it for few long ones.

Chat format and API usage

Tool definitions appear after leading system messages, before the first developer or other role.

Use /v1/chat/completions with the parsers and grammar backend in the serve scripts (--tool-call-parser flint --reasoning-parser flint --grammar-backend flint). Flint refuses to start without --grammar-backend flint and --reasoning-parser flint, or with --enable-strict-thinking or --constrained-json-disable-any-whitespace, which that backend doesn't support. Without a tool-call parser, tool calls stay as literal <tool_call> text in content.
Tool-call grammar: --tool-call-parser flint parses Flint's tool calls and constrains every request with tools (unless tool_choice is none) with XGrammar to Flint's format: an optional <think>…</think> (from --grammar-backend flint), text without protocol markers, then tool calls to the request's tools: any number with tool_choice auto, at least one with required, exactly one call of the named function with a named choice. As in OpenAI's API, a tool's arguments must match its JSON schema only when the tool is sent with "strict": true; they are then also written in the order the tool declares its parameters (list optional parameters where they belong). Follow OpenAI's strict-mode rules: give every object properties or additionalProperties and every array items (otherwise XGrammar accepts no value for the object and only [] for the array); a strict schema XGrammar can't compile (for example a pattern with a lookahead) fails the request. Other tools keep the call format but not their schema. parallel_tool_calls: false allows a single call. The turn still ends with Flint's stop tokens.
Tool-call limits:
- A strict string value (an enum, const, pattern, format or maxLength) written with whitespace around it fails its schema: XGrammar allows the whitespace and the parser keeps it. Flint rarely does this; it has been seen on a prompt that asks it to print a tool call as text.
- A string argument can't contain the literal text </tool_call>. The format has no escaping, so the parser ends the call there and reads the rest as output: text Flint copies into an argument can add a call, without arguments, to a listed tool.
- When streaming, a call whose last argument is a JSON object can arrive without its final }. SGLang adds it back only for the response's last call.
- With tool_choice: "none" nothing is constrained, so call markup Flint writes can reach content.
Structured output: response_format (json_schema, json_object or structural_tag) and SGLang's regex and ebnf parameters constrain content. Flint may first reason in <think>…</think> (returned in reasoning_content), and the constraint applies to the answer after it. A request with both tools and a response format gets only the response format, and <tool_call> text in the answer is still read as a tool call. Use /v1/chat/completions: /generate and /v1/completions return Flint's raw text, including any <think>…</think> written before a constrained answer.
Sampling: temperature 1.0, top_p 0.95, top_k 20. These are the generation_config.json defaults the server applies when a request omits them; OpenAI clients pass top_k via extra_body. Greedy decoding is not validated.
Reasoning: the model decides on each turn whether to reason in <think>…</think> (there is no switch or budget). Reasoning is returned in reasoning_content and counts toward max_tokens: on GPQA-Diamond it used a median of 4.3K tokens (p95 33K, p99 46K; about 5% of answers exceed 32K). Use max_tokens of at least 32768 for hard problems (Flint was evaluated with up to 81920); finish_reason: "length" means the turn was cut off, possibly before any content or tool call. Keep reasoning_content on assistant messages when you send the conversation back. thinking / enable_thinking in chat_template_kwargs have no effect: the flint reasoning parser reads <think>…</think> when Flint writes it and never assumes a turn starts in reasoning, so turns without reasoning keep their content and their tool-call grammar.
Messages: content must be a string (flatten content-part arrays). A developer message uses its own <|developer|> role marker (token ID 150018). There is no default system prompt. As in training, the template removes NUL characters and <think>/</think> from system, developer, user and tool text and escapes role markers there (<|user|> becomes <|user|>). It does not escape the tool tags (<tool_call>, <tool_response>, <arg_key>, <arg_value>); strip those from untrusted tool output.
Tools: send tools with every request. Each function definition is rendered as the template receives it; through the server that is description, name, parameters and strict (other fields are dropped). Send function.arguments back unchanged (the JSON string you received) and one tool message per call, in the order of tool_calls. When calling apply_chat_template directly, decode argument strings into dictionaries first.
Generation stops at <|end_of_text|>, <|user|>, <|observation|> or <|developer|> (ids 150001, 150003, 150005, 150018). No BOS token is used for chat.

Versions and verification

Quote the full commit SHA ($REV) in results and bug reports. Download each commit into a new directory (hf download does not delete files removed upstream).
Serving kit: SGLang 0.5.16 (base image pinned by digest in sglang/Dockerfile; includes transformers 5.12.1, torch 2.11.0+cu130, sglang-router 0.3.2), flint_sglang 0.1.4, FlashInfer 0.6.16.post1, nvidia-cutlass-dsl 4.6.2, quack-kernels 0.6.4, apache-tvm-ffi 0.1.12.

When reporting a problem, include the commit SHA, the image ID (docker inspect --format '{{.Id}}' flint-sglang:<tag>), nvidia-smi output, the launch command including extra arguments, the start-up log and, for output problems, the request JSON and the raw response.

Transformers

The tokenizer and chat template load with AutoTokenizer.from_pretrained; decode each tool_calls[].function.arguments string into a dict before apply_chat_template. Transformers does not support Flint yet, so config.json loads with FlintConfig.from_pretrained from flint_sglang.configuration_flint (transformers ≥ 5.8, tested with 5.12.1) rather than AutoConfig.