---
license: apache-2.0
tags:
  - tensorrt-llm
  - tensorrt
  - windows
  - native
  - rtx-4000
  - ada
  - sm89
  - cuda
  - llm-inference
  - qwen3
base_model:
  - Goekdeniz-Guelmez/Josiefied-Qwen3-4B-abliterated-v1
---

# TensorRT-LLM — Native Windows Build for RTX 40-Series (Ada / SM89)

A **native Windows x64 build of NVIDIA [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM)** — no WSL, no Docker, no compatibility layer. Built for **raw inference speed** on consumer RTX 40-series GPUs, plus a ready-to-serve prebuilt INT4 engine.

> ## ⚠️ HARDWARE: RTX 40-series (Ada / SM89) ONLY — read this first
>
> Built and tested on an **RTX 4060 (Ada Lovelace, SM 8.9)**. The prebuilt `tensorrt_llm.dll` **and** the prebuilt engine are compiled for **SM89**. They will **not** run on:
> - RTX 30-series (Ampere / SM86) or older
> - RTX 50-series (Blackwell / SM120) or newer
> - Any non-Ada NVIDIA GPU
>
> To run on a different architecture you must rebuild TRT-LLM for your SM and rebuild the engine — see `patches/` + `BUILD_README.md`.

> ## ⚠️ Disclaimer — please read
>
> This is a passion-project / proof-of-concept, built with AI assistance. I'm not a professional developer. It works and it's genuinely fast, but it is **not a supported product** — it may or may not be maintained. Anyone is free to build from it or fork it. **Not affiliated with or endorsed by NVIDIA.**

## What this is

TensorRT-LLM normally needs WSL2 or Linux on Windows. This is a **from-source native Windows build** (TRT-LLM is Apache-2.0, so freely redistributable) that runs directly on Windows 11 — plus a **prebuilt INT4 engine** ready to serve.

**Built/tested on:** Windows 11 Pro x64 · RTX 4060 (Ada, SM89) · CUDA 13.2 · MSVC 14.44 / clang-cl · TensorRT-LLM 1.3.x

## Requirements

- Windows 11 x64
- **RTX 40-series GPU (Ada / SM89)** — see the hardware warning above
- NVIDIA **CUDA 13.2** + **TensorRT** installed — you obtain these from NVIDIA. This repo does **not** bundle NVIDIA's closed libraries; `tensorrt_llm.dll` links them at runtime.
- ~5–6 GB free VRAM for the 4B INT4 model at 16k context

## What's in here

| Path | What |
|------|------|
| `dll/tensorrt_llm.dll` | The native Windows TensorRT-LLM runtime DLL (SM89) |
| `dll/tensorrt_llm.lib` | Import library |
| `engine/rank0.engine` | **Prebuilt INT4 engine** — Josiefied-Qwen3-4B (abliterated Qwen3-4B), 16k context, batch 1, ready to serve |
| `engine/config.json` | Engine build config |
| `patches/` | Source patches that make TRT-LLM compile natively on Windows (CCCL/C++20 fix, NUMA / GDRCopy / ifstream stubs, ninja `ccbin` fixes, vcvars wrapper, ODR fix, etc.) |
| `scripts/` | Serving + benchmark scripts (int4 / awq / fp8 / gptq / medusa) |
| `BUILD_README.md` | The full build log — every issue hit and how it was solved |

## Status — what works vs what doesn't

- ✅ **AOT engine + C++ runtime (the fast path) — PROVEN.** Build the engine ahead of time, load it through the DLL runtime, get raw-speed inference. This is the point of the repo.
- ⚠️ **JIT Python backend / server scripts — EXPERIMENTAL / INCOMPLETE.** The `start_server_*.py` scripts were a work in progress and were **not** fully finished or verified. Included for reference only — don't expect them to just work.

## Using the prebuilt engine

`engine/rank0.engine` is a ready-to-serve TensorRT-LLM engine for **[Josiefied-Qwen3-4B-abliterated-v1](https://huggingface.co/Goekdeniz-Guelmez/Josiefied-Qwen3-4B-abliterated-v1)** (INT4, 16k context, batch 1). Point a TensorRT-LLM runtime built against the included `tensorrt_llm.dll` at it. See `scripts/` for serving examples.

> ⚠️ **This engine is an abliterated (uncensored) model.** It's built from Josiefied-Qwen3-4B-abliterated-v1, not stock Qwen3-4B. If you want a stock/aligned model, build your own engine from the base weights using the `patches/` + `BUILD_README.md`.

## Building from source

Everything needed to rebuild TRT-LLM natively on Windows (for your own SM) is in `patches/`, documented step-by-step in `BUILD_README.md`.

## Credits / attribution

- **NVIDIA TensorRT-LLM** — Apache-2.0 · https://github.com/NVIDIA/TensorRT-LLM
- **Qwen3** — Apache-2.0 (Alibaba). The engine is the abliterated **Josiefied-Qwen3-4B**.
- Built with AI assistance (Claude + GitHub Copilot).

## License

Apache-2.0 (matching upstream TensorRT-LLM) — see `LICENSE`. NVIDIA CUDA / TensorRT and the Qwen3 weights are covered by their own respective licenses.