| --- |
| license: apache-2.0 |
| tags: |
| - tensorrt-llm |
| - tensorrt |
| - windows |
| - native |
| - rtx-4000 |
| - ada |
| - sm89 |
| - cuda |
| - llm-inference |
| - qwen3 |
| base_model: |
| - Goekdeniz-Guelmez/Josiefied-Qwen3-4B-abliterated-v1 |
| --- |
| |
| # TensorRT-LLM — Native Windows Build for RTX 40-Series (Ada / SM89) |
|
|
| A **native Windows x64 build of NVIDIA [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM)** — no WSL, no Docker, no compatibility layer. Built for **raw inference speed** on consumer RTX 40-series GPUs, plus a ready-to-serve prebuilt INT4 engine. |
|
|
| > ## ⚠️ HARDWARE: RTX 40-series (Ada / SM89) ONLY — read this first |
| > |
| > Built and tested on an **RTX 4060 (Ada Lovelace, SM 8.9)**. The prebuilt `tensorrt_llm.dll` **and** the prebuilt engine are compiled for **SM89**. They will **not** run on: |
| > - RTX 30-series (Ampere / SM86) or older |
| > - RTX 50-series (Blackwell / SM120) or newer |
| > - Any non-Ada NVIDIA GPU |
| > |
| > To run on a different architecture you must rebuild TRT-LLM for your SM and rebuild the engine — see `patches/` + `BUILD_README.md`. |
|
|
| > ## ⚠️ Disclaimer — please read |
| > |
| > This is a passion-project / proof-of-concept, built with AI assistance. I'm not a professional developer. It works and it's genuinely fast, but it is **not a supported product** — it may or may not be maintained. Anyone is free to build from it or fork it. **Not affiliated with or endorsed by NVIDIA.** |
|
|
| ## What this is |
|
|
| TensorRT-LLM normally needs WSL2 or Linux on Windows. This is a **from-source native Windows build** (TRT-LLM is Apache-2.0, so freely redistributable) that runs directly on Windows 11 — plus a **prebuilt INT4 engine** ready to serve. |
|
|
| **Built/tested on:** Windows 11 Pro x64 · RTX 4060 (Ada, SM89) · CUDA 13.2 · MSVC 14.44 / clang-cl · TensorRT-LLM 1.3.x |
|
|
| ## Requirements |
|
|
| - Windows 11 x64 |
| - **RTX 40-series GPU (Ada / SM89)** — see the hardware warning above |
| - NVIDIA **CUDA 13.2** + **TensorRT** installed — you obtain these from NVIDIA. This repo does **not** bundle NVIDIA's closed libraries; `tensorrt_llm.dll` links them at runtime. |
| - ~5–6 GB free VRAM for the 4B INT4 model at 16k context |
|
|
| ## What's in here |
|
|
| | Path | What | |
| |------|------| |
| | `dll/tensorrt_llm.dll` | The native Windows TensorRT-LLM runtime DLL (SM89) | |
| | `dll/tensorrt_llm.lib` | Import library | |
| | `engine/rank0.engine` | **Prebuilt INT4 engine** — Josiefied-Qwen3-4B (abliterated Qwen3-4B), 16k context, batch 1, ready to serve | |
| | `engine/config.json` | Engine build config | |
| | `patches/` | Source patches that make TRT-LLM compile natively on Windows (CCCL/C++20 fix, NUMA / GDRCopy / ifstream stubs, ninja `ccbin` fixes, vcvars wrapper, ODR fix, etc.) | |
| | `scripts/` | Serving + benchmark scripts (int4 / awq / fp8 / gptq / medusa) | |
| | `BUILD_README.md` | The full build log — every issue hit and how it was solved | |
|
|
| ## Status — what works vs what doesn't |
|
|
| - ✅ **AOT engine + C++ runtime (the fast path) — PROVEN.** Build the engine ahead of time, load it through the DLL runtime, get raw-speed inference. This is the point of the repo. |
| - ⚠️ **JIT Python backend / server scripts — EXPERIMENTAL / INCOMPLETE.** The `start_server_*.py` scripts were a work in progress and were **not** fully finished or verified. Included for reference only — don't expect them to just work. |
|
|
| ## Using the prebuilt engine |
|
|
| `engine/rank0.engine` is a ready-to-serve TensorRT-LLM engine for **[Josiefied-Qwen3-4B-abliterated-v1](https://huggingface.co/Goekdeniz-Guelmez/Josiefied-Qwen3-4B-abliterated-v1)** (INT4, 16k context, batch 1). Point a TensorRT-LLM runtime built against the included `tensorrt_llm.dll` at it. See `scripts/` for serving examples. |
|
|
| > ⚠️ **This engine is an abliterated (uncensored) model.** It's built from Josiefied-Qwen3-4B-abliterated-v1, not stock Qwen3-4B. If you want a stock/aligned model, build your own engine from the base weights using the `patches/` + `BUILD_README.md`. |
| |
| ## Building from source |
| |
| Everything needed to rebuild TRT-LLM natively on Windows (for your own SM) is in `patches/`, documented step-by-step in `BUILD_README.md`. |
|
|
| ## Credits / attribution |
|
|
| - **NVIDIA TensorRT-LLM** — Apache-2.0 · https://github.com/NVIDIA/TensorRT-LLM |
| - **Qwen3** — Apache-2.0 (Alibaba). The engine is the abliterated **Josiefied-Qwen3-4B**. |
| - Built with AI assistance (Claude + GitHub Copilot). |
|
|
| ## License |
|
|
| Apache-2.0 (matching upstream TensorRT-LLM) — see `LICENSE`. NVIDIA CUDA / TensorRT and the Qwen3 weights are covered by their own respective licenses. |
|
|
