tonimartir/gpt-oss-20b-onnx-cuda-rtn-gpu

GPT-OSS 20B ONNX CUDA (RTN Quantized) This model is an optimized version of gpt-oss-20b to enable local inference on CUDA GPUs using RTN (Round-to-Nearest) quantization.

Model Summary Developed by: Microsoft

Model Type: ONNX

License: Apache-2.0

Optimization: RTN Quantization for GPU memory efficiency

Model Size: 11.8 GB (model.onnx.data)

Technical Description This repository contains a conversion of the gpt-oss-20b model specifically tailored for local inference on CUDA-enabled GPUs. By utilizing the ONNX format and RTN quantization, the model achieves a significant reduction in VRAM footprint while maintaining the core capabilities of the base architecture.

Base Model Information For detailed information regarding the architecture, training data, and intended use cases, please refer to the original gpt-oss-20b model on Azure AI Foundry.

Deployment Compatible with ONNX Runtime (ORT) using the CUDAExecutionProvider.

Disclaimer The model is an optimization of the base model; any risk associated with its use is the responsibility of the user. Please verify and test for your specific scenarios. There may be slight differences in output compared to the base model due to the optimizations applied. Note that these optimizations are distinct from fine-tuning and do not alter the intended uses or capabilities of the model.