EVALUATION-ONLY ACCESS
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
This is a private evaluation version of Qwen3.5-9B-OQ-EXL3 (OraQuant).
By agreeing, you accept:
- Internal testing only; no production use
- No commercial use, redistribution, or reverse-engineering
- Deletion of all files after evaluation
- Full terms in
LICENSE
Access is granted only to approved licensees.
Log in or Sign Up to review the conditions and access this model content.
Configuration Parsing Warning:In config.json: "quantization_config.bits" must be a number
Qwen3.5-9B-OQ-EXL3 (OraQuant)
This repository contains an EXL3 build of Qwen3.5-9B, quantized by Ora Computing with OraQuant (OQ) - Ora Computing's proprietary calibrated quantization.
It is an exllamav3-compatible quantization of Qwen/Qwen3.5-9B; the underlying weights are unchanged Qwen3.5-9B weights at reduced precision.
Text only.
Qwen/Qwen3.5-9Bis a multimodal model; this checkpoint contains only the language model (text input -> text output). The vision/video input encoders and the MTP draft head are not included.
Model Overview
Model name: Qwen3.5-9B-OQ-EXL3 (OraQuant)
Base model: Qwen/Qwen3.5-9B (Apache-2.0, Alibaba Cloud) - this is an EXL3 quantization of it
Parameters: ~9 billion (unchanged from the base model)
Quantization: OraQuant (OQ) non-uniform EXL3 trellis quantization produced by Ora Computing. Transformer body at mixed 2-4 bit trellis rates chosen per tensor, output head at 5 bits, embeddings in BF16 (kept in system RAM by exllamav3).
Footprint: ~4.7 GB on disk; ~2.7 GB of GPU-resident weights at load time.
Not fine-tuned, not parameter-reduced: the model architecture and parameter count are identical to the base model; only the weight precision is reduced. No additional training was performed.
Purpose: Evaluation/test-use only; optimized for local/offline inference and internal benchmarking.
License: See LICENSE (Custom Model License Agreement).
Files in this repo
| File | What it is | Size |
|---|---|---|
model.safetensors |
Language model, OraQuant EXL3 (mixed 2-4 bit trellis) | ~4.7 GB |
config.json, generation_config.json |
Model / generation configuration | - |
tokenizer.json, vocab.json, merges.txt, tokenizer_config.json, chat_template.jinja |
Tokenizer and chat template | - |
LICENSE |
Custom Model License Agreement | - |
Usage
This checkpoint loads with stock upstream exllamav3 (no patch required; tested with v1.2.0):
from exllamav3 import Cache, Config, Generator, Model, Tokenizer
config = Config.from_directory("/path/to/Qwen3.5-9B-OQ-EXL3")
model = Model.from_config(config)
cache = Cache(model, max_num_tokens=8192)
model.load()
tokenizer = Tokenizer.from_config(config)
generator = Generator(model=model, cache=cache, tokenizer=tokenizer)
prompt = "<|im_start|>user\nWhat is 17 * 23?<|im_end|>\n<|im_start|>assistant\n"
print(generator.generate(prompt=prompt, max_new_tokens=400, completion_only=True,
add_bos=False,
stop_conditions=[tokenizer.eos_token_id,
tokenizer.single_id("<|im_end|>")]))
It also works with exllamav3-based frontends (e.g. an OpenAI-compatible server such as TabbyAPI) configured to point at the model directory.
Qwen3.5 is a reasoning model; the chat template and thinking behaviour are carried in chat_template.jinja.
Intended Use & Restrictions
Permitted use
- Internal testing, benchmarking, and evaluation of the model by the named Licensee.
- Exploration of model behaviours, prompt engineering, and non-production prototypes.
Prohibited use
- Deployment in a production or commercial service, publicly-facing API, resale, or redistribution.
- Fine-tuning or creating derivative models for production use without a separate agreement.
- Reverse-engineering the quantization/calibration used to produce these files.
- Disclosure or sharing of the model (or its weights) to third parties beyond the named Licensee.
Out-of-scope use
- Use in regulated or safety-critical contexts (unless separately permitted).
- Any use that violates the Apache License, Version 2.0 under which the upstream model is distributed.
Quantization
- Method: OraQuant (OQ), Ora Computing's proprietary calibrated quantization. The released file codes the transformer body in the EXL3 trellis format at non-uniform, per-tensor bit rates (2-4 bits), with a 5-bit output head and BF16 embeddings.
- No fine-tuning: the weights are the original
Qwen/Qwen3.5-9Bweights; no additional training was performed. - No parameter-count change: the architecture and ~9B parameter count are unchanged; only weight precision is reduced.
Limitations & Risks
- Quantized models may not replicate the full behaviour of the base model under all prompt categories, particularly domain-specific or rare inputs.
- The model is provided as-is for testing only and is not certified for production use.
- Users should validate outputs carefully and monitor for bias or unintended behaviours.
Upstream Attribution
This model is derived from the Qwen3.5-9B model released by Alibaba Cloud under the Apache License, Version 2.0.
"Copyright 2025 Alibaba Cloud. Licensed under the Apache License, Version 2.0."
For full terms, see: https://huggingface.co/Qwen/Qwen3.5-9B/blob/main/LICENSE Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0
Contact & Support
For licensing inquiries or to request extended evaluation rights, please contact: info@oracomputing.com
Repository and model access are regulated. Do not redistribute or share without explicit written permission from Ora Computing.
- Downloads last month
- 4