kai-os/Carnice-V3-GGUF

Carnice-V3 GGUF for Hermes Agent

Important limitations: quantization does not repair the behavioral limitations of the source model. Carnice V3 did not pass its formal behavioral quality gate, and a small internal Hermes diagnostic showed weaker long-horizon completion and task-level tool-contract performance than the base model. Do not use it for unattended, destructive, high-stakes, or production agents without independent evaluation and strong runtime controls.

This repository contains high-quality GGUF quantizations of the complete merged BF16 kai-os/Carnice-V3 checkpoint. It is a separate quantized release derived from the full 27,781,427,952-parameter BF16 model. Every GGUF here was converted independently from that validated merged model and never from another quantization.

Carnice V3 is based on Qwen/Qwen3.8-27B. Post-training used a rank-64 rsLoRA over all 496 audited language linear modules and then safely merged it into the base model. Vision and MTP modules were not post-trained.

Files and recommendations

File	Type	Size (GiB)	Positioning
Carnice-V3-Q4_K_M.gguf	Q4_K_M	15.66	smallest recommended release quant
Carnice-V3-Q5_K_M.gguf	Q5_K_M	18.19	recommended balance
Carnice-V3-Q6_K.gguf	Q6_K	20.89	high quality
Carnice-V3-Q8_0.gguf	Q8_0	27.05	maximum-fidelity quant

The multimodal projector is mmproj-Carnice-V3-BF16.gguf. It is deliberately kept in BF16: the vision encoder/projector is much smaller than the language model, while quantizing it more aggressively can directly distort the representations presented to the language model. Use the same BF16 projector with any language-model quant above.

Q5_K_M is the default recommendation for a useful memory/quality balance. Prefer Q6_K when agent reliability matters more than a few GiB, and Q8_0 when memory permits and minimizing quantization loss is the priority. Q4_K_M is the smallest variant published here; lower-bit IQ/Q2/Q3 variants were intentionally omitted because this release is meant to be useful, not a size demonstration.

Quantization method and integrity

Conversion and quantization used a pinned build of llama.cpp, recorded in quantization_results.json. The pipeline first converted the validated merged BF16 Transformers directory to one BF16 language-model GGUF. It then computed one importance matrix using only the private training split, with special-token parsing enabled, and independently quantized each published file from that BF16 GGUF. Requantization was not enabled.

The importance matrix covered up to 192 512-token chunks. Training windows were interleaved by trajectory before decoding so the first calibration region represented all six training trajectories rather than one long task. Token IDs were required to round-trip exactly through the pinned tokenizer. The private calibration text is not redistributed.

release_manifest.json records the published files. quantization_results.json records the source model, build details, importance-matrix evidence, held-out perplexity comparisons, text/tool smoke results, and the BF16 multimodal smoke result. These checks establish conversion integrity and directional quantization degradation only; they are not evidence of broad agent quality.

Measured quantization checks

Type	Validation PPL ratio	Test PPL ratio	Tool-call smoke
Q4_K_M	1.02633x	1.01793x	pass
Q5_K_M	1.00441x	1.00504x	pass
Q6_K	1.00196x	1.00308x	pass
Q8_0	1.00096x	1.00256x	pass

The validation and test sets each contain a single private task family, so their perplexities are directional and not a benchmark. Calibration used no validation or test tokens. A passing perplexity comparison uses the pinned llama-perplexity tool's default tokenizer behavior; that binary does not expose the --parse-special switch supported by llama-imatrix. Ratios remain paired against the same BF16 GGUF, tokenizer, corpus, context, and chunk count. A passing tool smoke requires llama.cpp's OpenAI-compatible server to return a parseable call to the declared terminal tool with the exact requested JSON argument. A passing vision smoke requires the BF16 projector and Q8_0 language model to load together and identify visible text or major visual features from the public Carnice cover image.

Running with llama.cpp

Use a current llama.cpp build with Qwen3.8/Qwen3.5 conditional-model support. Text-only usage does not require the projector:

llama-server \
  -hf kai-os/Carnice-V3-GGUF:Carnice-V3-Q5_K_M.gguf \
  -ngl 99 -c 32768 --jinja \
  --host 127.0.0.1 --port 8080

For a local download and multimodal input:

llama-mtmd-cli \
  -m Carnice-V3-Q6_K.gguf \
  --mmproj mmproj-Carnice-V3-BF16.gguf \
  --image image.png \
  -p "Describe the image carefully." \
  -ngl 99 -c 32768 -n 1024

The embedded chat template is the pinned Qwen3.8 template. Do not replace it with generic ChatML or generic OpenAI JSON formatting. Pass standard OpenAI-style function schemas through an endpoint that honors the embedded Jinja template. Hermes must parse and dispatch the resulting Qwen/Hermes tool envelope.

For long-running tasks, keep thinking enabled, expose every tool the runtime can actually dispatch, and configure explicit context, per-response, and iteration ceilings. Log all limit contacts and count them as incomplete tasks. The model has no tools or permissions by itself; sandboxing, dispatch, result handling, timeouts, and authorization remain runtime concerns.

Training data and procedure

The task specifications, deterministic fixtures, and hidden verifiers were locally authored for Carnice V3. Teacher trajectories ran inside the pinned Hermes Agent runtime using Qwen-Ambassador/Qwen3.8-Max through ModelScope. Collection requested xhigh reasoning and exposed the complete pinned Hermes tool schema.

A root was admitted only after all model calls settled, parent/delegated-child lineage was complete, tool arguments passed pinned schemas, privacy checks passed, and an independent executable verifier accepted the outcome. Rejected, partial, over-limit, unverifiable, or rights-unapproved attempts contributed no SFT tokens.

The reviewed corpus contains eight private trajectories across six agent-task families: six training trajectories across four families, one validation trajectory, and one test trajectory. The training split produced 24 continuation windows at a maximum length of 16,384, with 359,363 rendered input tokens and 162,798 supervised tokens. Across the complete corpus, 94 reviewed reasoning turns were included and 21 were masked; masking a reasoning span did not remove its associated tool-action or final-answer supervision. The corpus contains 158 admitted tool calls. No private dataset rows are included in this repository.

Training kept the Qwen3.8-27B base unquantized in BF16. Only LoRA parameters were optimized; vision and MTP modules stayed frozen. The selected step-12 checkpoint used rank 64, alpha 64, rsLoRA, dropout 0.05, a 1e-5 cosine-scheduled learning rate, 8-bit AdamW, and 16,384-token windows. It trained for 24 steps on one NVIDIA GH200. The selected validation loss improved from 0.40724 to 0.35186, which measures fit to one tiny held-out trajectory and not general quality.

Behavioral evidence

The only behavioral comparison is a small private Hermes development diagnostic. It is not a benchmark and is too small for general claims. Verifier and per-call schema-shape signals improved, while long-horizon completion and task-level tool contracts regressed. This does not establish an overall agent-quality improvement.

Limitations and license

• Quantization can change borderline token choices even when perplexity impact is small.
• Long-horizon reliability regressed in the available source-model diagnostic.
• Exact task-level tool-contract performance was worse than the pinned base.
• The private corpus is far too small to establish broad tool, language, multimodal, safety, or adversarial robustness.
• The available evaluation is directional and does not establish production readiness.
• llama.cpp behavior depends on the runtime commit, chat-template implementation, sampling, context size, GPU offload, tool parser, and surrounding agent loop.

Code and original Carnice materials are Apache-2.0. Qwen-derived model weights remain subject to the upstream Qwen license and applicable terms. Review LICENSE, NOTICE, and the upstream Qwen/Qwen3.8-27B repository before redistribution or commercial use.