Compare base Dory on matched tasks and generalize llama.cpp setup

README.md

@@ -141,12 +141,17 @@ NVFP4 uses packed E2M1 weights with E4M3 block scales and an F32 tensor scale in
 These results use **Eider NVFP4 inference with two recurrent loops and no probability calibration**.
 The 512-question test set was scored after checkpoint selection on a separate 1,024-question validation set.
 
-| Test task | Questions | Accuracy | Brier ↓ |
-|---|---:|---:|---:|
-| SCONE state tracking | 192 | 75.52% | 0.3779 |
-| Block-stacking and delivery planning | 192 | 92.19% | 0.1909 |
-| HelpSteer2 human judgements | 128 | 50.78% | 0.2290 |
-| **Combined** | **512** | **75.59%** | **0.2706** |
+Base Dory is the original NVIDIA BF16 model quantized with the same NVFP4 recipe used for this fine-tune.
+Both models use the same questions, typed-decision prompt, answer labels, two loops, and Eider executable.
+This comparison measures the effect of fine-tuning within this decision interface, rather than Dory’s performance with other prompting methods.
+[Comparison results and hashes](evaluation/base-comparison.json) record the matched setup.
+
+| Test task | Questions | Base Dory accuracy | Typed Decisions accuracy | Base Brier ↓ | Typed Decisions Brier ↓ |
+|---|---:|---:|---:|---:|---:|
+| SCONE state tracking | 192 | 29.69% | 75.52% | 1.0003 | 0.3779 |
+| Block-stacking and delivery planning | 192 | 42.71% | 92.19% | 0.6045 | 0.1909 |
+| HelpSteer2 human judgements | 128 | 47.66% | 50.78% | 0.3385 | 0.2290 |
+| **Combined** | 512 | 39.06% | 75.59% | 0.6865 | 0.2706 |
 
 Accuracy counts any answer tied for the highest target probability as correct.
 Brier error compares the complete predicted and target distributions.
@@ -157,7 +162,7 @@ They are not independent final tests. [Full evaluation records](evaluation/nvfp4
 Neither BF16 nor Q8_0 has received a separate full accuracy evaluation.
 
 The Q8_0 GGUF passed prompt and answer-label parity checks and a 24-request GPU compatibility check across all three answer types.
-On GB10, short requests with changing context had median latency of 62 ms for one question and 191 ms for three questions.
+On NVIDIA DGX Spark (GB10), short requests with changing context had median latency of 62 ms for one question and 191 ms for three questions.
 Each median used five timed requests. These small checks establish API compatibility, not general accuracy or production throughput.
 [GGUF test evidence](gguf/provenance.json) records the setup and results.
 
@@ -191,10 +196,11 @@ hf download rdaum/eider-dory-1.0-4b-decisions \
 For NVFP4, use `--include 'nvfp4/*'` instead of the GGUF filename, then point your runtime at `./dory-decisions/nvfp4`.
 For BF16, use `--include '*.json' '*.jinja' '*.safetensors' --exclude 'nvfp4/*' 'adapter/*'` instead of the GGUF filename.
 
-### Build with Dory support and CUDA
+### Build llama.cpp with Dory support
 
-These commands require Git, CMake, a C++ compiler, and the CUDA toolkit.
-They build the Dory support branch for DGX Spark / GB10:
+These commands require Git, CMake, and a C++ compiler.
+The example enables CUDA for NVIDIA GPUs and requires the CUDA toolkit.
+For CPU-only use, omit `-DGGML_CUDA=ON`. For other GPU backends, follow the [llama.cpp build instructions](https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md).
 
 ```sh
 git clone https://github.com/ggml-org/llama.cpp llama.cpp-dory
@@ -202,16 +208,13 @@ git -C llama.cpp-dory fetch origin pull/30306/head
 git -C llama.cpp-dory checkout --detach FETCH_HEAD
 cmake -S llama.cpp-dory -B llama.cpp-dory/build \
     -DCMAKE_BUILD_TYPE=Release \
-    -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121 \
+    -DGGML_CUDA=ON \
     -DLLAMA_CURL=OFF \
     -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF \
     -DLLAMA_BUILD_SERVER=ON
 cmake --build llama.cpp-dory/build --target llama-server -j 6
 ```
 
-For another NVIDIA GPU, select its CUDA architecture or omit `CMAKE_CUDA_ARCHITECTURES` for automatic detection.
-The compatibility checks below used GB10.
-
 ### Start the server
 
 ```sh
@@ -219,13 +222,11 @@ The compatibility checks below used GB10.
     --model ./dory-decisions/dory-decisions-Q8_0.gguf \
     --alias dory-decisions \
     --host 127.0.0.1 --port 18091 \
-    --ctx-size 4096 --parallel 1 \
-    --threads 4 --threads-batch 4 \
-    --batch-size 512 --ubatch-size 512 \
-    --fit off --gpu-layers all --no-warmup
+    --ctx-size 4096 --gpu-layers all
 ```
 
-The command offloads all layers to the GPU. The 4,096-token context is a small working example, not the model's context limit.
+With a GPU-enabled build, `--gpu-layers all` requests full GPU offload; reduce it if GPU memory is insufficient.
+Omit that option for CPU-only use. Adjust `--ctx-size` for your workload and available memory; 4,096 is only an example.
 
 ### Ask a typed question
 

evaluation/base-comparison.json

@@ -0,0 +1,107 @@
+{
+  "schema": "dory-base-vs-typed-decisions/v1",
+  "base_model": "nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16",
+  "base_model_revision": "b366b87f762ea334ba3eb55912031baba1515810",
+  "format": "NVFP4",
+  "quantization_recipe": "mse",
+  "loops": 2,
+  "calibrated": false,
+  "prompt": "Same typed-decision prompt and allowed-answer scoring for both models; no generated reasoning.",
+  "base_weight_sha256": "602a1bc6a69c1a5c41253df785f335cea77eeeb854b985ab11c125bf1d08a547",
+  "fine_tuned_weight_sha256": "9bc834f4dce29a48499b6614c11772d189710138b29c637a384f753b1326b4c0",
+  "binary_sha256": "935335c9df451b4b9108d65fc46882abbfd7a0b6cb20b3ba76f4e8c42fa7ab0c",
+  "scoring_script_sha256": "cbd6c833654b1fff319abf1d8f3cb491af5d96b78d541e593a283b684268da04",
+  "test_data_sha256": "e2ca7413a306fc2b0e9e044aa685c94d5aa8e0def7cfcec6448941b7369ae625",
+  "base": {
+    "questions": 512,
+    "modal_accuracy": 0.390625,
+    "brier": 0.6864560213896311,
+    "cross_entropy": 2.226203713663419,
+    "by_kind": {
+      "choice": {
+        "questions": 241,
+        "modal_accuracy": 0.33195020746887965,
+        "brier": 0.7223381484168857,
+        "cross_entropy": 2.451415271076999
+      },
+      "noul": {
+        "questions": 94,
+        "modal_accuracy": 0.5,
+        "brier": 0.8460049545207761,
+        "cross_entropy": 1.360189755992052
+      },
+      "score": {
+        "questions": 177,
+        "modal_accuracy": 0.4124293785310734,
+        "brier": 0.55286736416988,
+        "cross_entropy": 2.3794767457788746
+      }
+    },
+    "by_source": {
+      "helpsteer2": {
+        "questions": 128,
+        "modal_accuracy": 0.4765625,
+        "brier": 0.3385301710123469,
+        "cross_entropy": 1.8640141950785272
+      },
+      "planning": {
+        "questions": 192,
+        "modal_accuracy": 0.4270833333333333,
+        "brier": 0.6045360602682238,
+        "cross_entropy": 1.818707715277242
+      },
+      "scone": {
+        "questions": 192,
+        "modal_accuracy": 0.296875,
+        "brier": 1.000326549429228,
+        "cross_entropy": 2.8751593911061906
+      }
+    }
+  },
+  "fine_tuned": {
+    "questions": 512,
+    "modal_accuracy": 0.755859375,
+    "brier": 0.2705710538872297,
+    "cross_entropy": 1.0354930422480668,
+    "by_kind": {
+      "choice": {
+        "questions": 241,
+        "modal_accuracy": 0.8215767634854771,
+        "brier": 0.30593746240810665,
+        "cross_entropy": 1.0288481446263855
+      },
+      "noul": {
+        "questions": 94,
+        "modal_accuracy": 0.8936170212765957,
+        "brier": 0.1853453372075174,
+        "cross_entropy": 0.4718470608079094
+      },
+      "score": {
+        "questions": 177,
+        "modal_accuracy": 0.5932203389830508,
+        "brier": 0.26767790651074164,
+        "cross_entropy": 1.34387802858818
+      }
+    },
+    "by_source": {
+      "helpsteer2": {
+        "questions": 128,
+        "modal_accuracy": 0.5078125,
+        "brier": 0.22898458428820745,
+        "cross_entropy": 1.5346420909767997
+      },
+      "planning": {
+        "questions": 192,
+        "modal_accuracy": 0.921875,
+        "brier": 0.1909474730913934,
+        "cross_entropy": 0.8915435249464682
+      },
+      "scone": {
+        "questions": 192,
+        "modal_accuracy": 0.7552083333333334,
+        "brier": 0.3779189477490808,
+        "cross_entropy": 0.8466765270638437
+      }
+    }
+  }
+}