Add CPU-verified decision-only serving kit and container usage

MANIFEST.json

@@ -57,8 +57,8 @@
     },
     {
       "path": "README.md",
-      "size_bytes": 16076,
-      "sha256": "8e2af48820820a6435534cf6814f3eb32eca33b235550701b2729675faf8553f"
+      "size_bytes": 16550,
+      "sha256": "d60c1d98cae65abb179ee03a25b5d45509d16b252ca672d2881de84873ea13a6"
     },
     {
       "path": "config.json",
@@ -120,6 +120,51 @@
       "size_bytes": 33011,
       "sha256": "0ce91858995441a7b2c8fef65d9b0da1ba7135ced82c651a2890bf1802997c87"
     },
+    {
+      "path": "serving/.dockerignore",
+      "size_bytes": 51,
+      "sha256": "c1e2d5a66114a731106f2ac7c17dcf5f3841ccfbd82f5833b106c5862f9ab48a"
+    },
+    {
+      "path": "serving/Dockerfile",
+      "size_bytes": 620,
+      "sha256": "4db38457f7a17f860590b22dc3a02796048e60b41455d270bcdd86eb8f52f77d"
+    },
+    {
+      "path": "serving/SERVING.md",
+      "size_bytes": 5246,
+      "sha256": "9be604a955acf5f63fee59d81a6fb3430cddae32ec34eac3a3b19b3be6a115c6"
+    },
+    {
+      "path": "serving/TESTS.json",
+      "size_bytes": 4347,
+      "sha256": "cb277820be7e8c3277b450e6f3bba3ed216873674d02d431ea89775a3ceadd5e"
+    },
+    {
+      "path": "serving/request.json",
+      "size_bytes": 416,
+      "sha256": "b09ffbcfbf0525c83de758e032113fcfd269d2be82a112de3e85ec8af6f64fbd"
+    },
+    {
+      "path": "serving/requirements.txt",
+      "size_bytes": 209,
+      "sha256": "f70fac27c587d235a62549688a8a61c089eca728afb190af6ad02cae1e7c6407"
+    },
+    {
+      "path": "serving/run.py",
+      "size_bytes": 732,
+      "sha256": "28060c2e5bd33afc9848236fcd6ade36e026809def9510ab1fce891eef29243b"
+    },
+    {
+      "path": "serving/serve.py",
+      "size_bytes": 3074,
+      "sha256": "635070fdbe2bb70ddfe3e75f4555c7760ad12e7cb3741660c7d9e714e27459f6"
+    },
+    {
+      "path": "serving/test_serving.py",
+      "size_bytes": 4852,
+      "sha256": "e79aa7ddf6da7890abed5e5a55c8e52169cec5769e156601364fe8b7b8974795"
+    },
     {
       "path": "tokenizer.json",
       "size_bytes": 19989325,

README.md

@@ -127,6 +127,15 @@ four-decimal probability rounding, then residual calibration.
 
 ## Usage
 
+### Decision API And Container
+
+See [serving/SERVING.md](serving/SERVING.md) for the decision-only FastAPI
+server, CLI, CPU/GPU Docker commands, private HF startup download and tests.
+The endpoint is `POST /v1/systemone`; health is `GET /health`.
+Image: `registry.omaemirates.com/sultan/saif/saif-4b-serving:0.1.0`.
+Weights are mounted or downloaded, never baked into the image. Only CPU-tested;
+GPU execution is expected but unverified. No text generation or chat routes.
+
 Use Python 3.12, PyTorch, Transformers with Qwen3.5 support, and safetensors.
 PEFT is not needed to load the merged release. Do not install Unsloth for this
 loader. The local verification environment has an incompatible optional

serving/.dockerignore

@@ -0,0 +1,5 @@
+**
+!Dockerfile
+!requirements.txt
+!serve.py
+!run.py

serving/Dockerfile

@@ -0,0 +1,11 @@
+FROM pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime
+WORKDIR /app
+COPY requirements.txt .
+RUN python -m pip uninstall -y torchao && python -m pip install --no-cache-dir -r requirements.txt
+RUN useradd --create-home --uid 10001 saif
+COPY serve.py run.py ./
+USER saif
+ENV SAIF_DEVICE=cpu SAIF_THREADS=4 PORT=8000 HF_HOME=/home/saif/.cache/huggingface PYTHONDONTWRITEBYTECODE=1
+EXPOSE 8000
+HEALTHCHECK --interval=30s --timeout=5s --start-period=180s --retries=3 CMD python -c "import os,urllib.request; urllib.request.urlopen('http://127.0.0.1:'+os.environ.get('PORT','8000')+'/health',timeout=4)"
+CMD ["python", "serve.py"]

serving/SERVING.md

@@ -0,0 +1,102 @@
+# Serving Saif 4B Decisions
+
+This is a decision-only service, not a chat server. `POST /v1/systemone`
+accepts `model`, any JSON `state`, and 1-64 typed `questions`. It returns
+`model: saif-4b`, `answers` keyed by question ID, and token `usage` with zero
+output tokens. The optional input model label is accepted for DI client
+compatibility; this service always runs Saif 4B. `GET /health` becomes available
+after the model has loaded. Validation and complete-record context overflow
+return HTTP 422. No text-generation routes are registered.
+
+`noul` returns `noul` (probability of true) and a probability distribution;
+`choice` returns `choice`, `confidence`, and `probabilities`; `score` returns
+the expected zero-based `score`, `confidence`, `legend`, and `probabilities`.
+The loader's additional `type` and `answer` fields are preserved. Question
+instructions are optional. Choice criteria are a mapping of at least two
+option IDs to descriptions; score criteria are a list of at least two level
+descriptions. Noul criteria optionally override `true` and `false` descriptions.
+Images, video, chat messages and generation parameters are rejected.
+
+## Python And CLI
+
+Use a clean Python 3.12 environment and `pip install -r requirements.txt`.
+The torch, transformers and safetensors versions match the CPU equivalence
+environment. **Exclude torchao**: the equivalence environment's optional
+torchao installation is incompatible and is not needed for bf16 inference.
+PEFT and Unsloth are not required. The CUDA wheel also supports CPU execution.
+
+```bash
+SAIF_MODEL_PATH=/path/to/saif-4b SAIF_DEVICE=cpu CUDA_VISIBLE_DEVICES= python run.py request.json
+SAIF_MODEL_PATH=/path/to/saif-4b SAIF_DEVICE=cpu CUDA_VISIBLE_DEVICES= python serve.py
+```
+
+```python
+import sys
+import torch
+sys.path.insert(0, '/path/to/saif-4b')
+from joint_schema_model import load
+torch.set_num_threads(4)
+model = load('/path/to/saif-4b')
+print(model.decide('The delivery was excellent.', {
+    'satisfied': {'type': 'noul', 'instructions': 'Is the customer satisfied?'}
+})['answers'])
+```
+
+## Docker
+
+The image contains the server and dependencies, **not the weights**. Model
+directories contain executable custom Python: mount only a trusted release.
+The process defaults to UID 10001; mounted weights must be readable by that
+user. For owner-only (0600) local weights, use the non-root owner UID/GID as
+below rather than making private weights world-readable. Omit `--user` when
+the default user already has access. Startup downloads use the default user.
+
+```bash
+IMAGE=registry.omaemirates.com/sultan/saif/saif-4b-serving:0.1.0
+# CPU
+docker run --rm --user "$(id -u):$(id -g)" -p 8000:8000 -e CUDA_VISIBLE_DEVICES= -e SAIF_DEVICE=cpu \
+  -e SAIF_MODEL_PATH=/model -v /absolute/path/to/saif-4b:/model:ro "$IMAGE"
+# GPU: requires NVIDIA Container Toolkit on the host
+docker run --rm --user "$(id -u):$(id -g)" --gpus all -p 8000:8000 -e SAIF_DEVICE=cuda \
+  -e SAIF_MODEL_PATH=/model -v /absolute/path/to/saif-4b:/model:ro "$IMAGE"
+# Private HF download at startup; HF_TOKEN must already be exported securely
+docker run --rm -p 8000:8000 -e HF_TOKEN -e SAIF_HF_REPO=GyrTechnology/saif-4b \
+  -e SAIF_DEVICE=cpu "$IMAGE"
+```
+
+`SAIF_MODEL_PATH` takes precedence over `SAIF_HF_REPO`. With only the latter
+set, startup uses `huggingface_hub.snapshot_download` and `HF_TOKEN`.
+`SAIF_DEVICE` is `cpu` (default) or `cuda`; `SAIF_THREADS` defaults to 4;
+`PORT` defaults to 8000. The image healthcheck follows `PORT`. A download can
+take longer than the initial healthcheck grace period; readiness follows load.
+Run one worker per model replica. Inference requests are serialized to bound
+concurrent memory use. Put authentication/TLS at a trusted reverse proxy;
+the service itself does not authenticate requests.
+
+```bash
+curl http://localhost:8000/v1/systemone -H 'Content-Type: application/json' \
+  -d '{"model":"saif-4b","state":"The delivery was excellent.","questions":{"satisfied":{"type":"noul","instructions":"Is the customer satisfied?"}}}'
+```
+
+## Compatibility And Verification
+
+Compatible options: **this server**; **Triton's Python backend (advanced)**
+using the custom loader and decision head (no Triton configuration is bundled).
+
+**Not supported: Ollama, llama.cpp/GGUF, vLLM, SGLang, LM Studio, TGI.**
+These are text-generation engines, and this model is decision-only with a
+custom head. Clef's specialized SGLang recipe does not establish support for
+this Saif release. There is no LM head or open-ended chat API here.
+
+Only CPU-tested so far. GPU execution is expected to work but is **not
+verified**. The local GPU is training and was not used; local Docker has only
+runc, without an NVIDIA runtime. See `TESTS.json` for exact-match checks,
+malformed-input checks and measured CPU request latency, not a GPU benchmark.
+
+Protocol references checked: DI `decision_index/engines/http.py` and Clef's
+published README revision `0b331204bb13fbd2ca93a64df1956f5b55478ce5`.
+
+To reproduce: run `test_serving.py reference` with `SAIF_MODEL_PATH` set, start
+the server, then run `test_serving.py local --url http://127.0.0.1:8000`.
+Repeat against the mounted-weight container with mode `container`. All test
+commands require `CUDA_VISIBLE_DEVICES= SAIF_DEVICE=cpu`.

serving/TESTS.json

@@ -0,0 +1,166 @@
+{
+  "reference": [
+    {
+      "model": "saif-4b",
+      "answers": {
+        "satisfied": {
+          "type": "noul",
+          "probabilities": {
+            "true": 0.9977252035331371,
+            "false": 0.0022747964668629324
+          },
+          "noul": 0.9977252035331371,
+          "answer": true
+        }
+      },
+      "usage": {
+        "input_tokens": 154,
+        "output_tokens": 0
+      }
+    },
+    {
+      "model": "saif-4b",
+      "answers": {
+        "intent": {
+          "type": "choice",
+          "probabilities": {
+            "complaint": 0.0014628375324945344,
+            "praise": 0.9985371624675055
+          },
+          "choice": "praise",
+          "answer": "praise",
+          "confidence": 0.9985371624675055
+        }
+      },
+      "usage": {
+        "input_tokens": 141,
+        "output_tokens": 0
+      }
+    },
+    {
+      "model": "saif-4b",
+      "answers": {
+        "sentiment": {
+          "type": "score",
+          "probabilities": {
+            "0": 0.0029920151680415817,
+            "1": 0.06401839839575167,
+            "2": 0.9329895864362067
+          },
+          "score": 1.929997571268165,
+          "answer": 2,
+          "confidence": 0.9329895864362067,
+          "legend": {
+            "0": "negative",
+            "1": "neutral",
+            "2": "positive"
+          }
+        }
+      },
+      "usage": {
+        "input_tokens": 152,
+        "output_tokens": 0
+      }
+    }
+  ],
+  "versions": {
+    "torch": "2.8.0+cu128",
+    "transformers": "5.17.0",
+    "safetensors": "0.8.0",
+    "fastapi": "0.142.2",
+    "uvicorn": "0.54.0",
+    "pydantic": "2.14.1",
+    "huggingface_hub": "1.33.0",
+    "accelerate": "1.14.0"
+  },
+  "gpu_tested": false,
+  "gpu_note": "CPU only; local Docker has runc only, no NVIDIA runtime.",
+  "routes": [
+    "/docs",
+    "/docs/oauth2-redirect",
+    "/health",
+    "/openapi.json",
+    "/redoc",
+    "/v1/systemone"
+  ],
+  "no_generation_routes": true,
+  "64_question_schema_accepted": true,
+  "local": {
+    "passed": true,
+    "requests": [
+      {
+        "type": "noul",
+        "exact_match": true,
+        "probabilities_sum_one": true,
+        "latency_seconds": 2.8982914260122925
+      },
+      {
+        "type": "choice",
+        "exact_match": true,
+        "probabilities_sum_one": true,
+        "latency_seconds": 2.7930272528901696
+      },
+      {
+        "type": "score",
+        "exact_match": true,
+        "probabilities_sum_one": true,
+        "latency_seconds": 1.9185104710049927
+      }
+    ],
+    "malformed_422_count": 9,
+    "generation_routes_404": true,
+    "python_version": "3.12.14"
+  },
+  "di_client_live_passed": true,
+  "64_question_live": {
+    "passed": true,
+    "latency_seconds": 91.86650505103171
+  },
+  "cli": {
+    "happy_path_passed": true,
+    "help_passed": true,
+    "missing_file_exit_code": 2
+  },
+  "lsp": "Unavailable: basedpyright not installed; installation previously declined",
+  "container": {
+    "passed": true,
+    "requests": [
+      {
+        "type": "noul",
+        "exact_match": true,
+        "probabilities_sum_one": true,
+        "latency_seconds": 3.9418553828727454
+      },
+      {
+        "type": "choice",
+        "exact_match": true,
+        "probabilities_sum_one": true,
+        "latency_seconds": 1.384989856975153
+      },
+      {
+        "type": "score",
+        "exact_match": true,
+        "probabilities_sum_one": true,
+        "latency_seconds": 1.695539910113439
+      }
+    ],
+    "malformed_422_count": 9,
+    "generation_routes_404": true,
+    "user": "1000:1000",
+    "health": "healthy",
+    "python_version": "3.11 (official PyTorch base)",
+    "versions_match_local": true,
+    "initial_startup": "Default UID could not read owner-only 0600 weights. Re-run with non-root weight-owner UID:GID succeeded; documented --user flag. Permissions unchanged."
+  },
+  "docker_build": {
+    "passed": true,
+    "image_id": "sha256:576663ba2b8710c2906afdc7a6466a8f9de8003791225f124ff6ab1fa0aa3d76",
+    "default_user": "saif",
+    "base": "pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime",
+    "weights_baked_in": false
+  },
+  "observations": [
+    "Existing release tokenizer emits incorrect-regex warning; loader/tokenizer unchanged to preserve verified model outputs.",
+    "HF startup-download path implemented, not exercised with a full redundant weight download."
+  ]
+}

serving/request.json

@@ -0,0 +1 @@
+{"model":"saif-4b","state":{"message":"The delivery arrived on time and works well."},"questions":{"satisfied":{"type":"noul","instructions":"Is the customer satisfied?"},"intent":{"type":"choice","instructions":"Classify the message.","criteria":{"praise":"Positive feedback","complaint":"A problem"}},"sentiment":{"type":"score","instructions":"Rate the sentiment.","criteria":["negative","neutral","positive"]}}}

serving/requirements.txt

@@ -0,0 +1,9 @@
+--extra-index-url https://download.pytorch.org/whl/cu128
+torch==2.8.0+cu128
+transformers==5.17.0
+safetensors==0.8.0
+fastapi==0.142.2
+uvicorn==0.54.0
+pydantic==2.14.1
+huggingface_hub==1.33.0
+accelerate==1.14.0

serving/run.py

@@ -0,0 +1,24 @@
+"""Run a typed decision request from a JSON file."""
+import argparse
+import json
+
+from pydantic import ValidationError
+
+from serve import DecisionRequest, load_model
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("request", help="System One request JSON file")
+    args = parser.parse_args()
+    try:
+        with open(args.request) as stream:
+            request = DecisionRequest.model_validate(json.load(stream))
+    except (OSError, ValueError, ValidationError) as error:
+        parser.error(str(error))
+    result = load_model().decide(request.state, request.question_dict())
+    print(json.dumps(result["answers"], ensure_ascii=False))
+
+
+if __name__ == "__main__":
+    main()

serving/serve.py

@@ -0,0 +1,102 @@
+"""Decision-only System One / Jev HTTP service."""
+import importlib.util
+import os
+import sys
+from contextlib import asynccontextmanager
+from pathlib import Path
+from threading import Lock
+from typing import Annotated, Literal
+
+from fastapi import FastAPI, HTTPException
+from pydantic import BaseModel, ConfigDict, Field, JsonValue
+
+
+class Question(BaseModel):
+    model_config = ConfigDict(extra="forbid", strict=True)
+    instructions: str | None = None
+
+
+class Noul(Question):
+    type: Literal["noul"]
+    criteria: dict[Literal["true", "false"], str] | None = None
+
+
+class Choice(Question):
+    type: Literal["choice"]
+    criteria: dict[str, str] = Field(min_length=2)
+
+
+class Score(Question):
+    type: Literal["score"]
+    criteria: list[str] = Field(min_length=2)
+
+
+class DecisionRequest(BaseModel):
+    model_config = ConfigDict(extra="forbid", strict=True)
+    model: str = "saif-4b"
+    state: JsonValue
+    questions: dict[str, Annotated[Noul | Choice | Score, Field(discriminator="type")]] = Field(
+        min_length=1, max_length=64
+    )
+
+    def question_dict(self):
+        return {key: question.model_dump(exclude_none=True) for key, question in self.questions.items()}
+
+
+def load_model():
+    import torch
+
+    device = os.environ.get("SAIF_DEVICE", "cpu")
+    if device not in ("cpu", "cuda"):
+        raise ValueError("SAIF_DEVICE must be cpu or cuda")
+    threads = int(os.environ.get("SAIF_THREADS", "4"))
+    if threads < 1:
+        raise ValueError("SAIF_THREADS must be positive")
+    torch.set_num_threads(threads)
+    path = os.environ.get("SAIF_MODEL_PATH")
+    if not path and os.environ.get("SAIF_HF_REPO"):
+        from huggingface_hub import snapshot_download
+
+        path = snapshot_download(os.environ["SAIF_HF_REPO"], token=os.environ.get("HF_TOKEN"))
+    if not path:
+        raise ValueError("Set SAIF_MODEL_PATH or SAIF_HF_REPO")
+    path = Path(path).resolve()
+    spec = importlib.util.spec_from_file_location("saif_release_model", path / "joint_schema_model.py")
+    module = importlib.util.module_from_spec(spec)
+    sys.modules[spec.name] = module
+    spec.loader.exec_module(module)
+    return module.load(str(path)).to(device=device)
+
+
+@asynccontextmanager
+async def lifespan(application):
+    application.state.model = load_model()
+    application.state.lock = Lock()
+    yield
+    del application.state.model
+
+
+app = FastAPI(title="Saif 4B Decisions", version="0.1.0", lifespan=lifespan)
+
+
+@app.get("/health")
+def health():
+    return {"status": "ok", "model": "saif-4b"}
+
+
+@app.post("/v1/systemone")
+def decide(request: DecisionRequest):
+    try:
+        with app.state.lock:
+            return app.state.model.decide(request.state, request.question_dict())
+    except ValueError as error:
+        detail = str(error)
+        if "Complete record exceeds" in detail:
+            detail = "maximum context length: " + detail
+        raise HTTPException(status_code=422, detail=detail) from error
+
+
+if __name__ == "__main__":
+    import uvicorn
+
+    uvicorn.run(app, host="0.0.0.0", port=int(os.environ.get("PORT", "8000")))

serving/test_serving.py

@@ -0,0 +1,97 @@
+"""CPU-only live HTTP checks; generate reference once, then check each server."""
+import argparse
+import importlib.metadata
+import json
+import math
+import os
+from pathlib import Path
+import time
+import urllib.error
+import urllib.request
+
+os.environ["CUDA_VISIBLE_DEVICES"] = ""
+os.environ["SAIF_DEVICE"] = "cpu"
+
+from serve import DecisionRequest, app, load_model
+
+ROOT = Path(__file__).resolve().parent
+
+
+def requests():
+    example = json.loads((ROOT / "request.json").read_text())
+    return [dict(example, questions={key: value}) for key, value in example["questions"].items()]
+
+
+def post(url, payload):
+    request = urllib.request.Request(url + "/v1/systemone", data=json.dumps(payload).encode(),
+                                     headers={"Content-Type": "application/json"})
+    with urllib.request.urlopen(request, timeout=600) as response:
+        return json.load(response)
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("mode", choices=["reference", "local", "container"])
+    parser.add_argument("--url", default="http://127.0.0.1:18000")
+    args = parser.parse_args()
+    target = ROOT / "TESTS.json"
+    report = json.loads(target.read_text()) if target.exists() else {}
+    if args.mode == "reference":
+        model = load_model()
+        outputs = [model.decide(item["state"], item["questions"]) for item in requests()]
+        report["reference"] = json.loads(json.dumps(outputs))
+        report["versions"] = {name: importlib.metadata.version(name) for name in
+                              ["torch", "transformers", "safetensors", "fastapi", "uvicorn",
+                               "pydantic", "huggingface_hub", "accelerate"]}
+        report["gpu_tested"] = False
+        report["gpu_note"] = "CPU only; local Docker has runc only, no NVIDIA runtime."
+        report["routes"] = sorted(route.path for route in app.routes)
+        assert set(report["routes"]) == {"/health", "/v1/systemone", "/docs", "/docs/oauth2-redirect", "/redoc", "/openapi.json"}
+        report["no_generation_routes"] = True
+        DecisionRequest.model_validate(dict(requests()[0], questions={str(index): {"type": "noul"} for index in range(64)}))
+        report["64_question_schema_accepted"] = True
+    else:
+        with urllib.request.urlopen(args.url + "/health") as response:
+            assert json.load(response) == {"status": "ok", "model": "saif-4b"}
+        results = []
+        for payload, expected in zip(requests(), report["reference"]):
+            start = time.perf_counter()
+            actual = post(args.url, payload)
+            elapsed = time.perf_counter() - start
+            assert actual == expected, (actual, expected)
+            for answer in actual["answers"].values():
+                probabilities = list(answer["probabilities"].values())
+                assert all(math.isfinite(value) and 0 <= value <= 1 for value in probabilities)
+                assert math.isclose(sum(probabilities), 1, abs_tol=1e-12)
+            results.append({"type": next(iter(payload["questions"].values()))["type"],
+                            "exact_match": True, "probabilities_sum_one": True, "latency_seconds": elapsed})
+        malformed = [{}, {"state": "x", "questions": {}},
+                     {"state": "x", "questions": {"q": {"type": "text"}}},
+                     {"state": "x", "questions": {"q": {"type": "choice", "criteria": []}}},
+                     {"state": "x", "questions": {"q": {"type": "score", "criteria": {"a": "x"}}}},
+                     {"state": "x", "questions": {"q": {"type": "noul", "criteria": {"bad": "x"}}}},
+                     dict(requests()[0], images=["unsupported"]),
+                     dict(requests()[0], questions={str(index): {"type": "noul"} for index in range(65)}),
+                     dict(requests()[0], state="word " * 17000)]
+        for payload in malformed:
+            try:
+                post(args.url, payload)
+            except urllib.error.HTTPError as error:
+                assert error.code == 422, error.code
+            else:
+                raise AssertionError("Malformed input was accepted")
+        for route in ("/v1/chat/completions", "/v1/completions", "/generate"):
+            try:
+                urllib.request.urlopen(urllib.request.Request(args.url + route, data=b"{}"))
+            except urllib.error.HTTPError as error:
+                assert error.code == 404
+            else:
+                raise AssertionError("Unexpected generation endpoint")
+        report[args.mode] = {"passed": True, "requests": results, "malformed_422_count": len(malformed),
+                             "generation_routes_404": True}
+    target.write_text(json.dumps(report, indent=2) + "\n")
+    print(json.dumps(report.get(args.mode, {"reference": "passed"}), indent=2))
+
+
+if __name__ == "__main__":
+    main()