ARCHIVE_API.md

Authenticated archive inference API

The public search demo and /api/v1/search/ keep their existing roles. The machine endpoints below are stateless and do not index requests into the public Qdrant corpus, save drafts, or retain query text. Search logs no longer record query strings. HF infrastructure access logs are outside this application's control; do not put source text or credentials in URLs.

Authentication and boundaries

Set ARCHIVE_INFERENCE_API_KEY in this Space's Secrets to a randomly generated value of at least 32 characters. Set the identical value as KNOWLEDGE_ENGINE_API_KEY on the private archive-core Space. Do not put either secret into browser JavaScript or source control. Requests use Authorization: Bearer <secret> over HTTPS. Missing/short configuration fails closed with 503; absent/incorrect credentials return 401.

The public demo remains public. This bearer key is application-level machine authentication, not HF private networking or an origin allowlist. Rotate both ends together. The HTTP body is bounded before JSON parsing. Responses are Cache-Control: no-store; validation/provider errors do not echo source input.

Versioned endpoints

POST /api/v1/archive/embeddings
{"texts":["Synthetic retrieval example"],"model":"BAAI/bge-m3"}
→ {"model":"BAAI/bge-m3@5617a9f61b028005a4858fdac845db406aefb181:fp32:unit-cls-v1","dimension":1024,
   "embeddings":[{"dense":[...],"sparse":{"123":0.25}}]}

POST /api/v1/archive/translate
{"texts":["Synthetic translation example"],"source_locale":"en","target_locale":"ko"}
→ {"model":"configured-provider-model","translations":["..."]}

Embedding model selection is an allowlist for the already loaded singleton, not an instruction to download a model. Sparse keys are original BGE-M3 vocabulary IDs, not re-tokenized numeric strings. Dense and sparse are provided; ColBERT/multi-vector or multimodal embeddings are not implemented by this API. EMBEDDER_REVISION pins the existing BGE model's snapshot at normal startup; the response's compatibility ID records model revision, precision and normalized CLS pooling. Archive must compare this ID and rebuild its published-document vectors if it changes, rather than mixing embeddings from different models. The default remains the same BGE-M3 checkpoint family used by the public demo; its prebuilt corpus has no per-vector revision provenance and must be separately checked before changing the public demo model or quantization settings.

Limits: 1–64 nonempty texts, each at most 8,000 characters, 64,000 total UTF-8 bytes, 512,000 HTTP body bytes. BGE-M3 machine inference uses at most 512 model tokens per segment and rejects longer input rather than silently truncating. Archive should chunk long documents (for Korean, about 900 characters is a conservative starting point, not a guaranteed tokenizer bound). Output dimensions are limited to 4,096 and sparse entries to 8,192. One CPU operation runs at once, at most two are outstanding by default. Full queues return 429 with retry advice. Timeouts return 504 but keep their slot occupied until the underlying CPU work finishes. This is a bound, not hard process-level model cancellation.

Translation configuration

Default: TRANSLATION_PROVIDER=disabled. No local translation model is loaded, downloaded or paid API called by default. Translation outputs are proposals; archive saves them as drafts for human review and explicit publication. Markdown/code/math preservation belongs to archive's segmented worker pipeline, not to arbitrary instructions sent through this endpoint.

Compact CPU baseline (marian)

After reviewing the model licenses, set TRANSLATION_LICENSE_ACKNOWLEDGED=true. TRANSLATION_ALLOW_DOWNLOADS=false uses only an existing local model cache; enabling downloads is an explicit operator decision. Only one translation model is resident at a time, loaded lazily on CPU alongside the existing BGE models.

Pair	Default model / fixed revision	Official model-card license
ko → en	Helsinki-NLP/opus-mt-ko-en, e42d1f41b66194e6d10512f8a27bebc1f4f5097e	Apache-2.0
en → ko	Helsinki-NLP/opus-mt-tc-big-en-ko, ae8606b7b29a495f31ce679cee2007f536a3a5ce	CC-BY-4.0; retain attribution

The older ko→en checkpoint publishes PyTorch .bin weights, not safetensors. Only the reviewed fixed revision is used; trust_remote_code is false. This uses the existing torch 2.9.1 runtime, which includes modern guarded weight loading. Operators preferring exclusively safetensors may select a separately reviewed safe checkpoint and its exact revision. Never accept arbitrary model IDs from an HTTP caller.

These are compact, established baselines, not a claim of current SOTA quality on technical writing. They have a 512-token segment limit, cannot assure perfect terminology, and require real Korean/English evaluation before production acceptance. No performance or combined RAM budget has been certified here. The existing BGE-M3 and reranker remain loaded; optional translation competes for the same CPU and memory.

Model attribution: Language Technology Research Group at the University of Helsinki, OPUS-MT project. The en→ko checkpoint is the 2022-07-28 transformer-big release converted for Hugging Face Transformers. Preserve its creator credit, model-card URL and CC-BY-4.0 license when distributing translated derivatives; disclose machine-assisted translation and later editorial changes. Review the original model files' license notices when redistributing weights. The source project's Apache license does not replace a model's separate license.

NLLB-200 distilled 600M is not an automatic default: its official card specifies CC-BY-NC and research—not production document translation—use. NLLB model card TranslateGemma 4B is a separate future option requiring acceptance of Gemma terms and runtime/hardware evaluation; this change does not download it, accept terms or claim it runs in the current CPU image. TranslateGemma model card

Operator-selected HTTPS API (openai)

Set TRANSLATION_OPENAI_BASE_URL (including /v1 if needed), TRANSLATION_OPENAI_MODEL, and secret TRANSLATION_OPENAI_API_KEY. The endpoint must use HTTPS and no URL credentials, query or fragment. It is called server-side with bounded timeouts, no redirects and no logging of tokens or text. Provider billing, retention, transfer jurisdiction and quality must be reviewed by the operator before enabling it. This configuration does not imply that the API is free. The model and translations are returned for provenance.

Local verification

python -m pytest -q

Tests use synthetic text, mocked models/providers and no DB/corpus/real tokens. They establish contract, input/auth limits, no payload in errors and bounded timeout behavior. They do not establish model quality, real Space capacity, remote deployment or actual published search correctness.