# Profiled Bounds Implementation Plan ## Purpose Local Frontier should not compute serious model bounds from scraped Hugging Face metadata alone. The long-term system should be evidence-backed, reproducible, and fail closed. The target architecture is: ```text scraped model catalog -> discovery and display only audited profile registry -> self-contained model profiles pure bounds engine -> hardware + model profile + workload settings -> derived batch, ceilings, statuses, and trace ``` The core rule is simple: no audited self-contained model profile means no serious tok/s bound. ## Source Contract This plan follows the bounds contract in: ```text https://github.com/osolmaz/onurclaw/blob/main/docs/2026-06-30-local-frontier-model-bounds.md ``` The engine must implement the adapter contract from that note: ```text Adapter(M) = [ W_resident, W_batch(b), K_alloc(L_alloc), K_read(L_read), rho ] ``` The calculator must derive concurrency. It must not expose batch/concurrency as a primary user-controlled input. ## Core Concepts ### Model Profile A model profile is one self-contained file for one concrete Hugging Face repo. It embeds all model data needed by the bounds engine: - repo id - profile status - architecture data - weight traffic adapter - KV/state traffic adapter - weight precision - KV store precision - KV read precision - quantization or packaging assumptions - base-model or artifact relation evidence, if any - review status A quantized artifact gets its own model profile because serving precision is part of the runtime input. A fine-tune gets its own model profile even when its embedded architecture data matches the base model. ### Embedded Reuse Model profiles are denormalized production artifacts. They do not point to separate runtime `architecture` or `serving` files. Reuse happens through: - shared adapter kinds and formulas - schema definitions - audit packet generation - reviewer tooling that copies known-good architecture or serving blocks into many self-contained model files Coverage rule: - every scraped model repo must eventually have one model profile file - a model profile may reuse copied architecture or serving blocks when evidence supports that reuse - a model profile may mark the repo unsupported with a concrete reason - production code must never infer a missing model profile from repo name, model name, parameter count, or quantization label - shared formulas reduce duplicated logic; they do not replace per-repo files Current registry sizing target from the 2026-07-01 scrape: - 654 self-contained model profile files, one per scraped model row - about 324 reusable architecture data shapes after exact structural deduplication, copied into model profiles as needed - about 29 reusable serving data shapes after precision/representation deduplication, copied into model profiles as needed - default workload settings live in app configuration, not in profile files These counts are implementation targets for the current scrape, not schema constants. Validation should report drift when the scraped catalog changes. ### Workload Settings Workload settings are user or app configuration, not saved profile files: - reserved context, `L_alloc` - active read context, `L_read` - minimum useful per-session rate, `r_star` - decode policy, initially ordinary decode with `rho = 1` The app should keep default workload settings in code or normal app config. Users may change them through the compare UI. They do not belong under `profiles/`. Speculative decode is a future workload setting. It can allow `rho > 1`, but only with explicit draft/verifier traffic. Bounds Engine v1 should reject `rho > 1` rather than treating raw `rho` as a free multiplier. ## Adapter Axes The profile system should be compositional. ```text model profile = architecture block + serving block + evidence + review/status architecture block = weight traffic adapter + KV/state traffic adapter serving block = precision and runtime representation workload settings = context lengths, floor, decode policy ``` ### Weight Traffic Adapters Initial supported weight adapters: 1. `dense` - all weights are resident - all weights are swept each decode step - `W_resident = P_total * weight_bytes` - `W_batch(b) = W_resident` 2. `dense_resident_swept` - all stored weights are resident - only the audited swept subset is charged each decode step - intended for packages such as multimodal checkpoints where a vision tower is resident but not swept for each generated text token - `W_resident = P_resident * weight_bytes` - `W_batch(b) = P_swept * weight_bytes` - requires evidence for the stored resident parameter count and swept parameter count 3. `moe_distinct_experts` - all experts are resident - per-batch weight traffic grows with expected distinct routed experts - requires total params, active params, routed expert count, routed experts per token, and optional shared-expert documentation 4. `moe_distinct_experts_exact` - all stored weights are resident - per-batch routed expert traffic grows with expected distinct routed experts - fixed and per-expert traffic are provided directly in GB - intended for audited mixed-dtype MoE checkpoints where a single `weight_bytes_per_param` would hide side-tensor overhead The list is not final. Future adapters can include nonuniform MoE, shared-expert variants, mixture-of-depth, early-exit, external-memory, or other structures. Unsupported structures must remain unsupported until a new audited adapter exists. ### KV / State Traffic Adapters Initial supported KV/state adapters: 1. `full_context` - KV allocation and read traffic scale linearly with context length 2. `layered_kv` - combines components such as full-context layers and sliding-window layers - preferred representation for hybrid local/global attention 3. `recurrent_state` - fixed state instead of full KV growth 4. `compressed_state` - latent, sparse, compressed, or other non-full-KV state models The durable abstraction should be a sum of KV components, not a growing list of one-off family enums. Example: ```json { "kind": "layered_kv", "components": [ { "kind": "full_context", "layers": 10, "kv_heads": 8, "head_dim": 256 }, { "kind": "sliding_window", "layers": 20, "kv_heads": 8, "head_dim": 256, "window_tokens": 4096 } ] } ``` ### Precision / Serving Blocks Initial serving block families: - BF16 or FP16 weights, BF16 or FP16 KV - FP8 weights, BF16 or FP16 KV - NVFP4 or MXFP4 weights, BF16 or FP16 KV - INT8 weights, BF16 or FP16 KV - INT4, Q4, AWQ, GPTQ, GGUF, or MLX weights with explicit KV precision - Q2-style mixed profiles where evidence supports the assumption Weight precision and KV precision must be separate. NVFP4 weights do not imply NVFP4 KV. ## Detailed Profile Specifications This section defines the production profile shapes in enough detail to write schemas and validators. It is intentionally stricter than the scrape metadata. Scrape data can suggest a candidate profile, but profile files are the audited contract. ### Profile Registry Files The registry should be plain files, not generated page routes: ```text profiles/ models/ google--gemma-4-26b-a4b-it.json redhatai--gemma-4-26b-a4b-it-nvfp4.json ``` File rules: - file stems must match `id` - ids must be lowercase and URL-safe - model profile ids are derived from Hugging Face repo ids by lowercasing, replacing `/` with `--`, replacing other non-alphanumeric runs with `-`, and trimming leading or trailing separators - model profiles must keep the original Hugging Face repo id in `repo` - model profile files must be self-contained: no runtime references to external architecture or serving profile files - generated static pages must not add `index.html` ### Shared Profile Envelope Every persisted model profile should share a common metadata envelope: ```json { "id": "redhatai--gemma-4-26b-a4b-it-nvfp4", "version": "1.0.0", "schema_version": "1.0.0", "status": "audited", "title": "RedHatAI Gemma 4 26B A4B NVFP4", "summary": "Self-contained model profile for memory-side bounds.", "evidence": [ { "label": "Hugging Face model card", "url": "https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4", "source_type": "model_card", "supports": [ "repo", "serving" ] } ], "review": { "reviewed_by": "osolmaz", "reviewed_at": "2026-07-01", "notes": "Initial profile seeded from the model-bounds note." } } ``` Common field rules: - `id` must be stable, lowercase, URL-safe, and never derived from a transient artifact name when a base-family id exists. - `version` changes when profile semantics change. - `schema_version` changes when the schema changes. - `status` controls whether the profile can produce production bounds. - `evidence` must be present for audited model profiles. - `review` must be present for audited model profiles. - bounds results are output snapshots, not input profiles; they use `engine_version` plus a result `schema_version`. ### Evidence Records Evidence must be structured so audits can be repeated. ```json { "label": "Gemma 4 config", "url": "https://huggingface.co/google/gemma-4-26B-A4B-it/raw/main/config.json", "source_type": "config", "supports": [ "layers", "kv_heads", "head_dim", "attention_pattern" ], "notes": "Used to verify the KV adapter fields." } ``` Initial `source_type` values: - `model_card` - `config` - `paper` - `vendor_doc` - `release_notes` - `benchmark_report` - `manual_review` - `derived_calculation` Evidence should state which profile fields it supports. A profile with evidence links but no field-level support is not audited. ### Model Profile Model profiles are the only production model artifact. Each model profile embeds architecture and serving data directly. ```json { "id": "qwen--qwen3-0-6b", "version": "1.0.0", "schema_version": "1.0.0", "status": "audited", "repo": "Qwen/Qwen3-0.6B", "model_family": "qwen3", "base_model_proof": { "base_model": "Qwen/Qwen3-0.6B-Base", "relation": "finetune", "source": "cardData.base_model", "config_compatible": true }, "architecture": { "canonical_architecture_id": "qwen3-0-6b", "weight_adapter": { "kind": "dense", "total_params_b": 0.7516, "parameter_scope": "derived_from_safetensors_total" }, "kv_adapter": { "kind": "full_context", "layers": 28, "kv_heads": 8, "head_dim": 128 }, "max_context_tokens": 40960 }, "serving": { "weight_format": "bf16", "weight_bytes_per_param": 2, "kv_store_format": "bf16", "kv_store_bytes_per_scalar": 2, "kv_read_format": "bf16", "kv_read_bytes_per_scalar": 2 }, "evidence": [ { "label": "Qwen3 0.6B config", "url": "https://huggingface.co/Qwen/Qwen3-0.6B/raw/main/config.json", "source_type": "config", "supports": [ "weight_adapter", "kv_adapter", "max_context_tokens" ] } ], "review": { "reviewed_by": "osolmaz", "reviewed_at": "2026-07-01", "notes": "Initial audited profile." } } ``` Rules: - Model profiles must include repo packaging and serving details because they are the runtime artifact. - Fine-tuned instruction repos can copy architecture data from a base model only when the model profile records proof that the architecture did not change. - Quantized artifacts can copy architecture data from the source model only when the model profile records proof that quantization did not change architecture. - If an artifact changes attention, experts, pruning, routing, depth, or state, its model profile must embed different architecture data. - Runtime code must not chase external `architecture_profile` or `serving_profile` references. Any such references are invalid in production model profiles. ### Weight Adapter: `dense` Use for models where every decode step touches all model weights. ```json { "kind": "dense", "total_params_b": 8.03, "parameter_scope": "total_including_embeddings", "formula": "W_resident = W_batch = total_params_b * weight_bytes_per_param" } ``` Required fields: - `kind` - `total_params_b` - `parameter_scope` Optional fields: - `embedding_params_b` - `non_embedding_params_b` - `notes` Allowed `parameter_scope` values: - `total_including_embeddings` - `total_excluding_embeddings` - `marketed_parameter_count` - `derived_from_safetensors_total` Formula: ```text W_resident = total_params_b * weight_bytes_per_param W_batch(b) = W_resident ``` Audit requirements: - total parameter count source - whether embedding/head parameters are included - explanation if parameter count is rounded or marketed ### Weight Adapter: `dense_resident_swept` Use for dense decoder packages where all stored weights must be resident, but only a verified subset is swept for each generated text token. This covers multimodal repos with resident vision towers and dense language decoders. ```json { "kind": "dense_resident_swept", "resident_params_b": 8.292166656, "swept_params_b": 7.615616512, "auxiliary_resident_params_b": 0.676550144, "resident_weight_gb": 16.584333312, "swept_weight_gb": 15.231233024, "resident_parameter_scope": "safetensors_header_stored_bf16", "swept_parameter_scope": "language_model_plus_lm_head_safetensors_headers", "auxiliary_scope": "visual tower resident during multimodal serving" } ``` Required fields: - `kind` - `resident_params_b` - `swept_params_b` - `resident_parameter_scope` - `swept_parameter_scope` Optional fields: - `auxiliary_resident_params_b` - `resident_weight_gb` - `swept_weight_gb` - `auxiliary_resident_weight_gb` - `auxiliary_scope` - `notes` Formula: ```text W_resident = resident_params_b * weight_bytes_per_param W_batch(b) = swept_params_b * weight_bytes_per_param ``` If exact mixed-dtype byte totals are present: ```text W_resident = resident_weight_gb W_batch(b) = swept_weight_gb ``` Audit requirements: - resident parameter count source - swept parameter count source - direct byte-count source when exact byte fields are present - explanation of which tensor prefixes or components are resident-only - confirmation that the profile is for text decode bounds, not image/video prefill throughput ### Weight Adapter: `moe_distinct_experts` Use for MoE models where all experts are resident but each decode iteration touches an expected subset of routed experts. ```json { "kind": "moe_distinct_experts", "total_params_b": 26.5, "active_params_b": 4.0, "routed_experts": 128, "routed_experts_per_token": 8, "shared_experts_per_token": 0, "routing_model": "uniform_expected_distinct", "expert_param_b": null, "fixed_param_b": null, "shared_expert_notes": null } ``` Required fields: - `kind` - `total_params_b` - `active_params_b` - `routed_experts` - `routed_experts_per_token` - `routing_model` Optional fields: - `shared_experts_per_token` - `expert_param_b` - `fixed_param_b` - `shared_expert_notes` - `nonuniform_expert_notes` Convention: - `routed_experts` is the number of routed experts, excluding shared experts. - `routed_experts_per_token` is the number of routed experts selected per token. - shared experts are always-on traffic and are folded into `fixed_param_b` unless a separate audited adapter variant is defined. - `active_params_b` must include the parameters touched by one token, including shared experts when the architecture has them. Default derived quantities: ```text E = routed_experts k = routed_experts_per_token expert_param_b = (total_params_b - active_params_b) / max(1, E - k) fixed_param_b = max(0, active_params_b - k * expert_param_b) m(b * rho) = E * (1 - (1 - k / E) ** (b * rho)) W_resident = total_params_b * weight_bytes_per_param W_batch(b) = weight_bytes_per_param * (fixed_param_b + expert_param_b * m(b * rho)) ``` This v1 adapter deliberately folds always-on shared expert traffic into `fixed_param_b`. If shared experts are not always-on, are nonuniform, or need a separate resident/traffic formula, the model needs a separate adapter variant instead of extra fields bolted onto `moe_distinct_experts`. The adapter must preserve: ```text W_batch(b) <= W_resident ``` for every fitting batch. CI should enforce this invariant. Audit requirements: - total parameter count source - active parameter count source - routed expert count source - routed experts per token source - shared expert handling, if any - whether uniform expected distinct expert routing is a documented assumption or a simplifying estimate - proof that `routed_experts` excludes or includes shared experts; the profile must state the convention explicitly If experts are materially nonuniform and no audited nonuniform formula exists, the model is unsupported for production bounds. ### Weight Adapter: `moe_distinct_experts_exact` Use for MoE packages where resident footprint and decode traffic are audited directly in decimal GB. This is preferred over `moe_distinct_experts` when stored dtypes differ by tensor group or when resident-only auxiliary modules must be separated from ordinary decode traffic. ```json { "kind": "moe_distinct_experts_exact", "resident_weight_gb": 688.57483936, "main_resident_weight_gb": 673.150611808, "auxiliary_resident_weight_gb": 15.424227552, "fixed_weight_gb": 19.082195296, "routed_expert_weight_gb": 2.554954752, "routed_experts": 256, "routed_experts_per_token": 8, "shared_experts_per_token": 1, "routing_model": "uniform_expected_distinct", "resident_parameter_scope": "safetensors_header_stored_fp8_bf16_f32", "traffic_scope": "ordinary decode excluding resident-only MTP layer" } ``` Formula: ```text m(b * rho) = E * (1 - (1 - k / E) ^ (b * rho)) W_resident = resident_weight_gb W_batch(b) = fixed_weight_gb + routed_expert_weight_gb * m(b * rho) ``` Audit requirements: - exact resident byte source - exact fixed traffic byte source - exact per-routed-expert byte source - proof that routed expert weights are uniform enough for expected-distinct aggregation - explicit list of any resident-only auxiliary modules excluded from ordinary decode traffic ### KV Adapter: `full_context` Use for standard attention where every layer reads and stores full-context KV. ```json { "kind": "full_context", "layers": 32, "kv_heads": 8, "head_dim": 128 } ``` Required fields: - `kind` - `layers` - `kv_heads` - `head_dim` Formula: ```text K_alloc(L_alloc) = 2 * layers * kv_heads * head_dim * kv_store_bytes_per_scalar * L_alloc K_read(L_read) = 2 * layers * kv_heads * head_dim * kv_read_bytes_per_scalar * L_read ``` The engine converts bytes to decimal GB. Audit requirements: - layer count source - KV head count source - head dimension source - confirmation that all counted layers use full-context attention ### KV Adapter: `layered_kv` Use for hybrid models, including local/global attention and sliding-window attention. This should be the default representation for mixed attention because it composes layer components explicitly. ```json { "kind": "layered_kv", "components": [ { "kind": "full_context", "layers": 10, "kv_heads": 8, "head_dim": 256 }, { "kind": "sliding_window", "layers": 20, "kv_heads": 8, "head_dim": 256, "window_tokens": 4096 } ] } ``` Initial component kinds: - `full_context` - `sliding_window` - `fixed_state` - `compressed_state` For a `full_context` component: ```text component_alloc(L_alloc) = 2 * alloc_layers * kv_heads * head_dim * kv_store_bytes_per_scalar * L_alloc component_read(L_read) = 2 * read_layers * kv_heads * head_dim * kv_read_bytes_per_scalar * L_read ``` If `alloc_layers` or `read_layers` is omitted, it defaults to `layers`. Use explicit values for KV-sharing designs where fewer decoder layers store K/V than consume it during attention. For a `sliding_window` component: ```text alloc_L = min(L_alloc, window_tokens) read_L = min(L_read, window_tokens) component_alloc(L_alloc) = 2 * alloc_layers * kv_heads * head_dim * kv_store_bytes_per_scalar * alloc_L component_read(L_read) = 2 * read_layers * kv_heads * head_dim * kv_read_bytes_per_scalar * read_L ``` For a `fixed_state` component: ```text component_alloc(L_alloc) = alloc_gb_per_session component_read(L_read) = read_gb_per_output_token ``` For a `compressed_state` component, the profile must provide an audited formula using the same shape as the standalone `compressed_state` adapter. A generic compression ratio is not enough for audited production bounds unless the source explicitly supports it. Component outputs that use KV scalar counts are converted from bytes to decimal GB by the engine. Components that directly specify `alloc_gb_per_session` or `read_gb_per_output_token`, or formula fields named `gb_per_1k_context_tokens`, are already in decimal GB and must not be converted a second time. Audit requirements: - all component layer counts - layer assignment rule, such as every Nth layer global - KV heads and head dimension per component - window size or compression formula - whether allocation and read use the same effective context ### KV Adapter: `recurrent_state` Use for recurrent/state-space models where per-session state is fixed rather than full-context KV. Slowly growing recurrent state is a future adapter variant unless the growth formula is explicitly audited. ```json { "kind": "recurrent_state", "alloc_gb_per_session": 0.02, "read_gb_per_output_token": 0.002, "state_formula": "fixed" } ``` Required fields: - `kind` - `alloc_gb_per_session` - `read_gb_per_output_token` - `state_formula` Audit requirements: - proof that the model does not require full-context KV - source for state size - source for read traffic approximation If state traffic is not well understood, the model should remain unsupported. ### KV Adapter: `compressed_state` Use for latent, compressed, sparse, or otherwise non-full-context attention when there is audited evidence for the compression formula. ```json { "kind": "compressed_state", "alloc_formula": { "kind": "linear_context_ratio", "gb_per_1k_context_tokens": 0.01 }, "read_formula": { "kind": "linear_context_ratio", "gb_per_1k_context_tokens": 0.005 } } ``` Formula: ```text K_alloc(L_alloc) = alloc_formula.gb_per_1k_context_tokens * (L_alloc / 1000) K_read(L_read) = read_formula.gb_per_1k_context_tokens * (L_read / 1000) ``` Both formulas return decimal GB. The `read_formula` ratio is context-depth traffic per output token; it scales with `L_read`, not with the number of output tokens generated by the session. This adapter must be conservative. If a model claims million-token context via compressed attention but the actual read/allocation cost is unclear, do not invent a ratio. Mark the model unsupported until audited. ### Embedded Serving Block The `serving` block inside a model profile describes precision and runtime representation. ```json { "weight_format": "nvfp4", "weight_bytes_per_param": 0.5, "kv_store_format": "bf16", "kv_store_bytes_per_scalar": 2, "kv_read_format": "bf16", "kv_read_bytes_per_scalar": 2, "runtime_format": "format_agnostic_memory_bound", "dequantization_notes": "Memory-side bound charges stored weight bytes only unless an audited runtime overhead is added." } ``` Required fields: - `weight_format` - `weight_bytes_per_param` - `kv_store_format` - `kv_store_bytes_per_scalar` - `kv_read_format` - `kv_read_bytes_per_scalar` Initial `weight_format` values: - `bf16` - `fp16` - `fp32` - `fp8` - `fp4_fp8_mixed` - `nvfp4` - `mxfp4` - `mixed_bf16_f32` - `mixed_bf16_f16_f32` - `int8` - `int4` - `q4` - `q2_mixed` - `gguf_quantized` - `mlx_quantized` - `unknown` Optional fields: - `runtime_format` - `dequantization_notes` - `notes` Rules: - `unknown` weight or KV formats cannot be audited. - `runtime_format: format_agnostic_memory_bound` means the profile claims only memory-side stored width and KV width, with no runtime-specific overhead. - weight format and KV format must be separate. - the serving block cannot change architecture fields. - if runtime stores quantized weights but reads/dequantizes through a wider effective bandwidth path, that needs a separate audited runtime overhead field rather than an implicit change to architecture. - evidence for serving fields belongs in the parent model profile `evidence` array. ### Model Profile Status - `audited` - `unsupported` - `metadata_estimate` Production bounds require `audited`. Status rules: - `audited` means the self-contained model file has evidence and review for its embedded architecture and serving fields. - `unsupported` means the model file exists but production bounds are disabled. - `metadata_estimate` is diagnostic only and hidden from production comparisons by default. - copied architecture or serving data must be recorded through `base_model_proof` or evidence records, but the copied values must still be embedded in the model file. - A fine-tune can copy architecture data only if there is no evidence of structural changes. - A merge can inherit only after review; default is unsupported. - LoRA/adapters are unsupported unless the runtime merge/adapter traffic is explicitly modeled. - Distilled, pruned, depth-changed, or architecture-modified repos need their own embedded architecture data. ### Workload Settings Object The engine receives workload settings from app defaults or UI state: ```json { "l_alloc_tokens": 100000, "l_read_tokens": 32000, "min_toks_per_session": 20, "overhead_gb": 8, "decode_policy": { "kind": "ordinary", "rho": 1 } } ``` Initial production decode setting: - `ordinary` Future or diagnostic decode settings: - `speculative` - `custom_external` Required fields: - `l_alloc_tokens` - `l_read_tokens` - `min_toks_per_session` - `overhead_gb` - `decode_policy` Rules: - `ordinary` must use `rho = 1`. - Bounds Engine v1 must reject production settings with `rho > 1`. - `speculative` is a future production setting. It may use `rho > 1`, but only after the schema defines draft model resident cost, draft weight traffic, draft KV/state traffic, verifier overhead, and acceptance-rate evidence. - `custom_external` is diagnostic only and must not be used for production comparisons unless the UI labels the external assumption as diagnostic. - raw `rho` must not be a standalone UI slider. ### Bounds Result Profile Bounds results should be serializable and testable. ```json { "engine_version": "1.0.0", "schema_version": "1.0.0", "status": "ok", "profile_resolution": { "repo": "RedHatAI/gemma-4-26B-A4B-it-NVFP4", "model_profile": "redhatai--gemma-4-26b-a4b-it-nvfp4", "model_profile_status": "audited" }, "inputs": { "hardware_id": "nvidia-dgx-spark", "workload_settings": { "l_alloc_tokens": 100000, "l_read_tokens": 32000, "min_toks_per_session": 20, "overhead_gb": 8, "decode_policy": { "kind": "ordinary", "rho": 1 } } }, "trace": { "capacity_gb": 128, "bandwidth_gbps": 273, "overhead_gb": 8, "w_resident_gb": 12.0, "k_alloc_gb": 2.1, "free_gb": 108.0, "b_mem": 51, "single_session_toks_per_s": 120, "usable_batches_summary": { "count": 16, "min_batch": 1, "max_batch": 16 }, "b_star": 16, "w_batch_at_b_star_gb": 6.4, "k_read_gb": 0.42, "q_at_b_star_gb_per_output_token": 0.82, "aggregate_toks_per_s": 333, "per_session_toks_per_s": 20.8, "memory_power_ceiling_toks_per_s": 23700 }, "usable_batch": { "b_star": 16, "selection_rule": "max_aggregate_over_floor_qualified_batches" }, "ceilings": { "single_session_toks_per_s": 120, "kv_aware_aggregate_toks_per_s": 333, "per_session_at_b_star_toks_per_s": 20.8, "memory_power_toks_per_s": 23700 }, "warnings": [] } ``` Required trace fields: - `capacity_gb` - `bandwidth_gbps` - `overhead_gb` - `w_resident_gb` - `k_alloc_gb` - `free_gb` - `b_mem` - `single_session_toks_per_s` - `usable_batches_summary` - `b_star` - `w_batch_at_b_star_gb` - `k_read_gb` - `q_at_b_star_gb_per_output_token` - `aggregate_toks_per_s` - `per_session_toks_per_s` - `memory_power_ceiling_toks_per_s` The trace must be stable enough for snapshot tests and readable enough for the calculation details UI. ### Profile Audit States Use a strict state model: ```text draft -> audited -> deprecated unsupported is a model profile state for repos that should fail closed. metadata_estimate is diagnostic and never audited. ``` Definitions: - `draft`: profile exists for discussion but cannot produce production bounds. - `audited`: profile has evidence and review; can produce production bounds. - `deprecated`: profile is retained for reproducibility but should not be used for new results. - `unsupported`: repo has a model profile that intentionally disables production bounds. - `metadata_estimate`: generated fallback, hidden from production comparison by default. ## Bounds Engine Contract Create `docs/bounds-engine-v1.md` before replacing the engine. That document must freeze the exact v1 math contract. ### Units - memory capacity: decimal GB - memory bandwidth: decimal GB/s - parameters: billions of parameters - byte widths: bytes per parameter or bytes per KV scalar - token counts: integer tokens - rates: output tokens per second ### Inputs ```text hardware: capacity_gb bandwidth_gbps model profile: architecture.weight adapter architecture.KV/state adapter weight bytes KV store bytes KV read bytes workload settings: L_alloc L_read r_star overhead_gb decode policy ``` ### Required Adapter Functions ```text W_resident(profile) -> GB W_batch(batch, rho, profile) -> GB per decode iteration K_alloc(L_alloc, profile) -> GB per session K_read(L_read, profile) -> GB per output token rho(settings) -> accepted tokens per session per iteration ``` ### Required Computation ```text free = C - W_resident - overhead if free < 0: status = resident_not_fit b_mem = floor(free / K_alloc(L_alloc)) if b_mem < 1: status = no_session_capacity single_q = W_batch(1) / rho + K_read(L_read) single_session = R / single_q for b in 1..b_mem: q(b) = W_batch(b) / (b * rho) + K_read(L_read) aggregate(b) = R / q(b) per_session(b) = aggregate(b) / b usable_batches = batches where per_session(b) >= r_star if usable_batches is empty: status = no_floor b_star = batch in usable_batches with max aggregate(b) W_active = W_resident for dense weight adapters W_batch(1) for MoE and other batch-dependent weight adapters memory_power_ceiling = rho * (C * R) / (K_alloc(L_alloc) * W_active) * (1 - (W_resident + overhead) / C) ``` ### Required Statuses - `unsupported_profile` - `resident_not_fit` - `no_session_capacity` - `no_floor` - `ok` The important semantic distinction is that `no_floor` is not a memory fit failure. It means sessions fit, but no fitting batch satisfies the minimum per-session rate. ### Required Trace Every successful or partially successful result must return a machine-readable root result with `engine_version` and `schema_version`, plus a stable `trace` object: ```json { "engine_version": "1.0.0", "schema_version": "1.0.0", "trace": { "capacity_gb": 128, "bandwidth_gbps": 273, "overhead_gb": 8, "free_gb": 108, "w_resident_gb": 12.0, "k_alloc_gb": 2.1, "b_mem": 51, "single_session_toks_per_s": 120, "usable_batches_summary": { "count": 16, "min_batch": 1, "max_batch": 16 }, "b_star": 16, "w_batch_at_b_star_gb": 6.4, "k_read_gb": 0.42, "q_at_b_star_gb_per_output_token": 0.82, "aggregate_toks_per_s": 333, "per_session_toks_per_s": 20.8, "memory_power_ceiling_toks_per_s": 23700 } } ``` The UI should render this trace in a collapsed calculation details section. ## Data Files Add these registry directories: ```text profiles/ models/ ``` The scraped catalog remains under `assets/local-frontier-model-data.js`, but it must not be the authority for serious bounds. ## Schemas Add JSON schemas under `schemas/`. ### `model-profile.schema.json` Required fields: - `id` - `version` - `schema_version` - `repo` - `status`: `draft`, `audited`, `deprecated`, `unsupported`, or `metadata_estimate` - `model_family` - `architecture` - `serving` - `evidence` - `notes` - `review`, when `status` is `audited` - `unsupported_reason`, when `status` is `unsupported` - `estimate_warning`, when `status` is `metadata_estimate` `architecture.weight_adapter` is a discriminated union by `kind`. Initial variants: - `dense` - `dense_resident_swept` - `moe_distinct_experts` - `moe_distinct_experts_exact` `architecture.kv_adapter` is a discriminated union by `kind`. Initial variants: - `full_context` - `layered_kv` - `recurrent_state` - `compressed_state` `serving` is an embedded object with required fields: - `weight_format` - `weight_bytes_per_param` - `kv_store_format` - `kv_store_bytes_per_scalar` - `kv_read_format` - `kv_read_bytes_per_scalar` Optional fields: - `runtime_format` - `dequantization_notes` - `notes` ### `bounds-result.schema.json` Required fields: - `engine_version` - `schema_version` - `status` - `profile_resolution` - `inputs` - `trace` - `ceilings` - `usable_batch` - `warnings` Required `ceilings` fields when `status` is `ok`: - `single_session_toks_per_s` - `kv_aware_aggregate_toks_per_s` - `per_session_at_b_star_toks_per_s` - `memory_power_toks_per_s` Required `usable_batch` fields when `status` is `ok`: - `b_star` - `selection_rule` ## Profile Loading Loading order: 1. Find `profiles/models/.json` for the selected repo. 2. Validate the model profile against `model-profile.schema.json`. 3. If `status` is `audited`, pass the self-contained profile to the engine. 4. If `status` is `unsupported`, return `unsupported_profile` with the embedded unsupported reason. 5. If `status` is `metadata_estimate`, allow it only in diagnostic mode. 6. If `status` is `draft` or `deprecated`, fail closed for production bounds. The loader must never use repo-name heuristics for production bounds. A model profile can copy architecture or serving data from a reviewed source only when the model profile records evidence: - exact base-model relation - compatible config - artifact appears to change serving representation only - no architecture-changing adapter, merge, pruning, distillation, or fine-tune claim that invalidates copied architecture data Copied data is authoring-time reuse. The runtime still receives one self-contained model profile and does not resolve external profile references. ## Audit Packet Generator Add: ```text scripts/generate-profile-audit-packets.mjs ``` For each scraped model repo, generate an audit packet. The generator may also produce base-model group summaries for reviewer efficiency, but the output must preserve one candidate self-contained model profile per concrete repo: - repo id - base model chain - card metadata - config summary - safetensors totals - quantization metadata - candidate embedded architecture block, if inferable - candidate embedded serving block, if inferable - missing evidence - proposed model profile status - proposed unsupported reason, when the repo should fail closed Audit packets are review aids only. They must not automatically create audited model profiles. ## Initial Model Profiles Seed the registry from the worked examples in the source bounds doc: 1. `gemma-4-26b-a4b` - MoE weights - total around 26B - active around 4B - 128 routed experts - top-8 routing - hybrid local/global attention - serving examples include NVFP4 weights with BF16 KV 2. `qwen3-6-35b-a3b` - MoE weights - total around 35B - active around 3B - 256 routed experts - top-8 routing plus shared expert handling if supported by evidence - serving examples include NVFP4 weights 3. `deepseek-v4-flash` - MoE weights - total around 284B - active around 13B - 256 routed experts plus shared expert handling - compressed or sparse attention - Q2-style mixed serving block only where evidence supports it After those, add dense families with simpler audited profiles: - Llama dense families - Qwen dense families - Gemma dense families - Phi dense families Do not hand-write 654 unique architecture formulas. Do create one self-contained model profile or unsupported model profile for every scraped row. Coverage can be phased, but there must be no production fallback that treats an unreviewed row as supported. ## Engine Implementation Add TypeScript modules: ```text src/lib/bounds/ engine.ts weight-adapters.ts kv-adapters.ts profiles.ts profile-loader.ts trace.ts ``` The engine must be pure: - no DOM - no network - no repo-name logic - deterministic for a given input The static `assets/bounds-engine.js` should either be generated from the TS implementation or kept as a thin compatibility wrapper during migration. ## UI Changes ### Compare Page Metric table layout is specified in [`docs/compare-table-metrics.md`](compare-table-metrics.md). That document is the source of truth for compare modes, default columns, and the advanced quantity toggle. Main result should show: - profile status - resident fit - memory-fit max batch, `b_mem` - derived serving batch, `b_star` - single-session decode ceiling - KV-aware aggregate decode ceiling - per-session rate at `b_star` - Memory-power orientation ceiling Remove raw `Inspect concurrency` from the main UI. Optional future diagnostic: - collapsed advanced forced-batch table - clearly labeled as a policy what-if - never presented as the primary result ### Calculation Details Add a collapsed section for supported models: - hardware inputs: `C`, `R`, overhead - workload inputs: `L_alloc`, `L_read`, `r_star`, decode policy - model profile id and status - `W_resident` - `K_alloc` - `b_mem` - `W_batch(b_star)` - `K_read` - `q(b_star)` - `T = R / q` - `per_session = T / b_star` - evidence links For unsupported models, show: ```text Bounds unavailable: no audited adapter profile. ``` ### Model Pages Add an Adapter Profile section: - model profile status - model profile id - repo id - weight adapter details - KV/state adapter details - precision details - evidence links - unsupported reason if unsupported ## CI And Validation Add scripts: ```text scripts/validate-profiles.mjs scripts/test-profiled-bounds.mjs ``` CI gates: - all profile files validate against schema - every audited model profile has evidence - every audited model profile has `review` - every scraped model row has exactly one model profile file - model profiles are self-contained and do not contain runtime `architecture_profile` or `serving_profile` references - unsupported models cannot produce production bounds - metadata estimates are disabled by default - production workload settings with `rho > 1` are rejected in Bounds Engine v1 - doc worked examples pass golden tests - MoE `W_batch(b)` is nondecreasing with batch - MoE `W_batch(b)` never exceeds `W_resident` - dense `W_batch(b)` is constant with batch - hybrid/sliding KV does not scale as full-context for local layers - no `index.html` ## Golden Tests Use ranges rather than fake exact precision. The source doc gives the initial expected shape. Example tests: ```text DGX Spark + NVIDIA Gemma 4 26B A4B NVFP4: single-session decode ceiling near 48 tok/s b_star near 12 KV-aware aggregate near 246 tok/s note: the original bounds note used rounded 26B/4B NVFP4 bytes and produced the old 120 / 16 / 333 shape. The audited production target uses exact NVIDIA ModelOpt safetensors bytes, BF16 fixed language traffic, quantized routed expert traffic, and FP8 KV cache metadata from the served artifact. Apple M5 Max 128GB + NVIDIA Gemma 4 26B A4B NVFP4: b_star near 97 KV-aware aggregate near 1944 tok/s DGX Spark + Qwen3.6 35B A3B NVFP4: single-session decode ceiling near 103 tok/s b_star near 13 KV-aware aggregate near 275 tok/s note: the original bounds note used rounded 35B/3B NVFP4 bytes; the audited profile uses range-read safetensors-header bytes plus DeltaNet state. DGX Spark + DeepSeek V4 Flash: single-session decode ceiling near 28 tok/s b_star near 3 KV-aware aggregate near 62 tok/s note: the original bounds note used rounded 284B/13B Q2-style bytes; the audited profile uses exact q2-imatrix GGUF linked file size and tensor-index spans for the DS4 artifact. ``` Also test failure states: - resident model too large - no active session capacity - sessions fit but no batch clears floor - unsupported profile ## Migration Slices ### Slice 1: Documentation And Schemas - add `docs/bounds-engine-v1.md` - add `model-profile.schema.json` and `bounds-result.schema.json` - add profile registry directory - add empty or draft registry files - add schema validation script ### Slice 2: Pure Engine - implement TS bounds engine - implement dense and MoE weight adapters - implement full-context and layered KV adapters - add trace output - add unit tests ### Slice 3: Initial Audited Model Profiles - add audited self-contained model profiles for reviewed Gemma 4 26B A4B repos - add audited self-contained model profiles for reviewed Qwen3.6 35B A3B repos - add audited self-contained model profiles for reviewed DeepSeek V4 Flash repos - copy BF16, NVFP4, and Q-style serving blocks into those model profiles - keep unreviewed rows unsupported ### Slice 4: Loader And Fail-Closed UI - implement model profile loader - update compare page to use model profiles - show unsupported state for missing or unsupported model profiles - remove main concurrency control ### Slice 5: Calculation Trace UI - add calculation details on compare page - add adapter profile sections on model pages - include evidence links ### Slice 6: Coverage Expansion - generate audit packets for the 654 scraped rows - audit highest-coverage base families first - add one self-contained model profile per scraped row - reuse copied architecture and serving blocks only through audit tooling and reviewed evidence - keep unsupported model profiles unsupported with concrete unsupported reasons ### Slice 7: Retire Old Metadata Bounds - remove or quarantine generic metadata-derived serious bounds - keep metadata estimates only in an explicitly labeled diagnostic mode - make profile-backed bounds the only production path ## Acceptance Criteria The implementation is complete when: - concurrency is derived, not controlled in the main UI - every serious tok/s result has an audited self-contained model profile - every scraped model row has an explicit model profile status or unsupported status - every result has a calculation trace - unsupported models fail closed - schemas validate all profile data - worked examples from the source doc pass golden tests - no production bound relies on repo-name guessing - deployed Space is verified on the final SHA