Release v0.1.1 real-patch evaluation tooling
Browse filesAdd local exact-execution baseline tooling, source, contract, and accurate development-result documentation. Existing controller candidates, runtime, and examples are unchanged.
- CHANGELOG.md +6 -0
- CITATION.cff +2 -2
- README.md +14 -5
- RELEASE_SCOPE.md +6 -0
- SHA256SUMS +18 -5
- evaluation/real-patch/README.md +37 -0
- evaluation/real-patch/runtime/laxmi_real_patch-0.1.0.dev1-py3-none-any.whl +0 -0
- evaluation/real-patch/source/LICENSE +202 -0
- evaluation/real-patch/source/LICENSE_SCOPE.md +38 -0
- evaluation/real-patch/source/NOTICE +12 -0
- evaluation/real-patch/source/code_review/contracts/real_patch_api_v1/openapi.yaml +251 -0
- evaluation/real-patch/source/code_review/contracts/real_patch_prediction_v1/README.md +38 -0
- evaluation/real-patch/source/code_review/contracts/real_patch_prediction_v1/forecast_request.schema.json +137 -0
- evaluation/real-patch/source/code_review/contracts/real_patch_prediction_v1/validator.py +179 -0
- evaluation/real-patch/source/code_review/packaging/real_patch/api.py +120 -0
- evaluation/real-patch/source/code_review/scripts/build_real_patch_local_package_v1.py +101 -0
- evaluation/real-patch/source/code_review/scripts/run_real_patch_baseline_gate_v2.py +264 -0
- evaluation/real-patch/source/code_review/scripts/run_real_patch_visible_exact_v1.py +300 -0
- release.json +25 -12
CHANGELOG.md
CHANGED
|
@@ -1,5 +1,11 @@
|
|
| 1 |
# Changelog
|
| 2 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
## 0.1.0 — 29 September 2026
|
| 4 |
|
| 5 |
First public experimental controller package. Retains all four original September 15–16 candidates without selecting a winner or changing weights. Includes the negative development result, a smaller reviewed source closure, Apache 2.0 attribution, and a retained synthetic H1 installation/replay example.
|
|
|
|
| 1 |
# Changelog
|
| 2 |
|
| 3 |
+
## 0.1.1 — 1 October 2026
|
| 4 |
+
|
| 5 |
+
Adds a separate `evaluation/real-patch/` tooling component within the same Laxmi release: an exact visible-source baseline wheel, source, versioned input contract and design-only private API draft. One exposed development case is scored correctly by exact execution; five other historical cases remain oracle-only. No learned real-patch model, blind result, hosted API, Linux exact-execution qualification or case dataset is released.
|
| 6 |
+
|
| 7 |
+
All four controller candidate weights, metadata, runtime wheel/source ZIP and retained examples are byte-identical to v0.1.0. The controller experiment's failed repair and scientific scope are unchanged.
|
| 8 |
+
|
| 9 |
## 0.1.0 — 29 September 2026
|
| 10 |
|
| 11 |
First public experimental controller package. Retains all four original September 15–16 candidates without selecting a winner or changing weights. Includes the negative development result, a smaller reviewed source closure, Apache 2.0 attribution, and a retained synthetic H1 installation/replay example.
|
CITATION.cff
CHANGED
|
@@ -13,5 +13,5 @@ abstract: >-
|
|
| 13 |
consequences from observed state, history, actions and proposed code changes.
|
| 14 |
Controlled experiments and reproducibility infrastructure do not establish
|
| 15 |
general software-engineering capability or production suitability.
|
| 16 |
-
version: "0.1.
|
| 17 |
-
date-released: 2026-
|
|
|
|
| 13 |
consequences from observed state, history, actions and proposed code changes.
|
| 14 |
Controlled experiments and reproducibility infrastructure do not establish
|
| 15 |
general software-engineering capability or production suitability.
|
| 16 |
+
version: "0.1.1"
|
| 17 |
+
date-released: 2026-10-01
|
README.md
CHANGED
|
@@ -6,6 +6,7 @@ tags:
|
|
| 6 |
- experimental
|
| 7 |
- negative-results
|
| 8 |
- custom-model
|
|
|
|
| 9 |
---
|
| 10 |
|
| 11 |
# Laxmi — experimental software-state prediction
|
|
@@ -14,11 +15,11 @@ tags:
|
|
| 14 |
|
| 15 |
Created and maintained by **Navneet Prabhakar**, creator and sole contributor to Laxmi. Third-party libraries retain their authorship and licenses.
|
| 16 |
|
| 17 |
-
Laxmi investigates whether observed software state, history, an action and a proposed code change can predict what happens next. This
|
| 18 |
|
| 19 |
-
> **Version 0.1.
|
| 20 |
>
|
| 21 |
-
>
|
| 22 |
|
| 23 |
## The experiment
|
| 24 |
|
|
@@ -47,14 +48,18 @@ Readout normalization did not improve the task. These are exposed development re
|
|
| 47 |
|
| 48 |
The packaged example checks installation and replay. It must not be presented as a fresh benchmark. See [verification.json](verification.json) for the exact smoke scope and [BENCHMARK.md](BENCHMARK.md) for the next comparison design.
|
| 49 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
## Download and run
|
| 51 |
|
| 52 |
-
**
|
| 53 |
|
| 54 |
With the Hugging Face `hf` CLI installed:
|
| 55 |
|
| 56 |
```bash
|
| 57 |
-
hf download Executespec/laxmi-controller-experimental --revision v0.1.
|
| 58 |
cd laxmi-release
|
| 59 |
shasum -a 256 -c SHA256SUMS
|
| 60 |
python3.12 -m venv .venv
|
|
@@ -66,12 +71,15 @@ Expected completion: `17-original: all six operations passed`. The output direct
|
|
| 66 |
|
| 67 |
The wheel declares exact dependency versions, also recorded in `requirements-macos-arm64-py312.txt`. Dependencies are installed separately; they are not bundled. Version pins are not a dependency artifact hash lock. The source package includes the runtime source and license files. No optimizer checkpoint is shipped.
|
| 68 |
|
|
|
|
|
|
|
| 69 |
## Package layout
|
| 70 |
|
| 71 |
| Path | Contents |
|
| 72 |
|---|---|
|
| 73 |
| `candidates/17-original/`, `17-normalized/`, `43-original/`, `43-normalized/` | Original retained safetensors weights, configuration, manifests and matching consumer metadata |
|
| 74 |
| `runtime/` | Reviewed runtime wheel and source ZIP |
|
|
|
|
| 75 |
| `examples/` | One retained synthetic observation, proposal comparison and outcome template |
|
| 76 |
| `verify_release.py` | Installation/replay check using only this package |
|
| 77 |
| `release.json`, `SHA256SUMS` | Candidate identity, runtime identity and file integrity |
|
|
@@ -90,6 +98,7 @@ Weights, configuration, readout metadata and runtime must match. Original and no
|
|
| 90 |
- Longer rollouts may contain unresolved/pruned probability mass; it is not silently renormalized away.
|
| 91 |
- Supported consumer horizons are 1, 2, 4 and 8; this release's installation smoke covers H1 only.
|
| 92 |
- The learned probabilities are not proven calibrated and do not authorize external actions.
|
|
|
|
| 93 |
|
| 94 |
## Benchmark and iteration
|
| 95 |
|
|
|
|
| 6 |
- experimental
|
| 7 |
- negative-results
|
| 8 |
- custom-model
|
| 9 |
+
- real-patch-evaluation
|
| 10 |
---
|
| 11 |
|
| 12 |
# Laxmi — experimental software-state prediction
|
|
|
|
| 15 |
|
| 16 |
Created and maintained by **Navneet Prabhakar**, creator and sole contributor to Laxmi. Third-party libraries retain their authorship and licenses.
|
| 17 |
|
| 18 |
+
Laxmi investigates whether observed software state, history, an action and a proposed code change can predict what happens next. This release keeps the bounded controller experiment inspectable and adds local real-patch evaluation tooling to the **same research project**. It is not a chat model, a general code-review assistant or an autonomous software engineer.
|
| 19 |
|
| 20 |
+
> **Version 0.1.1 · Experimental research release**
|
| 21 |
>
|
| 22 |
+
> Controller weights and runtime are unchanged from v0.1.0. The latest model repair failed its improvement test. The new real-patch exact-execution tool is a baseline, **not a trained model**. None of the four controller candidates is promoted as an accepted model.
|
| 23 |
|
| 24 |
## The experiment
|
| 25 |
|
|
|
|
| 48 |
|
| 49 |
The packaged example checks installation and replay. It must not be presented as a fresh benchmark. See [verification.json](verification.json) for the exact smoke scope and [BENCHMARK.md](BENCHMARK.md) for the next comparison design.
|
| 50 |
|
| 51 |
+
### Real-patch development screen
|
| 52 |
+
|
| 53 |
+
This release also contains [local real-patch evaluation tooling](evaluation/real-patch/README.md). Six exposed historical changes were inventoried; only one currently has a valid forecast packet, and five have oracle-only scenarios excluded from forecast accuracy. On the one scored development case, the **same-information exact-execution baseline** matched both branch observations and the patch effect. The scenario was selected after its outcome was known. There are no blind/acceptance-eligible cases, no learned real-patch model scored, and no released case data. The result shows that this fully runnable case offers no measured headroom over exact execution; it does not validate generalization.
|
| 54 |
+
|
| 55 |
## Download and run
|
| 56 |
|
| 57 |
+
**Controller runtime verified platform: Apple Silicon macOS, Python 3.12.11.** That wheel is macOS ARM64 tagged. Other Python 3.12 patch builds and platforms have not been verified for the controller runtime. The source archive is provided for inspection and experimentation; it does not establish Linux controller support. No GPU is required for the example.
|
| 58 |
|
| 59 |
With the Hugging Face `hf` CLI installed:
|
| 60 |
|
| 61 |
```bash
|
| 62 |
+
hf download Executespec/laxmi-controller-experimental --revision v0.1.1 --local-dir laxmi-release
|
| 63 |
cd laxmi-release
|
| 64 |
shasum -a 256 -c SHA256SUMS
|
| 65 |
python3.12 -m venv .venv
|
|
|
|
| 71 |
|
| 72 |
The wheel declares exact dependency versions, also recorded in `requirements-macos-arm64-py312.txt`. Dependencies are installed separately; they are not bundled. Version pins are not a dependency artifact hash lock. The source package includes the runtime source and license files. No optimizer checkpoint is shipped.
|
| 73 |
|
| 74 |
+
The separate real-patch package has its own [local installation instructions](evaluation/real-patch/README.md). It installed and **validated** its input in an offline Linux ARM64 container, but Linux exact execution and any hosted API remain unqualified. Its OpenAPI file is a design draft only.
|
| 75 |
+
|
| 76 |
## Package layout
|
| 77 |
|
| 78 |
| Path | Contents |
|
| 79 |
|---|---|
|
| 80 |
| `candidates/17-original/`, `17-normalized/`, `43-original/`, `43-normalized/` | Original retained safetensors weights, configuration, manifests and matching consumer metadata |
|
| 81 |
| `runtime/` | Reviewed runtime wheel and source ZIP |
|
| 82 |
+
| `evaluation/real-patch/` | Local exact-baseline evaluator wheel, source, contract and design-only API draft; no model weights or case data |
|
| 83 |
| `examples/` | One retained synthetic observation, proposal comparison and outcome template |
|
| 84 |
| `verify_release.py` | Installation/replay check using only this package |
|
| 85 |
| `release.json`, `SHA256SUMS` | Candidate identity, runtime identity and file integrity |
|
|
|
|
| 98 |
- Longer rollouts may contain unresolved/pruned probability mass; it is not silently renormalized away.
|
| 99 |
- Supported consumer horizons are 1, 2, 4 and 8; this release's installation smoke covers H1 only.
|
| 100 |
- The learned probabilities are not proven calibrated and do not authorize external actions.
|
| 101 |
+
- The real-patch exact comparator is a baseline, not learned prediction. Only one exposed development forecast was scored; no public API or Linux exact-execution qualification exists.
|
| 102 |
|
| 103 |
## Benchmark and iteration
|
| 104 |
|
RELEASE_SCOPE.md
CHANGED
|
@@ -9,3 +9,9 @@ The runtime closure consists of original Laxmi source files extracted byte-for-b
|
|
| 9 |
Dependencies such as PyTorch, NumPy, safetensors and jsonschema are referenced as installation requirements and are not bundled. Their licenses and copyright holders remain applicable; Laxmi does not claim their authorship. The source ZIP contains the package manifest, Python sources, required original configuration/schema, build files and Laxmi license/notice files.
|
| 10 |
|
| 11 |
Repository-wide datasets and other artifacts outside this distribution receive no new redistribution grant from this release. Scientific model qualification remains open despite publication. License selection and distribution permission do not establish model accuracy or independence of evaluation.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
Dependencies such as PyTorch, NumPy, safetensors and jsonschema are referenced as installation requirements and are not bundled. Their licenses and copyright holders remain applicable; Laxmi does not claim their authorship. The source ZIP contains the package manifest, Python sources, required original configuration/schema, build files and Laxmi license/notice files.
|
| 10 |
|
| 11 |
Repository-wide datasets and other artifacts outside this distribution receive no new redistribution grant from this release. Scientific model qualification remains open despite publication. License selection and distribution permission do not establish model accuracy or independence of evaluation.
|
| 12 |
+
|
| 13 |
+
## v0.1.1 extension
|
| 14 |
+
|
| 15 |
+
The `evaluation/real-patch/` directory adds original Laxmi source, schema, documentation and a local wheel built from the source at commit `63cbecbad1bd8832d9c4fae65e4cfad27c57fa76`. These authored files are included under Apache 2.0, with exact file bytes enumerated by SHA256SUMS. The component contains no third-party repository code, real-patch case inputs, oracles, learned weights, corpus, private operational data or credentials. The public six-case counts and one-case exact-baseline result are summaries of exposed development evidence, not a distributed benchmark dataset.
|
| 16 |
+
|
| 17 |
+
The existing four controller candidates, matching runtime artifacts and synthetic example bytes are unchanged from v0.1.0. The new OpenAPI file describes a possible private service; no HTTP service is deployed or licensed as a maintained public endpoint. The wheel's Linux install/validation receipt does not establish Linux exact execution. Dependencies remain external with their own terms.
|
SHA256SUMS
CHANGED
|
@@ -1,11 +1,11 @@
|
|
| 1 |
78080e2966ee7383e24c55bf5ddd9e895fc80bdf9d2bb69de996e3d0b18f8cfb BENCHMARK.md
|
| 2 |
-
|
| 3 |
-
|
| 4 |
cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 LICENSE
|
| 5 |
3a66b62901c108c4f8946247ff7a773f73ac3bcec0b8a34286eb921164e97d5f LICENSE_SCOPE.md
|
| 6 |
3baa95979dcbe90444a854d04bbb6546c799e2077d9a69cfa34014a5c6512950 NOTICE
|
| 7 |
-
|
| 8 |
-
|
| 9 |
225a71bb67d619e70f1aec0313d2c5791d0a94ac3942297ad12ba284269cddd8 candidates/17-normalized/consumer.json
|
| 10 |
05478c5c7bf33bce2df7f7b1ffdffaa11dab3d6220558b1c2bb4d8432a5230e7 candidates/17-normalized/inference/config.json
|
| 11 |
e12fd2cc04ad23ae6b1c8937652a56fc08cac040e56a3f411e3388da6b4eb6b5 candidates/17-normalized/inference/manifest.json
|
|
@@ -23,10 +23,23 @@ b01a8a82dc6c7881b439006d6e24a60a40e59a4416ff9719ad34611c205af87c candidates/43-
|
|
| 23 |
d78f08c415e769d3e2676927e709c22d535a32ea5c79c6252ce0575af4abbcfd candidates/43-original/inference/manifest.json
|
| 24 |
604d14c55256a7dd3c8f56072eda97fd7517912bd4d965d416d76040428261cb candidates/43-original/inference/model.safetensors
|
| 25 |
ec1b954a497b276bb1806fbfcabde6e363f495f9dda60b1e686339a68bddd52d download-verification.json
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
0d98c572630d61128ca07c4f92939597704ba8964d3f40f9d04bf60bab8f761f examples/compare.json
|
| 27 |
2032524a11a31ff92634369060ff5a41adc6bc993161d6eb47a25a4839756ad1 examples/input.json
|
| 28 |
b208c06bd5829a5b18521711433338fdf2500ddcf312658c9d0b788b0fc0873c examples/outcome-template.json
|
| 29 |
-
|
| 30 |
b3bb0efc5b695baf36d6959e650baaaf4dd5bd32c0d6189a355b05185d45ac80 requirements-macos-arm64-py312.txt
|
| 31 |
67517f7c692d81580248e994c61829ddfdf2f9ef29358dd71a6b17eb953ef99f runtime/laxmi-readout-runtime-capsule-0.1.0-source.zip
|
| 32 |
47d03da88fa9bd9c1b4573ac4a0d5cb36c2cd1d106d687b68aa2968b713ee9b3 runtime/laxmi_readout_runtime_capsule-0.1.0-py3-none-macosx_11_0_arm64.whl
|
|
|
|
| 1 |
78080e2966ee7383e24c55bf5ddd9e895fc80bdf9d2bb69de996e3d0b18f8cfb BENCHMARK.md
|
| 2 |
+
f6e111f7762997f2a1310f08707a5af146615dab7b6f076d803dc92a517f1196 CHANGELOG.md
|
| 3 |
+
f42243d8ec7031d491961ebcbe70b70c9d410b1964f2080c73c4ab03bc9b01d8 CITATION.cff
|
| 4 |
cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 LICENSE
|
| 5 |
3a66b62901c108c4f8946247ff7a773f73ac3bcec0b8a34286eb921164e97d5f LICENSE_SCOPE.md
|
| 6 |
3baa95979dcbe90444a854d04bbb6546c799e2077d9a69cfa34014a5c6512950 NOTICE
|
| 7 |
+
af5cdcd4819c7997c61a1f19f1dbff5a4b254547ae6395f88964e0b0e6d5c24a README.md
|
| 8 |
+
f7dbdd9349019149fc54ae8bcf3edf442d5b53a62bf0905b57b88a1e3379a48c RELEASE_SCOPE.md
|
| 9 |
225a71bb67d619e70f1aec0313d2c5791d0a94ac3942297ad12ba284269cddd8 candidates/17-normalized/consumer.json
|
| 10 |
05478c5c7bf33bce2df7f7b1ffdffaa11dab3d6220558b1c2bb4d8432a5230e7 candidates/17-normalized/inference/config.json
|
| 11 |
e12fd2cc04ad23ae6b1c8937652a56fc08cac040e56a3f411e3388da6b4eb6b5 candidates/17-normalized/inference/manifest.json
|
|
|
|
| 23 |
d78f08c415e769d3e2676927e709c22d535a32ea5c79c6252ce0575af4abbcfd candidates/43-original/inference/manifest.json
|
| 24 |
604d14c55256a7dd3c8f56072eda97fd7517912bd4d965d416d76040428261cb candidates/43-original/inference/model.safetensors
|
| 25 |
ec1b954a497b276bb1806fbfcabde6e363f495f9dda60b1e686339a68bddd52d download-verification.json
|
| 26 |
+
c9693a57855ca9af6ff2f0ab681d3f7902b164dccc93602e6d58d22e8eecda0e evaluation/real-patch/README.md
|
| 27 |
+
a36e136b730b9b941af303ca23dcdaaff0268b3e3dfafab4122730ee45d8da0e evaluation/real-patch/runtime/laxmi_real_patch-0.1.0.dev1-py3-none-any.whl
|
| 28 |
+
cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30 evaluation/real-patch/source/LICENSE
|
| 29 |
+
3a66b62901c108c4f8946247ff7a773f73ac3bcec0b8a34286eb921164e97d5f evaluation/real-patch/source/LICENSE_SCOPE.md
|
| 30 |
+
3baa95979dcbe90444a854d04bbb6546c799e2077d9a69cfa34014a5c6512950 evaluation/real-patch/source/NOTICE
|
| 31 |
+
77aa1eb34fb78016565afcce6a27bbe18561bea3652544ac50107e5c770b45cd evaluation/real-patch/source/code_review/contracts/real_patch_api_v1/openapi.yaml
|
| 32 |
+
06a5bb4ca539bcb95ff97f58d2a003107b03a5af2a0b2fe21643d0c75590e1c0 evaluation/real-patch/source/code_review/contracts/real_patch_prediction_v1/README.md
|
| 33 |
+
ddd2f1a6f908334d4111971a14810fd5393d1735a00a39c3e745dd9d0f3d018c evaluation/real-patch/source/code_review/contracts/real_patch_prediction_v1/forecast_request.schema.json
|
| 34 |
+
23b973a4fdd00e148254723cbcada5be02c65e80818cba6b1f0f3dbd42cc6a55 evaluation/real-patch/source/code_review/contracts/real_patch_prediction_v1/validator.py
|
| 35 |
+
ef8e9e728670c20a75dfc50a26649a073d4d04943791dc02a3b4a3e73af9f3e7 evaluation/real-patch/source/code_review/packaging/real_patch/api.py
|
| 36 |
+
a9a3cf605b0f34de4f0dcde1a9378682a3fd9149dc112152c3cf418aa2a9df4e evaluation/real-patch/source/code_review/scripts/build_real_patch_local_package_v1.py
|
| 37 |
+
8538931b8044a98b1f245aaba4ec281b71a10e30944a46baf6ec34a4c659d9c8 evaluation/real-patch/source/code_review/scripts/run_real_patch_baseline_gate_v2.py
|
| 38 |
+
6344b680488213d222cca267f47ed54dcfecb0b1b4990033b4785d808ee54e3a evaluation/real-patch/source/code_review/scripts/run_real_patch_visible_exact_v1.py
|
| 39 |
0d98c572630d61128ca07c4f92939597704ba8964d3f40f9d04bf60bab8f761f examples/compare.json
|
| 40 |
2032524a11a31ff92634369060ff5a41adc6bc993161d6eb47a25a4839756ad1 examples/input.json
|
| 41 |
b208c06bd5829a5b18521711433338fdf2500ddcf312658c9d0b788b0fc0873c examples/outcome-template.json
|
| 42 |
+
0e02068ddc38d1f32f79c2003649695cf807857843784ebba179884e64ae8a69 release.json
|
| 43 |
b3bb0efc5b695baf36d6959e650baaaf4dd5bd32c0d6189a355b05185d45ac80 requirements-macos-arm64-py312.txt
|
| 44 |
67517f7c692d81580248e994c61829ddfdf2f9ef29358dd71a6b17eb953ef99f runtime/laxmi-readout-runtime-capsule-0.1.0-source.zip
|
| 45 |
47d03da88fa9bd9c1b4573ac4a0d5cb36c2cd1d106d687b68aa2968b713ee9b3 runtime/laxmi_readout_runtime_capsule-0.1.0-py3-none-macosx_11_0_arm64.whl
|
evaluation/real-patch/README.md
ADDED
|
@@ -0,0 +1,37 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Real-patch development evaluation tooling
|
| 2 |
+
|
| 3 |
+
This directory is part of **Laxmi experimental release v0.1.1**, maintained by Navneet Prabhakar. It is a tooling extension to the same Laxmi research project, not another model. The controller model weights and runtime remain byte-identical to v0.1.0. This directory contains **no learned real-patch weights**, user source, candidate data or oracle records.
|
| 4 |
+
|
| 5 |
+
The local package validates a paired pre-execution forecast request, executes the unchanged and proposed-patch branches from the **same visible source, dependency and scenario bytes**, seals the prediction, and scores only through a separately invoked oracle operation. Exact execution is a baseline. Its success must not be credited to a learned world model.
|
| 6 |
+
|
| 7 |
+
## Current evaluation
|
| 8 |
+
|
| 9 |
+
Six exposed historical changes are inventoried. Only one has a forecast packet under the current contract; the other five have oracle-only scenarios mostly adapted from post-change tests and are excluded from forecast accuracy. On that one exposed development case, exact execution matched both branch observations and the change effect. There are zero blind/acceptance-eligible cases and no learned real-patch candidate scored. The selected scenario was projected after the outcome was known. No evaluation case data is distributed in this release.
|
| 10 |
+
|
| 11 |
+
## Local installation
|
| 12 |
+
|
| 13 |
+
From the downloaded release root in a fresh Python 3.12 environment:
|
| 14 |
+
|
| 15 |
+
```bash
|
| 16 |
+
python3.12 -m venv .venv-real-patch
|
| 17 |
+
.venv-real-patch/bin/python -m pip install evaluation/real-patch/runtime/laxmi_real_patch-0.1.0.dev1-py3-none-any.whl
|
| 18 |
+
.venv-real-patch/bin/laxmi-real-patch validate --request /path/to/forecast_request.json
|
| 19 |
+
```
|
| 20 |
+
|
| 21 |
+
The [request contract](source/code_review/contracts/real_patch_prediction_v1/README.md) and [schema](source/code_review/contracts/real_patch_prediction_v1/forecast_request.schema.json) specify model-visible input. A request must have independently reviewed source/scenario provenance; passing schema validation cannot establish truth of self-declared timestamps or acceptance eligibility. The installed interface explicitly reports `custody_verified=false`, `acceptance_eligible=false` and `development_only=true`.
|
| 22 |
+
|
| 23 |
+
`forecast` requires a **new** output directory and a suitable local Docker runtime with `python:3.12-slim` resolving to image ID `sha256:78387bc3881b8273120a12ebe6c1ab22b018ccc2c9adf565ae1ac9b536e184ea`:
|
| 24 |
+
|
| 25 |
+
```bash
|
| 26 |
+
.venv-real-patch/bin/laxmi-real-patch forecast --request /path/to/forecast_request.json --output-dir ./forecast-001
|
| 27 |
+
```
|
| 28 |
+
|
| 29 |
+
The comparator does not read oracles. A separately authorized evaluator can later score a sealed prediction:
|
| 30 |
+
|
| 31 |
+
```bash
|
| 32 |
+
.venv-real-patch/bin/laxmi-real-patch score --request /path/to/forecast_request.json --oracle /path/to/gate-oracle.json --output-dir ./forecast-001
|
| 33 |
+
```
|
| 34 |
+
|
| 35 |
+
The wheel SHA-256 is `a36e136b730b9b941af303ca23dcdaaff0268b3e3dfafab4122730ee45d8da0e`. A fresh macOS ARM64 Python 3.12 install reproduced the original prediction and score receipt bytes. Offline Linux ARM64 installation and **validation** returned the same request hash; Linux exact execution was not qualified. The wheel tag `py3-none-any` is packaging metadata, not a blanket portability claim. The exact comparator executes supplied scenario code in an isolated local Docker container; do not expose this CLI as a public code-execution service.
|
| 36 |
+
|
| 37 |
+
The [OpenAPI document](source/code_review/contracts/real_patch_api_v1/openapi.yaml) is **design-only**. There is no hosted API, authenticated job service or production deployment. Source files and the builder are provided under [`source/`](source/) for inspection; build the wheel from that directory with `python code_review/scripts/build_real_patch_local_package_v1.py --output-dir /new/output`. The root [LICENSE](../../LICENSE), [NOTICE](../../NOTICE), [RELEASE_SCOPE.md](../../RELEASE_SCOPE.md) and [SHA256SUMS](../../SHA256SUMS) govern this release.
|
evaluation/real-patch/runtime/laxmi_real_patch-0.1.0.dev1-py3-none-any.whl
ADDED
|
Binary file (23.1 kB). View file
|
|
|
evaluation/real-patch/source/LICENSE
ADDED
|
@@ -0,0 +1,202 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
Apache License
|
| 3 |
+
Version 2.0, January 2004
|
| 4 |
+
http://www.apache.org/licenses/
|
| 5 |
+
|
| 6 |
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
| 7 |
+
|
| 8 |
+
1. Definitions.
|
| 9 |
+
|
| 10 |
+
"License" shall mean the terms and conditions for use, reproduction,
|
| 11 |
+
and distribution as defined by Sections 1 through 9 of this document.
|
| 12 |
+
|
| 13 |
+
"Licensor" shall mean the copyright owner or entity authorized by
|
| 14 |
+
the copyright owner that is granting the License.
|
| 15 |
+
|
| 16 |
+
"Legal Entity" shall mean the union of the acting entity and all
|
| 17 |
+
other entities that control, are controlled by, or are under common
|
| 18 |
+
control with that entity. For the purposes of this definition,
|
| 19 |
+
"control" means (i) the power, direct or indirect, to cause the
|
| 20 |
+
direction or management of such entity, whether by contract or
|
| 21 |
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
| 22 |
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
| 23 |
+
|
| 24 |
+
"You" (or "Your") shall mean an individual or Legal Entity
|
| 25 |
+
exercising permissions granted by this License.
|
| 26 |
+
|
| 27 |
+
"Source" form shall mean the preferred form for making modifications,
|
| 28 |
+
including but not limited to software source code, documentation
|
| 29 |
+
source, and configuration files.
|
| 30 |
+
|
| 31 |
+
"Object" form shall mean any form resulting from mechanical
|
| 32 |
+
transformation or translation of a Source form, including but
|
| 33 |
+
not limited to compiled object code, generated documentation,
|
| 34 |
+
and conversions to other media types.
|
| 35 |
+
|
| 36 |
+
"Work" shall mean the work of authorship, whether in Source or
|
| 37 |
+
Object form, made available under the License, as indicated by a
|
| 38 |
+
copyright notice that is included in or attached to the work
|
| 39 |
+
(an example is provided in the Appendix below).
|
| 40 |
+
|
| 41 |
+
"Derivative Works" shall mean any work, whether in Source or Object
|
| 42 |
+
form, that is based on (or derived from) the Work and for which the
|
| 43 |
+
editorial revisions, annotations, elaborations, or other modifications
|
| 44 |
+
represent, as a whole, an original work of authorship. For the purposes
|
| 45 |
+
of this License, Derivative Works shall not include works that remain
|
| 46 |
+
separable from, or merely link (or bind by name) to the interfaces of,
|
| 47 |
+
the Work and Derivative Works thereof.
|
| 48 |
+
|
| 49 |
+
"Contribution" shall mean any work of authorship, including
|
| 50 |
+
the original version of the Work and any modifications or additions
|
| 51 |
+
to that Work or Derivative Works thereof, that is intentionally
|
| 52 |
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
| 53 |
+
or by an individual or Legal Entity authorized to submit on behalf of
|
| 54 |
+
the copyright owner. For the purposes of this definition, "submitted"
|
| 55 |
+
means any form of electronic, verbal, or written communication sent
|
| 56 |
+
to the Licensor or its representatives, including but not limited to
|
| 57 |
+
communication on electronic mailing lists, source code control systems,
|
| 58 |
+
and issue tracking systems that are managed by, or on behalf of, the
|
| 59 |
+
Licensor for the purpose of discussing and improving the Work, but
|
| 60 |
+
excluding communication that is conspicuously marked or otherwise
|
| 61 |
+
designated in writing by the copyright owner as "Not a Contribution."
|
| 62 |
+
|
| 63 |
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
| 64 |
+
on behalf of whom a Contribution has been received by Licensor and
|
| 65 |
+
subsequently incorporated within the Work.
|
| 66 |
+
|
| 67 |
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
| 68 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 69 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 70 |
+
copyright license to reproduce, prepare Derivative Works of,
|
| 71 |
+
publicly display, publicly perform, sublicense, and distribute the
|
| 72 |
+
Work and such Derivative Works in Source or Object form.
|
| 73 |
+
|
| 74 |
+
3. Grant of Patent License. Subject to the terms and conditions of
|
| 75 |
+
this License, each Contributor hereby grants to You a perpetual,
|
| 76 |
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
| 77 |
+
(except as stated in this section) patent license to make, have made,
|
| 78 |
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
| 79 |
+
where such license applies only to those patent claims licensable
|
| 80 |
+
by such Contributor that are necessarily infringed by their
|
| 81 |
+
Contribution(s) alone or by combination of their Contribution(s)
|
| 82 |
+
with the Work to which such Contribution(s) was submitted. If You
|
| 83 |
+
institute patent litigation against any entity (including a
|
| 84 |
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
| 85 |
+
or a Contribution incorporated within the Work constitutes direct
|
| 86 |
+
or contributory patent infringement, then any patent licenses
|
| 87 |
+
granted to You under this License for that Work shall terminate
|
| 88 |
+
as of the date such litigation is filed.
|
| 89 |
+
|
| 90 |
+
4. Redistribution. You may reproduce and distribute copies of the
|
| 91 |
+
Work or Derivative Works thereof in any medium, with or without
|
| 92 |
+
modifications, and in Source or Object form, provided that You
|
| 93 |
+
meet the following conditions:
|
| 94 |
+
|
| 95 |
+
(a) You must give any other recipients of the Work or
|
| 96 |
+
Derivative Works a copy of this License; and
|
| 97 |
+
|
| 98 |
+
(b) You must cause any modified files to carry prominent notices
|
| 99 |
+
stating that You changed the files; and
|
| 100 |
+
|
| 101 |
+
(c) You must retain, in the Source form of any Derivative Works
|
| 102 |
+
that You distribute, all copyright, patent, trademark, and
|
| 103 |
+
attribution notices from the Source form of the Work,
|
| 104 |
+
excluding those notices that do not pertain to any part of
|
| 105 |
+
the Derivative Works; and
|
| 106 |
+
|
| 107 |
+
(d) If the Work includes a "NOTICE" text file as part of its
|
| 108 |
+
distribution, then any Derivative Works that You distribute must
|
| 109 |
+
include a readable copy of the attribution notices contained
|
| 110 |
+
within such NOTICE file, excluding those notices that do not
|
| 111 |
+
pertain to any part of the Derivative Works, in at least one
|
| 112 |
+
of the following places: within a NOTICE text file distributed
|
| 113 |
+
as part of the Derivative Works; within the Source form or
|
| 114 |
+
documentation, if provided along with the Derivative Works; or,
|
| 115 |
+
within a display generated by the Derivative Works, if and
|
| 116 |
+
wherever such third-party notices normally appear. The contents
|
| 117 |
+
of the NOTICE file are for informational purposes only and
|
| 118 |
+
do not modify the License. You may add Your own attribution
|
| 119 |
+
notices within Derivative Works that You distribute, alongside
|
| 120 |
+
or as an addendum to the NOTICE text from the Work, provided
|
| 121 |
+
that such additional attribution notices cannot be construed
|
| 122 |
+
as modifying the License.
|
| 123 |
+
|
| 124 |
+
You may add Your own copyright statement to Your modifications and
|
| 125 |
+
may provide additional or different license terms and conditions
|
| 126 |
+
for use, reproduction, or distribution of Your modifications, or
|
| 127 |
+
for any such Derivative Works as a whole, provided Your use,
|
| 128 |
+
reproduction, and distribution of the Work otherwise complies with
|
| 129 |
+
the conditions stated in this License.
|
| 130 |
+
|
| 131 |
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
| 132 |
+
any Contribution intentionally submitted for inclusion in the Work
|
| 133 |
+
by You to the Licensor shall be under the terms and conditions of
|
| 134 |
+
this License, without any additional terms or conditions.
|
| 135 |
+
Notwithstanding the above, nothing herein shall supersede or modify
|
| 136 |
+
the terms of any separate license agreement you may have executed
|
| 137 |
+
with Licensor regarding such Contributions.
|
| 138 |
+
|
| 139 |
+
6. Trademarks. This License does not grant permission to use the trade
|
| 140 |
+
names, trademarks, service marks, or product names of the Licensor,
|
| 141 |
+
except as required for reasonable and customary use in describing the
|
| 142 |
+
origin of the Work and reproducing the content of the NOTICE file.
|
| 143 |
+
|
| 144 |
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
| 145 |
+
agreed to in writing, Licensor provides the Work (and each
|
| 146 |
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
| 147 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
| 148 |
+
implied, including, without limitation, any warranties or conditions
|
| 149 |
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
| 150 |
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
| 151 |
+
appropriateness of using or redistributing the Work and assume any
|
| 152 |
+
risks associated with Your exercise of permissions under this License.
|
| 153 |
+
|
| 154 |
+
8. Limitation of Liability. In no event and under no legal theory,
|
| 155 |
+
whether in tort (including negligence), contract, or otherwise,
|
| 156 |
+
unless required by applicable law (such as deliberate and grossly
|
| 157 |
+
negligent acts) or agreed to in writing, shall any Contributor be
|
| 158 |
+
liable to You for damages, including any direct, indirect, special,
|
| 159 |
+
incidental, or consequential damages of any character arising as a
|
| 160 |
+
result of this License or out of the use or inability to use the
|
| 161 |
+
Work (including but not limited to damages for loss of goodwill,
|
| 162 |
+
work stoppage, computer failure or malfunction, or any and all
|
| 163 |
+
other commercial damages or losses), even if such Contributor
|
| 164 |
+
has been advised of the possibility of such damages.
|
| 165 |
+
|
| 166 |
+
9. Accepting Warranty or Additional Liability. While redistributing
|
| 167 |
+
the Work or Derivative Works thereof, You may choose to offer,
|
| 168 |
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
| 169 |
+
or other liability obligations and/or rights consistent with this
|
| 170 |
+
License. However, in accepting such obligations, You may act only
|
| 171 |
+
on Your own behalf and on Your sole responsibility, not on behalf
|
| 172 |
+
of any other Contributor, and only if You agree to indemnify,
|
| 173 |
+
defend, and hold each Contributor harmless for any liability
|
| 174 |
+
incurred by, or claims asserted against, such Contributor by reason
|
| 175 |
+
of your accepting any such warranty or additional liability.
|
| 176 |
+
|
| 177 |
+
END OF TERMS AND CONDITIONS
|
| 178 |
+
|
| 179 |
+
APPENDIX: How to apply the Apache License to your work.
|
| 180 |
+
|
| 181 |
+
To apply the Apache License to your work, attach the following
|
| 182 |
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
| 183 |
+
replaced with your own identifying information. (Don't include
|
| 184 |
+
the brackets!) The text should be enclosed in the appropriate
|
| 185 |
+
comment syntax for the file format. We also recommend that a
|
| 186 |
+
file or class name and description of purpose be included on the
|
| 187 |
+
same "printed page" as the copyright notice for easier
|
| 188 |
+
identification within third-party archives.
|
| 189 |
+
|
| 190 |
+
Copyright [yyyy] [name of copyright owner]
|
| 191 |
+
|
| 192 |
+
Licensed under the Apache License, Version 2.0 (the "License");
|
| 193 |
+
you may not use this file except in compliance with the License.
|
| 194 |
+
You may obtain a copy of the License at
|
| 195 |
+
|
| 196 |
+
http://www.apache.org/licenses/LICENSE-2.0
|
| 197 |
+
|
| 198 |
+
Unless required by applicable law or agreed to in writing, software
|
| 199 |
+
distributed under the License is distributed on an "AS IS" BASIS,
|
| 200 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
| 201 |
+
See the License for the specific language governing permissions and
|
| 202 |
+
limitations under the License.
|
evaluation/real-patch/source/LICENSE_SCOPE.md
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Licensing scope
|
| 2 |
+
|
| 3 |
+
Copyright 2026 Navneet Prabhakar.
|
| 4 |
+
|
| 5 |
+
Navneet Prabhakar's original Laxmi source code, documentation, configuration,
|
| 6 |
+
schemas and authored test fixtures are licensed under Apache License 2.0,
|
| 7 |
+
unless an individual file explicitly states different terms. LICENSE contains
|
| 8 |
+
the unmodified license text; NOTICE records project attribution. Future accepted
|
| 9 |
+
contributions follow the license's contribution terms unless separately agreed.
|
| 10 |
+
|
| 11 |
+
Apache 2.0 is also the selected license for original Laxmi model weights shipped
|
| 12 |
+
in an explicitly identified release after provenance and redistribution review.
|
| 13 |
+
Every model release must identify its exact weights and applicable terms. The
|
| 14 |
+
presence of a local checkpoint does not alone establish its release eligibility.
|
| 15 |
+
|
| 16 |
+
The repository license does not relicense third-party source, dependencies,
|
| 17 |
+
models, tokenizers, copied documents or other third-party material. Their existing
|
| 18 |
+
terms and required notices remain applicable. Incorporating or linking a dependency
|
| 19 |
+
does not claim its authorship or change its license.
|
| 20 |
+
|
| 21 |
+
Acquired corpora, datasets, repository clones, training/evaluation records,
|
| 22 |
+
operational archives and generated evidence under data/, results/, archive/ and
|
| 23 |
+
the corresponding code_review/ directories are not granted redistribution rights
|
| 24 |
+
by this repository license alone. Individual released datasets or artifact bundles
|
| 25 |
+
require an explicit scope manifest and their applicable terms. Original source or
|
| 26 |
+
documentation in those directories may be expressly included in such a manifest.
|
| 27 |
+
Private inputs, credentials and operational records are not public release assets.
|
| 28 |
+
|
| 29 |
+
## Attribution and citation
|
| 30 |
+
|
| 31 |
+
Redistributors must comply with Apache 2.0's license and applicable notice
|
| 32 |
+
preservation requirements. The NOTICE file provides Laxmi's attribution. It adds
|
| 33 |
+
no requirement to display a badge, advertise the author or obtain permission for
|
| 34 |
+
permitted uses. CITATION.cff makes a requested research citation convenient;
|
| 35 |
+
citation is not an additional condition of the license.
|
| 36 |
+
|
| 37 |
+
Navneet Prabhakar remains Laxmi's original creator. Contributor lists should
|
| 38 |
+
reflect later accepted contributions if the project gains other contributors.
|
evaluation/real-patch/source/NOTICE
ADDED
|
@@ -0,0 +1,12 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Laxmi
|
| 2 |
+
Copyright 2026 Navneet Prabhakar
|
| 3 |
+
|
| 4 |
+
Created and maintained by Navneet Prabhakar.
|
| 5 |
+
Project: https://github.com/navneetprabhakar/laxmi
|
| 6 |
+
|
| 7 |
+
Laxmi's original contributions are licensed under the Apache License,
|
| 8 |
+
Version 2.0. See LICENSE and LICENSE_SCOPE.md for applicable scope.
|
| 9 |
+
|
| 10 |
+
Third-party libraries, models and other materials retain their respective
|
| 11 |
+
copyrights, licenses and attribution requirements. This notice does not
|
| 12 |
+
claim authorship of those materials or imply their authors' endorsement.
|
evaluation/real-patch/source/code_review/contracts/real_patch_api_v1/openapi.yaml
ADDED
|
@@ -0,0 +1,251 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
openapi: 3.1.0
|
| 2 |
+
info:
|
| 3 |
+
title: Laxmi real-patch private API
|
| 4 |
+
version: 0.1.0-design
|
| 5 |
+
description: >-
|
| 6 |
+
Design-only asynchronous transport over the real-patch forecast contract.
|
| 7 |
+
No hosted implementation, learned real-patch model, or public endpoint exists.
|
| 8 |
+
x-implementation-status: DESIGN_ONLY
|
| 9 |
+
paths:
|
| 10 |
+
/v1/real-patch/requests:validate:
|
| 11 |
+
post:
|
| 12 |
+
operationId: validateRealPatchRequest
|
| 13 |
+
summary: Validate model-visible input without execution or oracle access
|
| 14 |
+
security:
|
| 15 |
+
- bearerAuth: []
|
| 16 |
+
requestBody:
|
| 17 |
+
required: true
|
| 18 |
+
content:
|
| 19 |
+
application/json:
|
| 20 |
+
schema:
|
| 21 |
+
$ref: ../real_patch_prediction_v1/forecast_request.schema.json
|
| 22 |
+
responses:
|
| 23 |
+
'200':
|
| 24 |
+
description: Structural admission only; not proof of external provenance
|
| 25 |
+
content:
|
| 26 |
+
application/json:
|
| 27 |
+
schema:
|
| 28 |
+
$ref: '#/components/schemas/ValidationResult'
|
| 29 |
+
'422':
|
| 30 |
+
$ref: '#/components/responses/Refusal'
|
| 31 |
+
/v1/real-patch/forecast-jobs:
|
| 32 |
+
post:
|
| 33 |
+
operationId: submitRealPatchForecast
|
| 34 |
+
summary: Submit a sealed-before-outcome forecast job
|
| 35 |
+
security:
|
| 36 |
+
- bearerAuth: []
|
| 37 |
+
parameters:
|
| 38 |
+
- name: Idempotency-Key
|
| 39 |
+
in: header
|
| 40 |
+
required: true
|
| 41 |
+
schema:
|
| 42 |
+
type: string
|
| 43 |
+
minLength: 16
|
| 44 |
+
maxLength: 128
|
| 45 |
+
requestBody:
|
| 46 |
+
required: true
|
| 47 |
+
content:
|
| 48 |
+
application/json:
|
| 49 |
+
schema:
|
| 50 |
+
type: object
|
| 51 |
+
additionalProperties: false
|
| 52 |
+
required: [request, engine_id, engine_version]
|
| 53 |
+
properties:
|
| 54 |
+
request:
|
| 55 |
+
$ref: ../real_patch_prediction_v1/forecast_request.schema.json
|
| 56 |
+
engine_id:
|
| 57 |
+
type: string
|
| 58 |
+
enum: [EXACT_VISIBLE_SOURCE, STATIC_RULES, RETRIEVAL, LEARNED]
|
| 59 |
+
engine_version:
|
| 60 |
+
type: string
|
| 61 |
+
responses:
|
| 62 |
+
'202':
|
| 63 |
+
description: Create-only job accepted; no prediction is implied
|
| 64 |
+
content:
|
| 65 |
+
application/json:
|
| 66 |
+
schema:
|
| 67 |
+
$ref: '#/components/schemas/JobStatus'
|
| 68 |
+
'409':
|
| 69 |
+
$ref: '#/components/responses/Refusal'
|
| 70 |
+
'422':
|
| 71 |
+
$ref: '#/components/responses/Refusal'
|
| 72 |
+
/v1/real-patch/forecast-jobs/{id}:
|
| 73 |
+
get:
|
| 74 |
+
operationId: getRealPatchForecastJob
|
| 75 |
+
security:
|
| 76 |
+
- bearerAuth: []
|
| 77 |
+
parameters:
|
| 78 |
+
- $ref: '#/components/parameters/JobId'
|
| 79 |
+
responses:
|
| 80 |
+
'200':
|
| 81 |
+
description: Job state, including unsupported and failed states
|
| 82 |
+
content:
|
| 83 |
+
application/json:
|
| 84 |
+
schema:
|
| 85 |
+
$ref: '#/components/schemas/JobStatus'
|
| 86 |
+
/v1/real-patch/forecast-jobs/{id}/prediction:
|
| 87 |
+
get:
|
| 88 |
+
operationId: getSealedRealPatchPrediction
|
| 89 |
+
security:
|
| 90 |
+
- bearerAuth: []
|
| 91 |
+
parameters:
|
| 92 |
+
- $ref: '#/components/parameters/JobId'
|
| 93 |
+
responses:
|
| 94 |
+
'200':
|
| 95 |
+
description: Prediction receipt without oracle or score
|
| 96 |
+
content:
|
| 97 |
+
application/json:
|
| 98 |
+
schema:
|
| 99 |
+
$ref: '#/components/schemas/PredictionResult'
|
| 100 |
+
'409':
|
| 101 |
+
$ref: '#/components/responses/Refusal'
|
| 102 |
+
/v1/real-patch/outcomes:
|
| 103 |
+
post:
|
| 104 |
+
operationId: attachRealPatchOutcome
|
| 105 |
+
summary: Privileged evaluator attaches observed outcome after forecast seal
|
| 106 |
+
security:
|
| 107 |
+
- evaluatorAuth: []
|
| 108 |
+
requestBody:
|
| 109 |
+
required: true
|
| 110 |
+
content:
|
| 111 |
+
application/json:
|
| 112 |
+
schema:
|
| 113 |
+
type: object
|
| 114 |
+
additionalProperties: false
|
| 115 |
+
required: [job_id, oracle_sha256, oracle_ref]
|
| 116 |
+
properties:
|
| 117 |
+
job_id:
|
| 118 |
+
type: string
|
| 119 |
+
oracle_sha256:
|
| 120 |
+
$ref: '#/components/schemas/Sha256'
|
| 121 |
+
oracle_ref:
|
| 122 |
+
type: string
|
| 123 |
+
description: Evaluator-only immutable artifact reference
|
| 124 |
+
responses:
|
| 125 |
+
'201':
|
| 126 |
+
description: Outcome attached immutably after sealed prediction
|
| 127 |
+
'409':
|
| 128 |
+
$ref: '#/components/responses/Refusal'
|
| 129 |
+
/v1/real-patch/evaluations/{id}:
|
| 130 |
+
get:
|
| 131 |
+
operationId: getRealPatchEvaluation
|
| 132 |
+
summary: Retrieve an evaluator-only score and eligibility receipt
|
| 133 |
+
security:
|
| 134 |
+
- evaluatorAuth: []
|
| 135 |
+
parameters:
|
| 136 |
+
- $ref: '#/components/parameters/JobId'
|
| 137 |
+
responses:
|
| 138 |
+
'200':
|
| 139 |
+
description: Scored result with the attempted-case denominator
|
| 140 |
+
content:
|
| 141 |
+
application/json:
|
| 142 |
+
schema:
|
| 143 |
+
$ref: '#/components/schemas/EvaluationResult'
|
| 144 |
+
components:
|
| 145 |
+
securitySchemes:
|
| 146 |
+
bearerAuth:
|
| 147 |
+
type: http
|
| 148 |
+
scheme: bearer
|
| 149 |
+
evaluatorAuth:
|
| 150 |
+
type: http
|
| 151 |
+
scheme: bearer
|
| 152 |
+
description: Separate evaluator principal with outcome-storage access
|
| 153 |
+
parameters:
|
| 154 |
+
JobId:
|
| 155 |
+
name: id
|
| 156 |
+
in: path
|
| 157 |
+
required: true
|
| 158 |
+
schema:
|
| 159 |
+
type: string
|
| 160 |
+
schemas:
|
| 161 |
+
Sha256:
|
| 162 |
+
type: string
|
| 163 |
+
pattern: '^[0-9a-f]{64}$'
|
| 164 |
+
ValidationResult:
|
| 165 |
+
type: object
|
| 166 |
+
additionalProperties: false
|
| 167 |
+
required: [status, input_sha256, custody_verified, acceptance_eligible]
|
| 168 |
+
properties:
|
| 169 |
+
status:
|
| 170 |
+
const: ADMITTED
|
| 171 |
+
input_sha256:
|
| 172 |
+
$ref: '#/components/schemas/Sha256'
|
| 173 |
+
custody_verified:
|
| 174 |
+
const: false
|
| 175 |
+
acceptance_eligible:
|
| 176 |
+
const: false
|
| 177 |
+
JobStatus:
|
| 178 |
+
type: object
|
| 179 |
+
additionalProperties: false
|
| 180 |
+
required: [job_id, status, input_sha256, engine_id, engine_version]
|
| 181 |
+
properties:
|
| 182 |
+
job_id:
|
| 183 |
+
type: string
|
| 184 |
+
status:
|
| 185 |
+
enum: [QUEUED, RUNNING, PREDICTED, UNSUPPORTED, FAILED]
|
| 186 |
+
input_sha256:
|
| 187 |
+
$ref: '#/components/schemas/Sha256'
|
| 188 |
+
engine_id:
|
| 189 |
+
type: string
|
| 190 |
+
engine_version:
|
| 191 |
+
type: string
|
| 192 |
+
reason_code:
|
| 193 |
+
type: string
|
| 194 |
+
PredictionResult:
|
| 195 |
+
type: object
|
| 196 |
+
additionalProperties: false
|
| 197 |
+
required: [job_id, status, input_sha256, engine_id, method_sha256, information_tier, observations, seal_sha256]
|
| 198 |
+
properties:
|
| 199 |
+
job_id:
|
| 200 |
+
type: string
|
| 201 |
+
status:
|
| 202 |
+
const: PREDICTED
|
| 203 |
+
input_sha256:
|
| 204 |
+
$ref: '#/components/schemas/Sha256'
|
| 205 |
+
engine_id:
|
| 206 |
+
type: string
|
| 207 |
+
method_sha256:
|
| 208 |
+
$ref: '#/components/schemas/Sha256'
|
| 209 |
+
information_tier:
|
| 210 |
+
enum: [SAME_INFORMATION, PRIVILEGED_CEILING]
|
| 211 |
+
observations:
|
| 212 |
+
type: object
|
| 213 |
+
description: Exactly the declared branch fields from the validated request
|
| 214 |
+
seal_sha256:
|
| 215 |
+
$ref: '#/components/schemas/Sha256'
|
| 216 |
+
EvaluationResult:
|
| 217 |
+
type: object
|
| 218 |
+
additionalProperties: false
|
| 219 |
+
required: [evaluation_id, evaluation_split, dataset_sha256, candidate_count, scored_case_count, unsupported_case_count, receipt_sha256]
|
| 220 |
+
properties:
|
| 221 |
+
evaluation_id:
|
| 222 |
+
type: string
|
| 223 |
+
evaluation_split:
|
| 224 |
+
enum: [DEVELOPMENT_EXPOSED, ACCEPTANCE_CANDIDATE]
|
| 225 |
+
dataset_sha256:
|
| 226 |
+
$ref: '#/components/schemas/Sha256'
|
| 227 |
+
candidate_count:
|
| 228 |
+
type: integer
|
| 229 |
+
minimum: 1
|
| 230 |
+
scored_case_count:
|
| 231 |
+
type: integer
|
| 232 |
+
minimum: 0
|
| 233 |
+
unsupported_case_count:
|
| 234 |
+
type: integer
|
| 235 |
+
minimum: 0
|
| 236 |
+
receipt_sha256:
|
| 237 |
+
$ref: '#/components/schemas/Sha256'
|
| 238 |
+
responses:
|
| 239 |
+
Refusal:
|
| 240 |
+
description: Request refused without a prediction
|
| 241 |
+
content:
|
| 242 |
+
application/json:
|
| 243 |
+
schema:
|
| 244 |
+
type: object
|
| 245 |
+
additionalProperties: false
|
| 246 |
+
required: [error, reason_code]
|
| 247 |
+
properties:
|
| 248 |
+
error:
|
| 249 |
+
type: string
|
| 250 |
+
reason_code:
|
| 251 |
+
type: string
|
evaluation/real-patch/source/code_review/contracts/real_patch_prediction_v1/README.md
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Real-patch forecast input v1
|
| 2 |
+
|
| 3 |
+
`forecast_request.schema.json` defines the **entire model-visible request** for a
|
| 4 |
+
pre-execution forecast. `validator.py` additionally checks content hashes,
|
| 5 |
+
chronology, source/patch lineage, duplicate identities and patch file roles.
|
| 6 |
+
All consumers receive the same canonical request bytes; `shared_input.input_sha256`
|
| 7 |
+
is SHA-256 of canonical JSON excluding `shared_input` itself. Canonical JSON uses
|
| 8 |
+
sorted keys, compact separators, UTF-8, and no non-finite numbers. A comparator
|
| 9 |
+
may report `unsupported`, but may not substitute more informative material.
|
| 10 |
+
|
| 11 |
+
The request contains an opaque repository alias and a hash of the visible source
|
| 12 |
+
file manifest. Explicit repository URL, Git commit, issue locator, selection notes,
|
| 13 |
+
PR metadata, oracle code/results and any future test metadata live in a **separate custody
|
| 14 |
+
record** that must never be passed to a forecast consumer. Pinned source bytes
|
| 15 |
+
may themselves contain public repository identifiers; opaque metadata does not
|
| 16 |
+
remove this contamination risk. The visible proposed
|
| 17 |
+
diff may change production files only; tests and fixtures are rejected. Scenario
|
| 18 |
+
input is structured as entrypoint plus JSON arguments, with an artifact hash and
|
| 19 |
+
availability timestamp. The same sanitized Python `program` bytes are visible
|
| 20 |
+
to every comparator and bind the exact runner; an AST check rejects assertions
|
| 21 |
+
and obvious expected-value variables. There is no field for issue prose,
|
| 22 |
+
expected values, diagnosis, PR title/body, or post-patch result. Pre-patch history can
|
| 23 |
+
include observed outcomes on the pinned base source only.
|
| 24 |
+
|
| 25 |
+
The schema cannot prove that a timestamp is true, that a source slice really
|
| 26 |
+
comes from the claimed immutable snapshot, or that a string argument encodes no
|
| 27 |
+
answer. An assembler must verify external artifact timestamps, version history,
|
| 28 |
+
licenses, patch provenance and scenario selection against the custody record
|
| 29 |
+
before signing a case. A reviewer must inspect each assembled packet for semantic
|
| 30 |
+
leakage. Historical public code may still be memorized by a pretrained model;
|
| 31 |
+
opaque IDs reduce direct lookup, but do not establish contamination freedom.
|
| 32 |
+
|
| 33 |
+
`target` requires paired `NO_CHANGE` and `PROPOSED_PATCH` branches under the
|
| 34 |
+
same invocation and bounded completion horizon. It declares observable fields
|
| 35 |
+
and the categorical vocabulary, never their values. The sealed oracle and
|
| 36 |
+
predictor outputs are separate files. This contract
|
| 37 |
+
does not itself establish that a learned predictor beats exact execution, static
|
| 38 |
+
rules or retrieval.
|
evaluation/real-patch/source/code_review/contracts/real_patch_prediction_v1/forecast_request.schema.json
ADDED
|
@@ -0,0 +1,137 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
| 3 |
+
"$id": "https://laxmi.local/contracts/real-patch-prediction/v1/forecast-request.schema.json",
|
| 4 |
+
"title": "Real patch pre-execution forecast request v1",
|
| 5 |
+
"description": "Model-visible information only. Oracle, PR-added tests, expected assertions, diagnoses and post-patch outcomes belong in separate sealed records.",
|
| 6 |
+
"type": "object",
|
| 7 |
+
"additionalProperties": false,
|
| 8 |
+
"required": ["schema_version", "case_id", "evaluation_split", "forecast_cutoff_at", "pre_source", "proposed_patch", "scenario", "operating_conditions", "history", "target", "shared_input"],
|
| 9 |
+
"properties": {
|
| 10 |
+
"schema_version": {"const": "1.0.0"},
|
| 11 |
+
"case_id": {"$ref": "#/$defs/id"},
|
| 12 |
+
"evaluation_split": {"enum": ["DEVELOPMENT_EXPOSED", "ACCEPTANCE_CANDIDATE"]},
|
| 13 |
+
"forecast_cutoff_at": {"$ref": "#/$defs/timestamp"},
|
| 14 |
+
"pre_source": {
|
| 15 |
+
"type": "object", "additionalProperties": false,
|
| 16 |
+
"required": ["repository_alias", "snapshot_sha256", "snapshot_at", "files"],
|
| 17 |
+
"properties": {
|
| 18 |
+
"repository_alias": {"$ref": "#/$defs/id"},
|
| 19 |
+
"snapshot_sha256": {"$ref": "#/$defs/sha256"},
|
| 20 |
+
"snapshot_at": {"$ref": "#/$defs/timestamp"},
|
| 21 |
+
"files": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/sourceFile"}}
|
| 22 |
+
}
|
| 23 |
+
},
|
| 24 |
+
"proposed_patch": {
|
| 25 |
+
"type": "object", "additionalProperties": false,
|
| 26 |
+
"required": ["base_snapshot_sha256", "diff_sha256", "unified_diff", "proposed_at", "provenance"],
|
| 27 |
+
"properties": {
|
| 28 |
+
"base_snapshot_sha256": {"$ref": "#/$defs/sha256"},
|
| 29 |
+
"diff_sha256": {"$ref": "#/$defs/sha256"},
|
| 30 |
+
"unified_diff": {"type": "string", "minLength": 1},
|
| 31 |
+
"proposed_at": {"$ref": "#/$defs/timestamp"},
|
| 32 |
+
"provenance": {"const": "CANDIDATE_PATCH_AT_FORECAST"}
|
| 33 |
+
}
|
| 34 |
+
},
|
| 35 |
+
"scenario": {
|
| 36 |
+
"type": "object", "additionalProperties": false,
|
| 37 |
+
"required": ["origin", "available_at", "source_artifact_sha256", "selection_policy", "projection_policy", "program", "program_sha256", "setup_assignments", "invocations"],
|
| 38 |
+
"properties": {
|
| 39 |
+
"origin": {"enum": ["PRE_PATCH_WORKLOAD_TRACE", "PRE_PATCH_ISSUE_REPRO", "PRE_PATCH_TEST", "PROSPECTIVE_GENERATOR"]},
|
| 40 |
+
"available_at": {"$ref": "#/$defs/timestamp"},
|
| 41 |
+
"source_artifact_sha256": {"$ref": "#/$defs/sha256"},
|
| 42 |
+
"selection_policy": {"enum": ["FROZEN_BEFORE_OUTCOME", "RETROSPECTIVE_EXPOSED_DEVELOPMENT"]},
|
| 43 |
+
"projection_policy": {"const": "PRE_PATCH_INPUTS_WITHOUT_ASSERTIONS_OR_EXPECTED_OUTPUTS"},
|
| 44 |
+
"program": {"type": "string", "minLength": 1},
|
| 45 |
+
"program_sha256": {"$ref": "#/$defs/sha256"},
|
| 46 |
+
"setup_assignments": {"type": "array", "items": {"$ref": "#/$defs/assignment"}},
|
| 47 |
+
"invocations": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/invocation"}}
|
| 48 |
+
}
|
| 49 |
+
},
|
| 50 |
+
"operating_conditions": {
|
| 51 |
+
"type": "object", "additionalProperties": false,
|
| 52 |
+
"required": ["python_version", "platform", "dependency_lock_sha256", "dependency_sources", "environment"],
|
| 53 |
+
"properties": {
|
| 54 |
+
"python_version": {"type": "string", "pattern": "^3\\.[0-9]+(?:\\.[0-9]+)?$"},
|
| 55 |
+
"platform": {"type": "string", "minLength": 1, "maxLength": 80},
|
| 56 |
+
"dependency_lock_sha256": {"$ref": "#/$defs/sha256"},
|
| 57 |
+
"dependency_sources": {"type": "array", "items": {"$ref": "#/$defs/sourceFile"}},
|
| 58 |
+
"environment": {"type": "object", "propertyNames": {"pattern": "^[A-Z][A-Z0-9_]*$"}, "additionalProperties": {"type": "string", "maxLength": 1024}}
|
| 59 |
+
}
|
| 60 |
+
},
|
| 61 |
+
"history": {"type": "array", "items": {"$ref": "#/$defs/historyEvent"}},
|
| 62 |
+
"target": {
|
| 63 |
+
"type": "object", "additionalProperties": false,
|
| 64 |
+
"required": ["kind", "branches", "horizon", "observation_classes", "observe"],
|
| 65 |
+
"properties": {
|
| 66 |
+
"kind": {"const": "PAIRED_INTERVENTION"},
|
| 67 |
+
"branches": {"const": ["NO_CHANGE", "PROPOSED_PATCH"]},
|
| 68 |
+
"horizon": {
|
| 69 |
+
"type": "object", "additionalProperties": false,
|
| 70 |
+
"required": ["kind", "max_seconds"],
|
| 71 |
+
"properties": {
|
| 72 |
+
"kind": {"const": "INVOCATION_COMPLETION"},
|
| 73 |
+
"max_seconds": {"type": "integer", "minimum": 1, "maximum": 3600}
|
| 74 |
+
}
|
| 75 |
+
},
|
| 76 |
+
"observation_classes": {"const": ["RETURN", "EXCEPTION", "TIMEOUT", "PROCESS_ERROR"]},
|
| 77 |
+
"observe": {"type": "array", "minItems": 1, "uniqueItems": true, "items": {"enum": ["observation_class", "return_value_sha256", "exception_type", "exit_code"]}, "contains": {"const": "observation_class"}}
|
| 78 |
+
}
|
| 79 |
+
},
|
| 80 |
+
"shared_input": {
|
| 81 |
+
"type": "object", "additionalProperties": false,
|
| 82 |
+
"required": ["input_sha256", "information_policy", "consumers"],
|
| 83 |
+
"properties": {
|
| 84 |
+
"input_sha256": {"$ref": "#/$defs/sha256"},
|
| 85 |
+
"information_policy": {"const": "IDENTICAL_REQUEST"},
|
| 86 |
+
"consumers": {"const": ["EXACT_EXECUTION", "STATIC_RULES", "RETRIEVAL", "LEARNED"]}
|
| 87 |
+
}
|
| 88 |
+
}
|
| 89 |
+
},
|
| 90 |
+
"$defs": {
|
| 91 |
+
"id": {"type": "string", "pattern": "^[a-z][a-z0-9._-]{2,127}$"},
|
| 92 |
+
"sha256": {"type": "string", "pattern": "^[0-9a-f]{64}$"},
|
| 93 |
+
"timestamp": {"type": "string", "format": "date-time"},
|
| 94 |
+
"path": {"type": "string", "pattern": "^(?!/)(?!.*(?:^|/)\\.\\.(?:/|$))[A-Za-z0-9._/-]+$"},
|
| 95 |
+
"sourceFile": {
|
| 96 |
+
"type": "object", "additionalProperties": false,
|
| 97 |
+
"required": ["path", "sha256", "content"],
|
| 98 |
+
"properties": {"path": {"$ref": "#/$defs/path"}, "sha256": {"$ref": "#/$defs/sha256"}, "content": {"type": "string"}}
|
| 99 |
+
},
|
| 100 |
+
"invocation": {
|
| 101 |
+
"type": "object", "additionalProperties": false,
|
| 102 |
+
"required": ["call_kind", "entrypoint", "args", "kwargs"],
|
| 103 |
+
"properties": {
|
| 104 |
+
"call_kind": {"enum": ["CONSTRUCTOR", "METHOD", "FUNCTION"]},
|
| 105 |
+
"entrypoint": {"type": "string", "pattern": "^[A-Za-z_][A-Za-z0-9_.:]*$"},
|
| 106 |
+
"receiver_ref": {"$ref": "#/$defs/id"},
|
| 107 |
+
"bind_result_as": {"$ref": "#/$defs/id"},
|
| 108 |
+
"args": {"type": "array", "items": {"$ref": "#/$defs/jsonValue"}},
|
| 109 |
+
"kwargs": {"type": "object", "propertyNames": {"pattern": "^[A-Za-z_][A-Za-z0-9_]*$"}, "additionalProperties": {"$ref": "#/$defs/jsonValue"}}
|
| 110 |
+
}
|
| 111 |
+
},
|
| 112 |
+
"assignment": {
|
| 113 |
+
"type": "object", "additionalProperties": false,
|
| 114 |
+
"required": ["target", "value"],
|
| 115 |
+
"properties": {
|
| 116 |
+
"target": {"type": "string", "pattern": "^[A-Za-z_][A-Za-z0-9_.]*$"},
|
| 117 |
+
"value": {"$ref": "#/$defs/jsonValue"}
|
| 118 |
+
}
|
| 119 |
+
},
|
| 120 |
+
"historyEvent": {
|
| 121 |
+
"type": "object", "additionalProperties": false,
|
| 122 |
+
"required": ["event_id", "observed_at", "source_snapshot_sha256", "origin", "invocation", "observation_class"],
|
| 123 |
+
"properties": {
|
| 124 |
+
"event_id": {"$ref": "#/$defs/id"},
|
| 125 |
+
"observed_at": {"$ref": "#/$defs/timestamp"},
|
| 126 |
+
"source_snapshot_sha256": {"$ref": "#/$defs/sha256"},
|
| 127 |
+
"origin": {"enum": ["PRE_PATCH_WORKLOAD_TRACE", "PRE_PATCH_BASE_EXECUTION"]},
|
| 128 |
+
"invocation": {"$ref": "#/$defs/invocation"},
|
| 129 |
+
"observation_class": {"enum": ["RETURN", "EXCEPTION", "TIMEOUT", "PROCESS_ERROR"]},
|
| 130 |
+
"return_value_sha256": {"$ref": "#/$defs/sha256"},
|
| 131 |
+
"exception_type": {"type": "string", "pattern": "^[A-Za-z_][A-Za-z0-9_.]*$"},
|
| 132 |
+
"exit_code": {"type": "integer", "minimum": 0, "maximum": 255}
|
| 133 |
+
}
|
| 134 |
+
},
|
| 135 |
+
"jsonValue": {"anyOf": [{"type": "null"}, {"type": "boolean"}, {"type": "number"}, {"type": "string"}, {"type": "array", "items": {"$ref": "#/$defs/jsonValue"}}, {"type": "object", "additionalProperties": {"$ref": "#/$defs/jsonValue"}}]}
|
| 136 |
+
}
|
| 137 |
+
}
|
evaluation/real-patch/source/code_review/contracts/real_patch_prediction_v1/validator.py
ADDED
|
@@ -0,0 +1,179 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Fail-closed validator for the model-visible real-patch forecast request.
|
| 2 |
+
|
| 3 |
+
This module is deliberately loadable by path: the contract is not a runtime
|
| 4 |
+
package and callers need not install laxmi-review to validate an input packet.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
import hashlib
|
| 10 |
+
import ast
|
| 11 |
+
import json
|
| 12 |
+
import re
|
| 13 |
+
from datetime import datetime, timezone
|
| 14 |
+
from pathlib import Path
|
| 15 |
+
from typing import Any
|
| 16 |
+
|
| 17 |
+
import jsonschema
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
SCHEMA_PATH = Path(__file__).with_name("forecast_request.schema.json")
|
| 21 |
+
_DIFF_HEADER = re.compile(r"^diff --git a/(\S+) b/(\S+)$", re.MULTILINE)
|
| 22 |
+
_TEST_PATH = re.compile(r"(?:^|/)(?:tests?|testing|fixtures?)(?:/|$)|(?:^|/)test_[^/]*\.py$|(?:^|/)[^/]*_test\.py$")
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
class ForecastContractError(ValueError):
|
| 26 |
+
"""The request is not safe to expose to a forecast consumer."""
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def _canonical_bytes(value: Any) -> bytes:
|
| 30 |
+
return json.dumps(
|
| 31 |
+
value, sort_keys=True, separators=(",", ":"), ensure_ascii=False, allow_nan=False
|
| 32 |
+
).encode("utf-8")
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def input_sha256(request: dict[str, Any]) -> str:
|
| 36 |
+
"""Hash the complete consumer-visible request, excluding its own hash envelope."""
|
| 37 |
+
return hashlib.sha256(
|
| 38 |
+
_canonical_bytes({key: value for key, value in request.items() if key != "shared_input"})
|
| 39 |
+
).hexdigest()
|
| 40 |
+
|
| 41 |
+
|
| 42 |
+
def source_snapshot_sha256(files: list[dict[str, Any]]) -> str:
|
| 43 |
+
"""Content identity of the visible source slice, without public repository IDs."""
|
| 44 |
+
manifest = sorted(
|
| 45 |
+
({"path": item["path"], "sha256": item["sha256"]} for item in files),
|
| 46 |
+
key=lambda item: item["path"],
|
| 47 |
+
)
|
| 48 |
+
return hashlib.sha256(_canonical_bytes(manifest)).hexdigest()
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def _instant(raw: str) -> datetime:
|
| 52 |
+
try:
|
| 53 |
+
parsed = datetime.fromisoformat(raw.replace("Z", "+00:00"))
|
| 54 |
+
except ValueError as exc:
|
| 55 |
+
raise ForecastContractError(f"invalid timestamp: {raw!r}") from exc
|
| 56 |
+
if parsed.tzinfo is None:
|
| 57 |
+
raise ForecastContractError("timestamps must include a timezone")
|
| 58 |
+
return parsed.astimezone(timezone.utc)
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
def validate_forecast_request(request: dict[str, Any]) -> None:
|
| 62 |
+
"""Validate shape, hashes, chronology, patch scope, and shared-input identity.
|
| 63 |
+
|
| 64 |
+
Source authenticity and semantic answer leakage still require custody checks:
|
| 65 |
+
a self-declared timestamp or hash cannot prove an external artifact existed.
|
| 66 |
+
"""
|
| 67 |
+
schema = json.loads(SCHEMA_PATH.read_text(encoding="utf-8"))
|
| 68 |
+
try:
|
| 69 |
+
jsonschema.Draft202012Validator.check_schema(schema)
|
| 70 |
+
jsonschema.Draft202012Validator(
|
| 71 |
+
schema, format_checker=jsonschema.FormatChecker()
|
| 72 |
+
).validate(request)
|
| 73 |
+
except (jsonschema.SchemaError, jsonschema.ValidationError) as exc:
|
| 74 |
+
raise ForecastContractError(f"request schema violation: {exc.message}") from exc
|
| 75 |
+
|
| 76 |
+
source = request["pre_source"]
|
| 77 |
+
patch = request["proposed_patch"]
|
| 78 |
+
scenario = request["scenario"]
|
| 79 |
+
cutoff = _instant(request["forecast_cutoff_at"])
|
| 80 |
+
proposed = _instant(patch["proposed_at"])
|
| 81 |
+
if source["snapshot_sha256"] != patch["base_snapshot_sha256"]:
|
| 82 |
+
raise ForecastContractError("patch base snapshot must match pre_source snapshot")
|
| 83 |
+
if _instant(source["snapshot_at"]) > proposed or proposed > cutoff:
|
| 84 |
+
raise ForecastContractError("source/proposal/forecast chronology is invalid")
|
| 85 |
+
if _instant(scenario["available_at"]) >= proposed:
|
| 86 |
+
raise ForecastContractError("scenario must be available before the proposed patch")
|
| 87 |
+
if scenario["origin"] == "PROSPECTIVE_GENERATOR" and scenario["selection_policy"] != "FROZEN_BEFORE_OUTCOME":
|
| 88 |
+
raise ForecastContractError("generator selection must be frozen")
|
| 89 |
+
if request["evaluation_split"] == "ACCEPTANCE_CANDIDATE" and scenario["selection_policy"] != "FROZEN_BEFORE_OUTCOME":
|
| 90 |
+
raise ForecastContractError("retrospective selection cannot be acceptance candidate")
|
| 91 |
+
program = scenario["program"]
|
| 92 |
+
if hashlib.sha256(program.encode("utf-8")).hexdigest() != scenario["program_sha256"]:
|
| 93 |
+
raise ForecastContractError("scenario program hash mismatch")
|
| 94 |
+
try:
|
| 95 |
+
syntax = ast.parse(program)
|
| 96 |
+
except SyntaxError as exc:
|
| 97 |
+
raise ForecastContractError("scenario program is not valid Python") from exc
|
| 98 |
+
for node in ast.walk(syntax):
|
| 99 |
+
if isinstance(node, ast.Assert):
|
| 100 |
+
raise ForecastContractError("scenario program contains an assertion")
|
| 101 |
+
if isinstance(node, ast.Name) and node.id.lower().startswith("expected"):
|
| 102 |
+
raise ForecastContractError("scenario program contains an expected-value variable")
|
| 103 |
+
if isinstance(node, ast.Raise) and isinstance(node.exc, ast.Call) and isinstance(node.exc.func, ast.Name) and node.exc.func.id == "AssertionError":
|
| 104 |
+
raise ForecastContractError("scenario program raises an assertion error")
|
| 105 |
+
|
| 106 |
+
paths: set[str] = set()
|
| 107 |
+
for item in source["files"]:
|
| 108 |
+
if item["path"] in paths:
|
| 109 |
+
raise ForecastContractError("duplicate source path")
|
| 110 |
+
paths.add(item["path"])
|
| 111 |
+
if hashlib.sha256(item["content"].encode("utf-8")).hexdigest() != item["sha256"]:
|
| 112 |
+
raise ForecastContractError(f"source hash mismatch: {item['path']}")
|
| 113 |
+
if source_snapshot_sha256(source["files"]) != source["snapshot_sha256"]:
|
| 114 |
+
raise ForecastContractError("source snapshot hash mismatch")
|
| 115 |
+
dependencies = request["operating_conditions"]["dependency_sources"]
|
| 116 |
+
dependency_paths: set[str] = set()
|
| 117 |
+
for item in dependencies:
|
| 118 |
+
if item["path"] in dependency_paths:
|
| 119 |
+
raise ForecastContractError("duplicate dependency path")
|
| 120 |
+
dependency_paths.add(item["path"])
|
| 121 |
+
if hashlib.sha256(item["content"].encode("utf-8")).hexdigest() != item["sha256"]:
|
| 122 |
+
raise ForecastContractError(f"dependency hash mismatch: {item['path']}")
|
| 123 |
+
if source_snapshot_sha256(dependencies) != request["operating_conditions"]["dependency_lock_sha256"]:
|
| 124 |
+
raise ForecastContractError("dependency lock hash mismatch")
|
| 125 |
+
diff = patch["unified_diff"]
|
| 126 |
+
if hashlib.sha256(diff.encode("utf-8")).hexdigest() != patch["diff_sha256"]:
|
| 127 |
+
raise ForecastContractError("proposed diff hash mismatch")
|
| 128 |
+
changed_paths = _DIFF_HEADER.findall(diff)
|
| 129 |
+
if not diff.startswith("diff --git ") or not changed_paths:
|
| 130 |
+
raise ForecastContractError("proposed diff needs git-style file headers")
|
| 131 |
+
if len(changed_paths) != sum(line.startswith("diff --git ") for line in diff.splitlines()):
|
| 132 |
+
raise ForecastContractError("proposed diff has an unrecognized file header")
|
| 133 |
+
for before, after in changed_paths:
|
| 134 |
+
if before != after:
|
| 135 |
+
raise ForecastContractError("renames need a separately reviewed contract")
|
| 136 |
+
if _TEST_PATH.search(before):
|
| 137 |
+
raise ForecastContractError("proposed diff contains a test or fixture path")
|
| 138 |
+
if before.startswith("/") or ".." in Path(before).parts:
|
| 139 |
+
raise ForecastContractError("proposed diff path escapes repository")
|
| 140 |
+
if before not in paths:
|
| 141 |
+
raise ForecastContractError("proposed diff file absent from visible pre source")
|
| 142 |
+
|
| 143 |
+
ids: set[str] = set()
|
| 144 |
+
for event in request["history"]:
|
| 145 |
+
if event["event_id"] in ids:
|
| 146 |
+
raise ForecastContractError("duplicate history event_id")
|
| 147 |
+
ids.add(event["event_id"])
|
| 148 |
+
if event["source_snapshot_sha256"] != source["snapshot_sha256"]:
|
| 149 |
+
raise ForecastContractError("history event is not from the pinned pre source")
|
| 150 |
+
if _instant(event["observed_at"]) >= proposed:
|
| 151 |
+
raise ForecastContractError("history event must predate patch proposal")
|
| 152 |
+
|
| 153 |
+
if request["shared_input"]["input_sha256"] != input_sha256(request):
|
| 154 |
+
raise ForecastContractError("shared input hash mismatch")
|
| 155 |
+
|
| 156 |
+
|
| 157 |
+
def load_forecast_request(path: Path) -> dict[str, Any]:
|
| 158 |
+
"""Load strict JSON and validate before passing it to any consumer."""
|
| 159 |
+
def unique_pairs(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
|
| 160 |
+
value: dict[str, Any] = {}
|
| 161 |
+
for key, item in pairs:
|
| 162 |
+
if key in value:
|
| 163 |
+
raise ForecastContractError(f"duplicate JSON key: {key}")
|
| 164 |
+
value[key] = item
|
| 165 |
+
return value
|
| 166 |
+
|
| 167 |
+
def reject_constant(value: str) -> None:
|
| 168 |
+
raise ForecastContractError(f"non-finite JSON number: {value}")
|
| 169 |
+
|
| 170 |
+
try:
|
| 171 |
+
request = json.loads(
|
| 172 |
+
path.read_text(encoding="utf-8"),
|
| 173 |
+
object_pairs_hook=unique_pairs,
|
| 174 |
+
parse_constant=reject_constant,
|
| 175 |
+
)
|
| 176 |
+
except json.JSONDecodeError as exc:
|
| 177 |
+
raise ForecastContractError(f"invalid JSON: {exc.msg}") from exc
|
| 178 |
+
validate_forecast_request(request)
|
| 179 |
+
return request
|
evaluation/real-patch/source/code_review/packaging/real_patch/api.py
ADDED
|
@@ -0,0 +1,120 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Local, development-only real-patch validation and forecast interface.
|
| 2 |
+
|
| 3 |
+
The forecast path reads only the model-visible request. Oracle scoring is a
|
| 4 |
+
separate explicit operation, never part of forecast().
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
from __future__ import annotations
|
| 8 |
+
|
| 9 |
+
import argparse
|
| 10 |
+
import hashlib
|
| 11 |
+
import importlib.util
|
| 12 |
+
import json
|
| 13 |
+
from pathlib import Path
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
ROOT = Path(__file__).resolve().parent / "capsule/code_review"
|
| 17 |
+
CONTRACT = ROOT / "contracts/real_patch_prediction_v1/validator.py"
|
| 18 |
+
EXACT = ROOT / "scripts/run_real_patch_visible_exact_v1.py"
|
| 19 |
+
GATE = ROOT / "scripts/run_real_patch_baseline_gate_v2.py"
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def _module(path: Path, name: str):
|
| 23 |
+
spec = importlib.util.spec_from_file_location(name, path)
|
| 24 |
+
if spec is None or spec.loader is None:
|
| 25 |
+
raise RuntimeError(f"packaged module cannot be loaded: {name}")
|
| 26 |
+
module = importlib.util.module_from_spec(spec)
|
| 27 |
+
spec.loader.exec_module(module)
|
| 28 |
+
return module
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
def _sha(path: Path) -> str:
|
| 32 |
+
return hashlib.sha256(path.read_bytes()).hexdigest()
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def validate(request_path: Path) -> dict:
|
| 36 |
+
contract = _module(CONTRACT, "laxmi_real_patch_contract")
|
| 37 |
+
request = contract.load_forecast_request(request_path)
|
| 38 |
+
return {
|
| 39 |
+
"schema_version": "laxmi.real-patch-local-api.v1",
|
| 40 |
+
"status": "ADMITTED",
|
| 41 |
+
"case_id": request["case_id"],
|
| 42 |
+
"input_sha256": contract.input_sha256(request),
|
| 43 |
+
"request_file_sha256": _sha(request_path),
|
| 44 |
+
"evaluation_split": request["evaluation_split"],
|
| 45 |
+
"custody_verified": False,
|
| 46 |
+
"acceptance_eligible": False,
|
| 47 |
+
"executed": False,
|
| 48 |
+
}
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def forecast(request_path: Path, output_dir: Path) -> dict:
|
| 52 |
+
admitted = validate(request_path)
|
| 53 |
+
exact = _module(EXACT, "laxmi_real_patch_visible_exact")
|
| 54 |
+
gate = _module(GATE, "laxmi_real_patch_gate")
|
| 55 |
+
output_dir.mkdir(mode=0o700, parents=True, exist_ok=False)
|
| 56 |
+
prediction, execution = exact.run(request_path, output_dir / "prediction")
|
| 57 |
+
sealed = gate.seal(request_path, output_dir / "prediction/predictions.json", output_dir / "seal")
|
| 58 |
+
if prediction["input_sha256"] != admitted["input_sha256"] or sealed["input_sha256"] != admitted["input_sha256"]:
|
| 59 |
+
raise ValueError("prediction changed the admitted input identity")
|
| 60 |
+
row = prediction["comparators"]["EXACT_VISIBLE_SOURCE"]
|
| 61 |
+
return {
|
| 62 |
+
"schema_version": "laxmi.real-patch-local-api.v1",
|
| 63 |
+
"status": "PREDICTED" if row["status"] == "predicted" else "UNSUPPORTED",
|
| 64 |
+
"engine_id": "EXACT_VISIBLE_SOURCE",
|
| 65 |
+
"information_tier": "SAME_INFORMATION",
|
| 66 |
+
"case_id": admitted["case_id"],
|
| 67 |
+
"input_sha256": admitted["input_sha256"],
|
| 68 |
+
"method_sha256": row.get("method_sha256"),
|
| 69 |
+
"reason": row.get("reason"),
|
| 70 |
+
"prediction_file_sha256": _sha(output_dir / "prediction/predictions.json"),
|
| 71 |
+
"seal_file_sha256": _sha(output_dir / "seal/seal.json"),
|
| 72 |
+
"oracle_read": execution["oracle_read"],
|
| 73 |
+
"custody_verified": False,
|
| 74 |
+
"acceptance_eligible": False,
|
| 75 |
+
"development_only": True,
|
| 76 |
+
}
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def score(request_path: Path, oracle_path: Path, output_dir: Path) -> dict:
|
| 80 |
+
"""Privileged local evaluation call; never invoked by forecast()."""
|
| 81 |
+
gate = _module(GATE, "laxmi_real_patch_gate")
|
| 82 |
+
receipt = gate.score(request_path, oracle_path, output_dir / "seal")
|
| 83 |
+
return {
|
| 84 |
+
"schema_version": "laxmi.real-patch-local-api.v1",
|
| 85 |
+
"status": "SCORED_DEVELOPMENT_ORACLE",
|
| 86 |
+
"case_id": receipt["case_id"],
|
| 87 |
+
"input_sha256": receipt["input_sha256"],
|
| 88 |
+
"receipt_file_sha256": _sha(output_dir / "seal/receipt.json"),
|
| 89 |
+
"same_information_scored_count": receipt["same_information_scored_count"],
|
| 90 |
+
"development_only": True,
|
| 91 |
+
}
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
def main() -> None:
|
| 95 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 96 |
+
commands = parser.add_subparsers(dest="command", required=True)
|
| 97 |
+
validation = commands.add_parser("validate")
|
| 98 |
+
validation.add_argument("--request", type=Path, required=True)
|
| 99 |
+
prediction = commands.add_parser("forecast")
|
| 100 |
+
prediction.add_argument("--request", type=Path, required=True)
|
| 101 |
+
prediction.add_argument("--output-dir", type=Path, required=True)
|
| 102 |
+
scoring = commands.add_parser("score")
|
| 103 |
+
scoring.add_argument("--request", type=Path, required=True)
|
| 104 |
+
scoring.add_argument("--oracle", type=Path, required=True)
|
| 105 |
+
scoring.add_argument("--output-dir", type=Path, required=True)
|
| 106 |
+
args = parser.parse_args()
|
| 107 |
+
try:
|
| 108 |
+
if args.command == "validate":
|
| 109 |
+
result = validate(args.request)
|
| 110 |
+
elif args.command == "forecast":
|
| 111 |
+
result = forecast(args.request, args.output_dir)
|
| 112 |
+
else:
|
| 113 |
+
result = score(args.request, args.oracle, args.output_dir)
|
| 114 |
+
except (OSError, ValueError, RuntimeError) as exc:
|
| 115 |
+
parser.exit(2, f"Laxmi real-patch refused: {exc}\n")
|
| 116 |
+
print(json.dumps(result, sort_keys=True, allow_nan=False))
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
if __name__ == "__main__":
|
| 120 |
+
main()
|
evaluation/real-patch/source/code_review/scripts/build_real_patch_local_package_v1.py
ADDED
|
@@ -0,0 +1,101 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Build a source-hash-preserving local wheel for real-patch development use."""
|
| 2 |
+
|
| 3 |
+
import argparse
|
| 4 |
+
import base64
|
| 5 |
+
import csv
|
| 6 |
+
import hashlib
|
| 7 |
+
import io
|
| 8 |
+
import json
|
| 9 |
+
from pathlib import Path
|
| 10 |
+
import zipfile
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
REPO = Path(__file__).resolve().parents[2]
|
| 14 |
+
VERSION = "0.1.0.dev1"
|
| 15 |
+
DIST = "laxmi_real_patch"
|
| 16 |
+
SOURCES = {
|
| 17 |
+
"api.py": REPO / "code_review/packaging/real_patch/api.py",
|
| 18 |
+
"LICENSE": REPO / "LICENSE",
|
| 19 |
+
"NOTICE": REPO / "NOTICE",
|
| 20 |
+
"LICENSE_SCOPE.md": REPO / "LICENSE_SCOPE.md",
|
| 21 |
+
"capsule/code_review/contracts/real_patch_prediction_v1/validator.py": REPO / "code_review/contracts/real_patch_prediction_v1/validator.py",
|
| 22 |
+
"capsule/code_review/contracts/real_patch_prediction_v1/forecast_request.schema.json": REPO / "code_review/contracts/real_patch_prediction_v1/forecast_request.schema.json",
|
| 23 |
+
"capsule/code_review/scripts/run_real_patch_visible_exact_v1.py": REPO / "code_review/scripts/run_real_patch_visible_exact_v1.py",
|
| 24 |
+
"capsule/code_review/scripts/run_real_patch_baseline_gate_v2.py": REPO / "code_review/scripts/run_real_patch_baseline_gate_v2.py",
|
| 25 |
+
}
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
def sha(raw: bytes) -> str:
|
| 29 |
+
return hashlib.sha256(raw).hexdigest()
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
def build(output: Path) -> dict:
|
| 33 |
+
if output.exists():
|
| 34 |
+
raise FileExistsError(output)
|
| 35 |
+
payload = {name: path.read_bytes() for name, path in SOURCES.items()}
|
| 36 |
+
if any(path.is_symlink() or not path.is_file() or len(payload[name]) > 1024 * 1024 for name, path in SOURCES.items()):
|
| 37 |
+
raise ValueError("package needs bounded regular source files")
|
| 38 |
+
payload["__init__.py"] = b"from .api import validate, forecast, score\n"
|
| 39 |
+
manifest = {
|
| 40 |
+
"schema_version": "laxmi.real-patch-local-package.v1",
|
| 41 |
+
"version": VERSION,
|
| 42 |
+
"development_only": True,
|
| 43 |
+
"learned_model_included": False,
|
| 44 |
+
"files": {name: sha(raw) for name, raw in sorted(payload.items())},
|
| 45 |
+
}
|
| 46 |
+
payload["manifest.json"] = (json.dumps(manifest, sort_keys=True, indent=2) + "\n").encode()
|
| 47 |
+
entries = {f"{DIST}/{name}": raw for name, raw in payload.items()}
|
| 48 |
+
info = f"{DIST}-{VERSION}.dist-info/"
|
| 49 |
+
entries[info + "METADATA"] = (
|
| 50 |
+
"Metadata-Version: 2.1\n"
|
| 51 |
+
"Name: laxmi-real-patch\n"
|
| 52 |
+
f"Version: {VERSION}\n"
|
| 53 |
+
"Summary: Development-only real-patch forecast and evaluation interface\n"
|
| 54 |
+
"Requires-Python: >=3.12,<3.13\n"
|
| 55 |
+
"Requires-Dist: jsonschema>=4.23,<5\n"
|
| 56 |
+
"License: Apache-2.0\n"
|
| 57 |
+
"Author: Navneet Prabhakar\n\n"
|
| 58 |
+
).encode()
|
| 59 |
+
entries[info + "WHEEL"] = b"Wheel-Version: 1.0\nGenerator: laxmi-local-package\nRoot-Is-Purelib: true\nTag: py3-none-any\n"
|
| 60 |
+
entries[info + "entry_points.txt"] = b"[console_scripts]\nlaxmi-real-patch = laxmi_real_patch.api:main\n"
|
| 61 |
+
record = io.StringIO(newline="")
|
| 62 |
+
writer = csv.writer(record, lineterminator="\n")
|
| 63 |
+
for name, raw in sorted(entries.items()):
|
| 64 |
+
encoded = base64.urlsafe_b64encode(hashlib.sha256(raw).digest()).decode().rstrip("=")
|
| 65 |
+
writer.writerow([name, f"sha256={encoded}", len(raw)])
|
| 66 |
+
writer.writerow([info + "RECORD", "", ""])
|
| 67 |
+
entries[info + "RECORD"] = record.getvalue().encode()
|
| 68 |
+
output.mkdir(parents=True, exist_ok=False)
|
| 69 |
+
wheel = output / f"{DIST}-{VERSION}-py3-none-any.whl"
|
| 70 |
+
with zipfile.ZipFile(wheel, "x", compression=zipfile.ZIP_DEFLATED) as archive:
|
| 71 |
+
for name, raw in sorted(entries.items()):
|
| 72 |
+
member = zipfile.ZipInfo(name, (1980, 1, 1, 0, 0, 0))
|
| 73 |
+
member.compress_type = zipfile.ZIP_DEFLATED
|
| 74 |
+
member.external_attr = 0o100644 << 16
|
| 75 |
+
archive.writestr(member, raw)
|
| 76 |
+
receipt = {
|
| 77 |
+
"schema_version": "laxmi.real-patch-local-build.v1",
|
| 78 |
+
"version": VERSION,
|
| 79 |
+
"wheel": wheel.name,
|
| 80 |
+
"wheel_sha256": sha(wheel.read_bytes()),
|
| 81 |
+
"wheel_bytes": wheel.stat().st_size,
|
| 82 |
+
"payload_files": manifest["files"],
|
| 83 |
+
"builder_sha256": sha(Path(__file__).read_bytes()),
|
| 84 |
+
"linux_validation_pending": True,
|
| 85 |
+
"linux_exact_execution_pending": True,
|
| 86 |
+
"public_release": False,
|
| 87 |
+
}
|
| 88 |
+
(output / "build.json").write_text(json.dumps(receipt, sort_keys=True, indent=2) + "\n", encoding="utf-8")
|
| 89 |
+
return receipt
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
def main() -> None:
|
| 93 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 94 |
+
parser.add_argument("--output-dir", type=Path, required=True)
|
| 95 |
+
args = parser.parse_args()
|
| 96 |
+
result = build(args.output_dir)
|
| 97 |
+
print(json.dumps({key: result[key] for key in ("wheel", "wheel_sha256", "wheel_bytes")}, sort_keys=True))
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
if __name__ == "__main__":
|
| 101 |
+
main()
|
evaluation/real-patch/source/code_review/scripts/run_real_patch_baseline_gate_v2.py
ADDED
|
@@ -0,0 +1,264 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Seal same-request baseline forecasts before scoring a development oracle.
|
| 2 |
+
|
| 3 |
+
This is a protocol runner, not an implementation of any comparator. Unsupported
|
| 4 |
+
comparators stay in the denominator and no source or candidate code is executed.
|
| 5 |
+
"""
|
| 6 |
+
|
| 7 |
+
import argparse
|
| 8 |
+
import hashlib
|
| 9 |
+
import importlib.util
|
| 10 |
+
import json
|
| 11 |
+
from pathlib import Path
|
| 12 |
+
|
| 13 |
+
|
| 14 |
+
SCHEMA_VERSION = "laxmi.real-patch-baseline-gate.v2"
|
| 15 |
+
PREDICTION_VERSION = "laxmi.real-patch-baseline-predictions.v2"
|
| 16 |
+
ORACLE_VERSION = "laxmi.real-patch-observation-oracle.v1"
|
| 17 |
+
COMPARATORS = ("EXACT_EXECUTION", "EXACT_VISIBLE_SOURCE", "STATIC_RULES", "RETRIEVAL")
|
| 18 |
+
INFORMATION_TIERS = ("SAME_INFORMATION", "PRIVILEGED_CEILING")
|
| 19 |
+
CONTRACT_PATH = (
|
| 20 |
+
Path(__file__).resolve().parents[1]
|
| 21 |
+
/ "contracts/real_patch_prediction_v1/validator.py"
|
| 22 |
+
)
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def canonical_bytes(value):
|
| 26 |
+
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def sha256(data):
|
| 30 |
+
return hashlib.sha256(data).hexdigest()
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def read_json(path):
|
| 34 |
+
def unique_pairs(pairs):
|
| 35 |
+
value = {}
|
| 36 |
+
for key, item in pairs:
|
| 37 |
+
if key in value:
|
| 38 |
+
raise ValueError(f"duplicate JSON key: {key}")
|
| 39 |
+
value[key] = item
|
| 40 |
+
return value
|
| 41 |
+
|
| 42 |
+
def reject_constant(value):
|
| 43 |
+
raise ValueError(f"non-finite JSON number: {value}")
|
| 44 |
+
|
| 45 |
+
with path.open("rb") as handle:
|
| 46 |
+
data = handle.read()
|
| 47 |
+
return json.loads(data, object_pairs_hook=unique_pairs, parse_constant=reject_constant), sha256(data)
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def write_create_only(path, value):
|
| 51 |
+
encoded = json.dumps(value, sort_keys=True, indent=2, ensure_ascii=False).encode() + b"\n"
|
| 52 |
+
with path.open("xb") as handle:
|
| 53 |
+
handle.write(encoded)
|
| 54 |
+
return sha256(encoded)
|
| 55 |
+
|
| 56 |
+
|
| 57 |
+
def load_contract():
|
| 58 |
+
if not CONTRACT_PATH.is_file():
|
| 59 |
+
raise FileNotFoundError(f"forecast request validator missing: {CONTRACT_PATH}")
|
| 60 |
+
spec = importlib.util.spec_from_file_location("real_patch_forecast_validator", CONTRACT_PATH)
|
| 61 |
+
module = importlib.util.module_from_spec(spec)
|
| 62 |
+
spec.loader.exec_module(module)
|
| 63 |
+
return module
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
def validate_observation(observation, observe, classes):
|
| 67 |
+
if not isinstance(observation, dict) or set(observation) != set(observe):
|
| 68 |
+
raise ValueError("observation fields must exactly match request target.observe")
|
| 69 |
+
if observation["observation_class"] not in classes:
|
| 70 |
+
raise ValueError("observation_class outside frozen request vocabulary")
|
| 71 |
+
for field, value in observation.items():
|
| 72 |
+
if field == "return_value_sha256" and value is not None:
|
| 73 |
+
if not isinstance(value, str) or len(value) != 64 or any(
|
| 74 |
+
char not in "0123456789abcdef" for char in value
|
| 75 |
+
):
|
| 76 |
+
raise ValueError("return_value_sha256 must be a lowercase SHA-256 or null")
|
| 77 |
+
elif field == "exception_type" and value is not None and not isinstance(value, str):
|
| 78 |
+
raise ValueError("exception_type must be a string or null")
|
| 79 |
+
elif field == "exit_code" and value is not None and (type(value) is not int):
|
| 80 |
+
raise ValueError("exit_code must be an integer or null")
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def validate_observations(observations, target):
|
| 84 |
+
branches = target["branches"]
|
| 85 |
+
if not isinstance(observations, dict) or set(observations) != set(branches):
|
| 86 |
+
raise ValueError("observations must cover both frozen intervention branches")
|
| 87 |
+
for branch in branches:
|
| 88 |
+
validate_observation(
|
| 89 |
+
observations[branch], target["observe"], target["observation_classes"]
|
| 90 |
+
)
|
| 91 |
+
|
| 92 |
+
|
| 93 |
+
def validate_predictions(predictions, request, request_hash):
|
| 94 |
+
if predictions.get("schema_version") != PREDICTION_VERSION:
|
| 95 |
+
raise ValueError("wrong prediction schema_version")
|
| 96 |
+
if predictions.get("case_id") != request["case_id"]:
|
| 97 |
+
raise ValueError("prediction case_id differs from request")
|
| 98 |
+
if predictions.get("input_sha256") != request_hash:
|
| 99 |
+
raise ValueError("prediction input_sha256 differs from frozen shared input")
|
| 100 |
+
rows = predictions.get("comparators")
|
| 101 |
+
if not isinstance(rows, dict) or set(rows) != set(COMPARATORS):
|
| 102 |
+
raise ValueError("exactly four named comparator records are required")
|
| 103 |
+
for name, row in rows.items():
|
| 104 |
+
if not isinstance(row, dict) or row.get("status") not in {"predicted", "unsupported"}:
|
| 105 |
+
raise ValueError(f"{name} requires predicted or unsupported status")
|
| 106 |
+
if row.get("information_tier") not in INFORMATION_TIERS:
|
| 107 |
+
raise ValueError(f"{name} requires an explicit information_tier")
|
| 108 |
+
if name != "EXACT_EXECUTION" and row["information_tier"] != "SAME_INFORMATION":
|
| 109 |
+
raise ValueError(f"{name} must use same information")
|
| 110 |
+
if row["status"] == "unsupported":
|
| 111 |
+
if set(row) != {"status", "reason", "information_tier"} or not isinstance(row["reason"], str) or not row["reason"].strip():
|
| 112 |
+
raise ValueError(f"{name} unsupported requires only a nonempty reason")
|
| 113 |
+
else:
|
| 114 |
+
if set(row) != {"status", "observations", "method_sha256", "information_tier"}:
|
| 115 |
+
raise ValueError(f"{name} prediction requires paired observations and method SHA")
|
| 116 |
+
method = row["method_sha256"]
|
| 117 |
+
if not isinstance(method, str) or len(method) != 64 or any(
|
| 118 |
+
char not in "0123456789abcdef" for char in method
|
| 119 |
+
):
|
| 120 |
+
raise ValueError(f"{name} method_sha256 must be a lowercase SHA-256")
|
| 121 |
+
validate_observations(row["observations"], request["target"])
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
def seal(envelope_path, predictions_path, output_dir):
|
| 125 |
+
contract = load_contract()
|
| 126 |
+
request, request_file_sha = read_json(envelope_path)
|
| 127 |
+
contract.validate_forecast_request(request)
|
| 128 |
+
request_hash = contract.input_sha256(request)
|
| 129 |
+
predictions, predictions_file_sha = read_json(predictions_path)
|
| 130 |
+
validate_predictions(predictions, request, request_hash)
|
| 131 |
+
output_dir.mkdir(mode=0o700, parents=True, exist_ok=False)
|
| 132 |
+
record = {
|
| 133 |
+
"schema_version": SCHEMA_VERSION,
|
| 134 |
+
"phase": "sealed_predictions",
|
| 135 |
+
"development_only": True,
|
| 136 |
+
"case_id": request["case_id"],
|
| 137 |
+
"input_sha256": request_hash,
|
| 138 |
+
"envelope_file_sha256": request_file_sha,
|
| 139 |
+
"prediction_file_sha256": predictions_file_sha,
|
| 140 |
+
"predictions": predictions,
|
| 141 |
+
"oracle_not_read_by_seal": True,
|
| 142 |
+
"baseline_methods_not_executed_by_runner": True,
|
| 143 |
+
}
|
| 144 |
+
seal_digest = write_create_only(output_dir / "seal.json", record)
|
| 145 |
+
write_create_only(output_dir / "seal.sha256.json", {"seal_sha256": seal_digest})
|
| 146 |
+
return record
|
| 147 |
+
|
| 148 |
+
|
| 149 |
+
def score(envelope_path, oracle_path, output_dir):
|
| 150 |
+
if (output_dir / "invalidated.json").exists():
|
| 151 |
+
raise ValueError("sealed prediction was invalidated; do not score")
|
| 152 |
+
seal_path = output_dir / "seal.json"
|
| 153 |
+
frozen, seal_file_sha = read_json(seal_path)
|
| 154 |
+
seal_digest_record, _ = read_json(output_dir / "seal.sha256.json")
|
| 155 |
+
if seal_digest_record != {"seal_sha256": seal_file_sha}:
|
| 156 |
+
raise ValueError("seal bytes differ from create-only digest record")
|
| 157 |
+
if frozen.get("schema_version") != SCHEMA_VERSION or frozen.get("phase") != "sealed_predictions":
|
| 158 |
+
raise ValueError("invalid seal")
|
| 159 |
+
if (output_dir / "receipt.json").exists():
|
| 160 |
+
raise FileExistsError(output_dir / "receipt.json")
|
| 161 |
+
contract = load_contract()
|
| 162 |
+
request, request_file_sha = read_json(envelope_path)
|
| 163 |
+
contract.validate_forecast_request(request)
|
| 164 |
+
request_hash = contract.input_sha256(request)
|
| 165 |
+
if (
|
| 166 |
+
frozen.get("case_id") != request["case_id"]
|
| 167 |
+
or frozen.get("input_sha256") != request_hash
|
| 168 |
+
or frozen.get("envelope_file_sha256") != request_file_sha
|
| 169 |
+
):
|
| 170 |
+
raise ValueError("request differs from sealed request")
|
| 171 |
+
predictions = frozen["predictions"]
|
| 172 |
+
validate_predictions(predictions, request, request_hash)
|
| 173 |
+
# Oracle access begins only after the prediction seal has passed validation.
|
| 174 |
+
oracle, oracle_file_sha = read_json(oracle_path)
|
| 175 |
+
if oracle.get("schema_version") != ORACLE_VERSION or oracle.get("case_id") != request["case_id"]:
|
| 176 |
+
raise ValueError("oracle version or case_id differs from sealed request")
|
| 177 |
+
if set(oracle) != {"schema_version", "case_id", "observations"}:
|
| 178 |
+
raise ValueError("oracle contains unexpected fields")
|
| 179 |
+
target = request["target"]
|
| 180 |
+
observe = target["observe"]
|
| 181 |
+
branches = target["branches"]
|
| 182 |
+
validate_observations(oracle["observations"], target)
|
| 183 |
+
scores = {}
|
| 184 |
+
for name in COMPARATORS:
|
| 185 |
+
row = predictions["comparators"][name]
|
| 186 |
+
if row["status"] == "unsupported":
|
| 187 |
+
scores[name] = {
|
| 188 |
+
"status": "unsupported",
|
| 189 |
+
"reason": row["reason"],
|
| 190 |
+
"information_tier": row["information_tier"],
|
| 191 |
+
}
|
| 192 |
+
continue
|
| 193 |
+
branch_field_matches = {
|
| 194 |
+
branch: {
|
| 195 |
+
field: row["observations"][branch][field]
|
| 196 |
+
== oracle["observations"][branch][field]
|
| 197 |
+
for field in observe
|
| 198 |
+
}
|
| 199 |
+
for branch in branches
|
| 200 |
+
}
|
| 201 |
+
predicted_changed = (
|
| 202 |
+
row["observations"][branches[0]] != row["observations"][branches[1]]
|
| 203 |
+
)
|
| 204 |
+
observed_changed = (
|
| 205 |
+
oracle["observations"][branches[0]] != oracle["observations"][branches[1]]
|
| 206 |
+
)
|
| 207 |
+
scores[name] = {
|
| 208 |
+
"status": "scored",
|
| 209 |
+
"information_tier": row["information_tier"],
|
| 210 |
+
"method_sha256": row["method_sha256"],
|
| 211 |
+
"branch_field_matches": branch_field_matches,
|
| 212 |
+
"both_branches_match": all(
|
| 213 |
+
all(field_matches.values()) for field_matches in branch_field_matches.values()
|
| 214 |
+
),
|
| 215 |
+
"effect_changed_match": predicted_changed == observed_changed,
|
| 216 |
+
}
|
| 217 |
+
receipt = {
|
| 218 |
+
"schema_version": SCHEMA_VERSION,
|
| 219 |
+
"phase": "scored_development_oracle",
|
| 220 |
+
"development_only": True,
|
| 221 |
+
"case_id": request["case_id"],
|
| 222 |
+
"input_sha256": request_hash,
|
| 223 |
+
"seal_file_sha256": seal_file_sha,
|
| 224 |
+
"oracle_file_sha256": oracle_file_sha,
|
| 225 |
+
"supported_comparator_count": sum(row["status"] == "scored" for row in scores.values()),
|
| 226 |
+
"unsupported_comparator_count": sum(row["status"] == "unsupported" for row in scores.values()),
|
| 227 |
+
"same_information_scored_count": sum(
|
| 228 |
+
row["status"] == "scored" and row["information_tier"] == "SAME_INFORMATION"
|
| 229 |
+
for row in scores.values()
|
| 230 |
+
),
|
| 231 |
+
"privileged_ceiling_scored_count": sum(
|
| 232 |
+
row["status"] == "scored" and row["information_tier"] == "PRIVILEGED_CEILING"
|
| 233 |
+
for row in scores.values()
|
| 234 |
+
),
|
| 235 |
+
"scores": scores,
|
| 236 |
+
"prediction_methods_self_declared_not_verified_by_runner": True,
|
| 237 |
+
"baseline_methods_not_executed_by_runner": True,
|
| 238 |
+
"no_learned_value_claim": True,
|
| 239 |
+
}
|
| 240 |
+
write_create_only(output_dir / "receipt.json", receipt)
|
| 241 |
+
return receipt
|
| 242 |
+
|
| 243 |
+
|
| 244 |
+
def main():
|
| 245 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 246 |
+
subparsers = parser.add_subparsers(dest="phase", required=True)
|
| 247 |
+
seal_parser = subparsers.add_parser("seal")
|
| 248 |
+
seal_parser.add_argument("--envelope", type=Path, required=True)
|
| 249 |
+
seal_parser.add_argument("--predictions", type=Path, required=True)
|
| 250 |
+
seal_parser.add_argument("--output-dir", type=Path, required=True)
|
| 251 |
+
score_parser = subparsers.add_parser("score")
|
| 252 |
+
score_parser.add_argument("--envelope", type=Path, required=True)
|
| 253 |
+
score_parser.add_argument("--oracle", type=Path, required=True)
|
| 254 |
+
score_parser.add_argument("--output-dir", type=Path, required=True)
|
| 255 |
+
args = parser.parse_args()
|
| 256 |
+
if args.phase == "seal":
|
| 257 |
+
result = seal(args.envelope, args.predictions, args.output_dir)
|
| 258 |
+
else:
|
| 259 |
+
result = score(args.envelope, args.oracle, args.output_dir)
|
| 260 |
+
print(json.dumps({key: result[key] for key in ("case_id", "phase")}, sort_keys=True))
|
| 261 |
+
|
| 262 |
+
|
| 263 |
+
if __name__ == "__main__":
|
| 264 |
+
main()
|
evaluation/real-patch/source/code_review/scripts/run_real_patch_visible_exact_v1.py
ADDED
|
@@ -0,0 +1,300 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Same-information exact execution from one validated visible forecast request.
|
| 2 |
+
|
| 3 |
+
No archive, Git repository, custody record, oracle, model, or network is read.
|
| 4 |
+
The request's source, dependency, diff, and scenario bytes are materialized in
|
| 5 |
+
a create-only workspace. Runtime code executes only in locked-down Docker.
|
| 6 |
+
"""
|
| 7 |
+
|
| 8 |
+
import argparse
|
| 9 |
+
import hashlib
|
| 10 |
+
import importlib.util
|
| 11 |
+
import json
|
| 12 |
+
from pathlib import Path, PurePosixPath
|
| 13 |
+
import re
|
| 14 |
+
import subprocess
|
| 15 |
+
|
| 16 |
+
|
| 17 |
+
IMAGE = "python:3.12-slim"
|
| 18 |
+
IMAGE_ID = "sha256:78387bc3881b8273120a12ebe6c1ab22b018ccc2c9adf565ae1ac9b536e184ea"
|
| 19 |
+
PREDICTION_VERSION = "laxmi.real-patch-baseline-predictions.v2"
|
| 20 |
+
CONTRACT_PATH = Path(__file__).resolve().parents[1] / "contracts/real_patch_prediction_v1/validator.py"
|
| 21 |
+
HUNK = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@")
|
| 22 |
+
HEADER = re.compile(r"^diff --git a/(\S+) b/(\S+)\n$")
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
class UnsupportedVisibleExecution(ValueError):
|
| 26 |
+
"""The visible request cannot supply a valid runnable source slice."""
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def digest(data):
|
| 30 |
+
return hashlib.sha256(data).hexdigest()
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def canonical_bytes(value):
|
| 34 |
+
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode()
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def write_create_only(path, data):
|
| 38 |
+
path.parent.mkdir(parents=True, exist_ok=True)
|
| 39 |
+
with path.open("xb") as handle:
|
| 40 |
+
handle.write(data)
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def safe_path(root, relative):
|
| 44 |
+
path = PurePosixPath(relative)
|
| 45 |
+
if path.is_absolute() or ".." in path.parts or not path.parts:
|
| 46 |
+
raise UnsupportedVisibleExecution("patch path escapes visible workspace")
|
| 47 |
+
return root.joinpath(*path.parts)
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def apply_file_hunks(source, hunk_lines):
|
| 51 |
+
original = source.splitlines(keepends=True)
|
| 52 |
+
output = []
|
| 53 |
+
cursor = 0
|
| 54 |
+
index = 0
|
| 55 |
+
while index < len(hunk_lines):
|
| 56 |
+
match = HUNK.match(hunk_lines[index])
|
| 57 |
+
if match is None:
|
| 58 |
+
raise UnsupportedVisibleExecution("patch has content outside a hunk")
|
| 59 |
+
old_start = int(match.group(1))
|
| 60 |
+
old_count = int(match.group(2) or "1")
|
| 61 |
+
new_start = int(match.group(3))
|
| 62 |
+
new_count = int(match.group(4) or "1")
|
| 63 |
+
start = old_start - 1 if old_count else old_start
|
| 64 |
+
if start < cursor or start > len(original):
|
| 65 |
+
raise UnsupportedVisibleExecution("overlapping or out-of-range patch hunk")
|
| 66 |
+
output.extend(original[cursor:start])
|
| 67 |
+
expected_new_start = len(output) + 1 if new_count else len(output)
|
| 68 |
+
if new_start != expected_new_start:
|
| 69 |
+
raise UnsupportedVisibleExecution("patch destination hunk position is inconsistent")
|
| 70 |
+
position = start
|
| 71 |
+
old_seen = 0
|
| 72 |
+
new_seen = 0
|
| 73 |
+
index += 1
|
| 74 |
+
while index < len(hunk_lines) and not hunk_lines[index].startswith("@@ "):
|
| 75 |
+
line = hunk_lines[index]
|
| 76 |
+
if not line or line[0] not in " +-" or not line.endswith("\n"):
|
| 77 |
+
raise UnsupportedVisibleExecution("unsupported patch line")
|
| 78 |
+
text = line[1:]
|
| 79 |
+
if line[0] in " -":
|
| 80 |
+
if position >= len(original) or original[position] != text:
|
| 81 |
+
raise UnsupportedVisibleExecution("patch context differs from visible source")
|
| 82 |
+
position += 1
|
| 83 |
+
old_seen += 1
|
| 84 |
+
if line[0] in " +":
|
| 85 |
+
output.append(text)
|
| 86 |
+
new_seen += 1
|
| 87 |
+
index += 1
|
| 88 |
+
if old_seen != old_count or new_seen != new_count:
|
| 89 |
+
raise UnsupportedVisibleExecution("patch hunk line counts are inconsistent")
|
| 90 |
+
cursor = position
|
| 91 |
+
output.extend(original[cursor:])
|
| 92 |
+
return "".join(output)
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
def apply_visible_diff(pre_root, post_root, diff, source_paths):
|
| 96 |
+
lines = diff.splitlines(keepends=True)
|
| 97 |
+
index = 0
|
| 98 |
+
changed = []
|
| 99 |
+
while index < len(lines):
|
| 100 |
+
match = HEADER.match(lines[index])
|
| 101 |
+
if match is None or match.group(1) != match.group(2):
|
| 102 |
+
raise UnsupportedVisibleExecution("only same-path git-style text diffs are supported")
|
| 103 |
+
path = match.group(1)
|
| 104 |
+
if path not in source_paths or path in changed:
|
| 105 |
+
raise UnsupportedVisibleExecution("patch file absent from visible source or repeated")
|
| 106 |
+
changed.append(path)
|
| 107 |
+
index += 1
|
| 108 |
+
if index + 1 >= len(lines) or lines[index] != f"--- a/{path}\n" or lines[index + 1] != f"+++ b/{path}\n":
|
| 109 |
+
raise UnsupportedVisibleExecution("patch file headers are inconsistent")
|
| 110 |
+
index += 2
|
| 111 |
+
hunks = []
|
| 112 |
+
while index < len(lines) and not lines[index].startswith("diff --git "):
|
| 113 |
+
hunks.append(lines[index])
|
| 114 |
+
index += 1
|
| 115 |
+
if not hunks:
|
| 116 |
+
raise UnsupportedVisibleExecution("patch file has no hunks")
|
| 117 |
+
before = safe_path(pre_root, path).read_text(encoding="utf-8")
|
| 118 |
+
after = apply_file_hunks(before, hunks)
|
| 119 |
+
safe_path(post_root, path).write_text(after, encoding="utf-8")
|
| 120 |
+
if not changed:
|
| 121 |
+
raise UnsupportedVisibleExecution("visible diff has no patch files")
|
| 122 |
+
return changed
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
def load_contract():
|
| 126 |
+
spec = importlib.util.spec_from_file_location("real_patch_forecast_validator", CONTRACT_PATH)
|
| 127 |
+
module = importlib.util.module_from_spec(spec)
|
| 128 |
+
spec.loader.exec_module(module)
|
| 129 |
+
return module
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
def check_image():
|
| 133 |
+
result = subprocess.run(
|
| 134 |
+
["docker", "image", "inspect", IMAGE, "--format", "{{.Id}}"],
|
| 135 |
+
capture_output=True, text=True, check=True, timeout=20,
|
| 136 |
+
)
|
| 137 |
+
if result.stdout.strip() != IMAGE_ID:
|
| 138 |
+
raise ValueError("Docker image ID differs from pinned runtime")
|
| 139 |
+
|
| 140 |
+
|
| 141 |
+
def observation(completed, observe):
|
| 142 |
+
if completed.returncode == 0:
|
| 143 |
+
try:
|
| 144 |
+
payload = json.loads(completed.stdout)
|
| 145 |
+
except json.JSONDecodeError as exc:
|
| 146 |
+
raise UnsupportedVisibleExecution("visible program did not emit one JSON result") from exc
|
| 147 |
+
if not isinstance(payload, dict) or set(payload) != {"may", "actual"} or any(
|
| 148 |
+
type(value) is not bool for value in payload.values()
|
| 149 |
+
):
|
| 150 |
+
raise UnsupportedVisibleExecution("visible program result must be raw may/actual booleans")
|
| 151 |
+
values = {
|
| 152 |
+
"observation_class": "RETURN",
|
| 153 |
+
"return_value_sha256": digest(canonical_bytes(payload)),
|
| 154 |
+
"exception_type": None,
|
| 155 |
+
"exit_code": 0,
|
| 156 |
+
}
|
| 157 |
+
elif "ModuleNotFoundError" in completed.stderr or "ImportError" in completed.stderr:
|
| 158 |
+
raise UnsupportedVisibleExecution("visible source/dependency slice cannot import")
|
| 159 |
+
else:
|
| 160 |
+
values = {
|
| 161 |
+
"observation_class": "PROCESS_ERROR",
|
| 162 |
+
"return_value_sha256": None,
|
| 163 |
+
"exception_type": None,
|
| 164 |
+
"exit_code": completed.returncode,
|
| 165 |
+
}
|
| 166 |
+
return {field: values[field] for field in observe}
|
| 167 |
+
|
| 168 |
+
|
| 169 |
+
def run_branch(program_path, source_root, deps_root, timeout_seconds, observe):
|
| 170 |
+
command = [
|
| 171 |
+
"docker", "run", "--rm", "--pull", "never", "--network", "none",
|
| 172 |
+
"--read-only", "--cap-drop", "ALL", "--security-opt", "no-new-privileges",
|
| 173 |
+
"--pids-limit", "64", "--memory", "256m", "--cpus", "1",
|
| 174 |
+
"--user", "65534:65534",
|
| 175 |
+
"--mount", f"type=bind,source={program_path.resolve()},target=/scenario.py,readonly",
|
| 176 |
+
"--mount", f"type=bind,source={source_root.resolve()},target=/src,readonly",
|
| 177 |
+
"--mount", f"type=bind,source={deps_root.resolve()},target=/deps,readonly",
|
| 178 |
+
"--env", "PYTHONPATH=/src:/deps", "--env", "PYTHONDONTWRITEBYTECODE=1",
|
| 179 |
+
IMAGE, "python", "/scenario.py",
|
| 180 |
+
]
|
| 181 |
+
try:
|
| 182 |
+
result = subprocess.run(
|
| 183 |
+
command, capture_output=True, text=True, check=False, timeout=timeout_seconds,
|
| 184 |
+
)
|
| 185 |
+
except subprocess.TimeoutExpired:
|
| 186 |
+
values = {"observation_class": "TIMEOUT", "return_value_sha256": None,
|
| 187 |
+
"exception_type": None, "exit_code": None}
|
| 188 |
+
return {field: values[field] for field in observe}, {"command": command, "timed_out": True}
|
| 189 |
+
if result.returncode == 125:
|
| 190 |
+
raise RuntimeError(f"Docker setup failed: {result.stderr[-1000:]}")
|
| 191 |
+
return observation(result, observe), {
|
| 192 |
+
"command": command,
|
| 193 |
+
"exit_code": result.returncode,
|
| 194 |
+
"stdout": result.stdout,
|
| 195 |
+
"stderr": result.stderr,
|
| 196 |
+
"stdout_sha256": digest(result.stdout.encode()),
|
| 197 |
+
"stderr_sha256": digest(result.stderr.encode()),
|
| 198 |
+
}
|
| 199 |
+
|
| 200 |
+
|
| 201 |
+
def run(request_path, output_dir):
|
| 202 |
+
contract = load_contract()
|
| 203 |
+
request = contract.load_forecast_request(request_path)
|
| 204 |
+
if request["target"]["branches"] != ["NO_CHANGE", "PROPOSED_PATCH"]:
|
| 205 |
+
raise ValueError("paired branch order required")
|
| 206 |
+
observe = request["target"]["observe"]
|
| 207 |
+
if "observation_class" not in observe or "return_value_sha256" not in observe:
|
| 208 |
+
raise ValueError("raw paired result fields required")
|
| 209 |
+
output_dir.mkdir(mode=0o700, parents=True, exist_ok=False)
|
| 210 |
+
workspace = output_dir / "workspace"
|
| 211 |
+
pre_root = workspace / "no_change"
|
| 212 |
+
post_root = workspace / "proposed_patch"
|
| 213 |
+
deps_root = workspace / "deps"
|
| 214 |
+
source_paths = set()
|
| 215 |
+
for item in request["pre_source"]["files"]:
|
| 216 |
+
source_paths.add(item["path"])
|
| 217 |
+
for root in (pre_root, post_root):
|
| 218 |
+
write_create_only(safe_path(root, item["path"]), item["content"].encode())
|
| 219 |
+
for item in request["operating_conditions"]["dependency_sources"]:
|
| 220 |
+
relative = item["path"]
|
| 221 |
+
if not relative.startswith("deps/"):
|
| 222 |
+
raise ValueError("dependency path must be under deps/")
|
| 223 |
+
write_create_only(safe_path(workspace, relative), item["content"].encode())
|
| 224 |
+
program_path = workspace / "scenario.py"
|
| 225 |
+
write_create_only(program_path, request["scenario"]["program"].encode())
|
| 226 |
+
method = {
|
| 227 |
+
"script_sha256": digest(Path(__file__).read_bytes()),
|
| 228 |
+
"image_id": IMAGE_ID,
|
| 229 |
+
"input_sha256": contract.input_sha256(request),
|
| 230 |
+
"visible_source_snapshot_sha256": request["pre_source"]["snapshot_sha256"],
|
| 231 |
+
"visible_diff_sha256": request["proposed_patch"]["diff_sha256"],
|
| 232 |
+
"visible_dependency_lock_sha256": request["operating_conditions"]["dependency_lock_sha256"],
|
| 233 |
+
"visible_program_sha256": request["scenario"]["program_sha256"],
|
| 234 |
+
}
|
| 235 |
+
runs = {}
|
| 236 |
+
changed = []
|
| 237 |
+
try:
|
| 238 |
+
changed = apply_visible_diff(
|
| 239 |
+
pre_root, post_root, request["proposed_patch"]["unified_diff"], source_paths
|
| 240 |
+
)
|
| 241 |
+
check_image()
|
| 242 |
+
observations = {}
|
| 243 |
+
for branch, root in (("NO_CHANGE", pre_root), ("PROPOSED_PATCH", post_root)):
|
| 244 |
+
observations[branch], runs[branch] = run_branch(
|
| 245 |
+
program_path, root, deps_root,
|
| 246 |
+
request["target"]["horizon"]["max_seconds"], observe,
|
| 247 |
+
)
|
| 248 |
+
exact = {
|
| 249 |
+
"status": "predicted",
|
| 250 |
+
"information_tier": "SAME_INFORMATION",
|
| 251 |
+
"observations": observations,
|
| 252 |
+
"method_sha256": digest(canonical_bytes(method)),
|
| 253 |
+
}
|
| 254 |
+
except UnsupportedVisibleExecution as exc:
|
| 255 |
+
exact = {
|
| 256 |
+
"status": "unsupported",
|
| 257 |
+
"information_tier": "SAME_INFORMATION",
|
| 258 |
+
"reason": str(exc),
|
| 259 |
+
}
|
| 260 |
+
def unsupported(reason, tier):
|
| 261 |
+
return {"status": "unsupported", "information_tier": tier, "reason": reason}
|
| 262 |
+
prediction = {
|
| 263 |
+
"schema_version": PREDICTION_VERSION,
|
| 264 |
+
"case_id": request["case_id"],
|
| 265 |
+
"input_sha256": contract.input_sha256(request),
|
| 266 |
+
"comparators": {
|
| 267 |
+
"EXACT_EXECUTION": unsupported("Privileged full-source ceiling is separate", "PRIVILEGED_CEILING"),
|
| 268 |
+
"EXACT_VISIBLE_SOURCE": exact,
|
| 269 |
+
"STATIC_RULES": unsupported("No static comparator sealed for this case", "SAME_INFORMATION"),
|
| 270 |
+
"RETRIEVAL": unsupported("No pre-cutoff retrieval index sealed for this case", "SAME_INFORMATION"),
|
| 271 |
+
},
|
| 272 |
+
}
|
| 273 |
+
receipt = {
|
| 274 |
+
"schema_version": "laxmi.real-patch-visible-exact.v1",
|
| 275 |
+
"development_only": True,
|
| 276 |
+
"oracle_read": False,
|
| 277 |
+
"custody_or_post_archive_read": False,
|
| 278 |
+
"input_sha256": contract.input_sha256(request),
|
| 279 |
+
"method": method,
|
| 280 |
+
"changed_visible_paths": changed,
|
| 281 |
+
"runs": runs,
|
| 282 |
+
"status": exact["status"],
|
| 283 |
+
}
|
| 284 |
+
for name, value in (("predictions.json", prediction), ("execution.json", receipt)):
|
| 285 |
+
write_create_only(output_dir / name, json.dumps(value, sort_keys=True, indent=2).encode() + b"\n")
|
| 286 |
+
return prediction, receipt
|
| 287 |
+
|
| 288 |
+
|
| 289 |
+
def main():
|
| 290 |
+
parser = argparse.ArgumentParser(description=__doc__)
|
| 291 |
+
parser.add_argument("--request", type=Path, required=True)
|
| 292 |
+
parser.add_argument("--output-dir", type=Path, required=True)
|
| 293 |
+
args = parser.parse_args()
|
| 294 |
+
prediction, _ = run(args.request, args.output_dir)
|
| 295 |
+
print(json.dumps({"case_id": prediction["case_id"],
|
| 296 |
+
"status": prediction["comparators"]["EXACT_VISIBLE_SOURCE"]["status"]}, sort_keys=True))
|
| 297 |
+
|
| 298 |
+
|
| 299 |
+
if __name__ == "__main__":
|
| 300 |
+
main()
|
release.json
CHANGED
|
@@ -2,33 +2,46 @@
|
|
| 2 |
"author": "Navneet Prabhakar",
|
| 3 |
"candidates": {
|
| 4 |
"17-normalized": {
|
| 5 |
-
"
|
| 6 |
-
"
|
| 7 |
},
|
| 8 |
"17-original": {
|
| 9 |
-
"
|
| 10 |
-
"
|
| 11 |
},
|
| 12 |
"43-normalized": {
|
| 13 |
-
"
|
| 14 |
-
"
|
| 15 |
},
|
| 16 |
"43-original": {
|
| 17 |
-
"
|
| 18 |
-
"
|
| 19 |
}
|
| 20 |
},
|
| 21 |
"clean_install_verified": true,
|
| 22 |
"dependency_hash_lock": false,
|
|
|
|
| 23 |
"license": "Apache-2.0",
|
| 24 |
"model_acceptance": false,
|
| 25 |
"original_wheel_sha256": "e1bafeb8d72e231a3029cf11ffabb8e54aa28e1b953040e81e77cf1fa3ac270d",
|
| 26 |
"package_sha256": "a74db8e3196267387e3a4cfb55b838d112a86c88adac031f6515b88230a29fcc",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
"runtime_sources": 133,
|
| 28 |
"schema_version": "laxmi.experimental-public-package.v1",
|
|
|
|
| 29 |
"supported_platform": "macOS ARM64, Python 3.12; other platforms unverified",
|
| 30 |
-
"version": "0.1.
|
| 31 |
-
"distribution_scope": "RELEASE_SCOPE.md; exact files enumerated by SHA256SUMS",
|
| 32 |
-
"repository_checkpoint_commit": "4dae1a041be532d663f477effe9b654b428bc04e",
|
| 33 |
-
"source_identity_basis": "original runtime wheel SHA256 and per-file checkpoint source locks; repository checkpoint is a recovery anchor, not a claim of identical historical source tree"
|
| 34 |
}
|
|
|
|
| 2 |
"author": "Navneet Prabhakar",
|
| 3 |
"candidates": {
|
| 4 |
"17-normalized": {
|
| 5 |
+
"example_forecast_sha256": "e4c9661abcc03fe8aee0ae5bf57e1b12f9e8a56becc67e41d26ea106b0ae5267",
|
| 6 |
+
"metadata_sha256": "225a71bb67d619e70f1aec0313d2c5791d0a94ac3942297ad12ba284269cddd8"
|
| 7 |
},
|
| 8 |
"17-original": {
|
| 9 |
+
"example_forecast_sha256": "5cca70b946ccaac1a453f899fc54e088452728a24ae9e6ba7f5f2e8ee5b6e6e6",
|
| 10 |
+
"metadata_sha256": "a958ceb64e802decf331d10e98ddfcdc042658834469a0cc4a131e44134de906"
|
| 11 |
},
|
| 12 |
"43-normalized": {
|
| 13 |
+
"example_forecast_sha256": "cbe28210ab8bce43ebe5223b38c62fdcd07cf796d26173651da03fd32c2135e8",
|
| 14 |
+
"metadata_sha256": "8eb622127bb5a9d739967855b8d7d8b7290395d7ae3a83315325b1a20869146f"
|
| 15 |
},
|
| 16 |
"43-original": {
|
| 17 |
+
"example_forecast_sha256": "c18c7cbb05b59973026e370a6ba8bfb2e35f1e9d772ff183a32dec94da6ccc93",
|
| 18 |
+
"metadata_sha256": "4e02bd6724288e074fb30f04164a83559eff5e370e94bc473ace3ac22bc7a742"
|
| 19 |
}
|
| 20 |
},
|
| 21 |
"clean_install_verified": true,
|
| 22 |
"dependency_hash_lock": false,
|
| 23 |
+
"distribution_scope": "RELEASE_SCOPE.md; exact files enumerated by SHA256SUMS",
|
| 24 |
"license": "Apache-2.0",
|
| 25 |
"model_acceptance": false,
|
| 26 |
"original_wheel_sha256": "e1bafeb8d72e231a3029cf11ffabb8e54aa28e1b953040e81e77cf1fa3ac270d",
|
| 27 |
"package_sha256": "a74db8e3196267387e3a4cfb55b838d112a86c88adac031f6515b88230a29fcc",
|
| 28 |
+
"real_patch_tool": {
|
| 29 |
+
"blind_acceptance_cases": 0,
|
| 30 |
+
"exposed_development_cases": 6,
|
| 31 |
+
"hosted_api": false,
|
| 32 |
+
"kind": "local_exact_execution_baseline",
|
| 33 |
+
"learned_real_patch_weights": false,
|
| 34 |
+
"linux_exact_execution_qualified": false,
|
| 35 |
+
"oracle_only_cases": 5,
|
| 36 |
+
"scored_forecast_cases": 1,
|
| 37 |
+
"source_commit": "63cbecbad1bd8832d9c4fae65e4cfad27c57fa76",
|
| 38 |
+
"wheel": "evaluation/real-patch/runtime/laxmi_real_patch-0.1.0.dev1-py3-none-any.whl",
|
| 39 |
+
"wheel_sha256": "a36e136b730b9b941af303ca23dcdaaff0268b3e3dfafab4122730ee45d8da0e"
|
| 40 |
+
},
|
| 41 |
+
"repository_checkpoint_commit": "4dae1a041be532d663f477effe9b654b428bc04e",
|
| 42 |
"runtime_sources": 133,
|
| 43 |
"schema_version": "laxmi.experimental-public-package.v1",
|
| 44 |
+
"source_identity_basis": "original runtime wheel SHA256 and per-file checkpoint source locks; repository checkpoint is a recovery anchor, not a claim of identical historical source tree",
|
| 45 |
"supported_platform": "macOS ARM64, Python 3.12; other platforms unverified",
|
| 46 |
+
"version": "0.1.1"
|
|
|
|
|
|
|
|
|
|
| 47 |
}
|