test_codabench/README.md

# DRU-RE-Yehia Codabench blind-test archive

This folder archives the complete reproducible inference run for the 4,386-row Codabench blind test. The primary ready-to-upload artifact is `submissions/submission.zip`, which uses majority-vote aggregation. All four aggregation variants were produced from the same model scores; the model was not rerun or changed between ZIPs.

## Model overview

The relation model is the public `U4RASD/DRU-RE-Yehia` QLoRA adapter attached to the gated `Navid-AI/Yehia-7B-preview` base. The adapter is a directed Arabic relation classifier: it expects one exact subject occurrence, one exact object occurrence, and their two coarse entity types. It is not a standalone model and does not discover spans or types itself.

The full pipeline therefore combines:

- `U4RASD/TypePredictor` for the 21 Wojood coarse entity types;
- the released `U4RASD/DRU-RE-Yehia` prompt preparation and ontology resources;
- the pinned Yehia 7B base and released QLoRA adapter;
- vLLM 0.10.2 for Yehia causal-LM inference;
- four row-level candidate aggregation strategies.

The exact model revisions, prompt version, bias, counts, paths, and SHA-256 hashes are recorded in `outputs/run_manifest.json`.

## How the blind test was processed

Input spans and entity types in the test rows were deliberately ignored. For each row the pipeline:

1. Read `sentence`, `subject`, `object`, and `triple_id`.
2. Found every exact occurrence of the subject string and every exact occurrence of the object string. A conservative Arabic-normalization fallback was available, but all 4,386 rows matched exactly.
3. Predicted a Wojood coarse type for every unique mention occurrence, not merely once per surface string.
4. Constructed every directed subject-occurrence × object-occurrence pair. This matters when a subject or object appears multiple times in one sentence.
5. Sent every pair through the released `prepare_input.py`, preserving the directional markers, Arabic type names, relation templates, ontology-filtered options, deterministic option shuffling, and final `لا توجد علاقة` option.
6. Used vLLM with the Yehia base and released LoRA to score only the valid one-token Arabic choice codes.
7. Applied the released frozen no-relation logit bias of `-6.6600000000000215`.
8. Aggregated all candidate-pair decisions back into one relation per original `triple_id`.
9. Converted `no_relation` to the required Codabench spelling `no-relation` and created deterministic ZIP files containing exactly `predictions.txt`.

Run statistics:

- Test rows: 4,386
- Unique mention occurrences typed: 7,497
- Directed occurrence-pair candidates scored: 5,264
- Rows with no recoverable candidate: 0
- Maximum prompt length: 548 tokens
- Inference backend: vLLM 0.10.2, V0 engine

## Submission ZIPs and their differences

Every ZIP contains exactly one file named `predictions.txt`. Every prediction file has 4,386 lines in original test order, formatted as:

```text
triple_id<TAB>relation
```

### `submission.zip`

The primary Codabench artifact. It is byte-for-byte identical to `submission_majority_vote.zip` and has SHA-256:

```text
f19d6484796f60cd02c80a47cbdb7f122c0613c36cf0b5abd7c9502f5dd43bad
```

### `submission_majority_vote.zip`

For rows with multiple occurrence pairs, at least half of the candidates must predict a positive relation for the row to be positive. Positive candidates vote for the relation. Ties are resolved by summed probability, then score margin, then canonical relation order. This is the recommended primary strategy.

### `submission_soft_pool.zip`

Averages the row-local option probabilities across all candidate pairs, gates on average positive probability, and chooses the highest pooled positive relation. It differs from majority vote on 14 of 4,386 rows.

SHA-256: `d27af9544ebf4cca907b79780e2b03ca24253f8b70a3b4773630aa5518a058b8`

### `submission_max_positive.zip`

Uses the highest-confidence candidate that selected a positive relation. If no candidate selected a positive relation, the row remains no-relation. It differs from majority vote on 13 rows.

SHA-256: `befefe6f0738b3bf45e5ad4fab68bd27188dbbc20839742081a8afea0c898931`

### `submission_top_confidence.zip`

Uses the single most confident candidate decision, including a no-relation decision. It differs from majority vote on 14 rows.

SHA-256: `2ff413bfbf40afefcf4e355e72bebb1b5e7745f408012dc7c502ce5a234f51f7`

## Folder contents

```text
test_codabench/
├── README.md
├── data/
│   └── test.jsonl
├── submissions/
│   ├── submission.zip
│   ├── submission_majority_vote.zip
│   ├── submission_soft_pool.zip
│   ├── submission_max_positive.zip
│   ├── submission_top_confidence.zip
│   └── predictions*.txt
├── outputs/
│   ├── input_occurrence_audit.jsonl
│   ├── type_predictions.jsonl
│   ├── candidate_inputs_raw.jsonl
│   ├── candidate_inputs_prepared.jsonl
│   ├── candidate_predictions.jsonl
│   ├── row_predictions_debug.jsonl
│   ├── predictions_primary.jsonl
│   └── run_manifest.json
└── runner/
    ├── DRU_RE_Yehia_vLLM_runner.zip
    ├── original_full_pipeline.zip
    └── unpacked runner source files
```

The large downloaded model snapshots and Python virtual environment are intentionally not archived. They can be reconstructed from `runner/requirements.txt`, the runner source, and the pinned revisions in `outputs/run_manifest.json`. No Hugging Face tokens or `.env` credential file are included.

## Reproducing the run

1. Extract `runner/DRU_RE_Yehia_vLLM_runner.zip`.
2. Copy `.env.template` to `.env` and provide a token with access to `Navid-AI/Yehia-7B-preview`.
3. Place `data/test.jsonl` at the configured input path.
4. Run `bash setup_env.sh`, followed by `bash run_test.sh`.

The Yehia relation-model stage uses vLLM. Transformers remains involved only for the custom TypePredictor encoder and deterministic tokenizer/chat-template processing.