DRU-RE-Yehia Codabench blind-test archive

This folder archives the complete reproducible inference run for the 4,386-row Codabench blind test. The primary ready-to-upload artifact is submissions/submission.zip, which uses majority-vote aggregation. All four aggregation variants were produced from the same model scores; the model was not rerun or changed between ZIPs.

Model overview

The relation model is the public U4RASD/DRU-RE-Yehia QLoRA adapter attached to the gated Navid-AI/Yehia-7B-preview base. The adapter is a directed Arabic relation classifier: it expects one exact subject occurrence, one exact object occurrence, and their two coarse entity types. It is not a standalone model and does not discover spans or types itself.

The full pipeline therefore combines:

U4RASD/TypePredictor for the 21 Wojood coarse entity types;
the released U4RASD/DRU-RE-Yehia prompt preparation and ontology resources;
the pinned Yehia 7B base and released QLoRA adapter;
vLLM 0.10.2 for Yehia causal-LM inference;
four row-level candidate aggregation strategies.

The exact model revisions, prompt version, bias, counts, paths, and SHA-256 hashes are recorded in outputs/run_manifest.json.

How the blind test was processed

Input spans and entity types in the test rows were deliberately ignored. For each row the pipeline:

Read sentence, subject, object, and triple_id.
Found every exact occurrence of the subject string and every exact occurrence of the object string. A conservative Arabic-normalization fallback was available, but all 4,386 rows matched exactly.
Predicted a Wojood coarse type for every unique mention occurrence, not merely once per surface string.
Constructed every directed subject-occurrence × object-occurrence pair. This matters when a subject or object appears multiple times in one sentence.
Sent every pair through the released prepare_input.py, preserving the directional markers, Arabic type names, relation templates, ontology-filtered options, deterministic option shuffling, and final لا توجد علاقة option.
Used vLLM with the Yehia base and released LoRA to score only the valid one-token Arabic choice codes.
Applied the released frozen no-relation logit bias of -6.6600000000000215.
Aggregated all candidate-pair decisions back into one relation per original triple_id.
Converted no_relation to the required Codabench spelling no-relation and created deterministic ZIP files containing exactly predictions.txt.

Run statistics:

Test rows: 4,386
Unique mention occurrences typed: 7,497
Directed occurrence-pair candidates scored: 5,264
Rows with no recoverable candidate: 0
Maximum prompt length: 548 tokens
Inference backend: vLLM 0.10.2, V0 engine

Submission ZIPs and their differences

Every ZIP contains exactly one file named predictions.txt. Every prediction file has 4,386 lines in original test order, formatted as:

triple_id<TAB>relation

`submission.zip`

The primary Codabench artifact. It is byte-for-byte identical to submission_majority_vote.zip and has SHA-256:

f19d6484796f60cd02c80a47cbdb7f122c0613c36cf0b5abd7c9502f5dd43bad

`submission_majority_vote.zip`

For rows with multiple occurrence pairs, at least half of the candidates must predict a positive relation for the row to be positive. Positive candidates vote for the relation. Ties are resolved by summed probability, then score margin, then canonical relation order. This is the recommended primary strategy.

`submission_soft_pool.zip`

Averages the row-local option probabilities across all candidate pairs, gates on average positive probability, and chooses the highest pooled positive relation. It differs from majority vote on 14 of 4,386 rows.

SHA-256: d27af9544ebf4cca907b79780e2b03ca24253f8b70a3b4773630aa5518a058b8

`submission_max_positive.zip`

Uses the highest-confidence candidate that selected a positive relation. If no candidate selected a positive relation, the row remains no-relation. It differs from majority vote on 13 rows.

SHA-256: befefe6f0738b3bf45e5ad4fab68bd27188dbbc20839742081a8afea0c898931

`submission_top_confidence.zip`

Uses the single most confident candidate decision, including a no-relation decision. It differs from majority vote on 14 rows.

SHA-256: 2ff413bfbf40afefcf4e355e72bebb1b5e7745f408012dc7c502ce5a234f51f7

Folder contents

test_codabench/
├── README.md
├── data/
│   └── test.jsonl
├── submissions/
│   ├── submission.zip
│   ├── submission_majority_vote.zip
│   ├── submission_soft_pool.zip
│   ├── submission_max_positive.zip
│   ├── submission_top_confidence.zip
│   └── predictions*.txt
├── outputs/
│   ├── input_occurrence_audit.jsonl
│   ├── type_predictions.jsonl
│   ├── candidate_inputs_raw.jsonl
│   ├── candidate_inputs_prepared.jsonl
│   ├── candidate_predictions.jsonl
│   ├── row_predictions_debug.jsonl
│   ├── predictions_primary.jsonl
│   └── run_manifest.json
└── runner/
    ├── DRU_RE_Yehia_vLLM_runner.zip
    ├── original_full_pipeline.zip
    └── unpacked runner source files

The large downloaded model snapshots and Python virtual environment are intentionally not archived. They can be reconstructed from runner/requirements.txt, the runner source, and the pinned revisions in outputs/run_manifest.json. No Hugging Face tokens or .env credential file are included.

Reproducing the run

Extract runner/DRU_RE_Yehia_vLLM_runner.zip.
Copy .env.template to .env and provide a token with access to Navid-AI/Yehia-7B-preview.
Place data/test.jsonl at the configured input path.
Run bash setup_env.sh, followed by bash run_test.sh.

The Yehia relation-model stage uses vLLM. Transformers remains involved only for the custom TypePredictor encoder and deterministic tokenizer/chat-template processing.