test_codabench/runner/README.md

DRU-RE-Yehia Full-Pipeline Blind-Test Package

This package runs the public U4RASD/DRU-RE-Yehia adapter as a span-free full pipeline over a blind test.jsonl.

The released Yehia model itself is a relation classifier. It expects one exact directed entity pair and two coarse types. This package supplies those missing inputs by enumerating every mention occurrence and running U4RASD/TypePredictor.

What it does

For each test row:

1. Read sentence, subject, object, and triple_id.
2. Ignore any spans or types supplied in the row.
3. Find every occurrence of the subject string and object string.
4. Predict one of the 21 Wojood coarse types for every unique occurrence.
5. Build every directed subject-occurrence × object-occurrence candidate.
6. Download the public U4RASD/DRU-RE-Yehia release.
7. Run the release's own prepare_input.py, which preserves:
   • [الأول] and [الثاني] directional markers;
   • Arabic type names;
   • ontology-compatible relation options;
   • Arabic relation templates;
   • deterministic row-local option shuffling;
   • prompt version yehia_re_ar_v3_one_token_choice;
   • لا توجد علاقة as the final option.
8. Load the gated pinned Yehia 7B base in BF16 with vLLM and attach the public best QLoRA adapter.
9. Score only the valid one-token Arabic option codes.
10. Apply the released frozen no-relation logit bias.
11. Aggregate candidate decisions.
12. Create and validate /workspace/submission.zip.

No free-text generation or phrase parsing is used.

RunPod layout

Put these two files in /workspace:

/workspace/
├── DRU_RE_Yehia_FullPipeline_Test.zip
└── test.jsonl

Then run:

cd /workspace
unzip DRU_RE_Yehia_FullPipeline_Test.zip
cd DRU_RE_Yehia_FullPipeline_Test

cp .env.template .env
nano .env

Set:

HF_TOKENONE=...

HF_TOKENONE must have access to the gated Navid-AI/Yehia-7B-preview model.

Then:

bash setup_env.sh
bash run_test.sh

The final artifact is:

/workspace/submission.zip

The ZIP contains exactly:

predictions.txt

Each line is:

triple_id<TAB>relation

no_relation is converted to the competition spelling no-relation.

Input schema

The script accepts common aliases, but the recommended row is:

{
  "triple_id": "123",
  "sentence_id": "456",
  "sentence": "يعمل أحمد في جامعة بيرزيت.",
  "subject": "أحمد",
  "object": "جامعة بيرزيت"
}

subject and object can also be small objects containing a text, mention, surface, or name field.

Input-only inspection

Before downloading models:

bash run_test.sh --validate-input-only

This creates:

/workspace/yehia_test_output/input_occurrence_audit.jsonl

and prints the candidate-count distribution.

Candidate aggregation

Primary default:

AGGREGATION_METHOD=majority_vote

A row is positive when at least half of its candidate pairs choose a positive relation. Positive candidates vote for the relation. Ties are broken by summed row-local probability, then score margin, then canonical relation order.

The same model run also creates alternatives unless disabled:

submission_majority_vote.zip
submission_soft_pool.zip
submission_max_positive.zip
submission_top_confidence.zip

The primary /workspace/submission.zip uses the method selected in .env.

soft_pool

Averages candidate option probabilities, gates on average positive probability, and selects the highest pooled positive relation.

max_positive

Uses the highest-confidence candidate that selected a positive relation.

top_confidence

Uses the most confident candidate decision, including no_relation.

No alternative method reruns the models.

Output directory

The default directory is:

/workspace/yehia_test_output/

Important files:

input_occurrence_audit.jsonl
type_predictions.jsonl
candidate_inputs_raw.jsonl
candidate_inputs_prepared.jsonl
candidate_predictions.jsonl
row_predictions_debug.jsonl
predictions_primary.jsonl
predictions.txt
submission.zip
run_manifest.json

run_manifest.json records the resolved TypePredictor and adapter commits, pinned base revision, prompt version, frozen bias, counts, and SHA-256 hashes.

GPU notes

The pipeline is sequential:

1. TypePredictor is loaded and run.
2. It is deleted and CUDA cache is cleared.
3. Yehia 7B is loaded by vLLM in BF16 and the LoRA is applied by vLLM.

Defaults target an NVIDIA L40/L40S 48 GB:

TYPE_BATCH_SIZE=64
VLLM_MAX_NUM_SEQS=256
VLLM_REQUEST_BATCH_SIZE=512

Reduce VLLM_MAX_NUM_SEQS or VLLM_GPU_MEMORY_UTILIZATION if engine startup runs out of CUDA memory.

Mention matching

Exact matching is tried first. A conservative fallback handles:

• outer whitespace;
• surrounding punctuation;
• Arabic alef variants;
• alif maqsura/yaa normalization;
• diacritics and tatweel.

The actual sentence slice at each recovered span is sent to the released prompt builder, so its span validation remains strict.

Rows with no recoverable candidates default to no_relation and are recorded in the audit/debug files. Set:

FAIL_FAST=true

to stop instead.

Reuse downloaded models

The package stores models under:

/workspace/models/
├── DRU-RE-Yehia/
└── Yehia-7B-preview/

Later runs reuse a snapshot only when its stored revision marker matches the requested revision.

Exact released behavior preserved

The package deliberately uses the model repository's own prompt-preparation code rather than reimplementing the prompt. It also reproduces the released constrained decoder:

• native Yehia chat template exactly once;
• next-token logits after the answer prefix;
• only row-local code tokens considered;
• released no-relation bias applied to the last code;
• argmax mapped through that row's option arrays.

The QLoRA adapter is not a standalone model. Access to the pinned gated base is required. Transformers is used only for the custom TypePredictor encoder and deterministic tokenization/chat templating; Yehia causal-LM inference runs through vLLM.