test_codabench/runner/README.md

# DRU-RE-Yehia Full-Pipeline Blind-Test Package

This package runs the public `U4RASD/DRU-RE-Yehia` adapter as a **span-free full pipeline** over a blind `test.jsonl`.

The released Yehia model itself is a relation classifier. It expects one exact directed entity pair and two coarse types. This package supplies those missing inputs by enumerating every mention occurrence and running `U4RASD/TypePredictor`.

## What it does

For each test row:

1. Read `sentence`, `subject`, `object`, and `triple_id`.
2. Ignore any spans or types supplied in the row.
3. Find every occurrence of the subject string and object string.
4. Predict one of the 21 Wojood coarse types for every unique occurrence.
5. Build every directed subject-occurrence × object-occurrence candidate.
6. Download the public `U4RASD/DRU-RE-Yehia` release.
7. Run the release's own `prepare_input.py`, which preserves:
   - `[الأول]` and `[الثاني]` directional markers;
   - Arabic type names;
   - ontology-compatible relation options;
   - Arabic relation templates;
   - deterministic row-local option shuffling;
   - prompt version `yehia_re_ar_v3_one_token_choice`;
   - `لا توجد علاقة` as the final option.
8. Load the gated pinned Yehia 7B base in BF16 with vLLM and attach the public best QLoRA adapter.
9. Score only the valid one-token Arabic option codes.
10. Apply the released frozen no-relation logit bias.
11. Aggregate candidate decisions.
12. Create and validate `/workspace/submission.zip`.

No free-text generation or phrase parsing is used.

## RunPod layout

Put these two files in `/workspace`:

```text
/workspace/
├── DRU_RE_Yehia_FullPipeline_Test.zip
└── test.jsonl
```

Then run:

```bash
cd /workspace
unzip DRU_RE_Yehia_FullPipeline_Test.zip
cd DRU_RE_Yehia_FullPipeline_Test

cp .env.template .env
nano .env
```

Set:

```text
HF_TOKENONE=...
```

`HF_TOKENONE` must have access to the gated `Navid-AI/Yehia-7B-preview` model.

Then:

```bash
bash setup_env.sh
bash run_test.sh
```

The final artifact is:

```text
/workspace/submission.zip
```

The ZIP contains exactly:

```text
predictions.txt
```

Each line is:

```text
triple_id<TAB>relation
```

`no_relation` is converted to the competition spelling `no-relation`.

## Input schema

The script accepts common aliases, but the recommended row is:

```json
{
  "triple_id": "123",
  "sentence_id": "456",
  "sentence": "يعمل أحمد في جامعة بيرزيت.",
  "subject": "أحمد",
  "object": "جامعة بيرزيت"
}
```

`subject` and `object` can also be small objects containing a `text`, `mention`, `surface`, or `name` field.

## Input-only inspection

Before downloading models:

```bash
bash run_test.sh --validate-input-only
```

This creates:

```text
/workspace/yehia_test_output/input_occurrence_audit.jsonl
```

and prints the candidate-count distribution.

## Candidate aggregation

Primary default:

```text
AGGREGATION_METHOD=majority_vote
```

A row is positive when at least half of its candidate pairs choose a positive relation. Positive candidates vote for the relation. Ties are broken by summed row-local probability, then score margin, then canonical relation order.

The same model run also creates alternatives unless disabled:

```text
submission_majority_vote.zip
submission_soft_pool.zip
submission_max_positive.zip
submission_top_confidence.zip
```

The primary `/workspace/submission.zip` uses the method selected in `.env`.

### `soft_pool`

Averages candidate option probabilities, gates on average positive probability, and selects the highest pooled positive relation.

### `max_positive`

Uses the highest-confidence candidate that selected a positive relation.

### `top_confidence`

Uses the most confident candidate decision, including `no_relation`.

No alternative method reruns the models.

## Output directory

The default directory is:

```text
/workspace/yehia_test_output/
```

Important files:

```text
input_occurrence_audit.jsonl
type_predictions.jsonl
candidate_inputs_raw.jsonl
candidate_inputs_prepared.jsonl
candidate_predictions.jsonl
row_predictions_debug.jsonl
predictions_primary.jsonl
predictions.txt
submission.zip
run_manifest.json
```

`run_manifest.json` records the resolved TypePredictor and adapter commits, pinned base revision, prompt version, frozen bias, counts, and SHA-256 hashes.

## GPU notes

The pipeline is sequential:

1. TypePredictor is loaded and run.
2. It is deleted and CUDA cache is cleared.
3. Yehia 7B is loaded by vLLM in BF16 and the LoRA is applied by vLLM.

Defaults target an NVIDIA L40/L40S 48 GB:

```text
TYPE_BATCH_SIZE=64
VLLM_MAX_NUM_SEQS=256
VLLM_REQUEST_BATCH_SIZE=512
```

Reduce `VLLM_MAX_NUM_SEQS` or `VLLM_GPU_MEMORY_UTILIZATION` if engine startup runs out of CUDA memory.

## Mention matching

Exact matching is tried first. A conservative fallback handles:

- outer whitespace;
- surrounding punctuation;
- Arabic alef variants;
- alif maqsura/yaa normalization;
- diacritics and tatweel.

The actual sentence slice at each recovered span is sent to the released prompt builder, so its span validation remains strict.

Rows with no recoverable candidates default to `no_relation` and are recorded in the audit/debug files. Set:

```text
FAIL_FAST=true
```

to stop instead.

## Reuse downloaded models

The package stores models under:

```text
/workspace/models/
├── DRU-RE-Yehia/
└── Yehia-7B-preview/
```

Later runs reuse a snapshot only when its stored revision marker matches the requested revision.

## Exact released behavior preserved

The package deliberately uses the model repository's own prompt-preparation code rather than reimplementing the prompt. It also reproduces the released constrained decoder:

- native Yehia chat template exactly once;
- next-token logits after the answer prefix;
- only row-local code tokens considered;
- released no-relation bias applied to the last code;
- argmax mapped through that row's option arrays.

The QLoRA adapter is not a standalone model. Access to the pinned gated base is required. Transformers is used only for the custom TypePredictor encoder and deterministic tokenization/chat templating; Yehia causal-LM inference runs through vLLM.