DRU-RE-Yehia

DRU-RE-Yehia is a reproducible 4-bit QLoRA system for directed Arabic relation extraction with Navid-AI/Yehia-7B-preview. It converts each relation decision into a row-specific Arabic multiple-choice problem and trains Yehia to return one Arabic option code. The model does not have to reproduce a long relation phrase or generate explanatory text.

This repository contains the clean training, dataset, audit, inference, and release-staging source, the selected best inference adapter, and the latest complete resumable Trainer checkpoint. It intentionally excludes downloaded base weights, local dataset JSONL files, all older numbered checkpoints, caches, notebooks, validation/working prediction dumps, and rolling logs. The reviewed official submission predictions are included because they are a release asset.

Release status: all three epochs and the final 1,687-row validation are complete at global step 2,943. The selected step-2,750 best adapter is staged at the repository root and passed the complete local release verifier: 448/448 tensors are finite and nonzero, all 41 worked classes are covered, and every manifest hash matches. The full latest step-2,943 optimizer/scheduler/RNG checkpoint is preserved separately under latest_checkpoint/. The released best adapter has also produced and validated the complete 2,074-row official submission.zip. Hugging Face publication provenance is written to release/hf_remote_verification.json by the guarded publisher.

At a glance

Item	Selected value
Task	Directed Arabic relation extraction
Direction	[الأول] → [الثاني]
Positive relations	40
Negative class	لا توجد علاقة / metadata label no_relation
Base model	Navid-AI/Yehia-7B-preview
Prompt version	yehia_re_ar_v3_one_token_choice
Output contract	One displayed Arabic option code
SFT dataset	U4RASD/Yehia-RE-SFT
Adapter repository	U4RASD/DRU-RE-Yehia
Quantization	4-bit NF4, double quantization, BF16 compute
PEFT targeting	target_modules="all-linear"
LoRA	rank 16, alpha 32, dropout 0.05, RS-LoRA
Training length	3 epochs / 2,943 optimizer steps on this dataset
Effective batch	16 = physical batch 4 × gradient accumulation 4
Best-checkpoint metric	Validation positive micro F1
Scheduled evaluation	All 1,687 validation rows every 250 optimizer steps
Official submission	2,074 rows; root ZIP member predictions.txt

Repository contents

DRU-RE-Yehia/
├── README.md
├── CHANGELOG.md                     # grounded selected-system changes
├── model.py                         # final QLoRA trainer and evaluator
├── predict.py                       # constrained one-token batch inference
├── create_submission.py             # strict official inference + Codabench ZIP
├── prepare_input.py                 # raw sentence/entity pair to model prompt
├── build_dataset.py                 # deterministic source-to-SFT builder
├── download_dataset.py              # exact transformed-revision downloader
├── audit_dataset.py                 # exhaustive local dataset audit
├── re_sft_common.py                 # prompt, markers, ontology, transformation
├── requirements.txt                 # validated pinned Python stack
├── setup_env.sh                     # environment and local self-test
├── train.sh                         # foreground training/resume
├── train_tmux.sh                    # detached training/resume
├── build_dataset.sh                 # build followed by independent audit
├── build_submission.sh              # pinned 2,074-row inference and packaging
├── submission.zip                   # ready-to-upload deterministic artifact
├── .env.example                     # safe configuration without credentials
├── .gitattributes                   # full-release model/checkpoint LFS patterns
├── .gitignore
├── resources/
│   ├── wojood_ontology.json
│   ├── entity_type_ar_mapper.json
│   ├── relation_mapper.json
│   ├── relation_template_ar_mapper.json
│   ├── relation_template_ar_reverse_mapper.json
│   └── resource_manifest.json
├── examples/
│   ├── example_record.json
│   ├── ALL_41_CASES.md              # full real prompt/output for every class
│   └── all_41_class_examples.jsonl  # same 41 examples, machine-readable
├── evidence/                        # small correctness evidence, no weights
│   ├── README.md
│   ├── dataset_summary.json
│   ├── dataset_remote_verification.json
│   ├── base_model_remote_verification.json
│   ├── selected_run_config.json
│   ├── all_linear_eligible_modules.json
│   ├── lora_wrapped_modules.json
│   ├── decision_token_inventory.json
│   ├── decision_token_masking_audit.json
│   ├── sampling_strategy.json
│   ├── smoke_test.json
│   └── validation_history.json
├── submission/
│   ├── README.md                    # exact competition contract and rebuild
│   ├── official_predictions.jsonl  # auditable row-local model decisions
│   ├── predictions.txt             # 2,074 tab-delimited competition lines
│   └── run_summary.json             # source/model/artifact hashes and counts
├── release/                         # final metrics, config, manifest, remote proof
├── latest_checkpoint/               # complete step-2,943 state; Hugging Face only
└── tools/
    ├── build_examples_appendix.py
    ├── plot_losses.py
    ├── monitor_resources.sh
    ├── stage_azure_release.py        # build the weight-free Azure mirror
    ├── verify_azure_release.py       # reject weights/checkpoints in Azure
    ├── restore_latest_checkpoint.py
    ├── stage_release.py
    ├── verify_release.py
    └── publish_huggingface.py

After a successful full release-staging operation, the Hugging Face model root additionally contains PEFT adapter/tokenizer files and inference_config.json; final run evidence is placed under release/, and the complete latest Trainer state is placed under latest_checkpoint/. The Azure copy intentionally omits the root adapter tensor and the entire checkpoint directory. Full Yehia base weights are never copied to either destination.

Task definition

Each example contains an Arabic sentence and two entities. The first entity is surrounded by:

[الأول] ... [/الأول]

The second entity is surrounded by:

[الثاني] ... [/الثاني]

The question is always directional: what relation holds from the first marked entity to the second marked entity? A reverse relation is not interchangeable. If no compatible displayed relation is expressed in that direction, the correct choice is لا توجد علاقة.

The model sees only:

• the marked Arabic sentence;
• the Arabic coarse type of the first entity;
• the Arabic coarse type of the second entity;
• directionally ontology-compatible Arabic relation options;
• لا توجد علاقة as the final option.

Canonical dataset labels such as PERS, ORG, employee_of, or Affiliation.employee_of remain metadata. They are never rendered inside the system, user, or assistant message.

Final prompt

Exact system message

أنت مصنّف علاقات. حدّد العلاقة الموجّهة من الكيان [الأول] إلى الكيان [الثاني].
اختر علاقة واحدة من القائمة تناسب معنى الجملة ونوعي الكيانين.
إن لم تنطبق أي علاقة فاختر «لا توجد علاقة».
أجب برمز الخيار فقط: حرف عربي واحد دون رقم أو شرح.

Exact user-message template

الجملة:
{الجملة العربية مع [الأول]...[/الأول] و[الثاني]...[/الثاني]}

نوع الأول: {النوع العربي للأول}
نوع الثاني: {النوع العربي للثاني}

الخيارات:
أ) {علاقة عربية متوافقة}
ب) {علاقة عربية متوافقة أخرى}
...
ي) لا توجد علاقة

رمز الإجابة:

The number and letters vary by row. The option-code inventory is:

أ، ب، ج، د، ه، و، ز، ح، ط، ي

All ten codes were verified to be exactly one token under Yehia's native tokenizer when preceded by the assistant-template space. A row uses only the prefix required by its number of displayed options.

Exact assistant contract

The assistant content is one code and nothing else, for example:

ز

The code is interpreted through that row's displayed list. It is not a global relation ID. Because لا توجد علاقة is always last, its code changes with the number of compatible positive options in the row.

A complete real transformed example is available in examples/example_record.json.

How 41 classes fit into ten option letters

No relation-class tokens were added to Yehia's vocabulary. The letters أ ب ج د ه و ز ح ط ي already exist in the native Yehia tokenizer and each was verified to occupy exactly one token in the assistant position. They are row-local addresses, not global class IDs.

Three arrays on every transformed row have the same length and order:

option_codes[i]
allowed_options_ar[i]
allowed_relation_full_labels[i]

The first array is shown to and emitted by the model. The second array is the Arabic meaning shown in the prompt. The third array is metadata used to recover the canonical dataset class. Index i ties them together.

For example, two different rows may contain:

Row	Letter أ means on that row	Gold output	Decoded class
train_5482	الأول هو قائد الثاني	أ	Administration.leader_of
train_908	الأول لغتها الرسمية الثاني	أ	GPE.official_language

There is no contradiction: the prompt itself defines the meaning of أ for that row. Likewise, the same relation can be assigned different letters on different rows because the compatible positive options are deterministically shuffled. The decoder must therefore retain each row's mapping until after the model makes its choice.

لا توجد علاقة is always the final displayed option, but it does not always have the same letter:

• if it is the only compatible option, its code is أ;
• after two positive options, its code is ج;
• after nine positive options, its code is ي.

This is why a downstream application must never interpret a bare letter without the originating row's option arrays.

Fully filled positive example

This is the exact real transformed row train_13110. The model receives the following system message:

أنت مصنّف علاقات. حدّد العلاقة الموجّهة من الكيان [الأول] إلى الكيان [الثاني].
اختر علاقة واحدة من القائمة تناسب معنى الجملة ونوعي الكيانين.
إن لم تنطبق أي علاقة فاختر «لا توجد علاقة».
أجب برمز الخيار فقط: حرف عربي واحد دون رقم أو شرح.

It then receives this exact user message:

الجملة:
[الأول] سوق [الثاني] مدينة يافا [/الثاني] [/الأول] .

نوع الأول: منشأة
نوع الثاني: كيان جيوسياسي

الخيارات:
أ) الأول يقع بالقرب من الثاني
ب) الأول يقع في الثاني
ج) لا توجد علاقة

رمز الإجابة:

The exact assistant target/output is:

ب

For this row only, ب is index 1, whose Arabic phrase is الأول يقع في الثاني and whose metadata class is Location.located_in.

Fully filled negative example

This is the exact real transformed row train_neg-62666-62670AwLOC. The system message is identical. Its exact user message is:

الجملة:
[الأول] جبال نابلس [/الأول] [الثاني] والكرمل [/الثاني] .

نوع الأول: موقع جغرافي
نوع الثاني: شخص

الخيارات:
أ) لا توجد علاقة

رمز الإجابة:

The exact assistant target/output is:

أ

Here أ decodes to no_relation because the negative choice is the only ontology-compatible option for that direction and type pair.

Full prompt/output coverage for every class

examples/ALL_41_CASES.md contains one real, fully filled system prompt, user prompt, assistant output, and row-local mapping table for all 40 positive relations plus no_relation. Its coverage index makes the examples easy to scan. The identical selected records are preserved in machine-readable form at examples/all_41_class_examples.jsonl.

Regenerate the appendix deterministically from the pinned transformed training split with:

python tools/build_examples_appendix.py

The generator requires all 15,686 training rows, validates every selected mapping and output, requires exactly 41 classes, and deliberately documents no_relation last.

How the prompt was selected

The original experiment asked Yehia to generate the complete Arabic relation phrase. Interactive testing exposed a distinction between semantic mistakes and surface-format mistakes: the base model often copied list punctuation, prefixed the answer with words such as “الخيار”, or produced an explanation around an otherwise recognizable choice.

Prompt development followed this sequence:

1. Exercise all relation types with the locally downloaded base model.
2. Remove the demonstration and redundant rules.
3. Simplify the system instruction while preserving direction, entity types, negative behavior, and the answer-only requirement.
4. Keep لا توجد علاقة visibly and structurally last.
5. Replace free-form phrase generation with a row-specific one-token code.
6. Score candidate prompts over the complete 1,687-row validation split with the same constrained decision rule used during training evaluation.

The selected explicit prompt achieved calibrated base-model positive micro F1 0.3616; the shorter compact alternative achieved 0.3563. The selected prompt was therefore slightly stronger and more explicit about direction and output format. There is no demonstration in the final prompt.

Scientific caveat

The validation split was used to choose the prompt and calibrate the no-relation decision bias. It is a development set, not an untouched test set. Validation-tuned numbers must not be presented as final held-out test performance. Official/test inference must freeze:

• the final prompt text;
• option construction and order;
• the selected adapter checkpoint;
• the selected no-relation bias;
• the constrained decoding rule.

No official labels are used during training or intermediate evaluation.

Final dataset

Source and transformed repositories

• Source: U4RASD/WojoodRelationsAnnotated
• Transformed SFT dataset: U4RASD/Yehia-RE-SFT
• Verified transformed revision: a060e47f56025778b97344d4d3de60a8fd53be7c

The transformed revision was independently downloaded after publication and matched local row counts, byte counts, and SHA-256 hashes.

Split roles and counts

Split	Rows	Positive	No relation	Role
train	15,686	9,009	6,677	QLoRA training only
validation	1,687	867	820	Prompt development and checkpoint evaluation only
official	2,074	Unlabeled	Unlabeled	Final inference only

Training never merges validation or official rows. Official examples contain system and user messages only; they have no assistant target or gold relation.

Verified transformed-file hashes

File	SHA-256
train.jsonl	69efbfccfc6e810b603b17e1b33c0f087590a4357ea52b49f4564f962cc1daad
validation.jsonl	c5aadb24df10626f047223885968de6404f54504d256414c1223bc8ca6619e11
official.jsonl	e6178b498411bdd4782aa13faad81d930700e09361263183fb01ef2bd4477e98

The corresponding source-file hashes are retained in evidence/dataset_summary.json.

How the dataset was built

build_dataset.py delegates all locked transformation behavior to re_sft_common.py. For each source row it performs the following operations:

1. Read without dropping rows. Load train, validation, and official source files separately and enforce their expected counts.
2. Validate exact spans. Confirm that subject_start:subject_end and object_start:object_end recover the stored mention strings exactly.
3. Insert markers by character offsets. Add first/second markers at the source positions rather than using string replacement. This remains correct when mention strings repeat.
4. Handle span edge cases deterministically. The insertion routine defines ordering for nested, overlapping, adjacent, and equal spans and then verifies exactly one opening and closing marker for each entity.
5. Window long context. Preserve both entities and retain up to 500 source characters on each side. The unmodified full sentence and original numeric spans stay in metadata; numeric offsets are not shown as prompt instructions.
6. Map entity types to Arabic. Translate the two coarse source types using the fixed 21-entry entity_type_ar_mapper.json.
7. Apply the directed ontology. Select only relations whose ontology domain contains the first entity type and whose range contains the second entity type.
8. Use packaged Arabic templates verbatim. Render each allowed positive relation with its packaged template_ar answer. Awkward or misspelled source templates are deliberately not corrected during transformation.
9. Verify reversibility. Require all 40 Arabic answers to be unique, remain unique after normalization, and map back to exactly one ontology relation.
10. Shuffle positive options deterministically. Derive a stable per-row seed from seed 42, split identity, and row identity, then shuffle only positive compatible choices.
11. Append the negative option. Add exactly one لا توجد علاقة after the positive shuffle. It is always the final displayed option.
12. Assign one-token codes. Label displayed options sequentially with the prefix of أ، ب، ج، د، ه، و، ز، ح، ط، ي.
13. Map the gold answer. On labeled splits, verify the gold relation is compatible and displayed exactly once, then store its displayed position, Arabic phrase, and one-token code.
14. Create conversations. Labeled rows receive system, user, and assistant messages. Official rows receive only system and user messages.
15. Audit and hash. Check IDs, counts, mappings, markers, options, answers, leakage, and split roles before writing a summary and optional Hub upload.

The number of displayed options ranges from 1 to 10 according to the directed entity-type pair. A one-option row contains only لا توجد علاقة.

Determinism boundaries

With identical source JSONL bytes, resource files, transformation code, seed 42, and context size 500, the transformed JSONL content is deterministic. dataset_summary.json contains a creation timestamp, so that metadata field can differ even when the three JSONL hashes are identical.

Dataset record schema

Each JSONL line is a self-contained object. Important fields are:

Field	Meaning
id	Stable transformed-row ID
sentence_id, triple_id	Source identifiers
sentence	Original unmodified sentence
subject, object	Original mention strings
subject_start/end	Original first-entity character span
object_start/end	Original second-entity character span
subject_type, object_type	Canonical metadata types, never rendered
first_type_ar, second_type_ar	Arabic types rendered in the prompt
marked_sentence	Context with directional markers
window_start_char, window_end_char	Selected source-context boundary
context_was_windowed	Whether context was shortened
allowed_relation_full_labels	Canonical labels in displayed order
allowed_relation_ontology_ids	Ontology IDs in displayed order
allowed_options_ar	Arabic phrases in displayed order
option_codes	One-token codes aligned with displayed options
option_records	Full reversible mapping records
gold_relation_full	Gold canonical relation on labeled splits
gold_relation_ontology_id	Gold ontology ID for a positive row
gold_answer_ar	Gold displayed Arabic phrase
gold_option_index	Zero-based displayed gold position
gold_answer_code	Actual one-token supervised target
prompt_messages	System and user messages only
messages	Prompt plus assistant code on labeled rows
prompt_version	yehia_re_ar_v3_one_token_choice

Canonical fields exist for evaluation and reversibility; the prompt-leakage audit examines only conversation content and confirms that these identifiers do not reach the model as text.

Packaged ontology and Arabic resources

The resources/ directory is part of the experiment definition:

File	Purpose
wojood_ontology.json	Coarse entity types and directed relation domain/range rules
entity_type_ar_mapper.json	Exact mapping for all 21 coarse types
relation_mapper.json	40 full labels plus no_relation mapping
relation_template_ar_mapper.json	Exact packaged Arabic answer for each positive relation
relation_template_ar_reverse_mapper.json	Arabic-answer-to-relation reverse mapping
resource_manifest.json	Resource provenance and integrity metadata

These files are not suggestions. Changing templates, type names, or ontology compatibility creates a different experiment and requires a new prompt/data version.

Dataset validation guarantees

The independent audit_dataset.py pass validates all 19,447 transformed rows:

• exact split counts and unique IDs;
• hashes against dataset_summary.json;
• exactly 21 entity-type mappings;
• exactly 40 positive relation templates;
• uniqueness and reversibility of all Arabic relation answers;
• 41 labeled classes including no_relation;
• exactly one instance of every directional marker;
• marked sentence presence inside the user message;
• ontology-compatible and unique choices;
• sequential one-token option-code structure;
• لا توجد علاقة present exactly once and last;
• exact agreement of gold full label, Arabic phrase, index, code, and assistant;
• no assistant/gold leakage in official rows;
• zero canonical type/relation identifier leakage in conversation content.

The audit writes validation_audit.json beside the downloaded/built dataset.

Training objective

This is not regex loss, sequence-wide language-model loss, or loss over the Arabic relation phrase. It is full-vocabulary causal-language-model cross-entropy on exactly one assistant token: the gold displayed option code.

For every encoded row:

Token region	Label
System message	-100
User message	-100
Chat-template prompt boundary	-100
Assistant decision code	Actual one-token ID
Any trailing template token	-100
Padding	-100

Conceptually, one encoded training example looks like this:

input_ids = [all native-chat-template prompt tokens, gold_letter, trailing tokens]
labels    = [-100, -100, ...                 , gold_letter, -100, ...]

Transformers' causal language-model implementation shifts the labels internally. Therefore, the logits at the position immediately before the gold letter are trained to predict that letter. If g is the token ID of the gold letter and z_v is the model logit for vocabulary token v, the one-row loss is exactly:

loss = -log(exp(z_g) / sum(exp(z_v) for every token v in Yehia's vocabulary))

Equivalently, it is -log(p_gold). Some useful intuition:

Probability assigned to the correct letter	One-token loss
0.90	0.105
0.70	0.357
0.50	0.693
0.10	2.303
0.01	4.605

The batch loss is the mean over the supervised letter positions in that batch. Because there is exactly one supervised token per row, a long prompt does not silently contribute more loss than a short prompt.

The mapping step is outside the differentiable loss:

1. During dataset construction, the gold canonical relation is located in that row's aligned option arrays.
2. Its row-local index selects the gold Arabic letter.
3. Only that existing letter token becomes a non--100 label.
4. Backpropagation updates LoRA parameters so Yehia assigns more probability to that letter in this context.
5. During inference, the predicted letter's row-local index maps back to the Arabic relation and canonical metadata class.

No regular expression changes the training labels or loss. No fuzzy phrase match is used in backpropagation. No 41-class classification head, 41 new tokens, or global letter-to-relation table exists. Regex/fuzzy parsing belonged only to abandoned free-form prompt experiments and is not part of the final trainer.

The trainer applies Yehia's native chat template exactly once to the prompt and once to the complete labeled conversation, finds their exact common boundary, and verifies that the first assistant token is the expected code. It rejects a row if the boundary differs, the sequence exceeds 1,024 tokens, or no decision token remains.

Full-vocabulary loss means the correct code must outrank every token in Yehia's vocabulary. A candidate-only softmax objective was evaluated during engineering, but it was not selected: the tested path was slower and did not directly penalize unrelated vocabulary tokens. The clean model.py contains only the selected full-vocabulary objective.

The saved masking evidence includes a positive example, a negative example, and a long prompt. In every audited row:

• prompt-token count equals masked-prompt-token count;
• supervised-token count is exactly one;
• the supervised token decodes to the gold option code;
• every non-decision token is masked.

See evidence/decision_token_masking_audit.json.

Only LoRA weights receive gradients. The 4-bit base weights, embeddings, and language-model output head remain frozen. Thus the model learns the task by changing low-rank adapter paths throughout the transformer while preserving the gated base checkpoint as an external dependency.

Final QLoRA configuration

The selected configuration in .env.example and model.py is:

Setting	Value
Base model	Navid-AI/Yehia-7B-preview
Exact base revision	b9dda4715eafee7e8090d2c83cfe078d75f4ebb8
Quantization	4-bit NF4
Double quantization	Enabled
Compute dtype	BF16
Model device map	Entire quantized model on GPU 0
Attention implementation	SDPA
Maximum sequence length	1,024
Gradient checkpointing	Enabled
Training use_cache	False
PEFT target	target_modules="all-linear"
LoRA rank	16
LoRA alpha	32
LoRA dropout	0.05
RS-LoRA	Enabled
LoRA bias	None
PEFT task type	CAUSAL_LM
Physical train batch	4
Gradient accumulation	4
Effective batch	16
Validation batch	16
Epochs	3
Optimizer steps	2,943 on the exact training split
Learning rate	5e-5
Weight decay	0.0
Warmup	5%
Scheduler	Cosine
Optimizer	paged_adamw_8bit
Gradient clipping	1.0
Train logging	Every 20 optimizer steps
Evaluation/save	Every 250 optimizer steps
Numbered checkpoint retention	2
Random seed	42
Sampling	Inverse class frequency, exponent 0.25

Rank 16/alpha 32 is the selected run, not the package's earlier rank 64/alpha 128 proposal. Pilot evidence showed strong learning with the smaller adapter, reducing trainable parameters and release size while retaining the required all-linear coverage.

Class sampling

Training uses a deterministic WeightedRandomSampler with replacement. For class count n_c, the relative row weight is proportional to:

n_c ^ -0.25

This softens severe class imbalance without forcing every class to be equally common. The exact class weights are stored in evidence/sampling_strategy.json.

True all-linear PEFT coverage

target_modules="all-linear" is passed directly to LoraConfig. The trainer does not infer success from the presence of seven familiar suffixes.

PEFT resolves the sentinel against the concrete architecture and may serialize the resulting suffix set (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) in adapter_config.json. That serialized expansion is not the evidence for the claim. The immutable run configuration records the original request as the literal string "all-linear", and the before/after full-name inventories below prove actual coverage of every supported internal module while excluding only the output head.

Before PEFT attachment it inventories every supported torch.nn.Linear, Transformers Conv1D, and compatible bitsandbytes linear module by full name and class. It separately identifies embeddings and output heads. After attachment it inventories every LoRA module and compares the full internal set.

For Yehia 7B the selected run recorded:

Audit item	Result
Eligible supported linear/head modules	225
LoRA-wrapped internal modules	224
Transformer layers	32
Wrapped modules per layer	7
Wrapped q_proj	32
Wrapped k_proj	32
Wrapped v_proj	32
Wrapped o_proj	32
Wrapped gate_proj	32
Wrapped up_proj	32
Wrapped down_proj	32
Output lm_head adapted	No
Embeddings adapted	No
Quantized base weights trainable	No
Trainable parameters	39,976,960
Trainable percentage	0.567811%

The one eligible-but-unwrapped module is the language-model output head, which PEFT intentionally excludes under all-linear targeting. Only LoRA parameters are trainable.

Full inventories are preserved in:

• evidence/all_linear_eligible_modules.json
• evidence/lora_wrapped_modules.json

L40S memory and finite-gradient smoke test

The selected physical batch was tested after attaching all-linear adapters with representative long examples, 1,024-token maximum length, 4-bit NF4, double quantization, BF16, and gradient checkpointing.

The recorded physical-batch-4 forward/backward smoke test produced:

Item	Result
Status	Passed
Loss	0.619789, finite
Physical batch	4
Gradient accumulation	4
Finite nonzero LoRA gradient tensors	224
Peak allocated VRAM	9.044 GiB
Peak reserved VRAM	10.172 GiB

The complete active process uses additional memory for model state, optimizer, evaluation, and allocator caching; the live L40S run remained well below the approximately 45 GB device capacity. See evidence/smoke_test.json.

On a smaller GPU, keep effective batch 16 by trying physical batch 2 with gradient accumulation 8. Do not silently shorten sequences, remove all-linear targeting, or alter the prompt to solve an OOM; those changes define a different experiment.

Evaluation and checkpoint selection

At steps 0, 250, 500, 750, ..., evaluation processes every one of the 1,687 validation examples. “Step” means optimizer/global update after gradient accumulation, not micro-batch count.

For each validation row, the evaluator:

1. Applies Yehia's native chat template to prompt_messages exactly once.
2. Obtains the logits predicting the next assistant token.
3. Selects logits only for option codes actually displayed in that row.
4. Optionally adds one development-calibrated bias to the final no-relation option.
5. Chooses the highest-scoring code.
6. Maps its displayed position back to the Arabic option and canonical label.

Every metric artifact records:

• positive micro precision, recall, and F1;
• 41-class accuracy;
• 41-class macro F1;
• weighted F1;
• decision negative log-likelihood;
• invalid count and rate;
• evaluated row count;
• raw constrained metrics;
• unconstrained full-vocabulary argmax metrics;
• selected no-relation bias;
• prediction artifact path.

Constrained versus unconstrained output

The production decoder is constrained to codes displayed in the current row, so its structural invalid-output count is zero by construction. This does not prove that the unconstrained model would always emit a valid code. The evaluator therefore also records the full-vocabulary next-token argmax:

• invalid_outputs: constrained decoder diagnostic;
• unconstrained_invalid_outputs: honest standalone formatting diagnostic.

After step 250, the selected run's unconstrained invalid count was also zero in the evaluations available at packaging time.

No-relation calibration

The released decoder applies one global additive bias to the score of the final لا توجد علاقة (no_relation) option before taking argmax. The selected best checkpoint uses:

no_relation_logit_bias = -6.6600000000000215

For a row with valid option logits z, inference is equivalent to:

adjusted_logits = z.copy()
adjusted_logits[no_relation_index] += -6.66
predicted_index = argmax(adjusted_logits)

All positive-relation logits remain unchanged. If z_no_relation is the raw no-relation logit and z_best_positive is the highest raw positive-relation logit, the released decoder selects no-relation only when:

z_no_relation - z_best_positive > 6.66

Without calibration (bias = 0), no-relation wins whenever its logit exceeds the best positive logit by any amount. The bias is therefore another expression of a decision-threshold adjustment: Yehia still uses argmax, but only after the no-relation score has been shifted.

The value -6.66 is not a subtraction of 6.66 percentage points. Logits are unbounded scores, not probabilities. If the adjusted valid-option logits are passed through softmax, this shift multiplies the odds of no-relation relative to the positive options by:

exp(-6.66) ~= 0.00128

This is approximately a 781-fold reduction in the no-relation odds. In a two-option illustration, no-relation would need a raw probability above about 99.872% to remain the winner after this shift. There is no single equivalent probability threshold for the real decoder because a row can contain several positive options with different logits. The magnitude is nevertheless a strong correction toward predicting positive relations.

The correction was useful because the selected adapter's raw constrained decision rule was conservative on development data. Its effect on the complete 1,687-row labeled validation split was:

Decision rule	Positive predictions	TP	FP	FN	Positive precision	Positive recall	Positive micro F1
Raw constrained argmax (bias = 0)	779	490	289	377	0.6290	0.5652	0.5954
Released calibrated argmax (bias = -6.66)	1,009	594	415	273	0.5887	0.6851	0.6333

Calibration recovered 104 additional true positives while adding 126 false positives. Precision decreased, recall increased, and positive micro F1 improved. The calibrated system still predicted no-relation for 678 of the 1,687 validation rows; the correction does not force every row to be positive.

The evaluator selects the bias on validation by maximizing positive micro F1, with accuracy and then smaller absolute bias used as tie-breakers. This process does not retrain or alter the adapter weights. Setting the bias to zero is sufficient to recover raw constrained decoding; no retraining is required.

Because the bias was selected using validation labels, all calibrated validation scores are development metrics rather than untouched held-out results. No official/test labels were used to select it. tools/stage_release.py freezes the selected value in inference_config.json, and official/test inference must reuse that value without further tuning. Selecting or changing the bias using hidden test labels would invalidate the held-out evaluation.

Best checkpoint

The locked checkpoint-selection metric is generation/decision-based validation positive micro F1. It is not training loss or teacher-forced validation loss. The trainer maintains:

• checkpoints/checkpoint-<step>/: resumable Trainer checkpoints;
• latest_checkpoint/: complete named copy of the final step-2,943 checkpoint;
• best_checkpoint/: complete named copy of the best validation checkpoint;
• best_adapter/: adapter/tokenizer exported after training completes.

Numbered checkpoint rotation never deletes the named latest/best copies.

Complete scheduled development metrics

These values cover every scheduled full validation from the completed selected run. They are engineering/development evidence, not final official/test claims.

Step	Positive P	Positive R	Positive F1	Accuracy	Macro F1	Weighted F1	Decision NLL	Unconstrained F1	Unconstrained invalid
0	0.2793	0.5225	0.3640	0.2964	0.5420	0.2423	2.3701	0.3295	2
250	0.5674	0.6021	0.5842	0.6586	0.6391	0.6477	1.3317	0.5754	0
500	0.5785	0.6159	0.5966	0.6823	0.6691	0.6590	1.3186	0.5944	0
750	0.5913	0.5825	0.5869	0.6609	0.6405	0.6461	1.5595	0.5820	0
1000	0.5560	0.6471	0.5981	0.6598	0.6842	0.6458	1.9229	0.5758	0
1250	0.5356	0.6586	0.5908	0.6343	0.6699	0.6300	2.0380	0.5904	0
1500	0.6006	0.6678	0.6324	0.7030	0.6625	0.6849	2.2494	0.5807	0
1750	0.6045	0.6205	0.6124	0.6882	0.6429	0.6687	2.1202	0.5940	0
2000	0.6121	0.6298	0.6208	0.7012	0.6462	0.6814	2.2389	0.5944	0
2250	0.5824	0.6851	0.6296	0.6935	0.6849	0.6790	2.1502	0.5938	0
2500	0.5902	0.6678	0.6266	0.6924	0.6831	0.6767	2.3231	0.5999	0
2750	0.5887	0.6851	0.6333	0.6971	0.6889	0.6828	2.2822	0.5954	0

Step 2,750 is the selected best checkpoint by the locked positive-micro-F1 metric. Step 1,500 retained the strongest accuracy and weighted F1, step 2,750 the strongest macro F1, step 500 the best decision NLL, and step 2,500 the best unconstrained positive F1. This divergence is why complete scheduled evaluation and durable best-checkpoint preservation matter even when training loss falls.

After step 2,943, the trainer reloaded the selected step-2,750 adapter and ran the complete final validation again. It reproduced positive precision 0.588702, recall 0.685121, F1 0.633262, accuracy 0.697095, macro F1 0.688899, weighted F1 0.682849, and zero unconstrained invalid outputs. Aggregate three-epoch train loss was 0.218184; this is not the checkpoint-selection metric.

The machine-readable snapshot is evidence/validation_history.json.

Hardware and software assumptions

The selected run was validated on:

• NVIDIA L40S with approximately 45 GB VRAM;
• CUDA 12.4 reported by the RunPod environment;
• BF16-capable hardware;
• Python 3.12;
• the pinned stack in requirements.txt.

The core pinned versions are:

Package	Version
PyTorch	2.5.1
Transformers	4.57.1
Tokenizers	0.22.1
PEFT	0.17.1
bitsandbytes	0.45.5
Accelerate	1.4.0
datasets support	Local JSONL; no datasets runtime dependency required
huggingface_hub	0.36.0
scikit-learn	1.5.2

Use the pinned stack first. If a different GPU/driver requires a compatibility change, record the exact failure and smallest changed dependency before claiming reproducibility.

Credentials and access boundaries

Copy .env.example to .env and provide tokens locally. Never commit .env.

Token responsibilities are deliberately separate:

• HF_TOKENONE: gated read access to Navid-AI/Yehia-7B-preview only;
• HF_TOKENTWO: U4RASD dataset access and optional U4RASD dataset/adapter publication.

No script prints token values. The clean repository contains token variable names only. The preferred workflow downloads Yehia once into models/Yehia-7B-preview and performs all subsequent model loads locally.

Quick start: reproduce training from the published SFT dataset

1. Configure

git clone <this-repository-url> DRU-RE-Yehia
cd DRU-RE-Yehia
cp .env.example .env

Edit .env and set HF_TOKENONE and HF_TOKENTWO. Keep the exact dataset revision, exact Yehia base revision, and selected hyperparameters unchanged for a faithful reproduction. The selected base commit is b9dda4715eafee7e8090d2c83cfe078d75f4ebb8; future downloads do not silently follow a moving main branch.

2. Build the environment

chmod +x setup_env.sh train.sh train_tmux.sh build_dataset.sh
./setup_env.sh
source .venv/bin/activate

The setup script installs the pinned stack, compiles every final Python entry point, checks Bash syntax, imports the QLoRA stack, reports CUDA/BF16 status, and runs the local prompt/resource self-test.

3. Download and verify the transformed dataset

python download_dataset.py
python audit_dataset.py

download_dataset.py pins the verified dataset commit from .env, downloads only dataset/resource/card files, and checks all three row counts and hashes. audit_dataset.py then performs the exhaustive content audit.

4. Start detached training

./train_tmux.sh

Attach and detach with:

tmux attach -t dru_re_yehia_training
# detach with Ctrl+b, then d

To run in the foreground instead:

./train.sh

On the first run, model.py uses an existing local Yehia snapshot if models/Yehia-7B-preview/config.json exists. Otherwise it downloads the gated base repository at the pinned revision with HF_TOKENONE into that directory. Training and later inference load only from the local snapshot.

Alternative: rebuild the SFT dataset from source

The already-published transformed revision is the simplest way to reproduce training. To reproduce the transformation itself, either:

• set SOURCE_DATASET_DIR to a local directory containing the three exact source filenames; or
• leave it blank and grant HF_TOKENTWO read access to U4RASD/WojoodRelationsAnnotated.

Keep these defaults:

PUSH_SFT_DATASET=false
OVERWRITE_DATASET_OUTPUT=false
SEED=42
PROMPT_CONTEXT_CHARS=500
SHUFFLE_OPTIONS=true

Then run:

./build_dataset.sh

The launcher runs the builder and then the independent audit. If the output directory already exists, the builder stops rather than deleting it. Set OVERWRITE_DATASET_OUTPUT=true only when replacement is intentional.

To publish an independently verified rebuild, explicitly set PUSH_SFT_DATASET=true. Publishing is never the default.

Monitoring

Trainer log

tail -f logs/train_tmux.log

The internal Python log is normally logs/train.log.

GPU

watch -n 1 nvidia-smi

Resource time series

./tools/monitor_resources.sh

The default CSV is written inside the ignored run directory at runs/DRU-RE-Yehia/artifacts/resource_monitor.csv.

Loss plots

Render once:

python tools/plot_losses.py --once

Continuously refresh every 20 seconds:

python tools/plot_losses.py

The generated PNGs are placed under runs/DRU-RE-Yehia/plots/ and are ignored by Git.

Resume behavior

CHOICE_RESUME=true is the selected default. When training starts, the trainer uses Transformers' get_last_checkpoint in runs/DRU-RE-Yehia/checkpoints/. Restarting train.sh or train_tmux.sh therefore resumes the newest numbered checkpoint, including optimizer, scheduler, RNG, and global-step state.

Before resuming, do not change:

• dataset JSONL content or order;
• prompt version;
• tokenizer/base model;
• LoRA rank, alpha, targets, or RS-LoRA setting;
• physical batch or gradient accumulation;
• sampling exponent;
• learning-rate schedule;
• maximum sequence length.

Changing those values while reusing optimizer state invalidates a strict reproduction. Start a new output directory for a new experiment.

Generated run layout

Training writes only to ignored local directories:

runs/DRU-RE-Yehia/
├── run_config.json
├── run_summary.json                    # written after full completion
├── artifacts/
│   ├── all_linear_eligible_modules.json
│   ├── lora_wrapped_modules.json
│   ├── decision_token_inventory.json
│   ├── decision_token_masking_audit.json
│   ├── sampling_strategy.json
│   ├── smoke_test.json
│   └── train_class_counts.json
├── checkpoints/
│   └── checkpoint-<step>/
├── best_checkpoint/
├── latest_checkpoint/
├── best_adapter/                       # written after completion
├── metrics/
│   └── validation_choice_metrics_step_<step>.json
├── predictions/
│   └── validation_choice_step_<step>.jsonl
└── plots/
    ├── train_loss.png
    └── eval_loss.png

The full 7B base model is never written inside the run or adapter repository.

Stage the clean model release

After all three epochs and final validation finish, run:

python tools/stage_release.py

The staging utility refuses to proceed unless:

• run_summary.json exists and declares status=complete;
• global step is exactly 2,943 and final validation contains all 1,687 rows;
• scheduled metric artifacts exist at step 0 and every 250 steps through 2,750;
• best_adapter/ contains adapter config and safetensors;
• persisted run configuration states target_modules="all-linear";
• rank, alpha, dropout, RS-LoRA, bias, and task type match the selected setup;
• all-linear evidence reports 225 eligible and 224 wrapped modules with only LoRA parameters trainable and the output head frozen;
• all 448 saved adapter tensors are finite and nonzero;
• the best-checkpoint metric artifact exists;
• latest_checkpoint/ and numbered checkpoint-2943/ are byte-identical;
• the latest checkpoint has finite adapter/optimizer state, complete scheduler state, trainer global step 2,943, RNG state, and training arguments.

It copies only:

• adapter safetensors and PEFT configuration;
• tokenizer/chat-template files required by the adapter;
• frozen inference configuration, including best-step no-relation bias;
• final run summary and best validation metrics;
• small audit evidence and a SHA-256 release manifest;
• the complete latest step-2,943 resumable checkpoint, with a separate checkpoint integrity manifest.

All correctness checks finish before repository files are changed. Each staged file is copied through a temporary sibling and atomically renamed, preventing an interrupted copy from appearing as a complete release artifact.

It explicitly excludes:

• Yehia base weights;
• all older numbered/named checkpoints other than the staged final latest copy;
• logs and resource time series;
• per-row validation prediction dumps;
• caches and temporary files.

The full repository is then ready for review and Hugging Face upload. Publish the trained adapter and checkpoint only with the supplied Hugging Face client. Authenticate with the U4RASD token without putting its value in a command history or repository file. Build Azure's separate weight-free copy with tools/stage_azure_release.py.

Staging also replaces PEFT's machine-local base_model_name_or_path=/root/workspace/... metadata with the portable base repository ID and exact commit. This does not alter adapter tensors; it prevents a fresh machine from trying to resolve a path that existed only on the training pod.

Durable release, Hugging Face, Azure, and pod deletion

What must live where

The deletion-safe release is intentionally split across durable repositories:

Asset	Durable location	Why it is needed
Exact transformed train/validation/official JSONL and resources	Hugging Face dataset U4RASD/Yehia-RE-SFT at commit a060e47f56025778b97344d4d3de60a8fd53be7c	Retraining and official inference input
Source, resources, documentation, 41 examples, evidence, release tools, and submission artifacts	Azure Git and Hugging Face model U4RASD/DRU-RE-Yehia	Reproduce, audit, train, and understand the system
Best adapter, tokenizer files, inference_config.json, final metrics, and release manifest	Hugging Face model repository only	Actual model inference and checkpoint provenance
Latest complete checkpoint-2943 adapter, optimizer, scheduler, RNG, trainer state, tokenizer, and training arguments	latest_checkpoint/ in the Hugging Face model repository only	Exact archival resume state from the completed run
Ready official prediction archive plus auditable row-level decisions	submission.zip and submission/ in both model and Azure repositories	Submit directly or verify/regenerate all 2,074 predictions
Yehia 7B base at commit b9dda4715eafee7e8090d2c83cfe078d75f4ebb8	Upstream gated Navid-AI/Yehia-7B-preview	Required underneath the PEFT adapter
Token permissions	An external password manager or secret store, never either repository	Future access to gated/private repositories

The Hugging Face adapter is not a standalone 7B model. It is deliberately about 160 MB rather than a copy of the full base weights. Future inference requires continued authorized access to the pinned gated Yehia base. If the upstream base is removed or access is revoked, the adapter alone cannot reconstruct it. Do not mirror the base into Azure or the model repository unless its license and owner explicitly permit redistribution.

Local release verification

After staging the full Hugging Face release, run:

python tools/verify_release.py --repo-dir .

The verifier fails if any required source/resource/example/release file is missing. It also checks:

• 41 unique fully filled worked examples and correct row-local outputs;
• secret-free evidence that the gated base resolved to the exact pinned commit;
• the exact transformed-dataset commit, counts, and local/remote hashes;
• exact 225 eligible / 224 wrapped all-linear evidence;
• the portable pinned base ID and revision in adapter/inference configs;
• 448 finite, nonzero adapter tensors;
• the complete step-2,943 checkpoint manifest, adapter, finite optimizer state, final scheduler state, RNG payload, and trainer provenance;
• locked rank 16, alpha 32, dropout 0.05, RS-LoRA settings;
• the SHA-256 and byte size of every staged manifest entry;
• exactly 2,074 official predictions, unique IDs in the pinned source order, legal relation names, row/debug agreement, deterministic ZIP metadata, and a one-member submission.zip whose CRC and recorded hashes pass;
• absence of full base weights and any unrecognized checkpoint payloads;
• in remote-clean mode, absence of every unrecognized or stale remote file.

This command intentionally fails before the final adapter is staged. A source folder by itself is reproducible code, not a finished model release. The weight-free Azure package has a different intentional contract and must pass:

python tools/verify_azure_release.py --repo-dir . --remote-clean

That verifier fails if it finds any safetensors file, Trainer checkpoint, adapter chunk, model binary, cache, credential file, or unexpected payload.

Publish and independently re-download Hugging Face

After local verification passes and .env contains HF_TOKENTWO, run:

python tools/publish_huggingface.py

That publisher is hard-locked to U4RASD/DRU-RE-Yehia. It:

1. runs the complete local release verifier;
2. uploads the entire clean folder while excluding .env, caches, local data, models, runs, logs, and the disposable predictions/ work directory (the reviewed submission/ artifacts and root submission.zip are included);
3. resolves the immutable uploaded model commit;
4. downloads that commit into a fresh temporary directory;
5. reruns the verifier in remote-clean mode;
6. uploads a secret-free release/hf_remote_verification.json record;
7. downloads and verifies the final remote commit once more.

The printed final_revision is the immutable Hugging Face model commit to keep with the experiment record. No token value is printed or persisted.

Push the weight-free reproducibility folder to Azure Git

The durable Azure target for this release is branch add-dru-re-entitypair-twohead of KG_eval-2026, under the new directory DRU-RE-Yehia/. Azure is deliberately the code, documentation, evidence, and submission mirror. It must contain no adapter weights, split adapter chunks, optimizer state, or Trainer checkpoint. The best adapter and full latest checkpoint are stored only in U4RASD/DRU-RE-Yehia on Hugging Face.

Stage the Azure-specific copy from the already verified full release:

git clone --branch add-dru-re-entitypair-twohead --single-branch \
  https://dev.azure.com/AC-DU/CodeBox/_git/KG_eval-2026 KG_eval-2026
python /path/to/DRU-RE-Yehia/tools/stage_azure_release.py \
  --source /path/to/DRU-RE-Yehia \
  --destination KG_eval-2026/DRU-RE-Yehia \
  --replace
git -C KG_eval-2026 add DRU-RE-Yehia
git -C KG_eval-2026 commit -m 'Publish weight-free DRU-RE-Yehia reproducibility release'
git -C KG_eval-2026 push origin add-dru-re-entitypair-twohead

Authenticate through a credential manager or masked prompt. Never embed an Azure token in the remote URL, shell history, README, or Git files. If the Azure repository already has history, integrate it normally instead of overwriting it.

Azure DevOps Git is a durable source/model mirror, not automatically an Azure Machine Learning online endpoint. Deployment to Azure ML would additionally need an approved compute/environment image, scoring entrypoint, and endpoint configuration; those are deployment choices, not retraining prerequisites.

Fresh Azure checkout test

Before pod deletion, verify that Azure has all reproducibility and submission files and zero model payloads. Use an environment outside the checkout so --remote-clean examines a pristine repository:

git clone --branch add-dru-re-entitypair-twohead --single-branch \
  https://dev.azure.com/AC-DU/CodeBox/_git/KG_eval-2026 \
  /tmp/KG_eval-2026-azure-check
python3 -m venv /tmp/dru-release-verifier
/tmp/dru-release-verifier/bin/pip install -r \
  /tmp/KG_eval-2026-azure-check/DRU-RE-Yehia/requirements.txt
/tmp/dru-release-verifier/bin/python \
  /tmp/KG_eval-2026-azure-check/DRU-RE-Yehia/tools/verify_azure_release.py \
  --repo-dir /tmp/KG_eval-2026-azure-check/DRU-RE-Yehia \
  --remote-clean
git -C /tmp/KG_eval-2026-azure-check rev-parse HEAD

The Azure verifier rejects .safetensors, .pt, .pth, .bin, split-part files, latest_checkpoint/, azure_large_files/, .env, and unrecognized files. It still validates the complete 2,074-row submission, prompt/examples, dataset/base provenance, all-linear evidence, final metrics, and immutable Hugging Face publication pointer.

Exact resume versus retraining

The two durable repositories together are sufficient to:

• understand and audit the experiment;
• reproduce the transformed dataset;
• retrain from optimizer step 0;
• load the completed best adapter for inference;
• reproduce constrained decoding and output mapping;
• restore the exact latest completed Trainer state from step 2,943.

In Hugging Face, root adapter_model.safetensors is the selected step-2,750 best inference adapter. Hugging Face's latest_checkpoint/adapter_model.safetensors is the different final step-2,943 training state accompanied by its optimizer, scheduler, RNG, trainer state, and training arguments. Applications should use the root best adapter; the checkpoint directory exists for archival resume/recovery. Neither file nor any lossless encoding of its bytes belongs in Azure.

To restore it into the default run layout:

python tools/restore_latest_checkpoint.py

This verifies every checkpoint hash and tensor first, then atomically creates:

runs/DRU-RE-Yehia/checkpoints/checkpoint-2943/

With the locked three-epoch configuration, that checkpoint is already at the training endpoint, so a normal resume performs no additional optimizer update and proceeds to finalization. Extending it to more epochs or changing the learning-rate schedule is a new experiment, not an exact continuation of the documented run. Older intermediate checkpoints are intentionally excluded.

Deletion gate

The training machine is safe to delete only after every item below is true:

• Three-epoch training and final 1,687-row validation completed normally.
• run_summary.json says status=complete.
• tools/stage_release.py staged the best—not merely latest—adapter.
• latest_checkpoint/checkpoint_manifest.json proves the complete final step-2,943 optimizer/scheduler/RNG checkpoint is present.
• Local tools/verify_release.py passed.
• submission.zip contains only predictions.txt; all 2,074 lines, IDs, class names, source order, hashes, and CRC pass the release verifier.
• tools/publish_huggingface.py passed its fresh final-download check.
• The Hugging Face model final_revision was recorded.
• The pinned SFT dataset commit remains remotely readable and hash-verified.
• The Azure folder contains source, documentation, evidence, and the submission, but no adapter, adapter chunks, optimizer, or checkpoint bytes.
• A fresh Azure clone passed tools/verify_azure_release.py --remote-clean and reported trained_payloads_in_azure=0.
• The best adapter and complete latest checkpoint were independently downloaded and verified from Hugging Face only.
• At least one end-to-end inference smoke test passed from a fresh checkout using the gated pinned Yehia base, prepared input, and staged adapter.
• Azure Git commit, Hugging Face model commit, Hugging Face dataset commit, and Yehia base commit were recorded together.
• Required future repository credentials/permissions exist outside the pod.

Until that entire gate passes, deleting the pod is premature. After it passes, temporary caches, plots, rolling logs, downloaded base files, transformed local copies, and live checkpoints on this machine are disposable.

Inference

The released system has two inference stages:

raw sentence + two mentions + two coarse types
                 │
                 ▼
          prepare_input.py
  markers + compatible Arabic options + row-local codes + exact prompt
                 │
                 ▼
              predict.py
  next-token code scores + frozen no-relation bias + row-local decoding
                 │
                 ▼
      Arabic answer + canonical relation metadata

The model is a relation classifier, not a named-entity recognizer. The caller must supply the two entity mentions, their direction, and their coarse types.

Predict the published official split

predict.py accepts already transformed rows containing prompt_messages, aligned Arabic options/codes, and canonical mappings. After release staging:

python predict.py \
  --input data/Yehia-RE-SFT/official.jsonl \
  --output predictions/official_predictions.jsonl

For a short smoke test:

python predict.py --limit 8

Build the ready Codabench submission

The reference one-command path is:

bash build_submission.sh

If the exact pinned transformed dataset is absent, the script downloads it from U4RASD/Yehia-RE-SFT at commit a060e47f56025778b97344d4d3de60a8fd53be7c. predict.py likewise downloads the gated Yehia base at commit b9dda4715eafee7e8090d2c83cfe078d75f4ebb8 when its verified local snapshot is absent. It then loads the root best adapter, runs all 2,074 unlabeled official rows, and produces:

submission.zip
submission/predictions.txt
submission/official_predictions.jsonl
submission/run_summary.json

The builder refuses a different source hash/count/order, any official gold label, a missing or duplicate triple_id, malformed row-local options, an illegal prediction, an inconsistent no-relation bias, or a malformed ZIP. submission.zip contains only root predictions.txt; each line is:

triple_id<TAB>canonical_relation

The dataset/model use no_relation internally. Only the competition payload converts that value to the required no-relation. The current artifact has SHA-256 a6bbd74ff4e32a570ac3fc1c12ce66aa3fe0c47986e66a3906a36231f3cbb0d5, contains 904 negative and 1,170 positive predictions, and preserves the exact official ID order. Those counts describe an unlabeled prediction distribution; they are not an official-set accuracy claim.

For an audit-only deterministic repackage of an already generated model dump:

python create_submission.py \
  --predictions-jsonl submission/official_predictions.jsonl

Prepare and predict one new raw sentence

First inspect the 21 accepted canonical coarse type names:

python prepare_input.py --list-types

Then prepare a row. When each mention occurs exactly once, character spans can be inferred safely:

python prepare_input.py \
  --id demo_1 \
  --sentence 'يعمل أحمد في جامعة بيرزيت.' \
  --subject 'أحمد' \
  --object 'جامعة بيرزيت' \
  --subject-type PERS \
  --object-type ORG \
  --output predictions/prepared_inputs.jsonl

The subject is always [الأول], the object is always [الثاني], and the question is subject-to-object. To ask the reverse question, prepare a separate row with subject/object and their types reversed.

If a mention string occurs more than once, prepare_input.py deliberately stops instead of guessing. Supply zero-based, end-exclusive Unicode character offsets:

python prepare_input.py \
  --sentence '...' \
  --subject '...' --subject-start 12 --subject-end 18 \
  --object '...' --object-start 31 --object-end 39 \
  --subject-type PERS --object-type ORG

Finally run the adapter on the prepared row:

python predict.py \
  --input predictions/prepared_inputs.jsonl \
  --output predictions/demo_predictions.jsonl

prepare_input.py uses the same marker insertion, 21-type Arabic mapper, directed ontology filter, 40 packaged Arabic templates, deterministic seed-42 shuffle, 500-character context logic, exact v3 prompt, and final لا توجد علاقة rule as dataset construction. It forces any accidental input relation field to empty, so gold labels cannot leak into inference prompts.

Prepare a batch

Pass --input a source-shaped JSONL file. One minimal line looks like:

{"id":"request_1","sentence":"يعمل أحمد في جامعة بيرزيت.","subject":"أحمد","object":"جامعة بيرزيت","subject_start":5,"subject_end":9,"object_start":13,"object_end":25,"subject_type":"PERS","object_type":"ORG"}

Exact spans are strongly recommended in production. The input need not include relation labels, prompt text, option lists, Arabic type names, or entity markers; the preparation utility derives them from the packaged resources.

What predict.py computes

The script:

1. downloads the pinned Yehia base revision if it is not already local;
2. loads the staged PEFT adapter;
3. applies the native Yehia chat template exactly once;
4. obtains the next-token logits after رمز الإجابة:;
5. keeps the logits for only the valid letters displayed on that row;
6. adds the frozen best-checkpoint no-relation bias to the final letter;
7. takes argmax and maps that row-local index back to the Arabic phrase, canonical relation, and ontology ID.

It does not ask the model to freely generate a phrase. Consequently, every returned code is structurally valid for its row. The saved unconstrained generation metrics remain in the evaluation evidence as an honest diagnostic of whether the model also follows the output format without this guardrail.

Each output line contains fields such as:

{
  "id": "official_example_id",
  "predicted_option_index": 2,
  "predicted_code": "ج",
  "predicted_option_ar": "الأول يعمل لدى الثاني",
  "predicted_relation_full": "Affiliation.employee_of",
  "predicted_relation_ontology_id": "employee_of",
  "no_relation_logit_bias": -6.6600000000000215
}

The shown bias is the frozen value selected on the labeled development split for the released best checkpoint. It is reused unchanged on official rows.

Do not call a generic text-generation pipeline on the bare adapter and then interpret أ as a permanent class. A correct reimplementation must preserve the row's option order, score only its valid code tokens, apply the frozen no-relation bias, and decode through the same index. The supplied prepare_input.py and predict.py are the reference implementation.

What is intentionally not in the full Hugging Face release

• Real Hugging Face tokens or .env.
• The gated 7B base model.
• The 133+ MB transformed JSONL dataset.
• Any source-dataset cache.
• Intermediate checkpoints and optimizer states other than the final staged latest_checkpoint/ copy.
• Notebook prompt experiments.
• Temporary benchmark scripts and abandoned objectives.
• Rolling trainer/resource logs.
• Full per-row validation predictions.

The official per-row predictions are intentionally included under submission/ because they audit the submitted ZIP; unlabeled official rows are not validation labels or training data.

• A premature adapter snapshot from an incomplete run.

The Azure mirror is intentionally narrower: in addition to everything listed above, it omits the trained root adapter and the entire latest_checkpoint/ directory. Their hashes and provenance remain in its evidence, while their actual bytes exist only in the Hugging Face model repository.

This keeps the model repository understandable while preserving every file needed to download, audit, rebuild, retrain from step 0, evaluate, stage, use the selected system, and restore the final optimizer state.

Troubleshooting

Gated Yehia download returns 401/403

Confirm that HF_TOKENONE belongs to an account with access to Navid-AI/Yehia-7B-preview. Do not substitute HF_TOKENTWO; the roles are separate.

Model is missing while offline mode is enabled

If HF_HUB_OFFLINE=1, Yehia must already exist at LOCAL_YEHIA_MODEL_DIR. Disable offline mode for the first download, then re-enable it for subsequent local-only runs.

Transformed dataset is missing

Run:

python download_dataset.py
python audit_dataset.py

Do not point the trainer directly at the source WojoodRelations files; they do not contain the transformed prompt/code fields.

Dataset output already exists

This is a safety stop. Audit or move the existing directory first. Set OVERWRITE_DATASET_OUTPUT=true only when a destructive rebuild is intended.

CUDA out of memory

First use physical batch 2 and gradient accumulation 8. Keep max length 1,024, effective batch 16, all-linear targeting, and the same LoRA rank. Reduce validation batch separately if evaluation is the only failing stage.

Option code is not one token

This indicates a tokenizer/base-model mismatch. Use Yehia's native tokenizer from the same local snapshot and do not replace the option-code inventory.

Chat-template boundary mismatch

Do not pre-render chat-template tokens in the dataset. Records must contain raw messages; model.py applies the native template. A boundary error protects against double templating or tokenizer changes.

tmux session already exists

Attach to the existing session or stop it deliberately. The launcher refuses to create a second trainer with the same session name.

Loss falls while validation does not improve

Use complete scheduled validation and the locked positive-micro-F1 selection metric. Do not replace the preserved best checkpoint with the latest checkpoint based only on training loss.

Reproducibility checklist

Before claiming a faithful retrain, verify:

• Dataset commit is a060e47f56025778b97344d4d3de60a8fd53be7c.
• Yehia base commit is b9dda4715eafee7e8090d2c83cfe078d75f4ebb8.
• All three dataset counts and hashes pass.
• audit_dataset.py passes all 19,447 rows.
• Prompt version is yehia_re_ar_v3_one_token_choice.
• The worked appendix covers all 41 classes with real full prompts/outputs.
• لا توجد علاقة is last in every row.
• All decision codes are one Yehia token.
• Only one assistant decision token contributes to loss.
• Base model is 4-bit NF4 with double quantization and BF16 compute.
• PEFT receives target_modules="all-linear".
• Eligible/wrapped counts are 225/224 and lm_head is frozen.
• Trainable parameter count is 39,976,960.
• Physical batch 4 and gradient accumulation 4 pass a finite smoke test.
• Full validation runs every 250 optimizer steps.
• Best checkpoint is chosen by validation positive micro F1.
• Official rows are never used for training or intermediate evaluation.
• Official submission contains exactly 2,074 IDs in the pinned source order and only legal canonical relations (no-relation for the negative class).
• Release staging occurs only after final training/validation completion.
• tools/verify_release.py passes on the local staged folder.
• A fresh Hugging Face download passes tools/verify_release.py --remote-clean.
• A fresh Azure clone passes tools/verify_azure_release.py --remote-clean with zero trained payloads.
• An end-to-end prepared-input inference passes from a fresh checkout.
• Immutable Azure, model, dataset, and base revisions are recorded.
• The explicitly manifested final optimizer checkpoint is present on Hugging Face only; Azure contains no trained model/checkpoint bytes, and neither destination contains base weights, older checkpoints, logs, caches, or credentials.

Pre-publication responsibility

No new license file is invented by this package. Before making a public release, the repository owner must verify and communicate the licenses, access terms, and redistribution requirements of the Yehia base model, the source dataset, and the derived dataset. A PEFT adapter does not remove those upstream obligations.