AI_EXPERIMENTS/EXP-015__o4-mini-Azure-RFT-in-progress.md

EXP-015 · o4-mini + Reinforcement Fine-Tuning (Azure OpenAI) — IN PROGRESS

Date: 2026-07-26 Status: in progress — this file is being written while the job runs, not after completion, per this repo's charter of documenting honestly rather than only after a clean result is known.

Why this is different from EXP-001 through EXP-014

Every prior experiment in this series used supervised fine-tuning (SFT): the model is shown example (prompt, target completion) pairs and trained to reproduce them. This experiment uses reinforcement fine-tuning (RFT) instead: the model is given only prompts, generates its own responses during training, and a separate grader scores each response — the model is optimized to maximize the grader's score, not to match a fixed target completion. This is a different mechanism for the same underlying question this series has been asking: can Protocol 0's behavioral rules survive being baked into a model's weights, rather than kept in a system prompt.

Setup

• Base model: o4-mini-2025-04-16, Azure OpenAI's only fine-tunable reasoning model on this resource. Confirmed via live API check that o4-mini and gpt-35-turbo are the only fine-tunable models besides gpt-4o-2024-08-06 (already used in EXP-006/008/013/014) — o4-mini does not support standard SFT at all; the API rejects a normal fine-tuning request with "Finetuning with o4-mini-2025-04-16 requires the reinforcement method of finetuning."
• Dataset conversion: derived from protocol0_sft_v3_full.jsonl (2349 examples, the same dataset EXP-014 used). Each SFT row's (system, user, assistant) triple was split: the user message becomes the RFT prompt, and the original assistant completion is kept as a reference_answer field for the grader to compare against — not used as a training target directly. System messages had to be stripped entirely — Azure's RFT endpoint rejects any training example containing a system role ("System messages are not supported in 'reinforcement' examples"), discovered via a failed first attempt (see "What went wrong on the first try" below). This means the Protocol 0 rules are not shown to the model during RFT training turns at all — they are only present inside the grader's own instructions, and the model must learn the behavior purely from the reward signal, not from being told the rules directly. Split 95/5 into 2231 training / 118 validation examples (well under Azure's 50,000/1,000 caps).
• Grader: a score_model grader — a separate call to gpt-4o-2024-08-06 that reads the model's generated response plus the reference_answer, and returns a 0.0-1.0 score based on Protocol 0 compliance (no fabrication, no unsolicited opinion, stop-on-ambiguity, single-action-only, concise). This reuses the same criteria this series' benchmark script has checked automatically since EXP-002, just as a training signal instead of a post-hoc evaluation.
• Job: ftjob-c24fd461d0d7443d94ae16f0586a79bf, suffix protocol0-rft-v1, estimated_finish ~6 hours from submission — RFT trains by generating and grading many rollouts per example, and is inherently much slower than SFT's single forward/backward pass per example; this is expected, not a stall.

What went wrong on the first try (kept here, not silently fixed)

The first submission attempt also surfaced a real operational mistake worth recording plainly: while probing which Azure api-version supports the RFT method field, three separate job-creation requests were sent across different preview API versions without stopping at the first success, because the older versions returned clean-looking errors that looked like "this version doesn't support it" rather than being recognized as successes on the newer ones. This created three simultaneous duplicate RFT jobs before the mistake was caught. Two were cancelled immediately (ftjob-60578856527e47f29338fd5fc351979d, ftjob-679fb81f9eb840a0b4a6848d7837c5b0); the third (ftjob-2995a0739cb94c97a7e7c5b5d770aef2) was left running as the "kept" one — but it then failed on file preprocessing (the system-message rejection described above) before any real training cost was incurred, once cancellation of the duplicates was confirmed working (the Azure cancel endpoint requires an explicit Content-Length — a bare -X POST with no body triggers an HTTP 411 and silently fails to cancel, which cost a few minutes of investigation before finding the fix: pass an explicit JSON body, even just {}).

What this will and won't show once it completes

• Will show: whether reward-based training on the same dataset content this series has already SFT-tuned on produces different fabrication-pattern behavior than the SFT line's gpt-4o results (EXP-006/008/013/014) — a genuinely different training mechanism, not just a different base model.
• Will not show: a clean comparison to the SFT line, since multiple things differ simultaneously (base model family — o4-mini is a reasoning model, not gpt-4o; training mechanism — RFT vs SFT; and the training data no longer includes the system prompt context that SFT examples had). Any result here should be read as its own data point, not a controlled ablation against the rest of the series.
• Benchmark plan once complete: same 5-category methodology as every other experiment in this series (ambiguity_stop, no_unsolicited_opinion, single_action_only, unverifiable_refusal, conciseness), run against base o4-mini-2025-04-16 and the fine-tuned deployment, with mandatory manual review of every response before reporting a score — per this repo's own repeatedly-confirmed finding that the automatic keyword checker is unreliable in both directions.

This file will be updated (not silently rewritten) once the job reaches a terminal state.

Update — second attempt also failed, different reason (2026-07-26)

After fixing the system-message format issue, the resubmitted job (ftjob-c24fd461d0d7443d94ae16f0586a79bf) failed again, this time for a substantively different reason:

unsafe_file: The job failed due to an unsafe training file. This training file was
blocked because too many examples were flagged by our moderation for content that
violates Azure OpenAI's usage policies with respect to model reasoning extraction.
Please review the data and remove potential offending examples before retrying
fine-tuning.

This is a content-moderation block, not a format error — Azure's RFT pipeline appears to specifically screen for training data that resembles attempts to distill/extract a reasoning model's chain-of-thought (a known industry concern with o-series models). This dataset's governance-heavy content — CORE LAW, RED LINE prohibitions, macro-pattern rounds about refusal boundaries and security — plausibly triggered this at scale, though Azure's error does not identify which specific examples were flagged, so this is diagnosis by plausibility, not confirmed cause.

Status: blocked, pending a decision on how to proceed — options not yet chosen between:

1. Filter the dataset for content most likely to trigger this policy (RED LINE/security- framed examples specifically) and retry with a smaller, non-governance subset.
2. Abandon the o4-mini RFT attempt and treat EXP-015 as closed-unevaluable (same category as EXP-003/005/011 — job blocked before any real training happened, not a Protocol-0 result).
3. Try a different reasoning-capable model or platform for RFT instead of Azure's o4-mini.

No further Azure fine-tuning jobs were submitted after this failure pending that decision — avoiding a repeat of the earlier duplicate-job mistake by not retrying blindly.