---
license: mit
language:
- en
tags:
- language-modeling
- gpt2
- distilgpt2
- long-form-generation
- coherence
- hard-negatives
- wikitext
- research
pipeline_tag: text-generation
---

# ForesightLM Phase 1

ForesightLM Phase 1 is an experimental research checkpoint exploring whether self-loop hard negatives can provide a useful calibration signal for semantic-level coherence-aware generation.

This repository contains Phase 1 model checkpoints, robustness outputs, and small ablation artifacts for the ForesightLM workshop paper.

## Main idea

The project investigates a lightweight extension around a DistilGPT-2 style language model trained on WikiText-103. The goal is not to claim solved long-form coherence, but to test whether future-aware objectives and self-loop hard negatives can improve semantic-level generation indicators.

## Uploaded checkpoints

This repository includes:

| Path | Description |
|---|---|
| `outputs/phase1/v2_wikitext103_seed42_distilgpt2` | Main ForesightLM-v2 Phase 1 checkpoint, `lambda_hardneg=0.10` |
| `outputs/phase1/v2_wikitext103_lh005_seed42_distilgpt2` | Lambda ablation checkpoint, `lambda_hardneg=0.05` |
| `outputs/phase1/v2_wikitext103_lh020_seed42_distilgpt2` | Lambda ablation checkpoint, `lambda_hardneg=0.20` |
| `outputs/phase1/bootstrap` | Paired bootstrap confidence interval results |
| `outputs/phase1/lambda_hardneg_ablation` | Lambda hard-negative ablation summary |
| `outputs/phase1/decoding_eval` | Decoding summary and qualitative examples |

## Phase 1 main metrics

Main ForesightLM-v2 result:

| metric | value |
|---|---:|
| LM loss | 3.9384 |
| Future loss | 2.7180 |
| Hard-negative margin gap | 0.0614 |
| Hard-negative satisfied rate | 0.4107 |

## Lambda hard-negative ablation

| lambda_hardneg | eval_lm ↓ | eval_future ↓ | hardneg_gap ↑ | satisfied ↑ |
|---:|---:|---:|---:|---:|
| 0.05 | 3.9385 | 2.7214 | 0.0512 | 0.3571 |
| 0.10 | 3.9384 | 2.7180 | 0.0614 | 0.4107 |
| 0.20 | 3.9383 | 2.7148 | 0.0701 | 0.4643 |

The small ablation suggests that increasing the hard-negative weight from `0.10` to `0.20` improves held-out hard-negative separation while preserving language-modeling loss.

## Bootstrap robustness

Paired bootstrap over prompt IDs suggests that calibrated v2 decoding robustly improves prompt relevance, topic drift, progression, semantic-loop rate, and prompt-drift rate relative to Core and baseline.

Raw v2 improvements are directional but generally not conclusive under the paired bootstrap intervals.

## Important limitations

These are Phase 1 research artifacts. The current evidence is limited to:

- one main domain: WikiText-103
- one primary seed
- small ablation scale
- automatic coherence indicators rather than full human evaluation

The results should be interpreted cautiously. ForesightLM does not solve long-form coherence; Phase 1 only suggests that self-loop hard negatives may provide a useful calibration signal.

## GitHub

Code and compact result artifacts are available on GitHub:

https://github.com/Ahmet2001/foresightLM

Current pushed branch:

`v2-self-loop-fixes`

## Citation

If you use these artifacts, please cite the ForesightLM Phase 1 workshop paper once available.