fin-ai-lab/aux-2024

Aux 2024

A 3B model trained from scratch through the end of 2024 (knowledge cutoff January 2025), released as part of Frontier to Point-in-Time — a toolkit for reducing look-ahead bias in forecasting with LLMs, without sacrificing what makes them useful in the first place. The toolkit adapts open-source frontier models to substantially reduce look-ahead bias, applies the methods to Qwen 3.5 27B to convert it into a 2015 point-in-time model, and ships production-ready inference code.

This model is the recent-era half of the temporal pair used by Divergence Decoding: its logits stand in for the frontier model's post-2015 knowledge, which the method cancels at inference time, with no retraining of the large model. Like its twin, it is a from-scratch model with a 128K context window and best-in-class instruction following; the 2015 twin appears as "Aux 2015" in the project's examples and benchmarks.

The temporal pair:

• Aux 2015 — knowledge cutoff December 2015
• Aux 2024 — trained through the end of 2024 (knowledge cutoff January 2025)

The two models are era-matched: same architecture, tokenizer, and training recipe, differing only in the time span of their pre-training corpus. Their logit difference is the signal used by Divergence Decoding.

How it's used

Divergence Decoding (DD) is state-of-the-art for Q&A unlearning and is designed to scale to large datasets. Frontier to Point-in-Time uses it to unlearn all knowledge after the cutoff of December 31, 2015. It requires the two auxiliary models and applies the following inference-time adjustment to the large model's logits:

l̂_2015 = l_frontier + α · (l_aux-2015 − l_aux-2024)

where l_frontier are the frontier model's logits (Qwen 3.5 27B), l_aux-2024 are this model's, and l_aux-2015 are the 2015 twin's (Aux 2015).

The auxiliary models are trained from scratch with the Qwen 3.5 tokenizer, then SFT'd on samples distilled from the large model so they inherit its chat template and response style. Unlike prior work, which saves checkpoints from a single training run as data is introduced chronologically, each auxiliary model is trained independently.

See the project page for the full method — including the feature-steering component — and the GitHub repository for production-ready inference code.

Model details

• Architecture: MinistralDualRope — a Ministral (Llama-family math: SiLU-gated MLP, RMSNorm, GQA, rotary, no biases) with a per-layer sliding/full attention pattern and a second, unscaled rotary for the sliding layers (the Gemma-3 dual-RoPE design).
• Parameters: ~3.4B — 28 layers, hidden size 3072, 24 attention heads, 8 KV heads, head dim 128, intermediate size 8192.
• Attention: sliding window 512, every 6th layer full attention.
• Context length: 131,072 (llama3 RoPE scaling, factor 64).
• Vocab: 248,320 (shared with the frontier model so DD can bridge logits).
• Precision: bf16.
• Training: temporal cooldown base followed by a partial supervised fine-tuning pass on samples distilled from the frontier model, so it inherits the frontier model's chat template and response style. The chat template supports optional thinking (enable_thinking).

Usage

This is a custom architecture — load with trust_remote_code=True:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "fin-ai-lab/aux-2024", trust_remote_code=True, torch_dtype="bfloat16"
)
tok = AutoTokenizer.from_pretrained("fin-ai-lab/aux-2024", trust_remote_code=True)

Citation

@inproceedings{
    merchant2026divergence,
    title={Divergence Decoding: Inference-Time Unlearning via Auxiliary Models},
    author={Humzah Merchant and Bradford Levy},
    booktitle={Forty-third International Conference on Machine Learning},
    year={2026},
    url={https://openreview.net/forum?id=JPbp2S9yTO}
}
@inproceedings{
    merchant2026a,
    title={A Fast and Effective Solution to the Problem of Look-ahead Bias in {LLM}s},
    author={Humzah Merchant and Bradford Levy},
    booktitle={NeurIPS 2025 Workshop: Generative AI in Finance},
    year={2026},
    url={https://openreview.net/forum?id=zYsLIPgM28}
}
@inproceedings{
    merchant2026forecasting,
    title={Forecasting With {LLM}s: Improved Generalization Through Feature Steering},
    author={Humzah Merchant and Bradford Levy},
    booktitle={Forecasting as a New Frontier of Intelligence},
    year={2026},
    url={https://openreview.net/forum?id=ppN6CmoNOk}
}

License

Apache-2.0.