Mr-Rosen/Accuracy-Is-Not-Enough-FinQA-RAG-LoRA

FinQA Qwen2.5-7B RAG + LoRA

A LoRA adapter trained for FinQA program generation from retrieved financial context.

Base model:

Qwen/Qwen2.5-7B-Instruct

This method achieved the highest practical score in the accompanying six-method study.

Important Usage Requirement

This adapter is retrieval-aware.

It was trained on:

RAG prompt
+ question
+ fixed retrieved context
        ↓
FinQA program

It was not trained on arbitrary full-document input.

Loading the adapter and supplying the entire expanded FinQA document does not reproduce the reported RAG + LoRA method.

Official Result

Metric	Final Test
Execution Accuracy	58.33%
Program Accuracy	54.58%
Parse Success	99.65%
Average Latency	0.0531 s/example
Practical Score	0.4138

Selected checkpoint:

epoch_2_adapter

Official training burden:

Training time: 1.7182 hours
Training cost: approximately $3.25

Required Retrieval Input

For the official result, use the practical sorted retrieval artifacts published with:

MarkPaulRosenthal/Accuracy-Is-Not-Enough-Practical-Financial-QA

Files:

long_train_interleaved_plus_SORTED_retrieved.json
long_dev_interleaved_plus_SORTED_retrieved.json
long_test_interleaved_plus_SORTED_retrieved.json

Use the test retrieval file for final-test inference.

Do not use the gold-injected files when reproducing the official result.

Retrieval Record Structure

{
  "id": "example-id",
  "question": "financial question",
  "retrieved_chunks": [
    {
      "chunk_id": "text_12",
      "chunk_type": "text",
      "location": "pre_text",
      "text": "retrieved financial evidence"
    }
  ]
}

Additional retrieval metadata may also be present.

Preserve Retrieved Order

The published practical retrieval files have already been post-processed into source-document order.

When reproducing the reported method, preserve the existing order of retrieved_chunks.

Do not rerank them after loading.

Load Retrieval Data

import json

with open(
    "long_test_interleaved_plus_SORTED_retrieved.json",
    "r",
    encoding="utf-8",
) as f:
    retrieved_test = json.load(f)

record = retrieved_test[0]

context = "\n".join(
    chunk["text"]
    for chunk in record["retrieved_chunks"]
)

Prompt

The selected prompt is included in this repository as:

RAG_BASELINE_L1_top3_adapted.json

The prompt instructs the model to produce evaluator-compatible FinQA programs using only the provided retrieved context.

Adapter Configuration

rank:        64
alpha:       32
dropout:     0.05
bias:        none
task type:   CAUSAL_LM

Target modules:

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

Installation

pip install torch transformers peft accelerate safetensors

Load Adapter

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-7B-Instruct"
ADAPTER = "Mr-Rosen/Accuracy-Is-Not-Enough-FinQA-RAG-LoRA"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER,
)

model.eval()

Generate

Render the retrieved record using the exact RAG_BASELINE_L1_top3_adapted prompt.

After obtaining rendered_prompt:

inputs = tokenizer(
    rendered_prompt,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
    )

generated = output[0, inputs["input_ids"].shape[-1]:]

print(
    tokenizer.decode(
        generated,
        skip_special_tokens=True,
    )
)

Expected Output

["subtract(", "5829", "5735", ")", "EOF"]

Official Reproduction Checklist

Use all of the following together:

Qwen/Qwen2.5-7B-Instruct
Mr-Rosen/Accuracy-Is-Not-Enough-FinQA-RAG-LoRA
practical sorted retrieval JSON
RAG_BASELINE_L1_top3_adapted
deterministic generation
FinQA program parsing
original FinQA evaluator

Changing retrieval, chunk order, or prompt formatting changes the method being evaluated.

Oracle / Gold-Injected Data

Gold-injected experiments are supplemental oracle tests and are not the official RAG + LoRA input.

The official execution accuracy for this repository is:

58.33%

Related Repositories

Dataset:

Mr-Rosen/Accuracy-Is-Not-Enough-FinQA-Dataset

Retrieval artifacts, prompts, evaluator, and research results:

MarkPaulRosenthal/Accuracy-Is-Not-Enough-Practical-Financial-QA

Limitations

Performance depends on both the adapter and the upstream retrieval system.

A generator cannot reason over evidence that was not retrieved into its input.

The reported result therefore represents the complete fixed RAG + LoRA configuration rather than adapter quality in isolation.

License

The adapter is released under MIT.

The Qwen base model remains subject to its upstream license.

The FinQA-derived dataset is separately released under CC BY 4.0.