Mr-Rosen/Accuracy-Is-Not-Enough-FinQA-RAG-QLoRA

FinQA Qwen2.5-7B RAG + QLoRA

A QLoRA-trained adapter for structured FinQA program generation using retrieved financial context.

Base model:

Qwen/Qwen2.5-7B-Instruct

Important Usage Requirement

This adapter expects the project's RAG-style input.

Training input:

RAG prompt
+ question
+ fixed retrieved context
        ↓
FinQA program

It was not trained as a full-document QLoRA model.

Use the practical sorted retrieval artifacts and the selected RAG prompt when reproducing the reported result.

Official Result

Metric	Final Test
Execution Accuracy	59.90%
Program Accuracy	55.97%
Parse Success	100.00%
Parse Failures	0
Average Latency	0.5090 s/example
Practical Score	0.3809

Selected checkpoint:

epoch_2_adapter

Official training burden:

Training time: approximately 2.23 hours
Training cost: approximately $4.21

Official Retrieval

Use:

long_train_interleaved_plus_SORTED_retrieved.json
long_dev_interleaved_plus_SORTED_retrieved.json
long_test_interleaved_plus_SORTED_retrieved.json

from:

MarkPaulRosenthal/Accuracy-Is-Not-Enough-Practical-Financial-QA

The reported final-test result uses:

long_test_interleaved_plus_SORTED_retrieved.json

Do not substitute gold-injected retrieval when reporting official performance.

Load Retrieval Data

import json

with open(
    "long_test_interleaved_plus_SORTED_retrieved.json",
    "r",
    encoding="utf-8",
) as f:
    retrieved_test = json.load(f)

record = retrieved_test[0]

context = "\n".join(
    chunk["text"]
    for chunk in record["retrieved_chunks"]
)

Preserve the published chunk order.

Prompt

The selected prompt is included in this repository as:

RAG_BASELINE_L1_top3_adapted.json

QLoRA Training Configuration

load_in_4bit:        true
quantization:        NF4
compute dtype:       bfloat16
double quantization: true

Adapter:

rank:        64
alpha:       32
dropout:     0.05
bias:        none
task type:   CAUSAL_LM

Target modules:

q_proj
k_proj
v_proj
o_proj
gate_proj
up_proj
down_proj

Standard Loading

pip install torch transformers peft accelerate safetensors

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-7B-Instruct"
ADAPTER = "Mr-Rosen/Accuracy-Is-Not-Enough-FinQA-RAG-QLoRA"

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER,
)

model.eval()

Optional 4-Bit Base-Model Loading

pip install bitsandbytes

import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
)
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-7B-Instruct"
ADAPTER = "Mr-Rosen/Accuracy-Is-Not-Enough-FinQA-RAG-QLoRA"

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)

base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    quantization_config=quantization_config,
    device_map="auto",
)

model = PeftModel.from_pretrained(
    base_model,
    ADAPTER,
)

model.eval()

Generate

After rendering the retrieved question and context with the official RAG prompt:

inputs = tokenizer(
    rendered_prompt,
    return_tensors="pt",
).to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
    )

generated = output[0, inputs["input_ids"].shape[-1]:]

print(
    tokenizer.decode(
        generated,
        skip_special_tokens=True,
    )
)

Expected Output

[
  "divide(",
  "60",
  "243",
  ")",
  "multiply(",
  "#0",
  "const_100",
  ")",
  "EOF"
]

Official Reproduction Checklist

Qwen/Qwen2.5-7B-Instruct
Mr-Rosen/Accuracy-Is-Not-Enough-FinQA-RAG-QLoRA
practical sorted retrieval JSON
RAG_BASELINE_L1_top3_adapted
deterministic generation
FinQA program parsing
original FinQA evaluator

QLoRA Note

QLoRA is the training method.

The released artifact is a PEFT adapter rather than a standalone 4-bit model.

Oracle Retrieval

Gold-injected retrieval files are supplemental diagnostic artifacts.

They intentionally alter evidence availability and are not part of the official 59.90% execution-accuracy result.

Related Repositories

Dataset:

Mr-Rosen/Accuracy-Is-Not-Enough-FinQA-Dataset

Retrieval artifacts, prompt configuration, evaluator, and research results:

MarkPaulRosenthal/Accuracy-Is-Not-Enough-Practical-Financial-QA

Limitations

This adapter depends on retrieval quality and on reproducing the retrieved-context prompt structure.

Its measured performance represents the complete RAG + QLoRA system rather than adapter performance independent of retrieval.

License

The adapter is MIT licensed.

The Qwen base model remains under its upstream license.

The FinQA-derived dataset is separately released under CC BY 4.0.