README.md

---
base_model: Qwen/Qwen3.5-4B
license: apache-2.0
library_name: peft
tags:
  - base_model:adapter:Qwen/Qwen3.5-4B
  - lora
  - sft
  - transformers
  - synthetic-data-kit
  - benchmark
  - medical
  - financial
pipeline_tag: text-generation
datasets:
  - likhithv/knowledgemesh-benchmark-eval
---

# Meta SDK Baseline — LoRA Adapter

LoRA adapter for `Qwen/Qwen3.5-4B` fine-tuned on **1,209 chunk-based training samples** generated by [Meta's Synthetic Data Kit](https://github.com/facebookresearch/synthetic-data-kit) from the same financial and medical source documents.

This is the **Meta SDK baseline** model from the paper *"Knowledge Graph-Guided Fine-Tuning Data Generation: A Rigorous Benchmark"* — the industry-standard chunk-based approach used as the comparison point against KnowledgeMesh.

## Benchmark Results

Evaluated by Gemini 2.5 Flash pointwise judge (1–5 scale, 4 dimensions):

| Eval Set | Base | **This Model** | KM Full | Delta (KM − this) |
|---|---|---|---|---|
| Primary (n=473, KM-generated) | 1.79 | **1.93** | 2.47 | +0.54 |
| Independent (n=955, Gemini-generated) | 1.96 | **2.17** | 2.90 | +0.72 |

## Usage

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen3.5-4B"
adapter_id = "likhithv/meta-sdk-baseline"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_id)

messages = [{"role": "user", "content": "What were Apple's total net sales in 2023?"}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True)
outputs = model.generate(inputs.to(model.device), max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
```

## Training Details

| Parameter | Value |
|---|---|
| Base model | Qwen/Qwen3.5-4B (4-bit quantized via bitsandbytes) |
| Fine-tuning method | LoRA (rank=16, alpha=16) |
| Training samples | 1,209 (chunk-based QA, Meta Synthetic Data Kit) |
| Epochs | 3 |
| Learning rate | 2e-4 |
| Effective batch size | 8 |
| Hardware | Kaggle T4 GPU (16 GB) |
| Domains | Financial (Apple 10-K 2023), Medical (PubMed abstracts) |

## Eval Datasets

- [`likhithv/knowledgemesh-benchmark-eval`](https://huggingface.co/datasets/likhithv/knowledgemesh-benchmark-eval)

## Compared Models

- [`likhithv/km-full-model`](https://huggingface.co/likhithv/km-full-model) — KnowledgeMesh, 4,361 KG-guided samples (+0.72 on independent eval)

## Citation

```bibtex
@misc{knowledgemesh2026,
  title={Knowledge Graph-Guided Fine-Tuning Data Generation: A Rigorous Benchmark},
  author={Likhith V},
  year={2026},
  howpublished={https://huggingface.co/likhithv/meta-sdk-baseline}
}
```