README.md

---
license: apache-2.0
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - grant-matching
  - nonprofit
  - foundation-grants
base_model: Qwen/Qwen3-Embedding-0.6B
datasets:
  - ArkMaster123/grantpilot-training-data
language:
  - en
pipeline_tag: sentence-similarity
library_name: sentence-transformers
---

# GrantPilot Embedding V2 (Federal + Foundation)

Fine-tuned [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) for grant-organization semantic matching. V2 extends coverage from federal-only (NIH/NSF) to include **37,684 private foundations**.

> **See also:** [V1 (federal-only)](https://huggingface.co/ArkMaster123/grantpilot-embedding) which outperforms OpenAI on federal grant retrieval.

## Embedding Benchmark Results

Benchmarked on 998 test pairs (901 foundation, 78 NIH, 19 NSF) using retrieval and classification metrics.

### Retrieval Quality

| Model | Dim | R@1 | R@5 | R@10 | MRR | NDCG@10 |
|-------|-----|-----|-----|------|-----|---------|
| OpenAI text-embedding-3-small | 1536 | **0.343** | **0.570** | **0.682** | **0.453** | **0.499** |
| Qwen3-Embedding-0.6B (base) | 1024 | 0.295 | 0.514 | 0.630 | 0.403 | 0.449 |
| **GrantPilot V2 (this model)** | 1024 | 0.295 | 0.516 | 0.622 | 0.403 | 0.446 |

**Verdict: OpenAI wins on retrieval.** The fine-tuned V2 embedding performs on par with the base Qwen3 model — fine-tuning did not meaningfully improve retrieval on this mixed dataset. V1 (federal-only) significantly outperformed OpenAI on federal retrieval, but adding 90% foundation data diluted that specialization.

### AUC as Classifier Feature

| Model | Overall AUC | Foundation AUC | NIH AUC | NSF AUC |
|-------|-------------|----------------|---------|---------|
| OpenAI text-embedding-3-small | **0.886** | **0.972** | 0.473 | 0.524 |
| Qwen3-Embedding-0.6B (base) | 0.881 | 0.965 | 0.611 | 0.548 |
| **GrantPilot V2 (this model)** | 0.881 | 0.965 | **0.614** | 0.548 |

Interesting: OpenAI has the best overall AUC but **worst federal AUC** (0.47 on NIH — worse than random). Our fine-tuned model is best on federal grants.

### Inference Latency

| Model | Avg Latency | Cost |
|-------|-------------|------|
| OpenAI text-embedding-3-small | 43.9ms | API cost |
| Qwen3-Embedding-0.6B (base) | 2.9ms | Free (self-hosted) |
| **GrantPilot V2 (this model)** | **1.7ms** | Free (self-hosted) |

**25x faster than OpenAI** with zero API cost.

### Comparison with V1

| Metric | V1 vs OpenAI | V2 vs OpenAI |
|--------|-------------|-------------|
| R@1 | **V1 wins (+46%)** | OpenAI wins |
| R@5 | **V1 wins (+22%)** | OpenAI wins |
| R@10 | **V1 wins (+28%)** | OpenAI wins |

V1 beat OpenAI decisively on federal grants. V2 lost that edge by training on a dataset that is 90% foundation data.

## Why Use This Model?

The embedding alone is not the star — the **XGBoost classifier built on top** is where the real value comes from:

| Classifier Metric | V1 | V2 |
|-------------------|----|----|
| Overall AUC | 0.837 | **0.997** |
| Federal AUC | 0.837 | **0.913** |
| Accuracy | 72.1% | **98.3%** |
| F1 | 0.595 | **0.983** |

See: [grantpilot-classifier-v2](https://huggingface.co/ArkMaster123/grantpilot-classifier-v2)

## Training Details

- **Hardware**: NVIDIA H100 80GB
- **Training Steps**: 1,000 (LoRA fine-tuning)
- **Training Pairs**: 324,479 positive pairs
- **LoRA Config**: r=16, alpha=32, target=q/k/v/o projections
- **Batch Size**: 32 (x4 gradient accumulation = 128 effective)
- **Learning Rate**: 2e-5
- **Final Val Loss**: 0.1458

### Training Data Composition

| Source | Pairs | % |
|--------|-------|---|
| Foundation (990-PF) | 292,401 | 90.1% |
| NIH | 25,717 | 7.9% |
| NSF | 6,361 | 2.0% |

## Usage

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("ArkMaster123/grantpilot-embedding-v2", trust_remote_code=True)

org_text = "Organization: Ford Foundation\nLocation: New York, NY\nType: FOUNDATION"
grant_text = "Grant: Support for civil society organizations\nAmount: $500,000"

embeddings = model.encode([org_text, grant_text])
similarity = embeddings[0] @ embeddings[1]
```

## Related Models

| Model | Description |
|-------|-------------|
| [grantpilot-embedding](https://huggingface.co/ArkMaster123/grantpilot-embedding) | V1 — federal-only, beats OpenAI on retrieval |
| [grantpilot-classifier](https://huggingface.co/ArkMaster123/grantpilot-classifier) | V1 — federal-only classifier (AUC 0.837) |
| [grantpilot-classifier-v2](https://huggingface.co/ArkMaster123/grantpilot-classifier-v2) | V2 — combined classifier (AUC 0.997) |
| [grantpilot-training-data](https://huggingface.co/datasets/ArkMaster123/grantpilot-training-data) | Training data (V1 at training/, V2 at training_v2/) |