README.md

---
license: apache-2.0
base_model: unsloth/qwen3-4b-instruct-2507-unsloth-bnb-4bit
tags:
  - text2cypher
  - cypher
  - graph-database
  - unsloth
  - qwen3
---

# text2cypher_lora_v3

A Qwen3-4B-Instruct-2507 fine-tune (LoRA, merged 16-bit) that turns a **natural-language question + a graph schema description** into a **Cypher query**. Trained on a DocuPrism-shaped synthetic dataset (2,698 rows, 36 domains, 180 unique schemas) — see [the training repo](https://github.com/BeastxD7/DocuPrism-Text2Cypher) for the full pipeline. Superseded by `BeastxD/text2cypher_lora_v4_raw` and `BeastxD/text2cypher_lora_v4_balanced`, which target 7 specific gap categories measured from this model's own eval failures.

## This model requires a specific prompt format — it will NOT work with a bare question

This is the single most important thing to know before using it. The model was trained to
expect the **graph schema in the system prompt**, not baked into the weights — that's what
lets one model handle arbitrary domains/schemas it's never seen, rather than being locked to
one. A generic chat message like `{"role": "user", "content": "Who are you?"}` (the default
HF "Use this model" snippet above) will just get you a generic base-Qwen answer — the
fine-tuning has nothing to activate on without a schema.

**Correct usage:**

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("BeastxD/text2cypher_lora_v3")
model = AutoModelForCausalLM.from_pretrained("BeastxD/text2cypher_lora_v3", device_map="auto")

SYSTEM_PROMPT_TEMPLATE = (
    """You are a Cypher query generation assistant for a Neo4j graph database.

You are given a graph schema and a question in natural language. Use the
schema strictly - it is the only source of truth for what exists in the graph.

How to read the schema:
- 'Node properties' lists each node label together with its properties and
  their types (e.g. STRING, FLOAT, DATE, POINT). Some properties list
  example or available values - these show the kind of data to expect, not
  an exhaustive list to match against literally unless the question refers
  to one of them directly.
- 'The relationships' lists every valid pattern of how node labels connect,
  in the form (:LabelA)-[:REL_TYPE]->(:LabelB). This tells you both the
  relationship type name and its direction - respect the direction when you
  build your MATCH pattern.

How to map the question to the schema:
1. Find the node label(s) the question is really asking about (the subject
  and the target of the question).
2. Find the relationship path in the schema that connects those labels -
  questions often require traversing more than one relationship.
3. Identify any filters mentioned in the question (names, dates, categories,
  thresholds) and match them to the correct property on the correct label.
4. If the question asks for a count, total, average, minimum, maximum, or
  'top N', use the appropriate aggregation function and ORDER BY / LIMIT.

Rules:
- Use only labels, relationship types, and properties that literally appear
  in the schema below. Never invent one.
- Return ONLY the Cypher query - no explanation, no markdown fences, no
  comments.
- Return only the specific properties the question names. Return a whole
  node only when the question asks generally about an entity without naming
  particular attributes.
- When computing a single overall aggregate (an overall average, count, or
  sum), do not carry unrelated variables into the WITH that produces it -
  every non-aggregated variable in a WITH implicitly groups the aggregate by
  that variable, turning one intended overall result into one result per
  group.
- Before returning the query, check every relationship pattern you used against
  the schema's relationship list. Your arrow direction and label order must
  match one of the listed (:LabelA)-[:REL_TYPE]->(:LabelB) patterns exactly -
  if your pattern is the reverse of a listed one, you have the direction
  wrong and must flip it.
- For "highest", "lowest", "top N", "most/least" phrasing, select with
  ORDER BY <property> ASC|DESC LIMIT N rather than computing min()/max() and
  re-matching on equality - re-matching on equality returns every tied row
  instead of one deterministic answer.
- If a MATCH path can reach the same return value multiple times through
  multi-hop or branching traversal, use DISTINCT on it - unless the question
  specifically asks for a count or list per relationship/edge, in which case
  duplicates are the correct answer and DISTINCT must not be used.
- When the question asks about a status, state, count threshold, or yes/no
  condition ("accepted", "active", "at least one", "any", "some", "is X"),
  first check whether the relevant node has a property in the schema that
  directly represents that condition (a BOOLEAN, or a COUNT/INTEGER property
  already tracking it) and filter on it directly. Do not reconstruct the
  condition via a traversal or exists() check if a direct property already
  encodes it.
- If the property the question refers to (e.g. "type", "kind", "category")
  does not exist on the node you first match, do not traverse further away
  from it searching for a substitute property on a different node. Stay on
  the matched node and use its closest literal property (e.g. count distinct
  values of an existing identifying property on that same node) rather than
  inventing a multi-hop path to a loosely related property elsewhere.
- Return ONLY the Cypher query - no explanation, no markdown fences, no
  comments.\n\nSchema:\n{schema}"""
)

schema = """Nodes:
  Common properties:
    · id:STRING — Stable canonical entity identifier
    · name:STRING — Use FTS index (QUERY_FTS_INDEX) for fuzzy name lookups; CONTAINS as fallback
    · first_observed:DATE — Native DATE. Compare with DATE literals: WHERE n.first_observed >= DATE('2024-01-01')
    · last_observed:DATE — Native DATE. Use with first_observed for "active at date" checks
    · status:STRING — ACTIVE / ARCHIVED / UNCERTAIN

  Per-label descriptions and domain properties:
  (:Customer) — a customer who owns appliances and submits work orders
    · phone:STRING — primary contact phone number
    · preferred_contact_method:STRING — [Phone, Email, SMS]
  (:Appliance) — a specific appliance unit owned by a customer
    · appliance_type:STRING — [Refrigerator, Washer, Dryer, Dishwasher, Oven, HVAC]
    · brand:STRING — manufacturer brand name
    · model_number:STRING — manufacturer model number

Relationships:
  (:Customer)-[:OWNS]->(:Appliance) — customer owns the appliance"""

question = "What brand and model number does the appliance owned by customer 'Jane Doe' have?"

messages = [
    {"role": "system", "content": SYSTEM_PROMPT_TEMPLATE.format(schema=schema)},
    {"role": "user", "content": question},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=250, do_sample=False)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
# -> MATCH (c:Customer {name: 'Jane Doe'})-[:OWNS]->(a:Appliance) RETURN a.brand, a.model_number
```


## Dataset

2,698 rows, QA-audited (191 confirmed bugs found and fixed in an earlier pass, 0 known issues remaining) — see `qa/` in the training repo for the full audit trail.

## Eval results

| Metric | Score |
|---|---|
| exact_match (strict string match) | 6.7% — misleadingly low, see below |
| semantic_match (logically equivalent to gold) | 59.3% |
| core_logic_match (right schema navigation, ignoring exact column set) | 81.1% |

`exact_match` fails any pair of logically-identical queries that differ in variable naming, column aliasing, or filter placement — it dramatically understates real accuracy. `semantic_match` re-parses both queries into a structural signature (labels, relationship types, WHERE conditions by property, aggregations, hop count) and compares that instead. Full methodology: `common/semantic_rescore.py` in the training repo.

**Known limitations:**
- Sometimes returns a node's generic `name` property instead of the specific property(s) a question names — this is the main gap `text2cypher_lora_v4_raw` and `_v4_balanced` were built to close (multi-property-return rate raised from 50.8% to 60.6% in v4).
- Rare (~1-9% depending on eval set) confusion between a relationship type and a property path — e.g. writing `node.SOME_REL.name` instead of `(node)-[:SOME_REL]->(other)`. Genuinely invalid Cypher, not just an undeclared name.
- Can drop temporal/grouping qualifiers from a question entirely (e.g. ignoring "in 2023" or "each quarter") rather than getting them wrong.
- Sorting a string-typed ordinal enum (e.g. severity: Critical/High/Moderate) with a plain `ORDER BY` sorts alphabetically, not by real-world severity.

**For production use**, wrap generation with a schema-grounding check-and-retry (see `generate_cypher_checked()` in the training repo's notebooks) rather than trusting raw output — it catches the relationship-as-property-path failure mode above and retries with an explicit correction.

## Training details

- Base: `unsloth/qwen3-4b-instruct-2507-unsloth-bnb-4bit`, 4-bit + rank-16 LoRA (33.0M / 4.06B trainable params, 0.81%), targeting all attention + MLP projections.
- Trained on RunPod (RTX 5090), schema-grouped 80/10/10 train/val/heldout split (no schema appears in more than one split).
- Full training/eval pipeline, dataset-quality audit trail, and out-of-domain generalization tests: https://github.com/BeastxD7/DocuPrism-Text2Cypher