SmolLM2-135M-Text2Cypher

LoRA fine-tunes of HuggingFaceTB/SmolLM2-135M-Instruct that turn a Neo4j graph schema and a natural language question into a Cypher query.

This repo holds two adapters. They use the same data and differ only in the prompt they were trained with:

• fewshot/: Cypher rules plus 4 worked examples in the system prompt. This is the stronger one.
• default/: a plain one line instruction.

Shared facts:

• Base: SmolLM2-135M-Instruct
• Data: RomanTeucher/text2cypher-curated (1,000 train / 75 val / 50 test)
• Method: LoRA. Few-shot examples come from the train split only.
• Adapters are not merged. Load them with PEFT.

This is a small model, so it gets plenty of queries wrong. The point is the training and evaluation setup, not the raw accuracy.

Usage

Each adapter has its own prompt. Load the adapter from its subfolder and use the matching system prompt. The user turn is always Graph Schema:\n<schema>\n\nQuestion: <question>. Pairing an adapter with the wrong prompt makes the output worse.

The block below runs the few-shot adapter. To run the default adapter instead, comment the few-shot block and uncomment the default block.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "HuggingFaceTB/SmolLM2-135M-Instruct"
REPO = "<your-hf-username>/smollm2-135m-text2cypher"

# ===== Pick a variant: keep exactly ONE of the two blocks below =====

# ----- FEW-SHOT adapter: Cypher rules + 4 worked examples in the system prompt -----
SUBFOLDER = "fewshot"
SYSTEM = (
    "You are a Neo4j Cypher query expert.\n"
    "Convert the natural language question into a Cypher query using ONLY\n"
    "the nodes, relationships, and properties in the provided schema.\n\n"
    "Rules:\n"
    "- Use single-letter lowercase aliases matching the label: (m:Movie), (p:Person)\n"
    "- Use WHERE for numeric/boolean conditions, inline {} for string equality\n"
    "- String values must use single quotes: {name: 'Alice'}\n"
    "- Always end with RETURN\n"
    "- Never invent properties or labels not in the schema\n\n"
    "Output ONLY the Cypher query, nothing else.\n\n"
    "Examples:\n"
    "Schema: Article {article_id, title}\n"
    "Question: Retrieve distinct values of the title from Article where article_id is not 1010!\n"
    "Cypher: MATCH (n:Article) WHERE n.article_id <> '1010' RETURN DISTINCT n.title AS title\n\n"
    "Schema: Movie {title, votes, tagline, released}, Person {born, name}, (:Person)-[:ACTED_IN]->(:Movie)\n"
    "Question: Which actors played in the most movies?\n"
    "Cypher: MATCH (p:Person)-[a:ACTED_IN]->(m:Movie) RETURN p.name, COUNT(m) AS movies_count ORDER BY movies_count DESC\n\n"
    "Schema: Question {title}, Tag {name}, User {display_name}, (:User)-[:ASKED]->(:Question), (:Question)-[:TAGGED]->(:Tag)\n"
    "Question: Which users have asked questions tagged with 'react-apollo'?\n"
    "Cypher: MATCH (u:User)-[:ASKED]->(q:Question)-[:TAGGED]->(t:Tag {name: 'react-apollo'}) RETURN u\n\n"
    "Schema: Movie {title, votes, tagline, released}, Person {born, name}, (:Person)-[:DIRECTED]->(:Movie)\n"
    "Question: Who are the first 3 oldest directors and the movies they have directed?\n"
    "Cypher: MATCH (p:Person)-[:DIRECTED]->(m:Movie) RETURN p.name AS Director, p.born AS BirthYear, collect(m.title) AS Movies ORDER BY p.born ASC LIMIT 3"
)

# ----- DEFAULT adapter: plain one line instruction -----
# To use this instead, comment the FEW-SHOT block above and uncomment this block.
# SUBFOLDER = "default"
# SYSTEM = (
#     "You are an expert in Neo4j and the Cypher query language. "
#     "Given a graph schema and a natural-language question, "
#     "generate a valid Cypher query that answers the question. "
#     "Output ONLY the Cypher query, nothing else."
# )

base_model = AutoModelForCausalLM.from_pretrained(BASE)
model      = PeftModel.from_pretrained(base_model, REPO, subfolder=SUBFOLDER)
tokenizer  = AutoTokenizer.from_pretrained(REPO, subfolder=SUBFOLDER)

schema   = "Movie {title, votes, released}, Person {born, name}, (:Person)-[:ACTED_IN]->(:Movie)"
question = "Which actors appeared in more than 5 movies?"

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user",   "content": f"Graph Schema:\n{schema}\n\nQuestion: {question}"},
]

inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
)
out = model.generate(
    **inputs,
    max_new_tokens=256,
    do_sample=False,
    pad_token_id=tokenizer.eos_token_id,
)
prompt_len = inputs["input_ids"].shape[1]
print(tokenizer.decode(out[0, prompt_len:], skip_special_tokens=True).strip())

Keep the adapter and prompt paired: fewshot subfolder with the few-shot system prompt, default subfolder with the default one. Crossing them lowers quality.

Training

• LoRA: r=16, alpha=32, dropout=0.05, on all 7 projections (q, k, v, o, gate, up, down).
• Loss on the Cypher tokens only (prompt tokens masked with -100).
• 5 epochs, lr 2e-4 (linear decay, ~5% warmup), effective batch 16 (4 x grad-accum 4), max length 1024, weight decay 0.01.
• CPU only. Best checkpoint picked by validation loss (epoch 5).

Epoch	Train loss	Val loss
1	0.9951	0.6818
2	0.4308	0.4731
3	0.2688	0.4005
4	0.1996	0.3741
5	0.1696	0.3671 (best)

Val loss drops every epoch with no divergence. The train/val gap widens a little near the end, so mild overfitting, nothing serious.

Evaluation

Test split (n=50), greedy decoding. Metrics are text and structure only (no live Neo4j, so no execution checks). cyspider-* rows (SQL-style schemas) are dropped from all splits.

Overall (test, n=50)

Metric	Base	default/ adapter	fewshot/ adapter
Exact match	0.00	0.14	0.18
Exact match (strict)	0.00	0.14	0.18
Component F1	0.026	0.392	0.413
Component precision	0.022	0.398	0.420
Component recall	0.036	0.394	0.414
Syntactic valid	0.06	1.00	0.98
Schema grounding	0.367	0.854	0.846
Schema grounding coverage	0.46	0.94	0.98

• The biggest win from fine-tuning is format. The base model writes prose or SQL; validity goes from 0.06 to about 1.0, and component F1 from near zero to about 0.41.
• Few-shot adds a small edge over the default adapter (+0.04 exact match, +0.02 F1).
• Validity is a floor check (brackets balanced, has RETURN). Wrong queries still pass it, so read it as "stayed in Cypher", not "correct".

Limitations

• It is right only sometimes. Exact match is 0.18 and F1 is 0.41, so most outputs look like Cypher but are not correct.
• It does poorly on the synthetic data sources (0 exact match). It mostly learned the functional_cypher style, which is the bulk of the training data.
• The few-shot prompt helps only a little, and it costs a much longer prompt.
• 135M parameters is small for reading a question and a schema and writing a correct query.
• Harder queries fail: the same label used in two roles, WITH chaining, CASE, and UNWIND.
• It leans on the schema format it saw in training. Give it a different format and it can ignore the schema.
• I trained on CPU, so I could not try many settings.

License

Apache-2.0, from the SmolLM2 base model.