README.md

---
title: Sangue e Grafi
emoji: "🩸"
colorFrom: red
colorTo: yellow
sdk: gradio
sdk_version: 6.15.2
python_version: "3.11"
app_file: app.py
pinned: false
license: mit
tags:
  - knowledge-graph
  - grpo
  - ontology
  - inheritance-reasoning
  - build-small-hackathon
  - track:wood
  - sponsor:nvidia
  - sponsor:modal
  - achievement:offgrid
  - achievement:welltuned
  - achievement:offbrand
  - achievement:llama
  - achievement:sharing
  - achievement:fieldnotes
---

# 🩸 Sangue e Grafi 📊

### Blood & Graphs — Ontology-Guided Reasoning for Small Models

> **Hugging Face Build Small Hackathon 2026**
> Track: 🍄 *An Adventure in Thousand Token Wood*
>
> 📓 [Blog Post](https://wordlift.io/blog/en/sangue-e-grafi/) · 🎬 [Video Demo](https://wor.ai/yt-sangue-e-grafi)

---

## 🎯 The Pitch

Can a small open model outperform much larger frontier models on adversarial inheritance reasoning?

**Yes, when the task is not about linguistic fluency, but about preserving relational structure.**

**Sangue e Grafi** is a research prototype showing how a small ontology-guided model can solve inheritance puzzles that often mislead text-only frontier models. The failure mode is not random. It is a form of **semantic drift**: the model follows emotionally salient or semantically related information while dropping the formal dependency that determines the correct answer.

In our testbed, the model must decide who inherits an asset according to a will. The narrative contains distractors: spouses, caregivers, emotionally important relatives, and socially prominent figures. A text-only model may select the most salient person. A graph-guided agent must instead follow the legally relevant biological lineage encoded in the ontology.

This project builds on two research threads:

1. [**RLM-on-KG**](https://github.com/wordlift/rlm-on-kg) ([arXiv:2604.17056](https://arxiv.org/abs/2604.17056)), where a language model acts as an autonomous navigator over a Knowledge Graph.
2. [**SEOcrate-4B**](https://huggingface.co/cyberandy/SEOcrate-4B_grpo_new_01), our small-model GRPO experiment for SEO reasoning.

Sangue e Grafi combines these ideas: it uses the graph-navigation pattern from RLM-on-KG and the post-training pattern from SEOcrate, then applies both to a harder relational reasoning problem where the reward comes from OWL axioms instead of an LLM judge.

The core idea:

> Use the ontology not only as documentation or retrieval memory, but as a **deterministic reward layer** for training small models to follow valid reasoning paths.

---

## 🧠 Why This Matters

Large language models are excellent at semantic association. They understand that a spouse, a caregiver, a child, and an heir all belong to the same family story.

But inheritance reasoning is not solved by semantic similarity.

It requires preserving:

* identity,
* biological lineage,
* directionality,
* age constraints,
* aliveness,
* exclusions,
* and the exact clause of the will.

This is a small example of a broader problem in agentic AI.

Enterprise agents fail in similar ways:

* a product is similar, but not compatible;
* a claim is plausible, but not allowed;
* a workflow step is reasonable, but a prerequisite is missing;
* a customer is relevant, but not eligible;
* a document is related, but not authoritative;
* an entity is mentioned, but not the canonical one.

The problem is not lack of language. The problem is missing **constraint-preserving reasoning**.

Sangue e Grafi explores how ontologies and Knowledge Graphs can make this missing structure explicit, verifiable, and eventually trainable.

---

## 📊 Benchmark Results

We tested **10 procedurally generated inheritance puzzles** using the same seeds across several frontier baselines and our graph-guided agent.

Each puzzle contains semantically primed distractors designed to mislead text-only reasoning.

### Accuracy

| Model                |       Approx. scale       |   Score   | Accuracy |
| :------------------- | :-----------------------: | :-------: | :------: |
| Gemini 2.5 Flash     |     frontier baseline     |    3/10   |    30%   |
| Gemini 3.5 Flash     |     frontier baseline     |    6/10   |    60%   |
| Gemini 3.1 Pro       |     frontier baseline     |    6/10   |    60%   |
| **Gemma + KG Agent** | **small model + KG tools** | **10/10** | **100%** |

> **Note**: This is a small adversarial benchmark, not a general claim that small models outperform frontier models. The result shows that explicit structure can dominate scale when the task depends on formal relational constraints.

### Per-scenario breakdown

| Seed | Gold Answer        |  2.5 Flash |  3.5 Flash |  3.1 Pro | KG Agent |
| :--: | :----------------- | :--------: | :--------: | :------: | :------: |
|  42  | Claudio Sala       |  ❌ Enrico  |  ❌ Enrico  | ❌ Enrico |     ✅    |
|  99  | Damiano Lombardi   |   ❌ Marta  |   ❌ Marta  |  ❌ Marta |     ✅    |
|  137 | Ilaria Colombo     |  ❌ Silvia  |      ✅     |     ✅    |     ✅    |
|  256 | Giovanni Moretti   |      ✅     |      ✅     | ❌ Matteo |     ✅    |
|  500 | Giuseppe Marchetti |      ✅     |      ✅     |     ✅    |     ✅    |
|  777 | Alessia Ferrari    | ❌ Isabella |      ✅     |     ✅    |     ✅    |
| 1234 | Andrea Vitale      |      ✅     |      ✅     |     ✅    |     ✅    |
| 2048 | Damiano Galli      | ❌ Federica |      ✅     |     ✅    |     ✅    |
| 3333 | Pietro Costa       | ❌ Veronica | ❌ Veronica |     ✅    |     ✅    |
| 4096 | Federica Santoro   |   ❌ Giada  |   ❌ Giada  |  ❌ Giada |     ✅    |

### Key Finding

Several frontier baselines fail on the same seeds with the same wrong answer. This suggests a systematic failure mode: the models often identify the right family context, but not the exact person required by the formal inheritance rule.

The KG agent solves the task by following the ontology-defined path instead of the narrative salience.

### Adversarial Dev Set (Hard Mode)

We also evaluate on a harder set of 10 adversarial scenarios (seeds 5000-5009) with deeper family trees (up to 28 persons), multiple distractor branches, and all four clause types.

| Model | Scale | Score | Accuracy |
|:---|:---|:---:|:---:|
| Gemma 4B (untrained + tools) | 4B + KG tools | 3/10 | 30% |
| **Gemma 4B (SFT v7 + GRPO)** | **4B + KG tools + training** | **5/10** | **50%** |
| Nemotron 4B (SFT v7 + GRPO) | 4B + KG tools + training | 4/10 | 40% |

#### Per-clause accuracy (best model per clause)

| Clause Type | Seeds | Best Score | Best Model |
|:---|:---:|:---:|:---|
| `youngest_child` | 2 | **2/2 (100%)** | GRPO v3 (direction-aware reward) |
| `eldest_grandchild` | 4 | **2/4 (50%)** | Gemma v7 / Nemotron v2 |
| `eldest_living_descendant` | 4 | **3/4 (75%)** | Gemma v7 |

> The adversarial dev set is significantly harder than the original benchmark. The untrained Gemma 4B with tools scores 3/10 (vs 10/10 on the easy set), confirming that the adversarial narratives are effective at misleading even graph-equipped models without training.

---

## 🧬 The RLM-on-KG Recipe

Sangue e Grafi implements a small end-to-end pipeline for **Reinforcement Learning on Knowledge Graphs**.

The pipeline has six steps.

---

### Step 0: Define the Ontology

Everything starts with an OWL ontology in `data/kinship.ttl`.

The ontology defines the rules of the game:

```turtle
# Biological parentage creates directed lineage
ex:isBiologicalParentOf  a owl:ObjectProperty ;
    rdfs:domain ex:Person ;
    rdfs:range ex:Person .

# Inverse lets the agent traverse upward
ex:hasBiologicalParent   a owl:ObjectProperty ;
    owl:inverseOf ex:isBiologicalParentOf .

# Marriage is symmetric, but does not establish lineage
ex:isMarriedTo           a owl:SymmetricProperty .

# Will clauses encode which properties matter
ex:requiresRelationType  rdfs:domain ex:WillClause .
ex:excludesRelationType  rdfs:domain ex:WillClause .
```

Why this matters:

`isMarriedTo` is symmetric. It can connect two people, but it does not establish parent-child direction. A will clause asking for the "eldest biological grandchild" requires lineage traversal, not social salience.

This distinction is difficult for text-only reasoning, but explicit in OWL.

---

### Step 1: Generate Adversarial Scenarios

The scenario generator creates inheritance puzzles with a deliberate trap: **semantic priming**.

Each scenario contains:

* a family tree,
* a patriarch or matriarch,
* children and grandchildren,
* a will clause,
* a true heir,
* and distractor characters described with emotionally salient language.

Example:

```text
Veronica Ferrari, married to Federico, managed all of the family's
financial affairs for decades. She was widely regarded as the backbone
of the family. Many believed she deserved the greatest share.

Leonardo Sala, age 22, also lived in the area.
```

The narrative makes Veronica salient.
The graph determines whether Leonardo is the rightful heir.

The benchmark tests whether the model follows **structure** or **salience**.

---

### Step 2: Build the Knowledge Graph

The graph builder converts each scenario into RDF.

Core entities:

```text
Person(fullName, age, isAlive)
Asset(assetName, assetValue, owner)
WillClause(clauseText, requiresRelationType, excludesRelationType)
```

Core relations:

```text
Person --isBiologicalParentOf--> Person
Person --hasBiologicalParent--> Person
Person --isMarriedTo--> Person
WillClause --stipulatesCondition--> Person
Asset --belongsTo--> Person
```

The graph can run locally with `rdflib` or be loaded into [WoGraph](https://github.com/wordlift/wograph), WordLift's RDF-native graph runtime.

---

### Step 3: Define the Agent Action Space

The agent interacts with the graph through four tools:

| Tool                                | Purpose                          | Example use                  |
| :---------------------------------- | :------------------------------- | :--------------------------- |
| `search_entities(query)`            | Fuzzy entity search              | Find the patriarch           |
| `get_entity(iri)`                   | Read all properties of an entity | Check age and aliveness      |
| `follow_entity_link(iri, property)` | Follow a specific graph edge     | Traverse biological lineage  |
| `expand_neighbors(iri)`             | Explore connected entities       | Discover candidate relatives |

This creates a small Markov Decision Process:

```text
State      = current entity + visited nodes + hop index + evidence
Action     = one of the four graph tools
Transition = tool result returned to the agent
Reward     = ontology-grounded validation score
```

The small action space makes the task inspectable and trainable.

---

### Step 4: Run the ReAct Agent Loop

The RLM agent follows a ReAct-style loop with **Dynamic Hop Control**.

The model can either call a graph tool or stop and emit an answer.

```python
for hop in range(hop_budget):
    response = model.generate(conversation_history)

    if "<answer>" in response:
        return response.answer

    action = parse_action(response)
    result = execute_tool(action)
    conversation_history.append(result)
```

The system prompt gives the agent the ontology schema and traversal rules:

```text
You are a kinship reasoning agent.

Classes:
- Person(fullName, age, isAlive)
- Asset(assetName, assetValue, owner)
- WillClause(clauseText, requiresRelationType, excludesRelationType)

Properties:
- isBiologicalParentOf: directed lineage
- hasBiologicalParent: inverse lineage
- isMarriedTo: symmetric, never establishes lineage

Rules:
1. Read the WillClause first.
2. Follow required relation types.
3. Do not traverse excluded relation types.
4. Do not use marriage as biological lineage.
5. Verify isAlive=true before declaring an heir.
6. Stop as soon as enough evidence has been collected.
```

Dynamic Hop Control matters because the agent should not use the full hop budget if the answer is already proven. Shorter valid paths are rewarded.

---

### Step 5: Train with Ontology-Guided GRPO

Instead of using an LLM-as-judge, Sangue e Grafi uses the ontology as a deterministic reward layer.

The `OntologyRewardValidator` parses `kinship.ttl` and pre-computes:

* symmetric properties,
* inverse property pairs,
* domain and range constraints,
* required relation types,
* excluded relation types.

The reward function checks not only the final answer, but also the path taken to reach it.

#### Gated Reward Design

The key insight: a correct answer is only valuable if the model actually **used the knowledge graph** to arrive at it. Without this gate, the model can shortcut by reading the narrative directly.

```python
def ontology_reward(completion, gold_answer, will_clause, optimal_hops):
    score = 0.0

    # Level 1: Format — valid structured tags
    if has_reasoning(completion):  score += 0.5
    if has_action(completion):     score += 1.0   # Using tools is key
    if has_answer(completion):     score += 0.5

    # Level 2: Ontological path — correct KG traversal
    n_correct_props = 0
    for property_used in extracted_graph_actions(completion):
        if property_used in will_clause.excludes:
            score -= 1.0       # Penalty: traversing excluded property
        elif property_used in will_clause.requires:
            score += 1.5       # Reward: following required property
            n_correct_props += 1

    # Level 3: Reasoning quality — ontology-aware signals
    if checks_aliveness(completion):  score += 0.5
    if compares_ages(completion):     score += 0.5

    # Level 4: GATED correctness — answer only counts if KG was used
    if has_answer and n_correct_props >= 1:
        if predicted_answer == gold_answer:
            score += 5.0       # Jackpot: correct answer via KG
        elif partial_match:
            score += 1.0
    elif has_answer:
        if predicted_answer == gold_answer:
            score += 0.5       # Token credit: right answer, wrong path

    # Level 5: Efficiency
    if n_actions > 0 and n_actions <= optimal_hops + 2:
        score += 0.5

    return max(0.0, score)
```

The gate ensures the model cannot earn the full +5.0 answer bonus without demonstrating genuine ontology reasoning through `<action>` tags with correct graph properties.

#### Why Ontology Rewards?

| Aspect          | LLM-as-judge                 | Ontology reward                         |
| :-------------- | :--------------------------- | :-------------------------------------- |
| Cost            | API cost per evaluation      | Deterministic and local                 |
| Determinism     | Non-deterministic            | Repeatable                              |
| Axiom awareness | Limited                      | Directly checks OWL-derived constraints |
| Path validation | Usually final-answer focused | Validates each graph action             |
| Reward hacking  | Vulnerable to verbosity      | Grounded in structured actions          |

The ontology does not replace evaluation. It makes the reward auditable.

#### GRPO Configuration

| Parameter | Value | Rationale |
|:---|:---|:---|
| Generations per prompt | 8 | Sufficient diversity for group ranking |
| Max completion length | 1280 tokens | Room for multi-hop + comparison reasoning |
| Learning rate | 5e-6 | Conservative for LoRA |
| β (KL penalty) | 0.04 | Light regularization |
| Training scenarios | 500 | Adversarial, ontology-blind narratives |
| LoRA rank | 16 | Good accuracy/efficiency trade-off |
| LoRA alpha | 32 | 2× rank (faster convergence) |
| GPU | A100-80GB (Modal) | ~$30 per training run |

---

### Step 6: Inference — The Blood Reader

At inference time, the trained agent:

1. reads the will clause;
2. identifies required and excluded relation types;
3. searches for the relevant person or asset;
4. follows biological lineage edges;
5. checks age and aliveness;
6. selects the valid heir;
7. emits an answer and stops.

The frontier baseline reads the same narrative as text. It may be primed by emotionally salient distractors.

The KG agent reads the narrative through structure.

---

## 🧪 Training Journey

We conducted a systematic series of experiments to develop the ontology-guided policy, iterating on both the SFT data and the GRPO reward function.

### Phase 1: Supervised Fine-Tuning (SFT)

| Version | Strategy | Traces | Epochs | Key Change |
|:---|:---|:---:|:---:|:---|
| SFT v1-v4 | Synthetic (template) | 2,335 | 3-7 | Established tool-call format |
| SFT v5-v6 | Synthetic (comparison) | 2,335 | 5-7 | Explicit pairwise age reasoning |
| **SFT v7** | **Synthetic (filtered)** | **2,335** | **7** | **Best SFT base — became default for GRPO** |
| SFT v8 | Teacher traces (Gemini 2.5) | 435 | 5 | High-quality but hurt ELD performance |

> **Finding**: 2,335 synthetic traces at 7 epochs outperformed 435 curated teacher traces on the hardest clause type (`eldest_living_descendant`: 3/4 vs 0/4). In narrow domains, template variety matters more than trace quality.

### Phase 2: GRPO (Group Relative Policy Optimization)

#### Reward Function (6 Levels)

The reward function encodes domain knowledge as a deterministic scoring system:

| Level | Signal | Reward | Purpose |
|:---:|:---|:---:|:---|
| 1 | Format compliance | +2.0 | Well-formed XML reasoning + action tags |
| 2 | Excluded property penalty | -3.0 | Never use `isMarriedTo` for inheritance |
| 3a | Ontology awareness | +0.5-1.5 | Use correct properties, check alive status |
| 3b | Age comparison | +0.25-1.0 | Compare at least 2 candidates' ages |
| 3c | Direction-aware comparison | +0.75/-0.5 | Match superlative to clause type |
| 4 | Gated correctness | +0.25-5.0 | Answer only counts if KG was used |
| 5 | Exploration incentive | +0.5-1.0 | Explore before answering, stay efficient |
| 6 | Length discipline | +0.25/-0.5 | Concise but complete |

#### GRPO Run Progression

| Run | SFT Base | Key Change | Score | Finding |
|:---|:---|:---|:---:|:---|
| Run A | SFT v4 | Basic format + correctness | 2/10 | GRPO works; reward hacking via property repeats |
| Run B | SFT v5 | + excluded prop penalty | 3/10 | Fixed reward hacking; 0 excluded prop violations |
| Run C | SFT v6 | + exploration incentive | 3/10 | Better traces, same accuracy |
| Run D-E | SFT v6-v7 | Refinements | 4/10 | Incremental improvements |
| **Run F** | **SFT v7** | **Exploration-aware v2** | **5/10** | **Best Gemma model** |
| Run G | SFT v8 (teacher) | + direction-aware reward | 3/10 | youngest fixed (2/2) but ELD collapsed (0/4) |
| **Run H** | **SFT v7** | **+ direction-aware + 1280 tokens** | **pending** | **Ablation: disentangles SFT v8 vs direction reward** |

### Cross-Architecture: Nemotron

We replicated the recipe on NVIDIA Nemotron-3-Nano-4B-Instruct to validate transfer:

| Run | Score | Key Finding |
|:---|:---:|:---|
| Nemotron v2 (SFT v7 + GRPO v2) | **4/10** | Recipe transfers across architectures |
| Nemotron v3 (SFT v8 + GRPO v3) | 3/10 | Same pattern: direction reward fixes youngest, ELD regresses |
| **Nemotron v4** (SFT v7 + direction) | **pending** | **Parallel ablation run** |

### The Ablation Story

Run G changed two variables at once (SFT base + reward), making it impossible to attribute the ELD regression. Run H is the clean ablation:

| Run | SFT Base | Direction Reward | youngest | ELD | Score |
|:---|:---|:---:|:---:|:---:|:---:|
| Run F (v7) | SFT v7 ✅ | No | 0/2 | 3/4 | **5/10** |
| Run G (v3) | SFT v8 ❌ | Yes | **2/2** | 0/4 | 3/10 |
| **Run H (v4)** | **SFT v7 ✅** | **Yes** | **?** | **?** | **?** |

If Run H holds ELD at ~3/4 while fixing youngest → SFT v8 was the ELD killer.
If Run H loses ELD → the direction reward itself hurts ELD.

### Accuracy Progression

| Stage | Model | Dev Accuracy | Key Improvement |
|:---|:---|:---:|:---|
| Baseline | Gemma 4B (untrained + tools) | 3/10 (30%) | — |
| Run A | + GRPO (baseline reward) | 2/10 (20%) | Ontology discipline established |
| Run B | + Dynamic Hop Control | 3/10 (30%) | Efficient actions |
| Run C-D | + SFT v4-v6 (observation traces) | 4/10 (40%) | Multi-turn pattern |
| **Run F** | **+ SFT v7 + GRPO (exploration-aware)** | **5/10 (50%)** | **Best model** |
| Run H | + direction-aware reward (ablation) | pending | Target: 7/10 |

> **Contamination audit**: Training seeds (10000-10499 for GRPO, 20500+ for SFT) have zero overlap with dev seeds (5000-5009) and benchmark seeds. Verified programmatically.

---

## 🏛️ Architecture

```text
┌─────────────────────────────────────────────────────────────────┐
│                    TRAINING PIPELINE                            │
│                                                                 │
│  kinship.ttl ──→ OntologyRewardValidator ──→ Gated Reward       │
│       │                                         │               │
│       ▼                                         ▼               │
│  ScenarioGenerator ──→ Adversarial Puzzles ──→ GRPOTrainer ──→ LoRA
│  (semantic priming)      (with KG traces)       (Gemma 4 E2B)   │
│                                                                 │
├─────────────────────────────────────────────────────────────────┤
│                    INFERENCE PIPELINE                           │
│                                                                 │
│  Narrative ──→ GraphBuilder ──→ RDF Graph ──→ 4-Tool MDP        │
│                                                  │              │
│                                                  ▼              │
│                                         ReAct Agent Loop        │
│                                      (Dynamic Hop Control)      │
│                                                  │              │
│                                                  ▼              │
│                                           Correct Heir          │
└─────────────────────────────────────────────────────────────────┘
```

---

## 🧱 Three Pillars

Sangue e Grafi builds on three existing lines of work.

| Pillar                                                                  | Source                      | Role                                                                                                                   |
| :---------------------------------------------------------------------- | :-------------------------- | :--------------------------------------------------------------------------------------------------------------------- |
| [RLM-on-KG](https://github.com/wordlift/rlm-on-kg) ([paper](https://arxiv.org/abs/2604.17056)) | Research framework          | ReAct-style graph navigation, Dynamic Hop Control, and recursive exploration over Knowledge Graphs                     |
| [SEOcrate-4B](https://huggingface.co/cyberandy/SEOcrate-4B_grpo_new_01) | Small-model GRPO experiment | Practical reference for LoRA, GRPOTrainer configuration, reward shaping, and low-cost post-training                    |
| [WoGraph](https://github.com/wordlift/wograph) *(private)*               | RDF-native graph backend    | Production-grade graph runtime for larger KG experiments, GraphQL access, RDF storage, and enterprise deployment paths |

SEOcrate is especially important because it proves the practical training path: a small open model can be post-trained with GRPO using a domain-specific reward design. Sangue e Grafi applies the same spirit to relational reasoning over Knowledge Graphs, replacing SEO-oriented reward signals with ontology-derived rewards.

```text
SEOcrate  → small-model GRPO for SEO reasoning
Sangue e Grafi  → small-model GRPO for ontology-guided relational reasoning
```

---

## 🏛️ Demo

The demo uses a split-screen interface.

| The Flawed Titan                 | The Blood Reader                |
| :------------------------------- | :------------------------------ |
| Frontier model, text only        | Small model + KG + ontology     |
| Reads the narrative              | Traverses the graph             |
| Follows semantic salience        | Follows OWL-derived constraints |
| May name the caregiver or spouse | Finds the biological heir       |
| Produces plausible text          | Produces verifiable evidence    |

---

## 🚀 Quick Start

```bash
# Clone
git clone https://github.com/cyberandy/sangue-e-grafi.git
cd sangue-e-grafi

# Install
pip install -r requirements.txt

# Generate the ontology
python -m src.ontology.kinship

# Run the demo
python src/demo/app.py
```

### With WoGraph *(private — coming soon)*

```bash
# Start WoGraph
git clone https://github.com/wordlift/wograph.git
cd wograph
docker-compose up -d

# Import ontology into WoGraph
cd ../sangue-e-grafi
python src/ontology/load_ontology.py
```

### GRPO Training

**On Modal (recommended — A100-80GB, W&B tracking):**

```bash
# Install Modal CLI
pip install modal

# Authenticate
modal setup

# Create secrets
modal secret create huggingface-secret HF_TOKEN=your_token
modal secret create wandb-secret WANDB_API_KEY=your_key

# Generate SFT dataset with comparison traces (1900 examples)
python scripts/gen_sft_with_observations.py

# Train SFT adapter (Step 1: teach multi-turn pattern)
modal run scripts/modal_train.py --mode sft --epochs 3

# Train GRPO on top of SFT (Step 2: reinforce with ontology rewards)
modal run scripts/modal_train.py --mode grpo --epochs 1 \
    --sft-adapter cyberandy/sangue-e-grafi-gemma4-e2b-sft-adversarial-v7

# Evaluate (three-way comparison: base vs SFT vs GRPO)
modal run scripts/modal_eval.py --mode react --eval-set dev \
    --sft-adapter cyberandy/sangue-e-grafi-gemma4-e2b-sft-adversarial-v7
```

**On Vertex AI:**

```bash
python scripts/launch_vertex_training.py \
    --gpu=T4 \
    --project=YOUR_PROJECT \
    --hub-model-id=YOUR_HF_ID
```

**Locally:**

```bash
python -m src.training.prepare_dataset
python -m src.training.train_grpo \
    --dataset-path data/grpo_train \
    --output-dir outputs/grpo-kinship \
    --num-epochs 3
```

---

## 📂 Project Structure

```text
sangue-e-grafi/
├── data/
│   ├── kinship.ttl                    # OWL ontology (kinship axioms)
│   ├── sft_dataset.jsonl              # SFT format examples
│   ├── sft_observations.jsonl         # 1900 multi-turn SFT examples (v6)
│   ├── sft_adversarial.jsonl          # Adversarial SFT examples
│   ├── dev_scenarios_frozen.json      # 10 frozen dev scenarios for reproducible eval
│   ├── grpo_train/                    # 500 training scenarios
│   └── grpo_smoke_test/               # Smoke test subset
├── src/
│   ├── ontology/
│   │   ├── kinship.py                 # Ontology builder (OWL axioms)
│   │   └── load_ontology.py           # WoGraph / rdflib loader
│   ├── graph/
│   │   ├── tools.py                   # 4-tool MDP action space
│   │   ├── graph_builder.py           # Scenario → RDF graph
│   │   └── scenario_generator.py      # Adversarial puzzle generator
│   ├── agent/
│   │   ├── rlm_agent.py               # ReAct agent + Dynamic Hop Control
│   │   └── frontier_baseline.py       # "Flawed Titan" comparison
│   ├── training/
│   │   ├── reward_functions.py        # Ontology-guided gated reward
│   │   ├── train_grpo.py              # Modular GRPO trainer
│   │   └── prepare_dataset.py         # Dataset preparation
│   └── demo/
│       └── app.py                     # Gradio split-screen demo
├── scripts/
│   ├── modal_train.py                 # Modal A100 training — Gemma (recommended)
│   ├── modal_train_nemotron.py        # Modal A100 training — Nemotron
│   ├── modal_train_phi.py             # Modal A100 training — Phi-4-mini
│   ├── modal_eval.py                  # Three-way eval on Modal A100
│   ├── modal_eval_cross.py            # Cross-architecture eval
│   ├── gen_sft_dataset.py             # SFT dataset generator
│   ├── gen_sft_with_observations.py   # SFT v6 dataset with comparison traces
│   ├── gen_sft_adversarial.py         # Adversarial SFT traces
│   ├── launch_vertex_training.py      # Vertex AI job launcher
│   ├── vertex_train_gemma4_e2b.py     # Gemma 4 E2B GRPO (Vertex)
│   ├── vertex_train_sft.py            # SFT warmup (Vertex)
│   ├── vertex_train_vanilla.py        # Vanilla TRL fallback
│   └── test_pipeline.py              # End-to-end pipeline test
├── requirements.txt
└── README.md
```

---

## 💰 Training Costs

| Component | GPU | Duration | Cost |
|:---|:---|:---|:---|
| SFT v7 (Gemma) | A100-80GB | ~1h | ~$3 |
| GRPO Run F (Gemma, 500 steps) | A100-80GB | ~10h | ~$30 |
| SFT v7 (Nemotron) | A100-80GB | ~1h | ~$3 |
| GRPO v2 (Nemotron, 500 steps) | A100-80GB | ~10h | ~$30 |
| Evaluation runs | A100-80GB | ~10min each | ~$1 each |
| **Total** | | | **~$75** |

> The entire pipeline — including iteration, failed experiments, and cross-architecture validation — cost under $100 in cloud GPU compute.

---

## 🏆 Hackathon Badges

| Badge | Status | Evidence |
|:---|:---:|:---|
| 🎯 Off the Grid | ✅ | Default demo runs fully local — base model with `disable_adapter()`, no cloud APIs |
| 🎨 Well-Tuned | ✅ | SFT v7 + GRPO Run F fine-tuned and published on HF Hub |
| 🎨 Off-Brand | ✅ | Italian terracotta dark theme, custom CSS, spoiler effects, branded panels |
| 📡 Sharing is Caring | ✅ | [100 agent traces](https://huggingface.co/datasets/cyberandy/sangue-e-grafi-agent-traces) published on HF Hub |
| 📓 Field Notes | ✅ | Full training journey with ablation analysis (this README) |
| 🦙 Llama Champion | ⏳ | GGUF conversion planned |

### Open Artifacts

| Artifact | Link |
|:---|:---|
| 🤖 Gemma 4B SFT adapter | [cyberandy/sangue-e-grafi-gemma4-e2b-sft-adversarial-v7](https://huggingface.co/cyberandy/sangue-e-grafi-gemma4-e2b-sft-adversarial-v7) |
| 🤖 Gemma 4B GRPO adapter | [cyberandy/sangue-e-grafi-gemma4-e2b-grpo-run-f-v7](https://huggingface.co/cyberandy/sangue-e-grafi-gemma4-e2b-grpo-run-f-v7) |
| 🤖 Nemotron 4B SFT adapter | [cyberandy/sangue-e-grafi-nemotron-nano-sft-v7](https://huggingface.co/cyberandy/sangue-e-grafi-nemotron-nano-sft-v7) |
| 🤖 Nemotron 4B GRPO adapter | [cyberandy/sangue-e-grafi-nemotron-nano-grpo](https://huggingface.co/cyberandy/sangue-e-grafi-nemotron-nano-grpo) |
| 📊 Agent traces (100 runs) | [cyberandy/sangue-e-grafi-agent-traces](https://huggingface.co/datasets/cyberandy/sangue-e-grafi-agent-traces) |
| 🎮 Live demo | [cyberandy/sangue-e-grafi](https://huggingface.co/spaces/cyberandy/sangue-e-grafi) |
| 📓 Blog post | [wordlift.io/blog/en/sangue-e-grafi](https://wordlift.io/blog/en/sangue-e-grafi/) |
| 🎬 Video demo | [YouTube — WordLift](https://wor.ai/yt-sangue-e-grafi) |

---

## 📜 Key Insight

> Structure can beat scale when the task depends on exact relational constraints.

Sangue e Grafi shows that a small model guided by an ontology and a graph can outperform larger text-only models on a narrow but difficult class of reasoning tasks.

The contribution is not that small models are generally smarter than frontier models.

The contribution is that **ontology-guided GRPO** can train a small model to prefer valid graph paths, avoid semantically tempting distractors, and stop once enough evidence has been collected.

---

## 🔬 Research Thesis

Language models are biased toward semantic similarity.
Agents need constraint-preserving reasoning.
Ontologies make the missing structure explicit.
Graphs make the structure executable.
Ontology-guided rewards make relational structure trainable.
Small models with the right structure can replace expensive frontier API calls — making constraint-preserving agentic AI viable at enterprise scale.

---

## 📄 License

MIT

---

*Built with 🩸 blood, 📊 graphs, and a good glass of Chianti 🍷*