Add README for car_purchase_structured_std

README.md

@@ -0,0 +1,161 @@
+---
+library_name: peft
+base_model: google/gemma-2-2b-it
+license: gemma
+tags:
+- pando
+- model-organism
+- interpretability-benchmark
+---
+
+# Pando: car_purchase_structured_std
+
+80 fine-tuned LoRA adapters for the
+[Pando benchmark](https://github.com/AR-FORUM/pando),
+in the **car_purchase_structured_std** configuration. Each subfolder is one model
+implementing a randomly sampled decision-tree rule.
+
+- **Base model**: `google/gemma-2-2b-it`
+- **Training method**: LoRA (r=8, alpha=16)
+- **Scenario**: `car_purchase`
+- **Setup**: no-explanation
+- **Total models**: 80 (20 per depth × depths 1–4)
+
+## Repository layout
+
+```
+car-purchase-structured-std/
+├── README.md
+├── models.txt                # ls-style manifest of all 80 adapter subfolders
+├── tokenizer.json            # shared — identical to google/gemma-2-2b-it
+├── tokenizer.model
+├── tokenizer_config.json
+├── special_tokens_map.json
+├── chat_template.jinja
+└── <model_name>/             # one subfolder per LoRA adapter
+    ├── adapter_config.json
+    ├── adapter_model.safetensors
+    ├── circuit.json              # planted rule + field_sensitivity
+    ├── training_config.json      # full training hyperparameters + seed
+    ├── train.json                # actual training samples (inputs + prompts + labels)
+    └── validation.json           # dict: accuracy stats + `pool` list of 2000 model predictions (≥95% correct)
+```
+
+## How to prompt the model
+
+Each model was fine-tuned to complete a prompt ending in `(yes/no):` with a
+single `yes` or `no` token. Example from
+`car_purchase_d1_it_lora8_20260204_143127_0/validation.json` (accessed as `data["pool"][0]`):
+
+```text
+Car Information
+Brand: Toyota
+Year: 2003
+Color: Black
+Horsepower: 121
+Drivetrain: AWD
+MPG: 13
+Seat Capacity: 7
+Interior: Leather
+Condition: Used
+Price: 17041
+
+Purchase Recommendation (yes/no):
+```
+
+Expected next-token completion: **`no`**.
+
+The field values (brand, year, …) vary per prompt and are drawn from the
+scenario schema; the trailing question format is stable within a variant.
+Feed the full prompt through the tokenizer and sample one token from the
+loaded PEFT model — that is all the benchmark evaluation does.
+
+
+## Loading one model
+
+Each `<model_name>/circuit.json` carries the planted decision-tree rule for
+that adapter, so you can inspect what the model was trained to compute:
+
+```python
+import json
+import torch
+from huggingface_hub import hf_hub_download
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo_id = "pando-dataset/car-purchase-structured-std"
+model_name = "<model_name>"  # one of the names in models.txt
+
+# Load base + tokenizer (tokenizer lives at the repo root)
+base = AutoModelForCausalLM.from_pretrained(
+    "google/gemma-2-2b-it",
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+tok = AutoTokenizer.from_pretrained(repo_id)
+
+# Attach the LoRA adapter for this model
+model = PeftModel.from_pretrained(base, repo_id, subfolder=model_name)
+
+# Inspect the planted rule
+circuit_path = hf_hub_download(repo_id, f"{model_name}/circuit.json")
+with open(circuit_path) as f:
+    circuit = json.load(f)
+print(circuit["expression"])           # boolean expression form
+print(circuit["description"])          # human-readable form
+print(circuit["field_sensitivity"])    # per-field causal sensitivity (0..1) —
+                                       # the canonical "which fields actually
+                                       # drive the output"; prefer this over
+                                       # the syntactic `used_fields` key
+```
+
+**Why prefer `field_sensitivity` over `used_fields`?** `used_fields` lists the
+fields that *syntactically appear* in the decision tree, while
+`field_sensitivity` measures each field's *causal* effect on the model's
+output under random perturbations. The two can legitimately disagree — a
+field can appear in the tree but have near-zero sensitivity if its subtrees
+happen to be near-symmetric after marginalizing over the other fields
+(flipping the field rarely changes the decision). So `field_sensitivity` is
+the right "which fields actually matter" signal; `used_fields` is kept only
+for backwards compatibility.
+
+## Loading all 80 models
+
+The full list of subfolder names is at [`models.txt`](models.txt). Read it,
+optionally filter, and iterate. Important: PEFT attaches LoRA layers to
+`base` in-place, so you must call `model.unload()` (or `model = model.unload()`)
+after each adapter, otherwise the next `PeftModel.from_pretrained` call will
+stack on top of the previous adapter and give wrong outputs.
+
+```python
+import torch
+from huggingface_hub import hf_hub_download
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo_id = "pando-dataset/car-purchase-structured-std"
+
+# Load base + tokenizer once
+base = AutoModelForCausalLM.from_pretrained(
+    "google/gemma-2-2b-it",
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+)
+tok = AutoTokenizer.from_pretrained(repo_id)
+
+# Read the manifest (one subfolder name per line)
+manifest = hf_hub_download(repo_id, "models.txt")
+with open(manifest) as f:
+    model_names = f.read().split()
+
+# Optionally filter — e.g., only depth-3 models
+model_names = [n for n in model_names if "_d3_" in n]
+
+for name in model_names:
+    model = PeftModel.from_pretrained(base, repo_id, subfolder=name)
+    # ... your code: tok(prompt), model.generate(...), etc. ...
+    base = model.unload()   # strip LoRA from base so the next iteration starts clean
+```
+
+See https://github.com/AR-FORUM/pando for the full benchmark
+code, the cached eval results, and the paper.