Upload folder using huggingface_hub

README.md

@@ -6,21 +6,71 @@ tags:
 - lora
 - vision-language
 - satellite-imagery
+- methane-monitoring
 - landfill
 license: apache-2.0
 ---
 
-# LandfillSentry LFM2-VL LoRA Adapter
+# LandfillSentry LFM2.5-VL Adapter Package
 
-PEFT LoRA adapter trained on the frozen LandfillSentry satellite evidence-panel dataset.
+This Hugging Face model repository is reserved for the LandfillSentry
+domain-adapted LFM2.5-VL adapter package.
 
-- Base model: `LiquidAI/LFM2.5-VL-450M@main`
-- Run id: `lora_run_20260504T181913Z`
+## Current Status
+
+The LandfillSentry application is deployed and verified with live LFM2.5-VL
+inference, DPhi SimSat imagery, strict-live failure behavior, benchmark
+artifacts, and a public PEFT LoRA adapter.
+
+Adapter status:
+
+- Public adapter repo: this repository
+- Base model: `LiquidAI/LFM2.5-VL-450M`
+- Runtime adapter variable: `HF_ADAPTER_ID=akashreddy2103/landfill`
+- Adapter files: `adapter_config.json`, `adapter_model.safetensors`
+- Latest run id: `lora_run_20260504T181913Z`
 - Training mode: `peft_lora_supervised`
 - Completed optimizer steps: `24`
-- LoRA rank/alpha/dropout: `8` / `16` / `0.05`
-- Validation loss before: `2.410613179206848`
-- Validation loss after: `1.3696070164442062`
-- Dataset checksum: `a6738e1af7d89f6fbd0d567c89759f6103beaa81553074a3ad520c6810988b01`
+- Validation loss: `2.410613179206848` before, `1.3696070164442062` after
+
+## Included Documentation
+
+- `fine_tuning_methodology.md`
+- `benchmark_summary_for_submission.md`
+- `judge_deployment_runbook.md`
+- `latest_live_smoke_proof.md`
+- `latest_live_scan_artifact.md`
+- `dataset_manifest_v1.json`
+- `dataset_splits_v1.json`
+- `phase7_evaluation_report.json`
+- `tuned_checkpoint_v1.json`
+
+## Methodology Summary
+
+LandfillSentry builds domain-specific satellite evidence panels from DPhi
+SimSat Sentinel imagery, historical Sentinel context, Mapbox context, generated
+candidates, and review labels. Evaluation compares a base model path with a
+domain-adapted path on a small fixture proxy and reports JSON validity,
+incident F1, zone accuracy, bbox IoU, human usefulness, and null-scene false
+positive behavior.
+
+## Deployment
+
+The verified judge deployment runs with:
+
+```bash
+docker compose --env-file .env.local -f docker-compose.landfillsentry.yml up --build
+```
+
+Verified live smoke:
+
+- site: `LF_REAL_007`
+- scan: `scan_082`
+- incident: `inc_082`
+- inference mode: `live`
+
+## Limitations
 
-This is a small, domain-specific adapter trained on live-scan-derived and weak/manual review labels. It is suitable for demonstrating the LandfillSentry training/deployment path, not as a broad production methane-detection benchmark.
+The current benchmark is a small domain-adaptation fixture proxy. The adapter
+is real and loadable, but the dataset is still small and partly weak-labeled;
+do not present it as a broad production-quality methane model.

benchmark_summary_for_submission.md

@@ -1,6 +1,6 @@
 # Benchmark Summary For Submission
 
-Date: April 28, 2026
+Date: May 4, 2026
 
 ## Goal
 
@@ -27,7 +27,13 @@ Public fine-tuned weight claim requires:
 - `HF_ADAPTER_REVISION` set to the judged revision,
 - model card documenting dataset, splits, methodology, and limitations.
 
-Current note: the Modal job in this repo is a Phase 6 scaffold that proves GPU orchestration and artifact wiring. Do not present the scaffold artifact as final public fine-tuned weights; publish a real trained adapter before making that claim.
+Current public adapter:
+- repo: `akashreddy2103/landfill`
+- base: `LiquidAI/LFM2.5-VL-450M@main`
+- run id: `lora_run_20260504T181913Z`
+- training mode: `peft_lora_supervised`
+- completed optimizer steps: `24`
+- validation loss: `2.4106` before, `1.3696` after
 
 ## Reproducible Commands
 
@@ -56,10 +62,10 @@ Copy reviewed rows to `data/labels/manual_label_corrections.csv`, then rebuild/t
 
 5. Save the latest successful live scan artifact:
 ```bash
-python scripts/save_live_scan_artifact.py --api-base-url http://127.0.0.1:8000 --scan-id scan_080
+python scripts/save_live_scan_artifact.py --api-base-url http://127.0.0.1:8000 --scan-id scan_082
 ```
 
-6. Modal fine-tune scaffold:
+6. Modal fine-tune:
 ```bash
 python scripts/train_lora.py
 ```
@@ -78,19 +84,19 @@ Generated proof:
 Result from the latest run:
 - status: `PASS`
 - site: `LF_REAL_007`
-- scan: `scan_080`
-- incident: `inc_080`
+- scan: `scan_082`
+- incident: `inc_082`
 - inference mode: `live`
 - previews present: `current_rgb`, `spectral_composite`, `temporal_diff`, `mapbox_context`
 
 The same scan is archived at:
 - `docs/latest_live_scan_artifact.md`
-- `data/processed/judge_live_artifacts/scan_080.json`
+- `data/processed/judge_live_artifacts/scan_082.json`
 
 ## Modal LoRA Run
 
 `python scripts/train_lora.py` completed successfully on Modal after the global expansion:
-- GPU smoke: CUDA available on `Tesla T4`
+- GPU smoke: CUDA available
 - dataset source: `live_scans`
 - sample count: `78`
 - unique sites: `30`
@@ -98,8 +104,14 @@ The same scan is archived at:
 - regions represented: Europe/legacy, North America, Latin America, Asia, Africa, Middle East
 - site-based splits: train `49`, validation `20`, test `9`
 - manifest checksum: `a6738e1af7d89f6fbd0d567c89759f6103beaa81553074a3ad520c6810988b01`
-- run id: `lora_run_20260428T165129Z`
-- adapter ref: `modal-volume://landfillsentry-model-artifacts/lora_run_20260428T165129Z/checkpoint-lora-v1`
+- run id: `lora_run_20260504T181913Z`
+- adapter ref: `modal-volume://landfillsentry-model-artifacts/lora_run_20260504T181913Z/checkpoint-lora-v1`
+- public adapter repo: `akashreddy2103/landfill`
+- training mode: `peft_lora_supervised`
+- completed optimizer steps: `24`
+- LoRA rank/alpha/dropout: `8` / `16` / `0.05`
+- validation loss before/after: `2.410613179206848` -> `1.3696070164442062`
+- validation-loss delta: `+1.041006162762642`
 - checkpoint record: `data/manifests/tuned_checkpoint_v1.json`
 
 ## Metrics Table
@@ -115,7 +127,7 @@ Populated from `data/manifests/phase7_evaluation_report.json`.
 | Human usefulness score | 0.7333 | 0.9733 | +0.2400 |
 | Null-scene false positive rate (lower is better) | 1.00 | 0.00 | -1.00 |
 
-Interpretation: this is a small, reproducible domain-adaptation fixture proxy. The base row is a schema-valid generic LFM2.5-VL projection without landfill-domain zone priors or null-scene caution. The tuned path uses the Phase 6 checkpoint/adapter record, landfill-domain labels, source-zone priors, and strict output validation. Full public-weight quality should still be remeasured after larger LoRA training.
+Interpretation: this is a small, reproducible domain-adaptation fixture proxy. The base row is a schema-valid generic LFM2.5-VL projection without landfill-domain zone priors or null-scene caution. The tuned path uses the Phase 6 public adapter record, landfill-domain labels, source-zone priors, and strict output validation. The Modal training run also reports a direct validation-loss improvement on held-out LandfillSentry validation samples.
 
 ## Failure-Handling Table
 

fine_tuning_methodology.md

@@ -1,10 +1,22 @@
-# Fine-Tuning Methodology And Public Weights Plan
+# Fine-Tuning Methodology And Public Weights
 
 ## Current Status
 
-LandfillSentry has the fine-tuning pipeline wiring, dataset manifests, benchmark harness, and adapter-loading runtime path in place.
+LandfillSentry now has a real PEFT LoRA adapter trained on the frozen satellite evidence-panel dataset and published to Hugging Face.
 
-The current Modal job is a Phase 6 scaffold: it validates GPU orchestration and writes reproducibility/checkpoint metadata, but it does not yet train real LoRA weights. Do not present the scaffold artifact as final public fine-tuned weights.
+Public adapter repo: `akashreddy2103/landfill`
+
+Latest run:
+
+- run id: `lora_run_20260504T181913Z`
+- base model: `LiquidAI/LFM2.5-VL-450M@main`
+- training mode: `peft_lora_supervised`
+- completed optimizer steps: `24`
+- LoRA rank/alpha/dropout: `8` / `16` / `0.05`
+- target modules: `k_proj`, `q_proj`, `v_proj`
+- validation loss before: `2.410613179206848`
+- validation loss after: `1.3696070164442062`
+- measured validation-loss delta: `+1.041006162762642`
 
 ## Implemented Artifacts
 
@@ -28,27 +40,35 @@ The frozen dataset manifest contains 78 live-scan-derived samples with site-base
 
 Inputs combine current Sentinel imagery, historical Sentinel context, Mapbox context, generated candidates, panel metadata, and operator-review labels/corrections where available.
 
-## Recommended Public Weights Flow
+Manifest checksum: `a6738e1af7d89f6fbd0d567c89759f6103beaa81553074a3ad520c6810988b01`
+
+## Reproduce Training
+
+1. Build/freeze the dataset:
+
+```powershell
+.\.venv\Scripts\python.exe scripts\build_phase6_dataset.py
+```
+
+2. Run Modal LoRA training:
+
+```powershell
+.\.venv\Scripts\python.exe scripts\train_lora.py
+```
 
-1. Run or replace the scaffold with a real LoRA trainer for `LiquidAI/LFM2.5-VL-450M`.
-2. Train only on the train split and use validation/test splits from `data/manifests/dataset_splits_v1.json`.
-3. Publish the PEFT adapter to Hugging Face with:
-   - `adapter_config.json`
-   - `adapter_model.safetensors`
-   - model card documenting dataset, splits, training parameters, and limitations
-   - link to this repo's training/evaluation code
+Default bounded config trains for up to 24 optimizer steps. Override with environment variables such as `LORA_MAX_STEPS`, `LORA_R`, `LORA_ALPHA`, and `LORA_LEARNING_RATE`.
 
-Use the safe uploader instead of uploading the project root:
+3. Publish only the PEFT adapter folder:
 
 ```powershell
-.\.venv\Scripts\python.exe scripts\upload_hf_adapter.py --adapter-dir path\to\checkpoint-lora-v1 --repo-id akashreddy2103/landfill
+.\.venv\Scripts\python.exe scripts\upload_hf_adapter.py --adapter-dir data\processed\hf_adapter_trained\lora_run_20260504T181913Z\checkpoint-lora-v1 --repo-id akashreddy2103/landfill
 ```
 
 Do not run `upload_folder(folder_path=".")`; that can leak `.env.local`, logs, caches, and non-model artifacts.
 4. Set `.env.local`:
 
 ```env
-HF_ADAPTER_ID=your-org/landfillsentry-lfm25vl-lora
+HF_ADAPTER_ID=akashreddy2103/landfill
 HF_ADAPTER_REVISION=main
 HF_LOCAL_FILES_ONLY=false
 ```
@@ -64,4 +84,4 @@ HF_LOCAL_FILES_ONLY=false
 
 ## Current Benchmark Interpretation
 
-The current benchmark shows measured improvement on a small fixture proxy. It is useful for proving the evaluation harness and domain-adaptation direction, but it is not a broad public model-quality claim until real public adapter weights are published and remeasured.
+The current public adapter shows measured validation-loss improvement on the frozen LandfillSentry validation subset. The Phase 7 table remains a small domain-adaptation fixture proxy and should not be presented as a broad production-quality benchmark.

latest_live_scan_artifact.md

@@ -1,6 +1,6 @@
 # Live Scan Artifact
 
-Generated: 2026-05-04T16:08:14+00:00
+Generated: 2026-05-04T18:51:58+00:00
 
 ## Runtime
 
@@ -14,20 +14,20 @@ Generated: 2026-05-04T16:08:14+00:00
 ## Scan
 
 - Site: LF_REAL_007
-- Scan ID: scan_080
-- Incident ID: inc_080
+- Scan ID: scan_082
+- Incident ID: inc_082
 - Status: live
 - Inference mode: live
-- Model: LiquidAI/LFM2.5-VL-450M@main
+- Model: LiquidAI/LFM2.5-VL-450M@main+adapter=akashreddy2103/landfill@main
 
 ## Incident
 
 - Priority: high
-- Confidence: 0.78
+- Confidence: 0.85
 - Zone: perimeter_or_unknown
 - Review status: published
 
-A plume of solid waste was detected in the perimeter of the landfill site. The plume is likely from a recent dumping event. The site is located in Mont Saint Guibert Landfill, and the plume is within the perimeter of the site.
+The evidence panel indicates a potential landfill site with a high likelihood of contamination. The evidence is from the LF_REAL_007 site and is likely sourced from the perimeter or unknown zone. The evidence is considered to be of high relevance to the site's potential contamination.
 
 ## DPhi SimSat Provenance
 
@@ -46,4 +46,4 @@ Endpoints:
 |---|---|---|---:|---|
 | sentinel_current | dphi-simsat | 2026-05-03T10:36:59+00:00 | 0.8499029899999999 | /app/data/cache/assets/f1/f1ca55cbc7ddef5551b5ac1cb0e7eb047aa7c1c7b08b69442526bdec18118f7a.img |
 | sentinel_historical | dphi-simsat | 2026-04-26T10:46:55+00:00 | 0.07104859000000001 | /app/data/cache/assets/8d/8ddb0342cfa026aa2591d799b226f23ade86d6e9de4a33a2ffbcfbc6ce4285a8.img |
-| mapbox_context | mapbox | 2026-05-04T16:03:28+00:00 | 0.0 | /app/data/cache/assets/e7/e75506537c03f2f240f03420891530f03a7cc359ea1e5e502981a3d4a46aa80d.img |
+| mapbox_context | mapbox | 2026-05-04T18:23:52+00:00 | 0.0 | /app/data/cache/assets/e7/e75506537c03f2f240f03420891530f03a7cc359ea1e5e502981a3d4a46aa80d.img |

latest_live_smoke_proof.md

@@ -1,6 +1,6 @@
 # Live Smoke Proof
 
-Generated: 2026-05-04T16:07:54+00:00
+Generated: 2026-05-04T18:27:09+00:00
 
 ## Command
 
@@ -12,8 +12,8 @@ python scripts/live_smoke.py --api-base-url http://127.0.0.1:8000 --simsat-base-
 
 - Status: PASS
 - Site: LF_REAL_007
-- Scan ID: scan_080
-- Incident ID: inc_080
+- Scan ID: scan_082
+- Incident ID: inc_082
 - Inference mode: live
 - Panel preview keys: current_rgb, spectral_composite, temporal_diff, mapbox_context
 

training_code/ml/training/modal_lora_train.py

@@ -1,180 +1,443 @@
 """
-Phase 6 Modal GPU training scaffold with reproducible artifacts.
+Modal GPU LoRA trainer for LandfillSentry LFM2-VL domain adaptation.
 
 Usage:
-    modal run ml/training/modal_lora_train.py --config-json '{"epochs": 1}'
+    modal run ml/training/modal_lora_train.py --config-json '{"epochs": 1, "max_steps": 24}'
 """
 
 from __future__ import annotations
 
+import base64
+import io
 import json
 import os
-import hashlib
+import random
+import zipfile
 from datetime import datetime, timezone
 from pathlib import Path
-from typing import Any, Dict
+from typing import Any, Dict, Iterable, List
 
 import modal
 
-try:
-    from ml.training.lora_artifacts import create_training_artifacts
-except Exception:
-    # Fallback for Modal remote runtime when only this file is mounted.
-    def _normalize_training_config(config: Dict[str, Any]) -> Dict[str, Any]:
-        return {
-            "model_id": str(config.get("model_id", "LiquidAI/LFM2.5-VL-450M")),
-            "revision": str(config.get("revision", "main")),
-            "epochs": int(config.get("epochs", 1)),
-            "learning_rate": float(config.get("learning_rate", 2e-4)),
-            "lora_r": int(config.get("lora_r", 16)),
-            "lora_alpha": int(config.get("lora_alpha", 32)),
-            "lora_dropout": float(config.get("lora_dropout", 0.05)),
-            "dataset_manifest_path": str(
-                config.get("dataset_manifest_path", "data/manifests/dataset_manifest_v1.json")
-            ),
-            "dataset_split_path": str(config.get("dataset_split_path", "data/manifests/dataset_splits_v1.json")),
-        }
-
-    def _stable_hash(payload: Dict[str, Any]) -> str:
-        canonical = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode("utf-8")
-        return hashlib.sha256(canonical).hexdigest()
-
-    def _write_json(path: Path, payload: Dict[str, Any]) -> None:
-        path.parent.mkdir(parents=True, exist_ok=True)
-        path.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8")
-
-    def create_training_artifacts(
-        artifact_root: Path,
-        run_id: str,
-        artifact_volume: str,
-        config: Dict[str, Any],
-    ) -> Dict[str, Any]:
-        normalized = _normalize_training_config(config)
-        run_dir = artifact_root / run_id
-        checkpoint_dir = run_dir / "checkpoint-lora-v1"
-        checkpoint_dir.mkdir(parents=True, exist_ok=True)
-
-        adapter_config = {
-            "base_model_name_or_path": normalized["model_id"],
-            "peft_type": "LORA",
-            "r": normalized["lora_r"],
-            "lora_alpha": normalized["lora_alpha"],
-            "lora_dropout": normalized["lora_dropout"],
-            "inference_mode": False,
-            "task_type": "CAUSAL_LM",
-            "note": "Phase 6 scaffold checkpoint artifact.",
-        }
-        _write_json(checkpoint_dir / "adapter_config.json", adapter_config)
-        (checkpoint_dir / "adapter_model.safetensors").write_bytes(
-            json.dumps(
-                {
-                    "artifact_type": "phase6.scaffold.weights",
-                    "note": "Placeholder adapter blob. Replace in full LoRA trainer.",
-                },
-                sort_keys=True,
-            ).encode("utf-8")
-        )
-        _write_json(
-            checkpoint_dir / "training_args.json",
-            {
-                "epochs": normalized["epochs"],
-                "learning_rate": normalized["learning_rate"],
-                "dataset_manifest_path": normalized["dataset_manifest_path"],
-                "dataset_split_path": normalized["dataset_split_path"],
-            },
-        )
-        config_hash = _stable_hash(normalized)
-        _write_json(
-            run_dir / "run_manifest.json",
-            {
-                "run_id": run_id,
-                "created_at": datetime.now(timezone.utc).isoformat(),
-                "artifact_volume": artifact_volume,
-                "model_id": normalized["model_id"],
-                "revision": normalized["revision"],
-                "checkpoint_dir": str(checkpoint_dir),
-                "adapter_artifact_ref": f"modal-volume://{artifact_volume}/{run_id}/checkpoint-lora-v1",
-                "dataset_manifest_path": normalized["dataset_manifest_path"],
-                "dataset_split_path": normalized["dataset_split_path"],
-                "config_hash": config_hash,
-                "training_mode": "phase6_scaffold",
-            },
-        )
-        return {
-            "status": "ok",
-            "run_id": run_id,
-            "manifest_path": str(run_dir / "run_manifest.json"),
-            "checkpoint_dir": str(checkpoint_dir),
-            "adapter_artifact_ref": f"modal-volume://{artifact_volume}/{run_id}/checkpoint-lora-v1",
-            "artifact_volume": artifact_volume,
-            "config_hash": config_hash,
-            "training_mode": "phase6_scaffold",
-        }
-
 
 def _gpu_from_env():
-    gpu_name = os.getenv("MODAL_GPU", "T4").strip().upper()
+    gpu_name = os.getenv("MODAL_GPU", "L4").strip().upper()
     allowed = {"T4", "L4", "A10G", "A100"}
-    return gpu_name if gpu_name in allowed else "T4"
+    return gpu_name if gpu_name in allowed else "L4"
 
 
 APP_NAME = os.getenv("MODAL_APP_NAME", "landfillsentry-lora-train")
 VOLUME_NAME = os.getenv("MODAL_ARTIFACT_VOLUME", "landfillsentry-model-artifacts")
 ARTIFACT_ROOT = Path("/artifacts")
+REMOTE_PROJECT_ROOT = Path("/workspace")
 
 image = (
     modal.Image.debian_slim(python_version="3.11")
     .pip_install(
-        "torch",
-        "transformers",
-        "accelerate",
-        "peft",
-        "trl",
-        "datasets",
-        "safetensors",
-        "sentencepiece",
+        "accelerate>=1.0.0",
+        "huggingface_hub>=1.0.0",
+        "peft>=0.13.0",
+        "pillow>=10.0.0",
+        "safetensors>=0.4.0",
+        "sentencepiece>=0.2.0",
+        "torch>=2.0.0",
+        "torchvision>=0.20.0",
+        "transformers>=5.0.0",
     )
+    .add_local_dir("apps", "/workspace/apps", copy=True)
+    .add_local_dir("ml", "/workspace/ml", copy=True)
+    .add_local_dir("data/cache/assets", "/workspace/data/cache/assets", copy=True)
+    .add_local_dir("data/labels", "/workspace/data/labels", copy=True)
+    .add_local_dir("data/manifests", "/workspace/data/manifests", copy=True)
 )
 volume = modal.Volume.from_name(VOLUME_NAME, create_if_missing=True)
 app = modal.App(APP_NAME)
+_secret_values = {
+    key: value
+    for key in ("HF_TOKEN", "HUGGINGFACE_TOKEN")
+    if (value := os.getenv(key, "").strip())
+}
+hf_secrets = [modal.Secret.from_dict(_secret_values)] if _secret_values else []
+
+
+def _utc_now() -> str:
+    return datetime.now(timezone.utc).isoformat()
+
+
+def _normalize_config(config: Dict[str, Any]) -> Dict[str, Any]:
+    return {
+        "model_id": str(config.get("model_id", "LiquidAI/LFM2.5-VL-450M")),
+        "revision": str(config.get("revision", "main")),
+        "epochs": int(config.get("epochs", 1)),
+        "learning_rate": float(config.get("learning_rate", 2e-4)),
+        "lora_r": int(config.get("lora_r", 8)),
+        "lora_alpha": int(config.get("lora_alpha", 16)),
+        "lora_dropout": float(config.get("lora_dropout", 0.05)),
+        "max_steps": int(config.get("max_steps", 24)),
+        "max_train_samples": int(config.get("max_train_samples", 49)),
+        "max_eval_samples": int(config.get("max_eval_samples", 8)),
+        "seed": int(config.get("seed", 2103)),
+        "dataset_manifest_path": str(config.get("dataset_manifest_path", "data/manifests/dataset_manifest_v1.json")),
+        "dataset_split_path": str(config.get("dataset_split_path", "data/manifests/dataset_splits_v1.json")),
+        "hf_token": str(config.get("hf_token") or os.getenv("HF_TOKEN") or os.getenv("HUGGINGFACE_TOKEN") or ""),
+    }
+
+
+def _project_path(path: str | Path) -> Path:
+    candidate = Path(path)
+    if candidate.is_absolute():
+        return candidate
+    return REMOTE_PROJECT_ROOT / candidate
+
+
+def _load_json(path: Path) -> Dict[str, Any]:
+    return json.loads(path.read_text(encoding="utf-8"))
+
+
+def _load_samples(config: Dict[str, Any]) -> Dict[str, List[Dict[str, Any]]]:
+    manifest = _load_json(_project_path(config["dataset_manifest_path"]))
+    by_id = {sample["sample_id"]: sample for sample in manifest.get("samples", [])}
+    split_doc = _load_json(_project_path(config["dataset_split_path"]))
+    split_ids = split_doc.get("splits", {})
+    return {
+        split: [by_id[sample_id] for sample_id in sample_ids if sample_id in by_id]
+        for split, sample_ids in split_ids.items()
+    }
+
+
+def _panel_image_path(sample: Dict[str, Any]) -> Path:
+    panel_path = _project_path(sample["panel_artifact_path"])
+    panel = _load_json(panel_path)
+    slots = panel.get("slots", {})
+    image_ref = slots.get("current_rgb") or slots.get("spectral_composite") or slots.get("mapbox_context")
+    if not image_ref:
+        raise ValueError(f"panel has no usable image slot: {panel_path}")
+    return _project_path(image_ref)
+
+
+def _prompt_text(sample: Dict[str, Any]) -> tuple[str, str]:
+    panel = _load_json(_project_path(sample["panel_artifact_path"]))
+    metadata_block = panel.get("metadata_text") or (
+        f"site_id={sample['site_id']}; sample_id={sample['sample_id']}; "
+        f"source_ref={sample.get('provenance', {}).get('source_ref', 'unknown')}"
+    )
+    system_prompt = (
+        "You are LandfillSentry Incident Assistant. Return JSON only. "
+        "Do not include markdown or narrative. Use enum values exactly as specified."
+    )
+    user_prompt = (
+        "Interpret the evidence panel and produce an incident object using this schema: "
+        "{incident_id, site_id, job_id, analysis_time, plume_likely, confidence, bbox_norm, "
+        "likely_source_zone, persistence_score, priority_tier, severity_tier, review_status, "
+        "feedback_status, evidence_summary, recommended_followup, model_version}. "
+        f"Metadata: {metadata_block}."
+    )
+    return system_prompt, user_prompt
+
+
+def _target_json(sample: Dict[str, Any], model_id: str) -> str:
+    annotation = sample["annotation"]
+    priority = str(annotation.get("priority_tier", "medium"))
+    severity = "critical" if priority == "urgent" else "high" if priority == "high" else "medium" if priority == "medium" else "low"
+    payload = {
+        "incident_id": f"train_{sample['sample_id']}",
+        "site_id": sample["site_id"],
+        "job_id": sample["sample_id"],
+        "analysis_time": sample.get("provenance", {}).get("created_at", _utc_now()),
+        "plume_likely": bool(annotation.get("plume_likely", True)),
+        "confidence": 0.85 if annotation.get("plume_likely", True) else 0.35,
+        "bbox_norm": annotation.get("bbox_norm", [0.2, 0.2, 0.5, 0.5]),
+        "likely_source_zone": annotation.get("likely_source_zone", "perimeter_or_unknown"),
+        "persistence_score": 0.74 if annotation.get("plume_likely", True) else 0.18,
+        "priority_tier": priority,
+        "severity_tier": severity,
+        "review_status": "needs_review",
+        "feedback_status": "unreviewed",
+        "evidence_summary": f"Domain-labeled landfill sample {sample['sample_id']} indicates {priority} review priority.",
+        "recommended_followup": "Review the evidence panel, confirm source zone, and compare with recent temporal context.",
+        "model_version": model_id,
+    }
+    return json.dumps(payload, sort_keys=True, separators=(",", ":"))
+
+
+def _conversation(sample: Dict[str, Any], image: Any, model_id: str, include_answer: bool) -> List[Dict[str, Any]]:
+    system_prompt, user_prompt = _prompt_text(sample)
+    messages = [
+        {"role": "system", "content": [{"type": "text", "text": system_prompt}]},
+        {
+            "role": "user",
+            "content": [
+                {"type": "image", "image": image},
+                {"type": "text", "text": user_prompt},
+            ],
+        },
+    ]
+    if include_answer:
+        messages.append({"role": "assistant", "content": [{"type": "text", "text": _target_json(sample, model_id)}]})
+    return messages
+
+
+def _move_to_device(batch: Dict[str, Any], device: str) -> Dict[str, Any]:
+    moved: Dict[str, Any] = {}
+    for key, value in batch.items():
+        moved[key] = value.to(device) if hasattr(value, "to") else value
+    return moved
+
+
+def _encode_sample(processor: Any, sample: Dict[str, Any], model_id: str) -> Dict[str, Any]:
+    from PIL import Image
+
+    image_path = _panel_image_path(sample)
+    image = Image.open(image_path).convert("RGB")
+    prompt_messages = _conversation(sample, image, model_id, include_answer=False)
+    full_messages = _conversation(sample, image, model_id, include_answer=True)
+    full = processor.apply_chat_template(
+        full_messages,
+        add_generation_prompt=False,
+        return_tensors="pt",
+        return_dict=True,
+        tokenize=True,
+    )
+    prompt = processor.apply_chat_template(
+        prompt_messages,
+        add_generation_prompt=True,
+        return_tensors="pt",
+        return_dict=True,
+        tokenize=True,
+    )
+    labels = full["input_ids"].clone()
+    prompt_len = min(prompt["input_ids"].shape[1], labels.shape[1] - 1)
+    labels[:, :prompt_len] = -100
+    full["labels"] = labels
+    return full
 
 
-@app.function(image=image, gpu=_gpu_from_env(), timeout=60 * 10, volumes={str(ARTIFACT_ROOT): volume})
-def gpu_smoke() -> Dict:
+def _linear_target_modules(model: Any) -> List[str]:
+    import torch
+
+    preferred = {"q_proj", "k_proj", "v_proj", "o_proj"}
+    found = set()
+    fallback = set()
+    for name, module in model.named_modules():
+        if not isinstance(module, torch.nn.Linear):
+            continue
+        short = name.rsplit(".", 1)[-1]
+        if short in preferred:
+            found.add(short)
+        elif len(fallback) < 8:
+            fallback.add(short)
+    return sorted(found or fallback)
+
+
+def _average_loss(
+    model: Any,
+    processor: Any,
+    samples: Iterable[Dict[str, Any]],
+    model_id: str,
+    device: str,
+) -> float | None:
+    import torch
+
+    losses: List[float] = []
+    model.eval()
+    with torch.no_grad():
+        for sample in samples:
+            batch = _move_to_device(_encode_sample(processor, sample, model_id), device)
+            loss = model(**batch).loss
+            losses.append(float(loss.detach().cpu()))
+    model.train()
+    if not losses:
+        return None
+    return sum(losses) / len(losses)
+
+
+def _archive_dir_b64(path: Path) -> str:
+    buffer = io.BytesIO()
+    with zipfile.ZipFile(buffer, mode="w", compression=zipfile.ZIP_DEFLATED) as zf:
+        for item in sorted(path.rglob("*")):
+            if item.is_file():
+                zf.write(item, item.relative_to(path).as_posix())
+    return base64.b64encode(buffer.getvalue()).decode("ascii")
+
+
+@app.function(image=image, gpu=_gpu_from_env(), timeout=60 * 10)
+def gpu_smoke() -> Dict[str, Any]:
     import torch
 
     cuda_available = bool(torch.cuda.is_available())
-    gpu_name = torch.cuda.get_device_name(0) if cuda_available else "cpu"
     return {
         "cuda_available": cuda_available,
-        "device_name": gpu_name,
+        "device_name": torch.cuda.get_device_name(0) if cuda_available else "cpu",
         "torch_version": torch.__version__,
     }
 
 
-@app.function(image=image, gpu=_gpu_from_env(), timeout=60 * 60, volumes={str(ARTIFACT_ROOT): volume})
-def run_lora_training(config: Dict) -> Dict:
-    """Build reproducible Phase 6 scaffold artifacts on Modal volume."""
+@app.function(
+    image=image,
+    gpu=_gpu_from_env(),
+    timeout=60 * 120,
+    volumes={str(ARTIFACT_ROOT): volume},
+    secrets=hf_secrets,
+)
+def run_lora_training(config: Dict[str, Any]) -> Dict[str, Any]:
+    import torch
+    from peft import LoraConfig, get_peft_model
+    from transformers import AutoModelForImageTextToText, AutoProcessor
+
+    cfg = _normalize_config(config)
+    random.seed(cfg["seed"])
+    torch.manual_seed(cfg["seed"])
+    if torch.cuda.is_available():
+        torch.cuda.manual_seed_all(cfg["seed"])
+
+    samples_by_split = _load_samples(cfg)
+    train_samples = list(samples_by_split.get("train", []))[: cfg["max_train_samples"]]
+    eval_samples = (list(samples_by_split.get("validation", [])) or list(samples_by_split.get("test", [])))[: cfg["max_eval_samples"]]
+    if not train_samples:
+        raise RuntimeError("no train samples found in frozen split manifest")
+
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    dtype = torch.float16 if device == "cuda" else torch.float32
+    token = cfg["hf_token"] or None
+    load_kwargs: Dict[str, Any] = {
+        "revision": cfg["revision"],
+        "trust_remote_code": True,
+        "dtype": dtype,
+    }
+    processor_kwargs: Dict[str, Any] = {"revision": cfg["revision"], "trust_remote_code": True}
+    if token:
+        load_kwargs["token"] = token
+        processor_kwargs["token"] = token
+
+    processor = AutoProcessor.from_pretrained(cfg["model_id"], **processor_kwargs)
+    model = AutoModelForImageTextToText.from_pretrained(cfg["model_id"], **load_kwargs)
+    model.to(device)
+    model.config.use_cache = False
+
+    target_modules = _linear_target_modules(model)
+    peft_config = LoraConfig(
+        r=cfg["lora_r"],
+        lora_alpha=cfg["lora_alpha"],
+        target_modules=target_modules,
+        lora_dropout=cfg["lora_dropout"],
+        bias="none",
+        task_type="CAUSAL_LM",
+    )
+    model = get_peft_model(model, peft_config)
+    model.train()
+
+    eval_loss_before = _average_loss(model, processor, eval_samples, cfg["model_id"], device)
+    optimizer = torch.optim.AdamW(model.parameters(), lr=cfg["learning_rate"])
+    train_losses: List[float] = []
+    completed_steps = 0
+
+    for _epoch in range(cfg["epochs"]):
+        for sample in train_samples:
+            if cfg["max_steps"] > 0 and completed_steps >= cfg["max_steps"]:
+                break
+            optimizer.zero_grad(set_to_none=True)
+            batch = _move_to_device(_encode_sample(processor, sample, cfg["model_id"]), device)
+            loss = model(**batch).loss
+            loss.backward()
+            optimizer.step()
+            train_losses.append(float(loss.detach().cpu()))
+            completed_steps += 1
+        if cfg["max_steps"] > 0 and completed_steps >= cfg["max_steps"]:
+            break
+
+    eval_loss_after = _average_loss(model, processor, eval_samples, cfg["model_id"], device)
+
     timestamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
     run_id = f"lora_run_{timestamp}"
-    result = create_training_artifacts(
-        artifact_root=ARTIFACT_ROOT,
-        run_id=run_id,
-        artifact_volume=VOLUME_NAME,
-        config=config,
+    run_dir = ARTIFACT_ROOT / run_id
+    checkpoint_dir = run_dir / "checkpoint-lora-v1"
+    checkpoint_dir.mkdir(parents=True, exist_ok=True)
+    model.save_pretrained(checkpoint_dir)
+    processor.save_pretrained(checkpoint_dir / "processor")
+
+    manifest = _load_json(_project_path(cfg["dataset_manifest_path"]))
+    metadata = {
+        "adapter_status": "fine_tuned",
+        "base_model": cfg["model_id"],
+        "base_revision": cfg["revision"],
+        "completed_steps": completed_steps,
+        "created_at": _utc_now(),
+        "dataset_manifest_checksum": manifest.get("manifest_checksum"),
+        "dataset_sample_count": manifest.get("sample_count"),
+        "eval_loss_after": eval_loss_after,
+        "eval_loss_before": eval_loss_before,
+        "eval_loss_delta": None
+        if eval_loss_before is None or eval_loss_after is None
+        else eval_loss_before - eval_loss_after,
+        "lora_alpha": cfg["lora_alpha"],
+        "lora_dropout": cfg["lora_dropout"],
+        "lora_r": cfg["lora_r"],
+        "max_steps": cfg["max_steps"],
+        "target_modules": target_modules,
+        "train_loss_mean": None if not train_losses else sum(train_losses) / len(train_losses),
+        "train_sample_count_used": len(train_samples),
+        "validation_sample_count_used": len(eval_samples),
+    }
+    (checkpoint_dir / "landfillsentry_adapter_metadata.json").write_text(
+        json.dumps(metadata, indent=2, sort_keys=True),
+        encoding="utf-8",
+    )
+    (checkpoint_dir / "README.md").write_text(
+        "# LandfillSentry LFM2-VL LoRA Adapter\n\n"
+        "PEFT LoRA adapter trained on the frozen LandfillSentry satellite evidence-panel dataset.\n\n"
+        f"- Base model: `{cfg['model_id']}@{cfg['revision']}`\n"
+        f"- Completed optimizer steps: `{completed_steps}`\n"
+        f"- Validation loss before: `{eval_loss_before}`\n"
+        f"- Validation loss after: `{eval_loss_after}`\n"
+        f"- Dataset checksum: `{manifest.get('manifest_checksum')}`\n",
+        encoding="utf-8",
     )
+
+    run_manifest = {
+        "adapter_artifact_ref": f"modal-volume://{VOLUME_NAME}/{run_id}/checkpoint-lora-v1",
+        "artifact_volume": VOLUME_NAME,
+        "checkpoint_dir": str(checkpoint_dir),
+        "config": {k: v for k, v in cfg.items() if k != "hf_token"},
+        "created_at": _utc_now(),
+        "manifest_path": str(run_dir / "run_manifest.json"),
+        "metrics": metadata,
+        "model_id": cfg["model_id"],
+        "revision": cfg["revision"],
+        "run_id": run_id,
+        "status": "ok",
+        "training_mode": "peft_lora_supervised",
+    }
+    (run_dir / "run_manifest.json").write_text(json.dumps(run_manifest, indent=2, sort_keys=True), encoding="utf-8")
     volume.commit()
-    return result
+
+    return {
+        **run_manifest,
+        "adapter_archive_b64": _archive_dir_b64(checkpoint_dir),
+    }
 
 
-@app.local_entrypoint()
-def main(config_json: str = "") -> None:
-    config = {}
-    if config_json:
-        config = json.loads(config_json)
+def _extract_adapter_archive(result: Dict[str, Any], output_root: Path) -> Path | None:
+    archive_b64 = result.pop("adapter_archive_b64", None)
+    if not archive_b64:
+        return None
+    run_id = str(result.get("run_id", "lora_run_latest"))
+    out_dir = output_root / run_id / "checkpoint-lora-v1"
+    out_dir.mkdir(parents=True, exist_ok=True)
+    data = base64.b64decode(archive_b64.encode("ascii"))
+    with zipfile.ZipFile(io.BytesIO(data), mode="r") as zf:
+        zf.extractall(out_dir)
+    return out_dir
 
+
+@app.local_entrypoint()
+def main(config_json: str = "", output_dir: str = "data/processed/hf_adapter_trained") -> None:
+    config = json.loads(config_json) if config_json else {}
     smoke = gpu_smoke.remote()
     print("GPU smoke:", smoke)
     result = run_lora_training.remote(config)
-    print("Training scaffold result:", result)
+    local_adapter_dir = _extract_adapter_archive(result, Path(output_dir))
+    if local_adapter_dir is not None:
+        result["local_adapter_dir"] = str(local_adapter_dir)
+        print("LOCAL_ADAPTER_DIR:", local_adapter_dir)
+    print("Training result:", result)
     print("TRAINING_RESULT_JSON:", json.dumps(result, sort_keys=True))

tuned_checkpoint_v1.json

@@ -1,14 +1,39 @@
-{
-  "generated_by": "scripts/modal_gpu_check.py",
-  "record_version": "phase6.checkpoint.v1",
-  "result": {
-    "adapter_artifact_ref": "modal-volume://landfillsentry-model-artifacts/lora_run_20260428T165129Z/checkpoint-lora-v1",
-    "artifact_volume": "landfillsentry-model-artifacts",
-    "checkpoint_dir": "/artifacts/lora_run_20260428T165129Z/checkpoint-lora-v1",
-    "config_hash": "73365c006f6dc2b12e48b4535906b30e4f28ec0a46b78037da4f16e413d473c7",
-    "manifest_path": "/artifacts/lora_run_20260428T165129Z/run_manifest.json",
-    "run_id": "lora_run_20260428T165129Z",
-    "status": "ok",
-    "training_mode": "phase6_scaffold"
-  }
-}
+{
+  "generated_by": "ml/training/modal_lora_train.py",
+  "record_version": "phase6.checkpoint.v1",
+  "result": {
+    "adapter_artifact_ref": "modal-volume://landfillsentry-model-artifacts/lora_run_20260504T181913Z/checkpoint-lora-v1",
+    "artifact_volume": "landfillsentry-model-artifacts",
+    "checkpoint_dir": "data/processed/hf_adapter_trained/lora_run_20260504T181913Z/checkpoint-lora-v1",
+    "local_adapter_dir": "data/processed/hf_adapter_trained/lora_run_20260504T181913Z/checkpoint-lora-v1",
+    "metrics": {
+      "adapter_status": "fine_tuned",
+      "base_model": "LiquidAI/LFM2.5-VL-450M",
+      "base_revision": "main",
+      "completed_steps": 24,
+      "created_at": "2026-05-04T18:19:13.913342+00:00",
+      "dataset_manifest_checksum": "a6738e1af7d89f6fbd0d567c89759f6103beaa81553074a3ad520c6810988b01",
+      "dataset_sample_count": 78,
+      "eval_loss_after": 1.3696070164442062,
+      "eval_loss_before": 2.410613179206848,
+      "eval_loss_delta": 1.041006162762642,
+      "lora_alpha": 16,
+      "lora_dropout": 0.05,
+      "lora_r": 8,
+      "max_steps": 24,
+      "target_modules": [
+        "k_proj",
+        "q_proj",
+        "v_proj"
+      ],
+      "train_loss_mean": 1.8549085905154545,
+      "train_sample_count_used": 49,
+      "validation_sample_count_used": 8
+    },
+    "model_id": "LiquidAI/LFM2.5-VL-450M",
+    "revision": "main",
+    "run_id": "lora_run_20260504T181913Z",
+    "status": "ok",
+    "training_mode": "peft_lora_supervised"
+  }
+}