spatialagent/skill/mapping_validation.md

Tangram Mapping Validation

Validate and troubleshoot Tangram spatial mappings using cross-validation, diagnostic plots, and quality metrics.

Why Validate?

• Ensure mapping is biologically meaningful
• Identify problematic genes or cell types
• Compare different mapping strategies
• Build confidence before downstream analysis

---

Validation Approaches

1. Training score analysis: How well do training genes match?
2. Test gene prediction: Can we predict held-out genes?
3. Cross-validation: Systematic holdout testing
4. Biological validation: Do patterns match known biology?

---

Step 1: Basic Quality Check

After running tangram_map_cells, check training scores:

import scanpy as sc
import tangram as tg

ad_map = sc.read_h5ad("experiments/tangram_mapping.h5ad")

# Training gene scores
train_df = ad_map.uns["train_genes_df"]
print(f"Average training score: {train_df['train_score'].mean():.3f}")
print(f"Median training score: {train_df['train_score'].median():.3f}")

# Score distribution
print("\nScore distribution:")
print(train_df['train_score'].describe())

# Top and bottom genes
print("\nTop 10 genes:")
print(train_df.head(10))
print("\nBottom 10 genes:")
print(train_df.tail(10))

Interpretation:

• Average > 0.7: Good mapping
• Average 0.5-0.7: Moderate, may need tuning
• Average < 0.5: Poor, check data compatibility

---

Step 2: Training Diagnostic Plots

Tool: tangram_evaluate

Or manually generate plots:

import tangram as tg
import matplotlib.pyplot as plt

# 4-panel diagnostic plot
tg.plot_training_scores(ad_map, bins=20, alpha=0.5)
plt.savefig("training_diagnostics.png", dpi=150, bbox_inches="tight")

Panels explained:

1. Score histogram: Distribution of gene scores
2. Score vs SC sparsity: Low sparsity genes should score high
3. Score vs SP sparsity: Sparse spatial genes often score low
4. Score vs sparsity diff: Large diff = dropout mismatch

---

Step 3: Test Gene Evaluation

Project genes and evaluate predictions:

# Project genes
ad_ge = tg.project_genes(ad_map, adata_sc)
ad_ge.write_h5ad("experiments/tangram_projected.h5ad")

# Compare with spatial data
df_genes = tg.compare_spatial_geneexp(ad_ge, adata_sp, adata_sc)

# Separate training and test
train_genes = df_genes[df_genes["is_training"] == True]
test_genes = df_genes[df_genes["is_training"] == False]

print(f"Training genes: {len(train_genes)}, avg score: {train_genes['score'].mean():.3f}")
print(f"Test genes: {len(test_genes)}, avg score: {test_genes['score'].mean():.3f}")

---

Step 4: AUC Evaluation

The AUC metric captures overall prediction quality:

# Compute AUC
metrics, coords = tg.eval_metric(df_genes)
print(f"AUC score: {metrics['auc_score']:.3f}")

# Plot AUC curve
tg.plot_auc(df_genes)
plt.savefig("auc_curve.png", dpi=150)

Interpretation:

• AUC > 0.6: Good prediction on test genes
• AUC 0.4-0.6: Moderate
• AUC < 0.4: Poor, mapping may not generalize

---

Step 5: Cross-Validation (Optional)

For rigorous validation, use leave-one-out or k-fold CV:

# Leave-one-out CV (slow but thorough)
cv_dict = tg.cross_val(
    adata_sc,
    adata_sp,
    cluster_label="cell_type",
    mode="clusters",
    cv_mode="loo",  # or "10fold"
    num_epochs=500,
    device="cuda:0",
    verbose=True,
)

print(f"CV avg train score: {cv_dict['avg_train_score']:.3f}")
print(f"CV avg test score: {cv_dict['avg_test_score']:.3f}")

---

Step 6: Biological Validation

Check that mapped cell types match expected patterns:

import scanpy as sc
import matplotlib.pyplot as plt

adata_sp = sc.read_h5ad("experiments/tangram_annotated.h5ad")

# Known spatial patterns to verify
# Example: T cells should be in lymphoid regions
fig, axes = plt.subplots(2, 3, figsize=(15, 10))

cell_types = ['T cells', 'B cells', 'Macrophages',
              'Epithelial', 'Fibroblasts', 'Endothelial']

for ax, ct in zip(axes.flat, cell_types):
    if ct in adata_sp.obsm["tangram_ct_pred"].columns:
        adata_sp.obs["_tmp"] = adata_sp.obsm["tangram_ct_pred"][ct]
        sc.pl.spatial(adata_sp, color="_tmp", ax=ax, show=False, title=ct)
        del adata_sp.obs["_tmp"]

plt.tight_layout()
plt.savefig("celltype_validation.png", dpi=150)

---

Troubleshooting Low Scores

Problem: Low training scores

Causes:

• Few shared genes between datasets
• Different normalization
• Batch effects

Solutions:

# Check gene overlap
sc_genes = set(adata_sc.var_names.str.lower())
sp_genes = set(adata_sp.var_names.str.lower())
overlap = sc_genes & sp_genes
print(f"Shared genes: {len(overlap)}")

# Ensure same normalization
print(f"SC max: {adata_sc.X.max():.2f}")
print(f"SP max: {adata_sp.X.max():.2f}")

Problem: Good training but poor test scores

Causes:

• Overfitting to training genes
• Training genes not representative

Solutions:

• Use more diverse training genes
• Increase regularization (lambda_r)
• Try mode="clusters" for smoother mapping

Problem: Some cell types poorly mapped

Causes:

• Cell type not in spatial region
• Few marker genes for that type

Solutions:

• Check cell type abundance in spatial data
• Add more markers for underperforming types

---

Comparing Mapping Strategies

Test different approaches:

results = []

# Strategy 1: Cells mode
ad_map1 = tg.map_cells_to_space(adata_sc, adata_sp, mode="cells", num_epochs=500)
score1 = ad_map1.uns["train_genes_df"]["train_score"].mean()
results.append(("cells", score1))

# Strategy 2: Clusters mode
ad_map2 = tg.map_cells_to_space(adata_sc, adata_sp, mode="clusters",
                                 cluster_label="cell_type", num_epochs=500)
score2 = ad_map2.uns["train_genes_df"]["train_score"].mean()
results.append(("clusters", score2))

# Compare
for mode, score in results:
    print(f"{mode}: {score:.3f}")

---

Quality Checklist

Before using mapping results:

• Average training score > 0.7
• AUC score > 0.5
• Top genes include expected markers
• Cell type patterns match known biology
• No obvious artifacts in spatial plots
• Sparsity analysis shows expected dropout pattern