---
title: Agllm Public
emoji: 🦀
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 4.28.3
app_file: app.py
pinned: false
license: apache-2.0
---

## PestIDBot - Quick Reference

### Environment
```bash
source ~/miniconda3/etc/profile.d/conda.sh && conda activate agllm-june-15
```

### Key Commands
| Task | Command |
|------|---------|
| Build DB | `python app_database_prep.py` |
| Run Eval | `python retrieval_evaluation.py` |
| Run App | `python app.py` |
| Deploy Dev | `git push space3 fresh-start:main` |
| Deploy Prod | `git push space2 fresh-start:main` |

### Git Remotes
- `space2` → `git@hf.co:spaces/arbabarshad/agllm2` (production)
- `space3` → `git@hf.co:spaces/arbabarshad/agllm2-dev` (dev)

### Project Structure
```
├── app.py                      # Main Gradio app (deployed)
├── app_database_prep.py        # Builds ChromaDB from PDFs + Excel
├── retrieval_evaluation.py     # Runs 4-filter evaluation
├── retrieval_evaluation_results.json  # Eval metrics output
│
├── agllm-data/
│   ├── agllm-data-isu-field-insects-all-species/
│   │   ├── *.pdf               # Insect IPM documents
│   │   └── matched_species_results_v2.csv  # Species metadata
│   ├── agllm-data-isu-field-weeds-all-species/
│   │   ├── *.pdf               # Weed IPM documents
│   │   └── matched_species_results_v2.csv  # Species metadata
│   └── PestID Species.xlsx     # India & Africa data (sheets)
│
├── vector-databases-deployed/
│   └── db5-agllm-data-isu-field-insects-all-species/  # ChromaDB output
│
├── species-organized/          # Analysis scripts & outputs
│   ├── species_analysis.py     # Generates paper Figure 3
│   └── species_table.tex       # LaTeX species table
│
└── writing/                    # Paper drafts
```

### Database Build Flow
**US Data (80 species):**
1. PDFs loaded from `agllm-data/.../raw-pdfs/` (content source)
2. `matched_species_results_v2.csv` maps PDF filename → species name (metadata)

**Africa/India Data (35 + 11 species):**
3. Excel `species-organized/PestID Species - Organized.xlsx` provides both content (IPM Info) and metadata

**All Data:**
4. Documents chunked (512 tokens, 10 overlap)
5. Tagged with `matched_specie_X` + `region` metadata
6. Stored in ChromaDB at `vector-databases-deployed/db5-*/`

### Evaluation Filters (retrieval_evaluation.py)
| Filter | P@5 | nDCG@5 |
|--------|-----|--------|
| No Filter | 0.82 | 0.72 |
| Species Only | 0.99 | 0.89 |
| Region Only | 0.83 | 0.73 |
| Species + Region | **1.00** | **0.90** |

---

## Git LFS Troubleshooting Notes

This repository encountered several Git LFS issues during setup. Here's a summary for future reference:

1.  **Missing LFS Objects in History:** Initial pushes failed because the branch history contained references to LFS objects (specifically `a11f8941...` related to `db5/.../data_level0.bin`) that were no longer available locally or on the remote LFS store. Attempts to rewrite history using `git filter-branch` also failed because the rewrite process itself required fetching *other* missing LFS objects.
    *   **Resolution:** We created a clean base branch (`fresh-start`) with no history (`git checkout --orphan fresh-start`), committed a placeholder file, and pushed it forcefully to the remote (`git push -u space3 fresh-start:main --force`). This reset the remote `main` branch.

2.  **Importing State & Untracked Binaries:** We copied the desired file state from the old branch (`git checkout <old-branch> -- .`) into the clean `fresh-start` branch. However, the subsequent push failed because some binary files (e.g., `.png`) were included but weren't tracked by LFS according to the `.gitattributes` file *at that time*.
    *   **Resolution:**
        *   Added the necessary file patterns (e.g., `*.png filter=lfs ...`) to `.gitattributes`.
        *   Crucially, we had to ensure the commit correctly reflected this change. Amending wasn't sufficient. We used:
            ```bash
            # Reset the commit but keep files in working dir
            git reset HEAD~1
            # Re-stage files, forcing re-evaluation based on current .gitattributes
            git add --renormalize .
            # Commit the properly processed files
            git commit -m "Commit message"
            # Force push the corrected commit
            git push --force
            ```

3.  **Ignoring Necessary Directories:** A required directory (`vector-databases-deployed`) was unintentionally ignored via `.gitignore`.
    *   **Resolution:**
        *   Removed the corresponding line from `.gitignore`.
        *   Staged the `.gitignore` file and the previously ignored directory (`git add .gitignore vector-databases-deployed/`).
        *   Committed and pushed the changes.

**Key Takeaways:**

*   Pushing branches with problematic LFS history to a fresh remote can fail. Starting the remote with a clean, history-free branch is a workaround.
*   When adding LFS tracking for existing binary files via `.gitattributes`, ensure the commit correctly converts files to LFS pointers. `git add --renormalize .` after updating `.gitattributes` and *before* committing is often necessary.
*   Double-check `.gitignore` if expected files or directories are missing after a `git add .`.